peopletrendin
Active Member
what is meant by robot.txt in seo?
robot.txt?
- Thread starter peopletrendin
- Start date
- Status
4 comments
OneManCrew
Active Member
The robots.txt is a text file you upload to the root of your site which guides and communicates with web crawlers.
With this file you can direct the robot crawler which areas of the website they should not scan or do scan.
Examples:
You can find a great tool to create the robots.txt file on WJunction - WJunction - Webmaster Forum - Wembaster Tools
With this file you can direct the robot crawler which areas of the website they should not scan or do scan.
Examples:
Code:
Allow visit to all files:
User-agent: *
Disallow:
Code:
Disallow the following directories:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/You can find a great tool to create the robots.txt file on WJunction - WJunction - Webmaster Forum - Wembaster Tools
Also you can look at the useful Google guide about robots.txt.
TheSHosting
Member
Just to add, note that rogue bots does not honor robots.txt files. So you will have to use other methods to block them and do not attempt to use robots.txt to block such agents.
Robot.txt File
If you don't want some of urls from your site to craweled by search engines put those pages in robot.txt file
If you don't want some of urls from your site to craweled by search engines put those pages in robot.txt file
- Status