Latest Announcemnent
New thread

robots.txt for your vBulletin forum

Status
Not open for further replies.
D

desiboy

Active Member
1,544
2008
7
0
robots.txt for your vBulletin forum

​
Well many of us might know what this does..but still there are some web masters who don't know about robots.txt and its importance .

Importance of robots.txt in short
When an search engine crawler (spider ) visits your site, it first looks at a file on your site called robots.txt and that file tells the spider, which web pages of your site should be crawled and indexed and which shouldn't be indexed.
Click to expand...
so what you have to do is create a new .txt file and rename it as robots.txt
and paste the code which is included below to it and upload it to your server .

If your forum in the main directory

ex :
Code: 
 http://www.MYFORUM.COM/
Code: 
User-Agent: *
allow: /
Disallow: /cgi-bin/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php
Case 2 : if its in a subdirectory / folder
Ex :
Code: 
http://www.MYSITE.com/forum
Code: 
User-Agent: *
allow: /
Disallow: /cgi-bin/
Disallow: /forum/ajax.php
Disallow: /forum/calendar.php
Disallow: /forum/cron.php
Disallow: /forum/editpost.php
Disallow: /forum/global.php
Disallow: /forum/image.php
Disallow: /forum/inlinemod.php
Disallow: /forum/joinrequests.php
Disallow: /forum/login.php
Disallow: /forum/member.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/moderator.php
Disallow: /forum/newattachment.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/postings.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/profile.php
Disallow: /forum/register.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendmessage.php
Disallow: /forum/showpost.php
Disallow: /forum/showgroups.php
Disallow: /forum/subscription.php
Disallow: /forum/threadrate.php
Disallow: /forum/usercp.php
Disallow: /forum/usernote.php
Also I have found that some web masters add their admincp directory to the robots.txt file which is not at all recommended , because it will tell the Hackers where your admincp is located.

We can even add sitemap to the robots.txt file
Code: 
Sitemap: http://www.YOURFORUM.com/sitemap.xml
Tut By DesiBoy !
Hope it helps

Thanks For Reading !! :))
 
7 comments
Status
Not open for further replies.
Back
Top