Google bot

[rickymartin]

Is there any other option to prevent crawler from crawling a site or page except robot.txt.

 

[perfectRDP]

How about adding <meta name="googlebot" content="noindex">

 

[Lock Down]

You can also use the .htaccess

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} (googlebot|bingbot|Baiduspider) [NC]
RewriteRule .* - [R=403,L]

 

[MWT]

You can also add:

[COLOR=#881280][FONT=monospace]<meta [/FONT][/COLOR][FONT=monospace]name[/FONT][COLOR=#881280][FONT=monospace]="[/FONT][/COLOR][FONT=monospace]robots[/FONT][COLOR=#881280][FONT=monospace]" [/FONT][/COLOR][FONT=monospace]content[/FONT][COLOR=#881280][FONT=monospace]="[/FONT][/COLOR][FONT=monospace]noindex, nofollow[/FONT][COLOR=#881280][FONT=monospace]" />
[/FONT][/COLOR]

 

[Tiny-LX]

I think there is also a restriction in web master tools also under the crawl section.

 

[floridaguy]

Did you try some options in .htaccess file ? i guess some wayout would be possible there.