Protecting your web site from search engines.

[Artashes]

While most of us, naturally, trying to get into search engines, is there a way to protect a certain URL/website from being crawled by search engine bots? I made a search for it, didn't find plenty of information/solutions on the subject.

Best,

 

[PVT-Jordan]

Definitely. In your robots.txt file, add a line (or a few lines) with the statement "Disallow:"

Here's a good site that shows you how it's structured.
http://www.searchengineworld.com/robots/robots_tutorial.htm

You can even restrict certain pages from certain bots.

This is used all the time to protect scripts, images, and a variety of info that people don't necessarily want so easily available.

 

[Artashes]

That was fast, thanks Jordan.

But would this work for an HTML page? Is there something, possibly, to add to Meta Tags that would disallow crawling?

Thank you.

 

[PVT-Jordan]

No problem. Glad I could help.

The robots file should work for any file type. If you wanted to use a meta tag for each page, you could use this code:

<META NAME="ROBOTS" CONTENT="NOINDEX">

 

[vito]

To elaborate a bit more on Jordan's last post,

<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">

The NOFOLLOW will instruct robots to not follow any links on the page.

Vito

 

[Artashes]

Thanks guys! Exactly what I was looking for.

 

[Exon]

Is Art making a personal movie page?

 

[Artashes]

Nope, just a personal page.

 

[aeromit]

You can also remove meta tags and to put your pages under user/pass restricted subdirectory. One more thing. Do not create any inbound links to your site. But even that may not help you to prevent top ranking in Google :thumbsup:

Changes they made messed up the whole search so don't be suprised if you get ranked when you donot want to

cheers

 

[HMarker]

Artashes, the easiest and most reliable way to protect certain url from search engines is to make those pages password protected so that only authorized people can view it. A simple login page can be coded with asp or php.

 

[CrazyTech]

Artashes, the easiest and most reliable way to protect certain url from search engines is to make those pages password protected so that only authorized people can view it. A simple login page can be coded with asp or php.

I disagree, that can be quite cumbersome to anyone who visits the site. Simply using the meta tag or robots.txt option will do the trick. I use them often and many of the scripts use them too to prevent certain directories from being accessed.

 

[storagedump.com]

Definitely. In your robots.txt file, add a line (or a few lines) with the statement "Disallow:"

Here's a good site that shows you how it's structured.
http://www.searchengineworld.com/robots/robots_tutorial.htm

You can even restrict certain pages from certain bots.

This is used all the time to protect scripts, images, and a variety of info that people don't necessarily want so easily available.

i didnt know that could be done thanks

 

[Artashes]

i didnt know that could be done thanks

Given the age of this thread, its closed.
Please consult the time of a particular thread prior to posting.