Robots Standard Exclusion

Zagor

Zagor

New member
Do you use robots.txt file on your web site? I don't have much experience with this. My question is what files and locations should I forbid and to what robots specifically, if any?
 
You just need to develop your site & your content should be fresh & unique. Then you need to submit your site to major search engine so that Google crawler can crawl your site.
 
For faster indexing of your site and effective search, you can bookmark your site to certain good social site and submit it in the several directories for getting backlinks. If the sites you submit to are dofollow sites , you invite Robots to crawl your site. as many time your pages are crawled by the bots, it get indexed by the google
 
Thanks, but yet I asked and no one seem to answered about what SPECIFIC files and locations is common to forbid for search engine robot crawling?
 
Do you have something you want to hide from search engines? If so, you should consider password protecting the information and using the noindex meta tag in addition to a robots.txt file.

Otherwise, there's no reason to use a robots.txt file.
 
The reason why I asked this is because people are always telling me that certain files are of no use at all for search engines, like JavaScript, CSS, flash, XML. So why not forbid them from the start and avoid their indexing?
 
So why not forbid them from the start and avoid their indexing?
Click to expand...
Did you read an article that told you to do this?

I'm pretty sure Google/Bing will be able to understand what it finds on your site and index it (or not) appropriately.

Again, there's no reason to worry about this stuff--just focus on building your website.
 
Yes I read about it more than once. And for the record I never worried about this before, I was just asking for your opinions.
 
You must log in or register to reply here.

Supporters

Back
Top