Benefits of Robots.txt File

A robots.txt file might contain rules like "Disallow: /admin/" (preventing crawlers from accessing the admin area) or "Allow: /" (allowing crawlers to access the entire site). You can protect certain aspects of your site from being crawled. What are your goals in finding out more on this?
 
A robots.txt file might contain rules like "Disallow: /admin/" (preventing crawlers from accessing the admin area) or "Allow: /" (allowing crawlers to access the entire site). You can protect certain aspects of your site from being crawled. What are your goals in finding out more on this?
Dumb question, what prevents a human from pulling up the file and finding your admin section? I haven't used robots.txt at all, if I have - it's been over 10-15 years ago.
 
Back
Top