here is a fun hack for website robot.txt files.
site:google.com "robots.txt" "disallow" filetype:txt
run that in a search string and you will get back the disallow strings for forced browsing, you can drop the site: modifier to get more data or change it to your target site.