So if you didnt know, over the holiday weekend I fell off a 15ft clif and messed up my foot real good. Well as I was sitting on the couch all weekend I wanted to make a script to convert robots.txt files that are on webservers inito a nice little clickable HTML map for reporting and pen-testing. A little bash hacking and I Have a nice little working script. so I present - I think the worlds first robots.txt to HTML page converter.

update: changed the raw code to a file as Im tired of google hits with linux commands

robotReporter.sh (1.73 KB)