I was trying to create a crawler and encountered a file : robots.txt Learn more.
This file is used to supply hints to the robots crawling the web and is available at the root of the website.
Mostly, I'm an experimenter trying out random stuff. My varied interests include technology, web, history, books, travel, movies, behavioral sciences etc.
Disclaimer: All the opinions expressed here are my own and are not of my employer.
Disclaimer: All the opinions expressed here are my own and are not of my employer.
Follow on Tumblr
Following
© 2008–2012 Powered by Tumblr
