In my last article I introduced the idea of having robots.txt be a dynamic document, rather than a static text file. The reason for this was to allow it to respond in real-time to the particulars of the robot that was making the request.
Full story »
