Tag:robots exclusion standard
All the articles with the tag "robots exclusion standard".
I, For One, Welcome Our Forthcoming New robots.txt Overlords
Despite my week-long Twitter consumption sabbatical (helped --- in part --- by the nigh week-long internet and power outage here in Maine), I still catch useful snippets from folks. My cow-orker @dabdine shunted a tweet by @terrencehart…