https://rud.is/b/2017/11/03/i-for-one-welcome-our-forthcoming-new-robots-txt-overlords/