blob: 377e147373a8c13aa372c6ebb1273dea514c2db5 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
# AI Crawler Index — robots.txt user-agent tokens, one per line, 2026-09-07
# https://www.pathwren.workers.dev/data/robots-tokens.txt
AdsBot-Google
AdsBot-Google-Mobile
AdsBot-Google-Mobile-Apps
AhrefsBot
AhrefsSiteAudit
AI2Bot
Ai2Bot-Dolma
aiHitBot
AIWebIndex
Amazonbot
Andibot
Anomura
anthropic-ai
APIs-Google
Applebot
Applebot-Extended
archive.org_bot
atlassian-bot
AwarioRssBot
AwarioSmartBot
Baiduspider
barkrowler
bedrockbot
bingbot
Bytespider
CCBot
ChatGPT-User
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
cohere-ai
cohere-training-data-crawler
Cotoyogi
Crawl4AI
Crawlspace
DataForSeoBot
Diffbot
dotbot
DuckAssistBot
DuckDuckBot
EchoboxBot
ExaSearchBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FeedFetcher-Google
FirecrawlAgent
Google-Agent
Google-CloudVertexBot
Google-CWS
Google-Extended
Google-GeminiNotebook
Google-InspectionTool
Google-Pinpoint
Google-Read-Aloud
Google-Safety
Google-Site-Verification
Googlebot
Googlebot-Image
Googlebot-News
Googlebot-Video
GoogleMessages
GoogleOther
GoogleOther-Image
GoogleOther-Video
GoogleProducer
GPTBot
ia_archiver
ICC-Crawler
ImagesiftBot
img2dataset
ISSCyberRiskCrawler
Kagibot
KlaviyoAIBot
LAIONDownloader
Lightpanda
Linguee Bot
Mediapartners-Google
meta-externalagent
meta-externalfetcher
Meta-WebIndexer
MistralAI-User
MJ12bot
MojeekBot
OAI-SearchBot
omgili
omgilibot
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Pinterestbot
Poseidon Research Crawler
QualifiedBot
QuillBot
Qwantbot
Qwantbot-news
Reflectionbot
rogerbot
SBIntuitionsBot
Scrapy
Screaming Frog SEO Spider
SemrushBot
SemrushBot-BA
SemrushBot-ESI
SemrushBot-FT
SemrushBot-OCOB
SemrushBot-SI
SemrushBot-SWA
SEOkicks
serpstatbot
SeznamBot
ShapBot
Sidetrade indexer bot
SiteAuditBot
Slackbot
Slackbot-LinkExpanding
SplitSignalBot
Storebot-Google
TerraCotta
Thinkbot
TikTokSpider
Timpibot
VelenPublicWebCrawler
Webzio-Extended
wpbot
YaK
YandexAdditional
YandexAdditionalBot
YandexBlogs
YandexBot
YandexCalendar
YandexComBot
YandexDirect
YandexFavicons
YandexImages
YandexMarket
YandexMedia
YandexMetrika
YandexMobileBot
YandexRenderResourcesBot
YandexScreenshotBot
YandexVideo
YandexWebmaster
Yeti
YouBot
|
