1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
OG: https://x[.]com/i/article/1869827152549691392
# Agents or Bots? Making Sense of AI on the Open Web
**Perplexity**
*Jul 19, 2024 · 9 min read*
---
AI is everywhere, but what kind of *everywhere*? A swirl of headlines says “agents” or “bots” are coming for web search, web traffic, and web content. What’s really happening beneath the jargon—and what parts should you care about?
Bots have shaped the dynamics of web traffic for decades, and now **AI-powered agents** are the new ambiguity. Here’s what makes them different (and what means trouble, opportunity, or hype at the boundary between bots and humans).
---
## Terms of Art: Agent vs. Bot
**Bots**:
Used for decades to refer to programmatic agents that crawl or interact with the web. Most common forms: search indexers (Googlebot, Bingbot), scrapers, uptime checkers, “badbots” probing for vulnerabilities.
**Agents**:
The current AI hype cycle has weaponized the term *agent*. Sometimes it means **AI models acting on instructions** (e.g., “tool use,” chain-of-thought, planning). Sometimes it’s broader—assistant, digital avatar, “robot friend.” Sometimes it’s marketing rebranding a bot or RPA workflow as “autonomous agent.”
> “Agent” is a squishy word. In user-facing products, it often means a chatbot wrapper on a LLM with some web access or plug-ins. In crawling and search, it means a more sophisticated (and possibly unpredictable) programmatic visitor.
The nomenclature, while fuzzy, matters: web infrastructure (like robots.txt, bot mitigation, analytics, and adtech) handle “bots” not “agents.” If traffic patterns shift, we need to get definitions right.
---
## Why This Distinction Matters
Web developers, sysadmins, and orgs depend on clear signals about *who/what* is using their content. Search engine bots obey and signal identity in User-Agent. Scrapers and API load testers may evade. Attack traffic can mimic or spoof anything. Traditionally, analytics tries to parse this out, sometimes with commercial bot detection.
**AI agents (as productized via LLMs and copilots):**
- Sometimes act like browsers (simulate user sessions)
- Sometimes act like bots (fetch web content programmatically)
- Sometimes act neither (query knowledge graphs / vector DBs behind walled gardens)
*Principal differences from previous generations of bots*:
- Unannounced, nonlinear fetch patterns
- Deep content digests (summarize, embed, “understand”)
- Potential to synthesize, repackage, and retain (possibly forever)
- Can present themselves as human (headless browser), bot, or something else
This upends web metric models, bot detection, and incentives around content/publication.
---
## A Fork in the Router: Bot and Agent Calls to the Web
A “bot call” to your site is just an HTTP GET (or series). The new wrinkle is: **AI agents can fetch at various levels of abstraction.**
- An LLM agent might:
- Crawl to collect reference content to “feed” an AI answer
- Simulate a user session to “see” web pages as a browser
- Screenshot, embed, and summarize to a remote vector store
- Pull structured data via APIs, scraping, or even browser automata (like Puppeteer)
If you run web infra:
- Traffic that looks legit but isn’t tied to a human session is growing
- Sometimes that traffic obeys robots.txt, sometimes it laughs at it
- Attribution is hard: the major LLMs may introduce “transparency” programs, but 3rd-party synth-agents likely will not
**What part of your content is being ingested, how often, and by whom?** These are the new observability and governance problems.
---
## Opting In/Out and Guardrails: The State of Control
Web infra has bot control conventions like `robots.txt`, bot user-agent signals, and some CAPTCHAs or anti-scraper defenses.
But:
- Modern LLMs (e.g., Perplexity, Gemini, ChatGPT Browsing) **may** obey `robots.txt`, or at least offer opt-outs. Open-source or rogue agents likely won’t.
- User-agent strings can be spoofed or obfuscated.
- AI agents can synthesize content **without directly visiting** (training on caches, search snippets, prior summaries).
- Preventing *all* unwanted non-human access = near-impossible on open web.
> Example: The classic “Perplexity bot” made headlines for allegedly grabbing paywalled or robots.txt-blocked content via indirect routes (API partners, cached snippets, etc.), not always classic aggressive crawling. Is it a bot? An agent? A middleman? The terminology—and enforcement—are murky.
Filtering, rate-limiting, legal enforcement, and even technical guarantees are limited. The open web is just that: open. And even with detection, what you do about it is up for grabs.
---
## The Near Future: Where Humans & AI Mix
- As **search shifts to AI answers**, human-attributable traffic drops, and synthesized content rises. Analytics, ad-tech, and tracking all get murkier.
- Many “agents” are just bespoke, private, or corporate deployments of public LLMs. If they reuse the same crawler infra as bots, the distinctions only matter for legal/perm/policy debates.
- Expect a short-term arms race between web operators (who want protection, attribution, revenue) and agent developers (who want open data, capability, scale).
- A new layer of “AI agent mitigation” tooling will rise, likely resembling but diverging from classic WAF/bot management.
*AI succeeded at being “everywhere” on the web. The next battleground is over “who/what” is everywhere on the web—and the ability to tell the difference is now a real competitive advantage.*
---
© 2024 Perplexity. See more at perplexity.ai.
|
