add: pplx response

1 files changed, 95 insertions, 0 deletions

diff --git a/2025/2025-08-05-pplx-response-to-cf.md b/2025/2025-08-05-pplx-response-to-cf.md
new file mode 100644
index 0000000..04ba58e
--- /dev/null
+++ b/2025/2025-08-05-pplx-response-to-cf.md
@@ -0,0 +1,95 @@
+OG: https://x[.]com/i/article/1869827152549691392
+
+# Agents or Bots? Making Sense of AI on the Open Web
+
+**Perplexity**
+*Jul 19, 2024 · 9 min read*
+
+---
+
+AI is everywhere, but what kind of *everywhere*? A swirl of headlines says “agents” or “bots” are coming for web search, web traffic, and web content. What’s really happening beneath the jargon—and what parts should you care about?
+
+Bots have shaped the dynamics of web traffic for decades, and now **AI-powered agents** are the new ambiguity. Here’s what makes them different (and what means trouble, opportunity, or hype at the boundary between bots and humans).
+
+---
+
+## Terms of Art: Agent vs. Bot
+
+**Bots**:  
+Used for decades to refer to programmatic agents that crawl or interact with the web. Most common forms: search indexers (Googlebot, Bingbot), scrapers, uptime checkers, “badbots” probing for vulnerabilities.
+
+**Agents**:  
+The current AI hype cycle has weaponized the term *agent*. Sometimes it means **AI models acting on instructions** (e.g., “tool use,” chain-of-thought, planning). Sometimes it’s broader—assistant, digital avatar, “robot friend.” Sometimes it’s marketing rebranding a bot or RPA workflow as “autonomous agent.”
+
+ > “Agent” is a squishy word. In user-facing products, it often means a chatbot wrapper on a LLM with some web access or plug-ins. In crawling and search, it means a more sophisticated (and possibly unpredictable) programmatic visitor.
+
+The nomenclature, while fuzzy, matters: web infrastructure (like robots.txt, bot mitigation, analytics, and adtech) handle “bots” not “agents.” If traffic patterns shift, we need to get definitions right.
+
+---
+
+## Why This Distinction Matters
+
+Web developers, sysadmins, and orgs depend on clear signals about *who/what* is using their content. Search engine bots obey and signal identity in User-Agent. Scrapers and API load testers may evade. Attack traffic can mimic or spoof anything. Traditionally, analytics tries to parse this out, sometimes with commercial bot detection.
+
+**AI agents (as productized via LLMs and copilots):**
+- Sometimes act like browsers (simulate user sessions)
+- Sometimes act like bots (fetch web content programmatically)
+- Sometimes act neither (query knowledge graphs / vector DBs behind walled gardens)
+
+*Principal differences from previous generations of bots*:
+- Unannounced, nonlinear fetch patterns
+- Deep content digests (summarize, embed, “understand”)
+- Potential to synthesize, repackage, and retain (possibly forever)
+- Can present themselves as human (headless browser), bot, or something else
+
+This upends web metric models, bot detection, and incentives around content/publication.
+
+---
+
+## A Fork in the Router: Bot and Agent Calls to the Web
+
+A “bot call” to your site is just an HTTP GET (or series). The new wrinkle is: **AI agents can fetch at various levels of abstraction.**
+
+- An LLM agent might:
+    - Crawl to collect reference content to “feed” an AI answer
+    - Simulate a user session to “see” web pages as a browser
+    - Screenshot, embed, and summarize to a remote vector store
+    - Pull structured data via APIs, scraping, or even browser automata (like Puppeteer)
+
+If you run web infra:
+- Traffic that looks legit but isn’t tied to a human session is growing
+- Sometimes that traffic obeys robots.txt, sometimes it laughs at it
+- Attribution is hard: the major LLMs may introduce “transparency” programs, but 3rd-party synth-agents likely will not
+
+**What part of your content is being ingested, how often, and by whom?** These are the new observability and governance problems.
+
+---
+
+## Opting In/Out and Guardrails: The State of Control
+
+Web infra has bot control conventions like `robots.txt`, bot user-agent signals, and some CAPTCHAs or anti-scraper defenses.
+
+But:
+- Modern LLMs (e.g., Perplexity, Gemini, ChatGPT Browsing) **may** obey `robots.txt`, or at least offer opt-outs. Open-source or rogue agents likely won’t.
+- User-agent strings can be spoofed or obfuscated.
+- AI agents can synthesize content **without directly visiting** (training on caches, search snippets, prior summaries).
+- Preventing *all* unwanted non-human access = near-impossible on open web.
+
+ > Example: The classic “Perplexity bot” made headlines for allegedly grabbing paywalled or robots.txt-blocked content via indirect routes (API partners, cached snippets, etc.), not always classic aggressive crawling. Is it a bot? An agent? A middleman? The terminology—and enforcement—are murky.
+
+Filtering, rate-limiting, legal enforcement, and even technical guarantees are limited. The open web is just that: open. And even with detection, what you do about it is up for grabs.
+
+---
+
+## The Near Future: Where Humans & AI Mix
+
+- As **search shifts to AI answers**, human-attributable traffic drops, and synthesized content rises. Analytics, ad-tech, and tracking all get murkier.
+- Many “agents” are just bespoke, private, or corporate deployments of public LLMs. If they reuse the same crawler infra as bots, the distinctions only matter for legal/perm/policy debates.
+- Expect a short-term arms race between web operators (who want protection, attribution, revenue) and agent developers (who want open data, capability, scale).
+- A new layer of “AI agent mitigation” tooling will rise, likely resembling but diverging from classic WAF/bot management.
+
+*AI succeeded at being “everywhere” on the web. The next battleground is over “who/what” is everywhere on the web—and the ability to tell the difference is now a real competitive advantage.*
+
+---
+
+© 2024 Perplexity. See more at perplexity.ai.