update: pplx

1 files changed, 27 insertions, 64 deletions

diff --git a/2025/2025-08-05-pplx-response-to-cf.md b/2025/2025-08-05-pplx-response-to-cf.md
index 04ba58e..f2b101d 100644
--- a/2025/2025-08-05-pplx-response-to-cf.md
+++ b/2025/2025-08-05-pplx-response-to-cf.md
@@ -7,89 +7,52 @@ OG: https://x[.]com/i/article/1869827152549691392
 
 ---
 
-AI is everywhere, but what kind of *everywhere*? A swirl of headlines says “agents” or “bots” are coming for web search, web traffic, and web content. What’s really happening beneath the jargon—and what parts should you care about?
+As the Internet evolves, so too do the ways in which we access and interact with information. In the earliest days of the web, automated bots played a simple, well-understood role: indexing sites for search, checking links, or scraping data according to clear rules set by website owners. 
 
-Bots have shaped the dynamics of web traffic for decades, and now **AI-powered agents** are the new ambiguity. Here’s what makes them different (and what means trouble, opportunity, or hype at the boundary between bots and humans).
+But with the rise of AI-powered assistants and user-driven agents, the boundary between what counts as "just a bot" and what serves the immediate needs of real people has become increasingly blurred. 
 
----
-
-## Terms of Art: Agent vs. Bot
-
-**Bots**:  
-Used for decades to refer to programmatic agents that crawl or interact with the web. Most common forms: search indexers (Googlebot, Bingbot), scrapers, uptime checkers, “badbots” probing for vulnerabilities.
-
-**Agents**:  
-The current AI hype cycle has weaponized the term *agent*. Sometimes it means **AI models acting on instructions** (e.g., “tool use,” chain-of-thought, planning). Sometimes it’s broader—assistant, digital avatar, “robot friend.” Sometimes it’s marketing rebranding a bot or RPA workflow as “autonomous agent.”
+## The Rise of Digital Assistants 
 
- > “Agent” is a squishy word. In user-facing products, it often means a chatbot wrapper on a LLM with some web access or plug-ins. In crawling and search, it means a more sophisticated (and possibly unpredictable) programmatic visitor.
+Modern Al assistants work fundamentally differently from traditional web crawling. When you ask Perplexity a question that requires current information—say, "What are the latest reviews for that new restaurant?"—the Al doesn't already have that information sitting in a database somewhere. Instead, it goes to the relevant websites, reads the content, and brings back a summary tailored to your specific question. 
+This is fundamentally different from traditional web crawling, in which crawlers systematically visit millions of pages to build massive databases, whether anyone asked for that specific information or not. User-driven agents, by contrast, only fetch content when a real person requests something specific, and they use that content immediately to answer the user's question. Perplexity's user-driven agents do not store the information or train with it. 
 
-The nomenclature, while fuzzy, matters: web infrastructure (like robots.txt, bot mitigation, analytics, and adtech) handle “bots” not “agents.” If traffic patterns shift, we need to get definitions right.
+## Why This Distinction Matters 
 
----
-
-## Why This Distinction Matters
+The difference between automated crawling and user-driven fetching isn't just technical—it's about who gets to access information on the open web. When Google's search engine crawls to build its index, that's different from when it fetches a webpage because you asked for a preview. Google's "user-triggered fetchers" prioritize your experience over robots.txt restrictions because these requests happen on your behalf. 
 
-Web developers, sysadmins, and orgs depend on clear signals about *who/what* is using their content. Search engine bots obey and signal identity in User-Agent. Scrapers and API load testers may evade. Attack traffic can mimic or spoof anything. Traditionally, analytics tries to parse this out, sometimes with commercial bot detection.
+The same applies to Al assistants. When Perplexity fetches a webpage, it's because you asked a specific question requiring current information. The content isn't stored for training—it's used immediately to answer your question. 
+When companies like Cloudflare mischaracterize user-driven Al assistants as malicious bots, they're arguing that any automated tool serving users should be suspect—a position that would criminalize email clients and web browsers, or any other service a would-be gatekeeper decided they don't like. 
 
-**AI agents (as productized via LLMs and copilots):**
-- Sometimes act like browsers (simulate user sessions)
-- Sometimes act like bots (fetch web content programmatically)
-- Sometimes act neither (query knowledge graphs / vector DBs behind walled gardens)
+This controversy reveals that Cloudflare's systems are fundamentally inadequate for distinguishing between legitimate Al assistants and actual threats. If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic. 
+This overblocking hurts everyone. Consider someone using Al to research medical conditions, compare product reviews, or access news from multiple sources. If their assistant gets blocked as a "malicious bot," they lose access to valuable information. 
 
-*Principal differences from previous generations of bots*:
-- Unannounced, nonlinear fetch patterns
-- Deep content digests (summarize, embed, “understand”)
-- Potential to synthesize, repackage, and retain (possibly forever)
-- Can present themselves as human (headless browser), bot, or something else
+The result is a two-tiered internet where your access depends not on your needs, but on whether your chosen tools have been blessed by infrastructure controllers, who will care more about your means. This undermines user choice and threatens the open web's accessibility for innovative services competing with established giants. 
 
-This upends web metric models, bot detection, and incentives around content/publication.
+## A Call for Clarity: How User Agents Actually Work 
 
----
+An Al assistant works just like a human assistant. When you ask an Al assistant a question that requires current information, they don't already know the answer. They look it up for you in order to complete whatever task you've asked. 
 
-## A Fork in the Router: Bot and Agent Calls to the Web
+On Perplexity and all other agentic Al platforms, this happens in real-time, in response to your request, and the information is used immediately to answer your question. It's not stored in massive databases for future use, and it's not used to train Al models. 
 
-A “bot call” to your site is just an HTTP GET (or series). The new wrinkle is: **AI agents can fetch at various levels of abstraction.**
+User-driven agents only act when users make specific requests, and they only fetch the content needed to fulfill those requests. This is the fundamental difference between a user agent and a bot. 
 
-- An LLM agent might:
-    - Crawl to collect reference content to “feed” an AI answer
-    - Simulate a user session to “see” web pages as a browser
-    - Screenshot, embed, and summarize to a remote vector store
-    - Pull structured data via APIs, scraping, or even browser automata (like Puppeteer)
+## Directly Addressing Cloudflare: A Question of Competence 
 
-If you run web infra:
-- Traffic that looks legit but isn’t tied to a human session is growing
-- Sometimes that traffic obeys robots.txt, sometimes it laughs at it
-- Attribution is hard: the major LLMs may introduce “transparency” programs, but 3rd-party synth-agents likely will not
+Cloudflare's recent blog post managed to get almost everything wrong about how modern Al assistants actually work. 
+In addition to misunderstanding 20-25M user agent requests are not scrapers, Cloudflare claimed that Perplexity was engaging in "stealth crawling," using hidden bots and impersonation tactics to bypass website restrictions. But the technical facts tell a different story. 
 
-**What part of your content is being ingested, how often, and by whom?** These are the new observability and governance problems.
+It appears Cloudflare confused Perplexity with 3-6M daily requests of unrelated traffic from BrowserBase, a third-party cloud browser service that Perplexity only occasionally uses for highly specialized tasks (less than 45,000 daily requests). 
 
----
+Because Cloudflare has conveniently obfuscated their methodology and declined to answer questions helping our teams understand, we can only narrow this down to two possible explanations. 
 
-## Opting In/Out and Guardrails: The State of Control
+1. Cloudflare needed a clever publicity moment and we-their own customer-happened to be a useful name to get them one. 
 
-Web infra has bot control conventions like `robots.txt`, bot user-agent signals, and some CAPTCHAs or anti-scraper defenses.
+2. Cloudflare fundamentally misattributed 3-6M daily requests from BrowserBase's automated browser service to Perplexity, a basic traffic analysis failure that's particularly embarrassing for a company whose core business is understanding and categorizing web traffic. 
 
-But:
-- Modern LLMs (e.g., Perplexity, Gemini, ChatGPT Browsing) **may** obey `robots.txt`, or at least offer opt-outs. Open-source or rogue agents likely won’t.
-- User-agent strings can be spoofed or obfuscated.
-- AI agents can synthesize content **without directly visiting** (training on caches, search snippets, prior summaries).
-- Preventing *all* unwanted non-human access = near-impossible on open web.
+Whichever explanation is the truth, the technical errors in Cloudflare's analysis aren't just embarrassing—they're disqualifying. When you misattribute millions of requests, publish completely inaccurate technical diagrams, and demonstrate a fundamental misunderstanding of how modern Al assistants work, you've forfeited any claim to expertise in this space. 
 
- > Example: The classic “Perplexity bot” made headlines for allegedly grabbing paywalled or robots.txt-blocked content via indirect routes (API partners, cached snippets, etc.), not always classic aggressive crawling. Is it a bot? An agent? A middleman? The terminology—and enforcement—are murky.
+This controversy reveals that Cloudflare's systems are fundamentally inadequate for distinguishing between legitimate Al assistants and actual threats. If you can't tell a helpful digital assistant from a malicious scraper, then you probably shouldn't be making decisions about what constitutes legitimate web traffic. 
 
-Filtering, rate-limiting, legal enforcement, and even technical guarantees are limited. The open web is just that: open. And even with detection, what you do about it is up for grabs.
-
----
-
-## The Near Future: Where Humans & AI Mix
-
-- As **search shifts to AI answers**, human-attributable traffic drops, and synthesized content rises. Analytics, ad-tech, and tracking all get murkier.
-- Many “agents” are just bespoke, private, or corporate deployments of public LLMs. If they reuse the same crawler infra as bots, the distinctions only matter for legal/perm/policy debates.
-- Expect a short-term arms race between web operators (who want protection, attribution, revenue) and agent developers (who want open data, capability, scale).
-- A new layer of “AI agent mitigation” tooling will rise, likely resembling but diverging from classic WAF/bot management.
-
-*AI succeeded at being “everywhere” on the web. The next battleground is over “who/what” is everywhere on the web—and the ability to tell the difference is now a real competitive advantage.*
-
----
+The bluster around this issue also reveals that Cloudflare's leadership is either dangerously misinformed on the basics of Al, or simply more flair than cloud. This matters because Cloudflare's customers include businesses of all types, companies who can't afford to trust their infrastructure with charlatan publicity stunts. 
 
-© 2024 Perplexity. See more at perplexity.ai.
+Even more embarrassing, Cloudflare published a technical diagram supposedly showing "Perplexity's crawling workflow" that bears no resemblance to how Perplexity actually works. If Cloudfare were truly interested in understanding the data they were seeing, how our systems work, or these fundamental concepts outlined above, they could have done what we encourage all Perplexity users to do. Just ask.