Are We Witnessing the End of the Open Web?
For many years, the internet has been hailed as an open ecosystem, a place where people, scholars, and publishers can freely share ideas. However, that foundation is under attack from a new digital weapons competition. Automated bots are taking over websites, harvesting content at industrial scales, and sometimes taking them offline as AI businesses expand their data-hungry systems.
The question is no longer hypothetical: Could AI bots reshape — or even dismantle — the internet as we know it?
The Rise of AI Web Scraping and LLM Training Data
Large language models (LLMs) like ChatGPT depend on enormous amounts of text data to function. To fuel these systems, AI companies deploy sophisticated web scrapers that systematically crawl websites, extracting articles, commentary, and public-interest reporting to train AI models.
Unlike traditional search engine crawlers, many of today’s AI scrapers:
- Ignore website “no scrape” directives
- Target fragile backend infrastructure
- Rotate identities using residential proxy networks
- Mask themselves as normal usersves like an automated extraction pipeline.
Since these bots frequently use anonymous residential IP addresses, it is quite difficult to tell them apart from real visitors. The outcome? Publishers have to protect themselves from traffic that mimics an automated extraction pipeline yet appears human.
This industrial-scale scraping is shifting the economics of the web.
The AI Pincer Movement: Traffic Loss and Infrastructure Strain
As more users turn to AI chatbots instead of visiting websites directly, publishers face a dual crisis:
- Declining traffic and advertising revenue
- Increased server strain from AI scraping bots
AI systems summarize and repackage journalism, reducing the need for users to click through to original sources. At the same time, massive scraping operations consume bandwidth, computing resources, and volunteer labor — especially harming independent and public-interest projects.
For smaller publishers, this is not just inconvenient. It is existential.
The Hidden Cost of Residential Proxy Networks
Residential proxy networks allow scrapers to rotate through real homeowner IP addresses. This makes automated data harvesting appear indistinguishable from organic web browsing.
The implications are significant:
- Websites cannot reliably block malicious scraping
- Defensive infrastructure becomes expensive
- Legitimate users may face increased restrictions
- Public-interest sites bear disproportionate costs
The actors extracting value — AI firms and training-data suppliers — are often insulated from these consequences. Meanwhile, publishers are forced into “permanent defence mode,” fighting a war of attrition they did not choose.
Will the Web Become Gated and Enclosed?
To survive, more websites may require proof of humanity — logins, subscriptions, CAPTCHAs, identity verification systems, and paywalls.
Over time, this defensive posture could accelerate:
- Gated content
- Limited public access
- Restricted participation
- Fragmentation of the open web
Instead of a freely accessible internet, we may see a landscape dominated by controlled ecosystems and walled gardens. Ironically, the tools designed to aggregate knowledge may end up limiting access to it.
AI Chatbots, News Consumption, and Information Integrity
The rise of generative AI is already reshaping how people consume information. Rather than reading multiple sources, users increasingly prompt chatbots for summaries, explanations, and news updates.
While AI agents can personalize and synthesize information efficiently, several concerns remain:
- AI hallucinations and misinformation
- Deceptive outputs
- Propaganda amplification
- Commercial bias and advertising integration
- Reduced exposure to diverse viewpoints
If AI companies prioritize monetization, the model may resemble the trajectory of social media platforms — where engagement and advertising revenue drive design decisions, sometimes at the expense of truth and trust.
As trust online becomes more fragile, users must scrutinize where their information originates and maintain direct relationships with credible publishers.
Surveillance Capitalism in the AI Era
The AI revolution does not replace surveillance capitalism — it deepens it.
When users interact with AI chatbots, they provide:
- Personal queries
- Emotional context
- Behavioral patterns
- Professional data
- Intellectual interests
These conversations become high-value data streams. The more intimate and detailed AI interactions become, the more concentrated power grows in the hands of platform operators.
Some initiatives aim to counterbalance this trend by giving individuals greater control over their data. Proposals such as data unions suggest that citizens could collectively steward their data, using it for mutual benefit rather than surrendering it to profit-driven AI labs.
In a world where data is the chokepoint of power, ownership and governance matter more than ever.
Environmental, Ethical, and Social Risks of the AI Arms Race
Beyond scraping and data extraction, the rapid expansion of AI raises broader systemic risks:
- Energy-intensive data centers and environmental strain
- Exploitation and abuse of AI systems
- “Therapy bots” and psychological manipulation
- Labor displacement and automation anxiety
- Intensifying geopolitical competition
Even industry leaders who express concern are often caught in a competitive race toward artificial superintelligence. This race is both commercial and national-security driven — and there is no clear endpoint.
Without oversight, the AI boom could produce consequences that far outweigh its benefits.
What Can Web Users and Citizens Do?
While AI is here to stay, individuals and communities still have agency. Practical steps include:
- Supporting independent media directly
- Subscribing to trusted newsletters
- Visiting original sources rather than relying solely on AI summaries
- Advocating for regulatory oversight of AI scraping practices
- Demanding transparency in AI training data
- Being mindful of personal data shared with AI systems
Active civic engagement will be essential to prevent the enclosure of the open web.
The Future of the Internet: Collapse or Transformation?
It may be premature to declare the “end of the internet.” But the structure of the web is undeniably shifting.
If current trends continue unchecked:
- Publishers may retreat behind paywalls
- Open access may shrink
- AI intermediaries may dominate knowledge distribution
- Corporate data extraction may intensify
However, thoughtful regulation, ethical AI development, and informed public pressure could redirect this trajectory.
The internet was not built solely by corporations. It was shaped by communities, researchers, journalists, and citizens. Its future will depend on whether those same stakeholders can assert meaningful control in the age of AI.
For now, staying informed, supporting trusted media, and demanding accountability may be the most powerful tools we have to protect the open web.
