Search engines and publishers used to operate on a simple deal: publishers would let the likes of Google trawl their site, and in return Google would give them traffic. With the advent of AI, that deal is all but dead, leaving the internet’s old model under existential threat. Ali Lyon meets Matthew Prince, the man hoping to save it.
When, in 2022, a group of major publishers first approached Matthew Prince about the existential threat that artificial intelligence posed to their industry, the man responsible for protecting a fifth of the internet was understandably sceptical.
After all, ever since the 15th century, when gainfully employed scribes warned that the printing press would mean the extinction of high-quality literature, writers and their bosses have fretted over technological developments spelling the end of their trade.
“I remember – distinctly – rolling my eyes and thinking, ‘God, media companies. They’re such Luddites. They’re always complaining about the next technology,’” recalls Prince, the co-founder of Cloudflare, a data juggernaut that acts the equivalent of a nightclub bouncer for the internet.
“What’s the big problem?”
But, filled with dread that their intellectual property was being scraped by AI firms without permission or providing them with traffic, the publishers were insistent. They demanded Prince and his team trawl through the data. What he discovered worried him.
“It was really stark,” he says. “10 years ago,for every two pages that Google scraped, they would send you one visitor. But over the last decade Google has transitioned from being less of a search engine and more of an answer generator. And what we found was that now, for every 19 pages that Google scrapes, they send you one visitor. So it has become almost 10 times harder to get traffic if you’re generating content today.”
That conversation – and the bleak story revealed by his firm’s deep bank of data – has kindled in Prince a pronounced fear over what the untrammelled rise of AI-based answer engines means not just for the future of publishing – but for the internet itself.
Now, the tech founder is taking it upon himself to try and stop it.
Cloudflare’s Matthew Prince: I feel like Cassandra
The thrust of Prince’s argument has its origins in the very nub of the quid pro quo agreement that has existed at the heart of every iteration and evolution of the internet to date.
Publishers – the deal went – set up websites to produce interesting content in the hope people would read it. They would allow search engines constantly to comb over that content, so that the likes of Google knew exactly what was on their website, enabling it to signpost users to relevant pages without having manually to type in the publisher’s web domain.
For decades, that arrangement worked well. It allowed Google and its competitors to simultaneously improve the quality of the results they provided users and obtain reams of data to package up for advertisers. Meanwhile publishers – including the storied newspaper at which your reporter is fortunate enough to be employed – could monetise the added traffic furnished on them by search engines, either via subscriptions or by selling their own adverts.
Crucially, even as the internet evolved from primarily comprising rudimentary pages of written content, to the advent of social media and smartphones, to the rapid proliferation of video publishing across Youtube and natively on publishers’ sites, the fundamentals of this deal remained the same: the more Google and its competitors felt viewers liked your content, the more it would recommend users to visit it.
Sam Altman’s OpenAI scrapes a site’s content 1,500 times for every one view
But then – in November 2022 – a little-known tech executive called Sam Altman launched a new artificial intelligence-powered answer engine known as ChatGPT that immediately had much of the Western world gripped. It also – in the eyes of Prince and his nervous publishing clients – spelled the end of an arrangement that for decades had acted as the foundations of the internet.
Altman’s new chatbot had managed to transfix the world by giving answers that had scraped a site 1,500 times for every user it sent to publishers’ sites. Anthropic – one of the more popular competitors that has launched in the wake of OpenAI’s launch – operates at a ratio of 40,000 scrapes for every visitor. In short, the AIs took the answers from elsewhere and presented it, cutting out the middle man, or publisher.
This rapid desertion of one of the internet’s founding principles struck Prince as “pathologically unfair”. And over the past three years, he has devoted much of his intellectual capacity and public appearances to warn of the threat it could pose to the news industry, democracy and – in his view – society as a whole.
“I think this is the most interesting question that the world is going to grapple with over the course of the next five years,” he says. “And I feel a little bit like Cassandra [from Greek mythology] yelling, ahead of time, that change is coming, and people don’t quite understand.”
The consequences of this are far more wide-reaching than a few online publishers having to tighten their belts or find new revenue streams. Over an hour-long interview Prince paints a dark vision of where the poor regulation of artificial intelligence could leave developed economies.
Every single consumer request in almost every field being funnelled through one or two AI companies would bestow an enormous amount of power in the hands of a minute number of people. And, he observes, a world where AI agents deal with each and every request that crosses one’s mind doesn’t leave much room for small, independent businesses – or vibrant high streets.
What will Google’s move into AI mean for publishers
His company’s own work in the area reached a crescendo last year, when Cloudflare announced it would automatically block AI crawlers from accessing content on their clients’ sites unless they granted them explicit permission.
Without knowledge of the scale of the business Prince has built, this might not sound like the kind of sea change required to move the needle on a paradigm shift of the scale he describes. But since it was established in 2010, his Silicon Valley-based cloud business has steadily amassed over 221,000 paying customers – protecting and servicing, as its founder phrases it, “a fifth of the internet”. It has a market capitalisation of £78bn, which, were it to be listed in London, would make it the seventh largest company in the FTSE 100, and employs roughly 4,500 staff worldwide.
But having successfully reined in the excesses of all standard AI platforms, Prince has now turned his guns on a new – and even more ambitious – scalp: the mothership of the internet, Google.
Even though according to Cloudflare’s own data, Google refers a higher proportion of internet users to publishers’ sites than other AI platforms, the Alphabet-owned search giant is fast becoming, in Prince’s eyes, “the problem child” of the internet. He says it has a monopoly that means his tool cannot block Google’s AI trawlers without also barring it from search scrapes – the aforementioned arrangement that drives a vast amount of eyeballs to publishers’ sites.
Google said it was driving more traffic than ever to publishers’ sites, despite Matthew Prince’s
How Matthew Prince plans to save the internet
“When it crawls for search, it’s also crawling for the answers [for its AI products],” he says. “And the problem with that is it gives them inherently an unfair advantage over the competition, because you can’t block the crawler for one thing without also blocking it for the other.”
A spokesman for Google said since launching its AI search, it was driving just as many clicks as it had ever done, more of which were classed as “high-quality” than before.
But that hasn’t stopped Prince from issuing a clarion call for publishers, regulators and tech companies to work together on a solution that is likely “part technological, part regulatory”, to ensure Google doesn’t navigate successfully from “one monopoly to another”.
Those solutions could derive from regulators – the UK’s Competition and Markets Authority has, in his eyes, been the most thoughtful of all watchdogs in the developed world. Or it could be legal and tech-based: Cloudflare is currently rolling out a user agreement, which commits Google only to scour through content in a manner that has been explicitly agreed to by the site owner.
Whatever the fix – and while he is wary of the consequences of over-regulating a technology that has the potential to be hugely beneficial – failing to do anything is not an option for the tech founder. Whether Matthew Prince is successful will depend on unwieldy governments becoming sufficiently alive to the scale and pace of change this paradigm shift will have on what is now by far and away the world’s most used technology.
“Even I was myopic at first,” he says. “I was like, ‘This isn’t a problem.’ Then I was like, ‘Oh, this is only a media problem.’ Now I’m starting to see this isn’t just affecting a few industries. It’s going to change basically everything.”