Generative AI (GenAI) relies on training Large Language Models (LLMs) with huge volumes of data. To do this, scrapers ongoingly sift through the Internet, automating the extraction of information that train LLMs. They scour the web across different domains – news sites, forums, and a lot more – to provide coherent responses when prompted.
GenAI Intercepting Web Traffic
For decades, businesses have mostly embraced crawlers and web scrapers from search engines. Who wouldn’t want their content to rise to the top of search engine results? But the dynamics have shifted with the introduction of LLMs. Instead of helping direct users from search engine results to websites, output from GenAI prompts often intercepts web traffic.
For example, according to a recent WSJ study, publishers have determined they’ll lose 20% to 40% of Google-generated traffic due to AI. Additionally, there are potential legal implications as LLMs often intake content and then disseminate it in their responses without proper payment, attribution, or licensing agreements to those who created it.
The publishing industry has been the first to openly speak out about the impact and legality of unauthorized scraping to train GenAI. Recently, Time Magazine teamed-up with OpenAI with a multiyear content licensing deal. Similarly, Reddit is trying to stop AI providers from scraping its forums without paying-up first after striking deals with OpenAI and Google.
