AI companies quietly source printed books for model training

When an email requesting quotes for more than 3,000 books landed in Pieter de Vries's inbox in Haarlem, the antiquarian bookseller assumed it was spam.

The sender, identifying herself as Natalia from a company called 2077AI, had attached a spreadsheet of ISBNs and asked for shipping estimates to China. De Vries ignored it, until a journalist investigating the request later told him it was connected to artificial intelligence.

Why are AI labs turning to physical books?

This spreadsheet featured 3,001 books, largely from 2020 to 2021, and was published by academic publishers such as Elsevier, Wiley, Routledge and Oxford University Press. According to de Vries, there were several other Dutch antiquarian booksellers that got the same request and treated it similarly.

The AI firms are running out of good content online, and hence, some are starting to use real books as an alternative.

The clearest precedent came from Anthropic, whose court records showed had purchased millions of physical books, cut their spines and scanned the pages in a programme internally called Project Panama, before discarding the originals.

A federal judge ruled that training AI models on legally purchased books constituted fair use, while separate claims tied to books downloaded from the LibGen and PiLiMi online libraries were resolved in a settlement.

An Anthropic spokesperson said its data comes from public web sources, commercial datasets and internal data and that the company does not acquire or destroy rare or antiquarian books.

What wasn’t clear from the court filing is how businesses manage to source their vast number of books in the first place. 

Earlier this year, the media outlet 404 Media reported on how the firm ISBNdb, which specialises in maintaining book metadata, had offered bulk purchasing services to AI labs but had quietly discontinued the service, stating to the publication Fortune that it had never been a live business venture. 

Publishers in Germany and Switzerland have spoken about similar bulk orders, whereas Zoom Books of Canada has denied any such ties to AI training programmes to Swiss broadcaster SRF.