The dirty data problem shaping AI model quality
From corrupted OCR scans in training corpora to ballooning memory costs during distillation, the quality of data going into AI models is quietly determining which tools actually work.
Most conversations about AI capability focus on model size, benchmark scores, or the latest architectural trick. Far less attention goes to what the models actually eat during training, and whether that food is any good. Two recent developments from Hugging Face and EleutherAI pull that curtain back, revealing that the data pipeline, not the model design, may be where the real quality battles are being fought.
The problem is not abstract. A language model trained on OCR-corrupted text learns at roughly 30 percent the efficiency of one trained on clean human transcriptions, according to the Talkie project cited by the FineBooks team. Separately, compressing large models through knowledge distillation, a process now standard across the industry, lives or dies on whether the training signal fed to the student model is accurate and affordable to produce. Both stories point at the same underlying tension: the data layer is fragile, and the industry has been largely papering over it.
FineBooks benchmarks 14 OCR models on 2,000+ historical pages
The FineBooks project, a collaboration between Hugging Face and EleutherAI, ran 14 open-weight OCR models across 2,165 pages of historical books and published the results as a public leaderboard. The target is the Biodiversity Heritage Library, which holds more than 64 million pages of digitised natural history documents. When EleutherAI released Common Pile last year, the largest openly licensed training corpus to date, it included around 300,000 public-domain books pulled from older OCR runs. The FineBooks authors argue that reprocessing those books with modern models is one of the highest-leverage improvements available to open AI development. The best model tested, dots.mocr at 3 billion parameters, hits a character error rate low enough for training purposes at under two dollars per thousand pages. Crucially, model size does not predict quality here: the 9-billion parameter Qwen3.5-9B scores lower than dots.mocr, while OvisOCR2 places second using only 0.9 billion parameters at 46 cents per thousand pages.
Why Common Pile and open corpora carry a hidden quality debt
The ground-truth dataset used to score FineBooks models comes from expert transcriptions done between 2011 and 2012 under the IMPACT and BHL-Europe projects, with an error rate of roughly one character per 2,000. That baseline reveals just how far older OCR pipelines have fallen short. The team plans to reprocess around 200,000 BHL documents with a top-performing model and release the resulting text as an open dataset. The evaluation framework itself is openly available, and new models are added to the leaderboard on a rolling basis. One caveat worth noting for anyone building on this work: the benchmark currently covers only Antiqua typefaces in English, French, German, and Latin. Fraktur, non-Latin scripts, and handwriting are not yet included, so the quality gains do not transfer automatically to all historical collections.
Knowledge distillation at Multiverse Computing hits a VRAM wall
On the model compression side, Multiverse Computing published research through Hugging Face on making knowledge distillation cheap enough to run without a fleet of GPUs. Distillation, where a smaller student model is trained to replicate the output distribution of a larger teacher, has become the standard route to deploying capable but affordable models. The problem is memory. Using a model like gpt-oss-120b, which has a vocabulary of over 200,000 tokens, the tensor holding the teacher's probability distribution across a sequence of 32,000 tokens at a batch size of four reaches around 50 gigabytes on its own. Add gradients, activations, and optimizer states, and a single training iteration can peak above 250 gigabytes of VRAM, more than an H200 can provide. The Multiverse Computing paper addresses this with two changes: caching the teacher's top-100 logits per token position so the teacher never needs to sit in memory alongside the student, and a fused chunked KL-divergence loss that processes data in pieces rather than materialising the full vocabulary-by-sequence matrix at once. The result keeps peak VRAM below 128 gigabytes, making long-context distillation possible on a single GPU.
What this means for teams choosing or building AI tools
For anyone evaluating AI tools or planning their own fine-tuning work, these two stories carry a practical message. Open training corpora carry quality debt that is only now being systematically measured and addressed. If a model was trained on Common Pile or similar public-domain book collections without reprocessing, its effective exposure to that content may be substantially lower than the raw page count suggests. At the same time, the distillation techniques that underpin many efficient commercial and open models are sensitive to how well the training signal is preserved under memory constraints. Cutting corners on either the input data or the compression pipeline compounds into capability gaps that benchmark numbers rarely surface directly. The next differentiation between AI tools may not come from who has the biggest model, but from who has cleaned up their data pipeline most thoroughly.
