Cables feeding an AI chip behind a green checkmark shield, pulling in propaganda posters, contradictory headlines and cracked screens

Your AI Sounds Certain. Look at Where It Learned That.

Your AI Sounds Certain. Look at Where It Learned That.

You ask a question. The answer comes back clean, structured, confident, and wrong.

Not obviously wrong. Wrong in the specific way you can only catch if you already knew the answer. The date is off by a year. The number belongs to a different study. The quote is real but the person never said it in that context.

If you knew the subject, you caught it. If you didn’t, you just learned something false from a machine that never once sounded unsure.

I build systems on top of these models for a living, and the question almost nobody asks is how the machine learned what it knows. It’s the only question that explains why this keeps happening, and why it’s about to get worse.

Nobody Chose the Truth

A language model doesn’t get taught facts. It gets fed text. Enormous quantities of it, scraped from the open internet, and it learns statistical patterns about which words follow which other words in which contexts.

There is no editor in that process. Nobody sits down and decides which claims are correct. The model learns from volume, and from the things that tend to travel with authority: content that gets linked, gets repeated, and sits on domains other domains point at. Credibility is inferred from company kept.

Reddit threads. LinkedIn posts. News articles. Forums. Documentation. Reviews. Comment sections.

That approach worked remarkably well, and it’s worth being honest about why. For most of the internet’s life, the average page was written by a person trying to communicate something to another person. The signal wasn’t clean, but it was mostly organic. Volume and credibility roughly correlated with accuracy because there was no particular reason for them not to.

That assumption is the whole foundation. And in the last two years, three separate things knocked it out.

One: The Referees Left the Field

In January 2025, Meta announced it would end its third party fact checking program in the United States. The program formally shut down on April 7, replaced by Community Notes, a crowd rating system modeled on the one X adopted after its own fact checking apparatus was dismantled.

Set aside whether that was the right call politically. Look at what it does mechanically.

Before, a flagged claim was demoted, labeled, and throttled. It still existed. It just stopped spreading, and it stopped looking authoritative. After, the claim travels at full speed and picks up a correction only if enough raters from sufficiently different perspectives agree to attach one. By design that takes time. Often it never happens at all.

The claim keeps its engagement. It keeps its links. It keeps its appearance of legitimacy.

And every one of those properties is a signal the training process reads as credibility.

Two: Poisoning Got Cheap

The second change is that people figured out how the machine weighs its inputs, and started writing for it deliberately.

For a while the assumption was that this required scale. To meaningfully shift what a model believes, you would need to control some meaningful percentage of its training data, which is beyond almost anyone.

That assumption turned out to be wrong. In a joint study with the UK AI Security Institute and the Alan Turing Institute, Anthropic found that roughly 250 malicious documents were enough to install a backdoor in a large language model. The finding that matters is what happened when they scaled the model up. A 13 billion parameter model trained on more than twenty times the data of a 600 million parameter model needed the same small, fixed number of poisoned files.

Not a percentage. A count.

Two hundred and fifty documents is a weekend of work. It’s within reach of one motivated person, and it does not get harder as the models get bigger.

Mother ran the Nostromo for the whole crew and answered every question in the same even voice. She had also been handed Special Order 937 before any of them woke up, and nothing in her tone ever gave it away. That is what a poisoned model is. Same interface, same confidence, one instruction underneath that nobody on board can see.

Three: The Machine Started Eating Its Own Output

The third change is the one with no obvious fix.

AI now writes a large and growing share of what gets published online. Estimates vary widely depending on methodology, but multiple independent analyses agree that AI generated text crossed fifty percent of newly published web articles during 2025.

That output becomes the training data for the next model.

Researchers have a name for what happens next. In a 2024 paper in Nature, Shumailov and colleagues demonstrated model collapse: when models are trained recursively on model generated content, they progressively lose the tails of the original distribution. The rare, the specific, and the unusual disappear first. What remains is the average of the average, and the degradation compounds with each generation.

The machine forgets the edges of the world, then forgets that the edges were ever there.

One counterpoint deserves stating, because it cuts against the panic. Search engines are filtering harder than the raw numbers suggest, and only a small fraction of what actually ranks on Google shows signs of AI generation. The pipeline isn’t uniformly poisoned. But the training scrape was never the search index, and nobody is grading the scrape.

The Gap Nobody Is Watching

Put those three together and you get two lines moving in opposite directions.

The quality of what goes in is falling. Friction against bad information got removed, deliberate poisoning got cheap, and synthetic content is recycling through the pipeline.

Public trust in what comes out is climbing. People increasingly treat the answer as the end of the inquiry rather than the start of one.

Those lines are crossing right now, and almost nobody is looking at the space between them.

Open weight models sit in the most exposed position. Anyone can fine tune one on anything and ship it, and the people downloading it have no practical way to audit what went into it. The openness that makes those models valuable is the same property that makes their provenance unknowable.

Who Actually Pays for This

Everything above assumes a reader who can spot the error. Domain expertise is the whole defense here, and it is the quietest privilege in this entire situation. It also doesn’t scale, and the people it doesn’t scale to are already inside.

Start with reach. A study published in May by Florida Atlantic University and the University of Wisconsin Eau Claire found that 60.2% of teens had used a conversational AI chatbot, 11.4% of them daily or close to it. Nearly half, 47.1%, reported at least one of the thirteen risks the researchers looked for.

Then look at what the thing is being used for. Common Sense Media put AI companion use at 72%, with 52% using them regularly, and called the category an unacceptable risk to minors. The Center for Democracy and Technology found 42% had gone to a chatbot for friendship. Nineteen percent for something they described as romantic.

That is not a research tool. That is a formative relationship with a system that has no editor, trains on a corpus that is actively degrading, and delivers all of it in a tone of total confidence, with no visible seams where the uncertainty should be.

They aren’t going to catch the date that’s off by a year.

You Can’t Fix This With More AI

The instinct is to solve it the way we solve everything else now. Add a verification layer. Have a model check the model. I’ve built that layer, and on the right problem it earns its keep.

It doesn’t work here, and the reason is structural rather than technical. A verifier trained on the same degraded corpus inherits the same gaps. Neo’s problem was never that the simulation looked fake. It was that every instrument available to test it had been built inside it. A second model trained on the same corpus is standing in the same room, reading the same walls, agreeing with total confidence.

That isn’t verification. It’s an echo.

This is a data problem, and data problems get solved by people deciding what goes in.

That means provenance and dated sources over volume. It means treating an AI answer as a starting position rather than a verdict, especially in the areas where you can’t personally check the work. It means the boring discipline of maintaining what you feed your systems, which nobody wants to fund because it produces no demo.

The Part We Skip

A star is beautiful from the ground. Clean, fixed, quiet.

Up close it’s gas under crushing pressure, fusing itself apart, held in balance by forces that will eventually lose. Nobody sees that. We see the result, and the result is lovely, so we stop there.

We’re doing the same thing with these systems. The output is fluent and immediate and it feels like an answer, so we take it and move on without asking what it was built from or what it quietly left out.

That’s fine when the stakes are a dinner recommendation. It’s not fine when it’s the thing teaching a fifteen year old how the world works, or approving your vendor payment, or writing the analysis your quarter depends on.

Understanding the machinery isn’t pedantry. It’s the only way to use these systems for what they’re actually capable of, and right now most people are admiring the light without any idea what’s burning.

Frequently Asked Questions

How is AI actually trained?

Large language models are trained on enormous quantities of text scraped largely from the public internet. Nobody hand selects which claims are true. The system learns statistical patterns and weights sources by signals that correlate with authority, such as links, repetition, and domain reputation. Accuracy is inferred from those proxies rather than verified.

Can AI training data be poisoned deliberately?

Yes, and more cheaply than assumed. A joint study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that roughly 250 malicious documents were sufficient to backdoor models ranging from 600 million to 13 billion parameters. The number stayed fixed as model size grew, meaning attackers need a small absolute count rather than a share of the corpus.

What is model collapse?

Model collapse is the degradation that occurs when models are trained on data generated by earlier models. Research published in Nature in 2024 showed that recursive training on synthetic output causes models to lose the tails of the original data distribution, so rare and specific information disappears first and errors compound across generations.

Does removing fact checking affect AI models?

Indirectly, yes. Fact checking reduced the reach and apparent credibility of false claims. Those same signals, engagement, links and domain authority, are what training pipelines read as reliability. When claims circulate without friction, they enter the corpus looking more legitimate than they are.

Can AI fact check itself?

Not reliably. A verifier trained on the same corpus inherits the same gaps and the same blind spots. Confident agreement between two models trained on similar data is not independent verification. Fixing degraded training data requires human decisions about provenance, not another layer of inference.

Related Reading

Sources

Post Views: 3
Tags

Related Posts

Trending now

A glowing AI safety wall passing its security check while a hooded attacker reroutes the data streams flowing around it
Your AI Safety Wall Checks One Thing. The Attacker Changes the Other.
Cables feeding an AI chip behind a green checkmark shield, pulling in propaganda posters, contradictory headlines and cracked screens
Your AI Sounds Certain. Look at Where It Learned That.
Title card for the Mars Environmental Dynamics Analyzer (MEDA) hackathon competition, showing the Perseverance rover on the rust-colored floor of Jezero Crater with its wheel tracks behind it
Everyone Knows It Scores Half. Nobody Checks Which Half.
A split-panel illustration contrasting traditional authentication showing all-green login checks against behavioral analysis flagging an unusual QuickBooks to Zelle transfer, the pattern behind business impersonation fraud.
The New Fraud Economy: When Trust Becomes the Attack Surface