What a language model actually does

The underlying operation of a language model is narrower and stranger than most descriptions suggest.

Next token prediction

Estimating a probability distribution over what comes next.

Which is the entire training objective.

Tokens

Text split into fragments rather than words.

Which explains several odd behaviours including difficulty with spelling.

Context window

How much text the model can consider at once.

Which is a hard limit rather than a soft one.

Why it produces fluent text

Fluency being what the objective directly rewards.

Which is not the same as accuracy.

Why the mechanism explains so much of the behaviour

A system trained solely to predict what comes next will be excellent at producing text that reads correctly and will have no separate mechanism for checking whether it is true.

Which is not a defect in the implementation; it is what the objective specifies.

Almost every strength and every failure mode people report follows from that single fact, and holding it in mind makes the behaviour far less mysterious.

Tokenisation and its consequences

Text split into fragments that do not align with letters or words.

Which is why counting characters, reversing strings and some arithmetic are unexpectedly difficult.

The model never sees individual letters in the way a person reading the text does.

Sampling

Output selected probabilistically from the predicted distribution.

Which is why the same prompt produces different answers.

Temperature

A setting controlling how much randomness is applied.

Which trades consistency against variety.

What it is not doing

Looking anything up, reasoning symbolically, or consulting a database.

Why the mechanics are worth understanding

These systems are described in language that suggests understanding, reasoning and knowledge, and the underlying operations are considerably more specific than those words imply. That gap is where most confusion, most disappointment and a substantial amount of misplaced trust originates.

Someone who knows that a language model predicts tokens, that it has no separate lookup step, that its context is a hard limit and that its confidence is unrelated to its accuracy will make better decisions about when to use one than someone working from the marketing description. Not because the technology is worse than claimed, but because knowing what a tool actually does is how anyone uses a tool well.

The pattern across all of this

Nearly every behaviour that people find surprising follows directly from a design choice that is documented and comprehensible. Invented citations follow from the training objective. Difficulty with spelling follows from tokenisation. Degradation in long conversations follows from context limits and attention behaviour. Cost structures follow from the fact that inference is not free.

None of these are mysteries or failures of implementation. They are consequences, and they are predictable once the mechanism is understood.

Where the reliable information is

Research papers, model documentation, technical blogs from the organisations building these systems, and independent evaluation work. Most of it is freely available and considerably more measured than either the promotional material or the criticism that circulates.

The field moves quickly enough that any general article, including this one, dates rapidly. Anything with consequences attached is worth checking against current documentation rather than against a description written at some point in the past.

What follows from all this in practice

Use these systems for work where the output can be checked, where fluency is genuinely useful, and where a draft that needs revision is more valuable than a blank page. Avoid relying on them for specific facts, citations, numbers or anything where a confident wrong answer is worse than no answer.

That is a narrower recommendation than the enthusiasm suggests and a much broader one than the dismissal allows. Both positions tend to be argued by people who have not looked closely at what the systems do, and the useful territory between them is large.

On the pace of change

Capabilities have moved quickly enough that specific claims about what models can and cannot do have a short shelf life. Several limitations described confidently a few years ago have been substantially reduced, and several that were expected to fall have not.

The underlying mechanisms have changed far less than the capabilities. Next token prediction, tokenisation, context limits and the absence of a verification step have all remained, which is why the same categories of failure keep recurring in new forms even as performance improves.

A note on evaluating claims

When a capability is announced, useful questions are: measured on what benchmark, compared against what, evaluated by whom, and reproducible by anyone outside the organisation making the claim. Those four questions dispose of a considerable amount of what circulates.

Independent replication in this field is under-resourced relative to the scale of the claims, which is a structural problem rather than a criticism of any particular organisation.