Chapter 2 — Understanding AI: A Primer for Educators

Last updated: July 5, 2026

A language model does exactly one thing: given a sequence of text, it predicts the next token — a token being a word fragment, so "unbelievable" might be read as "un / believ / able" — then appends its pick and repeats. The familiar shorthand, "it just predicts the next word," is both true and deeply misleading, because prediction at sufficient scale forces the model to compress human knowledge into hundreds of billions of parameters, yielding real capability alongside failure modes that are specific and learnable rather than random. Understanding those failure modes, not the architecture, is what educators actually need.

The system is a fluency machine before it is a truth machine. It was optimised to produce plausible continuations, and plausibility and truth usually coincide — but when they diverge it produces confident, fluent, well-formatted falsehood, with nothing to flag the difference, because to the mechanism there is none. This is why hallucination is structural rather than a bug awaiting a patch, and why it surfaces exactly where verification is least likely: precise numbers, niche people, exact quotes, page references, recent events, and anything the question falsely presupposes.

The canonical education example is the fabricated citation. A peer-reviewed audit found 55 percent of references generated by GPT-3.5, and 18 percent by GPT-4, were entirely invented — real-sounding authors, plausible titles, journals that exist, papers that do not. Retrieval-grounded tools have improved this, though not monotonically, and the lesson holds: the more a task depends on specific, verifiable facts, the more verification it needs. The classroom image is a brilliant colleague with no access to sources and a pathological unwillingness to say "I don't know" — brainstorm with them gladly, never let them write your reference list.

The most school-relevant consequence follows directly: AI detectors cannot work as advertised. They measure how predictable a text is — its perplexity — and simpler or formulaic prose is more predictable, so they flag it: the five-paragraph essay is practically designed to be predictable, and so is second-language writing. A Stanford study found detectors flagged an average of 61 percent of essays by real, human, non-native English writers as AI-generated, one of them flagging 98 percent, while light editing or a "write less predictably" prompt defeats detection in the other direction. A tool that punishes a student for writing plainly is not a safeguard; OpenAI retired its own detector within six months for exactly these reasons.

The interface adds a second layer of trouble by installing misconceptions. Models are sycophantic — an Anthropic-led study at ICLR 2024 found consistent sycophantic behaviour across five leading assistants, traced to the training signal itself: raters preferred agreeable answers, so training produced agreeable models. The failure is not hypothetical. In April 2025 OpenAI shipped a GPT-4o update that tipped so far into flattery — by its own account "validating doubts, fueling anger, urging impulsive actions" — that it rolled the update back within days. A model that flatters a student's wrong answer is failing in a way traceable directly to how it was trained.

Fluency also invites the ELIZA effect, the old habit of mistaking smooth language for understanding. Joseph Weizenbaum named it in 1966, when users attributed real comprehension to ELIZA, his therapist-parody program of a few hundred lines. A 2024 study in Neuroscience of Consciousness found 67 percent of respondents attributed some possibility of conscious experience to ChatGPT — and the attributions rose with frequency of use. Unstructured exposure deepens the illusion; only instruction about the mechanism corrects it.

The working vocabulary is therefore small and worth teaching explicitly. Treat these systems as stochastic — the same prompt yields different answers on different runs, because token selection involves controlled randomness — as jagged, acing a physics derivation while miscounting the letters in a word (an artefact of tokenisation, since the model reads fragments, not letters), and as confident regardless of correctness. Two mechanical facts complete the picture: the base model knows only its training data, frozen at a cutoff, unless retrieval is bolted on; and its context window weighs its beginning and end far more reliably than its middle.

Hold those three mental models and most of the interface's traps lose their grip. The demonstration that installs them takes twenty minutes: ask for page-numbered quotes from a book the class has read, ask about yesterday, ask the same question in two fresh chats, assert something false and watch it agree. What survives that exercise is the only durable safeguard — including against the temptation to rest a grade, or an accusation, on a detector's score.

Sources. [Primary empirical] Walters, W. H., & Wilder, E. I. "Fabrication and errors in the bibliographic citations generated by ChatGPT." Scientific Reports (2023). https://www.nature.com/articles/s41598-023-41032-5 (PMC: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10484980/) · [Primary empirical] Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. "GPT detectors are biased against non-native English writers." Patterns (2023). https://www.sciencedirect.com/science/article/pii/S2666389923001307 · [Primary empirical] Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. "AI generates covertly racist decisions about people based on their dialect." Nature (2024). https://www.nature.com/articles/s41586-024-07856-5