Chapter 4 — What the Research Says: AI and Learning Outcomes

Last updated: July 5, 2026

The research supports a narrow but useful claim: AI tutoring can produce real learning gains, but the effect lives in the design rather than the technology, and the honest numbers are smaller and shakier than the marketing implies. Getting this right means holding the good evidence and its limits at once. Kraft's calibration treats field-trial effects above 0.20 SD as large, so education claims routinely invoke numbers no credible intervention has produced.

The strongest trials share a structure. A Harvard trial (Kestin and colleagues) randomized introductory-physics students to learn two topics either in an expert-led active-learning class or with a GPT-4 tutor purpose-built for the course ("PS2 Pal"); students learned more than twice as much from the tutor, in less time. The World Bank's Edo State programme in Nigeria — six weeks, teachers guiding twice-weekly Copilot sessions — returned +0.31 SD on the composite and +0.23 SD on English, outperforming about 80 percent of rigorously evaluated developing-country interventions. A WhatsApp tutor in Ghana delivered around 0.36 SD over eight months on low-bandwidth phones. Stanford's Tutor CoPilot lifted the least-experienced human tutors most — 9 percentage points for students of the lowest-rated tutors, at roughly $20 a tutor per year. What these share is guidance, structure and a human in the loop — not unrestricted access to a chatbot.

Turkey supplied the mirror image, and it is the field's most policy-relevant result. Bastani and colleagues, in PNAS as "Generative AI without guardrails can harm learning," gave roughly a thousand high-schoolers vanilla ChatGPT during math practice; they solved 48 percent more problems correctly, then scored 17 percent worse than no-AI controls on the unassisted exam. A guarded "GPT Tutor" giving hints rather than answers erased the harm but produced no benefit either. The moderator was never the model but the guardrails.

The historical baseline keeps the excitement honest. Bloom's famous "two sigma" came from small dissertation studies confounding tutoring with mastery requirements, never replicated at scale; rigorous high-dosage human tutoring clusters around 0.3 to 0.4 SD. The pre-LLM intelligent-tutoring literature is deep and, at scale, humbling: VanLehn found step-based systems indistinguishable from human tutors, yet RAND's Cognitive Tutor Algebra trial across 147 schools found no first-year effect and only +0.21 SD in year two, and ASSISTments shrank to about 0.10 SD on replication. A defensible prior for a well-designed AI tutor is therefore modest.

The field's own quality-control problem makes humility mandatory. Its most-cited synthesis — Wang and Fan's 2025 meta-analysis, reporting an implausible g = 0.867 — was retracted in April 2026 over discrepancies the authors would not address, so a citation of it now works as a freshness test for whoever is quoting it. The surviving meta-analyses land lower and softer — g = 0.670, 0.577, 0.573, 0.45, a preregistered math-specific 0.534 — but rest on brief, small, researcher-made-test studies where sub-month interventions returned g = 0.735 against 0.376 for longer ones — exactly where novelty effects live.

The absences matter as much as the findings. Khanmigo, the most prominent AI tutor in American schools, still has no published randomised trial showing learning gains; the one peer-reviewed comparison, 69 physics undergraduates, found no difference from ordinary Google search, and Sal Khan himself conceded to Chalkbeat in April 2026 that for many students the tutor was "a non-event." Alpha School's "doubled learning velocity" likewise has no independent evaluation, and Stanford SCALE called the whole K-12 evidence base "extremely limited." That is not proof of failure — absence of evidence is not evidence of absence — but it should discipline procurement rhetoric rather than be papered over by a vendor case study.

Even the largest number wears a vendor badge. Google DeepMind's Sierra Leone trial — 1,763 students, teacher-led Gemini Guided Learning, +0.258 SD on blind-scored assessments — is exactly the guarded design the independent trials reward, yet it is vendor-run, not peer-reviewed, and its own data show stronger students gaining more. The mechanism work converges: in Gallegos's Chilean experiment, encouraging AI adoption changed nothing, while guidance on learning-oriented use raised final-exam scores 0.21 SD and pass rates 12 points. Teaching students how to use a tutor has better evidence than merely giving them one.

The operating rule, then, is to follow the design, not the technology: buy structured, guided, guardrailed use; distrust any headline effect above roughly 0.5 SD or any study shorter than a school term; and read every "years of learning" claim back into the standard deviation before believing it (Chapter 6). The evidence is genuinely encouraging and genuinely preliminary, and saying both at once is the whole skill.

Sources. [Primary empirical] Kestin, G., Miller, K., et al. "AI tutoring outperforms in-class active learning." Scientific Reports (2025). https://www.nature.com/articles/s41598-025-97652-6 · [Primary empirical] De Simone, M., et al. "From Chalkboards to Chatbots." World Bank Policy Research WP 11125 (2025). https://documents1.worldbank.org/curated/en/099548105192529324/pdf/IDU-c09f40d8-9ff8-42dc-b315-591157499be7.pdf · [Primary empirical] Bastani, H., Bastani, O., Sungu, A., et al. "Generative AI Without Guardrails Can Harm Learning." PNAS (2025). doi:10.1073/pnas.2422633122. https://www.pnas.org/doi/10.1073/pnas.2422633122