Does AI tutoring work? In the 2026 trials I read for this article, it did all three things: it raised test scores, it changed almost nothing, and in one university trial it lowered grades. The tool was rarely the deciding factor. The result turned on where learning was measured, what students were compared against, and how many of them used the tool at all.
That should change the first question a principal puts to a vendor. Picture the meeting (a hypothetical one). The deck says students who used the tutor improved. It is probably true. It also says almost nothing, because the students who open a tutor every week are the students most likely to have improved without one.
What do the 2026 trials actually show?
Five recent studies found a clear gain on a test, a small gain, a fall in grades and a saving of teacher time, and each is an honest reading of a different thing. I read each at its source on November 22, 2026; for the two longest papers I relied on the abstract and the results sections.
| Study | Who and what | Result | Hold in mind |
|---|---|---|---|
| Li and colleagues, preprint, November 14, 2026 | 7,200 students in 55 applied STEM classes at 19 universities; AI designed by each instructor for their own course; compared with the same courses' earlier AI-free assessments (pre-registered) | Learning up 0.32 SD (95% CI 0.06 to 0.59) on assessments the AI could not help with, mostly proctored in-person exams; engagement unchanged | University STEM only. The authors say other settings remain untested and few classes were observed beyond a first implementation |
| Liu and colleagues, University of Maryland, working paper, September 30, 2026 | 2,379 undergraduates and 30 instructors; a GPT-4o study assistant on the course site, switched on or off by instructor | Final grades down 0.37 SD; online course participation down 0.90 SD; about 15% of students offered the tool used it | One university. Students had free access to ChatGPT and Gemini anyway, and the tool ran in direct-instruction mode |
| Oreopoulos and Low, NBER working paper, August 2026 | 18 middle schools in Tennessee; Khanmigo inside daily remedial math sessions, for two years | 1.3 national percentile ranks per term, about 0.06 to 0.08 SD over a school year; gains "resemble those from Khan Academy practice without AI assistance" | The median student messaged the tutor on a third of practice days |
| Harrison and colleagues, preprint, September 13, 2026 | Medly in GCSE science, Years 9 and 10 in England; 644 students tested at the end; four weeks | Hedges' g 0.33 (95% CI 0.18 to 0.48) against 30 minutes a week of self-directed revision | Funded by Medly, which commissioned the evaluator. A five-question, 35-mark test the authors call not standardized; 30.7% attrition; findings called preliminary |
| Education Endowment Foundation, Aila trial, completed December 2025 | 86 schools teaching Years 3 to 6 in England; Oak National Academy's lesson-planning assistant; ten weeks | About 49 minutes a week saved on planning (roughly 2.5 hours against 3.3); resources no worse | It measures teacher time, not student learning. Used in about 7% of lessons in an average week |
Effect sizes are in standard deviations (SD), the usual yardstick in education research: how far the average student in one group landed from the average in the other, in units of how much students normally differ. A gain of 0.3 is about a third of the usual spread between students. A gain of 0.07 is under a tenth of it. Hedges' g, in the Medly study, is the same idea with a small-sample correction.
Why did the same technology give opposite results?
Because the studies differ in the three things that decide what a trial can see: where learning was measured, what the comparison group did, and how many students used the tool.
Start with the room. The 19-university trial measured learning where the AI could not help, mostly on proctored exams, and found a gain. Maryland looked at final grades. The Medly study used a short test that the authors say is not standardized, marked with AI and then approved by the class teacher. A result is only as strong as the room it was measured in.
Then the comparison. Medly's students beat a control group that spent the same thirty minutes a week on self-directed revision with resources of their choice, which is a fair fight. In the Tennessee study, the authors report gains that "resemble those from Khan Academy practice without AI assistance." In Chalkbeat's account of the paper, the researchers say the benefits "didn't seem to come from AI."
Then the use. In Maryland, about 15% of students offered the study assistant used it even once, so the measured effect of offering the tool is mostly the effect of a tool most students never opened. The authors' reading is that it "may displace engagement with existing learning activities without improving academic performance." They present that as a possibility, not a finding. In Tennessee, 96% of students tried Khanmigo, but the median student messaged it on only a third of the days they practiced, and in 17% of the exercises where they made a mistake. The messages were mostly bare answers or clicks on suggested prompts. Philip Oreopoulos, the first author, told Chalkbeat in August that for a keen, engaged student "it's ready to go," and added: "The issue is that most students don't use it that way."
The largest and best-measured gain came when instructors built the AI for their own courses. The Li authors conclude that AI can improve learning "when its adoption is frontline-driven." A tutor designed by the person who teaches the class is a different object from a tutor installed in it.
What is the Brochure Test?
The Brochure Test is three questions I ask of any claim that an AI tool improved learning: Where was the learning measured, somewhere the AI could not help, or on work it helped to produce? Compared with what, no help, a human, or another tool students already had? And of the students offered the tool, how many used it, and what did the rest do? A claim that survives all three is evidence. A claim that fails any of them is a brochure.
I offer it as my suggested way of reading a claim, not a validated instrument. The three questions run in a fixed order, because each one narrows the last: the room, the comparison, the use.
| Question | What it protects you from | What a thin answer sounds like | What the 2026 trials show |
|---|---|---|---|
| The room: where was learning measured? | Gains that live only inside the product | "Students improved on the platform's own quizzes" | Li used proctored exams. Medly used a non-standardized test marked with AI and approved by teachers |
| The comparison: against what? | Beating a control that was doing nothing | "Compared with last year's class" | Medly's control revised by choice for the same time. In Maryland, control students also had free general AI tools |
| The use: how many, how often, and the rest? | Averaging a few keen users with many absent ones | "Strong adoption" or "logins are up" | About 15% used the Maryland tool at all; the median Tennessee student messaged the tutor on a third of days |
Two lines go with it. Access is not use, and use is not learning. And a study of a tool is also a study of the school it was tried in.
What I See in Practice
Across the leadership teams I work with, the pattern is consistent: a vendor's summary gets read as the answer to a question the school never asked. The school buys access, then reports on access. A login count is the first number any dashboard produces and the least informative one, because it cannot tell you whether the student who logged in did any thinking.
I read research like this as a Forbes contributor and as the author of bestselling books on AI in education, and what leaders ask me afterward is rarely "does it work?" It is "what do I say to the teacher who loves it, and the governor who has read a headline?" The honest answer is the same for both: we are finding out, and here is how we will know. When a trust wants that argument put to a whole room, it is part of my education keynotes.
What can these studies not tell you?
They cannot tell you whether a tutoring product will work in your classrooms. Only two of the five involve students under 18 learning content, and one measures teacher time. Two are university studies, the Medly trial lasted four weeks, and only the Tennessee study follows students for as long as two years.
Four of the five were working papers or preprints when I read them, as far as the pages show, so none has yet been through journal peer review. Medly's evaluators disclose that Medly paid for the work and say no other connection exists; that is a reason to read carefully, not to dismiss it.
And the products change. Chalkbeat reports that Khan Academy has since redesigned its interface to weave Khanmigo into the content, which means the version in the trial is not the version a school would buy today. That is the same problem I describe in Your Approved AI Tool Just Changed. Is It Still Approved?, arriving through the research instead of the contract.
You might think a negative university result means schools should wait. It does not. One configuration at one university is a reason to test carefully. It is not a reason to refuse, which would be a decision made on less evidence than the one you are worried about.
How should a principal test an AI tutor before buying it?
Run a small, staggered pilot that answers the three questions before anything goes school-wide. Six steps, in this order:
- Name the measure first, and make it unaided. Pick an in-class check the tutor cannot help with, and decide when it will be sat. The strongest result above rested on proctored exams.
- Stagger the start. Half the classes begin now and half begin a term later, and the later group is your comparison. The 19-university trial used exactly this delay, so no student was denied the tool for good.
- Compare against what students already do. If they revise, practice or get help now, that is the baseline, not a blank page.
- Count use every week, by student, not logins. Decide before you start how many students must use the tool, and how often, for the result to mean anything.
- Look at the students who do not use it. What do they do instead, and does the pattern follow their prior attainment?
- Write the stop rule in advance. Include a fall in unaided performance, not only a failure to improve. Maryland is the reminder that a tool can leave students worse off on the measure that matters.
Then put every claim you are shown, from a vendor or from a colleague, on a card with these fields.
| Field | Fill in |
|---|---|
| The claim, in one sentence | |
| Measured by (unaided: yes or no) | |
| Compared with | |
| Students offered it / students who used it / days used per week | |
| Result, with the size of the effect | |
| Stop if | |
| Decision date and owner |
For the wider question of what evidence a board should expect, How Do You Prove AI Is Having an Impact? goes further, and What Should Schools Require Before Approving an AI Tool? covers the safety and data questions that come before any of this.
Does saving teachers' time count as evidence that students learn more?
No. It is a different claim, and a trial can establish one without the other. The Education Endowment Foundation's trial of Aila, run with the evaluator NFER across 86 schools, found teachers using it spent 76% as long preparing as teachers who did not, about 49 minutes a week less, and independent assessment found the resources no worse. That is a real finding about workload. The EEF itself writes that "time saved does not necessarily mean better planning or teaching," and Aila was used for only about 7% of lessons in an average week, with use falling over the ten weeks.
A workload claim passes the Brochure Test too, with the room changed: measure the hours before and after, with a comparison group, and count who used it. Does AI Reduce Teacher Workload? sets out what a school can measure.
The Leadership Question
The question for a principal is not whether AI tutoring works. It is whether you will know if it works here. Oreopoulos's own warning is that the same technology "can also be used to make your life easier," and that "making your life easier equals less effort, less learning." Whether a student uses a tutor to think harder or to think less is a fact about your classroom, and no study of someone else's can read it for you. Pair any pilot with a hard look at whether students are becoming dependent on AI, and judge the teachers' side of it as you would any training, by looking at the work.
A tutor most of your students never open has been installed, not tested.
If your leadership team is working through how to judge AI tools before it commits to them, this is the kind of work I support through AI strategy sessions for schools.
Sources and further reading
- Li, J., Bray, R., Martin, S. and colleagues, Instructor-designed AI improves learning in a randomized trial across 19 universities, arXiv preprint, November 14, 2026. 7,200 students, 55 classes, spring 2025 to spring 2026.
- Liu, J., Sweet, T. and colleagues, The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial, EdWorkingPaper 26-1598, Annenberg Institute at Brown University, September 30, 2026 (released October 2026). University of Maryland, College Park, fall 2025.
- Oreopoulos, P. and Low, N., One Click Away: AI Tutoring with Khanmigo in a Two-Year School Experiment, NBER Working Paper 35620, August 2026. Eighteen middle schools in Tennessee.
- Harrison, W., Khowaja, R., Dobson, E., Uwimpuhwe, G. and Higgins, S., Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science, arXiv preprint, September 13, 2026.
- Education Endowment Foundation, Does using Aila, an AI lesson-planning assistant, reduce the time teachers spend planning lessons?, evaluated by NFER, trial completed December 2025.
- Barnum, M., The problem with an AI tutor: Students don't want to use it, Chalkbeat, August 25, 2026.
Sources read on November 22, 2026. Studies are from England and the United States; results from one setting may not carry to another.
Dan Fitzpatrick is the founder of The AI Educator, a Forbes contributor and an international keynote speaker who helps school and system leaders lead through AI.


