A school should stop using an AI tool when it cannot show, with real student or staff work rather than opinion, that the school is better with the tool than without it. Review every tool once a term, give each one a verdict of keep, fix or retire, and treat retiring a tool properly as part of the job, not as a failure of it.
Open your approved list and find the last tool that came off it. If you cannot name one, the list has been a one-way door, and the people holding it open are the staff who still log in to things that no longer earn their place.
Why are schools being asked to stop tools now?
Because the people who write guidance have started asking what happens after the purchase.
On August 20, 2026, the US Department of Education sent states and districts a Dear Colleague letter on educational technology. EdSurge's report quotes Assistant Secretary Kirsten Baesler saying that technology use should be "purposeful, instructionally sound, and contributing to measurable student outcomes." EdTech Innovation Hub's account adds the line that matters here: schools should review implementation and outcomes after adoption, and change course or remove tools that show persistent shortcomings. I could not open the letter itself this week, so I am relying on those reports, and on GovTech's, for what it says. It is guidance, not regulation.
The gap it points at is real. The State Educational Technology Directors Association (SETDA) published its fifth annual state trends report on October 29, 2026, drawing on 73 survey responses from state education agency leaders in 45 states and one territory. In its press release, 67% of respondents said their state has a lot of edtech but does not always use it effectively, and 52% said their state collects no data on the use or effectiveness of edtech tools. These are state-level answers from a small group, not a count of districts. But if the level above you is not watching, the only review your tools will get is the one you run.
In England the pressure arrives from a different direction. The Department for Education's Generative AI: product safety standards, last updated on January 19, 2026, are written for developers, with a note that schools may find them useful in judging which products are safe. As far as I can find, they set no review cycle for schools. They do ask for a data protection impact assessment "during the full life cycle of the tool," and a life cycle has an end.
Sometimes the end is chosen for you. Newsweek reported on October 16, 2026 that New York City announced a one-year moratorium on AI for prekindergarten through eighth grade for 2026-27 and banned AI companion chatbots in every grade. A school that has already decided what it would stop, and why, meets news like that with a plan. A school that has not meets it with a scramble.
Why do approved lists only ever grow?
Because adding a tool has a champion, and removing one has nobody.
Someone asked for the tool. Someone trained the staff. Somebody's lessons now lean on it. Nobody's job description includes standing up in a meeting to say that it did not work, and the review slot in the calendar goes to the next adoption. The result is a list that behaves like a hallway closet.
The approval step has its own defense against this. The Two-Signature Rule asks every tool to arrive with a written condition and a date that would take it off the list. This article is about the day the date arrives.
What is the Vanishing Test?
The Vanishing Test is the question I suggest every tool answers at the end of a term, and I offer it as a way of thinking, not a validated instrument:
The Vanishing Test is three questions I ask of any AI tool at the end of a term: If it vanished tonight, who would notice first, and what would they have to do instead? Could we show, with work rather than opinion, that the school is better with it than without it? And what is it holding that we would have to get back? A tool that passes all three has earned another term. A tool that fails any of them has a decision waiting, and the decision is keep, fix or retire.
Who would notice first? Name a person, not a department. If the honest answer is "the teacher who asked for it," that is a legitimate answer for a tool one teacher relies on, and a thin one for a tool the whole school was told to adopt. If the answer is "nobody," you have found a tool that exists on your list and nowhere else.
Could we show it? Evidence means work: five pieces of feedback before the tool and five after, a set of lesson plans, a term of marking turnaround. The Department's news release, as GovTech reports it, distills its approach into five questions every edtech product should be able to answer: what learning problem it solves, when it should be used, for whom, for how long, and what evidence shows it improves student learning. Those were written for buying. They work better at the end of a term, when you have a term of evidence. Login counts do not answer them. Usage tells you the tool arrived; it does not tell you the school changed.
What is it holding? Student work, staff drafts, parent letters, a routine three colleagues now depend on. This question does not decide whether a tool stays. It decides how carefully the exit has to be run.
How do you run a term-end review?
Hold one 45-minute meeting per term, with the list on the table, one row per tool, chaired by the person who owns the list rather than the person who championed the tool.
Sort first, in the order that the approved-list sort uses: student-facing tools, then tools that touch student data, then the rest. A week before the meeting, send each tool's owner the three questions and ask them to bring three pieces of work. Ask the supplier for the usage reports its product already produces; the DfE standards expect products to give teachers data such as the duration of use by each learner, in reports that non-expert staff can understand. Then fill in one row per tool:
| Tool | The job it was approved for | Who would notice first | Evidence (work, not logins) | What it holds | Verdict | Next review |
|---|---|---|---|---|---|---|
| Name | One sentence | A person | Three samples, or "none" | Data, routines, dependents | Keep, fix or retire | A date |
The verdict rules are short. Keep needs evidence and a date for the next look. Fix needs a specific change and a deadline of one term: switch off a feature, change who uses it, change when. Retire needs an exit date. I would add one rule of my own: a tool may be fixed once. If it comes back next term needing the same repair, it is already leaving, and the school is only delaying the announcement.
What does a review look like for three tools?
It looks like three different verdicts from the same three questions. This is a hypothetical school, with invented tools described by their jobs.
| Tool | Who would notice first | Evidence | Holds | Verdict |
|---|---|---|---|---|
| Feedback assistant for Year 10 English essays | Three English teachers, within a week | Teacher-edited feedback samples show specific comments on 4 of 5 essays; marking turnaround shorter | Student essays | Keep. Review again in February. |
| Revision chatbot for Year 11 science | The head of science, eventually | Students describe it as "giving the answer"; two could not explain a result they reached with it | Student questions | Fix. Switch off worked solutions; students show their reasoning first. Review at Christmas. |
| Slide generator two teachers tried in September | Nobody | Neither teacher can show a lesson that used it | Nothing of substance | Retire. Remove from the list and the shared accounts this month. |
The second row is the one to study. The chatbot is popular, and popularity is the trap, a point I return to below.
How do you retire a tool without breaking anything?
Retire it on a date, with the job moved before the tool goes.
- Name the job and where it goes. Retiring the tool does not retire the job. If staff still need what the tool did, say what they will use instead, and make that route the easier one. A retired tool with no replacement sends the job underground, which is how shadow AI starts.
- Tell users before the day. Give the date and the replacement in the same message.
- Export what people need to keep. Student work, lesson resources and anything a colleague built on top of it.
- Ask the supplier in writing to delete the school's data, and file the confirmation. The DfE standards, as far as I can find, give no deletion timetable, so your agreement is the only place that promise can live.
- Remove it everywhere on the same day. The approved list, single sign-on and any shared accounts. Tell staff who signed up with personal accounts.
- Record why, in one sentence, and who decided. If students used it, tell parents. That sentence is what you show a governor when they ask what the school has stopped.
The Mistake I See Most Often
The mistake I see most often is a review that asks whether people like the tool.
Across the leadership teams I work with, the review conversation collects testimonials. Staff love it. Students ask for it. These are real signals, and they measure how pleasant the tool is. A tool can be pleasant and still do the part of the work you wanted students to do themselves, which is why the second question in the Vanishing Test asks for work. I have trained more than 150,000 educators across more than 30 countries, and enthusiasm is the least reliable witness I know. The people who love a tool most are rarely the people placed to see what it replaced.
As a school trustee, I read a good many reports on what a school has adopted. I read very few on what it has dropped. Boards that can see both columns are the ones I would trust with the next decision. The Decision Trail, the record of AI decisions a leadership team can show, counts "something that stopped" among its five visible signs for exactly this reason.
When should you keep a tool you cannot prove works?
Keep it when it removes a barrier rather than adds a benefit, or when the benefit arrives slowly, and write down what you will look for.
Assistive technology is the clear case. A tool that lets a student with an access need take part in a lesson is not asking to be measured against a test score; the question is whether the student can do the task without it, and for how long that will stay true. The Department's own examples of purposeful technology, as EdTech Innovation Hub lists them, include assistive technology. The same honesty applies elsewhere. "We cannot measure it yet" is a legitimate sentence only when it comes with a date and a named thing you will look at. "We did not look" is a different sentence, and it should not be allowed to pass for the first.
A tool that fails because students lean on it too hard is a separate case, and the repair is usually a limit on use, not a retirement. The question underneath it, whether your AI training builds capability or dependence, is worth asking in the same meeting.
What should leaders do this term?
Put the list in a table, set the review for the last fortnight of term, and name one tool you will fix or retire before you go on break.
Start with the student-facing tools. Send each owner the three questions a week ahead. Add the verdicts to the one-page appendix that a short school AI policy keeps for tools, and take the verdicts, including the retirements, to your governors or trustees as part of what you report to them about AI. When the board asks how you know your tools are working, the evidence you can show will be the reviews you ran.
If a tool changed under you this term, the Still-True Test belongs in the same meeting.
The next step
If your leadership team is working out which of its AI tools have earned another term, this is the kind of work I support through AI strategy sessions for schools. For a board session or a conference where leaders are working through these decisions together, see my keynotes for education leaders. I write about decisions like this in my newsletter for school leaders.
Sources and further reading
- EdSurge, "Department of Education Issues Long-Awaited Edtech Guidance for States and Districts," Ira Apfel, August 20, 2026. edsurge.com
- GovTech, "U.S. Department of Education: Ed Tech Must Prove Its Value," Julia Gilban-Cohen, August 21, 2026. govtech.com
- EdTech Innovation Hub, "US Education Department urges schools to judge EdTech by outcomes, not screen time," Emma Thompson, August 2026. edtechinnovationhub.com
- State Educational Technology Directors Association, "New Report: AI Remains Top State EdTech Priority Amid Growing Debate About Technology in Schools," October 29, 2026. setda.org
- Department for Education, "Generative AI: product safety standards," GOV.UK, last updated January 19, 2026. gov.uk
- Newsweek, "Map Shows States That Are Trying To Get AI Out of Schools," Hannah Parry, October 16, 2026. newsweek.com
Dan Fitzpatrick is the founder of The AI Educator, a Forbes contributor and the author of bestselling books on AI in education. Read more about Dan.


