3 years in the past, Khan Academy introduced Khanmigo, a generative AI-powered tutor for college kids and assistant for academics. Since then, we’ve labored ceaselessly to give a boost to its tutoring functions. Nowadays we’re sharing the result of our most up-to-date growth efforts.
Our efforts are incremental and ongoing, and we’re inspired by means of the six-percentage-point growth described beneath. Carried out throughout tens of millions of observe periods in keeping with day, the acquire interprets to a significant build up within the selection of scholars who be told from every tutoring interplay.
We imagine this sort of product-development procedure—carefully trying out every exchange, moderately measuring results, and discarding what doesn’t paintings—is very important for development efficient AI gear in schooling.
How We Find out about What Works
We accumulate proof about Khanmigo in numerous techniques, together with study room observations, instructor and pupil interviews, and research of pupil chat transcripts.
Over six months, from October 2025 to April 2026, Khan Academy ran a rigorous collection of product checks to grasp what adjustments may give a boost to Khanmigo’s effectiveness. The next findings summarize what we’ve discovered and the way we’re the usage of them to give a boost to Khanmigo.
How We Measure Luck
All through this paintings to give a boost to Khanmigo, we tracked 3 core metrics. Relying at the check, every metric served as both the principle function or a guardrail to verify we weren’t making improvements to one measurement on the expense of every other:
- Reaction latency: the time a pupil waits between asking a query and receiving Khanmigo’s reaction. A key part of conserving scholars engaged is having the interplay with Khanmigo really feel like a herbal dialog. Quicker responses stay scholars centered and are vital to creating the instrument really feel herbal.
- Subsequent-item correctness: whether or not the coed solutions the following difficulty appropriately after receiving tutoring. Subsequent-item correctness measures whether or not the coed appropriately responded the very subsequent difficulty at the similar talent all through the similar consultation with none assist from Khanmigo. It’s an immediate measure of unbiased finding out switch, no longer simply of efficiency with AI help.
- Cognitive engagement high quality: The use of a scale of “passive,” “energetic,” and “positive” rankings, an automatic evaluation of whether or not the change in every tutoring interplay used to be no less than energetic, which means the coed used to be reasoning and tasty as a substitute of simply passively receiving knowledge. You’ll learn extra about our paintings to measure cognitive engagement in ACL Anthology.
We additionally monitored further guardrail metrics in each check, together with cases of giving the solution away sooner than a pupil submitted a reaction, math error charges, and interactions in keeping with thread, as a way to make certain that adjustments weren’t inflicting accidental hurt in different places.
We run those experiments thru an A/B trying out platform that predicts whether or not the examined model will lead to higher metrics results than our keep an eye on model. This is known as its “probability to win.” If that chance is larger than .95 with none damaging affect on guardrail metrics, we put into effect the exchange.
Making the mathematics agent quicker with out sacrificing accuracy
When a pupil is operating thru a math difficulty, Khanmigo has a specialised machine that verifies calculations and assessments mathematical expressions in actual time. This “math agent” runs at the back of the scenes and is helping make certain that when Khanmigo responds to a pupil, the mathematics is proper. Decreasing the time the program takes to reply is essential. The fewer wait time a pupil reports between asking a query and receiving a reaction, the extra they keep engaged and the extra herbal the instrument feels.
We ran a sequence of product checks interested by lowering wait time whilst carefully tracking cognitive-engagement high quality and next-item correctness to make certain that quicker responses didn’t come on the expense of instructional high quality.
Effects:
- Switching the mathematics agent to a quicker AI style diminished reaction time by means of 0.3 seconds throughout 1.35 million tutoring threads over 12 days. Math accuracy held secure.
- Educating the mathematics agent to provide a extra concise reaction to Khanmigo diminished imply reaction time by means of 3 seconds throughout 352,000 tutoring threads over 5 days. A follow-up experiment by which we restricted the agent to specializing in the mathematics the coed had already executed as a substitute of additionally figuring out the remainder steps to get to the answer diminished latency by means of every other 400 milliseconds and diminished making a gift of the solution by means of 50%. Math accuracy held secure.
- Including a pre-check that made up our minds whether or not a math-verification step used to be even wanted sooner than invoking the mathematics agent diminished pointless calls to the machine throughout 1.04 million tutoring threads, slicing reaction time by means of about 0.3 seconds. Math accuracy held secure.
Key takeaway: We known a couple of levers for lowering Khanmigo’s reaction latency, together with a quicker style, shorter outputs, tighter time-outs, and smarter routing with out sacrificing the standard of the tuition. Those enhancements topic for pupil revel in—quicker responses stay scholars centered—and for price sustainability at scale.
The use of a pupil’s Khan Academy finding out historical past to give a boost to tutoring
When a pupil opens Khanmigo all through a convention workout, Khanmigo sees the issue they’re operating on. But it surely doesn’t mechanically understand how the coed has been functioning on that workout, what talent degree they’ve demonstrated, or the place they’ve been getting caught. We ran a sequence of product checks to judge whether or not giving Khanmigo get admission to to extra of a pupil’s Khan Academy finding out historical past, together with their fresh observe makes an attempt, demonstrated talent ranges, and prerequisite development, would assist it tutor extra successfully. A very powerful privateness word: Khanmigo complies with privateness laws, together with pupil knowledge privateness laws, equivalent to FERPA, COPPA, and state privateness regulations.
The principle result right here used to be next-item correctness: did the coed get the following difficulty proper after receiving tutoring?
What labored:
- Offering a abstract of the coed’s fresh problem-solving historical past on Khan Academy, together with what number of issues they tried not too long ago and which of them they were given proper and mistaken, stepped forward next-item correctness by means of +3.4% throughout 608,000 tutoring threads. There’s a 97.5% probability together with this data might be higher than no longer together with it within the total inhabitants of customers. All guardrail metrics held.
- Surfacing prerequisite talents the coed hasn’t but mastered and providing a temporary assessment sooner than the tougher difficulty stepped forward next-item correctness by means of 2.7% throughout 1.36 million tutoring threads. There’s a 98.5% probability of higher results than when this data isn’t integrated.
- Offering the total in-session dialog log. To begin with, together with the total in-session dialog log as a part of the tips to be had to the style as scholars persevered operating on talents didn’t result in measurable enhancements in pupil efficiency by itself. We then made two adjustments: 1) we put the dialog in undeniable textual content as a substitute of in hard-to-parse json, a knowledge switch structure and a pair of) we added all of the threads associated with this talent for the former 24 hours as a substitute of simply within the present consultation. When doing this, we discovered a 5.09% build up in cognitive engagement—a 99.4% probability of higher engagement with this data than with out it. We’re recently making plans experiments that contain extracting the pedagogically significant parts of the logs to cross to Khanmigo fairly than simply passing the uncooked logs.
What didn’t transfer the needle:
- Including examples of various difficulty sorts associated with the talent as a part of the suggested confirmed no impact.
- Offering extra related follow-up content material hyperlinks in keeping with the coed’s place within the Khan Academy content material confirmed no statistically vital exchange in next-item correctness. The exchange used to be rolled out as it did no hurt and modestly diminished reaction time, making the revel in rather quicker for college kids.
Key takeaway: When Khanmigo has get admission to to structured indicators from a pupil’s Khan Academy finding out report, equivalent to their fresh efficiency patterns and talent gaps, it produces measurably higher tutoring.
Our dedication to principled development
Throughout kind of 20 substantive product checks on this area over six months:
| What we attempted | Purpose | High quality guardrails |
|---|---|---|
| Making the mathematics agent quicker | Cut back reaction latency | Held secure |
| Giving Khanmigo structured Khan Academy finding out historical past | Enhance next-item correctness | Certain (+2.7% to +3.4%) |
| Giving Khanmigo hard-to-parse knowledge | Enhance next-item correctness | Impartial, no measurable impact |
Those product checks lined a complete of greater than 15 million tutoring threads throughout a six-month duration. Every check when compared the brand new model towards the present product revel in, and effects had been evaluated for his or her chance of fixing our key metrics sooner than any exchange used to be widely shipped.
The whole image is one in every of cautious, evidence-driven optimization. No unmarried growth by itself produced a dramatic bounce ahead, however jointly, this frame of labor has meaningfully stepped forward Khanmigo’s effectiveness and known more economical and quicker techniques to run it at scale.
A complete paper describing our metrics, infrastructure, and experiment effects might be revealed within the 27th International Conference in AI for Education.
