AI Tutoring Can Safely And Effectively Support Students: An Exploratory RCT In UK Classrooms LearnLM Team, Google & Eedi One-to-one tutoring is widely considered the gold standard for personalized education, yet it remainsprohibitively expensive to scale. To evaluate whether generative AI might help expand access to thisresource, we conducted an exploratory randomized controlled trial (RCT) with 𝑵 = 165 students acrossfive UK secondary schools. We integrated LearnLM—a generative AI model fine-tuned for pedagogy—intochat-based tutoring sessions on the Eedi mathematics platform. In the RCT, expert tutors directlysupervised LearnLM, with the remit to revise each message it drafted until they would be satisfiedsending it themselves. LearnLM proved to be a reliable source of pedagogical instruction, with supervisingtutors approving 76.4% of its drafted messages making zero or minimal edits (i.e., changing only one ortwo characters). This translated into effective tutoring support: students guided by LearnLM performedat least as well as students chatting with human tutors on each learning outcome we measured. Infact, students who received support from LearnLM were 5.5 percentage points more likely to solvenovel problems on subsequent topics (with a success rate of 66.2%) than those who received tutoringfrom human tutors alone (rate of 60.7%). In interviews, tutors highlighted LearnLM’s strength atdrafting Socratic questions that encouraged deeper reflection from students, with multiple tutors evenreporting that they learned new pedagogical practices from the model. Overall, our results suggestthat pedagogically fine-tuned AI tutoring systems may play a promising role in delivering effective,individualized learning support at scale. Keywords learning, efficacy, safety, artificial intelligence, tutoring, randomized controlled trial
Designing Safe And Relevant Generative Chats For Math Learning In Intelligent Tutoring Systems
Designing Safe And Relevant Generative Chats For Math Learning In Intelligent Tutoring Systems Abstract Large language models (LLMs) are flexible, personalizable, and available, which makes their use within Intelligent Tutoring Systems (ITSs) appealing. However, their flexibility creates risks: inaccuracies, harmful content, and non-curricular material. Ethically deploying LLM-backed ITSs requires designing safeguards that ensure positive experiences for students. We describe the design of a conversational system integrated into an ITS that uses safety guardrails and retrieval-augmented generation to support middle-grade math learning. We evaluated this system using red-teaming, offline analyses, an inclassroom usability test, and a field deployment. We present empirical data from more than 8,000 student conversations designed to encourage a growth mindset, finding that the GPT-3.5 LLM rarely generates inappropriate messages and that retrieval-augmented generation improves response quality. The student interaction behaviors we observe provide implications for designers—to focus on student inputs as a content moderation problem—and implications for researchers—to focus on subtle forms of bad content and creating metrics and evaluation processes. Keywords large language models, intelligent tutoring systems, safety, system designGeneralization, Natural language processing, CollaborationMultiple Choice Question, Large Language Models, Humanin-the-loop.-
Effective And Scalable Math Support: Experimental Evidence On The Impact Of An AI- Math Tutor In Ghana
Effective And Scalable Math Support: Experimental Evidence On The Impact Of An AI- Math Tutor In Ghana Abstract This study is a preliminary evaluation of the impact of receiving extra math instruction provided by Rori, an AI-powered math tutor accessible via WhatsApp, on the math performance of approximately 500 students in Ghana. Students assigned to both the control and treatment groups continued their normal classes with identical curricula and classroom hours. Students in the treatment group were given access to a phone for one hour a week – during their study hall period – and were allowed to use Rori to independently study math. All other aspects of the groups’ in-school experience were the same. We find that the math growth scores were substantially higher for the treatment group and statistically significant (p < 0.001). The effect size of 0.36 is considered large in the context of educational interventions: approximately equivalent to an extra year of learning. Importantly, Rori works on basic mobile devices connected to low-bandwidth data networks, and the marginal cost of providing Rori is approximately $5 per student, making it a potentially scalable intervention in the context of LMICs’ education systems. While the results should be interpreted judiciously, as they only report on year 1 of the intervention, they do suggest that chat-based tutoring solutions leveraging AI could offer a cost-effective and operationally efficient approach to enhancing learning outcomes for millions of students globally.Keywords LLMs, conversational agents, mobile-learning