Designing Safe And Relevant Generative Chats For Math Learning In Intelligent Tutoring Systems Abstract Large language models (LLMs) are flexible, personalizable, and available, which makes their use within Intelligent Tutoring Systems (ITSs) appealing. However, their flexibility creates risks: inaccuracies, harmful content, and non-curricular material. Ethically deploying LLM-backed ITSs requires designing safeguards that ensure positive experiences for students. We describe the design of a conversational system integrated into an ITS that uses safety guardrails and retrieval-augmented generation to support middle-grade math learning. We evaluated this system using red-teaming, offline analyses, an inclassroom usability test, and a field deployment. We present empirical data from more than 8,000 student conversations designed to encourage a growth mindset, finding that the GPT-3.5 LLM rarely generates inappropriate messages and that retrieval-augmented generation improves response quality. The student interaction behaviors we observe provide implications for designers—to focus on student inputs as a content moderation problem—and implications for researchers—to focus on subtle forms of bad content and creating metrics and evaluation processes. Keywords large language models, intelligent tutoring systems, safety, system designGeneralization, Natural language processing, CollaborationMultiple Choice Question, Large Language Models, Humanin-the-loop.-
Students’ Perceived Roles, Opportunities, And Challenges Of A Generative AI-Powered Teachable Agent: A Case Of Middle School Math Class
Students’ Perceived Roles, Opportunities, And Challenges Of A Generative AI-powered Teachable Agent: A Case Of Middle School Math Class Abstract Ongoing advancements in Generative AI (GenAI) have boosted the potential of applying long-standing “learning-by-teaching” practices in the form of a teachable agent (TA). Despite the recognized roles and opportunities of TAs, less is known about how GenAI could create synergy or introduce challenges in TAs and how students perceived the application of GenAI in TAs. This study explored middle school students’ perceived roles, benefits, and challenges of GenAI-powered TAs in an authentic mathematics classroom. Through classroom observation, focus-group interviews, and open-ended surveys of 108 sixth-grade students, we found that students expected the GenAI-powered TA to serve as a learning companion, facilitator, and collaborative problem-solver. Students also expressed the benefits and challenges of GenAI-powered TAs. This study provides implications for the design of educational AI and AI-assisted instruction. Keywords Generative AI in education; teachable agent; learning by teaching; AI-assisted instruction; Students’ perceptionsGeneralization, Natural language processing, CollaborationMultiple Choice Question, Large Language Models, Humanin-the-loop.-
Effective And Scalable Math Support: Experimental Evidence On The Impact Of An AI- Math Tutor In Ghana
Effective And Scalable Math Support: Experimental Evidence On The Impact Of An AI- Math Tutor In Ghana Abstract This study is a preliminary evaluation of the impact of receiving extra math instruction provided by Rori, an AI-powered math tutor accessible via WhatsApp, on the math performance of approximately 500 students in Ghana. Students assigned to both the control and treatment groups continued their normal classes with identical curricula and classroom hours. Students in the treatment group were given access to a phone for one hour a week – during their study hall period – and were allowed to use Rori to independently study math. All other aspects of the groups’ in-school experience were the same. We find that the math growth scores were substantially higher for the treatment group and statistically significant (p < 0.001). The effect size of 0.36 is considered large in the context of educational interventions: approximately equivalent to an extra year of learning. Importantly, Rori works on basic mobile devices connected to low-bandwidth data networks, and the marginal cost of providing Rori is approximately $5 per student, making it a potentially scalable intervention in the context of LMICs’ education systems. While the results should be interpreted judiciously, as they only report on year 1 of the intervention, they do suggest that chat-based tutoring solutions leveraging AI could offer a cost-effective and operationally efficient approach to enhancing learning outcomes for millions of students globally.Keywords LLMs, conversational agents, mobile-learning
Retrieval-Augmented Generation To Improve Math Question-Answering: Trade-Offs Between Groundedness And Human Preference
Interactive question-answering (QA) with tutors has shown to be an effective way for middle school math students to learn. While not all students have access to a tutor, large language models make it possible to automate portions of the tutoring process–including interactive QA to support students’ conceptual discussion of mathematical concepts. Some have questioned how LLM responses can be better aligned with a school’s curriculum. In this paper, Levonian and colleagues explore how retrieval-augmented generation (RAG) can help improve response quality by incorporating textbook information and other educational resources, while also identifying trade-offs of using RAG.