Analyzing Student Attention and Acceptance of Conversational AI for Math Learning: Insights from a Randomized Controlled Trial

Abstract

The significance of nurturing a deep conceptual understanding in math learning cannot be overstated. Grounded in the pedagogical strategies of induction, concretization, and exemplification (ICE), we designed and developed a conversational AI using both ruleand generation-based techniques to facilitate math learning. Serving as a preliminary step, this study employed an experimental design involving 151 U.S.-based college students to reveal students’ attention patterns, technology acceptance model, and qualitative feedback when using the developed ConvAI. Our findings suggest that participants in the ConvAI group generally exhibit higher attention levels than those in the control group, aside from the initial stage where the control group was more attentive. Meanwhile, participants appreciated their experience with the ConvAI, particularly valuing the ICE support features. Finally, qualitative analysis of participants’ feedback was conducted to inform future refinement and to inspire educational researchers and practitioners.