Math Multiple Choice Question Generation Via Human-Large Language Model Collaboration Abstract Multiple choice questions (MCQs) are a popular method for evaluating students’ knowledge due to their efficiency in administration and grading. Crafting high-quality math MCQs is a labor-intensive process that requires educators to formulate precise stems and plausible distractors. Recent advances in large language models (LLMs) have sparked interest in automating MCQ creation, but challenges persist in ensuring mathematical accuracy and addressing student errors. This paper introduces a prototype tool designed to facilitate collaboration between LLMs and educators for streamlining the math MCQ generation process. We conduct a pilot study involving math educators to investigate how the tool can help them simplify the process of crafting high-quality math MCQs. We found that while LLMs can generate well-formulated question stems, their ability to generate distractors that capture common student errors and misconceptions is limited. Nevertheless, a human-AI collaboration has the potential to enhance the efficiency and effectiveness of MCQ generation. Keywords Multiple Choice Question, Large Language Models, Human in-the-loop.Generalization, Natural language processing, CollaborationMultiple Choice Question, Large Language Models, Humanin-the-loop.-
The Neglected 15%: Positive Effects Of Hybrid Human-AI Tutoring Among Students With Disabilities
The Neglected 15%: Positive Effects Of Hybrid Human-AI Tutoring Among Students With Disabilities Abstract Incorporating human tutoring with AI holds promise for supporting diverse math learners. In the U.S., approximately 15% of students receive special education services, with limited previous research within AIED on the impact of AI-assisted learning among students with disabilities. Previous work combining human tutors and AI suggests that students with lower prior knowledge, such as lacking basic skills, exhibit greater learning gains compared to their more knowledgeable peers. Building upon this finding, we hypothesize that hybrid human-AI tutoring will have positive effects among students in inclusive settings, classrooms containing both students with and without identified disabilities. To investigate this hypothesis, we conduct a two-study quasi-experiment across two urban, low-income middle schools: one in Pennsylvania with 362 students and another in California comprised of 733 students, involving 27% and 16% of students with disabilities, respectively. Our findings indicate hybrid human-AI tutoring has positive effects on learning processes and outcomes among all students, including students with disabilities. The motivational benefits of the human tutoring treatment over a control group using solely math software show up in greater increases in practice and skill proficiency during math software use for students with disabilities than other students. In addition, the treatment yields significantly higher pre-post learning gains for all including students with disabilities whose relative gains trend higher and are statistically at least as high. These compelling findings support the promise of hybrid human-AI solutions and emphasize the collaborative focus among AIED to support diverse learners. Keywords Human-AI tutoring; Learning analytics; Equity; Inclusion.
Analyzing Student Attention And Acceptance Of Conversational AI For Math Learning: Insights From A Randomized Controlled Trial
Analyzing Student Attention And Acceptance of Conversational AI For Math Learning: Insights From A Randomized Controlled Trial Abstract The significance of nurturing a deep conceptual understanding in math learning cannot be overstated. Grounded in the pedagogical strategies of induction, concretization, and exemplification (ICE), we designed and developed a conversational AI using both ruleand generation-based techniques to facilitate math learning. Serving as a preliminary step, this study employed an experimental design involving 151 U.S.-based college students to reveal students’ attention patterns, technology acceptance model, and qualitative feedback when using the developed ConvAI. Our findings suggest that participants in the ConvAI group generally exhibit higher attention levels than those in the control group, aside from the initial stage where the control group was more attentive. Meanwhile, participants appreciated their experience with the ConvAI, particularly valuing the ICE support features. Finally, qualitative analysis of participants’ feedback was conducted to inform future refinement and to inspire educational researchers and practitioners. Keywords Technology design and development, Conversational AI, Large language models, Math learningGeneralization, Natural language processing, CollaborationMultiple Choice Question, Large Language Models, Humanin-the-loop.-
Automated Feedback For Student Math Responses Based On Multi-Modality And Fine-Tuning
Automated Feedback For Student Math Responses Based On Multi-Modality And Fine-Tuning Abstract Open-ended mathematical problems are a commonly used method for assessing students’ abilities by teachers. In previous automated assessments, natural language processing focusing on students’ textual answers has been the primary approach. However, mathematical questions often involve answers containing images, such as number lines, geometric shapes, and charts. Several existing computer-based learning systems allow students to upload their handwritten answers for grading. Yet, there are limited methods available for automated scoring of these image-based responses, with even fewer multi-modal approaches that can simultaneously handle both texts and images. In addition to scoring, another valuable scaffolding to procedurally and conceptually support students while lacking automation is comments. In this study, we developed a multi-task model to simultaneously output scores and comments using students’ multi-modal artifacts (texts and images) as inputs by extending BLIP, a multi-modal visual reasoning model. Benchmarked with three baselines, we fine-tuned and evaluated our approach on a dataset related to open-ended questions as well as students’ responses. We found that incorporating images with text inputs enhances feedback performance compared to using texts alone. Meanwhile, our model can effectively provide coherent and contextual feedback in mathematical settings. Keywords Generalization, Natural language processing, Collaborationanalytiopen-ended response, auto-scoring, automated comment, image response, multi-modality, fine-tuning
Retrieval-Augmented Generation To Improve Math Question-Answering: Trade-Offs Between Groundedness And Human Preference
Interactive question-answering (QA) with tutors has shown to be an effective way for middle school math students to learn. While not all students have access to a tutor, large language models make it possible to automate portions of the tutoring process–including interactive QA to support students’ conceptual discussion of mathematical concepts. Some have questioned how LLM responses can be better aligned with a school’s curriculum. In this paper, Levonian and colleagues explore how retrieval-augmented generation (RAG) can help improve response quality by incorporating textbook information and other educational resources, while also identifying trade-offs of using RAG.
Rewriting Math Word Problems With Large Language Models
In a recent study, math problems in Carnegie Learning’s MATHia adaptive learning software were rewritten by human authors and AI to improve clarity. Findings showed that readers spent less time reading rewritten human content and achieved higher mastery than did readers who read the original content. The team conducting the study also used GPT-4 to rewrite the same set of math word problems with the same guidelines that the human authors used. comparing zero-shot, few-shot and chain-of-thought prompting strategies. Overall, report analysis of human-written, original and GTP-written problems showed that GTP rewrites have the most optimal readability, lexical diversity and cohesion scores, though used more low frequency words. Carnegie Learning plans to present their outputs at randomized field trials in MATHia.
Scenario-Based Training And On-The-Job Support For Equitable Mentoring
Personalized Learning2 (PL2) is a professional mentoring platform created by researchers at Carnegie Mellon. Its goal is to improve workplace efficiency and utilize personalized learning to teach through situation-based instruction. PL2 combines both AI and mentor driven research training to help under-trained tutors with personalized learning. This platform includes social-emotional learning, math content and culturally responsive teaching practices to address the gap between historically marginalized students by training tutors to be more efficient and productive. PL2 offers a lower cost option for deliberate practice in order to increase impact and learning capacity of tutors.
Development Of Scenario-Based Mentor Lessons
This demonstration shows the recent advancement of scenario based tutor training and its focus on using the learn-by-doing approach. The 15 minute lessons outlined in this study use the predict-observe-explain inquiry method to develop tutor skills in helping student motivation. These methods are being developed within the Personalized Learning2 (PL2) program. PL2 is an app that combines student software with human tutors to improve mentoring ability. Enhancing mentor training will help to increase student ability while also maintaining low costs. This form of training works best when tutors have scenario based practice with response specific feedback.
An Evaluation Of Perceptions Regarding Mentor Competencies For Technology-Based Personalized Learning
This study discusses the development of Personalized Learning 2 (PL2), an online human mentoring system. PL2 uses student math learning data and mentor input to write custom feedback. This particular research is focused on finding a more efficient and research-based way to organize resources for PL2. 18 PL2 partner members completed a survey that revealed that Engaging and Motivating Students was the most important skill and Underingstanding Educational Norms and Policies was the least important. Reorganization will optimize mentor training and their ability to help students overcome barriers.
Computer-Supported Human Mentoring
In a recent study, math problems in Carnegie Learning’s MATHia adaptive learning software were rewritten by human authors and AI to improve clarity. Findings showed that readers spent less time reading rewritten human content and achieved higher mastery than did readers who read the original content. The team conducting the study also used GPT-4 to rewrite the same set of math word problems with the same guidelines that the human authors used. comparing zero-shot, few-shot and chain-of-thought prompting strategies. Overall, report analysis of human-written, original and GTP-written problems showed that GTP rewrites have the most optimal readability, lexical diversity and cohesion scores, though used more low frequency words. Carnegie Learning plans to present their outputs at randomized field trials in MATHia.