On LLM Prompting techniques for Arabic Language Arithmetic Reasoning

Math word problems (MWPs) are questions described through a written context, and the solution is found by using mathematical equations. Solving this kind of problem requires reasoning thinking to determine the correct answer. MWPs is used in education, scientific research and everyday life. Since the number of Arabic speakers exceeds 300 million, this study is focused on finding the most optimal approach for solving Arabic MWPs. Nowadays, using artificial intelligence to solve mathematical problems is among the most common modern approaches. Thus, choosing an appropriate LLM and prompting technique can provide highly efficient solutions and reduce errors.This Thesis introduces three new Arabic datasets to evaluate the performance of different LLMs to solve Arabic MWPs. AGSM8K, Qudurat, and ArabicMWPs were proposed to address the lack of resources for Arabic MWPs. AGSM8K with 7473 training questions and 1319 test questions is a dataset used in the main experiment using four closed source and two open source LLMs with three prompting techniques. These techniques are Manual Chain-of-Thought, Zero-shot CoT, and Self-consistency. The findings indicate that Gpt-4o with Self-consistency method achieves highest accuracy with 97.65%, precision 71.94%, recall 71.31%, and F1 score 71.50%. An Arabic LLM called ALLaM was used in this work and the highest accuracy achieved was 84.41% with Manual CoT prompting method for ArabicMWPs dataset. Combining LLMs with prompting engineering techniques produces accurate results that resemble human thinking, making it an effective way used in developing education and scientific research in solving Arabic mathematical tasks.

Supervisor: Prof. Ayed Salman
Convener: Prof. Sa’ed Abed
Examination Committee: Dr. Abdullah Alshaibani