CGSQuAD: Designing Arabic FAQs Chatbot using LLMs: Case Study: The College of Graduate Studies at Kuwait University (Prof. Ayed Salman)

Limited Arabic Natural Language Processing (NLP) resources exist despite its global spread.
Addressing this issue, this work presents CGSQuAD, a new high-quality Arabic SQuAD-like Question-Answering (QA) dataset for the College of Graduate Studies at Kuwait University (CGS of KU). Moreover, it presents detailed steps for different approaches to building that dataset. Furthermore, this work examines 12 different Large Language Models (LLMs). It presents the process of exploring the built dataset using LLMs by fine-tuning multiple extractive AraElectra QA models, AutoTrainning multiple extractive AraElectra QA models, testing OpenAI generative QA models, and evaluating each model's performance using some metrics like Exact Match (EM), F1, and Semantic Similarity. The dataset domain evolves around the scope of the CGS, containing 1.5K SQuAD-like question-answer pairs about the college guidelines. The results show that most finetuned models achieve F1 scores above 90% and EM scores above 87%. The generative models are tested and evaluated using semantic similarity and human evaluation. Semantic similarity is
47.33%, 44.53%, and 52.3% for GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo respectively. Human evaluation is 80.47%, 80%, and 87.13% for GPT-3.5 Turbo, GPT-4, and GPT-4 Turbo respectively. This work demonstrates the possibility of building a chatbot that serves graduate students at KU. It can be considered a valuable resource for future researchers in the Arabic NLP community.

Supervisor: Prof. Ayed Salman
Co-Supervisor: Prof. Fawaz Alanzi
Convener: Prof. Imtiaz Ahmad
Examination Committee: Prof. Imtiaz Ahmad and Assoc. Prof. Sa’ed Abed