|
This study presents ChatMIT, a domain-specific conversational system designed to support undergraduate recruitment at the Faculty of Mathematics and Information Technology at Lublin University of Technology. Three Retrieval-Augmented Generation architectures, namely Naive RAG, Hybrid RAG, and Self-RAG, were implemented and evaluated across three levels of question complexity using three large language models of varying parameter scales: Gemma 3 (27B), Llama 3 (70B), and gpt-oss (120B). Response quality was assessed by independent domain experts according to two criteria: factual correctness and linguistic quality, each evaluated using a five-point scale developed by the authors. The results indicate that the choice of RAG architecture is the primary factor influencing response quality, exerting a greater impact than model scale. Self-RAG substantially outperformed both Hybrid RAG and Naive RAG across all levels of question complexity, achieving mean factual correctness scores of 4.67/5, 2.77/5, and 2.33/5, as well as linguistic quality scores of 4.90/5, 4.37/5, and 4.33/5, respectively. Furthermore, the adoption of a zero-shot variant of Self-RAG constitutes a methodological contribution of this study, demonstrating that the adaptive properties of the original architecture can be preserved while significantly reducing deployment complexity, without the need for supervised training of a dedicated critic model. The findings confirm that RAG-based conversational systems represent an effective approach to automating information access in academic environments and offer considerable potential for deployment across diverse institutional domains and faculties.
|
