INTEGRATION OF LARGE LANGUAGE MODELS INTO CHATBOT-BASED CUSTOMER CALL PROCESSING SYSTEMS
DOI:
https://doi.org/10.32689/maup.it.2025.2.32Keywords:
customer service processing, hybrid model, semantic classification, large language models, BLEU Score, ROUGE-L Score, chatbot, customer service optimizationAbstract
Purpose of the work. To propose a hybrid model of automated text customer service that combines semantic classification with the generative capabilities of LLM to improve the accuracy, relevance, and naturalness of responses.Methodology. A system has been developed that selects one of three response mechanisms depending on the classification of the intent and emotional tone of the query: template rules, search matching, or LLM generation. Experimental verification was performed on a corpus of 7,500 queries (authentic and synthetic); evaluation was conducted using BLEU, ROUGE-L, and expert criteria for comprehensibility, naturalness, and user trust.Scientific novelty. For the first time, it has been shown that semantic routing in conjunction with LLM forms a more robust and adaptive system capable of correctly processing complex or emotionally charged queries. The proposed model outperformed baseline approaches in terms of accuracy (92.1%), BLEU 0.78, ROUGE-L 0.81, and the lowest failure rate, and also received the highest expert ratings.Conclusions. The hybrid model reduces operator workload, increases user satisfaction, and easily adapts to customer behavior dynamics, providing empathetic and effective responses. Its practical value has been confirmed by examples from e-commerce, banking, healthcare, and public services. Implementation challenges include integration with legacy systems, regular knowledge base updates, and moderation of generated content. Further research is focused on personalization, multimodal interaction, active learning, and optimization of computing resources, laying the foundation for the development of advanced chatbots in areas with a critical need for high-quality automated support.
References
Asai A., Min S., Zhong Z., Chen D. Retrieval-based language models and applications. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), July 2023. P. 41–46. URL: https://aclanthology.org/2023.acl-tutorials.6/ (date of access: 1.05.2025).
Hong M., Song Y., Jiang D., Wang L., Guo Z., Zhang C. J. Expanding chatbot knowledge in customer service: Context- aware similar question generation using large language models. arXiv preprint arXiv:2410.12444. URL: https://arxiv.org/abs/2410.12444 (date of access: 1.05.2025).
Jo S., Seo J. ProxyLLM: LLM-driven framework for customer support through text-style transfer. arXiv preprint arXiv:2412.09916. URL: https://arxiv.org/abs/2412.09916 (date of access: 1.05.2025).
Kaddour J., Harris J., Mozes M., Bradley H., Raileanu R., McHardy R. Challenges and applications of large language models. arXiv preprint arXiv:2307.10169. URL: https://arxiv.org/abs/2307.10169 (date of access: 1.05.2025).
Kruk F., Herath S., Choudhury P. BanglAssist: A Bengali-English generative AI chatbot for code-switching and dialect- handling in customer service. arXiv preprint arXiv:2503.22283. URL: https://arxiv.org/abs/2503.22283 (date of access: 1.05.2025).
Li X., Gao M., Zhang Z., Yue C., Hu H. Rule-based data selection for large language models. arXiv preprint arXiv:2410.04715. URL: https://arxiv.org/abs/2410.04715 (date of access: 1.05.2025).
Marr B. The amazing ways Duolingo is using AI and GPT-4. URL: https://bernardmarr.com/the-amazing-ways-duolingo-is-using-ai-and-gpt-4/?utm_source=chatgpt.com (date of access: 1.05.2025).
Naveed H., Khan A. U., Qiu S., Saqib M., Anwar S., Usman M. та ін. A comprehensive overview of large language models. arXiv preprint arXiv:2307.06435. URL: https://arxiv.org/abs/2307.06435 (date of access: 1.05.2025).
Nerella S., Bandyopadhyay S., Zhang J., Contreras M., Siegel S., Bumin A. та ін. Transformers and large language models in healthcare: A review. Artificial Intelligence in Medicine, 2024. Vol. 106. Art. 102900. DOI: https://doi.org/10.1016/j.artmed.2024.102900.
Rüdel T., Leidner J. L. Control in hybrid chatbots. arXiv preprint arXiv:2311.11701. URL: https://arxiv.org/abs/2311.11701 (date of access: 1.05.2025).
Wulf J., Meierhofer J. Utilizing large language models for automating technical customer support. arXiv preprint arXiv:2406.01407. URL: https://arxiv.org/abs/2406.01407 (date of access: 1.05.2025).
Xu Z., Cruz M. J., Guevara M., Wang T., Deshpande M., Wang X., Li Z. Retrieval-augmented generation with knowledge graphs for customer service question answering. Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, July 2024. P. 2905–2909. DOI: https://doi.org/10.1145/3626772.3661370.
Yao Y., Wang P., Tian B., Cheng S., Li Z., Deng S. та ін. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172. URL: https://arxiv.org/abs/2305.13172 (date of access: 1.05.2025).
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Софія ШВЕЦЬ
This work is licensed under a Creative Commons Attribution 4.0 International License.
