Analyzing Chatbot Interaction Patterns and Customer Satisfaction Classification Using Text Data
DOI:
https://doi.org/10.47709/brilliance.v6i4.9810Keywords:
Chatbot, Customer Satisfaction, Machine Learning, Support Vector Machine, Text Classification, TF-IDFAbstract
This study aims to identify recurring customer interaction patterns in the chatbot service of CV Mawar and independently evaluate text-based customer satisfaction classification using the Flipkart customer support dataset.
The primary dataset consisted of 137 chatbot interaction sessions from CV Mawar. After removing 10 sessions with empty transcripts, 127 sessions were manually coded and analyzed descriptively. Because reliable customer satisfaction labels were limited, a separate Flipkart dataset containing customer remarks and Customer Satisfaction (CSAT) scores was used for machine learning. Text classification was performed using Term Frequency–Inverse Document Frequency (TF-IDF) and a linear Support Vector Machine (SVM). The model was evaluated using accuracy, balanced accuracy, precision, recall, F1-score, and a confusion matrix.
Delivery estimates and product information or usage were the most frequent categories in CV Mawar, each representing 19.69% of the coded sessions. Only three usable sessions contained genuine service ratings, all rated “Good,” while no usable written feedback was available. The Flipkart classification model achieved 79.62% accuracy, 81.92% balanced accuracy, and a macro F1-score of 78.31%, outperforming the majority-class baseline accuracy of 68.74%.
The findings highlight common customer information needs in chatbot interactions and demonstrate the potential of text classification for analyzing customer feedback. However, the limited satisfaction ratings at CV Mawar and the use of an independent Flipkart dataset restrict conclusions about satisfaction among CV Mawar customers.
References
Aldunate, Á., Maldonado, S., Vairetti, C., & Armelini, G. (2022). Understanding customer satisfaction via deep learning and natural language processing. Expert Systems with Applications, 209, 118309. https://doi.org/10.1016/J.ESWA.2022.118309
Følstad, A., & Taylor, C. (2021). Investigating the user experience of customer service chatbot interaction: a framework for qualitative analysis of chatbot dialogues. Quality and User Experience, 6(1). https://doi.org/10.1007/s41233-021-00046-5
Hadwan, M., Al-Sarem, M., Saeed, F., & Al-Hagery, M. A. (2022). An Improved Sentiment Classification Approach for Measuring User Satisfaction toward Governmental Services’ Mobile Apps Using Machine Learning Methods with Feature Engineering and SMOTE Technique. Applied Sciences (Switzerland), 12(11). https://doi.org/10.3390/app12115547
Hand, D. J., & Anagnostopoulos, C. (2023). Notes on the H-measure of classifier performance. Advances in Data Analysis and Classification, 17(1), 109–124. https://doi.org/10.1007/s11634-021-00490-3
Hsu, C. L., & Lin, J. C. C. (2023). Understanding the user satisfaction and loyalty of customer service chatbots. Journal of Retailing and Consumer Services, 71, 103211. https://doi.org/10.1016/J.JRETCONSER.2022.103211
Jenneboer, L., Herrando, C., & Constantinides, E. (2022). The Impact of Chatbots on Customer Loyalty: A Systematic Literature Review. In Journal of Theoretical and Applied Electronic Commerce Research (Vol. 17, Number 1, pp. 212–229). MDPI. https://doi.org/10.3390/jtaer17010011
Jiang, H., Cheng, Y., Yang, J., & Gao, S. (2022a). AI-powered chatbot communication with customers: Dialogic interactions, satisfaction, engagement, and customer behavior. Computers in Human Behavior, 134, 107329. https://doi.org/10.1016/J.CHB.2022.107329
Madyatmadja, E. D., Sianipar, C. P. M., Wijaya, C., & Sembiring, D. J. M. (2023). Classifying Crowdsourced Citizen Complaints through Data Mining: Accuracy Testing of k-Nearest Neighbors, Random Forest, Support Vector Machine, and AdaBoost. Informatics, 10(4). https://doi.org/10.3390/informatics10040084
Nugroho, R. A., Warsito, B., & Hakim, A. R. (2023). Performance Comparison of Support Vector Machine and Random Forest Using Delta TF-IDF on Sentiment Classification of Tiket.com Reviews. 2023 International Seminar on Intelligent Technology and Its Applications (ISITIA), 632–637. https://doi.org/10.1109/ISITIA59021.2023.10221142
Ruan, Y., & Mezei, J. (2022). When do AI chatbots lead to higher customer satisfaction than human frontline employees in online shopping assistance? Considering product attribute type. Journal of Retailing and Consumer Services, 68, 103059. https://doi.org/10.1016/J.JRETCONSER.2022.103059
Sari, H., & Adinda, R. (2023). Examining Customer Experience in Using a Chatbot. International Journal of Asian Business and Information Management, 14(1). https://doi.org/10.4018/IJABIM.322438
Sumantiawan, D. I., Suseno, J. E., & Syafei, W. A. (2023). Sentiment Analysis of Customer Reviews Using Support Vector Machine and Smote-Tomek Links For Identify Customer Satisfaction. J. Sistem Info. Bisnis, 13(1), 1–9. https://doi.org/10.21456/vol13iss1pp1-9
Wulandari, D. A., Bachtiar, F. A., & Indriati, I. (2025). Aspect Based Sentiment Analysis on Shopee Application Reviews Using Support Vector Machine. Lontar Komputer?: Jurnal Ilmiah Teknologi Informasi, 15(02), 99. https://doi.org/10.24843/lkjiti.2024.v15.i02.p03
Xu, Y., Niu, N., & Zhao, Z. (2023). Dissecting the mixed effects of human-customer service chatbot interaction on customer satisfaction: An explanation from temporal and conversational cues. Journal of Retailing and Consumer Services, 74, 103417. https://doi.org/10.1016/J.JRETCONSER.2023.103417
Zhu, Y., Zhang, J., & Liang, J. (2023). Concrete or abstract: How chatbot response styles influence customer satisfaction. Electronic Commerce Research and Applications, 62, 101317. https://doi.org/10.1016/J.ELERAP.2023.101317
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Dhafin Rizky Trisandy, Rizka Dhini Kurnia, Purwita Sari, Ari Wedhasmara, Apriansyah Putra
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
