Aspect Discovery and Sentiment Analysis of Bibit Application Reviews Using BERTopic and Transformer Models

Abstract

Digital investment applications generate large volumes of user reviews that contain valuable information about user experiences and perceptions. However, general sentiment analysis does not indicate the specific aspects that contribute to user’s opinions. This study aims to identify user sentiment across aspects of the Bibit application using Aspect-Based Sentiment Analysis with a data-driven approach, compare the performance of Transformer models, and evaluate the effect of Class-Weighted Loss on sentiment classification. A total of 2.842 cleaned user reviews were analyzed using BERTopic to identify candidate aspects, which were subsequently interpreted and validated with the assistance of a Large Language Model (LLM). This process resulted in eight final aspects: UX/Usability, Transaction, Investment, Error/Performance, Account/Security, Feature/Product, Customer Service, and Promotion. Subsequently, 1,139 review-aspect instances were manually annotated as ground truth and used to evaluate five Transformer models, namely BERT, RoBERTa, IndoBERT, mBERT, and XLM-RoBERTa, under Unweighted and Class-Weighted Loss conditions. The results showed that IndoBERT with Unweighted Loss achieved the best performance, with an Accuracy of 85.09% and a Macro F1 of 82.74%. Class-Weighted Loss did not consistently improve performance across the evaluated models. At the aspect level, Transaction achieved a Macro F1 of 83.67%, while Customer Service obtained the lowest score of 60.00%. These findings indicate that the data-driven approach can identify relevant aspects from user reviews, while model selection and class-weighting strategies influence classification performance across aspects.