A Reliance Index Framework for Proactive Disengagement in Conversational AI
DOI:
https://doi.org/10.47709/brilliance.v6i4.9656Keywords:
Conversational AI, Cognitive offloading, Over-reliance, Human-AI trust, Reinforcement learning, Adaptive assistance, User autonomy, Explainable AI, Human-computer interactionAbstract
Conversational AI assistants now support everyday cognitive work, yet optimizing them for immediate helpfulness can encourage cognitive offloading and weaken users' long-term problem-solving skills. This study introduces proactive disengagement, a design approach in which an assistant detects over-reliance and deliberately steps back to protect user independence. Reliance is quantified with a composite Reliance Index (RI) that combines behavioral telemetry with self-reported mental effort across four subscales. The RI feeds a reinforcement learning framework that shifts assistance across four levels: direct answers, guided hints, reflective prompts, and temporary disengagement. Because human-subject data collection has not yet begun, the measurement and decision pipeline was validated on a synthetic dataset of 90 profiles split across three task domains. Over eight simulated weeks, the treatment group's mean RI declined from 0.55 to 0.41, whereas the baseline group remained essentially unchanged (0.54 to 0.56). Group differences were evaluated with independent-samples t-tests and Cohen's d. Treatment profiles also showed higher independent post-test performance (72.0% vs. 60.1%), comparable task accuracy (81.1% vs. 82.8%), and a modest, non-significant reduction in trust (5.30 vs. 5.61 on a 7-point scale). These simulated results confirm that the pipeline executes coherently; they are not evidence of real-world effectiveness. The paper contributes the RI instrument, an adaptive disengagement policy, ethical design principles for a “considerate AI”, and a pre-registered protocol for forthcoming human-subject experiments.
References
Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T., & Weld, D. S. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (pp. 1–16). ACM. https://arxiv.org/abs/2006.14779
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakc?, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. https://doi.org/10.1073/pnas.2422633122
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. https://doi.org/10.1145/3449287
Cao, S., & Huang, C.-M. (2022). Understanding user reliance on AI in assisted decision-making. Proceedings of the ACM on Human-Computer Interaction, 6(CSCW2), 1–23. https://doi.org/10.1145/3555572
Cheung, G. W., Cooper-Thomas, H. D., Lau, R. S., & Wang, L. C. (2024). Reporting reliability, convergent and discriminant validity with structural equation modeling: A review and best-practice recommendations. Asia Pacific Journal of Management, 41(2), 745–783. https://doi.org/10.1007/s10490-023-09871-y
Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gaševi?, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544
Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), Article 6. https://doi.org/10.3390/soc15010006
Grinschgl, S., Papenmeier, F., & Meyerhoff, H. S. (2021). Consequences of cognitive offloading: Boosting performance but diminishing memory. Quarterly Journal of Experimental Psychology, 74(9), 1477–1496. https://doi.org/10.1177/17470218211008060
Ibrahim, L., Collins, K. M., Kim, S. S. Y., Reuel, A., Lamparth, M., Feng, K., Ahmad, L., Soni, P., El Kattan, A., Stein, M., Swaroop, S., Padmakumar, V., Sucholutsky, I., Strait, A., Yang, D., Liao, Q. V., & Bhatt, U. (2025). Measuring and mitigating overreliance to build human-compatible AI (arXiv:2509.08010). arXiv. https://arxiv.org/abs/2509.08010
Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, Article 17458. https://doi.org/10.1038/s41598-025-97652-6
Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X. H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2025). Your brain on ChatGPT: Accumulation of cognitive debt when using an AI assistant for essay writing task (arXiv:2506.08872). arXiv. https://arxiv.org/abs/2506.08872
Lee, H.-P. H., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025). The impact of generative AI on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (Article 1121, pp. 1–22). ACM. https://doi.org/10.1145/3706598.3713778
Liu, C., Zhou, Q., Shen, X., Liu, X. B., Wu, T., & Chen, X. A. (2026). Behavioral indicators of overreliance during interaction with conversational language models. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. ACM. https://doi.org/10.1145/3772318.3790332
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35, 27730–27744.
Trella, A. L., Zhang, K. W., Nahum-Shani, I., Shetty, V., Doshi-Velez, F., & Murphy, S. A. (2022). Designing reinforcement learning algorithms for digital interventions: Pre-implementation guidelines. Algorithms, 15(8), Article 255. https://doi.org/10.3390/a15080255
UNESCO. (2021). Recommendation on the ethics of artificial intelligence. https://www.unesco.org/en/artificial-intelligence/recommendation-ethics
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023). Explanations can reduce overreliance on AI systems during decision-making. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW1), Article 129. https://doi.org/10.1145/3579605
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024). Design principles for generative AI applications. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Article 378, pp. 1–22). ACM. https://doi.org/10.1145/3613904.3642466
Yu, C., Liu, J., Nemati, S., & Yin, G. (2023). Reinforcement learning in healthcare: A survey. ACM Computing Surveys, 55(1), Article 5, 1–36.
Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: A systematic review. Smart Learning Environments, 11, Article 28. https://doi.org/10.1186/s40561-024-00316-7
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Wilianto, Yuliana, Albert Suwandhi, Albert Christian, Petrus Hemat Siregar
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
