Evaluating Informatics Engineering Learning Outcomes Using Weighted Rubrics to Compare Teacher and LLM Scores

Abstract

The increasing use of large language models (LLMs) in educational assessment highlights the need to examine scoring consistency between automated and teacher evaluations. This study analyzes assessment-score consistency and compares teacher and LLM evaluations using a weighted-rubric approach. A descriptive-analytical method was applied to the HARMOGEN-R dataset. First, 32,805 evaluation records were examined to assess consistency between recorded total scores and aggregated criterion scores. Second, 345 paired teacher–LLM responses were analyzed across five criteria using equal weights of 0.20. Evaluation metrics included mean absolute error (MAE), root mean squared error (RMSE), mean score difference, Pearson correlation, and Spearman rank correlation. The results showed that 32,618 evaluations (99.43%) had matching recorded totals and summed criterion scores. In the paired analysis, the mean teacher score was 52.583, compared with 49.996 for LLM scores. The MAE was 4.168, the RMSE was 9.328, and the mean difference (LLM minus teacher) was ?2.587. Pearson and Spearman correlations were 0.9781 and 0.9863, respectively (both p < 0.001). Despite these strong positive associations, score differences indicate that correlation does not necessarily establish agreement or interchangeability. The weighted-rubric approach provides a transparent method for score aggregation and identifying assessment discrepancies. However, equal weights were used as a baseline, and the criteria were not validated against official Course Learning Outcomes (CLOs) or Program Learning Outcomes (PLOs). Future research should investigate alternative weighting schemes, validate learning-outcome mapping, and assess score agreement across broader educational contexts.