Open Access

Daniel Schönle, Christoph Reich, Djaffar Ould-Abdeslam, Daniela Fiedler, Ute Harms, Johannes Poser

• This study evaluates machine learning for automating the evaluation of textual responses in virtual learning environments, particularly by applying advanced linguistic enhancement techniques. Techniques such as Transformer-based data augmentation, Part-of-Speech enhanced feature selection, and LinPair tokenisation were employed. The evaluation focused on classification quality and training efficiency using a synthetically created question-and-answer dataset, characterised by its limited sample size, extensive class range, and the complexity of identifying didactical elements. The findings indicate that while the Support Vector Machine (SVM) consistently outperforms the distilled version of the large language model Bidirectional Encoder Representations from Transformers (DistilBERT) in quality metrics, the integration of linguistic elements improved DistilBERT’s performance significantly achieving a 7.62% increase in F1-Score and a 17.02% rise in Hamming-Score. Despite these gains, DistilBERT recorded lower efficiency scores compared to SVM. This suggests that while SVM excels with synthetic data, Large Language Models demonstrate substantial potential in processing complex linguistic data when provided with linguistic information. These insights confirm the viability of both approaches as effective tools for automated assessments in educational settings.