The online judge system is a specialized learning management systems for program assignments commonly used in programming education. The behaviour data recorded by the system can reflect the most authentic thinking and learning situation of students. Mining and analysis the learning behaviour data can help teachers discover students with learning difficulties in a timely manner and design personalized teaching activities. Although the important value of learning behaviour data analysis in promoting the development of online education has attracted widespread attention and reflection, the mining techniques for these data have been underexplored. This study investigated the data on student self-directed learning behaviour in online programming learning and studied the identification method of self-directed learning behaviour patterns. We also explored the applicability of student behaviour differentiation analysis in academic performance warning, personalized teaching activity assistance design, and learning situation analysis. We collected online behaviour data from 152 students in the Algorithm Design and Analysis course at a comprehensive university in China. Students were asked to independently submit programs for 17 programming questions within 18 weeks, and the program used the automatic evaluation results on the online judge system as their regular grades for the course. In the end, 8259 program submission records were collected. We constructed an online programmer learning engagement evaluation model and event analysis, correlation analysis, and K-means algorithm to cluster these behavioural data to determine the differences in student learning engagement patterns. Using the model proposed in this study, 8259 program submission records were processed, and five significantly different programming learning behaviour patterns were identified, namely: progressive high input, high performance input, random input, low performance input, and low input. Thereby demonstrating the usability of the model in evaluating learner differentiated learning engagement patterns. We also propose application suggestions for differentiated student behaviour patterns in teaching scenarios.

Zuhua Dai, Shiya Lv, Qicheng Zhang. A Programming Learning Engagement Assessment Model and Teaching Application Based on Online Judgement System. Frontiers in Educational Research (2024) Vol. 7, Issue 10: 223-234. https://doi.org/10.25236/FER.2024.071036.

[1] Chai Y M, Lei Ch F. A survey of online learning behaviour research based on data mining technology[J]. Applied Research of Computers, 2018,35(05): 1287-1293. 

[2] Aydoğdu Ş. Predicting student final performance using artificial neural networks in online learning environments[J]. Education and Information Technologies, 2020,25(03): 1913-1927.

[3] Qiu F, Zhang G, Sheng X, Jiang L, Zhu L, Xiang Q, Chen P K. Predicting students’ performance in e-learning using learning process and behaviour data[J]. Scientific Reports, 2022,12(1): 453. 

[4] Tong Y, Zhan Z H. An evaluation model based on procedural behaviours for predicting MOOC learning performance): students’ online learning behaviour analytics and algorithms construction[J]. Interactive Technology and Smart Education, 2023,20(03): 291-312.

[5] Lin X. Clustering Research Based on Feature Selection in The Behaviour Analysis of MOOC Users [J]. Journal of Information Hiding and Multimedia Signal Processing, 2019,10: 147-155.

[6] Yan N, Au O T S. Online learning behaviour analysis based on machine learning[J]. Asian association of open universities journal, 2019,14(02): 97-106.

[7] Li Z H, Zhang Z L, Liu H. Research on evaluation method of online learning input based on Model integration [J]. China Distance Education, 2020(10): 9-16+60.

[8] Hu Q T, Wu W Y, Feng G T, Pan F, Qiu K X. Research on interpretability analysis of multimodal learning behaviour supported by deep learning[J]. Journal of Audio-Visual Education Research, 2021.42(11): 77-83.

[9] Wasik S, Antczak M, Badura J, Laskowski A, Sternal T. A Survey on Online Judge Systems and Their Applications [J]. ACM Computing Surveys (CSUR), 2018,51(01): 1-34.

[10] Xu B, Yan Sh, Jiang X, Feng S. SCFHpp: A student analysis model to identify students’ programming levels in online judge systems[J]. Symmetry, 2020,12(4): 601. 

[11] Bilegjargal D, Hsueh N L. Understanding Students’ Acceptance of Online Judge System in Programming Coursespp, A Structural Equation Modelling Approach[J]. IEEE Access, 2021,09: 152606-152615.

[12] Zhang Y, Li Z, Du B, Wu Y, Jiang H. Data Analysis of Online Judge System-Based Teaching Model[C]. In International Conference on Computer Science and Education, edited by W. Hong, and Y. Weng (2022): 531-543, Singapore. Springer Nature Singapore.

[13] Wang J Sh, Lin P Ch, Tang Zh Y, Chen S. How problem difficulty and order influence programming education outcomes in online judge systems[J]. Heliyon, 2023,9(11): e20947. 

[14] Yu J H, Chang X Z, Liu W, Huan Z. An online integrated programming platform to acquire students' behaviour data for immediate feedback teaching[J]. Computer Applications in Engineering Education, 2023, 31(3): 520-536. 

[15] Chuang Y, Chang T. H. Y. Analyzing Novice and Competent Programmers’ Problem-Solving Behaviours Using an Automated Evaluation System[J]. Science of Computer Programming, 2024, 103138.

[16] Sunday K, Ocheja P, Hussain S, Oyelere S, Samson B, Agbo F. Analyzing student performance in programming education using classification techniques[J]. International Journal of Emerging Technologies in Learning (iJET), 2020,15(02): 127-144. 

[17] Mai T T, Bezbradica M, Crane M. Learning behaviours data in programming education. Community analysis and outcome prediction with cleaned data[J]. Future Generation Computer Systems, 2022, 127: 42-55.

[18] Rico-Juan J R, Sánchez-Cartagena V M, Valero-Mas J J, Gallego A. Identifying Student Profiles Within Online Judge Systems Using Explainable Artificial Intelligence[J]. IEEE Transactions on Learning Technologies, 2023,16(6): 955-969. 

[19] Fu Q, Bai X, Zheng Y, Du R, Wang D, Zhang T. VisOJ: Real-time visual learning analytics dashboard for online programming judge[J]. The Visual Computer, 2023,39(6): 2393-2405. 

[20] Liu Y, Zhu R, Gao M, Personalized Programming Guidance Based on Deep Programming Learning Style Capturing[C]. In: Hong, W, Kanaparan, G. (eds) Computer Science and Education. Computer Science and Technology. ICCSE 2023. Communications in Computer and Information Science, 2023, Springer, Singapore. 

[21] Wu F T, Zhang Q. Learning behaviour engagement: definition, analytical framework and theoretical model[J]. China Educational Technology, 2018(01): 35-41.

[22] Jiang T C, Li J Y, Lv H Zh. Research of similarity model based on learning behaviour time series[J]. Journal of Qiqihar University (Natural Science Edition), 2019,35(6): 1-3+7.

[23] Lin X Q. Application of K⁃means clustering algorithm to online learning behaviour research in the era of big data[J]. Electronic Design Engineering, 2021.29(18): 181-184+193. 

[24] Bai L, Hu Y L, Zheng L. Design and practice of blended teaching model based on Learning Behaviour Analysis[J]. Computer Engineering and Science, 2018,40: 42-46.

[25] Wang Z R. Design and practice of the blended learning model based on an online judge system[J]. Journal of Computer Education, 2018(8): 126-129.