Analyzing the Use of Artificial Intelligence in EFL Listening Assessment - Pramono, Satrio Aji; Ilmi, Annisa Nurul; Fitrianingsih, Ihtiara; Utami, Amrih Bekti (Proceeding of International Seminar on Humanity, Education, and Language, 2025)
As AI integration in language assessment rises, concerns persist regarding its ability to accurately evaluate complex listening skills, such as pragmatic competence. Utilizing AI-generated questions tested on 38 English Education students, this research analyzed the technical quality of the questions, their difficulty levels, and the reliability of the test to examines the use of artificial intelligence (AI) in assessing English as a Foreign Language (EFL) listening skills. Findings reveal significant variability in question difficulty, low reliability indicated by a Cronbach’s alpha of 0.02, and the need for question revision due to low point-biserial correlations. Questionnaire responses also highlight mixed perceptions of AI’s role in language assessment. While AI holds promise for enhancing efficiency and personalization in assessment, the study emphasizes the need for a critical approach to its implementation, including further research with larger, more culturally diverse samples, and the development of advanced algorithms to better capture sociocultural and pragmatic degrees. Future work should explore hybrid models that combine AI and human evaluation to improve the fairness, reliability, and validity of language assessments.