Automated Framework for Communication Development in Autism Spectrum Disorder Using Whisper ASR and GPT-4o LLM

Abstract

Autism Spectrum Disorder (ASD) is a developmental condition impacting communication, social interaction, and behavior. Communication assessments for children with ASD are often conducted manually, making the process time-consuming, which can lead to delays in developing educational programs and a lack of standardization due to subjective evaluations. This study introduces an automated framework using Whisper and GPT-4o to enhance the efficiency and accuracy of evaluating communication abilities and language patterns in children with ASD. The research adopts a Research and Development (RnD) approach with the ASET model (Analyze, System Design, Execution, Testing), engaging children with mild and moderate verbal ASD and teachers from four autism schools in Daerah Istimewa Yogyakarta, Indonesia. Data were collected through interviews, classroom observations, audio recordings, and a matrix-based evaluation. Whisper was employed for automated transcription, integrated with GPT-4o for speaker diarization and communication analysis. Results showed an 89.1% reduction in analysis time compared to manual methods. Whisper achieved a low Word Error Rate (WER) for mild autism (average 5%) and a higher rate for moderate autism (average 23%). GPT-4o contributed to the process with high speaker diarization accuracy (93.9% for mild autism and 89.2% for moderate autism). The framework identified detailed communication improvements through the matrix-based evaluation, including verbal, pragmatic, semantic, sentence structure, and echolalia aspects. It provided insights previously undetected by teachers, such as specific developmental patterns in each aspect. The future research should integrate intonation and emotional analysis, refine diarization accuracy, and validate the approach across diverse populations.