Developing an ESD-Aligned Scientific Literacy Instrument for Pre-Service Physics Teachers: A Pilot  Rasch Measurement Study

Qisthi  Fariyani; Ani Rusilowati; Wiyanto

doi:10.21009/1.12113

Developing an ESD-Aligned Scientific Literacy Instrument for Pre-Service Physics Teachers: A Pilot Rasch Measurement Study

Authors

Qisthi Fariyani
Ani Rusilowati Faculty of Mathematics and Natural Sciences, Universitas Negeri Semarang, Indonesia
Wiyanto Faculty of Mathematics and Natural Sciences, Universitas Negeri Semarang, Indonesia.

DOI:

https://doi.org/10.21009/1.12113

Keywords:

Education for Sustainable Development, Rasch Measurement, Scientific Literacy, Sustainability Assessment

Abstract

Scientific literacy is important for helping students respond to global sustainability challenges. However, existing scientific literacy instruments have not fully integrated Education for Sustainable Development (ESD) and have limited psychometric evidence. This study aimed to develop an ESD-aligned scientific literacy instrument for pre-service physics teachers and examine its preliminary measurement quality using the Rasch model. The study used a research and development design based on the 4D model. The instrument consisted of 45 contextual items related to sustainability issues, including climate change, energy, and environmental problems. Six experts evaluated the content validity of the items using Aiken’s V. The instrument was then administered to 30 pre-service physics teachers. The data were analyzed using the Rasch model with Winsteps. The results showed high content validity, with Aiken’s V values ranging from 0.89 to 1.00. The Rasch analysis showed a person reliability of 0.80 and a Cronbach’s alpha of 0.84. However, item reliability was 0.45 and item separation was 0.90, indicating that the item calibration was still preliminary. The item-fit results showed Outfit MNSQ values of 0.58–1.37 and Outfit ZSTD values of −1.55 to 1.84. Point Measure Correlation values ranged from 0.17 to 0.48, indicating that some items had weaker relationships with the overall measure. The Wright Map showed imperfect alignment between person ability and item difficulty. PCA of residuals showed 16.8% raw variance explained by measures and a first-contrast eigenvalue of 4.40, which did not provide strong evidence of unidimensionality. Overall, the findings provide preliminary evidence that the instrument has potential for assessing ESD-aligned scientific literacy. Further studies with larger and more diverse samples are needed to confirm its measurement quality.

References

Allari, R. S., Hamdan, K., Albqoor, M. A., & Shaheen, A. (2022). Information literacy: Assessment of undergraduate and graduate nursing students. Reference Services Review, 50(2), 211–221. Scopus. https://doi.org/10.1108/RSR-09-2021-0052

Alom, M. M., & Ramalingappa, V. (2025). Development and Validation of a Digital Literacy Scale and the Evaluation of Post-Graduate Teacher Educators’ Digital Literacy. International Journal of Information and Education Technology, 15(10), 2249–2263. Scopus. https://doi.org/10.18178/ijiet.2025.15.10.2421

Balan, H. R., Bulay, M. L., & Raganas, N. S. (2025). Climate science literacy evaluation of senior high school students: A platform for science curriculum enhancement. International Journal of Evaluation and Research in Education, 14(5), 3650–3664. Scopus. https://doi.org/10.11591/ijere.v14i5.33760

Baltikian, M., Kärkkäinen, S., & Kukkonen, J. (2024). Assessment of scientific literacy levels among secondary school students in Lebanon: Exploring gender-based differences. Eurasia Journal of Mathematics, Science and Technology Education, 20(3). Scopus. https://doi.org/10.29333/ejmste/14279

Bond, T. (2015). Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Third Edition (3rd edition). Routledge. https://doi.org/10.4324/9781315814698

Boone, W. J., Staver, J. R., & Yale, M. S. (2014). Rasch Analysis in the Human Sciences. Springer. https://doi.org/10.1007/978-94-007-6857-4

Carroll, A. J., Borycz, J., Stephens, N. K., & Lowery, A. R. (2025). Scaffolded information literacy and data literacy instruction within undergraduate science and engineering laboratory courses: A longitudinal assessment. Journal of Academic Librarianship, 51(5). Scopus. https://doi.org/10.1016/j.acalib.2025.103124

Chang, T.-C., Lyu, Y.-M., Wu, H.-C., & Min, K.-W. (2024). Introduction of Taiwanese literacy-oriented science curriculum and development of an aligned scientific literacy assessment. Eurasia Journal of Mathematics, Science and Technology Education, 20(1). Scopus. https://doi.org/10.29333/ejmste/13930

Chung, C.-C., Lou, S.-J., & Cheng, Y.-M. (2024). Development and Assessment of Innovative Pedagogical Approaches through a Starlink DBL Course. International Journal of Engineering Education, 40(2), 255–267. Scopus.

Coppi, M., Fialho, I., & Cid, M. (2023). Developing a Scientific Literacy Assessment Instrument for Portuguese 3rd Cycle Students. Education Sciences, 13(9). Scopus. https://doi.org/10.3390/educsci13090941

Coppi, M., Fialho, I., & Cid, M. (2024). Validity evidence based on internal structure of a pilot scientific literacy assessment instrument. Meta: Avaliacao, 16(51), 364–385. Scopus. https://doi.org/10.22347/2175-2753v16i51.4330

Darman, D. R., Suhandi, A., Kaniawati, I., Samsudin, A., & Wibowo, F. C. (2024). Development and Validation of Scientific Inquiry Literacy Instrument (SILI) Using Rasch Measurement Model. Education Sciences, 14(3). Scopus. https://doi.org/10.3390/educsci14030322

Doan, A. T., Nguyen, P. V., Tran, H. T. B., Kongkaew, C., Kristina, S. A., Endarti, D., Sriram, S., & Vo, T. Q. (2025). Vietnamese version of the Health Literacy Questionnaire: Translation, linguistic validation, and assessment of cross-cultural adaptation. Health Promotion International, 40(5). Scopus. https://doi.org/10.1093/heapro/daaf149

Dong, F., & Kula, M. C. (2023). Digital device use and scientific literacy: An examination using Programme for International Student assessment (PISA) 2015 data. Education Economics, 31(3), 288–312. Scopus. https://doi.org/10.1080/09645292.2022.2063797

Dudley, D., & Cairney, J. (2023). How the Lack of Content Validity in the Canadian Assessment of Physical Literacy Is Undermining Quality Physical Education. Journal of Teaching in Physical Education, 42(4), 647–654. Scopus. https://doi.org/10.1123/jtpe.2022-0063

Escoz-Roldán, A., Gutiérrez-Pérez, J., & Meira-Cartea, P. A. (2020). Water and climate change, two key objectives in the agenda 2030: Assessment of climate literacy levels and social representations in academics from three climate contexts. Water (Switzerland), 12(1). Scopus. https://doi.org/10.3390/w12010092

Fan, W., & Li, Y. (2025). Assessment of postgraduate digital research literacy: Current competencies, desired skills, and enhancement strategies. Education and Information Technologies, 30(15), 21551–21571. Scopus. https://doi.org/10.1007/s10639-025-13622-5

Fang, X., & Zhang, Z. (2026). Examining the Contradictions Between the Theory and Practice of Physical Literacy Assessment: Toward a New Assessment Framework. Journal of Teaching in Physical Education, 45(1), 52–65. Scopus. https://doi.org/10.1123/jtpe.2023-0350

Fauzi, A., Saeﬁ, M., Kristiana, E., Adi, W. C., & Lestariani, N. (2021). Factor and Rasch Analysis on COVID-19 Genetics Literacy Assessment Instrument. Eurasia Journal of Mathematics, Science and Technology Education, 17(11). Scopus. https://doi.org/10.29333/ejmste/11264

Gellermann, D., Michel, H., & Harms, U. (2025). Between Two Worlds: Locating Climate Literacy between Modern Educational Frameworks and Assessment Needs. Mind, Brain, and Education, 19(2), 61–72. Scopus. https://doi.org/10.1111/mbe.70003

Istyadji, M. (2023). Conception of scientific literacy in the development of scientific literacy assessment tools: A systematic theoretical review. Journal of Turkish Science Education, 20(2), 281–308. Scopus. https://doi.org/10.36681/tused.2023.016

Kaur, M., & Dhiman, B. (2025). Performance Evaluation and Scientific Mapping in Sustainability-Focused Green Finance and Literacy. Anuario Do Instituto de Geociencias, 48. Scopus. https://doi.org/10.11137/1982-3908_2025_48_63945

Kruit, P. M., Bredeweg, B., & Nieuwelink, H. (2025). Enhancing students’ argumentation skills, content knowledge, and Nature of Science understanding through a web-based educational instrument in the context of socio-scientific issues. International Journal of Science Education, 47(6), 749–768. Scopus. https://doi.org/10.1080/09500693.2024.2348824

Linacre, J. M., Heinemann, A. W., Wright, B. D., Granger, C. V., & Hamilton, B. B. (1994). The structure and stability of the functional independence measure. Archives of Physical Medicine and Rehabilitation, 75(2), 127–132. https://doi.org/10.1016/0003-9993(94)90384-0

Liu, Y., Wang, J., Zhang, Z., Wang, J., Luo, T., Lin, S., Li, J., & Xu, S. (2024). Development and validation of an instrument for measuring civic scientific literacy. Disciplinary and Interdisciplinary Science Education Research, 6(1). Scopus. https://doi.org/10.1186/s43031-023-00092-3

O’Brien, T. C., Palmer, R., & Albarracin, D. (2021). Misplaced trust: When trust in science fosters belief in pseudoscience and the benefits of critical evaluation. Journal of Experimental Social Psychology, 96. Scopus. https://doi.org/10.1016/j.jesp.2021.104184

Okada, A. (2024). A self-reported instrument to measure and foster students’ science connection to life with the CARE-KNOW-DO model and open schooling for sustainability. Journal of Research in Science Teaching, 61(10), 2362–2404. Scopus. https://doi.org/10.1002/tea.21964

Ophoff, J. G., & Egger, C. (2021). Assessment Of German And Austrian Students’ Educational Research Literacy: Validation Of A Competency Test Based On Cross-National Comparisons. Studia Paedagogica, 26(4), 27–45. Scopus. https://doi.org/10.5817/SP2021-4-2

Qiao, C., Chen, Y., Guo, Q., & Yu, Y. (2024). Understanding science data literacy: A conceptual framework and assessment tool for college students majoring in STEM. International Journal of STEM Education, 11(1). Scopus. https://doi.org/10.1186/s40594-024-00484-5

Rico, A., Palacios-Agúndez, I., & Gonzalez-Larrañaga, G. (2026). A wikipedia-based Authentic Task About Biodiversity in Pre-service Primary Science Teacher Education: Evaluation, Re-design and Validation of a Multidimensional Rubric for K-12 Content. Research in Science Education, 56(2), 551–578. Scopus. https://doi.org/10.1007/s11165-025-10286-6

Sari, N. K., Mahanal, S., & Lestari, S. R. (2025). Constructing and validating a novel instrument to measure human physiology literacy in biology students. Journal of Applied Research in Higher Education, 1–23. Scopus. https://doi.org/10.1108/JARHE-07-2025-0523

Subali, B., Faizah, Z., & Sidiq, M. (2023). Indonesian national assessment support: Can RE-STEM Android app improve students’ scientific literacy skills? International Journal of Evaluation and Research in Education, 12(3), 1399–1407. Scopus. https://doi.org/10.11591/ijere.v12i3.24794

Sumintono, B., & Widhiarso, W. (2015). Aplikasi Pemodelan Rasch Pada Assessment Pendidikan. Trim Komunikata Publishing House.

Suwono, H., Maulidia, L., Saefi, M., Kusairi, S., & Yuenyong, C. (2022). The Development and Validation of an Instrument of Prospective Science Teachers’ Perceptions of Scientific Literacy. Eurasia Journal of Mathematics, Science and Technology Education, 18(1). Scopus. https://doi.org/10.29333/EJMSTE/11505

Thiagarajan, T. (1974). Instructional Development for Training Teachers of Exceptional Children: A Sourcebook.

Tseng, A. S., Bonilla, S., & MacPherson, A. (2021). Fighting “bad science” in the information age: The effects of an intervention to stimulate evaluation and critique of false scientific claims. Journal of Research in Science Teaching, 58(8), 1152–1178. Scopus. https://doi.org/10.1002/tea.21696

Watson, E., Gares, S., & Rempel, B. (2025). Is the Course Working? An Account of Our Development of an Instrument to Measure the Science Attitudes and Skills of Undergraduate Students Outside of Science Disciplines. Teaching and Learning Inquiry, 13. Scopus. https://doi.org/10.20343/teachlearninqu.13.29

Wong, G. K.-W., Reichert, F., & Law, N. (2023). Reorienting the assessment of digital literacy in the twenty-first century: A product-lifecycle and experience dependence perspective. Educational Technology Research and Development, 71(6), 2389–2412. Scopus. https://doi.org/10.1007/s11423-023-10278-1

Zhang, L., Liu, X., & Feng, H. (2023). Development And Validation Of An Instrument For Assessing Scientific Literacy From Junior To Senior High School. Disciplinary and Interdisciplinary Science Education Research, 5(1). Scopus. https://doi.org/10.1186/s43031-023-00093-2

Downloads

Published

2026-06-30

How to Cite

Fariyani, Q. ., Rusilowati, A., & Wiyanto. (2026). Developing an ESD-Aligned Scientific Literacy Instrument for Pre-Service Physics Teachers: A Pilot Rasch Measurement Study . Jurnal Penelitian & Pengembangan Pendidikan Fisika, 12(1), 149–166. https://doi.org/10.21009/1.12113