In the area of second language (L2) listening assessment, videos as input materials have been studied for several decades. However, with the exception of He’s (2022) study, there appears to be no L2 assessment research on interactive videos (i.e., videos with tasks embedded in the video timeline), despite the growing utility of such videos in online instructional settings and reported benefits for learners (Ketsman et al., 2018).
To explore the potential of using interactive videos for assessing L2 listening comprehension, we converted two original IELTS Listening tests into interactive video listening tests with embedded test items. A counterbalanced design (2 tests by 2 formats) was employed to administer the original audio-only paper-based listening tests and the interactive video-based listening tests to 65 L2 English-speaking participants at two Canadian universities.
In this mixed-methods study, participants’ performance on the two test formats was compared quantitatively using ANOVA and many-facet Rasch model (MFRM) analysis. Item-level response time on the interactive video listening tests was analysed using a mixed-effects model. Participants’ perceptions and preferences of both formats were elicited via focus group interviews and an end-of-the-experiment questionnaire.
The results revealed a statistically significant difference in total scores between the two IELTS Listening tests, with the interactive video version found to be more difficult than the audio-only version. Bias analyses in MFRM identified the items with differential item difficulties between the formats, with most of such items being more difficult in the interactive video format than in the audio-only format. Meanwhile, participants’ response time was found to be related to item difficulty, item length, and item types. Lastly, the qualitative data analyses demonstrated that while more than half of the participants preferred the audio-only format, many participants also valued specific features that were unique to the interactive videos.
Implications are discussed in relation to the design of interactive video-based listening tests and the construct(s) of L2 listening comprehension.