This study investigates the cognitive validity of two samples of IELTS lecture-listening material taken from past papers. In one condition, participants were asked to undertake the original test, and to provide a retrospective verbal report, explaining why they had chosen the answers that they had favoured. In a second condition, they were asked to take notes under the less constrained circumstances that obtain during a lecture, and then to report on them. The material was distributed on an ABAB principle so that the possible effects of recording and test method could be compared.
The scores obtained by individuals under ‘test’ conditions were compared with the extent to which the same individuals showed themselves capable of accurately reporting units of information in the freer ‘lecture’ condition. No clear correlation was demonstrated.
The verbal reports were then examined for evidence of the cognitive processes adopted by participants under test conditions, and were matched against conventional psycholinguistic accounts of first-language listening (see, for example, Brown 1995, Field 2009). A distinction was made between normal processes which might equally well be adopted by a native academic listener, strategic behaviour which aimed to compensate for problems of understanding; and test-specific behaviour representing the user’s response to characteristics of the test.
Evidence of the last raised concerns about cognitive validity. The protocols showed participants adopting specific routines that were tailored to the test method. They also provided considerable evidence of participants favouring test-wise strategies and attempting to exploit loopholes in the format of the test such as the availability of questions in a written form.
A third line of enquiry investigated participants’ responses to listening under the two conditions, in order to establish which they had found the more demanding. An unexpected result was the number of participants who found lecture listening less demanding than undertaking the test. Possible reasons are explored.