The study investigated whether examiners find the marking of coherence and cohesion (CC) in the IELTS Academic Writing Task 2 more difficult than the marking of the other criteria; what features of CC examiners are looking for in marking Academic Writing Task 2; the extent to which they differ in their marking of CC compared to their marking of the other criteria; whether qualifications and experience had an impact on assessment reliability; and how much current examiner training materials clarify understandings of CC.
The study involved think-aloud protocols and follow-up interviews with 12 examiners marking a set of 10 scripts, and a quantitative study with 55 examiners marking 12 scripts and completing a follow-up questionnaire.
The quantitative data revealed that examiner reliability was within the acceptable range for all four criteria. The marking of CC was slightly less reliable than the marking of Grammatical Range and Accuracy and Lexical Resource, but not significantly different to Task Response. No significant effects could be found for examiners’ qualifications or experience, which suggests that the training is effective. The findings showed that examiners found the marking of CC more difficult than the other criteria.
Examiners were conscientious in applying the band descriptors and used the terminology of the descriptors for CC most of the time. They also introduced other terms not explicitly used in the CC descriptors, such as ‘flow’, ‘structure’ and ‘linking words’, as well as the terms, ‘essay’, ‘introduction’ ‘conclusion’ and ‘topic sentence’. The introduction of terms such as these, together with variation in the degree to which examiners focused on particular features of CC, has implications for the construct validity of the test.
Suggestions for improving the construct validity include: possible fine tuning of the CC band descriptors; clarification of the expected rhetorical genre; further linguistic research to provide detailed analysis of CC in sample texts; and refinements to the training materials, including a glossary of key terms and sample scripts showing all cohesive ties.