Article in:Interpreting: Online-First Articles
Assessing spoken-language interpreting
The method of comparative judgement
In this study, we applied and evaluated a scoring method known as comparative judgement to assess spoken-language interpreting. This methodological exploration represents an extension of previous efforts to optimise scoring methods for assessing interpreting. Essentially, comparative judgement requires judges to compare two similar objects and make a binary decision about their relative qualities. To evaluate its reliability, validity and usefulness in the assessment of interpreting, we recruited two groups of judges (novice and experienced) to assess 66 two-way English/Chinese interpretations based on a computerised comparative judgement system. Our data analysis shows that the new method produced reliable and valid results across judge types and interpreting directions. However, the judges held polarised opinions about the method’s usefulness: while some considered it convenient, efficient and reliable, the opposite view was expressed by others. We discuss the results by providing an integrated analysis of the data collected, outline the perceived drawbacks and propose possible solutions to the drawbacks. We call for more evidence-based, substantive investigation into comparative judgement as a potentially useful method for assessing spoken-language interpreting in certain settings.
Keywords: spoken-language interpreting, comparative judgement, scoring method, interpreting assessment, interpreting quality
Published online: 01 October 2021
Barik, H. C.
Bradley, R. A. & Terry, M. E.
(2015) Investigating the reliability of adaptive comparative judgement. https://www.cambridgeassessment.org.uk/Images/232694-investigating-the-reliability-of-adaptive-comparative-judgment.pdf (Accessed 9 June 2021).
Bramley, T., Bell, J. & Pollitt, A.
(2012) Technical report on the development and pilot testing of the Certified Healthcare Interpreter™ (CHI™) examination for Arabic and Mandarin. https://cchicertification.org/uploads/CCHI_Technical_Report-CHI-Arabic_Mandarin.pdf (Accessed 9 June 2021).
Chen, J., Yang, H-B. & Han, C.
Cheung, A. K. F.
(2017) Using analytic rating scales to assess English–Chinese bi-directional interpreting: A longitudinal Rasch analysis of scale utility and rater behaviour. Linguistica Antverpiensia, New Series: Themes in Translation Studies 16, 196–215. https://lans-tts.uantwerpen.be/index.php/LANS-TTS/article/view/429/407
Hartley, A., Mason, I., Peng, G. & Perez, I.
(2003) Peer- and self-assessment in conference interpreter training. Centre for Languages, Linguistics and Area Studies. https://researchportal.hw.ac.uk/en/publications/peer-and-self-assessment-in-conference-interpreting-training
International School of Linguists
(2017) Diploma in Public Service Interpreting: Learner handbook. London, UK. https://www.islinguists.com/wp-content/uploads/2017/07/ISL-DPSI-Handbook-v4.2.pdf (Accessed 9 June 2021).
Jones, I. & Inglis, M.
Jones, I. & Wheadon, C.
Jones, I., Swan, M. & Pollitt, A.
Linacre, J. M.
McMahon, S. & Jones, I.
Myford, C. M. & Wolfe, E. W.
National Center for State Courts
(2019) Federal Court Interpreter Certification Examination for Spanish/English: Examinee handbook. https://www.prometric.com/sites/default/files/2019-09/fcice_examineehandbook.pdf (Accessed 9 June 2021).
Pollitt, A. & Murray, N. L.
PSI Services LLC
(2013) Development and validation of oral examinations for medical interpreter certification: Mandarin, Russian, Cantonese, Korean, and Vietnamese forms. https://nbcmi.memberclicks.net/assets/docs/tech-report-development-validation-language-forms.pdf (Accessed 9 June 2021).
Roberts, R. P.
Setton, R. & Dawrant, A.
Setton, R. & Motta, M.
Verhavert, S., Bouwer, R., Donche, V. & De Maeyer, S.
(2010) Assessing simultaneous interpreting: A study on test reliability and examiners’ assessment behavior. PhD thesis, Newcastle University.