Investigating rater severity/leniency in interpreter performance testing
A multifaceted Rasch measurement approach
Rater-mediated performance assessment (RMPA) is a critical component of interpreter certification testing systems worldwide. Given the acknowledged rater variability in RMPA and the high-stakes nature of certification testing, it is crucial to ensure rater reliability in interpreter certification performance testing (ICPT). However, a review of current ICPT practice indicates that rigorous research on rater reliability is lacking. Against this background, the present study reports on use of multifaceted Rasch measurement (MFRM) to identify the degree of severity/leniency in different raters’ assessments of simultaneous interpretations (SIs) by 32 interpreters in an experimental setting. Nine raters specifically trained for the purpose were asked to evaluate four English-to-Chinese SIs by each of the interpreters, using three 8-point rating scales (information content, fluency, expression). The source texts differed in speed and in the speaker’s accent (native vs non-native). Rater-generated scores were then subjected to MFRM analysis, using the FACETS program. The following general trends emerged: 1) homogeneity statistics showed that not all raters were equally severe overall; and 2) bias analyses showed that a relatively large proportion of the raters had significantly biased interactions with the interpreters and the assessment criteria. Implications for practical rating arrangements in ICPT, and for rater training, are discussed.
Keywords: performance testing, multifaceted Rasch measurement, rater severity/leniency, interpreter certification, rater training, rater variability
Published online: 03 September 2015
Cited by 18 other publications
Abdel Latif, Muhammad M. M.
Abdel Latif, Muhammad M. M.
Chen, Jing & Chao Han
Han, Chao & Kerui An
Han, Chao & Qin Fan
Han, Chao & Helen Slatyer
Han, Chao, Rui Xiao & Wei Su
Han, Chao & Xiao Zhao
This list is based on CrossRef data as of 29 august 2021. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.
Arocha, I.S. & Joyce, L.
Bond, T.G. & Fox, C.M.
Bonk, W.J. & Ockey, G.J.
Campbell, S. & Hale, S.
Certification Commission for Healthcare Interpreters
(2010) Job task analysis study and results. http://www.cchicertification.org/images/webinars/cchi%20jta%20report-public.pdf (accessed 22 May 2015).
(2011) Technical Report on the Development and Pilot Testing of the CCHI Examinations. http://www.cchicertification.org/images/pdfs/cchi%20technical%20report%20-%20public%20final.pdf (accessed 22 May 2015).
(2012) Technical Report on the Development and Pilot Testing of the Certified Healthcare Interpreter™ (CHI™) Examination for Arabic and Mandarin. http://www.cchicertification.org/images/pdfs/cchi%20arabic%20and%20mandarin%20technical%20report-final.pdf (accessed 22 May 2015).
(2014) Candidate’s Examination Handbook. http://www.cchicertification.org/images/pdfs/candidatehandbook.pdf (accessed 22 May 2015).
Elder, C., Barkhuizen, G., Knoch, U. & von Randow, J.
Fox, C. & Jones, J.
Hale, S., Garcia, I., Hlavac, J., Kim, M., Lai, M., Turner, B. & Slatyer, H.
(2012) Development of a conceptual overview for a new model for NAATI standards, testing and assessment. Sydney, Australia. http://www.naati.com.au/PDF/INT/INTFinalReport.pdf (accessed 22 May 2015).
Han, C. & Mehdi, R.
(2015) The effects of speech rate and accent on interpreter performance quality: A mixed-methods replication study. Manuscript submitted for publication.
IoL Educational Trust
(2010) Diploma in Public Service Interpreting: Handbook for candidates. London, UK. http://www.iol.org.uk/qualifications/DPSI/Handbook/DPSIHB11.pdf (accessed 22 May 2015).
Jacobs, E.A., Lauderdale, D.S., Meltzer, D., Shorey, J.M., Levinson, W. & Thisted, R.A.
(2013). A user’s guide to FACETS: Program manual 3.71.2. http://www.winsteps.com/a/facets-manual.pdf (accessed 22 May 2015).
Lu, M., Liu, C. & Gong, X.F.
Lumley, T. & McNamara, T.F.
Lunz, M.E. & Stahl, J.A.
Lynch, B.K. & McNamara, T.F.
McNamara, T.F. & Knoch, U.
(1998) Establishing a scheme for interpreter certification: The Norwegian experience. http://folk.uio.no/dianem/report1998.pdf (accessed 22 May 2015).
(2001) Measuring quality in interpreting: A report on the Norwegian Interpreter Certification Examination (NICE). Oslo, Norway. http://folk.uio.no/dianem/IntQuality-Internet.pdf (accessed 22 May 2015).
Myford, C.M. & Wolfe, E.W.
National Accreditation Authority for Translators and Interpreters
(2013) INT Project discussion paper. http://www.naati.com.au/PDF/INT/INT%20Project%20Discussion%20Paper%20-%20November%202013.pdf (accessed 22 May 2015).
National Association of the Deaf
(2014) NAD and RID releases the NAD-RID National Interpreter Certification (NIC) Credential Validity, Reliability, & Candidate Performance Report. http://nad.org/news/2014/2/nad-and-rid-release-performance-report (accessed 22 May 2015).
National Board of Certification for Medical Interpreters
(2014) The National Board of Certification for Medical Interpreters: Certified Medical Interpreter candidate handbook. http://www.certifiedmedicalinterpreters.org//sites/default/files/national-board-candidate-handbook.pdf (accessed 22 May 2015).
National Center for States Courts
(2013) Federal Court Interpreter Certification Examination for Spanish/English: Examinee handbook. http://www.ncsc.org/sitecore/content/microsites/fcice/home/About-the-program/~/media/Microsites/Files/FCICE/Final%20Examinee%20Handbook%201-23-2013%20for%20online.ashx (accessed 22 May 2015).
Office of China Accreditation Tests for Translators and Interpreters
PSI Services LLC
(2010) Development and validation of oral and written examinations for medical interpreter certification: Technical report. Burbank, California, USA. http://www.certifiedmedicalinterpreters.org//sites/default/files/oral-and-written-medical-interpreter-technical-report-final.pdf (accessed 22 May 2015).
(2013) Development and validation of oral examinations for Medical Interpreter Certification: Mandarin, Russian, Cantonese, Korean, and Vietnamese forms. http://www.certifiedmedicalinterpreters.org/sites/default/files/tech-report-development-validation-language-forms.pdf (accessed 22 May 2015).
(2006) Certification of health care interpreters in the United States: A primer, a status report and considerations for national certification. Los Angeles, CA. http://www.calendow.org/uploadedFiles/certification_of_health_care_interpretors.pdf (accessed 22 May 2015).
Russell, D. & Malcolm, K.
South African Translators’ Institute
(2007a) Guidelines: SASL interpreter accreditation testing. http://translators.org.za/sati_cms/downloads/dynamic/sati_accreditation_for_sasl_interpreting_english.pdf (accessed 22 May 2015).
(2007b) Guidelines: Simultaneous interpreter accreditation testing. http://translators.org.za/sati_cms/downloads/dynamic/sati_accreditation_for_sim_interpreting_english.pdf (accessed 22 May 2015).
Stansfield, C.W. & Hewitt, W.
Sudweeks, R., Reeve, S. & Bradshaw, W.S.
Turner, B., Lai, M. & Huang, N.
Upshur, J.A. & Turner, C.E.
Vermeiren, H., Gucht, J.V. & De Bontridder, L.
(2009) Standards as critical success factors in assessments: Certifying social interpreters in Flanders, Belgium. In C.V. Angelelli & H.E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins, 291–330.
(2005) T&I labor market in China. Sydney, Australia. http://www.ling.mq.edu.au/translation/lmtip_china.htm (accessed 22 May 2015).