Cover not available
Part of
Meaningful Language Test Scores: Research to enhance score interpretation
Edited by Spiros Papageorgiou and Venessa F. Manna
[Innovations in Language Learning and Assessment 1] 2023
► pp. 99–119

Chapter 6
Exploring the benefits and challenges of web-based approaches to standard setting

| Educational Testing Service

One aspect of standard setting that has been relatively consistent throughout its history is the interactional context in which expert panelists have made their judgments: face-to-face, in-person meetings. With the growth of the internet and acceleration of web-based technology, researchers have started considering how web-based approaches may address some of the inherent practical challenges for standard setting studies (see Katz et al., 2009). The worldwide pandemic of 2020 brought about an acceleration of the pervasiveness and use of web-based meeting software, making it not only a more attractive option for standard setting panels, but for a time potentially the only option. In this chapter, existing research on approaches to web-based standard setting is reviewed, and the procedural elements of three recent web-based standard setting studies conduct at ETS are summarized and compared. The accounts of web-based standard setting described in this chapter suggest that a web-based approach may be an attractive alternative to face-to-face meetings, not merely a pandemic-era necessity.

Article outline
  • Introduction
  • Implementing a web-based approach to standard setting: Three examples
    • Study 1: Hybrid (in-person, remote), August-September 2018
    • Study 2: Web-based (fully remote), April 2020
    • Study 3: Web-based (fully remote), January 2022
  • Typical elements of standard setting as implemented in web-based approaches
    • Introductory session
    • Pre-meeting work
    • Main meeting
    • Discussion of performance level descriptors (PLDs)
    • Training and practice in standard setting methods
    • Initial survey (training evaluation)
    • Round 1, 2, and 3 judgments
    • Round 1 and 2 discussion
    • Final survey
  • Conclusion
  • References
Published online: 29 June 2023
DOI logo
https://doi.org/10.1075/illa.1.06sch
References
American Council on the Teaching of Foreign Languages
(2012) ACTFL proficiency guidelines 2012. Author.
Centre for Canadian Language Benchmarks
(2012) Canadian language benchmarks: English as a second language for adults. Author.
Cizek, G. J., & Earnest, D. S.
(2015) Setting performance standards on tests. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 212–237). Routledge.Google Scholar
Council of Europe
(2001) Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press.
(2020) Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume with new descriptors. Retrieved on 8 February 2023 from [URL]
Hambleton, R. K., & Pitoniak, M. J.
(2006) Setting performance standards. In R. Brennan (Ed.), Educational measurement (4th ed., pp. 433–470). Praeger.Google Scholar
Harvey, A. L.
(2000, April) Comparing onsite and online standard setting methods for multiple levels of standards. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.Google Scholar
Harvey, A. L., & Way, W. D.
(1999, April) A comparison of web-based standard setting and monitored standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.Google Scholar
Katz, I. R., & Tannenbaum, R. J.
(2014) Comparison of web-based and face-to-face standard setting using the Angoff Method. Journal of Applied Testing Technology, 15(1), 1–17.Google Scholar
Katz, I. R., Tannenbaum, R. J., & Kannan, P.
(2009) Virtual standard setting. CLEAR Exam Review, 20(2), 19–27.Google Scholar
Knoch, U., & Frost, K.
(2016) Linking the GEPT Writing sub-test to the Common European Framework of Reference (CEFR) (LTTC-GEPT Research Reports No. RG–08). The Language Training and Testing Center.Google Scholar
Kolias, C.
(2022, May 18) Virtual standard setting: The benefits, the challenges, and the way forward. EALTA webinar.Google Scholar
Papageorgiou, S., Wu, S., Hsieh, C.-N., Tannenbaum, R. J., & Cheng, M.
(2019) Mapping the TOEFL iBT® test scores to China’s Standards of English Language Ability: Implications for score interpretation and use (Research Report No. TOEFL-RR–89). Educational Testing Service. DOI logoGoogle Scholar
Schmidgall, J.
(2021) Mapping the redesigned TOEIC Bridge® test scores to proficiency levels of the Common European Framework of Reference for Languages (Research Memorandum No. RM–21–01). ETS.Google Scholar
Schmidgall, J., Tolentino, F., & Hille, K.
in press). Mapping the redesigned TOEIC Bridge® test scores to proficiency levels of the Canadian Language Benchmarks (Research Memorandum No. RM-xx-xx). ETS.
Schnipke, D. L., & Becker, K. A.
(2007) Making the test development process more efficient using web-based virtual teams. CLEAR Exam Review, 18, 13–17.Google Scholar
Tannenbaum, R. J., & Cho, Y.
(2014) Critical factors to consider in evaluation standard-setting studies to map language test scores to frameworks of language proficiency. Language Assessment Quarterly, 7(3), 233–249. DOI logoGoogle Scholar
Tannenbaum, R. J., & Katz, I. R.
(2013) Standard setting. In K. Geisinger et al. (Eds.), Testing and assessment in psychology, Vol. 3: Testing and assessment in school psychology and education (pp. 455–478). American Psychological Association. DOI logoGoogle Scholar
Tannenbaum, R. J., & Wiley, E. C.
(2008) Linking English-language test scores onto the Common European Framework of Reference: An application of standard-setting methodology (ETS Research Report No. RR–08–34). ETS. DOI logoGoogle Scholar
Way, W. D., & McClarty, K. L.
(2012) Standard setting for computer-based assessments. In G. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 451–466). Routledge.Google Scholar
Zieky, M. J.
(2001) So much as changed: How the setting of cutscores has evolved since the 1980s. In G. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19–52). Lawrence Erlbaum Associates.Google Scholar
Zieky, M.
(2012) So much has changed: An historical overview of setting cut scores. In G. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 15–32). Routledge.Google Scholar