Chapter 6
Exploring the benefits and challenges of web-based approaches to
standard setting
One aspect of standard setting that has been relatively
consistent throughout its history is the interactional context in which
expert panelists have made their judgments: face-to-face, in-person
meetings. With the growth of the internet and acceleration of web-based
technology, researchers have started considering how web-based
approaches may address some of the inherent practical challenges for
standard setting studies (see Katz
et al., 2009). The worldwide pandemic of 2020 brought about
an acceleration of the pervasiveness and use of web-based meeting
software, making it not only a more attractive option for standard
setting panels, but for a time potentially the only option. In this
chapter, existing research on approaches to web-based standard setting
is reviewed, and the procedural elements of three recent web-based
standard setting studies conduct at ETS are summarized and compared. The
accounts of web-based standard setting described in this chapter suggest
that a web-based approach may be an attractive alternative to
face-to-face meetings, not merely a pandemic-era necessity.
Article outline
- Introduction
- Implementing a web-based approach to standard setting: Three
examples
- Study 1: Hybrid (in-person, remote), August-September
2018
- Study 2: Web-based (fully remote), April 2020
- Study 3: Web-based (fully remote), January 2022
- Typical elements of standard setting as implemented in web-based
approaches
- Introductory session
- Pre-meeting work
- Main meeting
- Discussion of performance level descriptors (PLDs)
- Training and practice in standard setting methods
- Initial survey (training evaluation)
- Round 1, 2, and 3 judgments
- Round 1 and 2 discussion
- Final survey
- Conclusion
-
References
References (22)
References
American Council on
the Teaching of Foreign
Languages. (2012). ACTFL
proficiency guidelines
2012. Author.
Centre for Canadian
Language
Benchmarks. (2012). Canadian
language benchmarks: English as a second language for
adults. Author.
Cizek, G. J., & Earnest, D. S. (2015). Setting
performance standards on
tests. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook
of test development (2nd
ed., pp. 212–237). Routledge.
Council of
Europe. (2001). Common
European Framework of Reference for Languages: Learning,
teaching,
assessment. Cambridge University Press.
Council of
Europe. (2020). Common
European Framework of Reference for Languages: Learning,
teaching, assessment. Companion volume with new
descriptors. Retrieved on 8
February 2023 from [URL]
Hambleton, R. K., & Pitoniak, M. J. (2006). Setting
performance
standards. In R. Brennan (Ed.), Educational
measurement (4th
ed., pp. 433–470). Praeger.
Harvey, A. L. (2000,
April). Comparing onsite and online
standard setting methods for multiple levels of
standards. Paper presented at the annual meeting of the National Council on Measurement in Education, New Orleans, LA.
Harvey, A. L., & Way, W. D. (1999,
April). A comparison of web-based
standard setting and monitored standard
setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Montreal, Canada.
Katz, I. R., & Tannenbaum, R. J. (2014). Comparison
of web-based and face-to-face standard setting using the
Angoff Method. Journal of
Applied Testing
Technology, 15(1), 1–17.
Katz, I. R., Tannenbaum, R. J., & Kannan, P. (2009). Virtual
standard setting. CLEAR Exam
Review, 20(2), 19–27.
Knoch, U., & Frost, K. (2016). Linking
the GEPT Writing sub-test to the Common European Framework
of Reference (CEFR) (LTTC-GEPT
Research Reports No.
RG–08). The Language Training and Testing Center.
Kolias, C. (2022,
May 18). Virtual standard setting: The
benefits, the challenges, and the way
forward. EALTA webinar.
Papageorgiou, S., Wu, S., Hsieh, C.-N., Tannenbaum, R. J., & Cheng, M. (2019). Mapping
the TOEFL iBT® test scores to China’s Standards
of English Language Ability: Implications for score
interpretation and use (Research
Report No.
TOEFL-RR–89). Educational Testing Service.
Schmidgall, J. (2021). Mapping
the redesigned TOEIC Bridge®
test scores to proficiency levels of the Common
European Framework of Reference for
Languages (Research
Memorandum No.
RM–21–01). ETS.
Schmidgall, J., Tolentino, F., & Hille, K. (in
press). Mapping the
redesigned TOEIC Bridge®
test scores to proficiency levels of the Canadian
Language
Benchmarks (Research
Memorandum No.
RM-xx-xx). ETS.
Schnipke, D. L., & Becker, K. A. (2007). Making
the test development process more efficient using web-based
virtual teams. CLEAR Exam
Review, 18, 13–17.
Tannenbaum, R. J., & Cho, Y. (2014). Critical
factors to consider in evaluation standard-setting studies
to map language test scores to frameworks of language
proficiency. Language
Assessment
Quarterly, 7(3), 233–249.
Tannenbaum, R. J., & Katz, I. R. (2013). Standard
setting. In K. Geisinger et al. (Eds.), Testing
and assessment in psychology, Vol. 3: Testing and assessment
in school psychology and
education (pp. 455–478). American Psychological Association.
Tannenbaum, R. J., & Wiley, E. C. (2008). Linking
English-language test scores onto the Common European
Framework of Reference: An application of standard-setting
methodology (ETS Research Report
No.
RR–08–34). ETS.
Way, W. D., & McClarty, K. L. (2012). Standard
setting for computer-based
assessments. In G. Cizek (Ed.), Setting
performance standards: Foundations, methods, and
innovations (2nd
ed., pp. 451–466). Routledge.
Zieky, M. J. (2001). So
much as changed: How the setting of cutscores has evolved
since the
1980s. In G. Cizek (Ed.), Setting
performance standards: Concepts, methods, and
perspectives (pp. 19–52). Lawrence Erlbaum Associates.
Zieky, M. (2012). So
much has changed: An historical overview of setting cut
scores. In G. Cizek (Ed.), Setting
performance standards: Foundations, methods, and
innovations (2nd
ed., pp. 15–32). Routledge.