Validating a test of L2 routine formulae to detect pragmatics learning in stay abroad
Carsten Roever | The University of Melbourne
Yuki Higuchi | Sophia University
Miyuki Sasaki | Waseda University
Tomoko Yashima | Kansai University
Makiko Nakamuro | Keio University
There has long been interest in the effect of stay abroad on pragmatic development in second language (L2) pragmatics research. However, few testing instruments exist to assess pragmatics learning in stay abroad and provide information about learners’ development to help institutions evaluate the success of stay abroad programs. In this study, we describe the adaptation and validation of Roever’s (2005) test of second language routine formulae to be used large-scale with Japanese stay-abroad learners. We follow Kane’s (2006) argument-based approach to validation, evaluating the first four inferences: Domain description, Evaluation, Generalization, and Explanation. We also investigate the test’s practicality within the institutional setting of intended use. We find evidence supporting the use of the test for providing information on pragmatics learning during stay abroad. The test can differentiate between low- and high-ability learners, shows adequate reliability, and reflects the roles of proficiency and exposure in pragmatics learning in stay abroad contexts. The test is also practical and requires few resources. Further steps, such as the investigation of the Extrapolation and Decision inference, are discussed.
Keywords: stay abroad, study abroad, L2 pragmatics, routine formulae, conventional expressions, argument-based validation
Article outline
- 1.Introduction
- 2.Background
- 2.1Pragmatic development in study abroad: The case of routine formulae
- 2.2Past tests of routine formula recognition and the argument-based approach
- 3.This study
- 4.Method
- 4.1Participants
- 4.2Instruments
- 4.3Procedures
- 4.4Data analysis
- 5.Results
- 5.1RQ1 Practicality: What is the cost of creating, administering, and scoring the test?
- 5.2RQ2 Domain description: Do items sample the domain of routine formula knowledge sufficiently based on expert judgment?
- 5.3RQ3 evaluation: Can item scoring be argued to follow clear procedures, is computation of total scores defensible, and does item difficulty spread from easy to difficult?
- 5.4RQ4 generalization: Is the sample of items representative of the universe of generalization based on acceptable Cronbach’s alpha reliability?
- 5.5Explanation
- 5.5.1RQ5a Criterion: Does the test correlate moderately to highly with a general proficiency measure like TOEFL?
- 5.5.2RQ5b group differences 1: Does recognition of routine formulae increase with length of stay abroad?
- 5.5.3RQ5c explanation: Group differences 2: Do test-takers with stay abroad outperform a proficiency-matched group without stay abroad?
- 6.Discussion
- 7.Conclusion
- Acknowledgements
-
References
Published online: 04 March 2022
https://doi.org/10.1075/ap.20015.roe
https://doi.org/10.1075/ap.20015.roe
References
Bachman, L. F., & Palmer, A. S.
Baker-Smemoe, W., Dewey, D. P., Bown, J., & Martinsen, R. A.
Bardovi-Harlig, K.
Bardovi-Harlig, K., & Bastos, M. T.
Bardovi-Harlig, K., & Dörnyei, Z.
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E.
Brecht, R., Davidson, D., & Ginsberg, R.
Chapelle, C. A., Enright, M. K., & Jamieson, J.
Coulmas, F.
Davidson, D. E.
Di Silvio, F., Donovan, A., & Malone, M. E.
Ebel, R. L.
Geeslin, K. L., & Schmidt, L. B.
Hulstijn, J. H.
Ikeda, N.
(2017) Measuring L2 oral pragmatic abilities for use in social contexts: Development and validation of an assessment instrument for L2 pragmatics performance in university settings (Unpublished doctoral dissertation). The University of Melbourne, Australia.
Kane, M. T.
Matsumura, S.
Messick, S.
Myles, F., & Cordier, C.
Pawley, A., & Syder, F. H.
Powers, D. E., Roever, C., Huff, K., & Trapani, C.
Ren, W.
Roever, C., & Phakiti, A.
Roever, C., Wang, S., & Brophy, S.
Sánchez-Hernández, A., & Alcón-Soler, E.
Schauer, G. A.
Schauer, G.
Schenker, T.
Sell, F., Renkwitz, K., Sickinger, P., & Schneider, K. P.
Shively, R. L.
Sireci, S., & Faulkner-Bond, M.
Taguchi, N.
Taguchi, N., Li, S., & Xiao, F.
Taguchi, N., Xiao, F., & Li, S.
Tannenbaum, R. J., & Baron, P. A.
Tare, M., Golonka, E., Lancaster, A. K., Bonilla, C., Doughty, C. J., Belnap, R. K., & Jackson, S. R.
Tullock, B., & Ortega, L.
Xiao, F.
Xu, Y.
Yang, J.