Article in:
Applied Pragmatics: Online-First Articles

Validating a test of L2 routine formulae to detect pragmatics learning in stay abroad

| The University of Melbourne
| Sophia University
| Waseda University
| Kansai University
| Keio University
There has long been interest in the effect of stay abroad on pragmatic development in second language (L2) pragmatics research. However, few testing instruments exist to assess pragmatics learning in stay abroad and provide information about learners’ development to help institutions evaluate the success of stay abroad programs. In this study, we describe the adaptation and validation of Roever’s (2005) test of second language routine formulae to be used large-scale with Japanese stay-abroad learners. We follow Kane’s (2006) argument-based approach to validation, evaluating the first four inferences: Domain description, Evaluation, Generalization, and Explanation. We also investigate the test’s practicality within the institutional setting of intended use. We find evidence supporting the use of the test for providing information on pragmatics learning during stay abroad. The test can differentiate between low- and high-ability learners, shows adequate reliability, and reflects the roles of proficiency and exposure in pragmatics learning in stay abroad contexts. The test is also practical and requires few resources. Further steps, such as the investigation of the Extrapolation and Decision inference, are discussed.
Keywords: stay abroad, study abroad, L2 pragmatics, routine formulae, conventional expressions, argument-based validation
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1Pragmatic development in study abroad: The case of routine formulae
    • 2.2Past tests of routine formula recognition and the argument-based approach
  • 3.This study
  • 4.Method
    • 4.1Participants
    • 4.2Instruments
    • 4.3Procedures
    • 4.4Data analysis
  • 5.Results
    • 5.1RQ1 Practicality: What is the cost of creating, administering, and scoring the test?
    • 5.2RQ2 Domain description: Do items sample the domain of routine formula knowledge sufficiently based on expert judgment?
    • 5.3RQ3 evaluation: Can item scoring be argued to follow clear procedures, is computation of total scores defensible, and does item difficulty spread from easy to difficult?
    • 5.4RQ4 generalization: Is the sample of items representative of the universe of generalization based on acceptable Cronbach’s alpha reliability?
    • 5.5Explanation
      • 5.5.1RQ5a Criterion: Does the test correlate moderately to highly with a general proficiency measure like TOEFL?
      • 5.5.2RQ5b group differences 1: Does recognition of routine formulae increase with length of stay abroad?
      • 5.5.3RQ5c explanation: Group differences 2: Do test-takers with stay abroad outperform a proficiency-matched group without stay abroad?
  • 6.Discussion
  • 7.Conclusion
  • Acknowledgements
  • References
Published online: 04 March 2022
https://doi.org/10.1075/ap.20015.roe
References
Bachman, L. F.
(2004) Statistical analyses for language assessment. Cambridge University Press. CrossrefGoogle Scholar
Bachman, L. F., & Palmer, A. S.
(2010) Language assessment in practice: Developing language tests and justifying their use in the real world. Oxford University Press.Google Scholar
Baker-Smemoe, W., Dewey, D. P., Bown, J., & Martinsen, R. A.
(2014) Variables affecting L2 gains during study abroad. Foreign Language Annals, 47 (3), 464–486. CrossrefGoogle Scholar
Bardovi-Harlig, K.
(2009) Conventional expressions as a pragmalinguistic resource: Recognition and production of conventional expressions in L2 pragmatics. Language Learning, 59 (4), 755–795. CrossrefGoogle Scholar
Bardovi-Harlig, K., & Bastos, M. T.
(2011) Proficiency, length of stay, and intensity of interaction and the acquisition of conventional expressions in L2 pragmatics. Intercultural Pragmatics, 8 1, 347–384. CrossrefGoogle Scholar
Bardovi-Harlig, K., & Dörnyei, Z.
(1998) Do language learners recognize pragmatic violations? Pragmatic vs. grammatical awareness in instructed L2 learning. TESOL Quarterly, 32 1, 233–259. CrossrefGoogle Scholar
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E.
(2015) The effect of instruction on pragmatic routines in academic discussion. Language Teaching Research, 19 (3), 324–350. CrossrefGoogle Scholar
Barron, A.
(2019) Pragmatic development and stay abroad. Journal of Pragmatics, 146 1, 43–53. CrossrefGoogle Scholar
Brecht, R., Davidson, D., & Ginsberg, R.
(1995) Predictors of foreign language gain during study abroad. In B. F. Freed (Ed.), Second language acquisition in a study abroad context (pp. 37–66). John Benjamins. CrossrefGoogle Scholar
Chapelle, C. A.
(2021) Argument-based validation in testing and assessment. Sage. CrossrefGoogle Scholar
Chapelle, C. A., Enright, M. K., & Jamieson, J.
(Eds.) (2008) Building a validity argument for the Test of English as a Foreign Language. Routledge.Google Scholar
Coulmas, F.
(1979) On the sociolinguistic relevance of routine formulae. Journal of Pragmatics, 3 1, 239–266. CrossrefGoogle Scholar
Davidson, D. E.
(2010) Study abroad: When, how long, and with what results? New data from the Russian front. Foreign Language Annals, 43 (1), 6–26. CrossrefGoogle Scholar
Di Silvio, F., Donovan, A., & Malone, M. E.
(2014) The effect of study abroad homestay placements: Participant perspectives and oral proficiency gains. Foreign Language Annals, 47 (1), 168–188. CrossrefGoogle Scholar
Ebel, R. L.
(1964) The social consequences of educational testing. In Proceedings of the 1963 invitational conference on testing problems (pp. 130–143). Educational Testing Service.Google Scholar
ETS
(2010) Linking TOEFL iBT scores to IELTS scores – A research report. ETS.Google Scholar
Geeslin, K. L., & Schmidt, L. B.
(2018) Study abroad and L2 learner attitudes. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 385–405). Routledge. CrossrefGoogle Scholar
Hulstijn, J. H.
(2011) Language proficiency in native and nonnative speakers: An agenda for research and suggestions for second-language assessment. Language Assessment Quarterly, 8 (3), 229–249. CrossrefGoogle Scholar
(2015) Language proficiency in native and non-native speakers: Theory and research (Vol. 411). John Benjamins Publishing Company. CrossrefGoogle Scholar
Ikeda, N.
(2017) Measuring L2 oral pragmatic abilities for use in social contexts: Development and validation of an assessment instrument for L2 pragmatics performance in university settings (Unpublished doctoral dissertation). The University of Melbourne, Australia.
Kane, M. T.
(2006) Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed.) (pp. 17–64). American Council on Education/Praeger Publishers.Google Scholar
Matsumura, S.
(2001) Learning the rules for offering advice: A quantitative approach to second language socialization. Language Learning, 51 (4), 635–679. CrossrefGoogle Scholar
(2003) Modelling the relationships among interlanguage pragmatic development, L2 proficiency, and exposure to L2. Applied Linguistics, 24 (4), 465–491. CrossrefGoogle Scholar
Messick, S.
(1989) Validity. In R. L. Linn (Ed.), Educational Measurement (3rd ed.) (pp. 13–103). American Council on Education & Macmillan.Google Scholar
Myles, F., & Cordier, C.
(2017) Formulaic sequence (FS) cannot be an umbrella term in SLA: Focusing on psycholinguistic FSs and their identification. Studies in Second Language Acquisition, 39 (1), 3–28. CrossrefGoogle Scholar
Pawley, A., & Syder, F. H.
(1983) Two puzzles for linguistic theory: Nativelike selection and nativelike fluency. In J. C. Richards & R. W. Schmidt (Eds.), Language and communication (pp. 191–226). Longman.Google Scholar
Powers, D. E., Roever, C., Huff, K., & Trapani, C.
(2003) Validating LanguEdge Courseware Scores Against Faculty Ratings and Student Self-assessments. (ETS Research Report RR-03-11). Educational Testing Service. CrossrefGoogle Scholar
Ren, W.
(2013) The effect of study abroad on the pragmatic development of the internal modification of refusals. Pragmatics, 23 1, 715–741. CrossrefGoogle Scholar
(2015) L2 pragmatic development in study abroad contexts. Peter Lang. CrossrefGoogle Scholar
(2019) Pragmatic development of Chinese during study abroad: A cross-sectional study of learner requests. Journal of Pragmatics, 146 1, 137–149. CrossrefGoogle Scholar
Roever, C.
(2005) Testing ESL pragmatics. Peter Lang. CrossrefGoogle Scholar
(2012) What learners get for free (and when): Learning of routine formulae in ESL and EFL environments. ELT Journal, 66 (1), 10–21. CrossrefGoogle Scholar
Roever, C., Fraser, C., & Elder, C.
(2014) Testing ESL sociopragmatics. Peter Lang. CrossrefGoogle Scholar
Roever, C., & Phakiti, A.
(2018) Quantitative methods in language assessment and acquisition research. Routledge. CrossrefGoogle Scholar
Roever, C., Wang, S., & Brophy, S.
(2014) Learner background factors and learning of second language pragmatics. International Review of Applied Linguistics, 52 (4), 377–401. CrossrefGoogle Scholar
Sánchez-Hernández, A., & Alcón-Soler, E.
(2019) Pragmatic gains in the study abroad context: Learners’ experiences and recognition of pragmatic routines. Journal of Pragmatics, 146 1, 54–71. CrossrefGoogle Scholar
Schauer, G. A.
(2006) Pragmatic awareness in ESL and EFL contexts: Contrast and development. Language Learning, 56 (2), 269–318. CrossrefGoogle Scholar
Schauer, G.
(2009) Interlanguage pragmatic development: The study abroad context. Bloomsbury Publishing.Google Scholar
Schenker, T.
(2018) Making short-term study abroad count – Effects on German language skills. Foreign Language Annals, 51 (2), 411–429. CrossrefGoogle Scholar
Sell, F., Renkwitz, K., Sickinger, P., & Schneider, K. P.
(2019) Measuring pragmatic competence on the functional and lexical level: The development of German high-school students’ requests during a stay abroad in Canada. Journal of Pragmatics, 146 1, 106–120. CrossrefGoogle Scholar
Shively, R. L.
(2013) Learning to be funny in Spanish during study abroad: L2 humor development. Modern Language Journal, 97 1, 930–946. CrossrefGoogle Scholar
(2015) Developing interactional competence during study abroad: Listener responses in L2 Spanish. System, 48 1, 86–98. CrossrefGoogle Scholar
Sireci, S., & Faulkner-Bond, M.
(2014) Validity evidence based on test content. Psicothema, 26 (1), 100–107. CrossrefGoogle Scholar
Sinclair, J.
(1991) Corpus, concordance and collocation. Oxford University Press.Google Scholar
Taguchi, N.
(2014) Development of interactional competence in Japanese as a second language: Use of incomplete sentences as interactional resources. Modern Language Journal, 98 (2), 518–535. CrossrefGoogle Scholar
Taguchi, N., Li, S., & Xiao, F.
(2013) Production of formulaic expressions in L2 Chinese: A developmental investigation in a study abroad context. Chinese as a Second Language Research, 2 (1), 23–58. CrossrefGoogle Scholar
Taguchi, N., Xiao, F., & Li, S.
(2016) Effects of intercultural competence and social contact on speech act production in a Chinese study abroad context. Modern Language Journal, 100 1, 775–796. CrossrefGoogle Scholar
Tannenbaum, R. J., & Baron, P. A.
(2011) Mapping TOEFL® ITP scores onto the Common European Framework of Reference. ETS.Google Scholar
Tare, M., Golonka, E., Lancaster, A. K., Bonilla, C., Doughty, C. J., Belnap, R. K., & Jackson, S. R.
(2018) The role of cognitive aptitudes in a study abroad language-learning environment. In C. Sanz & A. Morales-Font (Eds.), The Routledge handbook of study abroad research and practice (pp. 406–420). Routledge. CrossrefGoogle Scholar
Tullock, B., & Ortega, L.
(2017) Fluency and multilingualism in study abroad: Lessons from a scoping review. System, 71 1, 7–21. CrossrefGoogle Scholar
Xiao, F.
(2015) Adult second language learners’ pragmatic development in the study-abroad context: A review. Frontiers: The Interdisciplinary Journal of Study Abroad, 25 1, 132–149. CrossrefGoogle Scholar
Xu, Y.
(2019) Changes in interlanguage complexity during study abroad: A meta-analysis. System, 80 1, 199–211. CrossrefGoogle Scholar
Yang, J.
(2016) CFL learners’ recognition and production of pragmatic routine formulae. Chinese as a Second Language. The Journal of the Chinese Language Teachers Association, USA, 51 (1), 29–61. CrossrefGoogle Scholar
Youn, S. J.
(2015) Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32 (2), 199–225. CrossrefGoogle Scholar