Cover not available
Part of
Instructed Second Language Acquisition Research Methods
Edited by Laura Gurzynski-Weiss and YouJin Kim
[Research Methods in Applied Linguistics 3] 2022
► pp. 149–180

Chapter 7
Pragmatics

Assessing learning outcomes in instructional studies

| Northern Arizona University
| Daegu National University of Education

The field of instructed SLA (ISLA) has grown rapidly in recent years to examine how systematic manipulations of instructional conditions can lead to the development of second language (L2) knowledge and use (Loewen & Sato, 2017). Following this trend, L2 pragmatics researchers have implemented various instructional methods and examined their effectiveness using experimental designs (Taguchi, 2015; Taguchi & Roever, 2017). The critical issue in this practice is how to assess learning outcomes in order to make a claim that certain instructional methods can produce robust pragmatic knowledge. To address this question, this chapter presents an overview of assessment methods used in analyzing pragmatics learning outcomes. The chapter surveys instructed L2 pragmatics studies published in the last four decades to identify common assessment methods (i.e., discourse completion tasks and role-play tasks). We provide step-by-step illustrations of expert studies in order to demonstrate how researchers design an assessment task, evaluate learning, and interpret results. The chapter concludes with a discussion of the challenges and limitations of current methods and provides direction for addressing such challenges in pragmatics research.

Keywords: pragmatics, instructional studies, learning outcomes, assessment, DCT, role-play
Article outline
  • 1.Introduction: What is pragmatics and why is it important?
  • 2.What we know and what we need to know about pragmatics in ISLA
    • 2.1Current trends in instructed pragmatics
  • 3.Data elicitation and interpretation options for pragmatics
    • 3.1Discourse completion tasks (DCTs)
      • 3.1.1Characteristics of DCTs
      • 3.1.2Pros and cons of DCTs
      • 3.1.3Step-by-step DCT illustration
    • 3.2Role-play tasks
      • 3.2.1Characteristics of role-play tasks
      • 3.2.2Pros and cons of role-play tasks
      • 3.2.3Step-by-step role play illustration
  • 4.Advice to future pragmatics researchers in ISLA: How to improve assessment methods of learning outcomes
    • 4.1Incorporating prosody as instructional targets and learning outcomes
    • 4.2Improving authenticity of DCTs and role-plays via technology applications
    • 4.3Enhancing consequentiality in role-play interaction
  • 5.Troubleshooting ISLA pragmatics research
    • Common question 1: How can we choose the most appropriate method to assess learning outcomes? Should we choose DCTs or role-plays?
    • Common question 2: How can we determine assessment criteria and score bands when evaluating learning outcomes?
  • 6.Conclusions
  • 7.Further reading and additional resources
    • 7.1Suggested books
    • 7.2Suggested journals, professional organizations, and websites
  • Note
  • References
  • Appendix
Published online: 8 December 2022
DOI logo
https://doi.org/10.1075/rmal.3.07tag
References
Alemi, M., & Haeri, N. S.
(2020) Robot-assisted instruction of L2 pragmatics: Effects on young EFL learners’ speech act performance. Language Learning & Technology, 24 (2), 86–103.Google Scholar
Al-Gahtani, S., & Roever, C.
(2012) Proficiency and sequential organization of L2 requests. Applied Linguistics, 33 (1), 42–65. DOI logoGoogle Scholar
(2018) Proficiency and preference organization in second language refusals. Journal of Pragmatics, 129 , 140–153. DOI logoGoogle Scholar
Barón, J., Celaya, M. L., & Levkina, M.
(2020) Learning pragmatics through tasks: When interaction plays a role. Applied Pragmatics, 2 (1), 1–25. DOI logoGoogle Scholar
Blum-Kulka, S., House, J., & Kasper, G.
(1989) Cross-cultural pragmatics: Requests and apologies. Ablex.Google Scholar
Boersma, P., & Weenink, D.
(2016) Praat: Doing phonetics by computer (Version 6.0.11) [Software]. Available from [URL]
Chen, S.
(2019, October 4–5). Developing an L2 pragmatic speaking test using conversation analysis findings [Paper presentation]. Annual Midwest Association of Language Testers (MwALT) conference, Indiana University Bloomington.
Cohen, A. D., & Shively, R. L.
(2007) Acquisition of requests and apologies in Spanish and French: Impact of study abroad and strategy-building intervention. Modern Language Journal, 91 (2), 189–212. DOI logoGoogle Scholar
Eslami, Z. R., Mirzaei, A., & Dini, S.
(2015) The role of asynchronous computer mediated communication in the instruction and development of EFL learners’ pragmatic competence. System, 48 , 99–111. DOI logoGoogle Scholar
Félix-Brasdefer, J. C.
(2007) Pragmatic development in the Spanish as a FL classroom: A cross-sectional study of learner requests. Intercultural Pragmatics, 4 (2), 253–286. DOI logoGoogle Scholar
Golato, A.
(2003) Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24 (1), 90–121. DOI logoGoogle Scholar
Grabowski, K. C.
(2009) Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking [Unpublished doctoral dissertation]. Columbia University.
Gumperz, J. J.
(1982) Discourse strategies. Cambridge University Press. DOI logoGoogle Scholar
Gumperz, J. J., & Cook-Gumperz, J.
(1982) Introduction: Language and the communication of social identity. In J. J. Gumperz (Ed.), Language and social identity (pp. 1–21). Cambridge University Press.Google Scholar
Halenko, N.
(2013) Using computer animation to assess and improve spoken language skills. In Conference proceedings: ICT for language learning (p. 286). Libreriauniversitaria.Google Scholar
Halenko, N., & Jones, C.
(2017) Explicit instruction of spoken requests: An examination of pre-departure instruction and the study abroad environment. System, 68 , 26–37. DOI logoGoogle Scholar
Hashimoto, B. J., & Nelson, K.
(2020) Using a corpus in creating and evaluating a DCT. Applied Pragmatics, 2 (1), 80–120. DOI logoGoogle Scholar
Hassall, T.
(2020) Preference structure in request sequences: What about role-play? Journal of Pragmatics, 155 , 321–333. DOI logoGoogle Scholar
Huth, T.
(2010) Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal, 94 (4), 537–553. DOI logoGoogle Scholar
Kang, O., Kermad, A., & Taguchi, N.
(2021) The interplay of proficiency and study abroad experience on prosody of L2 speech acts. Journal of Second Language Pronunciation, 7 (3), 343–369. DOI logoGoogle Scholar
Kasper, G.
(2006) Beyond repair: Conversation analysis as an approach to SLA. AILA Review, 19 (1), 83–99. DOI logoGoogle Scholar
(2008) Data collection in pragmatics research. In H. Spencer-Oatey (Ed.), Culturally speaking: Culture, communication and politeness theory (pp. 279–302). Continuum.Google Scholar
Kasper, G., & Dahl, M.
(1991) Research methods in interlanguage pragmatics. Studies in Second Language Acquisition, 13 (2), 215–247. DOI logoGoogle Scholar
Kasper, G., & Youn, S. J.
(2018) Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review, 9 (4), 589–616. DOI logoGoogle Scholar
Kim, M., Lee, H., & Kim, Y.
(2018) Learning of Korean honorifics through collaborative tasks: Comparing heritage and foreign language learners. In N. Taguchi & Y. Kim (Eds.), Task-based approaches to teaching and assessing pragmatics (pp. 27–54). John Benjamins. DOI logoGoogle Scholar
Li, S.
(2012) The effects of input-based practice on pragmatic development of requests in L2 Chinese. Language Learning, 62 (2), 403–438. DOI logoGoogle Scholar
Loewen, S., & Sato, M.
(Eds.) (2017) The Routledge handbook of instructed second language acquisition. Routledge. DOI logoGoogle Scholar
Nguyen, H. T.
(2018) Interactional practices across settings: From classroom role-plays to workplace patient consultations. Applied Linguistics, 39 (2), 213–235.Google Scholar
Okada, Y.
(2010) Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42 (6), 1647–1668. DOI logoGoogle Scholar
Plonsky, L., & Zhuang, J.
(2019) A meta-analysis of second language pragmatics instruction. In N. Taguchi (Ed.), Routledge handbook of SLA and pragmatics (pp. 287–307). Routledge.Google Scholar
Pomerantz, A., & Heritage, J.
(2013) Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Wiley-Blackwell.Google Scholar
Rockey, C., Tiegs, J., & Fernández, F.
(2020) Mobile application use in technology-enhanced DCTs. CALICO, 37 (1), 85–108. DOI logoGoogle Scholar
Schmidt, R.
(1993) Consciousness, learning and interlanguage pragmatics. In G. Kasper & S. Blum-Kulka (Eds.), Interlanguage pragmatics (pp. 21–42). Oxford University Press.Google Scholar
Stokoe, E.
(2013) The (in)authenticity of simulated talk: Comparing role-played and actual interaction and the implications for communication training. Research on Language and Social Interaction, 46 (2), 165–185. DOI logoGoogle Scholar
Swain, M., & Lapkin, S.
(1998) Interaction and second language learning: Two adolescent French immersion students working together. Modern Language Journal, 82 (1), 320–337. DOI logoGoogle Scholar
Sydorenko, T., Jones, Z. W., Daurio, P., & Thorne, S. L.
(2020) Beyond the curriculum: Extended discourse practice through self-access pragmatics simulations. Language Learning & Technology, 24 (2), 48–69.Google Scholar
Sykes, J. M., & González-Lloret, M.
(2020) Exploring the interface of interlanguage (L2) pragmatics and digital spaces. CALICO Journal, 37 (1), i–xv. DOI logoGoogle Scholar
Taguchi, N.
(2015) Instructed pragmatics at a glance: Where instructional studies were, are, and should be going. Language Teaching, 48 (1), 1–50. DOI logoGoogle Scholar
(2021) Application of immersive virtual reality (VR) to pragmatics data collection methods: Insights from interviews. CALICO, 38 (2), 181–201. DOI logoGoogle Scholar
Taguchi, N., Hirschi, K., & Kang, O.
(2021) Longitudinal L2 development in the prosodic marking of pragmatic meaning: Prosodic changes in L2 speech acts and individual factors. Studies in Second Language Acquisition, 1–16. Advance online publication. DOI logoGoogle Scholar
Taguchi, N., & Kim, Y.
(2016) Collaborative dialogue in learning pragmatics: Pragmatic-related episodes as an opportunity for learning request-making. Applied Linguistics, 37 (3), 416–437. DOI logoGoogle Scholar
Taguchi, N., Li, Q., & Tang, X.
(2017) Learning Chinese formulaic expressions in a scenario-based interactive environment. Foreign Language Annals, 50 (4), 641–660. DOI logoGoogle Scholar
Taguchi, N., & Roever, C.
(2017) Second language pragmatics. Oxford University Press.Google Scholar
Taguchi, N., & Sykes, J. M.
(Eds.) (2013) Technology in interlanguage pragmatics research and teaching. John Benjamins. DOI logoGoogle Scholar
Takimoto, M.
(2020) Investigating the effects of cognitive linguistic approach in developing EFL learners’ pragmatic proficiency. System, 89 . DOI logoGoogle Scholar
Verschueren, J.
(1999) Understanding pragmatics. Oxford University Press.Google Scholar
Youn, S. J.
(2020) Pragmatic variables in role-play design for the context validity of assessing interactional competence. Papers in Language Testing and Assessment, 9 (1), 95–127.Google Scholar