Chapter 7
Pragmatics
Assessing learning outcomes in instructional studies
The field of instructed SLA (ISLA) has grown rapidly in recent years to examine how systematic manipulations of instructional conditions can lead to the development of second language (L2) knowledge and use (Loewen & Sato, 2017). Following this trend, L2 pragmatics researchers have implemented various instructional methods and examined their effectiveness using experimental designs (Taguchi, 2015; Taguchi & Roever, 2017). The critical issue in this practice is how to assess learning outcomes in order to make a claim that certain instructional methods can produce robust pragmatic knowledge. To address this question, this chapter presents an overview of assessment methods used in analyzing pragmatics learning outcomes. The chapter surveys instructed L2 pragmatics studies published in the last four decades to identify common assessment methods (i.e., discourse completion tasks and role-play tasks). We provide step-by-step illustrations of expert studies in order to demonstrate how researchers design an assessment task, evaluate learning, and interpret results. The chapter concludes with a discussion of the challenges and limitations of current methods and provides direction for addressing such challenges in pragmatics research.
Article outline
- 1.Introduction: What is pragmatics and why is it important?
- 2.What we know and what we need to know about pragmatics in ISLA
- 2.1Current trends in instructed pragmatics
- 3.Data elicitation and interpretation options for pragmatics
- 3.1Discourse completion tasks (DCTs)
- 3.1.1Characteristics of DCTs
- 3.1.2Pros and cons of DCTs
- 3.1.3Step-by-step DCT illustration
- 3.2Role-play tasks
- 3.2.1Characteristics of role-play tasks
- 3.2.2Pros and cons of role-play tasks
- 3.2.3Step-by-step role play illustration
- 4.Advice to future pragmatics researchers in ISLA: How to improve assessment methods of learning outcomes
- 4.1Incorporating prosody as instructional targets and learning outcomes
- 4.2Improving authenticity of DCTs and role-plays via technology applications
- 4.3Enhancing consequentiality in role-play interaction
- 5.Troubleshooting ISLA pragmatics research
- Common question 1: How can we choose the most appropriate method to assess learning outcomes? Should we choose DCTs or role-plays?
- Common question 2: How can we determine assessment criteria and score bands when evaluating learning outcomes?
- 6.Conclusions
- 7.Further reading and additional resources
- 7.1Suggested books
- 7.2Suggested journals, professional organizations, and websites
-
Note
-
References
-
Appendix
References (47)
References
Alemi, M., & Haeri, N. S. (2020). Robot-assisted instruction of L2 pragmatics: Effects on young EFL learners’ speech act performance. Language Learning & Technology,
24
(2), 86–103.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Al-Gahtani, S., & Roever, C. (2012). Proficiency and sequential organization of L2 requests. Applied Linguistics,
33
(1), 42–65. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Al-Gahtani, S., & Roever, C. (2018). Proficiency and preference organization in second language refusals. Journal of Pragmatics,
129
, 140–153. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Blum-Kulka, S., House, J., & Kasper, G. (1989). Cross-cultural pragmatics: Requests and apologies. Ablex.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Boersma, P., & Weenink, D. (2016). Praat: Doing phonetics by computer (Version 6.0.11) [Software]. Available from [URL]
Chen, S. (2019, October 4–5). Developing an L2 pragmatic speaking test using conversation analysis findings [Paper presentation]. Annual Midwest Association of Language Testers (MwALT) conference, Indiana University Bloomington.
Cohen, A. D., & Shively, R. L. (2007). Acquisition of requests and apologies in Spanish and French: Impact of study abroad and strategy-building intervention. Modern Language Journal,
91
(2), 189–212. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Eslami, Z. R., Mirzaei, A., & Dini, S. (2015). The role of asynchronous computer mediated communication in the instruction and development of EFL learners’ pragmatic competence. System,
48
, 99–111. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Félix-Brasdefer, J. C. (2007). Pragmatic development in the Spanish as a FL classroom: A cross-sectional study of learner requests. Intercultural Pragmatics,
4
(2), 253–286. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Golato, A. (2003). Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics,
24
(1), 90–121. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Grabowski, K. C. (2009). Investigating the construct validity of a test designed to measure grammatical and pragmatic knowledge in the context of speaking [Unpublished doctoral dissertation]. Columbia University.
Gumperz, J. J. (1982). Discourse strategies. Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Gumperz, J. J., & Cook-Gumperz, J. (1982). Introduction: Language and the communication of social identity. In J. J. Gumperz (Ed.), Language and social identity (pp. 1–21). Cambridge University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Halenko, N. (2013). Using computer animation to assess and improve spoken language skills. In Conference proceedings: ICT for language learning (p. 286). Libreriauniversitaria.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Halenko, N., & Jones, C. (2017). Explicit instruction of spoken requests: An examination of pre-departure instruction and the study abroad environment. System,
68
, 26–37. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hassall, T. (2020). Preference structure in request sequences: What about role-play? Journal of Pragmatics,
155
, 321–333. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Huth, T. (2010). Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal,
94
(4), 537–553. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G. (2008). Data collection in pragmatics research. In H. Spencer-Oatey (Ed.), Culturally speaking: Culture, communication and politeness theory (pp. 279–302). Continuum.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G., & Dahl, M. (1991). Research methods in interlanguage pragmatics. Studies in Second Language Acquisition,
13
(2), 215–247. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kasper, G., & Youn, S. J. (2018). Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review,
9
(4), 589–616. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Li, S. (2012). The effects of input-based practice on pragmatic development of requests in L2 Chinese. Language Learning,
62
(2), 403–438. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Loewen, S., & Sato, M. (Eds.). (2017). The Routledge handbook of instructed second language acquisition. Routledge. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Nguyen, H. T. (2018). Interactional practices across settings: From classroom role-plays to workplace patient consultations. Applied Linguistics,
39
(2), 213–235.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Okada, Y. (2010). Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics,
42
(6), 1647–1668. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Plonsky, L., & Zhuang, J. (2019). A meta-analysis of second language pragmatics instruction. In N. Taguchi (Ed.), Routledge handbook of SLA and pragmatics (pp. 287–307). Routledge.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pomerantz, A., & Heritage, J. (2013). Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Wiley-Blackwell.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Rockey, C., Tiegs, J., & Fernández, F. (2020). Mobile application use in technology-enhanced DCTs. CALICO,
37
(1), 85–108. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Schmidt, R. (1993). Consciousness, learning and interlanguage pragmatics. In G. Kasper & S. Blum-Kulka (Eds.), Interlanguage pragmatics (pp. 21–42). Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Stokoe, E. (2013). The (in)authenticity of simulated talk: Comparing role-played and actual interaction and the implications for communication training. Research on Language and Social Interaction,
46
(2), 165–185. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Swain, M., & Lapkin, S. (1998). Interaction and second language learning: Two adolescent French immersion students working together. Modern Language Journal,
82
(1), 320–337. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sydorenko, T., Jones, Z. W., Daurio, P., & Thorne, S. L. (2020). Beyond the curriculum: Extended discourse practice through self-access pragmatics simulations. Language Learning & Technology,
24
(2), 48–69.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sykes, J. M., & González-Lloret, M. (2020). Exploring the interface of interlanguage (L2) pragmatics and digital spaces. CALICO Journal,
37
(1), i–xv. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N. (2015). Instructed pragmatics at a glance: Where instructional studies were, are, and should be going. Language Teaching,
48
(1), 1–50. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N. (2021). Application of immersive virtual reality (VR) to pragmatics data collection methods: Insights from interviews. CALICO,
38
(2), 181–201. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N., Hirschi, K., & Kang, O. (2021). Longitudinal L2 development in the prosodic marking of pragmatic meaning: Prosodic changes in L2 speech acts and individual factors. Studies in Second Language Acquisition, 1–16. Advance online publication. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N., & Kim, Y. (2016). Collaborative dialogue in learning pragmatics: Pragmatic-related episodes as an opportunity for learning request-making. Applied Linguistics,
37
(3), 416–437. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N., Li, Q., & Tang, X. (2017). Learning Chinese formulaic expressions in a scenario-based interactive environment. Foreign Language Annals,
50
(4), 641–660. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Taguchi, N., & Roever, C. (2017). Second language pragmatics. Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Takimoto, M. (2020). Investigating the effects of cognitive linguistic approach in developing EFL learners’ pragmatic proficiency. System,
89
. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Verschueren, J. (1999). Understanding pragmatics. Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Youn, S. J. (2020). Pragmatic variables in role-play design for the context validity of assessing interactional competence. Papers in Language Testing and Assessment,
9
(1), 95–127.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by (2)
Cited by two other publications
Taguchi, Naoko
2024.
Technology-enhanced language learning and pragmatics: Insights from digital game-based pragmatics instruction.
Language Teaching 57:1
► pp. 57 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
Kim, YouJin, Sanghee Kang, Meredith D’Arienzo & Naoko Taguchi
2023.
Comparing traditional and task-based approaches to teaching pragmatics: Task design processes and learning outcomes.
Language Teaching Research ![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 5 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.