Article published in:
Applied Pragmatics
Vol. 2:1 (2020) ► pp. 80–120

Using a corpus in creating and evaluating a DCT

| Northern Arizona University
| Northern Arizona University
Discourse Completion Tasks (DCTs) have been one of the most popular tools in pragmatics research. Yet, many have criticized DCTs for their lack of authenticity (e.g., Culpeper, Mackey, & Taguchi, 2018; Nguyen, 2019). We propose that corpora can serve as resources in designing and evaluating DCTs. We created a DCT using advice-seeking prompts from the Q+A corpus (Baker & Egbert, 2016). Then, we administered the DCT to 33 participants. We evaluated the DCT by (1) comparing the linguistic form and the semantic content of the participants’ DCT responses (i.e., advice-giving expressions) with authentic data from the corpus; and (2) interviewing the participants about the instrument quality. Chi-square tests between DCT data and corpus data revealed no significant differences in advice-giving expressions in terms of both the overall level of directness (χ2 [2, N = 660] = 6.94, p = .03, V = .10) and linguistic realization (χ2 [8, N = 660] = 17.75, p = .02, V = .16), and showed a significant difference but small effect size in terms of semantic content (χ2 [6, N = 512] = 30.35, p < .01, V = .24). Taken together with the interview data, our findings indicate that corpora are useful in designing DCTs.
Keywords: discourse completion task, corpus-based instrument, advice-giving, technology-mediated communication, authenticity, pragmatics
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1DCTs
    • 2.2Criticism of DCTs
    • 2.3Applications of corpora in instrument design and pragmatics research
    • 2.4The speech act of advice-giving
  • 3.The present study
  • 4.Methods
    • 4.1Participants
    • 4.2Corpus
    • 4.3DCT instrument
    • 4.4Comparison of advice-giving strategies between DCT and corpus
      • 4.4.1Coding schemes for advice-giving strategies
      • 4.4.2Statistical analysis of advice-giving strategies
    • 4.5Interview
      • 4.5.1Interview procedures
      • 4.5.2Thematic analysis of interview data
  • 5.Results
    • 5.1Thematic analysis of interview data
    • 5.2Statistical analysis of coded advice-giving strategies in the DCT and corpus data
  • 6.Discussion
    • 6.1Similarities and differences between DCT elicited and naturalistic advice-giving strategies
    • 6.2Designing a DCT with characteristics consistent with the target register to produce an authentic instrument
  • 7.Implications for pragmatics research
    • 7.1Limitations
    • 7.2Future directions
  • Acknowledgments
  • Notes
  • References
Published online: 10 March 2020
https://doi.org/10.1075/ap.19009.has
References
Aijmer, K., & Rühlemann, C.
(Eds.) (2015) Corpus pragmatics. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
Alcón, E., & Safont, P.
(2001) Occurrence of exhortative speech acts in ELT materials and natural speech data: A focus on request, suggestion and advice realization strategies. Studies in English Language and Linguistics, 31, 5–22.Google Scholar
Bachman, L. F.
(2002) Some reflections on task-based language performance assessment. Language Testing, 19(4), 453–476. CrossrefGoogle Scholar
Bachman, L. F., & Palmer, A. S.
(1996) Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.Google Scholar
Baker, P., & Egbert, J.
(Eds.) (2016) Triangulating methodological approaches in corpus linguistic research. New York, NY: Routledge. CrossrefGoogle Scholar
Bardovi-Harlig, K.
(2018) Matching modality in L2 pragmatics research design. System, 751, 13–22. CrossrefGoogle Scholar
Bardovi-Harlig, K., Mossman, S., & Su, Y.
(2017) The effect of corpus-based instruction on pragmatic routines. Language Learning & Technology, 21(3), 76–103.Google Scholar
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E.
(2015) Developing corpus-based materials to teach pragmatic routines. TESOL Journal, 6(3), 499–526. CrossrefGoogle Scholar
Beebe, L. M., & Cummings, M. C.
(1996) Natural speech act data versus written questionnaire data: How data collection method affects speech act performance. In S. Gass & J. Neu (Eds.), Speech acts across cultures: Challenges to communication in a second language (pp. 65–86). Berlin: Mouton de Gruyter.Google Scholar
Biber, D., & Conrad, S.
(2019) Register, genre, and style. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
Biber, D., Conrad, S., Reppen, R., Byrd, P., Helt, M., Clark, V., … Urzua, A.
(2004) Representing language use in the university: Analysis of the TOEFL 2000 spoken and written academic language corpus. (ETS TOEFL Monograph Series, MS-25). Princeton, NJ: Educational Testing Service.Google Scholar
Billmyer, K., & Varghese, M.
(2000) Investigating instrument-based pragmatic variability: effects of enhancing discourse completion tests. Applied Linguistics, 21(4), 517–552. CrossrefGoogle Scholar
Blum-Kulka, S.
(1980) Learning to say what you mean in a second language: A study of the speech act performance of learners of Hebrew as a second language. Applied Linguistics, 31, 29–59. CrossrefGoogle Scholar
Boyatzis, R. E.
(1998) Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.Google Scholar
Braun, V., & Clarke, V.
(2006) Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. CrossrefGoogle Scholar
Brown, P., & Levinson, S. C.
(1987) Politeness: Some universals in language use. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
Canli, Z., & Canli, B.
(2013) Keep calm and say sorry!: The use of apologies by EFL teachers in Turkish and English. Educational Process: International Journal, 2(1), 36–46. CrossrefGoogle Scholar
Chapelle, C. A.
(2012) Validity argument for language assessment: The framework is simple…. Language Testing, 29(1), 19–27. CrossrefGoogle Scholar
Cohen, A. D., & Olshtain, E.
(1994) Researching the production of second language speech acts. In E. T. Tarone, S. M. Gass, & A. D. Cohen (Eds.), Research methodology in second language acquisition (pp. 143–156). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Culpeper, J., Mackey, A., & Taguchi, N.
(2018) Second language pragmatics: From theory to methods. New York, NY: Routledge. CrossrefGoogle Scholar
Cutrona, C. E. & Suhr, J. A.
(1994) Social support communication in the context of marriage: An analysis of couples’ supportive interactions. In B. R. Burleson, T. L. Albrecht, & I. G. Sarason (Eds.), Communication of social support: Messages, interactions, relationships, and community (pp. 113–135). Thousand Oaks, CA: Sage.Google Scholar
DeCapua, A., & Dunham, J. F.
(2007) The pragmatics of advice giving: Cross-cultural perspectives. Intercultural Pragmatics, 4(3), 319–342. CrossrefGoogle Scholar
Eslami-Rasekh, Z.
(2005) Raising the pragmatic awareness of language learners. ELT Journal, 59(3), 199–208. CrossrefGoogle Scholar
Flöck, I., & Geluykens, R.
(2015) Speech acts in corpus pragmatics: A quantitative contrastive study of directives in spontaneous and elicited discourse. In J. Romero-Trillo (Ed.), Yearbook of corpus linguistics and pragmatics 2015: Current approaches to discourse and translation studies (pp. 7–37). Cham: Springer. CrossrefGoogle Scholar
Golato, A.
(2003) Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24(1), 90–121. CrossrefGoogle Scholar
Goldsmith, D. J.
(2004) Communicating social support. New York, NY: Cambridge University Press. CrossrefGoogle Scholar
Grabowski, K. C.
(2007) Reconsidering the measurement of pragmatic knowledge using a reciprocal written test format. Columbia University Working Papers in TESOL & Applied Linguistics, 7(1), 1–48.Google Scholar
Halenko, N., & Jones, C.
(2011) Teaching pragmatic awareness of spoken requests to Chinese EAP learners in the UK: Is explicit instruction effective? System, 39(2), 240–250. CrossrefGoogle Scholar
Hartford, B. S., & Bardovi-Harlig, K.
(1992) Experimental and observational data in the study of interlanguage pragmatics. Pragmatics and Language Learning, 31, 33–52.Google Scholar
Hinkel, E.
(1997) Appropriateness of advice: DCT and multiple choice data. Applied Linguistics, 18(1), 1–26. CrossrefGoogle Scholar
Hong, C. Y., & Shih, S. C.
(2013) Proficiency and complaints: Analyses of production and perceptions. Intergrams, 14(1), 1–20.Google Scholar
Johnston, B., Kasper, G., & Ross, S.
(1998) Effect of rejoinders in production questionnaires. Applied Linguistics, 19(2), 157–182. CrossrefGoogle Scholar
Kasper, G., & Dahl, M.
(1991) Research methods in interlanguage pragmatics. Studies in Second Language Acquisition, 13(2), 215–247. CrossrefGoogle Scholar
Kouper, I.
(2010) The pragmatics of peer advice in a Live Journal community. Language@ internet, 7, article 1.Google Scholar
Labben, A.
(2016) Reconsidering the development of the discourse completion test in interlanguage pragmatics. Pragmatics: Quarterly Publication of the International Pragmatics Association (IPrA), 26(1), 69–91. CrossrefGoogle Scholar
Martínez-Flor, A. M.
(2003) Non-native speakers’ production of advice acts: The effects of proficiency. Revista Española de Lingüística Aplicada, 161, 139–153.Google Scholar
McHugh, M. L.
(2013) The chi-square test of independence. Biochemia Medica, 23(2), 143–149. CrossrefGoogle Scholar
Nguyen, T. T. M.
(2019) Data collection methods in L2 pragmatics research: An overview. In N. Taguchi (Ed.), The Routledge handbook of SLA and pragmatics (pp. 195–211). New York, NY: Routledge.Google Scholar
Parvaresh, V., & Tavakoli, M.
(2009) Discourse completion tasks as elicitation tools: How convergent are they. The Social Sciences, 4(4), 366–373.Google Scholar
Patton, M. Q.
(1990) Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.Google Scholar
Plonsky, L., & Oswald, F. L.
(2014) How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. CrossrefGoogle Scholar
R Core Team
(2013) R: A language and environment for statistical computing (Version 3.4.3) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
Schauer, G. A., & Adolphs, S.
(2006) Expressions of gratitude in corpus and DCT data: Vocabulary, formulaic sequences, and pedagogy. System, 34(1), 119-134.Google Scholar
Staples, S., & Fernández, J.
(2018) Corpus linguistics approaches to L2 pragmatics. In N. Taguchi (Ed.), The Routledge handbook of second language acquisition and pragmatics (241–254). Routledge: London.Google Scholar
Woodfield, H.
(2008) Problematising discourse completion tasks: Voices from verbal report. Evaluation & Research in Education, 21(1), 43–69. CrossrefGoogle Scholar
Yuan, Y.
(2001) An inquiry into empirical pragmatics data-gathering methods: Written DCTs, oral DCTs, field notes, and natural conversations. Journal of Pragmatics, 33(2), 271–292. CrossrefGoogle Scholar
(2002) Compliments and compliment responses in Kunming Chinese. Pragmatics: Quarterly Publication of the International Pragmatics Association (IPrA), 12(2), 183–226. CrossrefGoogle Scholar
Cited by

Cited by 3 other publications

Hashimoto, Brett, Daniel Keller, Ekaterina Sudina, Katherine Yaw, Jesse Egbert & Luke Plonsky
2020. Research in progress: Applied linguistics at Northern Arizona University, USA. Language Teaching 53:2 ► pp. 227 ff. Crossref logo
Taguchi, Naoko
2022. Dispreferred speech acts in virtual reality: Analysis of tone choices and hesitations. System ► pp. 102793 ff. Crossref logo
Taguchi, Naoko
2022. Immersive Virtual Reality for Pragmatics Task Development. TESOL Quarterly 56:1 ► pp. 308 ff. Crossref logo

This list is based on CrossRef data as of 03 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.