Cover not available
Article published In:
Applied Pragmatics
Vol. 2:1 (2020) ► pp.80–120

Using a corpus in creating and evaluating a DCT

| Northern Arizona University
| Northern Arizona University

Discourse Completion Tasks (DCTs) have been one of the most popular tools in pragmatics research. Yet, many have criticized DCTs for their lack of authenticity (e.g., Culpeper, Mackey, & Taguchi, 2018; Nguyen, 2019). We propose that corpora can serve as resources in designing and evaluating DCTs. We created a DCT using advice-seeking prompts from the Q+A corpus (Baker & Egbert, 2016). Then, we administered the DCT to 33 participants. We evaluated the DCT by (1) comparing the linguistic form and the semantic content of the participants’ DCT responses (i.e., advice-giving expressions) with authentic data from the corpus; and (2) interviewing the participants about the instrument quality. Chi-square tests between DCT data and corpus data revealed no significant differences in advice-giving expressions in terms of both the overall level of directness (χ2 [2, N = 660] = 6.94, p = .03, V = .10) and linguistic realization (χ2 [8, N = 660] = 17.75, p = .02, V = .16), and showed a significant difference but small effect size in terms of semantic content (χ2 [6, N = 512] = 30.35, p < .01, V = .24). Taken together with the interview data, our findings indicate that corpora are useful in designing DCTs.

Keywords: discourse completion task, corpus-based instrument, advice-giving, technology-mediated communication, authenticity, pragmatics
Article outline
  • 1.Introduction
  • 2.Literature review
    • 2.1DCTs
    • 2.2Criticism of DCTs
    • 2.3Applications of corpora in instrument design and pragmatics research
    • 2.4The speech act of advice-giving
  • 3.The present study
  • 4.Methods
    • 4.1Participants
    • 4.2Corpus
    • 4.3DCT instrument
    • 4.4Comparison of advice-giving strategies between DCT and corpus
      • 4.4.1Coding schemes for advice-giving strategies
      • 4.4.2Statistical analysis of advice-giving strategies
    • 4.5Interview
      • 4.5.1Interview procedures
      • 4.5.2Thematic analysis of interview data
  • 5.Results
    • 5.1Thematic analysis of interview data
    • 5.2Statistical analysis of coded advice-giving strategies in the DCT and corpus data
  • 6.Discussion
    • 6.1Similarities and differences between DCT elicited and naturalistic advice-giving strategies
    • 6.2Designing a DCT with characteristics consistent with the target register to produce an authentic instrument
  • 7.Implications for pragmatics research
    • 7.1Limitations
    • 7.2Future directions
  • Acknowledgments
  • Notes
  • References
Published online: 10 March 2020
DOI logo
https://doi.org/10.1075/ap.19009.has
References
Aijmer, K., & Rühlemann, C.
(Eds.) (2015) Corpus pragmatics. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Alcón, E., & Safont, P.
(2001) Occurrence of exhortative speech acts in ELT materials and natural speech data: A focus on request, suggestion and advice realization strategies. Studies in English Language and Linguistics, 31, 5–22.Google Scholar
Bachman, L. F.
(2002) Some reflections on task-based language performance assessment. Language Testing, 19(4), 453–476. DOI logoGoogle Scholar
Bachman, L. F., & Palmer, A. S.
(1996) Language testing in practice: Designing and developing useful language tests. Oxford: Oxford University Press.Google Scholar
Baker, P., & Egbert, J.
(Eds.) (2016) Triangulating methodological approaches in corpus linguistic research. New York, NY: Routledge. DOI logoGoogle Scholar
Bardovi-Harlig, K.
(2018) Matching modality in L2 pragmatics research design. System, 751, 13–22. DOI logoGoogle Scholar
Bardovi-Harlig, K., Mossman, S., & Su, Y.
(2017) The effect of corpus-based instruction on pragmatic routines. Language Learning & Technology, 21(3), 76–103.Google Scholar
Bardovi-Harlig, K., Mossman, S., & Vellenga, H. E.
(2015) Developing corpus-based materials to teach pragmatic routines. TESOL Journal, 6(3), 499–526. DOI logoGoogle Scholar
Beebe, L. M., & Cummings, M. C.
(1996) Natural speech act data versus written questionnaire data: How data collection method affects speech act performance. In S. Gass & J. Neu (Eds.), Speech acts across cultures: Challenges to communication in a second language (pp. 65–86). Berlin: Mouton de Gruyter.Google Scholar
Biber, D., & Conrad, S.
(2019) Register, genre, and style. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Biber, D., Conrad, S., Reppen, R., Byrd, P., Helt, M., Clark, V., … Urzua, A.
(2004) Representing language use in the university: Analysis of the TOEFL 2000 spoken and written academic language corpus. (ETS TOEFL Monograph Series, MS-25). Princeton, NJ: Educational Testing Service.Google Scholar
Billmyer, K., & Varghese, M.
(2000) Investigating instrument-based pragmatic variability: effects of enhancing discourse completion tests. Applied Linguistics, 21(4), 517–552. DOI logoGoogle Scholar
Blum-Kulka, S.
(1980) Learning to say what you mean in a second language: A study of the speech act performance of learners of Hebrew as a second language. Applied Linguistics, 31, 29–59. DOI logoGoogle Scholar
Boyatzis, R. E.
(1998) Transforming qualitative information: Thematic analysis and code development. Thousand Oaks, CA: Sage.Google Scholar
Braun, V., & Clarke, V.
(2006) Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. DOI logoGoogle Scholar
Brown, P., & Levinson, S. C.
(1987) Politeness: Some universals in language use. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Canli, Z., & Canli, B.
(2013) Keep calm and say sorry!: The use of apologies by EFL teachers in Turkish and English. Educational Process: International Journal, 2(1), 36–46. DOI logoGoogle Scholar
Chapelle, C. A.
(2012) Validity argument for language assessment: The framework is simple…. Language Testing, 29(1), 19–27. DOI logoGoogle Scholar
Cohen, A. D., & Olshtain, E.
(1994) Researching the production of second language speech acts. In E. T. Tarone, S. M. Gass, & A. D. Cohen (Eds.), Research methodology in second language acquisition (pp. 143–156). Hillsdale, NJ: Lawrence Erlbaum Associates.Google Scholar
Culpeper, J., Mackey, A., & Taguchi, N.
(2018) Second language pragmatics: From theory to methods. New York, NY: Routledge. DOI logoGoogle Scholar
Cutrona, C. E. & Suhr, J. A.
(1994) Social support communication in the context of marriage: An analysis of couples’ supportive interactions. In B. R. Burleson, T. L. Albrecht, & I. G. Sarason (Eds.), Communication of social support: Messages, interactions, relationships, and community (pp. 113–135). Thousand Oaks, CA: Sage.Google Scholar
DeCapua, A., & Dunham, J. F.
(2007) The pragmatics of advice giving: Cross-cultural perspectives. Intercultural Pragmatics, 4(3), 319–342. DOI logoGoogle Scholar
Eslami-Rasekh, Z.
(2005) Raising the pragmatic awareness of language learners. ELT Journal, 59(3), 199–208. DOI logoGoogle Scholar
Flöck, I., & Geluykens, R.
(2015) Speech acts in corpus pragmatics: A quantitative contrastive study of directives in spontaneous and elicited discourse. In J. Romero-Trillo (Ed.), Yearbook of corpus linguistics and pragmatics 2015: Current approaches to discourse and translation studies (pp. 7–37). Cham: Springer. DOI logoGoogle Scholar
Golato, A.
(2003) Studying compliment responses: A comparison of DCTs and recordings of naturally occurring talk. Applied Linguistics, 24(1), 90–121. DOI logoGoogle Scholar
Goldsmith, D. J.
(2004) Communicating social support. New York, NY: Cambridge University Press. DOI logoGoogle Scholar
Grabowski, K. C.
(2007) Reconsidering the measurement of pragmatic knowledge using a reciprocal written test format. Columbia University Working Papers in TESOL & Applied Linguistics, 7(1), 1–48.Google Scholar
Halenko, N., & Jones, C.
(2011) Teaching pragmatic awareness of spoken requests to Chinese EAP learners in the UK: Is explicit instruction effective? System, 39(2), 240–250. DOI logoGoogle Scholar
Hartford, B. S., & Bardovi-Harlig, K.
(1992) Experimental and observational data in the study of interlanguage pragmatics. Pragmatics and Language Learning, 31, 33–52.Google Scholar
Hinkel, E.
(1997) Appropriateness of advice: DCT and multiple choice data. Applied Linguistics, 18(1), 1–26. DOI logoGoogle Scholar
Hong, C. Y., & Shih, S. C.
(2013) Proficiency and complaints: Analyses of production and perceptions. Intergrams, 14(1), 1–20.Google Scholar
Johnston, B., Kasper, G., & Ross, S.
(1998) Effect of rejoinders in production questionnaires. Applied Linguistics, 19(2), 157–182. DOI logoGoogle Scholar
Kasper, G., & Dahl, M.
(1991) Research methods in interlanguage pragmatics. Studies in Second Language Acquisition, 13(2), 215–247. DOI logoGoogle Scholar
Kouper, I.
(2010) The pragmatics of peer advice in a Live Journal community. Language@ internet, 7, article 1.Google Scholar
Labben, A.
(2016) Reconsidering the development of the discourse completion test in interlanguage pragmatics. Pragmatics: Quarterly Publication of the International Pragmatics Association (IPrA), 26(1), 69–91. DOI logoGoogle Scholar
Martínez-Flor, A. M.
(2003) Non-native speakers’ production of advice acts: The effects of proficiency. Revista Española de Lingüística Aplicada, 161, 139–153.Google Scholar
McHugh, M. L.
(2013) The chi-square test of independence. Biochemia Medica, 23(2), 143–149. DOI logoGoogle Scholar
Nguyen, T. T. M.
(2019) Data collection methods in L2 pragmatics research: An overview. In N. Taguchi (Ed.), The Routledge handbook of SLA and pragmatics (pp. 195–211). New York, NY: Routledge.Google Scholar
Parvaresh, V., & Tavakoli, M.
(2009) Discourse completion tasks as elicitation tools: How convergent are they. The Social Sciences, 4(4), 366–373.Google Scholar
Patton, M. Q.
(1990) Qualitative evaluation and research methods (2nd ed.). Newbury Park, CA: Sage.Google Scholar
Plonsky, L., & Oswald, F. L.
(2014) How big is “big”? Interpreting effect sizes in L2 research. Language Learning, 64(4), 878–912. DOI logoGoogle Scholar
R Core Team
(2013) R: A language and environment for statistical computing (Version 3.4.3) [Computer software]. Vienna, Austria: R Foundation for Statistical Computing.Google Scholar
Schauer, G. A., & Adolphs, S.
(2006) Expressions of gratitude in corpus and DCT data: Vocabulary, formulaic sequences, and pedagogy. System, 34(1), 119-134.Google Scholar
Staples, S., & Fernández, J.
(2018) Corpus linguistics approaches to L2 pragmatics. In N. Taguchi (Ed.), The Routledge handbook of second language acquisition and pragmatics (241–254). Routledge: London.Google Scholar
Woodfield, H.
(2008) Problematising discourse completion tasks: Voices from verbal report. Evaluation & Research in Education, 21(1), 43–69. DOI logoGoogle Scholar
Yuan, Y.
(2001) An inquiry into empirical pragmatics data-gathering methods: Written DCTs, oral DCTs, field notes, and natural conversations. Journal of Pragmatics, 33(2), 271–292. DOI logoGoogle Scholar
(2002) Compliments and compliment responses in Kunming Chinese. Pragmatics: Quarterly Publication of the International Pragmatics Association (IPrA), 12(2), 183–226. DOI logoGoogle Scholar