Chapter published in:
Task-Based Approaches to Teaching and Assessing Pragmatics
Edited by Naoko Taguchi and YouJin Kim
[Task-Based Language Teaching 10] 2018
► pp. 218–246

Chapter 9
Task design and validity evidence for assessment of L2 pragmatics in interaction

| Northern Arizona University
This study examines how clear communicative goals and authentic interaction are ensured in task-based pragmatic assessment practices, particularly in designing role-play assessment tasks and developing task-appropriate rating criteria. Employing a concurrent mixed methods design, conversation analysis (CA) and multi-faceted Rasch measurement were used to investigate whether task-independent interactional features are elicited from role-play interactions and how interaction-specific rating categories function quantitatively. The CA findings indicated that various interactional organizations emerged from role-play task interactions as examinees oriented to the different degree of imposition and contextual variables embedded in the role-plays. Further, the interactional features included in the rating criteria created a substantial amount of variance in distinguishing varying degrees of pragmatic performance. The findings are discussed in terms of designing valid pragmatic assessment tasks and what features need to be included in developing rating criteria to ensure the validity evidence of task-based pragmatic assessment.
Article outline
  • Introduction
  • Background
    • Assessing pragmatic competence in interaction and validity challenges
    • Role-plays as assessment tasks and eliciting validity evidence
    • The present study
  • Methods
    • Participants
      • Examinees
      • Interlocutors
      • Raters
    • Test instruments
      • Role-plays
      • Rating criteria
    • Procedures
      • Test administration
      • Rater training
      • Rating
    • Data analysis
      • Conversation analysis
      • Multi-faceted Rasch measurement
  • Results
    • Conversation Analysis (CA) findings
      • Recommendation letter request role-play
      • Additional advising time request role-play
      • Refusing a professor’s request role-play
      • Deciding a meeting time
      • Deciding a meeting mode
    • Multi-faceted Rasch Measurement analysis
  • Discussion
  • Conclusion
  • References
Published online: 15 August 2018
https://doi.org/10.1075/tblt.10.09you
References
Al-Gahtani, S., & Roever, C.
(2012) Proficiency and sequential organization of L2 requests. Applied Linguistics, 33, 42–65. CrossrefGoogle Scholar
(2014) Insert and post-expansion in L2 Arabic requests. System, 42, 189–206. CrossrefGoogle Scholar
Atkinson, J., & Heritage, J.
(Eds.) (1984) Structures of social action. Cambridge: Cambridge University Press.Google Scholar
Bardovi-Harlig, K., & Hartford, B. S.
(Eds.) (2005) Interlanguage pragmatics: Exploring institutional talk. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Bond, T. G., & Fox, C. M.
(2007) Applying the Rasch model (2nd ed.). New York, NY: Routledge.Google Scholar
Celce-Murcia, M.
(2007) Rethinking the role of communicative competence in language teaching. In E. Alcón Soler & M. P. Safont Jordà (Eds.), Intercultural language use and language learning (pp. 41–57). Dordrecht: Springer. CrossrefGoogle Scholar
Celce-Murcia, M., Dörnyei, Z., & Thurrell, S.
(1995) Communicative competence: A pedagogically motivated model with content specifications. Issues in Applied Linguistics, 6, 5–35.Google Scholar
Chapelle, C. A.
(2012) Validity argument for language assessment: The framework is simple… Language Testing, 29, 19–27. CrossrefGoogle Scholar
Chapelle, C., Enright, M., & Jamieson, J.
(Eds.) (2008) Building a validity argument for the Test of English as a Foreign Language. New York, NY: Routledge.Google Scholar
Eckes, T.
(2011) Introduction to many-Facet Rasch measurement. Frankfurt: Peter Lang.CrossrefGoogle Scholar
Grabowski, K.
(2013) Investigating the construct validity of a role-play test designed to measure grammatical and pragmatic knowledge at multiple proficiency levels. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 149–171). Houndmills: Palgrave Macmillan. CrossrefGoogle Scholar
Heritage, J.
(1984) A change-of-state token and aspects of its sequential placement. In M. J. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 299–345). Cambridge: Cambridge University Press.Google Scholar
Hudson, T. D., Detmer, E., & Brown, J. D.
(1992) A framework for testing cross-cultural pragmatics (Technical Report #2). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.Google Scholar
(1995) Developing prototype measures of cross-cultural pragmatics (Technical Report #7). Honolulu, HI: University of Hawai‘i, Second Language Teaching and Curriculum Center.Google Scholar
Hughes, A.
(1989) Testing for language teachers. Cambridge: Cambridge University Press.Google Scholar
Huth, T.
(2010) Can talk be inconsequential? Social and interactional aspects of elicited second-language interaction. Modern Language Journal, 94, 537–553. CrossrefGoogle Scholar
Kane, M.
(1992) An argument-based approach to validity. Psychological Bulletin, 112, 527–535. CrossrefGoogle Scholar
(2006) Validation. In R. L. Brennan (Ed.), Educational measurement (pp. 17–64). Westport, CT: American Council on Education.Google Scholar
Kasper, G.
(2006) Speech acts in interaction: Towards discursive pragmatics. In K. Bardovi-Harlig, C. Félix-Brasdefer & A. S. Omar (Eds.), Pragmatics and language learning (pp. 281–314). Honolulu, HI: Second Language Teaching and Curriculum Center, University of Hawai‘i.Google Scholar
Kasper, G., & Rose, K. R.
(2002) Pragmatic development in a second language. Malden, MA: Blackwell Publishing.Google Scholar
Kasper, G.
(2009) L2 pragmatic development. In W. C. Ritchie & T. K. Bhatia (Eds.), New handbook of second language acquisition (pp. 259–295). Leeds: Emerald.Google Scholar
Kasper, G., & Ross, S.
(2013) Assessing second language pragmatics: An overview and introductions. In S. J. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 1–40). Houndmills: Palgrave Macmillan. CrossrefGoogle Scholar
Kasper, G., & Youn, S. J.
(2017) Transforming instruction to activity: Roleplay in language assessment. Applied Linguistics Review. Advance online publication. CrossrefGoogle Scholar
Linacre, J. M.
(2014) Facets Rasch measurement computer program, version 3.71.4. Chicago, IL: Winsteps.com.Google Scholar
Liu, J.
(2007) Developing a pragmatic test for Chinese EFL learners. Language Testing, 24, 391–415. CrossrefGoogle Scholar
Lunz, M. E., Wright, B. D., & Linacre, J. M.
(1990) Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3, 331–345. CrossrefGoogle Scholar
McNamara, T.
(1996) Measuring second language performance. New York, NY: Addison Wesley Longman.Google Scholar
McNamara, T., & Knoch, U.
(2012) The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing, 29, 555–576. CrossrefGoogle Scholar
Norris, J. M.
(2009) Task-based teaching and testing. In M. H. Long & C. J. Doughty (Eds.), Handbook of language teaching (pp. 578–594). Oxford: Blackwell. CrossrefGoogle Scholar
(2016) Current uses for task-based language assessment. Annual Review of Applied Linguistics, 36, 230–244.CrossrefGoogle Scholar
Okada, Y.
(2010) Role-play in oral proficiency interviews: Interactive footing and interactional competencies. Journal of Pragmatics, 42, 1647–1668. CrossrefGoogle Scholar
Okada, Y., & Greer, T.
(2013) Pursuing a relevant response in oral proficiency interview role plays. In S. Ross & G. Kasper (Eds.), Assessing second language pragmatics (pp. 288–310). Houndmills: Palgrave Macmillan. CrossrefGoogle Scholar
Pomerantz, A.
(1984) Agreeing and disagreeing with assessments: Some features of preferred/dispreferred turn shapes. In J. M. Atkinson & J. Heritage (Eds.), Structures of social action: Studies in conversation analysis (pp. 57–101). Cambridge: Cambridge University Press.Google Scholar
Pomerantz, A., & Heritage, J.
(2013) Preference. In J. Sidnell & T. Stivers (Eds.), The handbook of conversation analysis (pp. 210–228). Chichester: Wiley-Blackwell.Google Scholar
Purpura, J. E., Brown, J. D., & Schoonen, R.
(2015) Improving the validity of quantitative measures in applied linguistics research. Language Learning, 65, 37–75. CrossrefGoogle Scholar
Roever, C.
(2006) Validation of a web-based test of ESL pragmalinguistics. Language Testing, 23, 229–256. CrossrefGoogle Scholar
(2011) Testing of second language pragmatics: Past and future. Language Testing, 28, 463–481. CrossrefGoogle Scholar
Ross, S. J., & Kasper, G.
(Eds.) (2013) Assessing second language pragmatics. Houndmills: Palgrave Macmillan. CrossrefGoogle Scholar
Sacks, H.
(1992) Lectures on conversation (2 Vols.). Oxford: Blackwell.Google Scholar
Schegloff, E. A.
(2007) Sequence organization in interaction: A primer in conversation analysis. Cambridge: Cambridge University Press. CrossrefGoogle Scholar
Schumacker, R. E.
(1999) Many-facet Rasch analysis with crossed, nested, and mixed designs. Journal of Outcome Measurement, 3, 323–338.Google Scholar
Taguchi, N., & Roever, C.
(2017) Second language pragmatics. Oxford: Oxford University Press.Google Scholar
Teddlie, C., & Tashakkori, A.
(2006) A general typology of research designs featuring mixed methods. Research in the Schools, 13(1), 12–28.Google Scholar
Walters, F. S.
(2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing, 24, 155–183. CrossrefGoogle Scholar
Youn, S. J.
(2010) Task-based needs analysis of L2 pragmatics in an EAP context: Implications for teaching. Unpublished manuscript, University of Hawai‘i.Google Scholar
(2015) Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32, 199–225. CrossrefGoogle Scholar