Chapter 9. Task design and validity evidence for assessment of L2 pragmatics in interaction

This study examines how clear communicative goals and authentic interaction are ensured in task-based pragmatic assessment practices, particularly in designing role-play assessment tasks and developing task-appropriate rating criteria. Employing a concurrent mixed methods design, conversation analysis (CA) and multi-faceted Rasch measurement were used to investigate whether task-independent interactional features are elicited from role-play interactions and how interaction-specific rating categories function quantitatively. The CA findings indicated that various interactional organizations emerged from role-play task interactions as examinees oriented to the different degree of imposition and contextual variables embedded in the role-plays. Further, the interactional features included in the rating criteria created a substantial amount of variance in distinguishing varying degrees of pragmatic performance. The findings are discussed in terms of designing valid pragmatic assessment tasks and what features need to be included in developing rating criteria to ensure the validity evidence of task-based pragmatic assessment.