Cover not available
Article published In:
Journal of Second Language Studies
Vol. 6:1 (2023) ► pp.61–94

Developing and evaluating a contextualized interactional competence rating scale based on a metaphorical conceptualization

A pragmatic mixed-method approach

| Iowa State University of Science and Technology, United States

Rating scales developed to measure interactional competence (IC) are mainly data-driven which can incur unaffordable costs for assessment practitioners with limited resources, such as universities using this test for placement purposes. A cheaper alternative is to use models proposed in the literature to develop such rating scales. This mixed-methods study evaluates a contextualized analytic IC rating scale developed based on Galaczi and Taylor’s (2018) metaphorical conceptualization of IC. A many-facet Rasch analysis indicated this rating scale could separate examinees into three levels of ability, which was lower than the expected four levels. To explore this finding, raters were interviewed about their experiences of using this rating scale. The findings show the potential and limitations of developing contextualized IC ratings scales based on literature and possible avenues for future research. They also suggest a direct interaction between assessment and instructional practices of IC with implications for language teaching.

Keywords: interactional competence, paired oral discussion, rating scale, oral proficiency assessment, interactional competence definition evaluation, pragmatism, mixed-methods evaluation, instruction of interactional competence
Article outline
  • 1.Introduction
  • 2.Literature review
  • 3.Material and methods
    • 3.1Participants
    • 3.2Materials
      • 3.2.1Videos
      • 3.2.2Rating scale
    • 3.3Procedures
  • 4.Results
    • 4.1Quantitative analysis
    • 4.2Qualitative analysis
  • 5.Discussion
  • 6.Conclusion
  • References
Published online: 10 January 2023
DOI logo
https://doi.org/10.1075/jsls.22003.nei
References
Bachman, L. F.
(2004) Fundamental considerations in language testing (7. impr). Oxford Univ. Press.Google Scholar
Bachman, L. F., & Palmer, A. S.
(1996) Language testing in practice. Oxford University Press.Google Scholar
Bonk, W. J., & Ockey, G. J.
(2003) A many-facet Rasch analysis of the second language group oral discussion task. Language Testing, 20 (1), 89–110. DOI logoGoogle Scholar
Borger, L.
(2019) Assessing interactional skills in a paired speaking test. Apples – Journal of Applied Language Studies, 13 (1), 151–174. DOI logoGoogle Scholar
Canale, M.
(1983) From communicative competence to communicative langauge pedagogy. In J. C. Richards & R. W. Schmidt (Eds.), Language and communication (pp. 2–28). Longman.Google Scholar
Canale, M., & Swain, M.
(1980) Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics, 1 (1), 1–47. DOI logoGoogle Scholar
Creswell, J. W.
(2010) Mapping th edeveloping landscape of mixed methods research. In A. Tashakkori & C. Teddlie (Eds.), Sage handbook of mixed methods in social and behavioral research (2nd ed., pp. 45–68). SAGE Publications. DOI logoGoogle Scholar
Dörnyei, Z.
(2007) Research methods in applied linguistics: Quantitative, qualitative, and mixed methodologies. Oxford University Press.Google Scholar
Ducasse, A. M., & Brown, A.
(2009) Assessing paired orals: Raters’ orientation to interaction. Language Testing, 26 (3), 423–443. DOI logoGoogle Scholar
Eckes, T.
(2015) Introduction to Many-Facet Rasch Measurement: Analyzing and evaluating rater-mediated assessments. Peter Lang GmbH, Internationaler Verlag der Wissenschaften.Google Scholar
Ezzy, D.
(2002) Qualitative analysis: Practice and innovation. Routledge.Google Scholar
Fulcher, G.
(2010) Practical language testing. Hodder Education.Google Scholar
Galaczi, E. D.
(2014) Interactional Competence across Proficiency Levels: How do Learners Manage Interaction in Paired Speaking Tests? Applied Linguistics, 35 (5), 553–574. DOI logoGoogle Scholar
Galaczi, E., & Taylor, L.
(2018) Interactional Competence: Conceptualisations, Operationalisations, and Outstanding Questions. Language Assessment Quarterly, 15 (3), 219–236. DOI logoGoogle Scholar
Galaczi, E., & Taylor, L. B.
(2021) Measuring interactional competence. In P. Winke & T. Brunfaut (Eds.), The Routledge handbook of second language acquisition and language testing (pp. 338–348). Routledge.Google Scholar
Hughes, R.
(2007) Testing the visible: Literate biases in oral language testing. Journal of Applied Linguistics and Professional Practice, 1 (3), 295–309. DOI logoGoogle Scholar
Hymes, D.
(1972) On communicative competence. In J. B. Pride & J. Holmes (Eds.), Sociolinguistics: Selected readings (pp. 53–73). Penguin.Google Scholar
Kim, H.
(2018) What constitutes professional communication in aviation: Is language proficiency enough for testing purposes? Language Testing, 35 (3), 403–426. DOI logoGoogle Scholar
Kramsch, C.
(1986) From Language Proficiency to Interactional Competence. The Modern Language Journal, 70 (4), 366–372. DOI logoGoogle Scholar
Lam, D. M. K.
(2018) What counts as “responding”? Contingency on previous speaker contribution as a feature of interactional competence. Language Testing, 35 (3), 377–401. DOI logoGoogle Scholar
(2021) Don’t Turn a Deaf Ear: A Case for Assessing Interactive Listening. Applied Linguistics, 42 (4), 740–764. DOI logoGoogle Scholar
Lantolf, J. P.
(Ed.) (2000) Sociocultural theory and second language learning. Oxford Univ. Press.Google Scholar
Lazaraton, A.
(1996) Interlocutor support in oral proficiency interviews: The case of CASE. Language Testing, 13 (2), 151–172. DOI logoGoogle Scholar
Linacre, J. M.
(1998) FACETS 3.17. Computer Program. MESA Press.Google Scholar
(2021) Winsteps® Rasch measurement computer program (5.1.1) [Computer software]. [URL]
Mackey, A., & Gass, S. M.
(2010) Second language research: Methodology and design (Repr). Routledge.Google Scholar
May, L.
(2011) Interactional Competence in a Paired Speaking Test: Features Salient to Raters. Language Assessment Quarterly, 8 (2), 127–145. DOI logoGoogle Scholar
May, L., Nakatsuhara, F., Lam, D., & Galaczi, E.
(2020) Developing tools for learning oriented assessment of interactional competence: Bridging theory and practice. Language Testing, 37 (2), 165–188. DOI logoGoogle Scholar
McNamara, T. F., & Roever, C.
(2006) Language testing: The social dimension. Blackwell Pub.Google Scholar
Merriam, S. B., & Tisdell, E. J.
(2016) Qualitative research: A guide to design and implementation (Fourth edition). Jossey-Bass.Google Scholar
Nakatsuhara, F., May, L., Lam, D., & Galaczi, E.
(2018) Learning oriented feedback in the development and assessment of interactional competence. Reserach Notes, 70 1, 1–68.Google Scholar
Ockey, G. J.
(2012) Item response theory. In G. Fulcher & F. Davidson (Eds.), The Routledge handbook of language testing (pp. 336–349).Google Scholar
Ockey, G. J., & Chukharev-Hudilainen, E.
(2021) Human versus computer partner in the paired oral discussion test. Applied Linguistics, 42(5), 924–944. DOI logoGoogle Scholar
Ockey, G. J., Koyama, D., Setoguchi, E., & Sun, A.
(2015) The extent to which TOEFL iBT speaking scores are associated with performance on oral language tasks and oral ability components for Japanese university students. Language Testing, 32 (1), 39–62. DOI logoGoogle Scholar
O’Sullivan, B.
(2000) Exploring gender and oral proficiency interview performance. System, 28 (3), 373–386. DOI logoGoogle Scholar
O’Sullivan, B., Weir, C. J., & Saville, N.
(2002) Using observation checklists to validate speaking-test tasks. Language Testing, 19 (1), 33–56. DOI logoGoogle Scholar
Pekarek Doehler, S., & Pochon-Berger, E.
(2011) Developing “methods” for interaction: A cross-sectional study of disagreement sequences in French L2. In J. K. Hall, J. Hellermann, & S. P. Doehler (Eds.), L2 interactional competence and development (pp. 206–243). Multilingual Matters. DOI logoGoogle Scholar
(2015) The development of L2 interactional competence: Evidence from turn-taking organization, sequence organization, repair organization, an preference organization. In T. Cadierno & S. W. Eskildsen (Eds.), Usage-based perspectives on second language learning (pp. 233–268). De Gruyter Mouton. DOI logoGoogle Scholar
Plough, I., Banerjee, J., & Iwashita, N.
(2018) Interactional competence: Genie out of the bottle. Language Testing, 35 (3), 427–445. DOI logoGoogle Scholar
Roever, C., & Ikeda, N.
(2021) What scores from monologic speaking tests can(not) tell us about interactional competence. Language Testing, 026553222110033. DOI logoGoogle Scholar
Roever, C., & Kasper, G.
(2018) Speaking in turns and sequences: Interactional competence as a target construct in testing speaking. Language Testing, 35 (3), 331–355. DOI logoGoogle Scholar
Ross, S.
(2018) Listener response as a facet of interactional competence. Language Testing, 35 (3), 357–375. DOI logoGoogle Scholar
Sandlund, E., & Greer, T.
(2020) How do raters understand rubrics for assessing L2 interactional engagement? A comparative study of CA-and non-CA-formulated performance descriptors. Papers in Language Testing and Assessment: An International Journal of the Association for Language Testing and Assessment of Australia and New Zealand, 9 (1), 128–163.Google Scholar
Sandlund, E., Sundqvist, P., & Nyroos, L.
(2016) Testing L2 Talk: A Review of Empirical Studies on Second-Language Oral Proficiency Testing: Testing L2 Talk. Language and Linguistics Compass, 10 (1), 14–29. DOI logoGoogle Scholar
Vo, S.
(2021) Evaluating interactional competence in interview and paired discussion tasks: A rater cognition study. TESOL Journal, 12 (2). DOI logoGoogle Scholar
Wright, B. D.
(1996) Localy dependency, correlations and prinicpal components. Rasch Measurement Transactions, 10 1, 509–511.Google Scholar
Wu, S., & Ma, Z.
(2020) How is Chinese reading affected by under-specification and over-specification? Evidence from self-paced reading experiments. Journal of Pragmatics, 1551, 213–233. DOI logoGoogle Scholar
Youn, S. J.
(2015) Validity argument for assessing L2 pragmatics in interaction using mixed methods. Language Testing, 32 (2), 199–225. DOI logoGoogle Scholar
Young, R.
(2008) Language and interaction: An advanced resource book. Routledge.Google Scholar