Cover not available
Article published In:
TASK
Vol. 2:2 (2022) ► pp.184–217

Designing speaking tasks for different assessment goals

The complex relationship between cognitive task complexity, language performance, and task accomplishment

ORCID logo | Waseda University, Japan
 | ETS Japan
This study explored the role of cognitive task complexity in designing speaking assessment tasks. A sample of 120 English learners at different proficiency levels engaged in four narrative tasks with distinct levels/types of cognitive complexity. Performances were assessed using linguistic measures and a task accomplishment rubric. Findings revealed that one of the middle-complexity tasks with moderate intrinsic, moderate germane, and low extraneous complexity elicited the best linguistic performances overall, while the lowest complexity task elicited the highest task accomplishment ratings. Focusing on distinguishing among learner abilities, however, the highest complexity task demonstrated the best discrimination. Cognitive task complexity was also found to moderate the relationship between task accomplishment ratings and linguistic measures. Implications for research into task design, learner performance, and assessment purpose are considered.
Keywords: assessment task design, CALF measures, cognitive task complexity, discrimination, task accomplishment
Article outline
  • Introduction
  • Task-Based Language Assessment, task design, and cognitive task complexity
  • Task performance: Linguistic indices or task accomplishment?
  • Task design for different purposes
  • The current study
    • Methods
    • Participants
    • Materials
      • Narrative tasks
      • Cloze test
    • Procedures
    • Data scoring, coding, and analysis
      • Task accomplishment ratings
      • Linguistic indices
      • Syntactic complexity
      • Accuracy
      • Lexis
      • Fluency
      • Inter-coder reliability
      • Cloze test
      • Statistical analysis
  • Results
    • Proficiency test scores
    • Learner task performance
    • Correlations among performance measures and task accomplishment
    • Discrimination
  • Discussion
    • Role of task design in maximizing task performance
    • Role of task design in discriminating among proficiency levels
    • Role of task design in understanding the relationship among performance measures
  • Limitations
  • Conclusion
  • Acknowledgements
  • Note
  • References
Published online: 24 January 2023
DOI logo
https://doi.org/10.1075/task.21020.sas
References (48)
References
Brown, J. D. (1980). Relative merits of four methods for scoring cloze tests. The Modern Language Journal, 64 1, 311–317. DOI logoGoogle Scholar
(2005). Testing in language programs: a comprehensive guide to English language assessment. McGraw-Hill College.Google Scholar
Brown, J. D., & Grüter, T. (2020). The same cloze for all occasions? Using the Brown (1980) cloze test for measuring proficiency in SLA research. International Review of Applied Linguistics. DOI logoGoogle Scholar
Brown, J. D., & Hudson, T. (1998). The alternatives in language assessment. TESOL quarterly, 32 (4), 653–675. DOI logoGoogle Scholar
Brown, J. D., Hudson, T., Norris, J. M., & Bonk, W. (2002). Investigating task-based second language performance assessment. University of Hawai‘i Press.Google Scholar
De Jong, N. H. (2018). Fluency in second language testing: Insights from different disciplines. Language Assessment Quarterly, 15 (3), 237–254. DOI logoGoogle Scholar
De Jong, N. H., Steinel, M., Florijn, A. F., Schoonen, R., & Hulstijn, J. H. (2012). The effect of task complexity on functional adequacy, fluency and lexical diversity in speaking performances of native and non-native speakers. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA (pp. 121–142). John Benjamins. DOI logoGoogle Scholar
Doughty, C., & Pica, T. (1986). “Information gap” tasks: Do they facilitate second language acquisition? TESOL Quarterly, 20 (2), 305–325. DOI logoGoogle Scholar
Educational Testing Service. (2019). TOEFL iBT® test speaking rubrics. [URL]
Freed, B. F., Segalowitz, N., & Dewey, D. P. (2004). Learning context and its effects on second language acquisition. Studies in Second Language Acquisition, 26 1, 275–301.Google Scholar
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I. (2019). Multiple approaches to complexity in second language research. Second language research, 35 (1), 3–21. DOI logoGoogle Scholar
Housen, A., Kuiken, F., & Vedder, I. (Eds.). (2012). Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA. John Benjamins. DOI logoGoogle Scholar
Hunt, K. W. (1970). Syntactic maturity in school children and adults. Monographs of the Society for Research in Child Development, 35 1, 1–67. DOI logoGoogle Scholar
Koizumi, R. (2012). Relationships between text length and lexical diversity measures: Can we use short texts of less than 100 tokens? Vocabulary Learning and Instruction, 1 (1), 60–69. DOI logoGoogle Scholar
Kuiken, F., & Vedder, I. (2017). Functional adequacy in L2 writing: Towards a new rating scale. Language Testing, 34 (3), 321–336. DOI logoGoogle Scholar
Kuiken, F., Mos, M., & Vedder, I. (2005). Cognitive task complexity and second language writing performance. Eurosla Yearbook, 5 (1), 195–222. DOI logoGoogle Scholar
Kuiken, F., Vedder, I., & Gilabert, R. (2010). Communicative adequacy and linguistic complexity in L2 writing. Communicative proficiency and linguistic development: Intersections between SLA and language testing research, EUROSLA Monograph Series 1 1, 81–100.Google Scholar
Kyle, K., Crossley, S. A., & McNamara, D. S. (2016). Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing. Language Testing, 33 (3), 319–340. DOI logoGoogle Scholar
Lee, J. (2020). Task closure and task complexity effects on L2 written performance. Journal of Second Language Writing, 50 1, 100777. DOI logoGoogle Scholar
Long, M. H., & Norris, J. M. (2000). Task-based language teaching and assessment. In M. Byram (Ed.), Encyclopedia of language teaching (pp. 597–603). Routledge.Google Scholar
McNamara, T. (2000). Language testing. Oxford University Press.Google Scholar
Norris, J. M. (2001). Identifying rating criteria for task-based EAP assessment. In T. D. Hudson & J. D. Brown (Eds.), A focus on language test development: Expanding the language proficiency construct across a variety of tests (pp. 163–204). University of Hawai‘i Press.Google Scholar
(2010, September). Understanding instructed SLA: Constructs, contexts, and consequences. Plenary address delivered at the annual conference of the European Second Language Association (EUROSLA), Reggio Emilia, Italy.
(2015). Statistical significance testing in second language research: Basic problems and some solutions. In J. M. Norris, S. Ross & R. Schoonen (Eds.), Improving and extending quantitative reasoning in second language research (pp. 95–124). Wiley-Blackwell.Google Scholar
(2016). Current uses for task-based language assessment. Annual Review of Applied Linguistics, 36 1, 230–244. DOI logoGoogle Scholar
Norris, J. M., & East, M. (2021). Task-based language assessment. In M. J. Ahmadian & M. Long (Eds.), The Cambridge Handbook of Task-Based Language Teaching. Cambridge University Press. DOI logoGoogle Scholar
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in Instructed SLA: The case of complexity. Applied Linguistics, 30 (4), 555–578. DOI logoGoogle Scholar
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based L2 performance assessment. Language Testing, 19 (4), 395–418. DOI logoGoogle Scholar
Pallotti, G. (2009). CAF: Defining, refining and differentiating constructs. Applied Linguistics, 30 (4), 590–601. DOI logoGoogle Scholar
Révész, A. (2014). Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics, 35 1, 87–92. DOI logoGoogle Scholar
Révész, A., Ekiert, M., & Torgersen, E. N. (2016a). The effects of complexity, accuracy, and fluency on communicative adequacy in oral task performance. Applied Linguistics, 37 (6), 828–848.Google Scholar
Révész, A., Michel, M., & Gilabert, R. (2016b). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgments: A validation study. Studies in Second Language Acquisition, 38 (4), 703–737. DOI logoGoogle Scholar
Robinson, P. (2001a). Task complexity, cognitive resources, and syllabus design: a triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 287–318). Cambridge University Press. DOI logoGoogle Scholar
(2001b). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22 1, 27–57. DOI logoGoogle Scholar
Robinson, P., & Ross, S. (1996). The development of task-based assessment in English for academic purposes programs. Applied Linguistics, 17 (4), 455–476. DOI logoGoogle Scholar
Sasayama, S. (2015). Validating the assumed relationship between task design, cognitive complexity, and second language task performance. Unpublished doctoral dissertation, Georgetown University.
(2016). Is a ‘complex’ task really complex? Validating the assumption of cognitive task complexity. The Modern Language Journal, 100 (1), 231–254. DOI logoGoogle Scholar
Sasayama, S., & Norris, J. M. (2019). Unravelling cognitive task complexity: Learning from learners’ perspectives on task characteristics and L2 performance. In Z. Wen & M. J. Ahmadian (Eds.), Researching L2 task performance and pedagogy: In honour of Peter Skehan (pp. 95–132). John Benjamins. DOI logoGoogle Scholar
Sasayama, S., Garcia Gomez, P., & Norris, J. M. (2021). Designing efficient L2 writing assessment tasks for low-proficiency learners of English. TOEFL Research Report. Princeton, NJ: Educational Testing Service. DOI logoGoogle Scholar
Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press.Google Scholar
(2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30 1, 510–532. DOI logoGoogle Scholar
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 87 1, 245–251. DOI logoGoogle Scholar
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction, 4 1, 295–312. DOI logoGoogle Scholar
Treffers-Daller, J., Parslow, P., & Williams, S. (2018). Back to basics: How measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics, 39 (3), 302–327.Google Scholar
Van Gorp, K., & Deygers, B. (2014). Task-based language assessment. In A. Kunnan (Ed.), The companion to language assessment, 2 1 (pp. 578–593). John Wiley & Sons.Google Scholar
Vasylets, O., Gilabert, R., & Manchon, R. M. (2020). Task modality, communicative adequacy and CAF measures: The moderating role of task complexity. In Manchon, R. M. (Ed.), Writing and language learning: Advancing research agendas (pp. 183–206). John Benjamins. DOI logoGoogle Scholar
Xi, X., & Norris, J. M. (Eds.). (2021). Assessing academic English for higher education admissions. Routledge. DOI logoGoogle Scholar
Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics, 31 (2), 236–259. DOI logoGoogle Scholar
Cited by (1)

Cited by one other publication

Norris, John M., Shoko Sasayama & Michelle Kim
2023. Simulating Real‐World Context in an Email Writing Task: Implications for Task‐Based Language Assessment. ETS Research Report Series 2023:1 ► pp. 1 ff. DOI logo

This list is based on CrossRef data as of 7 november 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.