Article published In:
TASKVol. 2:2 (2022) ► pp.184–217
Designing speaking tasks for different assessment goals
The complex relationship between cognitive task complexity, language performance, and task accomplishment
This study explored the role of cognitive task complexity in designing speaking assessment tasks. A sample of 120 English learners at different proficiency levels engaged in four narrative tasks with distinct levels/types of cognitive complexity. Performances were assessed using linguistic measures and a task accomplishment rubric. Findings revealed that one of the middle-complexity tasks with moderate intrinsic, moderate germane, and low extraneous complexity elicited the best linguistic performances overall, while the lowest complexity task elicited the highest task accomplishment ratings. Focusing on distinguishing among learner abilities, however, the highest complexity task demonstrated the best discrimination. Cognitive task complexity was also found to moderate the relationship between task accomplishment ratings and linguistic measures. Implications for research into task design, learner performance, and assessment purpose are considered.
Article outline
- Introduction
- Task-Based Language Assessment, task design, and cognitive task complexity
- Task performance: Linguistic indices or task accomplishment?
- Task design for different purposes
- The current study
- Methods
- Participants
- Materials
- Narrative tasks
- Cloze test
- Procedures
- Data scoring, coding, and analysis
- Task accomplishment ratings
- Linguistic indices
- Syntactic complexity
- Accuracy
- Lexis
- Fluency
- Inter-coder reliability
- Cloze test
- Statistical analysis
- Results
- Proficiency test scores
- Learner task performance
- Correlations among performance measures and task accomplishment
- Discrimination
- Discussion
- Role of task design in maximizing task performance
- Role of task design in discriminating among proficiency levels
- Role of task design in understanding the relationship among performance measures
- Limitations
- Conclusion
- Acknowledgements
- Note
-
References
References (48)
References
Brown, J. D. (1980). Relative merits of four methods for scoring cloze tests. The Modern Language Journal,
64
1, 311–317. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, J. D. (2005). Testing in language programs: a comprehensive guide to English language assessment. McGraw-Hill College.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, J. D., & Grüter, T. (2020). The same cloze for all occasions? Using the Brown (1980) cloze test for measuring proficiency in SLA research. International Review of Applied Linguistics. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, J. D., & Hudson, T. (1998). The alternatives in language assessment. TESOL quarterly,
32
(4), 653–675. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Brown, J. D., Hudson, T., Norris, J. M., & Bonk, W. (2002). Investigating task-based second language performance assessment. University of Hawai‘i Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
De Jong, N. H. (2018). Fluency in second language testing: Insights from different disciplines. Language Assessment Quarterly,
15
(3), 237–254. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
De Jong, N. H., Steinel, M., Florijn, A. F., Schoonen, R., & Hulstijn, J. H. (2012). The effect of task complexity on functional adequacy, fluency and lexical diversity in speaking performances of native and non-native speakers. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA (pp. 121–142). John Benjamins. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Doughty, C., & Pica, T. (1986). “Information gap” tasks: Do they facilitate second language acquisition? TESOL Quarterly,
20
(2), 305–325. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Educational Testing Service. (2019). TOEFL iBT® test speaking rubrics. [URL]
Freed, B. F., Segalowitz, N., & Dewey, D. P. (2004). Learning context and its effects on second language acquisition. Studies in Second Language Acquisition,
26
1, 275–301.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Housen, A., De Clercq, B., Kuiken, F., & Vedder, I. (2019). Multiple approaches to complexity in second language research. Second language research,
35
(1), 3–21. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Hunt, K. W. (1970). Syntactic maturity in school children and adults. Monographs of the Society for Research in Child Development,
35
1, 1–67. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Koizumi, R. (2012). Relationships between text length and lexical diversity measures: Can we use short texts of less than 100 tokens? Vocabulary Learning and Instruction,
1
(1), 60–69. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kuiken, F., & Vedder, I. (2017). Functional adequacy in L2 writing: Towards a new rating scale. Language Testing,
34
(3), 321–336. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kuiken, F., Vedder, I., & Gilabert, R. (2010). Communicative adequacy and linguistic complexity in L2 writing. Communicative proficiency and linguistic development: Intersections between SLA and language testing research, EUROSLA Monograph Series
1
1, 81–100.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Kyle, K., Crossley, S. A., & McNamara, D. S. (2016). Construct validity in TOEFL iBT speaking tasks: Insights from natural language processing. Language Testing,
33
(3), 319–340. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Lee, J. (2020). Task closure and task complexity effects on L2 written performance. Journal of Second Language Writing,
50
1, 100777. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Long, M. H., & Norris, J. M. (2000). Task-based language teaching and assessment. In M. Byram (Ed.), Encyclopedia of language teaching (pp. 597–603). Routledge.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
McNamara, T. (2000). Language testing. Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M. (2001). Identifying rating criteria for task-based EAP assessment. In T. D. Hudson & J. D. Brown (Eds.), A focus on language test development: Expanding the language proficiency construct across a variety of tests (pp. 163–204). University of Hawai‘i Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M. (2010, September). Understanding instructed SLA: Constructs, contexts, and consequences. Plenary address delivered at the annual conference of the European Second Language Association (EUROSLA), Reggio Emilia, Italy.
Norris, J. M. (2015). Statistical significance testing in second language research: Basic problems and some solutions. In J. M. Norris, S. Ross & R. Schoonen (Eds.), Improving and extending quantitative reasoning in second language research (pp. 95–124). Wiley-Blackwell.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M. (2016). Current uses for task-based language assessment. Annual Review of Applied Linguistics,
36
1, 230–244. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M., & East, M. (2021). Task-based language assessment. In M. J. Ahmadian & M. Long (Eds.), The Cambridge Handbook of Task-Based Language Teaching. Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in Instructed SLA: The case of complexity. Applied Linguistics,
30
(4), 555–578. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Norris, J. M., Brown, J. D., Hudson, T. D., & Bonk, W. (2002). Examinee abilities and task difficulty in task-based L2 performance assessment. Language Testing,
19
(4), 395–418. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Pallotti, G. (2009). CAF: Defining, refining and differentiating constructs. Applied Linguistics,
30
(4), 590–601. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Révész, A. (2014). Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics,
35
1, 87–92. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Révész, A., Ekiert, M., & Torgersen, E. N. (2016a). The effects of complexity, accuracy, and fluency on communicative adequacy in oral task performance. Applied Linguistics,
37
(6), 828–848.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Révész, A., Michel, M., & Gilabert, R. (2016b). Measuring cognitive task demands using dual-task methodology, subjective self-ratings, and expert judgments: A validation study. Studies in Second Language Acquisition,
38
(4), 703–737. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Robinson, P. (2001a). Task complexity, cognitive resources, and syllabus design: a triadic framework for examining task influences on SLA. In P. Robinson (Ed.), Cognition and second language instruction (pp. 287–318). Cambridge University Press. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Robinson, P. (2001b). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics,
22
1, 27–57. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Robinson, P., & Ross, S. (1996). The development of task-based assessment in English for academic purposes programs. Applied Linguistics,
17
(4), 455–476. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sasayama, S. (2015). Validating the assumed relationship between task design, cognitive complexity, and second language task performance. Unpublished doctoral dissertation, Georgetown University.
Sasayama, S. (2016). Is a ‘complex’ task really complex? Validating the assumption of cognitive task complexity. The Modern Language Journal,
100
(1), 231–254. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sasayama, S., Garcia Gomez, P., & Norris, J. M. (2021). Designing efficient L2 writing assessment tasks for low-proficiency learners of English. TOEFL Research Report. Princeton, NJ: Educational Testing Service. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Skehan, P. (1998). A cognitive approach to language learning. Oxford University Press.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Skehan, P. (2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics,
30
1, 510–532. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin,
87
1, 245–251. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning and Instruction,
4
1, 295–312. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Treffers-Daller, J., Parslow, P., & Williams, S. (2018). Back to basics: How measures of lexical diversity can help discriminate between CEFR levels. Applied Linguistics,
39
(3), 302–327.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Van Gorp, K., & Deygers, B. (2014). Task-based language assessment. In A. Kunnan (Ed.), The companion to language assessment,
2
1 (pp. 578–593). John Wiley & Sons.![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Xi, X., & Norris, J. M. (Eds.). (2021). Assessing academic English for higher education admissions. Routledge. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Yu, G. (2010). Lexical diversity in writing and speaking task performances. Applied Linguistics,
31
(2), 236–259. ![DOI logo](https://benjamins.com/logos/doi-logo.svg)
![Google Scholar](https://benjamins.com/logos/google-scholar.svg)
Cited by (1)
Cited by one other publication
Norris, John M., Shoko Sasayama & Michelle Kim
2023.
Simulating Real‐World Context in an Email Writing Task: Implications for Task‐Based Language Assessment.
ETS Research Report Series 2023:1
► pp. 1 ff.
![DOI logo](//benjamins.com/logos/doi-logo.svg)
This list is based on CrossRef data as of 5 july 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers.
Any errors therein should be reported to them.