Cover not available
Article published In:
Journal of Second Language Studies
Vol. 6:1 (2023) ► pp.95–118

Pitfalls of production data analysis for investigating L2 cognitive mechanism

An ontological realism perspective

 | Chuo University
 | Matsuyama University
 | Kansai University
This article addresses the pitfalls of performance analysis in investigating cognitive processing during second language (L2) learning. The problems that we discuss in this paper are twofold: (1) Assuming psychological variables to be ontological entities without meeting the criteria for ontological reality and (2) Inappropriateness of assessing abilities based on learner’s speaking or writing performance to investigate cognitive processes. By addressing these problems, we argue that some latent variables postulated by observing L2 performance do not exist in reality and emphasize the difficulty of interpreting cognitive mechanisms through performance analysis. We also enumerate some problems that arise from the epistemological perspectives of previous research practice (e.g., the bifurcation of contradictory hypotheses and their indeterminacy). Finally, two alternative approaches treating L2 performance are proposed. The implications of this line of discussion for future research are also discussed.
Keywords: complexity, accuracy, fluency, ontological realism, performance analysis
Article outline
  • 1.Introduction
  • 2.Ontological perspective on performance analysis
  • 3.Skepticism of CAF as ontological entities
  • 4.L2 performance as superficial phenomena that form a complex structure
  • 5.Implications for future research
  • Notes
  • References
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.

For any use beyond this license, please contact the publisher at [email protected].

Published online: 22 March 2022
DOI logo
https://doi.org/10.1075/jsls.21013.fuk
References (83)
References
Biber, D., Gray, B., & Poonpon, K. (2011). Should we use characteristics of conversation to measure grammatical complexity in L2 writing development? TESOL Quarterly, 31 (1), 5–35. DOI logoGoogle Scholar
Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: A structural equation perspective. Psychological Bulletin, 110 (2), 305–314. DOI logoGoogle Scholar
Borsboom, D., & Cramer, A. O. J. (2013). Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology, 9 1, 91–121. DOI logoGoogle Scholar
Borsboom, D., Cramer, A. O. J., Kievit, R. A., Zand Scholten, A., & Franić, S. (2009). The end of construct validity. In R. W. Lissitz (Ed.), The concept of validity (pp. 135–170). Charlotte, NC: Information Age Publishers.Google Scholar
Borsboom, D., Mellenbergh, G. J., & van Heerden, J. (2004). The concept of validity. Psychological Review, 111 (4), 1061–1071. DOI logoGoogle Scholar
Branigan, H. P., & Pickering, M. J. (2017). An experimental approach to linguistic representation. Behavioral and Brain Sciences, 40 1, e282. DOI logoGoogle Scholar
Bulté, B., & Housen, A. (2018). Syntactic complexity in L2 writing: Individual pathways and emerging group trends. International Journal of Applied Linguistics, 28 (1), 147–164. DOI logoGoogle Scholar
Cartwright, N. (1983). How the laws of physics lie. Oxford: Clarendon Press. DOI logoGoogle Scholar
Churchland, P. (1995). The engine of reason, the seat of the soul: A philosophical journey into the brain. Cambridge, MA: MIT Press. DOI logoGoogle Scholar
Coltman, T., Devinney, T. M., Midgley, D. F., & Venaik, S. (2008). Formative versus reflective measurement models: Two applications of formative measurement. Journal of Business Research, 61 (12), 1250–1262. DOI logoGoogle Scholar
Declerck, M., & Kormos, J. (2012). The effect of dual task demands and proficiency on second language speech production. Bilingualism: Language and Cognition, 15 (4), 782–796. DOI logoGoogle Scholar
Edwards, J. R., & Bagozzi, R. P. (2000). On the nature and direction of relationships between constructs and measures. Psychological Methods, 5 (2), 155–174. DOI logoGoogle Scholar
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press.Google Scholar
Fornell, C., & Bookstein, F. L. (1982). Two structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of Marketing Research, 19 (4), 440–452. DOI logoGoogle Scholar
Foster, P., & Tavakoli, P. (2009). Native speakers and task performance: Comparing effects on complexity, fluency, and lexical diversity. Language learning, 59 (4), 866–896. DOI logoGoogle Scholar
Fukuta, J., Goto, A., Kawaguchi, Y., Kurita, A., & Murota, D. (2018). Syntactically–driven algorithmic processing of PP-attachment ambiguity in a second language. International Review of Applied Linguistics in Language Teaching, 56 (3), 253–277. DOI logoGoogle Scholar
Fukuta, J., Goto, A., Kawaguchi, Y., Murota, D., & Kurita, A. (2015). Japanese EFL learners’ implicit knowledge and algorithmic processing of dative alternation: Perspective from syntactic priming in reading comprehension. ARELE: Annual Review of English Language Education in Japan, 26 1, 221–236. DOI logoGoogle Scholar
Fukuta, J., & Yamashita, J. (2015). Effects of cognitive demands on attention orientation in L2 oral production. System, 53 1, 1–12. DOI logoGoogle Scholar
García Mayo, M. D. P. (Ed.). (2006). Investigating tasks in formal language learning. Multilingual Matters. DOI logoGoogle Scholar
Gilabert, R. (2007a). The simultaneous manipulation of task complexity along planning time and (+ / - here-and-now): Effects on L2 oral production. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 44–68). Clevedon, UK: Multilingual Matters.Google Scholar
(2007b). Effects of manipulating task complexity on self-repairs during L2 oral production, International Review of Applied Linguistics in Language Teaching, 45 (3), 215–240. DOI logoGoogle Scholar
Gilabert, R., Barón, J., & Llanes, À. (2009). Manipulating cognitive complexity across task types and its impact on learners’ interaction during oral performance. International Review of Applied Linguistics in Language Teaching, 47 (3), 367–395. DOI logoGoogle Scholar
Hacking, I. (1983). Representing and intervening: Introductory topics in the philosophy of science. Cambridge: Cambridge University Press. DOI logoGoogle Scholar
Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E. (2004). Is Syntax Separate or Shared Between Languages?: Cross-Linguistic Syntactic Priming in Spanish-English Bilinguals. Psychological Science, 15 (6), 409–414. DOI logoGoogle Scholar
Hiver, P., & Al-hoorie, A. H. (2016). A dynamic ensemble for second language research: Putting complexity theory into practice. The Modern Language Journal, 100 (4), 741–756. DOI logoGoogle Scholar
Housen, A., & Kuiken, F. (2009). Complexity, accuracy, and fluency in second language acquisition. Applied Linguistics, 30 (4), 461–473. DOI logoGoogle Scholar
Housen, A., Kuiken, F., & Vedder, I. (Eds.). (2012). Dimensions of L2 performance and proficiency: Complexity, accuracy and fluency in SLA. Philadelphia: John Benjamins. DOI logoGoogle Scholar
Inhelder, B., & Piaget, J. (1958). An essay on the construction of formal operational structures. the growth of logical thinking: From childhood to adolescence. A. Parsons & S. Milgram (Trans.). New York: Basic Books. DOI logoGoogle Scholar
Ishikawa, T. (2007). The effect of manipulating task complexity along the [+ / - here-and-now] dimension on L2 written narrative discourse. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 136–156). Clevedon, UK: Multilingual Matters.Google Scholar
(2008). The effect of task demands of intentional reasoning on L2 speech performance. The Journal of Asia TEFL, 5 1, 29–63.Google Scholar
Izumi, E., Uchimoto, K., & Isahara, H. (2004). The NICT-JLE Corpus: Exploiting the language learners’ speech database for research and education. International Journal of the Computer, the Internet and Management, 12 (2), 119–125.Google Scholar
Jackson, D. O., & Suethanapornkul, S. (2013). The Cognition Hypothesis: A synthesis and meta-analysis of research on second language task complexity. Language Learning, 63 (2), 330–367. DOI logoGoogle Scholar
Jolliffe, I. T. (2002). Principal component analysis (2nd ed). New York: Springer.Google Scholar
Kaan, E., & Chun, E. (2018). Priming and adaptation in native speakers and second-language learners. Bilingualism: Language and Cognition, 21 (2), 228–242. DOI logoGoogle Scholar
Kaan, E., Futch, C., Fuertes, R. F., Mujcinovic, S., & Fuente, E. Á. D. L. (2019). Adaptation to syntactic structures in native and nonnative sentence comprehension. Applied Psycholinguistics, 40 (1), 3–27. DOI logoGoogle Scholar
Kim, Y. (2009). The effects of task complexity on learner-learner interaction. System, 37 1, 254–268. DOI logoGoogle Scholar
Koizumi, R., & In’nami, Y. (2014). Modeling complexity, accuracy, and fluency of Japanese Learners of English: A structural equation modeling approach. JALT Journal, 36 (1), 25–46. DOI logoGoogle Scholar
Kormos, J., & Trebits, A. (2012). The role of task complexity, modality, and aptitude in narrative task performance. Language Learning, 62 1, 439–472. DOI logoGoogle Scholar
Kuiken, F., & Vedder, I. (2007). Task complexity and measures of linguistic performance in L2 writing. International Review of Applied Linguistics in Language Teaching, 45 1, 261–284. DOI logoGoogle Scholar
(2008). Cognitive task complexity and written output in Italian and French as a foreign language. Journal of Second Language Writing, 17 1, 48–60. DOI logoGoogle Scholar
(2012). Syntactic complexity, lexical variation and accuracy as a function of task complexity and proficiency level in L2 writing and speaking. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions of L2 performance and proficiency: Complexity, accuracy, and fluency in SLA (pp. 143–170). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Lambert, C., & Kormos, J. (2014). Complexity, accuracy, and fluency in task-based L2 research: Toward more developmentally based measures of second language acquisition. Applied Linguistics, 35 (5), 607–614. DOI logoGoogle Scholar
Lambert, C., Kormos, J., & Minn, D. (2017). Task repetition and second language speech processing. Studies in Second Language Acquisition, 39 (1), 167–196. DOI logoGoogle Scholar
Larsen-Freeman, D. (2006). The emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of English. Applied Linguistics, 27 (4), 590–619. DOI logoGoogle Scholar
Laudan, L. (1981). A confutation of convergent realism. Philosophy of Science, 48 (1), 19–49. DOI logoGoogle Scholar
Markus, K. A., & Borsboom, D. (2013). Frontiers of test validity theory: Measurement, causation, and meaning. Abingdon, UK: Routledge. DOI logoGoogle Scholar
Mehnert, U. (1998). The effects of different lengths of time for planning on second language performance. Studies in Second Language Acquisition, 20 (1), 83–108. DOI logoGoogle Scholar
Nishimura, Y., Kawaguchi, Y., & Abe, D. (2018). The relationship between complexity and fluency in L2 writing: An approach using network analysis. Language Education & Technology, (55), 171–198.Google Scholar
Nishimura, Y., Tamura, Y., & Fukuta, J. (2018). Network structures in L2 oral performance: A learner corpus study. ARELE: Annual Review of English Language Education in Japan, 29 1, 113–128. DOI logoGoogle Scholar
Norris, J. M., & Ortega, L. (2009). Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics, 30 (4), 555–578. DOI logoGoogle Scholar
Ong, J., & Zhang, L. J. (2010). Effects of task complexity on the fluency and lexical complexity in EFL students’ argumentative writing. Journal of Second Language Writing, 19 (4), 218–233. DOI logoGoogle Scholar
Pallotti, G. (2009). CAF: Defining, refining, and differentiating constructs. Applied Linguistics, 30 (4), 590–601. DOI logoGoogle Scholar
(2015). A simple view of linguistic complexity. Second Language Research, 31 (1), 117–134. DOI logoGoogle Scholar
Pickering, M. J., & Branigan, H. P. (1999). Syntactic priming in language production. Trends in cognitive sciences, 3 (4), 136–141. DOI logoGoogle Scholar
Plonsky, L., & Kim, Y. (2016). Task-Based Learner Production: A Substantive and Methodological Review. Annual Review of Applied Linguistics, 361, 73–97. DOI logoGoogle Scholar
Polat, B., & Kim, Y. (2014). Dynamics of complexity and accuracy: A longitudinal case study of advanced untutored development. Applied Linguistics, 35 (2), 184–207. DOI logoGoogle Scholar
Révész, A. (2014). Towards a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and processes. Applied Linguistics, 35 (1), 87–92. DOI logoGoogle Scholar
Reynolds, C. W. (1987). Flocks, herds and schools: A distributed behavioral model. SIGGRAPH ’87: Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing Machinery . pp. 25–34. DOI logo
Robinson, P. (1995). Task complexity and second language narrative discourse. Language Learning, 45 (1), 99–140. DOI logoGoogle Scholar
(2001). Task complexity, task difficulty, and task production: Exploring interactions in a componential framework. Applied Linguistics, 22 (1), 27–57. DOI logoGoogle Scholar
(2003). Attention and memory during SLA. In C. J. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 631–678). Malden, MA: Blackwell. DOI logoGoogle Scholar
(2007). Criteria for classifying and sequencing pedagogic tasks. In M. P. García Mayo (Ed.), Investigating tasks in formal language learning (pp. 7–26). Clevedon, UK: Multilingual Matters.Google Scholar
Robinson, P. J. (2011). Second language task complexity, the cognition hypothesis, language learning, and performance. In P. J. Robinson (Ed.), Second language task complexity: Researching the cognition hypothesis of language learning and performance (pp. 2–38). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Romeijn, J. W. (2008). The all-too-flexible abductive method: ATOM’s normative status. Journal of Clinical Psychology, 64 (9), 1023–1036. DOI logoGoogle Scholar
Schmittmann, V. D., Cramer, A. O. J., Waldorp, L. J., Epskamp, S., Kievit, R. A., & Borsboom, D. (2013). Deconstructing the construct: A network perspective on psychological phenomena. New Ideas in Psychology, 31 (1), 43–53. DOI logoGoogle Scholar
Sheppard, C. (2004). The measurement of second language production: The validity of fluency, accuracy, and complexity. ICU Language Research Bulletin, 19 1, 139–156.Google Scholar
Skehan, P. (1996). A framework for the implementation of task-based instruction. Applied Linguistics, 17 (1), 38–62. DOI logoGoogle Scholar
(1998). A cognitive approach to language learning. Oxford: Oxford University Press.Google Scholar
(2009). Modelling second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied Linguistics, 30 (4), 510–532. DOI logoGoogle Scholar
(2014). The context researching a processing perspective on task performance. In P. Skehan (Ed.), Processing perspectives on task performance (pp. 1–26). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Skehan, P., & Foster, P. (1997). Task type and task processing conditions as influences on foreign language performance. Language Teaching Research, 1 (3), 185–211. DOI logoGoogle Scholar
(1999). The influence of task structure and processing conditions on narrative retellings. Language learning, 49(1), 93–120. DOI logoGoogle Scholar
(2005). Strategic and on-line planning: The influence of surprise information and task time on second language performance. In R. Ellis (Ed.), Planning and task performance in a second language (pp. 193–216). Amsterdam: John Benjamins. DOI logoGoogle Scholar
Spoelman, M., & Verspoor, M. (2010). Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics, 31 (4), 532–553. DOI logoGoogle Scholar
Tavakoli, P., & Foster, P. (2011). Task design and second language performance: The effect of narrative type on learner output. Language Learning, 61 1, 37–72. DOI logoGoogle Scholar
Van Fraassen, B. (1980). The scientific image. Oxford: Oxford University Press. DOI logoGoogle Scholar
Vercellotti, M. L. (2017). The development of complexity, accuracy, and fluency in second language performance: A longitudinal study. Applied Linguistics, 38 (1), 90–111. DOI logoGoogle Scholar
Wolfe-Quintero, K., Inagaki, S., & Kim, H.-Y. (1998). Second language development in writing: Measures of fluency, accuracy, and complexity. Honolulu University of Hawai’i, Second Language Teaching and Curriculum Center.Google Scholar
Wright, C. (2020). Effects of task type on L2 Mandarin fluency development. Journal of Second Language Studies, 3 (2), 157–179. DOI logoGoogle Scholar
Xu, T. S., Zhang, L. J., & Gaffney, J. S. (2021). Examining the relative effectiveness of task complexity and cognitive demands on students’ writing in a second language. Studies in Second Language Acquisition. Advance online publication. DOI logoGoogle Scholar
Yu, H., & Lowie, W. (2019). Dynamic paths of complexity and accuracy in second language speech: A longitudinal case study of Chinese learners. Applied Linguistics, 41 (6), 855–877. DOI logoGoogle Scholar
Yuan, F., & Ellis, R. (2003). The effects of pre-task planning and on-line planning on fluency, complexity and accuracy in L2 monologic oral production. Applied Linguistics, 24 (1), 1–27. DOI logoGoogle Scholar
Cited by (2)

Cited by two other publications

Alamer, Abdullah, Florian Schuberth & Jörg Henseler
2024. When and how to use confirmatory composite analysis (CCA) in second language research. Studies in Second Language Acquisition 46:2 ► pp. 597 ff. DOI logo
Wagner, Thomas
2023. Implications for the Classroom. In English Interlanguage Morphology, ► pp. 113 ff. DOI logo

This list is based on CrossRef data as of 22 november 2024. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.