Article published In:
Journal of Second Language StudiesVol. 6:1 (2023) ► pp.95–118
Pitfalls of production data analysis for investigating L2 cognitive mechanism
An ontological realism perspective
This article addresses the pitfalls of performance analysis in investigating cognitive processing during second language (L2) learning. The problems that we discuss in this paper are twofold: (1) Assuming psychological variables to be ontological entities without meeting the criteria for ontological reality and (2) Inappropriateness of assessing abilities based on learner’s speaking or writing performance to investigate cognitive processes. By addressing these problems, we argue that some latent variables postulated by observing L2 performance do not exist in reality and emphasize the difficulty of interpreting cognitive mechanisms through performance analysis. We also enumerate some problems that arise from the epistemological perspectives of previous research practice (e.g., the bifurcation of contradictory hypotheses and their indeterminacy). Finally, two alternative approaches treating L2 performance are proposed. The implications of this line of discussion for future research are also discussed.
Keywords: complexity, accuracy, fluency, ontological realism, performance analysis
Article outline
- 1.Introduction
- 2.Ontological perspective on performance analysis
- 3.Skepticism of CAF as ontological entities
- 4.L2 performance as superficial phenomena that form a complex structure
- 5.Implications for future research
- Notes
-
References
Available under the Creative Commons Attribution-NonCommercial (CC BY-NC) 4.0 license.
For any use beyond this license, please contact the publisher at [email protected].
References
Biber, D., Gray, B., & Poonpon, K.
(2011) Should
we use characteristics of conversation to measure grammatical complexity in L2 writing
development? TESOL
Quarterly,
31
(1), 5–35. 
Bollen, K., & Lennox, R.
(1991) Conventional
wisdom on measurement: A structural equation perspective. Psychological
Bulletin,
110
(2), 305–314.
Borsboom, D., & Cramer, A. O. J.
(2013) Network
analysis: An integrative approach to the structure of psychopathology. Annual Review of
Clinical
Psychology,
9
1, 91–121.
Borsboom, D., Cramer, A. O. J., Kievit, R. A., Zand Scholten, A., & Franić, S.
(2009) The
end of construct validity. In R. W. Lissitz (Ed.), The
concept of
validity (pp. 135–170). Charlotte, NC: Information Age Publishers.
Borsboom, D., Mellenbergh, G. J., & van Heerden, J.
(2004) The
concept of validity. Psychological
Review,
111
(4), 1061–1071.
Branigan, H. P., & Pickering, M. J.
(2017) An
experimental approach to linguistic representation. Behavioral and Brain
Sciences,
40
1, e282.
Bulté, B., & Housen, A.
(2018) Syntactic
complexity in L2 writing: Individual pathways and emerging group trends. International Journal
of Applied
Linguistics,
28
(1), 147–164.
Cartwright, N.
(1983) How
the laws of physics lie. Oxford: Clarendon Press.
Churchland, P.
(1995) The
engine of reason, the seat of the soul: A philosophical journey into the brain. Cambridge, MA: MIT Press.
Coltman, T., Devinney, T. M., Midgley, D. F., & Venaik, S.
(2008) Formative
versus reflective measurement models: Two applications of formative measurement. Journal of
Business
Research,
61
(12), 1250–1262.
Declerck, M., & Kormos, J.
(2012) The
effect of dual task demands and proficiency on second language speech production. Bilingualism:
Language and
Cognition,
15
(4), 782–796.
Edwards, J. R., & Bagozzi, R. P.
(2000) On
the nature and direction of relationships between constructs and measures. Psychological
Methods,
5
(2), 155–174.
Ellis, R.
(2003) Task-based
language learning and teaching. Oxford: Oxford University Press.
Fornell, C., & Bookstein, F. L.
(1982) Two
structural equation models: LISREL and PLS applied to consumer exit-voice theory. Journal of
Marketing
Research,
19
(4), 440–452.
Foster, P., & Tavakoli, P.
(2009) Native
speakers and task performance: Comparing effects on complexity, fluency, and lexical
diversity. Language
learning,
59
(4), 866–896.
Fukuta, J., Goto, A., Kawaguchi, Y., Kurita, A., & Murota, D.
(2018) Syntactically–driven
algorithmic processing of PP-attachment ambiguity in a second language. International Review of
Applied Linguistics in Language
Teaching,
56
(3), 253–277.
Fukuta, J., Goto, A., Kawaguchi, Y., Murota, D., & Kurita, A.
(2015) Japanese
EFL learners’ implicit knowledge and algorithmic processing of dative alternation: Perspective from syntactic priming in
reading comprehension. ARELE: Annual Review of English Language Education in
Japan,
26
1, 221–236.
Fukuta, J., & Yamashita, J.
(2015) Effects
of cognitive demands on attention orientation in L2 oral
production. System,
53
1, 1–12.
García Mayo, M. D. P.
(Ed.) (2006) Investigating
tasks in formal language learning. Multilingual Matters.
Gilabert, R.
(2007a) The
simultaneous manipulation of task complexity along planning time and (+ / - here-and-now): Effects on L2 oral
production. In M. P. García Mayo (Ed.), Investigating
tasks in formal language
learning (pp. 44–68). Clevedon, UK: Multilingual Matters.
(2007b) Effects
of manipulating task complexity on self-repairs during L2 oral production, International Review
of Applied Linguistics in Language
Teaching,
45
(3), 215–240.
Gilabert, R., Barón, J., & Llanes, À.
(2009) Manipulating
cognitive complexity across task types and its impact on learners’ interaction during oral
performance. International Review of Applied Linguistics in Language
Teaching,
47
(3), 367–395.
Hacking, I.
(1983) Representing
and intervening: Introductory topics in the philosophy of
science. Cambridge: Cambridge University Press.
Hartsuiker, R. J., Pickering, M. J., & Veltkamp, E.
(2004) Is
Syntax Separate or Shared Between Languages?: Cross-Linguistic Syntactic Priming in Spanish-English
Bilinguals. Psychological
Science,
15
(6), 409–414.
Hiver, P., & Al-hoorie, A. H.
(2016) A
dynamic ensemble for second language research: Putting complexity theory into practice. The
Modern Language
Journal,
100
(4), 741–756.
Housen, A., & Kuiken, F.
(2009) Complexity,
accuracy, and fluency in second language acquisition. Applied
Linguistics,
30
(4), 461–473.
Housen, A., Kuiken, F., & Vedder, I.
(Eds.) (2012) Dimensions
of L2 performance and proficiency: Complexity, accuracy and fluency in
SLA. Philadelphia: John Benjamins.
Inhelder, B., & Piaget, J.
(1958) An
essay on the construction of formal operational structures. the growth of logical thinking: From childhood to
adolescence. A. Parsons & S. Milgram (Trans.). New York: Basic Books.
Ishikawa, T.
(2007) The
effect of manipulating task complexity along the [+ / - here-and-now] dimension on L2 written narrative
discourse. In M. P. García Mayo (Ed.), Investigating
tasks in formal language
learning (pp. 136–156). Clevedon, UK: Multilingual Matters.
(2008) The
effect of task demands of intentional reasoning on L2 speech performance. The Journal of Asia
TEFL,
5
1, 29–63.
Izumi, E., Uchimoto, K., & Isahara, H.
(2004) The
NICT-JLE Corpus: Exploiting the language learners’ speech database for research and
education. International Journal of the Computer, the Internet and
Management,
12
(2), 119–125.
Jackson, D. O., & Suethanapornkul, S.
(2013) The
Cognition Hypothesis: A synthesis and meta-analysis of research on second language task
complexity. Language
Learning,
63
(2), 330–367.
Jolliffe, I. T.
(2002) Principal
component analysis (2nd ed). New York: Springer.
Kaan, E., & Chun, E.
(2018) Priming
and adaptation in native speakers and second-language learners. Bilingualism: Language and
Cognition,
21
(2), 228–242.
Kaan, E., Futch, C., Fuertes, R. F., Mujcinovic, S., & Fuente, E. Á. D. L.
(2019) Adaptation to
syntactic structures in native and nonnative sentence comprehension. Applied
Psycholinguistics,
40
(1), 3–27.
Khezrlou, S.
(2020) The
role of task repetition with direct written corrective feedback in L2 writing complexity, accuracy and
fluency. Journal of Second Language
Studies,
3
(1), 31–54.
Kim, Y.
(2009) The
effects of task complexity on learner-learner
interaction. System,
37
1, 254–268.
Koizumi, R., & In’nami, Y.
(2014) Modeling
complexity, accuracy, and fluency of Japanese Learners of English: A structural equation modeling
approach. JALT
Journal,
36
(1), 25–46.
Kormos, J., & Trebits, A.
(2012) The
role of task complexity, modality, and aptitude in narrative task performance. Language
Learning,
62
1, 439–472.
Kuiken, F., & Vedder, I.
(2007) Task
complexity and measures of linguistic performance in L2 writing. International Review of
Applied Linguistics in Language
Teaching,
45
1, 261–284.
(2008) Cognitive
task complexity and written output in Italian and French as a foreign language. Journal of
Second Language
Writing,
17
1, 48–60.
(2012) Syntactic
complexity, lexical variation and accuracy as a function of task complexity and proficiency level in L2 writing and
speaking. In A. Housen, F. Kuiken & I. Vedder (Eds.), Dimensions
of L2 performance and proficiency: Complexity, accuracy, and fluency in
SLA (pp. 143–170). Amsterdam: John Benjamins.
Lambert, C., & Kormos, J.
(2014) Complexity,
accuracy, and fluency in task-based L2 research: Toward more developmentally based measures of second language
acquisition. Applied
Linguistics,
35
(5), 607–614.
Lambert, C., Kormos, J., & Minn, D.
(2017) Task
repetition and second language speech processing. Studies in Second Language
Acquisition,
39
(1), 167–196.
Larsen-Freeman, D.
(2006) The
emergence of complexity, fluency, and accuracy in the oral and written production of five Chinese learners of
English. Applied
Linguistics,
27
(4), 590–619.
Laudan, L.
(1981) A
confutation of convergent realism. Philosophy of
Science,
48
(1), 19–49.
Markus, K. A., & Borsboom, D.
(2013) Frontiers
of test validity theory: Measurement, causation, and meaning. Abingdon, UK: Routledge.
Mehnert, U.
(1998) The
effects of different lengths of time for planning on second language performance. Studies in
Second Language
Acquisition,
20
(1), 83–108.
Nishimura, Y., Kawaguchi, Y., & Abe, D.
(2018) The
relationship between complexity and fluency in L2 writing: An approach using network
analysis. Language Education &
Technology, (55), 171–198.
Nishimura, Y., Tamura, Y., & Fukuta, J.
(2018) Network
structures in L2 oral performance: A learner corpus study. ARELE: Annual Review of English
Language Education in
Japan,
29
1, 113–128.
Norris, J. M., & Ortega, L.
(2009) Towards
an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied
Linguistics,
30
(4), 555–578.
Ong, J., & Zhang, L. J.
(2010) Effects
of task complexity on the fluency and lexical complexity in EFL students’ argumentative
writing. Journal of Second Language
Writing,
19
(4), 218–233.
Pallotti, G.
(2009) CAF:
Defining, refining, and differentiating constructs. Applied
Linguistics,
30
(4), 590–601.
(2015) A
simple view of linguistic complexity. Second Language
Research,
31
(1), 117–134.
Pickering, M. J., & Branigan, H. P.
(1999) Syntactic
priming in language production. Trends in cognitive
sciences,
3
(4), 136–141.
Plonsky, L., & Kim, Y.
(2016) Task-Based Learner Production: A Substantive and Methodological Review. Annual Review of Applied Linguistics, 361, 73–97.
Polat, B., & Kim, Y.
(2014) Dynamics
of complexity and accuracy: A longitudinal case study of advanced untutored
development. Applied
Linguistics,
35
(2), 184–207.
Révész, A.
(2014) Towards
a fuller assessment of cognitive models of task-based learning: Investigating task-generated cognitive demands and
processes. Applied
Linguistics,
35
(1), 87–92.
Reynolds, C. W.
(1987) Flocks,
herds and schools: A distributed behavioral model. SIGGRAPH ’87: Proceedings of the
14th Annual Conference on Computer Graphics and Interactive Techniques. Association for Computing
Machinery
. pp. 25–34.
Robinson, P.
(1995) Task
complexity and second language narrative discourse. Language
Learning,
45
(1), 99–140.
(2001) Task
complexity, task difficulty, and task production: Exploring interactions in a componential
framework. Applied
Linguistics,
22
(1), 27–57.
(2003) Attention
and memory during SLA. In C. J. Doughty & M. H. Long (Eds.), The
handbook of second language
acquisition (pp. 631–678). Malden, MA: Blackwell.
(2007) Criteria
for classifying and sequencing pedagogic tasks. In M. P. García Mayo (Ed.), Investigating
tasks in formal language
learning (pp. 7–26). Clevedon, UK: Multilingual Matters.
Robinson, P. J.
(2011) Second
language task complexity, the cognition hypothesis, language learning, and
performance. In P. J. Robinson (Ed.), Second
language task complexity: Researching the cognition hypothesis of language learning and
performance (pp. 2–38). Amsterdam: John Benjamins.
Romeijn, J. W.
(2008) The
all-too-flexible abductive method: ATOM’s normative status. Journal of Clinical
Psychology,
64
(9), 1023–1036.
Schmittmann, V. D., Cramer, A. O. J., Waldorp, L. J., Epskamp, S., Kievit, R. A., & Borsboom, D.
(2013) Deconstructing
the construct: A network perspective on psychological phenomena. New Ideas in
Psychology,
31
(1), 43–53.
Sheppard, C.
(2004) The
measurement of second language production: The validity of fluency, accuracy, and
complexity. ICU Language Research
Bulletin,
19
1, 139–156.
Skehan, P.
(1996) A
framework for the implementation of task-based instruction. Applied
Linguistics,
17
(1), 38–62.
(1998) A
cognitive approach to language learning. Oxford: Oxford University Press.
(2009) Modelling
second language performance: Integrating complexity, accuracy, fluency, and lexis. Applied
Linguistics,
30
(4), 510–532.
(2014) The
context researching a processing perspective on task
performance. In P. Skehan (Ed.), Processing
perspectives on task
performance (pp. 1–26). Amsterdam: John Benjamins.
Skehan, P., & Foster, P.
(1997) Task
type and task processing conditions as influences on foreign language performance. Language
Teaching
Research,
1
(3), 185–211.
(1999) The influence of task structure and processing conditions on narrative retellings. Language learning, 49(1), 93–120.
(2005) Strategic
and on-line planning: The influence of surprise information and task time on second language
performance. In R. Ellis (Ed.), Planning
and task performance in a second
language (pp. 193–216). Amsterdam: John Benjamins.
Spoelman, M., & Verspoor, M.
(2010) Dynamic
patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of
Finnish. Applied
Linguistics,
31
(4), 532–553.
Tavakoli, P., & Foster, P.
(2011) Task
design and second language performance: The effect of narrative type on learner
output. Language
Learning,
61
1, 37–72.
Van Fraassen, B.
(1980) The
scientific image. Oxford: Oxford University Press.
Vercellotti, M. L.
(2017) The
development of complexity, accuracy, and fluency in second language performance: A longitudinal
study. Applied
Linguistics,
38
(1), 90–111.
Wolfe-Quintero, K., Inagaki, S., & Kim, H.-Y.
(1998) Second
language development in writing: Measures of fluency, accuracy, and complexity. Honolulu University of Hawai’i, Second Language Teaching and Curriculum Center.
Wright, C.
(2020) Effects
of task type on L2 Mandarin fluency development. Journal of Second Language
Studies,
3
(2), 157–179.
Xu, T. S., Zhang, L. J., & Gaffney, J. S.
(2021) Examining
the relative effectiveness of task complexity and cognitive demands on students’ writing in a second
language. Studies in Second Language Acquisition. Advance online publication.
Yu, H., & Lowie, W.
(2019) Dynamic
paths of complexity and accuracy in second language speech: A longitudinal case study of Chinese
learners. Applied
Linguistics,
41
(6), 855–877.
Yuan, F., & Ellis, R.
(2003) The
effects of pre-task planning and on-line planning on fluency, complexity and accuracy in L2 monologic oral
production. Applied
Linguistics,
24
(1), 1–27.