Cover not available
Part of
Teachability and Learnability across Languages
Edited by Ragnar Arntzen, Gisela Håkansson, Arnstein Hjelde and Jörg-U. Keßler
[Processability Approaches to Language Acquisition Research & Teaching 6] 2019
► pp. 137–159

Chapter 7
Grammatical accuracy and complexity in a speaking proficiency test

| University of Gothenburg
The Common European Framework of Reference has stimulated research on the relation between linguistic competence and communicative competence. Such studies have often used linguistic measures from the Complexity-Accuracy-Fluency (CAF) paradigm to objectively tap into the linguistic competence of language learners; others have used Processability Theory. This study contributes to this research by measuring grammatical accuracy and complexity, in terms of both CAF and level of processability, in learners of second language Swedish taking a speaking proficiency test. The results show that a higher level of processability primarily enables learners to use a more complex, rather than a more accurate, language. Nevertheless, assessors focus primarily on accuracy. This raises questions about the reliability of communicative language testing, and the article argues for a more psycholinguistic and experimental approach to language testing.
Article outline
  • 1.Introduction
  • 2.Background
    • 2.1The Common European Framework of Reference (CEFR)
    • 2.2Complexity, accuracy and language proficiency
    • 2.3Level of processability and language proficiency
  • 3.Method
    • 3.1The test
    • 3.2Transcribing and coding
    • 3.3Statistical analysis
  • 4.Results
    • 4.1Descriptive statistics
    • 4.2Correlations between assessments
    • 4.3Correlations between measures
    • 4.4Correlations between assessments and measures
  • 5.Explaining unexpected cases
    • 5.1Selecting typical and non-typical cases
    • 5.2Comparing typical and non-typical cases
  • 6.Summary and discussion
  • 7.Conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 6 June 2019
DOI logo
https://doi.org/10.1075/palart.6.07age
References (56)
References
Albrechtsen, D., Henriksen, B. & Faerch, C. 1980. Native speaker reactions to learners’ spoken interlanguage 1. Language Learning 30(2): 365–396. DOI logoGoogle Scholar
Axelsson, M. 1994. Noun Phrase Development in Swedish as a Second Language. A Study of Adult Learners Acquiring Definiteness and the Semantics and Morphology of Adjectives. PhD dissertation, Stockholm University.Google Scholar
Bachman, L. F. 1990. Fundamental Considerations in Language Testing. Oxford: OUP.Google Scholar
Bachman, L. F. & Cohen, A. D. 1998. Language testing – SLA interfaces: An update. In Interfaces between Second Language Acquisition and Language Testing Research, L. F. Bachman & A. D. Cohen (eds), 1–31. Cambridge: CUP.Google Scholar
Bachman, L. F. & Palmer, A. S. 2010. Language Assessment in Practice: Developing Language Assessments and Justifying their Use in the Real World. Oxford: OUP.Google Scholar
Baten, K. & Håkansson, G. 2015. The development of subordinate clauses in German and Swedish as L2s: A theoretical and methodological comparison. Studies in Second Language Acquisition 37: 517–547. DOI logoGoogle Scholar
Bulté, B. & Housen, A. 2012. Defining and operationalising L2 complexity. In Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA [Language Learning & Language Teaching 32], A. Housen, F. Kuiken & I. Vedder (eds), 21–46. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Council of Europe 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge: CUP.Google Scholar
De Jong, N. H., Steinel, M. P., Florijn, A. F., Schoonen, R. & Hulstijn, J. H. 2012. Facets of speaking proficiency. Studies in Second Language Acquisition 34(1): 5–34. DOI logoGoogle Scholar
Eckes, T. 2009. Many-facets Rasch measurement. In Reference Supplement to the Manual for Relating Language Examinations to the Common European Framework of Reference for Language: Learning, Teaching, Assessment (Section H), S. Takala (ed.), 1–52. Strasbourg: Council of Europe.Google Scholar
Eklund Heinonen, M. 2009. Processbarhet på Prov: Bedömning av Muntlig Språkfärdighet hos Vuxna Andraspråksinlärare (Processability in Tests: Assessment of Oral Proficiency in Adult Second Language Learners). PhD dissertation, Uppsala University.Google Scholar
Ellis, R. 2008. Investigating grammatical difficulty in second language learning: Implications for second language acquisition research and language testing. International Journal of Applied Linguistics 18(1): 4–22. DOI logoGoogle Scholar
Ellis, R. & Barkhuizen, G. 2005. Analysing Learner Language. Oxford: OUP.Google Scholar
FIFA = Fédération Internationale de Football Association 2014/2015. Laws of the Game. Zurich: Fédération Internationale de Football Association.Google Scholar
Flyman Mattsson, A. & Håkansson, G. 2010. Bedömning av Svenska som Andraspråk: En Analysmodell Baserad på Grammatiska Utvecklingsstadier (Assessing Second Language Swedish: An Analytical Model Based on Grammatical Developmental Stages). Lund: Studentlitteratur.Google Scholar
Folkuniversitetet 2014. Swedex: Swedish examinations. <[URL]> (3 September 2014).
2010. Swedex B2 – Kriterier – Tala 2010-12-20. Kommentarer till bedömningskriterierna för muntlig uppgift [Swedex B2 – Criteria – Speaking 20 December 2010. Comments to the assessing criteria for the speaking task]. <[URL]> (1 May 2014).
Foster, P., Tonkyn, A. & Wigglesworth, G. 2000. Measuring spoken language: A unit for all reason. Applied Linguistics 21(3): 354–375. DOI logoGoogle Scholar
Fulcher, G. 2003. Testing Second Language Speaking. London: Longman.Google Scholar
2000. The ‘communicative’ legacy in language testing. System 28: 483–497. DOI logoGoogle Scholar
Glahn, E., Håkansson, G., Hammarberg, B., Holmen, A., Hvenekilde, A. & Lund, K. 2001. Processability in Scandinavian second language acquisition. Studies in Second Language Acquisition 23: 389–416. DOI logoGoogle Scholar
Granfeldt, J. & Ågren, M. 2013. Stages of processability and levels of proficiency in the Common European Framework of Reference for Languages. The case of L3 French. In Language Acquisition and Use in Multilingual Contexts: Theory and Practice, A. Flyman Mattsson & C. Norrby (eds), 28–38. Lund: Lund University Press.Google Scholar
Green, A. 2014. Exploring Language Assessment and Testing: Language in Action. Abingdon: Routledge.Google Scholar
Gyllstad, H., Granfeldt, H., Bernardini, P. & Källkvist, M. 2014. Linguistic correlates to communicative proficiency levels of the CEFR. In Eurosla Yearbook 14, L. Roberts, I. Vedder & J. H. Hulstjin (eds), 1–30. Amsterdam: John Benjamins.Google Scholar
Hammarberg, B. & Viberg, Å. 1977. The place-holder constraint, language typology, and teaching of Swedish to immigrants. Studia Linguistica 31: 106–163. DOI logoGoogle Scholar
Helsen, W., Gilis, B. & Weston, M. 2006. Errors in judging “offside” in association football: Test of the optical error versus the perceptual flash-lag hypothesis. Journal of Sports Sciences 24(5): 521–528. DOI logoGoogle Scholar
Housen, A., Kuiken, F. & Vedder, I. 2012. Complexity, accuracy and fluency: Definitions, measurements and research. In Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA [Language Learning & Language Teaching 32], A. Housen, F. Kuiken & I. Vedder (eds), 1–20. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Hulstijn, J. H. 2007. The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. The Modern Language Journal 91(4): 663–667. DOI logoGoogle Scholar
Hulstijn, J. H., Schoonen, R., De Jong, N. H., Steinel, M. P. & Florijn, A. 2011. Linguistic competences of Dutch as a second language at the B1 and B2 levels of speaking proficiency of the Common European Framework of Reference for Language (CEFR). Language Testing 29(2): 203–221. DOI logoGoogle Scholar
Hyltenstam, K. 1978. Variation in Interlanguage Syntax. PhD dissertation, Lund University.Google Scholar
Hymes, D. 1972. On communicative competence. In Sociolinguistics, J. Pride & J. Holmes (eds), 269–293. Harmondsworth: Penguin.Google Scholar
Iwashita, N., Brown, A., McNamara, T. & O’Hagan, S. 2008. Assessed levels of second language speaking proficiency: How distinct? Applied Linguistics 29(1): 24–49. DOI logoGoogle Scholar
Jin, T. & Mak, B. 2013. Distinguishing features in scoring L2 Chinese speaking performance: How do they work? Language Testing 30(1): 23–47. DOI logoGoogle Scholar
Jin, T., Mak, B. & Zhou, P. 2012. Confidence scoring of speaking performance: How does fuzziness become exact? Language Testing 29(1): 43–65. DOI logoGoogle Scholar
Kuiken, F. & Vedder, I. 2014. Raters’ decisions, rating procedures, and rating scales. Language Testing 31(3): 279–284. DOI logoGoogle Scholar
Luoma, S. 2004. Assessing Speaking. Cambridge: CUP. DOI logoGoogle Scholar
MacWhinney, B. 2000. The CHILDES Project: Tools for Analyzing Talk, 3rd edn. Mahwah, NJ: Lawrence Erlbaum Associates.Google Scholar
Martin, M. 2013. The complex simple: A problematic adjective in the CEFR writing scales. Nordand 8(2): 63–86.Google Scholar
Martin, M., Mustonen, S., Reiman, N., & Seilonen, M. 2010. On becoming an independent user. In Communicative Proficiency and Linguistic Development: Intersections between SLA and Language Testing Research [Eurosla Monographs Series 1], I. Bartning, M. Martin & I. Vedder (eds), 57–79. Paris: Eurosla.Google Scholar
Norris, J. & Ortega, L. 2009. Towards an organic approach to investigating CAF in instructed SLA: The case of complexity. Applied Linguistics 30(4): 555–578. DOI logoGoogle Scholar
Philipsson, A. 2007. Interrogative Clauses and Verb Morphology in L2 Swedish. Theoretical Interpretations of Grammatical Development and Effects of Different Elicitation Techniques. PhD dissertation, Stockholm University.Google Scholar
Pienemann, M. 1998. Language Processing and Second Language Development: Processability Theory [Studies in Bilingualism 15]. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Pienemann, M. & Håkansson, G. 1999. A unified approach toward the development of Swedish as L2. A processability account. Studies in Second Language Acquisition 21: 383–420. DOI logoGoogle Scholar
Platzack, C. 2001. The vulnerable C-domain. Brain and Language 77: 364–377. DOI logoGoogle Scholar
R Core Team 2015. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL: <[URL]>
RStudio Team 2015. RStudio: Integrated Development for R. R Studio. Boston MA. <[URL]>
Salameh, E.-K., Håkansson, G. & Nettelbladt, U. 1996. The acquisition of Swedish as a second language in a group of Arabic-speaking pre-school children: Word order pattern and phrasal morphology. Logopedics Phoniatrics Vocology 21(3–4): 163–170. DOI logoGoogle Scholar
Sandlund, E. & Sundqvist, P. 2014. Provuppgiftshantering som social praktik: En jämförelse mellan interaktionsanalys och bedömardata för muntligt prov i engelska (Task management as a social practise: A comparison between interaction analysis and assessment data from an English speaking test). In Språk i Undervisning: Rapport från ASLA:s Vårsymposium, Linköping, 11–12 maj 2012 (Language in teaching. Proceedings from ASLA’s spring symposium, Linköping, May 11–12, 2012), C. Rosén, P. Simfors & A.-K. Sundberg (eds), 125–138. Linköping: Linköping University.Google Scholar
Skehan, P. 1998. A Cognitive Approach to Language Learning. Oxford: OUP.Google Scholar
Spoelman, M. & Verspoor, M. 2010. Dynamic patterns in development of accuracy and complexity: A longitudinal case study in the acquisition of Finnish. Applied Linguistics 31(4): 532–553. DOI logoGoogle Scholar
Tomita, Y. & Spada, N. 2013. Form-focused instruction and learner investment in L2 communication. The Modern Language Journal 97(3): 591–610. DOI logoGoogle Scholar
Tonkyn, A. 2012. Measuring and perceiving change in oral complexity, accuracy and fluency. Examining instructed learners’ short-term gains. In Dimensions of L2 Performance and Proficiency: Complexity, Accuracy and Fluency in SLA [Language Learning & Language Teaching 32], A. Housen, F. Kuiken, & I. Vedder (eds), 1–20. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Van Moere, A. 2012. A psycholinguistic approach to oral language assessment. Language Testing 29(3): 325–344. DOI logoGoogle Scholar
Viberg, Å. 1990. Bisatser i inlärarperspektiv (Sub-clauses from the learners’ perspective). In Andra Symposiet i Svenska som Andraspråk i Göteborg 1989 (The second symposium for Swedish as a second language in Gothenburg 1989), G. Tingbjörn (ed.), 338–362. Stockholm: Scriptor.Google Scholar
Watkins, M. W. & Pacheco, M. 2001. Interobserver agreement in behavioral research: Importance and calculation. Journal of Behavioral Education 10(4): 205–212. DOI logoGoogle Scholar
Wijers, M. 2015. Forgotten factors in the development of dependent clauses in Swedish as a second language. Working papers in Scandinavian Syntax 95: 33–43.Google Scholar