Cover not available
Article published In:
Methodologies for intra-sentential code-switching research
Edited by Amaia Munarriz-Ibarrola, M. Carmen Parafita Couto and Emma Vanden Wyngaerd
[Linguistic Approaches to Bilingualism 8:1] 2018
► pp. 67–97

Using two-alternative forced choice tasks and Thurstone’s law of comparative judgments for code-switching research

ORCID logo | University of Southern Mississippi
| University of Illinois at Chicago
| Leiden University Center for Linguistics
ORCID logo | Universitat Autónoma de Barcelona

This article argues that 2-alternative forced choice tasks and Thurstone’s law of comparative judgments (Thurstone, 1927) are well suited to investigate code-switching competence by means of acceptability judgments. We compare this method with commonly used Likert scale judgments and find that the 2-alternative forced choice task provides granular details that remain invisible in a Likert scale experiment. In order to compare and contrast both methods, we examined the syntactic phenomenon usually referred to as the Adjacency Condition (AC) (apud Stowell, 1981), which imposes a condition of adjacency between verb and object. Our interest in the AC comes from the fact that it is a subtle feature of English grammar which is absent in Spanish, and this provides an excellent springboard to create minimal code-switched pairs that allow us to formulate a clear research question that can be tested using both methods.

Keywords: two-alternative forced choice and Thurstone’s law, code-switching, acceptability judgment
Article outline
  • 1.Introduction
    • 1.1Acceptability judgments in linguistic research
    • 1.2Two-alternative forced choice judgements
    • 1.3Thurstone’s law of comparative judgment
  • 2.Motivation
    • 2.1The AC
    • 2.2Research question and hypotheses
    • 2.3Methodology preview: The role of code-switching
  • 3.Experiment 1: Judgments and Likert scales
    • 3.1Method
      • 3.1.1Participants
      • 3.1.2Materials
      • 3.1.3Procedure
    • 3.2Results
    • 3.3Preliminary Discussion
  • 4.Experiment 2: 2AFC and Thurstone’s law
    • 4.1First session: Method
      • 4.1.1Participants
      • 4.1.2Materials
      • 4.1.3Procedure
    • 4.2Results of session 1
      • 4.2.1Unilingual sentences
      • 4.2.2Code switched-sentences
      • 4.2.3Calculation of the Thurstone measure
    • 4.3Second session: Method
      • 4.3.1Participants, Materials, and Procedure
      • 4.3.2Results of session 2
  • 5.Discussion
  • 6.Conclusions
  • Notes
  • Appendix
    • Sentences used in Experiments 1 and 2
      • Monolingual base sentences
        • Block 1
        • Block 2
    • Code switched sentences used in Experiments 1 and 2
      • Condition A: Vsp+ADVsp+OBJen
        • Block 1
        • Block 2
      • Condition B: Ven+ADVsp+OBJsp
        • Block 1
        • Block 2
      • Condition C: Vsp+ADVen+OBJen
        • Block 1
        • Block 2
      • Condition D: Ven+ADVen+OBJsp
        • Block 2
  • References
Published online: 18 September 2017
DOI logo
https://doi.org/10.1075/lab.16030.sta
References
Anderson, T.
(2006) Spanish-English bilinguals’ attitudes toward code-switching: proficiency, grammaticality, and familiarity. Doctoral dissertation, The Pennsylvania State University, State College, Pennsylvania.Google Scholar
Bader, M., & Häussler, J.
(2010) Toward a model of grammaticality judgments. Journal of Linguistics, 461, 273–330. DOI logoGoogle Scholar
Badiola, L., Delgado, R., Sande, A., & Stefanich, S.
(2017) Code-switching attitudes and their effects on acceptability judgment tasks. Linguistic Approaches to Bilingualism, 8(1). DOI logoGoogle Scholar
Bard, E. G., Robertson, D. & Sorace, A.
(1996) Magnitude estimation of linguistic acceptability. Language, 721, 32–68. DOI logoGoogle Scholar
Bialystok, E.
(1979) Explicit and implicit judgements of L2 grammaticality. Language Learning, 291, 81–103. DOI logoGoogle Scholar
Bock, R. D., & Jones, L. V.
(1968) The measurement and prediction of judgment and choice. San Francisco, CA: Holden-Day.Google Scholar
Bošković, Ž.
(1997) The Syntax of Nonfinite Complementation: An Economy Approach. Cambridge, MA: MIT Press.Google Scholar
Brown, T. C., & Peterson, G. L.
(2009) An enquiry into the method of paired comparison: reliability, scaling, and Thurstone’s Law of Comparative Judgment. Fort Collins, CO: U.S. Department of Agriculture, Forest Service.Google Scholar
Brysbaert, M., New, B., & Keuleers, E.
(2012) Adding Part of Speech information to the SUBTLEXUS word frequencies. Behavior Research Methods, 441, 991–997. DOI logoGoogle Scholar
Bullock, B. E. & Toribio, A. J.
(2012) The Cambridge Handbook of Linguistic Code-switching. Cambridge, UK: Cambridge University Press.Google Scholar
Caroll, J. M., Bever, T. G., & Pollack, C. R.
(1981) The non-uniqueness of linguistic intuitions. Language, 571, 368–383. DOI logoGoogle Scholar
Cattelan, M.
(2012) Models for paired comparison data: a review with emphasis on dependent data, Statistical Science, 271, 412–433 DOI logoGoogle Scholar
Chomsky, N.
(1957) Syntactic Structures. The Hague, the Netherlands: Mouton. DOI logoGoogle Scholar
(1986) Knowledge of Language: Its nature, origin and use. New York: Praeger.Google Scholar
(1995) The Minimalist Program, Cambridge, MA: MIT Press.Google Scholar
(2000) Minimalist Inquiries: The Framework. In R. Martin, D. Michaels & J. Uriagereka (Eds.), Step by Step: Essays in Minimalist Syntax in Honor of Howard Lasnik, 89–155. Cambridge, MA: MIT Press.Google Scholar
Cowart, W.
(1996) Experimental Syntax: Applying Objective Methods to Sentence Judgments. California: Sage Publications Inc.Google Scholar
David, H. A.
(1988) The method of paired comparisons (2nd ed.). New York, NY: Oxford University Press.Google Scholar
Duchon, A., Perea, M., Sebastián-Gallés, N., Martí, A., & Carreiras, M.
(2013) EsPal: One-stop shopping for Spanish word properties, Behavior Research Methods, 451, 1246–1258 DOI logoGoogle Scholar
Ebert, S. & Koronkiewicz, B.
2017). Monolingual stimuli as a foundation for analyzing code-switching data, Linguistic Approaches to Bilingualism, 8(1). DOI logo
Edwards, L.
(1957) Techniques of Attitude Scale Construction. Englewood Cliffs, New Jersey: Prentice-Hall. DOI logoGoogle Scholar
Engen, T.
(1971) Psychophysics, Vol. II: Scaling methods. In J. Kling & L. Riggs (Eds.), Woodworth and Schlosberg’s experimental psychology, 89–91. New York: Holt, Rinehart, & Winston.Google Scholar
Featherston, S.
(2005a) Magnitude estimation and what it can do for your syntax: Some WH-constructions in German. Lingua, 1151, 1525–50. DOI logoGoogle Scholar
(2005b) Universals and grammaticality: wh-constraints in German and English. Linguistics, 431, 667–711. DOI logoGoogle Scholar
Featherson, S.
(2008) Thermometer judgments as linguistic evidence. In C. M. Riehl & A. Rothe, (Eds.), Was ist linguistische Evidenz? Aachen: Shaker Verlag, 69–89.Google Scholar
Giancaspro, D.
(2013) L2 Learners’ and Heritage Speakers’ Judgments of Code-Switching at the Auxiliary-VP Boundary. Selected Proceedings of the 16th Hispanic Linguistics Symposium, J. Cabrelli Amaro et al. (Eds.), 56–69. Somerville, MA: Cascadilla Proceedings Project.Google Scholar
Gibson, E., Piantadosi, S., & Fedorenko, K.
(2011) Using Mechanical Turk to Obtain and Analyze English Acceptability Judgments. Language and Linguistics Compass 51, 509–524. DOI logoGoogle Scholar
Gigerenzer, G., & Richter, H.
(1990) Context effects and their interaction with development: Area judgments. Cognitive Development, 51, 235–264. DOI logoGoogle Scholar
Gigerenzer, G., Krauss, S. & Vitouch, O.
(2004) The null ritual: What you always wanted to know about significance testing but were afraid to ask. In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences. Thousand Oaks, CA: Sage. DOI logoGoogle Scholar
Green, D. M. & Swets, J. A.
(1966) Signal Detection Theory and Psychophysics, Wiley, New York.Google Scholar
Haegeman, L.
(1991) Introduction to government and binding theory. Oxford: Blackwell.Google Scholar
Johnson, K.
(1991) Object Positions. Natural Language and Linguistic Theory, 91, 577–637. DOI logoGoogle Scholar
Jones, L. V., & Thissen, D. A.
(2007) A History and Overview of Psychometrics. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics, volume 26: Psychometrics, 1–27. New York, NY: Elsevier.Google Scholar
Kratzer, A.
(1996) Severing the External Argument from its Verb, in J. Rooryck & L. Zaring (Eds.), Phrase Structure and the Lexicon. Dordrecht: Kluwer Academic Publishers. DOI logoGoogle Scholar
Labov, W.
(1972) Some principles of linguistic methodology, Language in Society 11, 97–120. DOI logoGoogle Scholar
(1975) What is a linguistic fact? Lisse: Peter de Ridder. DOI logoGoogle Scholar
López, L.
(2012) Indefinite objects: scrambling, choice functions and differential marking. Cambridge, Mass: MIT Press. DOI logoGoogle Scholar
Luce, R. D.
(2002) A psychophysical theory of intensity proportions, joint presentations, and matches. Psychological Review, 1091. 520–532. DOI logoGoogle Scholar
MacSwan, J.
(1999) A minimalist approach to intrasentential code switching: Spanish-Nahuatl bilingualism in Central Mexico. New York: Garland.Google Scholar
Montag, E. D.
(2006) Empirical formula for creating error bars for the method of paired comparisons. Journal of Electronic Imaging, 151, 222–230. DOI logoGoogle Scholar
Mohan, B. A.
(1977) Acceptability testing and fuzzy grammar. In S. Greenbaum (Ed.), Acceptability in language, 133–148. The Hague: MoutonGoogle Scholar
Muysken, P.
(2000) Bilingual Speech: A typology of code-mixing. Cambridge: Cambridge University PressGoogle Scholar
Munarriz, A. & Parafita Couto, M. C.
(2014) ¿Cómo estudiar el cambio de código ? Incorporación de diferentes metodologías en el caso de varias comunidades bilingües. Lapurdum, 181, 43–73 DOI logoGoogle Scholar
Nagata, H.
(1988) The relativity of linguistic intuition: The effect of repetition on grammaticality judgments. Journal of Psycholinguistic Research, 1711, 1–17. DOI logoGoogle Scholar
Narens, L.
(1996) A theory of ratio magnitude estimation. Journal of Mathematical Psychology, 401, 109–129 DOI logoGoogle Scholar
Nunnally, J. C.
(1967) Psychometric theory. New York: McGraw Hill.Google Scholar
Onar Valk, P.
(2014) Convergent developments in Dutch Tukish word order – A comparative study using ‘elicited production’ and ‘judgment’ data: Converging evidence?, Applied Linguistics Review, 51, 353–374. DOI logoGoogle Scholar
Oxford University Language Centre
Placement Tests.” lang.ox.ac.uk. [URL] (accessed July 1st, 2015)
Parafita Couto, M. C., Deuchar, M., & Fusser, M.
(2015) How do Welsh-English bilinguals deal with conflict? Adjective-noun order resolution. In: G. Stell, K. Yakpo (Eds.), Code-switching at the crossroads between structural and sociolinguistic perspectives, 65–84. Boston: Mouton de Gruyter.Google Scholar
Parafita Couto, M. C., Munarriz, A., Epelde, I., Deuchar, M., & Oyharçabal, B.
(2015) Gender conflict resolution in Spanish-Basque mixed DPs. Bilingualism, Language and Cognition, 181, 304–323. DOI logoGoogle Scholar
Parraga, C. A.
(2015) Perceptual Psychophysics. In G. Cristobal, M. Keil, & L. Perrinet (Eds.), Biologically-Inspired Computer Vision: Fundamentals and Applications, 81–108. New York, NY: Wiley. DOI logoGoogle Scholar
Peer, E., Vosgerau, J., & Acquisti, A.
(2014) Reputation as a sufficient condition for data quality on Amazon Mechanical Turk. Behavior Research Methods, 461, 1023–1031 DOI logoGoogle Scholar
Pollock, J. Y.
(1989) Verb Movement, Universal Grammar, and the Structure of IP. Linguistic Inquiry, 201, 365–424Google Scholar
Reber, A.
(1995) The Penguin dictionary of psychology. New York: Penguin Books.Google Scholar
Ross, J. R.
(1979) Where’s English? In C. J. Fillmore, D. Kemper, & W. S. Wang (Eds.), Individual differences in language ability and language behavior, 127–163. New York: Academic Press. DOI logoGoogle Scholar
Serratrice, L., Sorace, A., Filiaci, F., & Baldo, M.
(2009) Bilingual children’s sensitivity to specificity and genericity: Evidence from metalinguistic awareness. Bilingualism: Language and Cognition, 121, 239–257. DOI logoGoogle Scholar
Schütze, C. T.
(2016) The empirical base of linguistics. Grammaticality judgments and linguistic methodology. Chicago & London: University of Chicago Press. DOI logoGoogle Scholar
Schütze, C. T., & Sprouse, J.
(2013) Judgment Data. In R. J. Podesva and D. Sharma (Eds.), Research Methods in Linguistics, pp. 27–50. Cambridge, England: Cambridge University Press.Google Scholar
Shin, F., Goodall, G., Michel, D. & Beecher, H.
(2012) Is Magnitude Estimation worth the trouble? In J. Choi, E. Alan Hogue, J. Punske, D. Tat, J. Schertz & A. Truman (Eds.), Proceedings of the 29th West Coast Conference on Formal Linguistics (WCCFL29). 328–336. Somerville, MA: Cascadilla Proceedings Project.Google Scholar
Snow, C., & Meijer, G.
(1977) On the secondary nature of syntactic intuitions. In S. Greenbaum (Ed.), Acceptability in language, 163–177. The Hague, the Netherlands: Mouton.Google Scholar
Sorace, A.
(1996) The use of acceptability judgments in second language acquisition research. In W. C. Ritchie and T. K. Bhatia (Eds.), Handbook of second language acquisition, pp. 375–409. San Diego, CA: Academic Press.Google Scholar
(2010) Magnitude estimation in language acquisition research. In S. Unsworth & E. Blom (Eds.), Experimental Methods in Language Acquisition, pp. 57–72. Amsterdam: John Benjamins. DOI logoGoogle Scholar
Sorace, A., & Keller, F.
(2004) Gradience in linguistic data. Lingua, 1151, 1497–1524. DOI logoGoogle Scholar
Sorace, A., Serratrice, L., Filiaci, F., & Baldo, M.
(2009) Discourse conditions on subject pronoun realization: Testing the linguistic intuitions of older bilingual children. Lingua, 1191, 460–477. DOI logoGoogle Scholar
Sprouse, J.
(2007) Continuous acceptability, categorical grammaticality, and experimental syntax. Biolinguistics, 11, 118–129.Google Scholar
(2011) A Test of the Cognitive Assumptions of Magnitude Estimation: Commutativity does not Hold for Acceptability Judgments. Language, 871, 274–288. DOI logoGoogle Scholar
Sprouse, J. & Almeida, D.
(2011) Power in acceptability judgment experiments and the reliability of data in syntax. Ms., University of California, Irvine & Michigan State University.Google Scholar
(2012) The empirical status of data in syntax: A reply to Gibson and Fedorenko. Language and Cognitive Processes. iFirst. 1–7.Google Scholar
Sprouse, J., Schütze, C. T. & Almeida, D.
(2013) A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001–2010. Lingua, 1341, 219–248. DOI logoGoogle Scholar
Stevens, S. S.
(1946) On the Theory of Scales of Measurement. Science, 1031, 677–680. DOI logoGoogle Scholar
(1956) The direct estimation of sensory magnitudes: loudness. The American journal of psychology, 691, 1–25 DOI logoGoogle Scholar
Stowell, T.
(1981) Origins of phrase structure. PhD dissertation, Massachusetts Institute of Technology, Cambridge, MA.Google Scholar
Stokes, W.
(1974) All of the work on quantifier-negation isn’t convincing. In M. W. La Galy, R. A. Fox, & A. Bruck (Eds.), Papers from the tenth regional meeting, Chicago Linguistic Society, 692–700. Chicago: Chicago Linguistic Society.Google Scholar
Thurstone, L.
(1927) A Law of Comparative Judgment. Psychological Review, 341, 273–286. DOI logoGoogle Scholar
Tikofsky, R. S., & Reiff, D. G.
(1970) Structural apperception in the absence of syntactic constraints, Language and Speech, 131, 240–253. DOI logoGoogle Scholar
Tsukida, K. & Gupta, M. R.
(2011) How to Analyze Paired Comparison Data (UWEE Technical Report Number UWEETR-2011–0004) Seattle, University of Washington.Google Scholar
Torgerson, W. S.
(1958) Theory and methods of scaling. New York, NY: John Wiley & SonsGoogle Scholar
Toribio, A. J.
(2001) On the emergence of bilingual code-switching competence. Bilingualism: Language and Cognition, 41, 203–231. DOI logoGoogle Scholar
Weskott, T. & Fanselow, G.
(2008) Variance and informativity in different measures of linguistic acceptability. Proceedings of the West Coast Conference on Formal Linguistics (WCCFL), 271, 431–439.Google Scholar
Weskott, T., & Fanselow, G.
(2011) On the Informativity of Different Measures of Linguistic Acceptability. Language, 871, 249–273 DOI logoGoogle Scholar
Cited by

Cited by 24 other publications

Balam, Osmer, María del Carmen Parafita Couto & Hans Stadthagen-González
2020. Bilingual verbs in three Spanish/English code-switching communities. International Journal of Bilingualism 24:5-6 ► pp. 952 ff. DOI logo
Bellamy, Kate, M. Parafita Couto & Hans Stadthagen-Gonzalez
2018. Investigating Gender Assignment Strategies in Mixed Purepecha–Spanish Nominal Constructions. Languages 3:3 ► pp. 28 ff. DOI logo
Bellamy, Kate & M. Carmen Parafita Couto
2022. Chapter 2. Gender assignment in mixed noun phrases. In The Acquisition of Gender [Studies in Bilingualism, 63], ► pp. 13 ff. DOI logo
Bellamy, Kate, Hans Stadthagen-Gonzalez & M. Carmen Parafita Couto
2022. Él Code-Switches More Than tú y yo: New Data for the Subject Pronoun-Verb Switch Constraint. Languages 7:1 ► pp. 22 ff. DOI logo
Camacho, José
2018. The Interpretation of Adjective-N Sequences in Spanish Heritage. Languages 3:4 ► pp. 46 ff. DOI logo
De Nicolás, Irati & Luis López
2022. Adjective Placement in English/Spanish Mixed Determiner Phrases: Insights from Acceptability Judgments. Languages 7:1 ► pp. 54 ff. DOI logo
Ebert, Shane & Bryan Koronkiewicz
2018. Monolingual stimuli as a foundation for analyzing code-switching data. Linguistic Approaches to Bilingualism 8:1 ► pp. 25 ff. DOI logo
Gerasimova, Anastasia & Ekaterina Lyutikova
2020. Intralingual Variation in Acceptability Judgments and Production: Three Case Studies in Russian Grammar. Frontiers in Psychology 11 DOI logo
Greidanus Romaneli, Miriam, Ivo H. G. Boers & M. Carmen Parafita Couto
2021. Gender in Dutch/Portuguesecodeswitching. Belgian Journal of Linguistics 35 ► pp. 131 ff. DOI logo
Kootstra, Gerrit Jan, Ton Dijkstra & Janet G. van Hell
2020. Interactive Alignment and Lexical Triggering of Code-Switching in Bilingual Dialogue. Frontiers in Psychology 11 DOI logo
Koronkiewicz, Bryan
2019. Control Stimuli in Experimental Code-Switching Research. Languages 4:3 ► pp. 67 ff. DOI logo
Koronkiewicz, Bryan
2022. Adverbs in Spanish–English code-switching: Comparing verb raising and non-raising. International Journal of Bilingualism 26:2 ► pp. 227 ff. DOI logo
López, Luis
2020. Bilingual Grammar, DOI logo
Munarriz-Ibarrola, Amaia, Maria del Carmen Parafita Couto & Emma Vanden Wyngaerd
2018. Methodologies for intra-sentential code-switching research. Linguistic Approaches to Bilingualism 8:1 ► pp. 1 ff. DOI logo
Pablos, Leticia, M. Carmen Parafita Couto, Bastien Boutonnet, Amy de Jong, Marlou Perquin, Annelies de Haan & Niels O. Schiller
2019. Adjective-noun order in Papiamento-Dutch code-switching. Linguistic Approaches to Bilingualism 9:4-5 ► pp. 710 ff. DOI logo
Pablos, Leticia, M. Carmen Parafita Couto, Bastien Boutonnet, Amy de Jong, Marlou Perquin, Annelies de Haan & Niels O. Schiller
2021. Adjective-noun order in Papiamento-Dutch code-switching. In Psycholinguistic Approaches to Production and Comprehension in Bilingual Adults and Children [Benjamins Current Topics, 117], ► pp. 211 ff. DOI logo
Parafita Couto, Maria Carmen & Hans Stadthagen-Gonzalez
2019. El book or the libro? Insights from acceptability judgments into determiner/noun code-switches. International Journal of Bilingualism 23:1 ► pp. 349 ff. DOI logo
Saad, George, Marian Klamer & Francesca Moro
2019. Identifying agents of change: Simplification of possessive marking in Abui-Malay bilinguals. Glossa: a journal of general linguistics 4:1 DOI logo
Sheehan, Michelle, Martin Schäfer & Maria Carmen Parafita Couto
2019. Crowdsourcing and Minority Languages: The Case of Galician Inflected Infinitives1. Frontiers in Psychology 10 DOI logo
Stadthagen-González, Hans, M Carmen Parafita Couto, C Alejandro Párraga & Markus F Damian
2019. Testing alternative theoretical accounts of code-switching: Insights from comparative judgments of adjective–noun order. International Journal of Bilingualism 23:1 ► pp. 200 ff. DOI logo
Suurmeijer, Luuk, M. Carmen Parafita Couto & Marianne Gullberg
2020. Structural and Extralinguistic Aspects of Code-Switching: Evidence From Papiamentu-Dutch Auditory Sentence Matching. Frontiers in Psychology 11 DOI logo
Valdés Kroff, Jorge R., Rosa E. Guzzardo Tamargo & Paola E. Dussias
2018. Experimental contributions of eye-tracking to the understanding of comprehension processes while hearing and reading code-switches. Linguistic Approaches to Bilingualism 8:1 ► pp. 98 ff. DOI logo
Vanden Wyngaerd, Emma
2020. C0 and Dutch-English code-switching. Ampersand 7 ► pp. 100060 ff. DOI logo
Vaughan-Evans, Awel, Maria Carmen Parafita Couto, Bastien Boutonnet, Noriko Hoshino, Peredur Webb-Davies, Margaret Deuchar & Guillaume Thierry
2020. Switchmate! An Electrophysiological Attempt to Adjudicate Between Competing Accounts of Adjective-Noun Code-Switching. Frontiers in Psychology 11 DOI logo

This list is based on CrossRef data as of 23 april 2022. Please note that it may not be complete. Sources presented here have been supplied by the respective publishers. Any errors therein should be reported to them.