Article published In:
ITL - International Journal of Applied LinguisticsVol. 127/128 (2000) ► pp.1–36
Vol. 127/128 (2000) ► pp.1–36
Tests as a Second Language Research Method
Their Types, Reliability, Validity, and Variable Research Results
In second language acquisition (SLA) research various tests are employed for theory building. However, what tests should be administered to participants in a research setting? In order to address the issue, this study was conducted to examine the reliability, validity of tests themselves, and the difference in results according to the difference of test-types, focusing on cross-sectional restrictive relative clause acquisition and relative clause test-types. Among the variety of relative clause tests, four test-types that appear frequently in recent SLA academic publications were selected. The four test-types were Translation, Cloze Procedure, Grammaticality Judgment, and Sentence Combining. The results from 120 Japanese students indicate the following. First, Sentence Combining shows high reliability in internal consistency and the highest validity. Second, the research results change across the different test-types. Third, the concept of 'test-type (task) related interlanguage variability' should be explained by the combination of quality-related issues such as 'measurement error' and the cognitive demands each test-type requires of the subjects. Implications for the practical issues of language test construction for SLA research, educational evaluation, and the directions for further research are also discussed.
References
AARTS, F. & SCHILS, E.
(1995) : Relative clauses, the accessibility hierarchy and the contrastive analysis hypothesis. Internationa Review of Applied Linguistics 331, 47–63.
AKAGAWA, Y.
(1992) : Avoidance of relative clauses by Japanese high school students. JACET Bulletin 231, 1–18.
ALDERSON, J.C., CLAPHAM, C, WALL, D.
(1995) : Language test construction and evaluation. Cambridge: Cambridge University Press.
ARTHUR, B.
(1980) : Gauging the boundaries of second language competence : A study of learner judgment. Language Learning 301, 178–194. 
BACHMAN, L.F.
(1990) : Fundamental considerations in language testing. Oxford: Oxford University Press.
BAILEY, N., EISENSTEIN, M. & MADDEN, C.
(1976) : The development of wh- questions in adult second language learners. In Fanslow, J. & Crymes, R. (Eds.), ON TESOL '761, 1-9. Washington, D.C: TESOL.
BEEBE, L.M.
(1980) : Sociolinguistic variation and style shifting in second Language acquisition. Language Learning 301, 178–194.
(1988) : Five sociolinguistic approaches to second language acquisition. In Beebe, L.M. (Ed.), Issues in second language acquisition : Multiple perspectives (pp. 43–77). New York : Newbury House.
BROWN, J.D.
(1988) : Understanding research in second language learning : A teacher's guide to statistics and research design. London : Cambridge University Press.
(1996) : Testing in language programs. Upper Saddle River, NJ : Prentice Hall Regents.
BUCK, G.
(1992) : Translation as a language testing procedure : Does it work? Language Testing 91, 123–148.
CELCE-MURCIA, M. & LARSEN-FREEMAN, D.E.
(1983) : The grammar book-An ESL/EFL teacher's course. Rowley, MA : Newbury House.
CHAUDRON, C.
(1983) : Research on metalinguistic judgments : A review of theory, methods, and results. Language Learning 331, 343–377.
DEKEYSER, R.
(1990) : Towards a valid measurement of monitored knowledge. Language Testing 71, 147–157.
DlCKERSON, L.J.
(1975) : The learner's interlanguage as a system of variable rules. TESOL Quarterly 91, 401–407.
DUFF, A. P.
(1993) : Tasks and interlanguage : An SLA perspective. In Crookes, G. & Gass, S.M. (Eds.), Tasks and language learning : Integrating theory & practice (pp. 57–95). Bristol, PA : Multilingual Matters.
DULAY, H., BURT, M. & KRASHEN, S.D.
(1982) : Language two. Oxford : Oxford University Press.
ECKMAN, F.R., BELL, L. & NELSON, D.
(1988) : On the generalization of relative clause instruction in the acquisition of English as a second language. Applied Linguistics 91, 1–20.
EBEL, R.L.
(1979) : Essentials of educational measurement (Third edition.). Prentice-Hall, Inc.
EISENSTEIN, M., BAILEY, N. & MADDEN, C.
(1982) : It takes two : Contrasting tasks and contrasting structures. TESOL Quarterly 161, 381–391.
ELLIS, R.
(1986) : Sources of variability in interlanguage. Applied Linguistics 61, 118–131.
(1991) : Grammar judgments and second language acquisition. Studies in Second Language Acquisition 131, 161–186.
(1994) Data, theory, and applications in second language acquisition research. In Ellis, R. The study of second language acquisition (pp. 669–691). Oxford : Oxford University Press.
FULCHER, G.
(1995) : Variable competence in second language acquisition : A problem for research methodology? System 231, 25–33.
GASS, S.
(1979) : Language transfer and universal grammatical relations. Language Learning 291, 327–344.
(1980) : An investigation of systematic transfer in adult second language learners. In Scarcella, R.C., & Krashen, S.D. (Eds.), Research in second language acquisition (pp. 132–141). Rowley, MA : Newbury House.
(1982) : From theory to practice. In Hines, M. & Rutherford, W. (Eds.), On TESOL '81. Washington, D.C. : TESOL, 129–139.
(1983) : The development of L2 intuitions. TESOL Quarterly 171, 273–291.
GASS, S. & SELINKER, L.
(1994) : Second language acquisition : An introductory course. Hillsdale, NJ : Lawrence Erlbaum Associates.
HATCH, E. & LAZARTON, A.
(1991) : The research manual. New York : Newbury House.
HENNING, G.
(1987) : A guide to language testing : Development, evaluation, research. Boston, MA : Heinle & Heinle Publishers.
HUEBER, T.
(1985) : System and variability in interlanguage syntax. Language Learning 351, 141–163.
HULL, J.
(1986) Task variation in interlanguage performance : Does it Affect monitoring? Working Papers 5. Department of ESL. The University of Hawaii at Manoa.
HULSTIJN, J.
(1989): A cognitive view on interlanguage variability. In Eisenstein (Ed.), The dynamic interlanguage : Empirical studies in second language variation (pp. 17–31). New York : Plenum Press.
HYLTENSTAM, K.
(1983) : Data type and second language variability. In Rongbon, H. (ed.), Psycholinguistics and foreign language learning (pp. 57–74). Abo : Abo Akademi.
IKEBE, Y.
(1990) : A study on the acquisition of relative clauses by Japanese learners of English. Unpublished MA thesis, Hiroshima University.
INOI, S.
(1991) : Variation in interlanguage with special reference to articles and pronouns. Annual Review of English Language Education in Japan 21, 1–10.
ITO, A.
(1995) : A study on the effects of difference of test-types on interlanguage performance of Japanese EFL learners (Manuscript). A paper presented at 26th CASELE Conference at Yamaguchi University, Japan.
(1996a) : Testing English tests : A language proficiency perspective. JALT Journal181, 183–197.
(1996b) : A study on the variability of test performance of Japanese EFL learners : A combination of two theoretical frameworks. CELES Bulletin 261, 227–234.
(1997): Japanese EFL learners' test-type related interlanguage variability. JALT Journal 191, 89–105.
(1998) : The author responds : More on Test-Type. JALT Journal 201, 89–90.
(1999) : A study of test-type related variability of interlanguage performance among Japanese EFL learners : A focus on relative clause tests. Unpublished Ph.D. Dissertation, Hiroshima University.
JONES, A.
(1991) : Generalization in the acquisition of relative clauses in English. Annual Report of Studies : The Faculty of Letters of Jissen Women's University 331, 1–39.
KAMEYAMA, T.
(1987) : Bunpo test niokeru test keishiki niyoru keitaiso seitoritsu no henka. (The change of accuracy rate in grammar tests by the difference of test-types). CELES Bulletin 171, 248–252.
KAWAUCHI, C.
(1988) : Universal processing of relative clauses by adult learners of English. JACET Bulletin 191, 19–36.
KEENAN, E.L.
(1975) : Variation in universal grammar. In Fasold, R. & Shuy, R. (Eds.), Analyzing variation in language (pp. 136–148). Washington, D.C. : Georgetown University Press.
KEENAN, E.L., & COMRIE, B.
(1977) : Noun phrase accessibility and universal grammar. Linguistic Inquiry 81, 63–99.
KLEIN-BRALEY, C.
(1982) : Die Ubersetzung als Testverfahren in der Staatprufung fur Lerramtskandidaten. Neusprachliche Mitteilungen 21, 94–97.
(1987) : Fossil at large : Translation as a language testing procedure. Colloquium on language testing research, April, 6–9.
KRASHEN, S.D.
(1977a) : The monitor model for adult second language performance. In Burt, M., Dulay, H. & Finoccjiaro, M., Viewpoints on English as a second language (pp. 152–161). New York : Regents.
(1977b) : Some issues relating to the monitor model. In Brown, D.H., Yorio, CA., & Crymes, R.H. (Eds.), ON TESOL '771, 144-158. Washington, D.C. : TESOL.
(1978a) : Individual variation in the use of the monitor. In Ritchie, W.C. (Ed.), Second language acquisition research : Issues and implications (pp. 175–183). New York : Academic Press.
(1978b) : The monitor model for second language acquisition. In Gingras, R.C. (Ed.), Second language acquisition and foreign language teaching (pp. 1–26). New York : Center for Applied Linguistics.
(1981) : Second language acquisition and second language learning. Oxford : Pergamon Press.
(1982) : Principles and practice in second language acquisition. Oxford : Pergamon Press.
(1985) : The input hypothesis : Issues and implications. London : Longman.
KRASHEN, S.D. & TERRELL, T.
(1983) : The natural approach : Language acquisition in the classroom. Oxford : Pergamon Press.
KUBOTA, M.
(1993) : Accuracy order and frequency order of relative clauses as used by Japanese senior high school students of EFL. IRLT Bulletin 71, 27–53.
KURODA, T.
(1986) : A research on cross-language influence in Japanese EFL learners. Unpublished MA thesis, Hyogo University of Teacher Education.
KUNO, S.
(1976) : Subject, theme, and the speaker's empathy - a reexamination of relativization phenomena. In Li, C. (Ed.), Subject and topic (pp. 417–444). Academic Press.
LEE, Y.O., KRASHEN, S.D., and GRIBBONS, B.
(1995) : The effects of reading on the acquisition of English relative clauses. I.T.L. Review of Applied Linguistics113–114, 263–273.
LARSEN-FREEMAN, D.E.
(1975): The acquisition of grammatical morphemes by adult ESL students. TESOL Quarterly 91, 409–419.
LARSEN-FREEMAN, D. LONG, M.H.
(1991) : An introduction to second language acquisition research. London : Longman.
LOSCHKY, L. & BLEY-VROMAN, R.
(1993) : Grammar and task-based methodology. In Crookes, G. & Gass, S.M. (Eds.), Tasks and language learning : Integrating theory and practice (pp. 123–167). Bristol, PA : Multilingual Matters.
LONG, M.H. & SATO, C.J.
(1984) : Methodological issues in interlanguage studies : An interactionist perspective. In A. Davies, A., C. Criper., & A. P. R. Howatt, (Eds.), Interlanguage (pp. 253–279). Edinburgh : Edinburgh University Press.
MAKONI, S.B.
(1996) : Variation in unplanned discourse. International Review of Applied Linguistics 341, 167–181.
MOCHIZUKI, A.
(1996) : A comparison of C-tests with four different texts : Their reliability and validity. A paper presented at 22nd FELES Conference at Tohoku Gakuin Unviersity, Japan.
OHBA, H.
(1987) : Task variation riron ni tsuite. (On the task variation theory). CELES Bulletin 171, 43–49.
(1994a): Nihonjin eigo gakushusha no chukangengo kahen-sei: Task keishiki no kanten kara. (Japanese EFL learners' interlanguage variability : With reference to the effects of task-type) CELES Bulletin 241, 187–192.
(1994b): Test keishiki no chigai niyoru eigo gakushusha no performance no kahensei. (Task-related variability in interlanguage by Japanese learners of English) STEP Bulletin 61, 34–48.
(1995) : The learning order of English relative clauses by Japanese senior high school students in an instruction-only environment. Journal of Health Sciences University of Hokkaido, 211, 19–35.
OHTOMO, K.
(1994) : Chapter 17. Gengo test to dai 2 gengo shutoku. (Language testing and second language acquisition). In Koike, I. (Ed.), Dai 2 gengo shutoku kekyu ni motozuku saishin no eigo kyoiku. (The latest English language education based on second language acquisition research) (pp. 300–312). Tokyo : Taishukans-hoten.
(1996) : Komoku oto riron nyumon. (An introduction to item response theory). Tokyo : Taishukanshoten.
PAVESI, M.
(1986) : Markedness, discourse modes, and relative clause formation in a formal and an informal context. Studies in Second Language Acquisition 81, 93–105.
QUIRK, R., GREENBAUM., LEECH, G., & SVARTVIK, J.
(1985): A comprehensive grammar of the English language. London : Longman.
SADIGHI, F.
(1994) : The acquisition of English relative clauses by Chinese, Japanese, and Korean adult native speakers. International Review of Applied Linguistics 321, 141–153.
SAJJADl, S. & TAHRIRIAN, M.H.
(1992): Task variability and interlanguage use. International Review of Applied Linguistics 301, 35–44.
SATO, C.J.
(1985) : Task variation in interlanguage phonology. In S. Gass. & Madden, C.G. (Eds.). Input on Second Language Acquisition (pp.181–196). Rowley, MA : Newbury House.
SCHACHTER, J.
(1974): An error in error analysis. Language Learning 241, 205–214.
SCHACHTER, J., TYSON, A., & DIFFEY, F.J.
(1976): Learner intuitions of grammaticality. Language Learning 261, 67–76.
SCHMIDT, M.
(1980) : Coordinate structures and language universals in interlanguage. Language Learning 301, 397–416.
SCHMIDT, R.W.
(1987) : Sociolinguistic variation and language transfer in phonology. In Ioup, G. & Weinberger, S.H. (Eds.), Interlanguage phonology (pp.365–377). Rowley, MA : Newbury House.
SHOHAMY, E.
(1994) : The role of language tests in the construction and validation of second-language acquisition theories. In E.E. Tarone, Gass, S.M., and Cohen, A.D. (Eds.), Research methodology in second language acquisition (pp.133–142). Hillsdale, NJ : Lawrence Erlbaum Associates.
SHIZUKA, T.
(1993) : Task variation and accuracy predictor in interlanguage phonology production. Bulletin of Kanto-Koshin-etsu. English Language Education Society 71, 63–77.
Statistical Product and Service Solutions Windows 7.5 version.
(1966) Chicago, IL : SPSS Inc.
STAUBLE, A.
(1984) : A comparison of a Spanish-English and a Japanese English second language continuum: Negation and verb morphology. In Andersen, R. (Ed.), Second languages : A Cross-linguistic perspective (pp. 323–353). Rowley, MA: Newbury House.
SWAIN, M.
(1993) : Second language testing and second language acquisition : Is there a conflict with traditional psychometrics? Language Testing 101, 193–207.
TAKAMIYAGI, T.
(1991) : A study of task related variation in interlanguage by Japanese university students in an instruction only environment. Joetsu University of Education Kenkyu Ronshu. Bulletin of Language Studies 61, 47–64.
TARONE, E.
(1979) : Interlanguage as chameleon. Language Learning 291, 181–191.
(1982) : Systematicity and attention in interlanguage. Language Learning 321, 142–164
(1983) : On the variability of interlanguage system. Applied Linguistics 41, 142–164.
TARONE, B.
(1985) : Variability in interlanguage use : A study of style-shifting in morphology and syntax. Language Learning 351, 373–403.
TARONE, E.
(1988) : Variation in interlanguage. London : Edward Arnold Publishers.
(1989): Accounting for style-shifting in interlanguage. In S. Gass, Madden, C., Preston, D. & Selinker, L. (Eds.), Variation in second language acquisition : Psycholinguistic issues (pp. 13–21). Clevedon, England: Multilingual Matters.
TARONE, E. & PARRISH, B.
(1988) : Task-related variation in interlanguage : The case of articles. Language Learning 381, 21–44.
TITFORD, C.
(1983) : Translation and testing. International Review of Applied Linguistics 111, 312–319.
TOMITA, Y.
(1988) : Nihonjin kokosei no chukan gengo nitsuiteno ichikousatsu : Keitaiso shutoku junjo kenkyu. (A study on high school students' interlanguage : With reference to the acquisition order of morphologies). CELES Bulletin 181, 208–213.