Ch. 8 | Exercise 1

Chapter 8 | Exercise 1

Case study 1
‘Do subtitles really help in learning a foreign language?’

This case study is inspired by Winke et al. (2010), who studied the effect of captions (subtitles) on vocabulary learning. The data set captions in Rling simulates their experimental data for the second-year learners of Spanish (English native speakers). There were 47 participants, who watched some video fragments twice and later took a vocabulary test. The videos contained unknown words. Each participant was randomly assigned to one of the following test conditions: captions presented only during the first time the subjects watched the video, captions presented only during the second time,  no captions presented, and captions presented both times. Are there any differences in the vocabulary test performance depending on whether the captions were presented and how?

The data set captions in Rling contains 47 observations (subjects) and two variables. The variable test represents the vocabulary test scores. The variable cap_group shows four groups: “cap_both” (captions presented both times), “cap_first” (captions presented during the first time), “cap_second” (captions presented during the second time) and “no_cap” (no captions).


Make a box plot of the vocabulary test scores depending on the experimental group. Does it suggest any differences between the groups?


Test the assumptions of one-way ANOVA. Are they met?


Carry out a one-way ANOVA. What is the F-score? Is it significant?


Perform a post-hoc test. Which group means are significantly different from one another?

Load the data and make a box plot:

> library(Rling) > data(captions) > boxplot(captions$test ~ captions$cap_group)

The plot shows that the group that did not see the captions performed worse in the lexical test than the other groups. The plot also shows that the dispersion varies substantially, with the captions-first group having the broadest range and the no caption group having the smallest dispersion.

There are no indications that the observations or groups are dependent. To test the assumption of normality for all four groups, you can use the function by() and the Shapiro test:

> by(captions$test, captions$cap_group, shapiro.test) captions$cap_group: cap_both Shapiro-Wilk normality test data: dd[x, ] W = 0.9131, p-value = 0.3382 [output omitted]

All p-values are greater than 0.05, so we do not find violations of normality.

To test the assumptions of homogeneity of variance, you can use the Levene test:

> leveneTest(captions$test, captions$cap_group) Levene's Test for Homogeneity of Variance (center = median) Df F value Pr(>F) group 3 3.8013 0.01671 * 43 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

The Levene test shows that the variance is heterogeneous.

Finally, the design is not balanced: the experimental groups have a different number of participants.

> summary(captions$cap_group) cap_both cap_first cap_second no_cap 9 14 16 8

Since the assumption of homogeneity is violated, we will use oneway.test() with a correction for heterogeneous variance:

> oneway.test(test ~ cap_group, data = captions) One-way analysis of means (not assuming equal variances) data: test and cap_group F = 18.8435, num df = 3.000, denom df = 21.166, p-value = 3.472e-06

The omnibus F-test shows that at least one pair of groups has significantly different means: F(3, 21.17) = 18.84, p < 0.001.

Since the samples are not equal and the variance is not homogeneous, we will use a non-parametric post hoc test:

> library(nparcomp) > npar <- nparcomp(test ~ cap_group, data = captions, type = "Tukey") > npar$Analysis Comparison Estimator Lower Upper Statistic p.Value 1 p( cap_both , cap_first ) 0.456 0.174 0.770 -0.32754432 0.998707910 2 p( cap_both , cap_second ) 0.507 0.191 0.817 0.04897639 0.999999974 3 p( cap_both , no_cap ) 0.111 0.012 0.564 -2.30835809 0.104734102 4 p( cap_first , cap_second ) 0.612 0.320 0.840 0.97610398 0.845417751 5 p( cap_first , no_cap ) 0.138 0.026 0.490 -2.65144208 0.042346763 6 p( cap_second , no_cap ) 0.031 0.003 0.267 -3.67050218 0.001150837

We find that the no-caption group is significantly different from the groups where the captions were presented first and second.