An Evaluation of the Big Word Club Vocabulary Program
Susan E Mayer and Ariel Kalil, University of Chicago
Philip Oreopolous, University of Toronto

Executive Summary for Schools

We evaluated the effectiveness of the Big Word Club (BWC), a web-based program of activities intended to help elementary school-aged children learn new vocabulary words by introducing one new word per day throughout the school year. We estimate whether prekindergarten and kindergarten children in schools randomly assigned to participate in the Big Word Club scored higher than students in a control group of schools on an assessment of receptive vocabulary based on words included in the BWC program (the BWC Assessment). We also assessed students using a standardized test of receptive vocabulary called the Peabody Picture Vocabulary Test 4 (PPVT-4).

In treatment group schools, teachers were given access to the BWC web site. In control group schools, teachers were not given access to BWC. The evaluation was conducted with 818 pre-kindergarten and kindergarten students in 47 schools across Arizona, Colorado and Texas. Schools and students were balanced across treatment and control groups in key characteristics such as school type, student gender, and grade level. All students were assessed using the BWC Assessment 17 weeks and 25 weeks after the treatment group schools were given access to the BWC web site. During the time of the 25-week BWC Assessment, students were also assessed with the standardized PPVT-4.

The extent to which teachers used the BWC every day as it was intended varied. Web logins were tracked by BWC, and while this is not a perfect measure of usage, it was our best data for teachers’ use of the program. At the time we began the 25-week assessments there had been a maximum of 112 school days. By this time, treatment group teachers had logged in an average of 60 days. Two treatment group teachers never logged in. 45% of teachers had at least 50 logins, 36% had at least 75 logins and 25% had at least 100 logins. This suggests that about half of teachers had logged into the BWC as much as half the days that it was available. In all estimates of the effect of the BWC, students in treatment schools where teachers never logged in are included in the analysis. If more teachers used the BWC everyday as it was intended, the results may be different.

The BWC Assessment was modeled after the standardized PPVT-4 and contains words from weeks 1-16 of the BWC program. Words for later weeks were not included as teachers would not have been able to cover them in 17 weeks. We assessed 818 students with the BWC Assessment at 17 weeks after treatment group teachers first had access to the BWC. Results show that children in classrooms with access to the BWC for 17 weeks performed better than the control group at a statistically significant level. In percentile terms, the results show that a child at the 50th percentile would experience up to a 13% gain, jumping to the 63rd percentile after having access to the BWC program, on average. A child at the 80th percentile would experience up to a 7% gain, jumping to the 87th percentile after 17 weeks of BWC access, on average.

We assessed 603 previously assessed students with the very same BWC Assessment at 25 weeks after treatment teachers first had access to the BWC. Results of this second assessment round show that children in treatment classrooms retained their advantage over the control group
on the BWC Assessment. The magnitude of the results were very consistent with the 17-week results showing that children retained their knowledge of the words they learned 8 weeks later. We also assessed students at 25 weeks after treatment using the PPVT-4. Children in classrooms with access to the BWC also scored higher on the PPVT-4, but the difference was not statistically significant.

The evaluation detected no statistically significant difference in the treatment results for children by gender, whether the school was private, whether the students were observed to have English as a second language or whether the students were observed to have special needs and whether the school was designated as Title 1.

The results of the BWC evaluation are similar to other interventions that have been shown to improve the receptive and expressive vocabulary of pre-kindergarten and kindergarten children. However, these other interventions included a large number of curriculum programs that are much more expensive and demanding of teacher time. The BWC is a low cost and relatively low intensity intervention that requires no teacher training. This suggests that the BWC may be a cost-effective program to increase vocabulary, especially when used regularly.