A school-based program designed to enhance vocabulary knowledge and comprehension among middle school students, with special attention to English language learners.
Blueprints: Inconclusive Evidence
Sharon Vaughn
The University of Texas at Austin, College of Education
This school-based program designed to enhance vocabulary knowledge and comprehension among English language learners. The program is implemented during social studies class (i.e., typically 50 minutes a day and 5 days a week) for approximately 9 to 12 weeks, and includes: (a) an overview and vocabulary instruction, (b) the use of brief videos and purposeful discussion to build concepts, (c) the use of graphic organizers and other writing activities to build comprehension and vocabulary through writing, and (d) structure paired grouping.
This school-based program is implemented during social studies class (i.e., typically 50 minutes a day and 5 days a week) for approximately 9 to 12 weeks. It includes: (a) an overview and vocabulary instruction, (b) the use of brief videos and purposeful discussion to build concepts, (c) the use of graphic organizers and other writing activities to build comprehension and vocabulary through writing, and (d) structure paired grouping. The unit lessons are designed around one or two central ideas that serve as organizing concepts to help the teacher focus the events and ideas in each unit. The daily instructional routine consists of the following exercises: (a) a brief overview of the "big idea," (b) explicit vocabulary instruction that integrates paired students' discussion of the word, (c) discussion built around a short video clip (2-4 min) that complements the day's reading, (d) a teacher-led or paired student reading assignment followed by generating and answering questions, and (e) a wrap-up activity in the form of a graphic organizer or other writing exercise.
In both studies (Vaughn et al., 2009), as compared to students in control, students in treatment earned higher scores in:
Vaughn et al. (2009) conducted two studies over two consecutive school years (2006~2007 & 2007~2008), using different, non-overlapping samples. In both samples, Vaughn et al. (2009) conducted a randomized controlled trial to evaluate whether the program enhances students' vocabulary knowledge and comprehension. In the first sample, a total of 381 seventh graders from two participating schools were assigned to either treatment (7 classes, 176 students) or control (8 classes, 205 students). In the second sample, a total of 507 seventh graders from two participating schools were randomly assigned to either treatment (9 classes, 273 students) or control (8 classes, 234 students). A pretest and a posttest were assessed before and after the 9-12 week intervention.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
Sharon VaughnThe University of Texas at Austin, College of EducationAustin, TX USASRVaughn@teachnet.edb.utexas.edu
A school-based program designed to enhance vocabulary knowledge and comprehension among middle school students, with special attention to English language learners.
Middle Schools with a high proportion of English Language Learners in Texas.
*Risk/Protective Factor was significantly impacted by the program
This school-based program designed to enhance vocabulary knowledge and comprehension among English language learners. The program is implemented during social studies class (i.e., typically 50 minutes a day and 5 days a week) for approximately 9 to 12 weeks, and includes: (a) an overview and vocabulary instruction, (b) the use of brief videos and purposeful discussion to build concepts, (c) the use of graphic organizers and other writing activities to build comprehension and vocabulary through writing, and (d) structure paired grouping.
This school-based program is implemented during social studies class (i.e., typically 50 minutes a day and 5 days a week) for approximately 9 to 12 weeks. It includes: (a) an overview and vocabulary instruction, (b) the use of brief videos and purposeful discussion to build concepts, (c) the use of graphic organizers and other writing activities to build comprehension and vocabulary through writing, and (d) structure paired grouping. The unit lessons are designed around one or two central ideas that serve as organizing concepts to help the teacher focus the events and ideas in each unit. The daily instructional routine consists of the following exercises: (a) a brief overview of the "big idea," (b) explicit vocabulary instruction that integrates paired students' discussion of the word, (c) discussion built around a short video clip (2-4 min) that complements the day's reading, (d) a teacher-led or paired student reading assignment followed by generating and answering questions, and (e) a wrap-up activity in the form of a graphic organizer or other writing exercise.
Vaughn et al. (2009) identified four instructional practices associated with improved outcomes for English language learners: (a) research-based vocabulary and concept instruction, (b) the use of media to build comprehension and concept knowledge, (c) the use of graphic organizers, and (d) structured peer-pairings.
Vaughn et al. (2009) conducted two studies over two consecutive school years (2006~2007 & 2007~2008), using different, non-overlapping samples. In both samples, Vaughn et al. (2009) conducted a randomized controlled trial to evaluate whether the program enhances students' vocabulary knowledge and comprehension. In the first sample, a total of 381 seventh graders from two participating schools were assigned to either treatment (7 classes, 176 students) or control (8 classes, 205 students). In the second sample, a total of 507 seventh graders from two participating schools were randomly assigned to either treatment (9 classes, 273 students) or control (8 classes, 234 students). A pretest and a posttest were assessed before and after the 9-12 week intervention.
In both studies (Vaughn et al., 2009), as compared to students in control, students in treatment earned higher scores in vocabulary knowledge and comprehension.
In both studies (Vaughn et al., 2009), as compared to students in control, students in treatment earned higher scores in:
Not conducted.
In the first study, Vaughn et al. (2009) reported various calculations of effect sizes that ranged from medium to large (g = .50-1.28) depending on whether the effect sizes were estimated using the pooled within-group standard deviation at the pretest or posttest as the measure of standard deviation, and whether adjusted or unadjusted means were included in the calculations. This was consistent with the second study, in which Vaughn et al. (2009) reported small-medium to large effect sizes (g = .33-1.01).
Two studies conducted by the same research team (Vaughn et al., 2009) shared the same limitations:
Blueprints: Inconclusive Evidence
Evaluation Methodology
Design:
Recruitment: Not provided other than all social studies teachers (n=4) were identified by the school principal and selected to participate in the study. These teachers taught both treatment and control social studies classes.
Assignment: A total of 381 seventh graders from two participating schools were assigned to either treatment (7 classes, 176 students) or control (8 classes, 205 students). Teachers' class sections were randomly assigned to conditions.
Attrition: In adding up the sample sizes reported for each outcome in Table 2 (including summing ELL and non-ELL students together) and comparing them to the sample size at assignment (n = 381), the number of students at the pretest was 346 for both comprehension and vocabulary for an overall student attrition rate of 9% at the pretest. In addition, the number of students at the posttest was 334 for both comprehension and vocabulary for an overall attrition rate of 12% at the posttest.
Sample: Students' information was not provided other than that 25% were designated as English language learners. In terms of participating schools, both schools were considered to have a substantial number of English language learners. At both schools, at least 65% were Latino. The proportion of students who qualified for the free or reduced-price lunch program ranged from 70% to 82%.
Measures: Researcher-developed measures were used to assess student outcomes. The measure covered students' understanding of the content taught during the instruction and served as an indicator of growth in social studies learning. It consisted of vocabulary matching items and comprehension questions. Reliability/validity was not reported.
The vocabulary section had 20 items that included definitions that had to be matched with vocabulary terms used within the context of a sentence that contained social studies information.
The comprehension section included 10 questions asking students to identify and explain the big ideas of the social studies units taught during the instruction.
Analysis: All analyses were conducted using three-level, hierarchical linear models in HLM 6.06. For all HLM analyses, Vaughn et al. (2009) reported results for fixed effects of treatment based on robust standard errors. The three-level models included variability because of students within section, sections within teachers, and teachers. In all models, treatment was entered at Level 2 (i.e., at the section level). Random effects due to teachers were limited to effects on the intercept, that is, the average value across all sections for that teacher. Pretest scores were included as covariates.
Intent-to-Treat: It appears that all participants were analyzed according to the condition in which they were assigned and all available data were utilized in the analysis which is in line with intent-to-treat protocol, but this was not explicitly stated.
Outcomes
Implementation Fidelity: Four researchers conducted three observations for treatment and four observations for control over the course of the intervention to ensure fidelity of treatment and to determine if there was any contamination of instruction in the control classes. Observations and teacher reports confirmed that materials and practices designed for treatment classes were not used in the control classes. In total, there were four derived fidelity scores: average quality (0-2), organization (1-4), classroom management (1-4), and global ratings of instruction (1-3). Analyses of group differences on these fidelity ratings (see table 1) indicated a significant difference on ratings of organization between treatment and control, favoring treatment. Vaughn et al. (2009), however, reported "Overall, in both studies, the teachers were rated fairly similarly during treatment and control instruction" (p. 309).
Baseline Equivalence: Baseline equivalence was tested only on pretest scores and there were no significant differences between treatment and control on either outcome measure.
Differential Attrition: Not conducted.
Posttest: At the posttest, Vaughn et al. (2009) found that students in treatment earned higher scores than students in control both in vocabulary and comprehension. For both vocabulary and comprehension, treatment effect did not interact with student status as English language leaner, indicating that English language learners and non-English language learners benefitted from the program equally.
Long-Term: Not conducted.
Evaluation Methodology
Design:
Recruitment: Not provided other than all social studies teachers (n=4) were identified by the school principal and selected to participate in the study. One of the two schools in the first study participated in the second study again.
Assignment: A total of 507 seventh graders from two participating schools were randomly assigned to either treatment (9 classes, 273 students) or control (8 classes, 234 students). Teachers' class sections were randomly assigned to conditions.
Attrition: In adding up the sample sizes reported for each outcome in Table 2 (including summing ELL and non-ELL students together) and comparing them to the sample size at assignment (n = 507), the number of students at the pretest was 414 for both comprehension and vocabulary for an overall student attrition rate of 18% at the pretest. In addition, the number of students at the posttest was 453 for both comprehension and vocabulary for an overall attrition rate of 11% at the posttest.
Sample: Students' information was not provided other than that 21% were English language learners. In terms of participating schools, one school from the first study participated in the second study again. The other school was new to the study and had 51% Latino students. The percentage of students who were eligible for free or reduced-price lunch ranged from 68% ~ 85%.
Measures: Researcher-developed measures were used to assess student outcomes. The measure covered students' understanding of the content taught during the instruction and served as an indicator of growth in social studies learning. It consisted of vocabulary matching items and comprehension questions. Reliability/validity was not reported.
The vocabulary section had 20 items that included definitions that had to be matched with vocabulary terms used within the context of a sentence that contained social studies information.
The comprehension section included 10 questions asking students to identify and explain the big ideas of the social studies units taught during the instruction.
Analysis: All analyses were conducted using three-level, hierarchical linear models in HLM 6.06. For all HLM analyses, Vaughn et al. (2009) reported results for fixed effects of treatment based on robust standard errors. The three-level models included variability because of students within section, sections within teachers, and teachers. In all models, treatment was entered at Level 2 (i.e., at the section level). Random effects due to teachers were limited to effects on the intercept, that is, the average value across all sections for that teacher. Pretest scores were included as covariates.
Intent-to-Treat: It appears that all participants were analyzed according to the condition in which they were assigned and all available data were utilized in the analysis which is in line with intent-to-treat protocol, but this was not explicitly stated.
Outcomes
Implementation Fidelity: Four researchers conducted three observations for treatment and four observations for control over the course of the intervention to ensure fidelity of treatment and to determine if there was any contamination of instruction in the control classes. Observations and teacher reports confirmed that materials and practices designed for treatment classes were not used in the control classes. In total, there were four derived fidelity scores: average quality (0-2), organization (1-4), classroom management (1-4), and global ratings of instruction (1-3). Analyses of group differences on these fidelity ratings (see table 1) indicated no significant differences in average ratings between treatment and control sections in Study 2. Vaughn et al. (2009) also reported "Overall, in both studies, the teachers were rated fairly similarly during treatment and control instruction" (p. 309).
Baseline Equivalence: Baseline equivalence was tested only on pretest scores and there were no significant differences between treatment and control on either outcome measure.
Differential Attrition: Not conducted.
Posttest: Posttest: At the posttest, Vaughn et al. (2009) found that students in treatment earned higher scores than students in control both in vocabulary and comprehension. For both vocabulary and comprehension, treatment effect did not interact with student status as English language leaner, indicating that English language learners and non-English language learners benefitted from the program equally.
Long-Term: Not conducted.