Sign Up For Newsletter

Blueprints For Healthy Youth Development logo

Targeted Reading Intervention

A classroom-based intervention intended to improve reading skills for kindergarten and first graders through individualized instruction.

Program Outcomes

  • Academic Performance
  • Preschool Communication/ Language Development

Program Type

  • Academic Services
  • School - Individual Strategies
  • Teacher Training

Program Setting

  • School

Continuum of Intervention

  • Indicated Prevention

Age

  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • All

Endorsements

Blueprints: Promising

Program Information Contact

Dr. Lynne Vernon-Feagans
University of North Carolina at Chapel Hill
301K Peabody Hall, #3500
Chapel Hill, NC 27599-3500
(919) 843-5623
lynnevf@email.unc.edu
targetedreadingintervention.org

Program Developer/Owner

Dr. Lynn Vernon-Feagans
University of North Carolina at Chapel Hill


Brief Description of the Program

Targeted Reading Intervention has the classroom teacher - rather than a specialized tutor or educator - deliver individualized instruction to struggling readers in regular kindergarten and first-grade classrooms. The instruction takes the form of 15-minute one-on-one instructional sessions in the regular classroom until the child makes rapid progress, and the teacher can go on to instruct another struggling reader. Teachers use a variety of word and comprehension strategies to improve reading and emphasize both identification of letter sounds in words and comprehension of words in text. The program intends primarily to help struggling readers, particularly those in rural, low-wealth communities where teachers have limited access to professional development and students have little access to intervention services. Along with helping struggling readers, the program aims to provide effective professional development for teachers, with the intention to benefit all students in the classroom.

Outcomes

Amendum et al. (2011)

  • Struggling readers in the experimental schools had higher posttest scores than struggling readers in the control schools on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.
  • Both struggling and non-struggling readers in the experimental schools had higher posttest scores than struggling and non-struggling readers in the control schools on three of the four reading measures: Word Attack, Word-Letter Identification, and Passage Comprehension.

Vernon-Feagans et al. (2009, 2010)

  • Both struggling and non-struggling readers in the experimental schools had higher posttest scores than struggling and non-struggling readers in the control schools on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.
  • In some cases, struggling readers improved enough to catch-up with their non-struggling peers.

Vernon-Feagans et al. (2018)

As compared to control students, students in the treatment classrooms showed significant improvements in:

  • Letter-word identification
  • Word attack
  • Spelling of sounds
  • Passage comprehension

Brief Evaluation Methodology

Most studies of Targeted Reading Intervention randomly assigned a small number of schools (from 4 to 16) to an experimental condition implementing the reading program and a control condition teaching reading as usual. The schools came from rural disadvantaged areas of the southeastern and southwestern United States. Within each kindergarten and first-grade classroom in the experimental and control schools, five struggling readers and five non-struggling readers were selected. These students were assessed in the fall at the beginning of the school year and in the spring near the end of the school year. Analyses compared experimental and control schools on measures of reading outcomes in the spring for both struggling and non-struggling readers.

In Study 5 (Vernon-Feagans et al., 2018), assignment was conducted at the classroom level. The authors randomized 100 classrooms in 10 schools. Within each classroom, students found to be at risk of struggling with reading during beginning-of-year screenings were included in the sample (N = 556). Results for measures of literacy skills were compared for the treatment and control groups after one school year of implementation.

Study 1

Amendum, S. J., Vernon-Feagans, L. V., & Ginsberg, M. C. (2011). The effectiveness of a technologically facilitated classroom-based early reading intervention. The Elementary School Journal, 112(1), 107-131.


Risk Factors

School: Poor academic performance

Protective Factors

School: Instructional Practice, Opportunities for prosocial involvement in education


* Risk/Protective Factor was significantly impacted by the program

See also: Targeted Reading Intervention Logic Model (PDF)

Race/Ethnicity/Gender Details

The treatment proved similarly effective for demographic subgroups.

Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.

Start-Up Costs

Initial Training and Technical Assistance

Entire school systems are encouraged to implement Targeted Reading Intervention. The program begins with attendance at a 3-day summer institute by all involved teachers. The cost of the institute ranges from $15,000 to $20,000.

Curriculum and Materials

Materials cost approximately $400 per classroom. Webcam capacity is needed for each classroom.

Licensing

None required.

Other Start-Up Costs

Computer with web access if not already available in the classroom.

Intervention Implementation Costs

Ongoing Curriculum and Materials

Once initial start-up investment is made, replacement costs are minimal.

Staffing

The program is implemented by classroom teachers during the regular school day.

Other Implementation Costs

None.

Implementation Support and Fidelity Monitoring Costs

Ongoing Training and Technical Assistance

The program consists primarily of ongoing coaching offered remotely by graduate students working under the supervision of the developer. The estimated cost of this support is $5,000 to $7,500 per teacher per year for the first two years. After the first two years, training and coaching costs would reduce to between $15,000 and $30,000 per district per year, depending on size, location and ongoing need.

Fidelity Monitoring and Evaluation

Initial fidelity monitoring is included in the coaching cost. After the second year, periodic fidelity visits are recommended at a cost of $500 per day plus travel. Three visits per year per district are recommended.

Ongoing License Fees

None.

Other Implementation Support and Fidelity Monitoring Costs

No information is available

Other Cost Considerations

The size of the implementation can result in scale savings on some costs.

Year One Cost Example

A first year implementation for a school district training 50 kindergarten, first and second grade teachers serving 250 struggling readers could expect to incur the following costs:

Summer Institute $20,000.00
Materials @ $400 per classroom $20,000.00
Coaching and support @ $7500 per classroom $375,000.00
Total One Year Cost $415,000.00

If the program directly reached 250 struggling students during the year, the cost per student would be $1,660.

Funding Overview

When Targeted Reading Intervention decides to widely disseminate the program by charging school districts for its implementation, the developer expects Title I to be the primary source of funding. In addition, as a curriculum enhancement, state and local funding for both curriculum and teacher training could be accessed to support implementation. However, with the high initial investment that will be required, support from foundations and the local business community might be needed for the initial training and coaching costs.

Funding Strategies

Improving the Use of Existing Public Funds

No information is available

Allocating State or Local General Funds

State and local school system funding could be used to implement and especially sustain Targeted Reading Intervention. Curriculum and teacher training funds should be considered. State administered Race to the Top funding should also be explored.

Maximizing Federal Funds

Formula Funds: Title I would be the primary federal Education Department source of funding given that the intervention is targeted at low wealth rural schools with large numbers of underachieving reading students. Rural supplements, special education and RTI funding from federal sources should also be explored.

Discretionary Grants: Discretionary grants targeting rural schools, innovations in education and special education are among many offered by the Department of Education that might provide support for Targeted Reading Intervention.

Foundation Grants and Public-Private Partnerships

Foundations will be an important source of start-up support for Targeted Reading Intervention. For example, the Rural Schools and Community Trust channels funding from the Crane, Pew, and Kellogg Foundations, which might be available to school districts wanting to implement Targeted Reading Intervention.

Debt Financing

No information is available

Generating New Revenue

No information is available

Data Sources

All information comes from the responses to a questionnaire submitted by the developer of Targeted Reading Intervention, Dr. Lynne Vernon-Feagans, to the Annie E. Casey Foundation.

Program Developer/Owner

Dr. Lynn Vernon-FeagansUniversity of North Carolina at Chapel HillSchool of EducationFrank Porter Graham Child Development InstituteFrank Porter Graham Child Development InstituteChapel HillChapel Hill27599-3500USA(919) 966-5484(919) 843-5623(919) 962-1533lynnevf@email.unc.edu targetedreadingintervention.org

Program Outcomes

  • Academic Performance
  • Preschool Communication/ Language Development

Program Specifics

Program Type

  • Academic Services
  • School - Individual Strategies
  • Teacher Training

Program Setting

  • School

Continuum of Intervention

  • Indicated Prevention

Program Goals

A classroom-based intervention intended to improve reading skills for kindergarten and first graders through individualized instruction.

Population Demographics

The early grades of elementary school children (K-1 grades).

Target Population

Age

  • Late Childhood (5-11) - K/Elementary

Gender

  • Both

Race/Ethnicity

  • All

Race/Ethnicity/Gender Details

The treatment proved similarly effective for demographic subgroups.

Other Risk and Protective Factors

The risk factors or factors to be changed by the program relate to individualized instruction for the student and professional development for the classroom teacher.

Risk/Protective Factor Domain

  • School

Risk/Protective Factors

Risk Factors

School: Poor academic performance

Protective Factors

School: Instructional Practice, Opportunities for prosocial involvement in education


*Risk/Protective Factor was significantly impacted by the program

See also: Targeted Reading Intervention Logic Model (PDF)

Brief Description of the Program

Targeted Reading Intervention has the classroom teacher - rather than a specialized tutor or educator - deliver individualized instruction to struggling readers in regular kindergarten and first-grade classrooms. The instruction takes the form of 15-minute one-on-one instructional sessions in the regular classroom until the child makes rapid progress, and the teacher can go on to instruct another struggling reader. Teachers use a variety of word and comprehension strategies to improve reading and emphasize both identification of letter sounds in words and comprehension of words in text. The program intends primarily to help struggling readers, particularly those in rural, low-wealth communities where teachers have limited access to professional development and students have little access to intervention services. Along with helping struggling readers, the program aims to provide effective professional development for teachers, with the intention to benefit all students in the classroom.

Description of the Program

The program has the classroom teacher - rather than a specialized tutor or educator - deliver individualized instruction to struggling readers in regular kindergarten and first-grade classrooms. The instruction takes the form of 15-minute sessions offered daily over the course of the school year. Teachers use a variety of word strategies to improve reading and emphasize both identification of letter sounds in words and comprehension of words in text. The program intends to help struggling readers, particularly those in rural, low-wealth communities where teachers have limited access to professional development and students have little access to intervention services. Along with helping struggling readers, the program aims to provide effective professional development for teachers, with the intention to benefit all students in the classroom.

Five elements were identified in the literature as effective learning strategies for struggling readers: 1) explicit instruction in the alphabetic principle combined with reading for meaning, 2) intervention in the first few grades of elementary school, 3) small-group or one-on-one intensive instruction, 4) an effective emotional and cognitive relationship with the teacher, and 5) individualized instruction that matches the child's level of skill and particular needs.

Each 15-minute session includes the following: 1) re-reading in which the teacher asks the student to re-read material to develop fluency, 2) word work in which the teacher uses multi-sensory strategies to help the child manipulate, say, and write words, and 3) guided oral reading in which the teacher selects a reading appropriate for the student's level and allows the child to summarize what they have read and answer questions about the story.

The program also aims to encourage the professional development of teachers by having them work individually with students and receive coaching from program consultants about their work with students. Weekly, each program teacher uses a laptop computer with a webcam in the classroom so that the program consultant can see the teacher working with the student and give real-time feedback. In addition, weekly team meetings are conducted via webcam with the program consultant and program teachers to deliver ongoing professional development and problem-solving for particular children.

Theoretical Rationale

A transactional theory of development posits that teachers and students (or adults and children more generally) influence each other in dynamic cycles that over time can lead to positive or negative outcomes. Teachers change by actually experiencing the gains that children are making in reading. Positive experiences with the student, in turn, enhance teacher efficacy and transforms the way s/he teaches reading to all children. Such outcomes are enhanced by personalized interaction with an individual student or a small group of students. Thus, professional development of teachers in reading development and individualized instruction provides the means to transform teacher instruction.

Theoretical Orientation

  • Skill Oriented

Brief Evaluation Methodology

Most studies of Targeted Reading Intervention randomly assigned a small number of schools (from 4 to 16) to an experimental condition implementing the reading program and a control condition teaching reading as usual. The schools came from rural disadvantaged areas of the southeastern and southwestern United States. Within each kindergarten and first-grade classroom in the experimental and control schools, five struggling readers and five non-struggling readers were selected. These students were assessed in the fall at the beginning of the school year and in the spring near the end of the school year. Analyses compared experimental and control schools on measures of reading outcomes in the spring for both struggling and non-struggling readers.

In Study 5 (Vernon-Feagans et al., 2018), assignment was conducted at the classroom level. The authors randomized 100 classrooms in 10 schools. Within each classroom, students found to be at risk of struggling with reading during beginning-of-year screenings were included in the sample (N = 556). Results for measures of literacy skills were compared for the treatment and control groups after one school year of implementation.

Outcomes (Brief, over all studies)

Kindergarten and first-grade students participating in the program intervention showed greater improvement on several reading measures than students in the control schools. For struggling and non-struggling readers whose teachers participated in the program, struggling readers generally, but not consistently, showed stronger gains. In a later study focused on students at risk of struggling with reading (Vernon-Feagans et al., 2018), students in treatment classrooms showed significantly higher literacy skills than their control group counterparts as determined by four subscores of the Woodcock-Johnson III test.

Outcomes

Amendum et al. (2011)

  • Struggling readers in the experimental schools had higher posttest scores than struggling readers in the control schools on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.
  • Both struggling and non-struggling readers in the experimental schools had higher posttest scores than struggling and non-struggling readers in the control schools on three of the four reading measures: Word Attack, Word-Letter Identification, and Passage Comprehension.

Vernon-Feagans et al. (2009, 2010)

  • Both struggling and non-struggling readers in the experimental schools had higher posttest scores than struggling and non-struggling readers in the control schools on all four Woodcock Johnson reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds.
  • In some cases, struggling readers improved enough to catch-up with their non-struggling peers.

Vernon-Feagans et al. (2018)

As compared to control students, students in the treatment classrooms showed significant improvements in:

  • Letter-word identification
  • Word attack
  • Spelling of sounds
  • Passage comprehension

Mediating Effects

None reported.

Effect Size

Averaged across studies, effect sizes of the program fell in the medium range. They ranged from a low of .31 to a high of .72 in the main study (Amendum et al., 2011), from .41 to .57 in Study 2 (Vernon-Feagans et al., 2009), from .34 to .72 in Study 3 (Vernon-Feagans et al., 2009), and from .38 to .61 in Study 4 (Vernon-Feagans et al., 2010). Hedge's g effect sizes ranged from 0.16 to 0.28 in Study 5 (Vernon-Feagans et al., 2018).

Generalizability

The samples came from rural schools in the southeastern and southwestern United States that received Title I funding and had a largely disadvantaged student population. The results did not vary by gender or race/ethnicity, but the program may not generalize to urban schools, more advantaged children, and other regions of the country. One study (Vernon-Feagans et al., 2018) focused on children determined to be at risk of struggling with reading during screenings at the beginning of the academic year.

Potential Limitations

The main study (Amendum et al., 2011) had a strong design but still faced some limitations:

  • Baseline equivalence was not fully established.
  • One experimental school that withdrew after assignment because of problems with technology should have, if at all possible, been included in the assessments and analysis to meet intent-to-treat requirements.
  • The analysis, while adjusting for clustering within schools and classrooms, treats condition membership as an individual-level variable when randomization was done at the school level.
  • No long-term effects were examined.

Problems in the design and analysis in the other studies (Vernon-Feagans et al., 2009, 2010) are more serious, in large part because the papers did not discuss some key methodological issues.

  • Baseline equivalence was not examined.
  • Differential attrition was not examined (although imputation of missing data may counter any bias).
  • The score on a measure of intervention duration and quality was not high (only 3 on a scale from 1 to 5).
  • The analysis, while adjusting for clustering within schools and classrooms, treats condition membership as an individual-level variable when randomization was done at the school level.
  • The analyses did not control for baseline outcomes.
  • No long-term effects were examined.

A study using randomization at the classroom level addressed some limitations of previous studies (Study 5, Vernon-Feagans, 2018). Limitations included:

  • No formal tests for differential or baseline by condition attrition
  • No long-term effects were examined

Endorsements

Blueprints: Promising

Peer Implementation Sites

Bearfield Primary School
145 Herford County High School Road
Ahoskie, NC 27910
(252) 209-6140
Contact: Principal Julie Shields

Program Information Contact

Dr. Lynne Vernon-Feagans
University of North Carolina at Chapel Hill
301K Peabody Hall, #3500
Chapel Hill, NC 27599-3500
(919) 843-5623
lynnevf@email.unc.edu
targetedreadingintervention.org

References

Study 1

Certified Amendum, S. J., Vernon-Feagans, L. V., & Ginsberg, M. C. (2011). The effectiveness of a technologically facilitated classroom-based early reading intervention. The Elementary School Journal, 112(1), 107-131.

Study 2

Vernon-Feagans, L., Amendum, S., Kainz, K., Ginsberg, M., Wood, T., & Bock, A. (2009). The Targeted Reading Intervention (TRI): A classroom teacher tier 2 intervention to help struggling readers in early elementary school. In Evidence for Interventions for Struggling Readers. Symposium conducted at the annual meeting of the Society for Research on Educational Effectiveness, Washington, D.C.

Study 3

Vernon-Feagans, L., Amendum, S., Kainz, K., Ginsberg, M., Wood, T., & Bock, A. (2009). The Targeted Reading Intervention (TRI): A classroom teacher tier 2 intervention to help struggling readers in early elementary school. In Evidence for Interventions for Struggling Readers. Symposium conducted at the annual meeting of the Society for Research on Educational Effectiveness, Washington, D.C.

Study 4

Vernon-Feagans, L., Kainz, K., Hedrick, A., Ginsberg, M., & Amendum, S. (In press). The Targeted Reading Intervention: A classroom teacher professional development program to promote effective teaching for struggling readers in kindergarten and first grade. The Journal of Educational Psychology.

Study 5

Vernon-Feagans, L., Bratsch-Hines, M., Varghese, C., Cutrer, E. A., & Garwood, J. D. (2018). Improving struggling readers' early literacy skills through a tier 2 professional development program for rural classroom teachers: The Targeted Reading Intervention. The Elementary School Journal, 118(4), 525-548.

Study 1

Evaluation Methodology

Design: The study used a cluster randomized design of eight schools from five school districts in the southwestern United States. Schools were matched and paired before randomization, but one school withdrew after assignment to the experimental group because of problems using the program technology. The seven remaining schools included 26 experimental and 17 control classrooms at kindergarten and first-grade levels. The study offered no information on selection of the schools for participation or the representativeness of rural schools.

Within the classrooms, students with severe disability or unable to speak at least some conversational English were excluded. Of the remaining students, teachers and program consultants used assessments of reading skills and knowledge of the child's progress in school to rank each child from lowest to highest performance. In both the experimental and control classrooms, five children reading below grade level (i.e., struggling readers) were randomly selected for study, as were five children reading above grade level (i.e., non-struggling readers). These two groups together included 10 students per classroom.

The combination of experimental and control schools with struggling and non-struggling readers defined four conditions: 1) struggling readers in experimental schools (n = 112), 2) non-struggling readers in experimental schools (n = 125), 3) struggling readers in control schools (n = 63), and 4) non-struggling readers in control schools (n = 64). Only struggling students in the experimental schools received the treatment, but the professional development of teachers giving the instruction should also help non-struggling readers in the experimental schools.

Baseline assessment occurred in the fall of the school year, and post-intervention assessment occurred in the following spring. Attrition was low. With 364 students having scores at baseline, the sample dropped to 350 (96%) at posttest. The experimental school that withdrew early (but after assignment) because of problems with technology was not included in the assessments and analysis.

Sample Characteristics: All schools received Title I funding. Minority students comprised 57-98% of the students in the schools and most students (57-78%) were eligible for free or reduced-price lunch.

Measures: The key outcome measures came from four subsets of the Woodcock-Johnson Test of Achievement III battery:

  1. Word Attack to measure skills in applying phonetic analysis to pronouncing unfamiliar words,
  2. Letter-Word Identification to measure letter and word identification skills,
  3. Passage Comprehension to measure understanding, and
  4. Spelling of Sounds to measure spelling ability.

Reliabilities for the four scales ranged from .74 to .91.

The Peabody Picture Vocabulary Test - Third Edition, which served as a predictor rather than outcome, measures vocabulary knowledge and has an alpha reliability for elementary students of .92-.95.

Analysis: In an intent-to-treat analysis, multiple imputation was used to deal with missing data. ANCOVAs included sociodemographic variables, the Peabody Picture Vocabulary Test scores, treatment condition, and an interaction between treatment condition and the Peabody Picture Vocabulary Test scores. The models also treated pretest outcome scores as covariates. Planned contrasts were used to make comparisons across subsets of the four conditions.

The authors state that all estimates came from hierarchical linear models that accounted for the nested structure of the data, estimating random intercepts for classrooms and allowing for school-specific variation between classrooms within schools. This procedure dealt with clustering effects both for schools and classrooms. However, it appeared that, outside adjustment of the standard errors for clustering, the randomization variable distinguishing between experimental and control schools was measured at the individual level. Based on a check of the t-distribution, the reported probabilities associated with particular t-values must have used degrees of freedom roughly equal to the number of students (364) rather than the number of schools (7). This may have biased tests in a way that overstated the significance of program effects.

Outcomes

Implementation Fidelity: The teachers participated in four professional-development activities that contributed to program fidelity: 1) a summer institute, 2) weekly or biweekly coaching sessions via the webcam, 3) weekly meetings via webcam about the reading progress of individual children, and 4) monthly or bimonthly webcam meetings to discuss strategies and content for improving the program.

Quantitative measures assessing the duration and quality of program practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped teachers' use of the tools and faithfulness to program strategies. With the two measures averaged into one, the mean on a scale from 1 to 5 equaled 3.41.

Baseline Equivalence: The study compared all participants (struggling and non-struggling readers) across the experimental and control schools and compared all participants across the experimental and control schools. The comparisons of struggling readers revealed significant differences by race, with white students more likely to attend the experimental schools. The models therefore controlled for race. The comparisons of schools revealed that the experimental schools scored significantly higher at baseline on the Letter-Word Identification scores and Spelling of Sounds scores and marginally significantly higher on Word Attack (p = .07) and Passage Comprehension (p =.13). These differences indicate a pattern of non-equivalence. With only seven schools, deviations from randomness can be expected, and the study controlled for baseline outcomes. Regardless, randomization may not have worked well with the small number of schools.

Differential Attrition: None reported, but the posttest lost only 4% of participants, and missing data were imputed for the small number of students not present for the spring assessment.

Posttest: The struggling readers did better at posttest in the experimental schools than in the control schools on all four reading measures: Word Attack, Word-Letter Identification, Passage Comprehension, and Spelling of Sounds. Effect sizes ranged from .40 to .72. For both struggling and non-struggling readers combined, the experimental schools did better than the control schools on three of the four measures: Word Attack, Word-Letter Identification, and Passage Comprehension. Effect sizes ranged from .31 to .61. The effects for all readers suggest that non-struggling readers benefitted indirectly from the program. However, the program worked no better for students scoring low at baseline on the Peabody Picture Vocabulary Test than those scoring high.

Longterm: The study did not gather follow-up data or examine sustained effects.

Study 2

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization at the school level. Six elementary schools in two poor rural counties in the southeastern United States participated in the study. Before randomization, the six schools were divided into three pairs based on matching characteristics of size, percent free and reduced-price lunch, percent minority, and involvement in the Reading First program. Random assignment selected one school out of each pair for the experimental condition and one school out of each pair for the control condition. No information was provided on how the schools were selected for participation or the representativeness of rural schools more generally.

Within each of the schools, the study examined kindergarten and first grade classrooms. For the six schools, there were 14 experimental classrooms, and 18 control classrooms.

In each classroom, teachers used assessment data and their judgment to identify children whose reading skills were below and above grade level. The study then selected five struggling readers and five non-struggling readers in each experimental and control classroom. Struggling readers in the experimental classrooms received the Targeted Reading Intervention, while other groups received reading instruction as usual. The total number of participants was 276.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school years was not discussed. The pretest occurred in the fall and the posttest occurred in the spring of the study year.

All schools and classrooms participated throughout the study. Of the 276 participants, 270 (98%) had baseline scores and 252 (91%) had posttest scores on key measures.

Sample Characteristics: The sampled students were diverse but generally disadvantaged. Approximately 60% of the students were African American and less than 10% of the mothers had a college degree. Most of the students were eligible for free or reduced-price lunch.

Measures: The key outcome measures came from two subsets of the Woodcock-Johnson Test of Achievement III battery: Word Attack to measure skills in applying phonetic analysis to pronouncing unfamiliar words and Letter-Word Identification to measure letter and word identification skills.

Two other subtests from the Comprehensive Test of Phonological Processing served as predictors of the outcomes: phonological awareness and rapid color naming.

Analysis: The analyses used multiple imputation techniques, under the assumption that data were missing at random, to replace missing data. Random intercept regression models accounted for clustering within classrooms and schools but, as described in Study 1, the analysis was still conducted at the individual level. Perhaps most problematic, the analysis does not mention controlling for baseline outcomes.

Outcomes

Implementation Fidelity: Teachers implementing TRI met face-to-face over three days in a summer institute and met in-person biweekly with TRI consultants during the academic year.

Quantitative measures assessing the duration and quality of TRI practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped the teachers' use of the tools and faithfulness to TRI strategies. The two scales were averaged into one. The average score of 3 (on a scale of 1 to 5) showed mixed success in implementation of the program.

Baseline Equivalence: Not examined.

Differential Attrition: Not examined, although multiple imputation of missing data might counter bias due to attrition.

Posttest: Struggling and non-struggling readers in the experimental schools scored significantly higher than struggling and non-struggling readers in the control schools on the two achievement test measures of Word Attack (d=.41) and Letter-Word Identification (d=.50). For struggling readers, the experimental group scored significantly higher than the control group on Letter-Word Identification (d=.57), but not on Word Attack.

Another test indicated that struggling readers who scored low on the Rapid Color Naming and Phonological Awareness measures gained the most from the intervention on Word Attack but not Letter-Word Identification.

Overall, the evidence across both outcome variables unexpectedly showed more consistent benefits of the intervention for all students than for struggling readers.

Longterm: The study did not gather follow-up data or examine sustained effects.

Effect Size: Effect sizes of .41 to .57 fall in the medium range.

Generalizability: The sample of disadvantaged rural schools in the southeastern United States limits the generalizability to more advantaged and urban schools in other parts of the country.

Limitations: Problems in the design and analysis are serious, in large part because the paper did not discuss some key methodological issues.

  • Baseline equivalence was not examined.
  • Differential attrition was not examined (although imputation of missing data may counter bias).
  • The score on the measure of intervention duration and quality was not high (only 3 on a scale from 1 to 5).
  • The analysis, while adjusting for clustering within schools and classrooms, was done at the individual level when randomization was done at the school level.
  • The analysis did not control for baseline outcomes.

Study 3

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization done at the school level. Four elementary schools in rural counties in Texas and New Mexico were selected to participate. Before randomization, the four schools were divided into two pairs based on matching characteristics of size, percent free and reduced-price lunch, percent minority, and involvement in the Reading First program. Random assignment selected one school out of each pair for the experimental condition and one school out of each pair for the control condition. No information was provided on how the schools were selected for participation or the representativeness of rural schools more generally. Within each of the schools, the study examined kindergarten and first grade classrooms. For the four schools, there were 26 experimental classrooms and 17 control classrooms.

In each classroom, teachers used assessment data and their judgment to identify children whose reading skills were below and above grade level. The study then selected five struggling readers and five non-struggling readers in each experimental and control classroom. Both struggling and non-struggling readers in the experimental classrooms received the Targeted Reading Intervention, while both groups of readers in the control schools received reading instruction as usual. The total number of participants was 359.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school years is not discussed. The pretest occurred in the fall and the posttest occurred in the spring of the school year.

All schools and classrooms participated throughout the study. The study mentioned missing data but gives no specific figures on students who left the study or failed to complete pretest and posttest measures.

Sample Characteristics: The sampled students were diverse but generally disadvantaged. About one-third of the students were white, 25% African American, and 35% other. Additionally, less than 15% of the mothers had a college degree. Most of the students were eligible for free or reduced-price lunch.

Measures: The key outcome measures came from four subtests of the Woodcock-Johnson Test of Achievement III battery: Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds. In addition, Peabody Picture Vocabulary Test (3rd Edition) was used as an outcome measure.

Analysis: The analyses used multiple imputation techniques, under the assumption that data were missing at random, to replace missing data. Random intercept regression models accounted for clustering within classrooms and schools but, as described in Study 1, the analysis was still conducted at the individual level. Perhaps most problematic, the analysis did not appear to control for baseline outcomes.

Outcomes

Implementation Fidelity. As in Study 2, teachers implementing TRI met face-to-face over three days in a summer institute. They also met with program consultants during the academic year but did so via a webcam rather than in person.

Quantitative measures assessing the duration and quality of TRI practices came from reports by teachers when meeting with program consultants. The duration measure tapped the total number of weeks spent with an experimental child. The quality measure tapped the teachers' use of the tools and faithfulness to TRI strategies. The two scales were averaged into one. The average score of 3 (on a scale of 1 to 5) showed mixed success in implementation of the program.

Baseline Equivalence. Not examined.

Differential Attrition. Not examined, although multiple imputation of missing data might counter bias due to attrition.

Posttest: The results showed program benefits for struggling and non-struggling readers on the four outcome variables (Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds). For both struggling and non-struggling readers, the experimental group scored significantly higher than the control group on three of four measures: Word Attack (d=.35), Letter-Word Identification (d=.34), and Passage Comprehension (d=.61). For struggling readers, the experimental group scored significantly higher than the control group on all four measures: Word Attack (d=.52), Letter-Word Identification (d=.52), and Passage Comprehension (d=.72), and Spelling of Sounds (d=.40). However, no evidence emerged that the benefits of the intervention were higher for struggling readers with lower scores on the Peabody Picture Vocabulary.

Longterm: The study did not gather follow-up data or examine sustained effects.

Effect Size: Effect sizes of .34 to .72 fall in the medium range.

Generalizability: The sample of disadvantaged rural schools in the southwestern United States limits the generalizability to more advantaged and urban schools in other parts of the country.

Limitations

Problems in the design and analysis are serious, in large part because the paper did not discuss some key methodological issues.

  • Baseline equivalence was not examined.
  • Differential attrition was not examined (although imputation of missing data may counter any bias).
  • The score on the measure of intervention duration and quality was not high (only 3 on a scale from 1 to 5).
  • The analysis, while adjusting for clustering within schools and classrooms, was done at the individual level when randomization was done at the school level.
  • The analysis did not control for baseline outcomes.

Study 4

Evaluation Methodology

Design: The study used a cluster randomized control design with randomization at the school level. Sixteen elementary schools in five rural counties participated in the study. Before randomization, the sixteen schools were divided into eight pairs based on matching characteristics of size, percent minority, percent free and reduced-price lunch, and involvement in the Reading First program. All the schools received Title I funding. Prior to randomization, one school withdrew because of difficulties with the technology required to effectively implement the intervention. Within each of the 15 schools (75 classrooms), the study examined kindergarten and first grade classrooms.

Within each kindergarten and first-grade classroom, teachers with the aid of the program consultants used mandated state assessment data and classroom performance during the beginning of the school year to identify children as struggling or non-struggling readers. Five struggling and five non-struggling readers were then randomly selected in each experimental and control classroom. The combination of experimental and control schools with struggling and non-struggling readers defined four conditions: 1) struggling readers in experimental schools (n = 194), 2) non-struggling readers in experimental schools (n = 206), 3) struggling readers in control schools (n = 116), and 4) non-struggling readers in control schools (n = 132). Only struggling readers in the experimental schools received the intervention, but non-struggling readers in the experimental schools may benefit as well from the professional development of teachers implementing the program. The total number of participants was 648.

The program was delivered to struggling readers in one-on-one (or sometimes small group) sessions by the classroom teacher within the regular classroom. Each session lasted about 15 minutes. The number of sessions per week or school year was not discussed. The pretest occurred in the fall and posttest occurred in the spring of the study year.

All but one school participated throughout the study. Of the 648 students, some baseline measures contained missing data (for example, one measure had data on 578 or 89% of the students). The study did not report the sample size at posttest or give figures on attrition.

Sample Characteristics: The sampled students were diverse, but generally disadvantaged. About 50% of the children were from minority backgrounds, and half were boys. Maternal education was generally low as the average was a high school degree with just a few mothers with a college degree.

Measures: The key outcome measures came from four subtests of the Woodcock-Johnson Tests of Achievement III battery: Word Attack, Letter-Word Identification, Passage Comprehension, and Spelling of Sounds. In addition, Peabody Picture Vocabulary Test (3rd Edition) was used as an outcome measure.

Analysis: Multilevel ANCOVA models with imputed missing data treated students as nested within classrooms (the school level added no significant variation and was not used). As described in Study 1, however, the analysis appeared to have been conducted at the individual level. The models controlled for baseline outcome scores and sociodemographic characteristics as covariates.

Outcomes

Implementation Fidelity. Teacher fidelity was assessed through weekly online reporting procedures and ratings of quality done over the course of the year by consultants. Fidelity was at or above 80% for all teachers.

Baseline Equivalence. Not examined.

Differential Attrition. Not examined.

Posttest. The key hypothesis of this study compared the struggling readers in the experimental classrooms with the struggling readers in the control classrooms. Controlling for baseline scores, struggling readers in the experimental schools scored significantly higher than struggling readers in the control schools on four of the five achievement test measures: Word Attack (d=.38), Letter-Word Identification (d=.61), Passage Comprehension (d=.58), and Spelling of Sounds (d=.42). Significant differences were not found for the Peabody Picture Vocabulary Test.

A secondary hypothesis concerned the narrowing of differences between the struggling and non-struggling readers in the experimental classrooms. In support of the hypothesis and the effectiveness of the program, differences between the two groups did not reach significance at posttest for three of the five measures. In contrast, differences between struggling and non-struggling readers in the control schools remained significant for four of five measures.

Longterm: The study did not gather follow-up data or examine sustained effects.

Effect Size: Effect sizes of .38 to .61 fall in the medium range.

Generalizability: The sample of disadvantaged rural schools in an unidentified part of the country limits the generalizability to more advantaged and urban schools in other parts of the country.

Limitations

  • The study failed to present information on the analysis sample size, baseline equivalence, and differential attrition.
  • The analysis, while adjusting for clustering within schools and classrooms, was done at the individual level when randomization was done at the school level.

Study 5

Evaluation Methodology

Design:

Recruitment: The researchers selected 10 low-income schools in three school districts in a rural part of the southeastern United States. All schools received Title I funding and had a majority of students eligible for free or reduced price lunch. The study included all kindergarten and first grade classrooms in these schools. Within those classrooms, they selected students who were classified as struggling readers through age-appropriate screenings including the AimsWeb, Dynamic Indicators of Basic Early Literacy Skills (DIBELS), and Woodcock-Johnson III. They attempted to include about 3 struggling readers per classroom in each of the two years. In other words, the study focused on the subset of students in these classrooms that were most at risk with regards to reading difficulties. Recruited teachers participated for 2 years whenever possible, but their students were followed for just one academic year.

Assignment: The study used cluster randomization at the classroom level. Over the 3 years of the study, 50 kindergarten and 50 first grade classrooms were randomly assigned to either a treatment or control condition. Of the kindergarten classrooms, 26 were assigned to treatment and 24 were assigned to control. Of the first-grade classrooms, 29 were assigned to treatment and 21 were assigned to control.

The assignment resulted in a total of 67 teachers participating in treatment classrooms and 52 teachers participating in the control group. Note that teachers leaving the schools were replaced, resulting in more teachers than classrooms. The total number of struggling readers identified and included in these classrooms was 305 in the treatment condition and 251 in the control group. In the treatment classrooms, teachers received the professional development and implemented the program with the learners identified as needing reading support. Control group participants continues with business as usual.

Attrition: Assessments occurred at baseline in the fall of the school year and at posttest in the spring of the same school year. About 18% of teachers (22% of the treatment teachers and 15% of the control group teachers) left the study between the first and second year of their participation. The authors stated that they left for reasons unrelated to the study. Replacement teachers were trained as needed and maintained in the corresponding condition. Overall student attrition was at 7.2%, or 6.6% of treatment students and 8% of control group students.

Sample: The sample was roughly divided between kindergarten and first grade students and was roughly 55% male. About 54% of the students were Black, 25% were White, and 16% were Hispanic. The authors also provided information on the sample of teachers in the study. Nearly all were female and around 80% were White. The majority had an elementary education certification, and nearly a third had a master's degree. They averaged roughly 9 years of teaching experience.

Measures: To assess students' literacy outcomes, the researchers used four subtests of the Woodcock-Johnson III: letter-word identification, word attack, spelling of sounds, and passage comprehension. The authors cite reliability estimates for these sub-scores from other sources ranging from 0.81 to 0.98 for children from 5 to 7 years of age. Trained graduate students did the assessments, but it was not clear if they were aware of student assignments.

Analysis: Given clustering of students in classrooms and schools, the researchers used 3-level hierarchical linear models (student, classrooms, schools) to estimate program effects. Each of the four outcome measures was examined in a separate analysis with baseline test scores and demographic variables included as covariates. To address concerns over attrition, multiple imputation was used with 20 datasets.

Intent-to-Treat: The researchers used all randomized students in the analyses.

Outcomes

Implementation Fidelity: Research assistants coded a random sample of two videos for each struggling reader in order to assess adherence to and quality of program implementation. About 15% of videos were double coded, resulting in a Cohen's kappa of 0.78 for adherence and 0.84 for quality. Coders indicated that adherence to program activities and strategies was at about 84% and that there was high quality of implementation in 61% of the sessions observed. There was also some evidence that implementation quality improved significantly from the first to the second year of the study. However, dosage was limited. Teachers reported an average of 17 one-on-one sessions with each child in the treatment condition, when the goal had been daily sessions for 6-8 weeks.

Baseline Equivalence: The researchers found no significant demographic differences between the treatment and control condition participants, with tests conducted for both teachers and participating students. They also found no differences in baseline student outcome measures between treatment and control children.

Differential Attrition: Overall student attrition was at 7.2% over the school year of their participation, or 6.6% of the treatment students and 8.0% of the control students. Given the similar attrition rates across conditions, the authors did not present formal tests for differential attrition.

Posttest: After one school year of implementation, as compared to the control group, students in treatment classrooms scored significantly higher on all four outcome measures of letter-word identification, word attack, spelling of sounds, and passage comprehension.

Long-Term: Not included

Contact

Blueprints for Healthy Youth Development
University of Colorado Boulder
Institute of Behavioral Science
UCB 483, Boulder, CO 80309

Email: blueprints@colorado.edu

Sign up for Newsletter

If you are interested in staying connected with the work conducted by Blueprints, please share your email to receive quarterly updates.

Blueprints for Healthy Youth Development is
currently funded by Arnold Ventures (formerly the Laura and John Arnold Foundation) and historically has received funding from the Annie E. Casey Foundation and the Office of Juvenile Justice and Delinquency Prevention.