An Early Childhood (preschool and kindergarten) program designed to promote academic learning and prosocial behaviors by supporting the development of executive function and other regulation-related skills.
Blueprints: Promising
Tools of the Mind
Email: information@toolsofthemind.org
Website: www.toolsofthemind.org
Deborah Leong, Ph.D.
Tools of the Mind
Tools of the Mind (Tools) is an early childhood curriculum and a professional development program that supports children's cognitive, social, and emotional development with a special emphasis on the development of executive function and self-regulation. At the heart of Tools curriculum are instructional strategies designed to promote the development of mature make-believe play as a primary context for the development of self-regulation in preschool- and kindergarten-aged children. Teachers participate in professional development, including training workshops and regular contact with Tools staff over the course of the Core Year 1 and optional Years 2 and 3.
Tools of the Mind (Tools) is an early childhood curriculum and a professional development program that is based on Vygotskian theory and developmental science and is informed by neuroscience research. Tools offers a comprehensive PreK & K curriculum that supports children's cognitive, social, and emotional development with a special emphasis on the development of executive function and self-regulation. In Tools classrooms, children are given multiple opportunities to practice self-regulation in specific ("focal") activities where self-regulation is a focus as well as in the activities where self-regulation strategies are embedded in the context of academic tasks. At the heart of Tools curriculum are instructional strategies designed to promote the development of mature make-believe play as a primary context for the development of self-regulation in preschool- and kindergarten-aged children.
Teachers receive professional development delivered during workshops and through regular contacts with Tools staff. Teachers participate in Year 1 core professional development and may continue with additional training in Years 2 and 3. To provide additional guidance in assessing children's development and differentiating instruction, curriculum resources and tools are available to teachers through Tools' digital applications and platforms.
In the evaluation certified by Blueprints (Blair & Raver, 2014; Blair et al., 2018), teachers received two years of professional development.
In the first study (Blair & Raver, 2014; Blair et al., 2018), as compared to the control group, the treatment group had significantly better scores at posttest or follow-up on measures of:
The second study (Barnett et al., 2008) found significant treatment effects for:
The third study (Diamond et al., 2007) found significant treatment effects for:
The fourth study (Farran & Wilson, 2014) found no positive treatment effects and possible negative treatment effects on:
The fifth study (Solomon et al., 2018) found no significant treatment effects for any of the primary outcomes.
The sixth study (Diamond et al., 2019) found that, compared to the control group, kindergarten students in the intervention group showed significantly improved:
The seventh study (Hsueh et al., 2014; Mattera et al., 2013; Morris et al., 2014) did not find any significant positive results for child, teacher, or classroom outcomes among 3-year-olds. Four-year-olds had significant intervention effects on:
Blair and Raver (2014) used a block-randomized control trial to assign 29 schools to either a treatment or a business-as usual control group. The sample included 759 children in 79 kindergarten classrooms. The study conducted a pretest early in the academic year, a posttest in the spring of the academic year, and a 6-month follow-up in the fall of first grade. It evaluated outcomes measuring executive function, stress response, academic performance, and classroom behavior.
A second study (Barnett et al., 2008) assigned students and teachers randomly to treatment and control classrooms. It included 274 3- and 4-year-old children in the treatment (n=106) and control (n=168) groups, but after randomization only 224 consented to participate. The study conducted assessments in the fall and spring of the intervention year and evaluated language and pre-literacy skills and nonverbal problem solving, in addition to problem behaviors.
A third study (Diamond et al., 2007) randomly assigned students to either the Tools of the Mind intervention or the district's version of Balanced Literacy. It included 147 preschoolers, 62 in the Balanced Literacy program and 85 in Tools of the Mind. Students participated in the intervention for 1-2 years; posttest assessments were completed in May and June of Year 2. All measures addressed executive function.
A fourth study (Farran & Wilson, 2014) randomly assigned schools, within blocks, to either the Tools of the Mind intervention or the district's standard curriculum. The sample included 60 classrooms with 498 children in the treatment condition and 379 in the comparison condition. The pretest was conducted with children at the beginning of their preschool year, the posttest at the end of the preschool year, and two long-term follow-up assessments were conducted at the end of the first and second years of primary school. The study used direct measures of pre-academic skills and executive function and teacher-observed measures of language, social behavior, and self-regulation.
A fifth study (Solomon et al., 2018) randomly assigned 20 childcare centers, each with one participating classroom (total N=256), to receive the Tools of the Mind intervention or a play-based control condition that did not emphasize self-regulation. Child executive function and teacher- and parent-reported child behavior were measured at baseline, eight months after baseline, and post-intervention.
In a sixth study, Diamond et al. (2019) used a clustered experimental design to examine 352 kindergarten students in 18 kindergarten classrooms. Schools were randomly assigned to either the intervention group or the business-as-usual control group. Classroom teachers completed a baseline questionnaire about each child's attitudes and behaviors in the fall of the kindergarten year and standardized assessments measured academic performance. The posttest occurred in the spring of the kindergarten year. The primary outcomes included measures of child academic performance, behavior, and self-regulatory skills.
A seventh study (Hsueh et al., 2014) examined only the "play" portion of the intervention and did not test the full Tools of the Mind program. This study randomly assigned 104 Head Start centers with 461 participating 3-year-old children and 1,299 4-year-old children to either the treatment or control group. The pretest came early in the fall, a posttest in the spring of the next year, and a kindergarten follow-up one year later. The study evaluated social-emotional and behavioral outcomes and pre-academic skills. Although 3-year-olds and 4-year-olds participated in the study in mixed classrooms, they were analyzed separately.
Study 1
Blair, C., & Raver, C. C. (2014). Closing the achievement gap through modification of neurocognitive and neuroendocrine function: Results from a cluster randomized controlled trial of an innovative approach to the education of children in kindergarten. PLOS One, 9, 1-13.
Blair, C., McKinnon, R. D., & Daneri, M. P. (2018). Effect of the Tools of the Mind kindergarten program on children's social and emotional development. Early Childhood Research Quarterly, 48, 52-61.
Individual: Antisocial/aggressive behavior*, Stress
School: Poor academic performance
Individual: Problem solving skills, Prosocial behavior, Skills for social interaction
School: Instructional Practice
*
Risk/Protective Factor was significantly impacted by the program
See also: Tools of the Mind Logic Model (PDF)
In Study 1, (Blair & Raver, 2014; Blair et al., 2018) the program was not tested for differences in its effects by gender, race, or ethnicity
In Studies 2 and 3 (Barnett et al., 2008; Diamond et al., 2007) the sample was predominantly (92,6%) Latino/Hispanic with Spanish as the primary home language for the majority of children (69%). The program was not tested for differences in its effects by gender, race, or ethnicity
In Study 4, Farran and Wilson (2014) did not find differential effects of the intervention on gender or race/ethnicity.
In Study 5, Solomon et al. (2018) did not test for program differences by race or gender.
In Study 6, Diamond et al. (2019) did not test for differences in program effects by race, ethnicity, or gender.
In Study 7, Hsueh et al. (2014) did not test for differences by gender, race, or ethnicity.
About Us (Full Description):
Tools of the Mind is a research-based early childhood program that embeds development of executive functions and self-regulation into the design of all teaching and learning activities. Tools leverages make-believe play and a classroom culture of co-regulation and peer scaffolding to build the foundational skills children need to be successful in school and life.
First developed and pilot tested in Denver, Colorado in 1990s, Tools of the Mind has expanded to many other states and is being currently implemented by early childhood programs and public-school districts in 23 states and 3 countries. Although beneficial for all children, Tools has been most widely used in programs that serve children at risk of low achievement outcomes related to low-income/poverty status. A majority of the programs continue to be in districts with a high proportion of at-risk families. Tools has a version of the program and many years of experience with families whose home language is Spanish.
Tools of the Mind can be used in multiple settings serving children ages 3-6. Examples of such settings include but are not limited to:
Tools of the Mind can be used in single-age as well as multi-age classrooms (e.g., preschool classrooms with children ages 3-5). Tools of the Mind is typically used during school year, although its use can be extended to a summer program (e.g., for children attending Tools preschool and going into Kindergarten). Tools of the Mind can be used in programs serving typically developing children as well as the programs serving children with special needs. In the past, Tools of the Mind had been used by programs with varying ratios of children with special needs and typically developing children (e.g., inclusion, reverse inclusion, self-contained special needs). The educational backgrounds and experience of Tools-trained teachers also vary; those who have received the training include experienced teachers with master's degrees, Head Start assistants with GED certificates or high school diplomas, and graduates of alternative teacher education programs such as Teach for America.
Tools of the Mind is grounded in the Vygotskian theory of development and learning and also incorporates the contributions of several generations of post-Vygotskian scholars including one of the founding fathers of neuropsychology - Alexander Luria. Since Vygotskians identify intentionality and self-regulation as major outgrowths of the early childhood years that are necessary prerequisites for school success, promoting self-regulation is one of the main focus areas in Tools of the Mind activities. The design of these activities is based on the Vygotsky-Luria ideas of the development of intentionality and is also informed by the latest findings in the field of developmental neuropsychology and especially the research on the development of executive functions.
As a comprehensive curriculum, Tools of the Mind contains multiple activities targeting all major areas of child development and intended to be used in large group, in small groups, as well as by pairs of children and by individual children. Many of the small group activities are designed to be played by children as a game with minimal or no support by the teacher. In these games, children practice literacy and math concepts while at the same time learning to follow increasingly complex sets of rules. All games included in the Tools of the Mind curriculum share some common characteristics that make these games compatible with the Tools goal of building classroom community: children and teams do not compete with each other but only with their own past results and children are not prohibited but are instead encouraged to help each other when somebody does not know the answer. To help children develop intentional self-regulated behaviors, self-regulation practice is embedded in all activities across subject areas. This practice takes different forms as children engage in other-regulation working with their partners, use private speech (self-talk) to regulate their own behavior, or make use of external (tangible or pictorial) mediators to self-direct their attention to specific aspects of a task.
All Tools activities are designed to be multi-level so that children of varying abilities can engage in social interactions with each other over the same content. This design prevents ability grouping and promotes the sense of community in all children. Tools teachers learn to support children of different abilities within the same activity by providing differentiated scaffolding to all of them. This makes Tools curriculum an ideal fit for multi-age classrooms as well as the inclusion classrooms. Another distinct feature of all Tools activities is that these are designed to "grow with a child" from early preschool years into kindergarten constantly presenting children with new levels of challenges.
Attention to play as the primary context for development of self-regulation is one of the hallmarks of Tools of the Mind. In Tools preschool classrooms, children spend a significant part of their day playing and play is modeled, scaffolded, and organized in a particular way to ensure that children create an imaginary situation, act out well-defined roles and follow the rules built in the pretend scenario. In Tools kindergarten, children continue to acquire and refine their play skills as they engage in play-dramatization, creating and acting out pretend scenarios based on literature they study in class. Strategic choice of the literature ensures that children use their play to master science and social studies content along with rich vocabulary.
The pedagogy used by Tools of the Mind teachers is based on the concept of individualized scaffolding within each child's Zone of Proximal Development (ZPD) that the teacher has determined through Dynamic Assessment. Dynamic assessment is used to establish both the child's current level of achievement and a child's potential ability to attain higher levels. When used in a classroom, Dynamic Assessment consists of a series of prompts and hints provided by the teacher. Skills and understandings that children demonstrate as a result of minimal assistance are considered closer to being developed and are then chosen as a focus of immediate instruction.
Teachers in the Tools classrooms use the results of Dynamic Assessment to determine which scaffolding strategy works best with each individual child or with a group of children with like needs. These scaffolds are described in the Teacher Manual and taught to the teachers during the training sessions. In addition to the range of scaffolding strategies aimed at typically developing children, special strategies are included to support children with special needs and dual language learners. Tools teachers also learn to use Dynamic Assessment to determine whether the level of support can be removed, as the child is more and more capable of doing the activity on his/her own. Scaffolding strategies go beyond one-on-one interactions and include the use of external mediators, private speech and specially designed interactions with peers. Children's increasing levels of self-regulation allow Tools teachers minimize whole group instruction and spend more time providing scaffolding to individual children, especially the ones who need extra support or extra challenges.
Training and Technical Assistance
Public, charter, and private schools as well as Head Start programs partner with Tools to launch the curriculum and engage in the accompanying professional development. In addition, partnership with Tools involves joint analysis of district- and Tools-collected data resulting in partners making developmentally appropriate instructional decisions leading to improved child outcomes. There are also additional Continuing Education opportunities after the Core PD is completed and opportunities to participate and present in Community of Practice. Part of the professional development process is the creation of a vibrant and strong community of practice by creating a structure for each program partner that fits the specific needs and organizational structure of that program to support teachers and holding data meetings so that the quality of the program remains high.
During Year 1, participants attend a 2-day in-person workshop in the beginning of a school year and 3 more one-day workshops throughout the year (30 hours total). The workshops content covers the major principles of Tools' educational philosophy, its research base as well as "why's" and "how to's" of the key Tools' activities. Additional information on Tools activities including the guidance on how to provide individualized scaffolding is available in the iScaffold - a professional development tool the teachers can access on the eTools section of Tools of the Mind website or download to their iPads. In-person or virtual technical assistance is provided by Tools staff who visit the classrooms in-between the workshops and provide telephone consultations.
After completing the Year 1 core professional development, districts have the opportunity to continue to partner with Tools through participating in Year 2 and Year 3 training. The focus of Year 2 is becoming more skillful practitioners: Developing 5 core teaching capabilities, using data to improve instruction and child outcomes. During Year 2, teachers attend 2 full-day workshops (12 hours) and receive in-person or virtual Technical Assistance from Tools staff. Teachers who want to receive Tools endorsements have an option to participate in Year 3 training. The focus of Year 3 is growth and sustainability: Continuous improvement in data-driven instruction, application of Tools' core capabilities and reflecting on practice and latest developments with colleagues and Tools staff.
Participants in Tools of the Mind core workshop series development are eligible to apply for college credits based on professional development hours completed and subsequent work associated. In addition to this, Tools of the Mind has an internal endorsement and certification process that educators have an opportunity to participate in after completing the first two years of core professional development and classroom implementation. The endorsement process begins with a one-day workshop designed where educators are introduced to the Tools' fidelity instrument and pathway to endorsement. Teachers have the opportunity to reflect on practice in other classrooms and their own and prepare for their endorsement submission. Endorsed teachers can go on to get certified in each of the five core teaching capabilities in Tools. This certification signals excellence in that capability and certified teachers will be leaders, sharing case studies from their classrooms and ways they are innovating using the Tools approach in their own regions.
Source: Washington State Institute for Public Policy
All benefit-cost ratios are the most recent estimates published by The Washington State Institute for Public Policy for Blueprint programs implemented in Washington State. These ratios are based on a) meta-analysis estimates of effect size and b) monetized benefits and calculated costs for programs as delivered in the State of Washington. Caution is recommended in applying these estimates of the benefit-cost ratio to any other state or local area. They are provided as an illustration of the benefit-cost ratio found in one specific state. When feasible, local costs and monetized benefits should be used to calculate expected local benefit-cost ratios. The formula for this calculation can be found on the WSIPP website.
Core Year 1 and Technical Assistance for Kindergarten
$3750/classroom includes:
Core Year 1 and Technical Assistance for PreK
$3,750/classroom includes:
Curriculum and Materials: Kindergarten
Each classroom makes a 1-time purchase of a Tools of the Mind Kit (produced by and purchased through Lakeshore© Learning Materials Inc). The Lakeshore Kit includes all the materials needed to implement activities (e.g., phonological awareness activities, phonics activities, fiction books, science materials, and writing activities). Materials that need to be replenished can be printed off eTools-Tools' web-based portal. The Lakeshore© Learning Literacy and Math Kits are $599/each per classroom (or combined set for $1100). Some classrooms begin with Literacy Kit purchase and later purchase the Math kits.
Curriculum and Materials: PreK
Each classroom makes a 1-time purchase of a Tools of the Mind Kit (produced by and purchased through Lakeshore© Learning Materials Inc). The Lakeshore Kit includes all the materials needed to implement activities (e.g., literacy activities, math activities, linear calendar, etc.). Materials that need to be replenished can be printed off eTools-Tools' web-based portal. The Lakeshore© Tools of the Mind Kit is $749.00 per classroom.
For Kindergarten Only
Tools' digital applications deepen the Tools approach and activities. If a school district or program chooses to take advantage of Tools' digital applications & platforms, each classroom should have:
Inclusive with training costs. Licensing/Subscription costs may be applicable to classrooms continuing to access certain Tools applications (including iScaffold and the PowerTools Literacy Solutions).
Tools offers the opportunity for in-person or virtual meetings with school district teachers, administrators and coaches as well as classroom visits prior to beginning implementation.
Kindergarten classrooms are designed to have 8 center areas; Tools works in many contexts, from ones that only have desks which are multi purposed to create 'centers', to Kindergarten classroom environments with rich base layer materials in clearly established centers.
Tools' digital applications deepen the Tools approach and activities. If a school district or program chooses to take advantage of Tools' digital applications and platforms, each classroom should have:
No additional ongoing physical material costs associated. Teachers can access printable materials on eTools.
Beyond the years in which teachers engage in training, subscription/licensing costs may be applied to access Tools' digital applications and platforms, including the PowerTools Literacy Solutions (PowerTools Reading System and Developmental Writing Assessments), iScaffold, and eTools. Subscription costs depend on current and past training engagement and/or data partnerships with Tools of the Mind.
Kindergarten
Required: 1 lead teacher per classroom to attend the professional development workshop.
Highly Recommended: Each classroom has at least a .5 assistant who attends professional development workshops with their lead or co-teacher.
Recommended: A dedicated coach or Master Teacher who supports all teachers implementing Tools; we can work with administrators to identify creative solutions if this is not in place.
PreK
Required: 1 lead teacher per classroom to attend the professional development workshops and work with Tools of the Mind team lead to support implementation.
Highly Recommended: A teacher assistant who attends professional development workshops with their lead teacher
Recommended: Specialists working with Tools children in push-in or pull-out programs (e.g., special educators) who attend professional development workshops
Recommended: A dedicated coach or Master Teacher who supports all teachers implementing Tools; we can work with administrators to identify creative solutions if this is not in place.
We recommend that administrators participate in one or more workshops to understand the Tools approach. Administrators can also take advantage of our digital platforms, such as the PowerTools Literacy Solutions, to monitor live progress of their students' reading and writing outcomes (subscription is included as part of the Year 1 and 2 workshops).
There are options to receive additional Technical Assistance (TA) support (beyond TA days included in the contracted professional development package)-in-person or virtual-where a Tools of the Mind team members work directly with teachers and other staff to customize implementation and increase fidelity.
The TA Residency Model is also available, where a Tools of the Mind team member is embedded into the school district or program for a period of time, to deepen implementation and fidelity.
Costs for TA support are dependent on number of classrooms and visits required.
Programs can have the local team become certified to endorse teachers using the Tools fidelity of implementation measure as part of their Year 3 work; classrooms can become endorsed; teachers can go further to be certified in each or all of Tools' 5 core capabilities.
N/A
No information is available
No information is available
This example assumes that a school district would adopt the Tools of the Mind program to use in 10 Kindergarten classrooms with 2 teachers per classroom. Costs included are for materials to implement the program and for on-site training. Not included in the training expense is the cost for trainer travel; this expense should be negotiated directly with the Tools of the Mind organization.
See Summary Statement for alternate expenses for PreK.
| Training: $3,750 X 10 classrooms | $37,500.00 |
| Tools of the Mind Kit (literacy and math): $1,100 X 10 classrooms | $11,000.00 |
| Total One Year Cost | $48,500.00 |
With 200 students in 10 Kindergarten classrooms, the additional cost to the district (beyond their existing staffing and operations cost) is approximately $242.50/child (exclusive of travel expenses).
With 180 students in 10 PreK classrooms, the additional cost to the district (beyond their existing staffing and operations cost) is approximately $250.00/child (exclusive of travel expenses). This includes professional development and curriculum at $3,750 per classroom and the Tools of the Mind Kit (literacy and math) at $749 per classroom.
No information is available
Tools partners with programs and local foundations to get the needed support for launching Tools in Year 1. For example, Tools of the Mind is currently partnering with the Colorado Department of Education and the Denver Public Schools system.
The Tools of the Mind program/curriculum is fee for service.
Deborah Leong, Ph.D.President/Co-founderTools of the Mind26093 Thea Gulch Rd.Golden, CO 80403USA(303) 279-5589dleong@toolsofthemind.org www.toolsofthemind.org
An Early Childhood (preschool and kindergarten) program designed to promote academic learning and prosocial behaviors by supporting the development of executive function and other regulation-related skills.
The program targets children in preschool (3-5 years of age) and kindergarten (5-6 years of age).
In Study 1, (Blair & Raver, 2014; Blair et al., 2018) the program was not tested for differences in its effects by gender, race, or ethnicity
In Studies 2 and 3 (Barnett et al., 2008; Diamond et al., 2007) the sample was predominantly (92,6%) Latino/Hispanic with Spanish as the primary home language for the majority of children (69%). The program was not tested for differences in its effects by gender, race, or ethnicity
In Study 4, Farran and Wilson (2014) did not find differential effects of the intervention on gender or race/ethnicity.
In Study 5, Solomon et al. (2018) did not test for program differences by race or gender.
In Study 6, Diamond et al. (2019) did not test for differences in program effects by race, ethnicity, or gender.
In Study 7, Hsueh et al. (2014) did not test for differences by gender, race, or ethnicity.
Starting from preschool, children engage in planning their activities. As preschoolers plan their play they learn to anticipate and resolve potential conflicts with peers which assists children in developing social competence, emotional and problem-solving skills. In kindergarten, children practice planning and monitoring skills as they create learning plans, set their learning goals and discuss their performance relative to these goals during weekly conferences with their teacher. In addition, specifically designed dyadic activities engage children in monitoring and evaluating their partner's actions, which promotes development of metacognitive skills and reflective thinking along with acquiring positive models of communication and cooperation.
Individual: Antisocial/aggressive behavior*, Stress
School: Poor academic performance
Individual: Problem solving skills, Prosocial behavior, Skills for social interaction
School: Instructional Practice
*Risk/Protective Factor was significantly impacted by the program
Tools of the Mind (Tools) is an early childhood curriculum and a professional development program that supports children's cognitive, social, and emotional development with a special emphasis on the development of executive function and self-regulation. At the heart of Tools curriculum are instructional strategies designed to promote the development of mature make-believe play as a primary context for the development of self-regulation in preschool- and kindergarten-aged children. Teachers participate in professional development, including training workshops and regular contact with Tools staff over the course of the Core Year 1 and optional Years 2 and 3.
Tools of the Mind (Tools) is an early childhood curriculum and a professional development program that is based on Vygotskian theory and developmental science and is informed by neuroscience research. Tools offers a comprehensive PreK & K curriculum that supports children's cognitive, social, and emotional development with a special emphasis on the development of executive function and self-regulation. In Tools classrooms, children are given multiple opportunities to practice self-regulation in specific ("focal") activities where self-regulation is a focus as well as in the activities where self-regulation strategies are embedded in the context of academic tasks. At the heart of Tools curriculum are instructional strategies designed to promote the development of mature make-believe play as a primary context for the development of self-regulation in preschool- and kindergarten-aged children.
Teachers receive professional development delivered during workshops and through regular contacts with Tools staff. Teachers participate in Year 1 core professional development and may continue with additional training in Years 2 and 3. To provide additional guidance in assessing children's development and differentiating instruction, curriculum resources and tools are available to teachers through Tools' digital applications and platforms.
In the evaluation certified by Blueprints (Blair & Raver, 2014; Blair et al., 2018), teachers received two years of professional development.
Tools of the Mind is based on the Vygotskian approach to learning and development that emphasizes the importance of social interactions in developing intentional self-regulated behaviors. This educational philosophy is consistently applied to all components of the curriculum, from instructional activities derived from post-Vygotskian research, to the emphasis on mature make-believe play as the leading activity of preschool and kindergarten-aged children, to the systematic use of dynamic assessment to inform day-to-day teaching and individualized scaffolding, and to the approach to school readiness that focuses on the cognitive and social-emotional development that enables children to engage in the learning activities of the primary grades. The consistency in the approach allows for the stability of curriculum implementation in different settings, with children of different abilities, and by different staff members working with these children.
Blair and Raver (2014) used a block-randomized control trial to assign 29 schools to either a treatment or a business-as usual control group. The sample included 759 children in 79 kindergarten classrooms. The study conducted a pretest early in the academic year, a posttest in the spring of the academic year, and a 6-month follow-up in the fall of first grade. It evaluated outcomes measuring executive function, stress response, academic performance, and classroom behavior.
A second study (Barnett et al., 2008) assigned students and teachers randomly to treatment and control classrooms. It included 274 3- and 4-year-old children in the treatment (n=106) and control (n=168) groups, but after randomization only 224 consented to participate. The study conducted assessments in the fall and spring of the intervention year and evaluated language and pre-literacy skills and nonverbal problem solving, in addition to problem behaviors.
A third study (Diamond et al., 2007) randomly assigned students to either the Tools of the Mind intervention or the district's version of Balanced Literacy. It included 147 preschoolers, 62 in the Balanced Literacy program and 85 in Tools of the Mind. Students participated in the intervention for 1-2 years; posttest assessments were completed in May and June of Year 2. All measures addressed executive function.
A fourth study (Farran & Wilson, 2014) randomly assigned schools, within blocks, to either the Tools of the Mind intervention or the district's standard curriculum. The sample included 60 classrooms with 498 children in the treatment condition and 379 in the comparison condition. The pretest was conducted with children at the beginning of their preschool year, the posttest at the end of the preschool year, and two long-term follow-up assessments were conducted at the end of the first and second years of primary school. The study used direct measures of pre-academic skills and executive function and teacher-observed measures of language, social behavior, and self-regulation.
A fifth study (Solomon et al., 2018) randomly assigned 20 childcare centers, each with one participating classroom (total N=256), to receive the Tools of the Mind intervention or a play-based control condition that did not emphasize self-regulation. Child executive function and teacher- and parent-reported child behavior were measured at baseline, eight months after baseline, and post-intervention.
In a sixth study, Diamond et al. (2019) used a clustered experimental design to examine 352 kindergarten students in 18 kindergarten classrooms. Schools were randomly assigned to either the intervention group or the business-as-usual control group. Classroom teachers completed a baseline questionnaire about each child's attitudes and behaviors in the fall of the kindergarten year and standardized assessments measured academic performance. The posttest occurred in the spring of the kindergarten year. The primary outcomes included measures of child academic performance, behavior, and self-regulatory skills.
A seventh study (Hsueh et al., 2014) examined only the "play" portion of the intervention and did not test the full Tools of the Mind program. This study randomly assigned 104 Head Start centers with 461 participating 3-year-old children and 1,299 4-year-old children to either the treatment or control group. The pretest came early in the fall, a posttest in the spring of the next year, and a kindergarten follow-up one year later. The study evaluated social-emotional and behavioral outcomes and pre-academic skills. Although 3-year-olds and 4-year-olds participated in the study in mixed classrooms, they were analyzed separately.
The first study (Blair & Raver, 2014; Blair et al., 2018) found significantly better scores among the treatment group as compared to the control group for measures of applied problems (math), reading, and vocabulary. One teacher-rated measure of aggression and conduct problems in the classroom was significantly lower for the intervention group than the control group. It also found significantly better scores among the treatment group for several measures executive function: working memory, reaction speed, and rapid naming.
The second study (Barnett et al., 2008) found a significant treatment effect on teacher-rated problem behaviors when compared to the control group. In addition, the study found a marginally significant effect for a measure of receptive and expressive language test of Spanish-speaking children.
The third study (Diamond et al., 2007) found significant treatment effects for 4 of 5 tests of executive function; however, the study did not include baseline measures. The study found that the intervention had a stronger effect on more demanding tests of executive function.
The fourth study (Farran & Wilson, 2014) found no significant treatment effects for any measure at either posttest or long-term follow-up. The study found possible negative treatment effects at long-term follow-up as compared to the control condition.
The fifth study (Solomon et al., 2018) found no significant treatment effects for any of the primary outcomes tested (child executive function, parent- and teacher-rated behavioral difficulties, and teacher-rated social and behavioral competencies).
The sixth study (Diamond et al., 2019) found the intervention students had significantly improved performance on reading skills, writing skills, and prosocial behavior, as well as increased levels of self-regulation and self-control. There were no differences between students in reports of their academic mathematic competence.
The seventh study (Hsueh et al., 2014; Mattera et al., 2013; Morris et al., 2014) did not find any significant intervention effects for 3-year-old children, teacher, or classroom outcomes. For 4-year-old children, the study found a significant intervention effect for one measure of classroom environment, one measure of teacher behavior, and one measure of child social behavior, but no significant effects for child academic skills.
In the first study (Blair & Raver, 2014; Blair et al., 2018), as compared to the control group, the treatment group had significantly better scores at posttest or follow-up on measures of:
The second study (Barnett et al., 2008) found significant treatment effects for:
The third study (Diamond et al., 2007) found significant treatment effects for:
The fourth study (Farran & Wilson, 2014) found no positive treatment effects and possible negative treatment effects on:
The fifth study (Solomon et al., 2018) found no significant treatment effects for any of the primary outcomes.
The sixth study (Diamond et al., 2019) found that, compared to the control group, kindergarten students in the intervention group showed significantly improved:
The seventh study (Hsueh et al., 2014; Mattera et al., 2013; Morris et al., 2014) did not find any significant positive results for child, teacher, or classroom outcomes among 3-year-olds. Four-year-olds had significant intervention effects on:
None of the studies formally analyzed mediating effects.
The first study (Blair & Raver, 2014) reported mostly small effect sizes, ranging from .08-.25 for significant main effects. The second study (Barnett et al., 2008) reported a moderate effect size (-.50) for the significant problem behavior outcome in the multilevel model. The third study (Diamond et al., 2007) reported medium effect sizes on the more demanding EF tasks: β=+0.28 for the Dots-Incongruent task; β=+0.31 for Standard Flanker task; and β=+0.39 for the Reverse Flanker task. The fourth study (Farran & Wilson, 2014) reports no significant treatment effects. In the fifth study, Solomon et al. (2018) found no significant treatment effects. Diamond et al. (2019), in the sixth study, reported small to large odds ratios ranging from 0.95 to 8.70 for the primary outcomes. The seventh study (Hsueh et al., 2014) did not obtain significant effects for any of the outcomes for 3-year-olds. Effect sizes for 4-year-olds ranged from medium-high (.68) for teacher behavior to very small for child behavior (.12).
In the first study (Blair & Raver, 2014), the geographic extent of the sample was unclear, although 12 total school districts participated. The second (Barnett et al., 2008) and third (Diamond et al., 2007) studies are limited to a single school. The fourth study (Farran & Wilson, 2014) was conducted in two states with diverse school districts. In the fifth study, Solomon et al. (2018) can generalize their sample to other preschool-aged children attending YMCA childcare centers in Ontario, Canada. In Study 6, Diamond et al. (2019) examined kindergarten students in 18 schools in the Vancouver and Surrey school districts, British Columbia, Canada. The seventh study (Hsueh et al., 2014; Morris et al., 2014) had a large geographic coverage and a large number of Head Start centers.
Study 1: Blair and Raver (2014); Blair et al. (2018)
Study 2: Barnett et al. (2008)
Study 3: Diamond et al. (2007)
Study 4: Farran and Wilson (2014)
Study 5: Solomon et al. (2018)
Study 6: Diamond et al. (2019)
Study 7: Hsueh et al. (2014); Morris et al. (2014)
Blueprints: Promising
Tools has a number of peer implementation sites with different demographics in over 20 different states across the country. Educators who want to visit these sites may contact Tools at information@toolsofthemind.org with information about their specific school or program, and Tools staff connects them with a site matching their characteristics.
Tools of the Mind
Email: information@toolsofthemind.org
Website: www.toolsofthemind.org
Certified Blair, C., & Raver, C. C. (2014). Closing the achievement gap through modification of neurocognitive and neuroendocrine function: Results from a cluster randomized controlled trial of an innovative approach to the education of children in kindergarten. PLOS One, 9, 1-13.
Certified Blair, C., McKinnon, R. D., & Daneri, M. P. (2018). Effect of the Tools of the Mind kindergarten program on children's social and emotional development. Early Childhood Research Quarterly, 48, 52-61.
Barnett, W. S., Jung, K., Yarosz, D. J., Thomas, J., Hornbeck, A., Stechuk, R., & Burns, S. (2008). Educational effects of the Tools of the Mind curriculum: A randomized trial. Early Childhood Research Quarterly, 23, 299-313.
Diamond, A., Barnett, W. S., Thomas, J., & Munro, S. (2007). Preschool program improves cognitive control. Science, 318, 1387-1388.
Farran, D. C., & Wilson, S. J. (2014). Achievement and self-regulation in pre-kindergarten classrooms: Effects of the Tools of the Mind curriculum. Manuscript submitted for publication.
Solomon, T., Plamondon, A., O'Hara, A., Finch, H., Goco, G., Chaban, P., ... & Tannock, R. (2018). A cluster randomized-controlled trial of the impact of the Tools of the Mind curriculum on self-regulation in Canadian preschoolers. Frontiers in psychology, 8, 2366.
Diamond, A., Lee, C., Senften, P., Lam, A. & Abbott, D. (2019). Randomized control trial of Tools of the Mind: Marked benefits to kindergarten children and their teachers. PLoS ONE, 14(9): e0222447. https://doi.org/10.1371/journal.pone.0222447
Hsueh, J., Lowenstein, A. E., Morris, P., Mattera, S. K., & Bangser, M. (2014). Impacts of social-emotional curricula on three-year-olds: Exploratory findings from the Head Start CARES demonstration. Washington, DC: Office of Planning, Research, and Evaluation, U.S. Department of Health and Human Services.
Mattera, S., Lloyd, C. M., Fishman, M., & Bangser, M. (2013). A first look at the Head Start CARES Demonstration: Large-scale implementation of programs to improve children's social-emotional competence. OPRE Report 2013-47. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
Morris, P., Mattera, S. K., Castells, N., Bangser, M., Bierman, K., & Raver, C. (2014). Impact findings from the Head Start CARES demonstration: National evaluation of three approaches to improving preschoolers' social and emotional competence. Washington, DC: Office of Planning, Research, and Evaluation, U.S. Department of Health and Human Services.
Evaluation Methodology
Design:
Recruitment: The study contacted 12 school districts that agreed to participate. The 29 schools within the school districts agreed to be randomly assigned and participate fully in the project. The study then attempted to recruit 6 students in each of 79 kindergarten classrooms through flyers sent to children's homes. The first 6 students whose parents consented to participate were selected. Over the two years of study, the total sample size reached 759.
Assignment: The study utilized a block randomized design. Schools were randomized within blocks balancing school district, the percent of students eligible for free or reduced-price lunch, school size, state test scores, and number of kindergarten classrooms. The study noted an exception "for 2 schools in the participating districts in which all of the district's kindergarten classrooms were housed in one facility. For these two schools, random assignment was at the classroom level and the school was treated as two schools for analysis purposes." Kindergarten classrooms assigned to the control condition (N = 37) continued with business as usual, while classrooms assigned to the treatment condition (N = 42) implemented program activities.
Attrition: The study conducted three assessments, the first around October (used as a pretest although the program would have started at the beginning of the school year), the second around March (used as a posttest although the program would not have ended until the end of the school year), and the last in the fall of the following year (approximately 6-month follow-up). Blair and Raver (2014) reported approximately 5% student attrition (N=34) from fall to spring; these dropouts were replaced in the spring with other students who had been participating in the program since the start. The total sample size was N = 723 in the fall and N = 725 in the spring. However, the smaller sample sizes listed in Tables 1 and 2 for many of the measures indicate additional missing data.
Blair et al. (2018) examined teacher-reported data that had more attrition. Of the 759 enrolled students, 667 (88%) had data in the spring and 554 (73%) had data in first grade. Table 2 additionally shows considerable missing data at the baseline assessment in the fall of kindergarten, with sample sizes ranging from 534 to 616.
Sample: Students eligible for free or reduced-price lunch ranged by school from 5-92%, and about 15% of the schools in the sample were considered high-poverty schools. No information was provided on the characteristics of the sample students.
Measures: Blair and Raver (2014) measured academic ability using the Woodcock-Johnson III (WJ-III) subtest of applied problems (math) and letter-word (reading) at all assessment points, the WJ-III reading vocabulary subtest at posttest and 6-month follow-up, and the Expressive One Word Picture Vocabulary Test at pretest (both measures of vocabulary ability). In addition, the study measured general reasoning with the Raven Colored Progressive Matrices.
Measures relating to executive functions included cognitive flexibility and inhibitory control with the Hearts & Flowers test, Flanker with Reverse Flanker test, and the Dimensional Change test in the NIH Toolbox. The study measured attention control with the dot-probe test and neuroendocrine function by taking two measures of cortisol and alpha amylase at different times of day at all assessment points. Finally, the study measured reaction speed by timing responses on a number of the above tests (it is not clear which).
Teachers provided six student measures (Blair et al., 2018). Note that kindergarten teachers delivered the program, making the posttest assessments non-independent, but that ratings by first-grade teachers, after the end of the program, should be independent. For the first two outcomes, the Teacher Social Competence Rating Scale measured aggression/conduct scores and self-regulation scores. Alpha reliabilities for the two scales across the three assessments ranged from .71-.95. Third, the Strengths and Difficulties Questionnaire measured general behavioral problems (alphas of .84-.91). Fourth, the Emotional Regulation Checklist measured emotional regulation (alphas of .92-.93). Fifth, the Student-Teacher Relationship Scale-Short Form measured teacher-student relationship quality (alphas of .87-.88). Sixth, the Social Skills Rating System rates student academic competence relative to other students in the classroom (alphas of .95).
Analysis: Blair and Raver (2014) analyzed the data using multilevel models with children nested within schools. For all analyses, the treatment versus control group status was included as a binary indicator along with the pretreatment measure of the outcome variable and child-level and school-level covariates. Although the study also analyzed change scores, they did not report them. The study found that data were missing at random, so used full information maximum likelihood (FIML) estimation.
Blair et al. (2018) used mixed linear modeling with children nested within schools, the unit of random assignment. The models included the baseline score and several sociodemographic covariates that varied across conditions at baseline. The FIML estimates addressed potential bias associated with missing data and adjusted standard errors to account for the non-independence of children nested within schools.
Intent-to-Treat: All participants were included at each assessment point with the FIML estimates.
Outcomes
Implementation Fidelity: The study did not provide any discussion of implementation fidelity.
Baseline Equivalence: Page 8 of Blair and Raver (2014) states that "we examined treatment and control group differences on all variables assessed in the fall of kindergarten to determine if differences between the groups were present at baseline. No differences were observed." Blueprints calculations using the means, standard deviations, and Ns in fact show that, for 21 baseline outcome measures listed in Tables 1 and 2 for the Fall, there were no significant differences between the control group and intervention group means.
In contrast, Blair et al. (2018) noted several baseline differences in race, parent education, and eligibility for free and reduced-price lunch - all used as controls in the analysis. However, there were no significant differences between conditions on the outcome variables.
Differential Attrition: Attrition was low in Blair and Raver (2014), and the study reported that missing data were not related to treatment or control condition. The baseline equivalence tests, which could not include the missing data, showed no differences, and full information maximum likelihood estimation adjusted for potential bias in the estimates resulting from missing data.
Blair et al. (2018) stated that those missing data at the spring of kindergarten or at the fall of first grade did not differ from the rest of the sample in regards to treatment status, age, parental education, and the percentage of free or reduced-price lunch eligible children.
Posttest: The results are difficult to follow and Figures 1 and 2 show confidence intervals based on 1 standard error unit, although the text says the figures show 95% confidence intervals.
At posttest, one of the four measures of academic ability showed significant improvement for the treatment group, as compared to the control group: applied problems (d = .13), but reading, reasoning, and vocabulary did not. At the 6-month follow-up, the significant treatment effect for math at posttest fell to insignificance, while the insignificant treatment effect on reading and vocabulary emerged as significant (d = .14 and .10, respectively). The treatment effect on the reasoning measure did not reach significance at either posttest or follow-up.
At posttest, three of five measures of executive function were significantly improved for the treatment students, as compared to the control students: working memory (d = .14), reaction time (d = .12), and rapid naming (d = .08). The posttest measures of stress response did not differ significantly across conditions.
When testing for moderation by poverty status of the school, the study found that the treatment had significant benefits in high-poverty schools for reasoning, vocabulary, and attention. Moderation for the alpha amylase measure of stress response showed significantly lower scores for the treatment group in high poverty schools, but the cortisol measure of stress response showed significantly higher scores for the treatment group in high poverty schools.
Blair et al. (2018) first examined six teacher ratings of students at posttest, finding significant effects on all but academic competence. The intervention group scored significantly lower than the control group on aggression and general behavioral problems and significantly higher on self-regulation, emotional regulation, and teacher-child relationship. Effects were small (d = .15-.19). At the six-month follow-up, when the ratings were independent, only the aggression measure remained significant (d = -.25). Tests for moderation at both assessments found no differences in program impact by school poverty status.
Long-Term: The study did not include a long-term follow-up.
Evaluation Methodology
Design:
Recruitment: One preschool in a high-poverty, urban school district in New Jersey was selected for participation. All 274 children in the 18 participating classrooms of 3- to 4-year-old children were invited to participate, of those 80% (n=224) gave consent.
Assignment: All classrooms on one floor were used for the treatment condition (n=7) and all classrooms on the next floor were used for the control condition (n=11). Teachers were randomly assigned to classrooms within blocks to prevent confounding by factors such as teacher certification. Students were also randomly placed into classrooms. However, consent for students came after assignment, and lack of consent was higher for the control group (n = 40, 24%) than for the intervention group (n = 10, 9%).
Attrition: Since randomization occurred before consent was sought, many attritors dropped out before the first assessment point. Of the 224 who consented to participate, three families requested that their children be transferred from Tools to the district curriculum and one family requested the reverse; these four children did not participate in the study. One child in each group moved out of the district prior to assessment, four in each group were not pretested, and six children in the intervention group and five children in the control group had moved. One child in each group was not tested due to absences. At posttest there were 205 subjects (92% of those consenting, 74% of those randomized).
Sample: The sample consisted of 54% 4-year-olds and 46% 3-year-olds. There were fewer females (47%) than males (53%). Within the sample, 69% reported their primary home language as Spanish. The sample was largely Hispanic (92.6%), with a small number of Black (2.2%), Asian (3.7%), and multi-racial (1.5%) children.
Measures:
The study used:
In addition, teachers rated internalizing and externalizing behaviors with the Problem Behaviors Scale of the SSRS.
Finally, the study used the Early Childhood Environmental Rating Scale - Revised (ECERS), a measure of global classroom quality; Supports for Early Literacy Assessment (SELA), a study of the quality of literacy instruction; and the Preschool Classroom Implementation (PCI), a measure of use of scaffolding by teachers.
Analysis: The study conducted two analyses. The first was a regression analysis with posttest as the dependent variable and pretest, the intervention, and the child's primary language as independent variables. The second analysis was a hierarchical linear model with treatment at the classroom level used as a fixed effect to control for clustering.
Intent-to-Treat: Four students were excluded for transferring to the non-assigned condition. Otherwise, the analysis used all students with data.
Outcomes
Implementation Fidelity: Classroom observations showed that the intervention classrooms fully implemented all environmental features except writing, while control classrooms did not fully implement any of the key environmental features. Coding of videotapes showed Tools teachers, but not control group teachers, were found to direct significantly more "group talk" than "individual talk."
Baseline Equivalence: The study reported no significant differences between conditions at pretest for gender, ethnicity, primary home language, or outcome measures or on a parent questionnaire of prior child care attendance, frequency of being read to in the home, and parent-reported knowledge of numbers, letters, and colors.
Differential Attrition: Twenty percent of those randomized did not consent to participation. Analysis of variance between those children whose parents declined to participate and those who agreed showed no significant main effects of attrition or interactions between attrition and the intervention. Among those who did consent to participate, the study reported no significant differences between attritors and completors on demographic measures.
Posttest: The analysis using the hierarchical linear model (the more conservative analysis) reported a significant intervention effect only for the SSRS measure of internalizing and externalizing problem behaviors (p<.05). The OLPT measure of oral language skills of Spanish-speaking children was marginally significant (p<.10). However, the study reported that because the HLM results showed high correlation with the regression results, and the ICCs were less than .005, the HLM model may not be necessary. In the regression results, the SSRS measure remained significant, as were the Peabody Picture Vocabulary Test and OLPT measures (p<.05).
For classroom measures, the study reported a significant intervention effect for all three measures: Early Childhood Environmental Rating Scale (p=.003), Supports for Early Literacy Assessment (p=.001), and Preschool Classroom Implementation (p=.002).
Long-Term: The study did not include a long-term follow-up.
Study 3 may be the same sample as Study 2, but it is difficult to tell from the information provided.
Evaluation Methodology
Design:
Recruitment: One preschool in a high-poverty, urban school district was selected for participation. The sample includes 147 children in 18 participating classrooms. No other information on recruitment is provided.
Assignment: The treatment condition included 85 children and the control condition included 62 children. Stratified random assignment of teachers and assistant minimized confounds due to teacher characteristics.
Attrition: The study does not provide information on attrition in the sample.
Sample: The sample consisted of 54% 4-year-olds and 46% 3-year-olds. There were fewer females (47%) than males (53%). Within the sample, 69% reported their primary home language as Spanish. The sample was largely Hispanic (92.6%), with a small number of Black (2.2%), Asian (3.7%), and multi-racial (1.5%) children.
Measures: The study used the Flanker Task and the Dots Task, both of which are measures of executive function. The Flanker Task included the Reverse Flanker Task, switching the area of focus. The Dots task included the study used the Dots-Incongruent task, which required remembering a rule and inhibition, and the Dots-Mixed task, which combined congruent and incongruent tasks.
Analysis: The study conducted multiple regression analyses with age, gender, curriculum, and years in curriculum as independent variables.
Intent-to-Treat: The study does not provide information on those included or excluded from the analytical sample.
Outcomes
Implementation Fidelity: The study does not provide information on implementation fidelity.
Baseline Equivalence: The study did not measure outcomes at baseline and no information on demographics is provided.
Differential Attrition: The study does not provide information on attrition.
Posttest: The study reports significantly better scores for the intervention children, as compared to the control children, in the Dots-Incongruent Task, the Flanker Task, the Reverse Flanker Task, and the Dots-Mixed Task. Almost twice as many Tools as control children achieved greater than 75% correct on the Dots-Mixed Task. On the Reverse Flanker Task, the control children performed near chance (65% correct), but the Tools children averaged 84% correct
Long-Term: The study did not include a long-term follow-up.
Evaluation Methodology
Design:
Recruitment: The study recruited eligible school districts in two southern states. The sample included 60 schools in 4 districts in one state and one in the other. A total of 877 children consented to participate (498 in the treatment condition and 379 in the control condition). The consent rate in Tools classrooms was 88% and in control classrooms was 76%.
Assignment: The study assigned schools to conditions. The four smallest school districts were treated as individual blocks. The 22 schools in the large, urban district were divided into five blocks based on the number of classrooms in each school and the experience of the teachers. Within each of those blocks, half the schools were assigned to the treatment condition and half to the control condition.
Attrition: Of the 877 children who consented to participate in the study, 866 (98.75%) recorded at least one pretest measure. At posttest, 821 (93.61%) children recorded at least one measure, 811 (92.47%) children at one-year follow-up, and 778 (88.71%) at two-year follow-up.
Sample: The sample consisted of 56% males. The average age was 54.35 months. The sample was 40% White, 26% Black, 25% Hispanic, and had small numbers of Asian and mixed-race children.
Measures: The study used seven subtests of the Woodcock Johnson III Tests of Achievement, a direct measure of pre-academic skills. The study created a composite measure of academic achievement using the seven measures: letter-word identification, spelling, oral comprehension, picture vocabulary, academic knowledge, applied problems, and quantitative concepts. The study used direct measures to assess executive function. These measures were the Dimensional Change Card Sort task to measure attention shifting capabilities, the Copy Design task to measure attention and visual-spatial skills, the Corsi block-tapping task to measure working memory, the Peg Tapping and Hand-Toes-Knees-Shoulders task to measure inhibitory control.
In addition, the study used teacher reports of social behavior, self-regulation, and language ability. The Cooper-Farran Behavior Rating Scales were used to measure interpersonal skills and work-related skills and the Adaptive Language Inventory was used to measure children's comprehension and use of language in the classroom. Finally, assessors reported self-regulatory behaviors using the Self-Regulation Assessor Rating scale.
Analysis: The study conducted three-level nested regression models, with students at Level 1, classrooms at Level 2, and district randomization blocks at Level 3. Analysis included pretest, age at pretest assessment, interval between assessments, gender, English-language learner status, and individualized education program status as covariates.
Intent-to-Treat: The study included all cases for which there was data.
Outcomes
Implementation Fidelity: The study reports high levels of implementation fidelity.
Baseline Equivalence: The study reports no significant differences between conditions on outcome measures at baseline, however there were small but significant differences in race/ethnicity and age.
Differential Attrition: The study reports no significant differences in attrition by condition.
Posttest: At posttest, only the oral comprehension subscale showed a significant treatment effect, favoring the control condition. No other measures of pre-academic skills, direct reports of self-regulation, or teacher or assessor ratings of social behavior showed significant treatment effects.
Long-Term: At the one-year follow-up, the study reported significant negative effects on the letter word and quantitative concepts subscales, in addition to the composite achievement score for the treatment group as compared to the control group. At the two-year follow-up, the study reports a negative treatment effect on the spelling subscale for the treatment group as compared to the control group. At the one-year follow-up, only one direct measure of self-regulation showed a significant negative treatment effect and at the two-year follow-up the measures of copy design and the composite score of self-regulation showed negative treatment effects. Finally, the teacher and assessor ratings showed no significant treatment effects at either the one- or two-year follow-ups.
Evaluation Methodology
Design:
Recruitment: Participants were recruited from a network of childcare sites operated by YMCA Canada, specifically located in areas where local elementary schools were not scheduled to offer the full-day kindergarten (FDK) program. Based on these criteria, 20 childcare sites (one classroom per site) were eligible to participate. Children from these sites were eligible to participate if they were 3 or 4 years of age, had a sufficient grasp of English, and did not have any serious developmental challenges. Of the 260 children assessed for eligibility, 1 did not meet inclusion criteria and 3 declined to participate. Teachers were eligible if they were accredited Early Childhood Education teachers and were not expecting to take a leave of absence during the study period.
Participants were recruited in two consecutive cohorts due to high attrition mid-way through the program. Cohort A included 195 participants recruited at baseline, and Cohort B included 61 additional participants recruited from eight months after baseline, after randomization had already taken place.
Assignment: In Cohort A, childcare sites were randomly assigned to the intervention or control conditions (n=10 sites and n=106 children in the intervention condition; n=10 sites and n=89 children in the control condition). In Cohort B, children who were already attending school at the participating sites but were too young to participate at the time of recruiting transferred into study classrooms and received the curriculum that was already being implemented (n=42 in intervention classrooms, n=19 in control classrooms).
Attrition: Assessments were conducted at baseline (Cohort A only), 8 months after baseline, and post-intervention (15 months after baseline). Among those in Cohort A, attrition from baseline to Time 2 was 40% (n=78 dropouts of the initial n=195 participants). The authors claim the high rate of attrition was due to children leaving preschool altogether to attend the newly initiated full-day kindergarten program in the area. An additional cohort of n=61 participants was therefore recruited at Time 2, yielding an overall sample of n=178. Attrition from Time 2 to Time 3 was 11% in the overall sample (n=19 dropouts of n=178 at Time 2) and 11% among participants in Cohort A (n=13 dropouts of n=117 at Time 2).
Sample:
The sample consisted of preschoolers attending YMCA childcare centers in Ontario, Canada. There were 105 males and 90 females in Cohort A, whereas there were 33 males and 28 females in Cohort B. The mean age of participants in Cohort A in the intervention condition was 45.1 months and in the control condition was 45.9 months. The mean age of participants in Cohort B in the intervention condition was 42.4 months and in the control condition was 43.5 months.
Measures:
Primary outcome measures included executive function (two well-established behavioral tasks completed by children) and child behavior (two surveys completed by parents and teachers). Children also completed measures of vocabulary, literacy, and numeracy at baseline (Cohort A) or 8 months after baseline (Cohort B).
Executive function tasks included the Day-Night Stroop task, scored using conventional methods, and the Head-to-Toes task, scored using conventional methods as well as an additional method to bring the data "more in line" with the other data. This produced three analysis outcomes from these two tasks. Child behavior surveys included the teacher- and parent-reported Strengths and Difficulties Questionnaire, from which the Total Difficulties Score was used for analysis, and the teacher-reported Social Competence and Behavior Evaluation Scale, from which three subscales were used for analysis (anger/aggression, anxiety/withdrawal, and social competence). This yielded eight total outcomes computed for analysis: three child executive function outcomes (based on scores from the Day/Night and Head-To-Toes tasks), parent- and teacher-rated behavioral difficulties, teacher-rated anxiety/withdrawal, teacher-rated anger/aggression, and teacher-rated social competence. No reliability information was reported for the present study sample.
Analysis:
Main analyses were conducted using multilevel modeling (MLM) to account for nesting of the data, in which children (level 1) were nested within site (level 2). Due to the relatively low number of sites, Bayesian estimation was utilized and allowed for the inclusion of participants with missing data as well as non-completers. In examining the main effect of curriculum (i.e., intervention vs. control) on each of the primary outcomes, child sex, child age at final assessment, and initial levels of the outcome being tested were included as covariates. All analyses were conducted on Cohort A only as well as Cohorts A and B combined and did not change any pattern of results.
Intent-to-Treat: All available data were used in all analyses. Bayesian estimation was used to account for missing data.
Outcomes
Implementation Fidelity: Intervention adherence was measured using the Tools Implementation Checklist (TIC) completed by members of the research team at three time points (7, 11, and 14 months after baseline).
Baseline Equivalence: For Cohort A, teacher-rated anxiety/withdrawal was significantly higher among children in the intervention condition at baseline relative to children in the control condition, though the effect size for this difference was small. No significant baseline differences were found between children in the intervention and control conditions for Cohort B.
Differential Attrition: For Cohort A, attrition rates were similar in the intervention group (44%) and the control group (49%), and attrition did not vary by age or any of the baseline outcome measures. For Cohort B, attrition rates were similar in the intervention group (12%) and the control group (5%), though whether attrition was related to age or baseline outcome measures was not examined for this cohort.
Posttest: No significant intervention effects were found for any of the primary outcomes tested. Moderator analyses revealed no evidence that intervention effects varied as a function of children's initial language ability, but effects did vary by initial hyperactivity/inattention, such that children in the intervention condition with high initial hyperactivity/inattention showed significantly greater improvements in one of three executive function measures compared to children in the control condition with high initial hyperactivity/inattention.
Long-Term: Not examined.
Evaluation Methodology
Design:
Recruitment: Participants were kindergarten students from public schools in Vancouver and Surrey, Canada. All schools in these two school districts were invited to participate, and both principals and teachers had to agree to implement the program if selected for the treatment group. Kindergarten classes in 20 schools and with 352 kindergarten students were initially included in the study.
Assignment: Prior to randomization, schools were matched into pairs on kindergarten teacher's years of experience and training, student ethnicity, percent of students eligible for free lunch, and the language used at home. The 20 schools were then randomly assigned within pairs to either the intervention group or control group. The control schools appear to have continued with their usual curriculum of utilizing whole group activities, rewards, and time-outs, which were not used in the intervention group. No classrooms in the control group utilized activities resembling those in the intervention group. However, in order to treat the control and treatment teachers comparably, control teachers were offered 3 days of workshops on topics of their choosing, were provided an assistant for 90 minutes daily, and received an allowance of $1,000 to purchase classroom supplies.
Assessments/Attrition: Assessments occurred at baseline (pre-treatment) in the fall of the kindergarten year, within the first month of school. A posttest came 8 months later, at the end of the program in the spring of the kindergarten year. Two schools that were part of a matched pair dropped out of the study a couple of months into the school year, and were not replaced by other schools, resulting in nine intervention schools (student n = 172) and nine control schools (student n = 180). No other information was provided regarding students lost to follow-up. It appears from the text that incomplete teacher reports on students led to missing data, but the extent of the loss was unclear.
Sample: Across all teachers, the mean years of teaching was 15.5 with a range between 1-29 years. The mean years of teaching kindergarten was 7.5 years with a range between 1-15 years. Across all classrooms, approximately 19.5 students were in each class, 51% of students were female, and the average age at baseline was 5.2 years.
Measures: The primary outcomes of reading and writing skills were measured using British Columbia's objective, standardized assessment tools. However, the study gave no details on the tool or on who did the scoring. Otherwise, several teacher-reported measures of student attitudes and behaviors were obtained: social inclusion and other prosocial behavior, attention-regulation, and self-control. Kindergarten teachers both rated the children on these behavioral and attitudinal measures and delivered the program. Additionally, teachers' feelings about teaching were measured based on a 10-point scale with 1 being excited about teaching and 10 being exhausted and burned out.
Analysis: The effect of the intervention was tested using either a Poisson loglinear model when the outcome was skewed or a binary logistic model when the outcome was dichotomous. GEE models adjusted for nesting of students within schools, and a probability level of .025 was used to adjust for multiple tests. The models controlled for children's English as a Second Language (ESL) status, free-lunch status, and teachers' years of experience. However, only ESL was used in the formal, reported analyses.
Intent-to-Treat: The primary analysis appeared to follow an intent-to-treat approach that used all available data, but the study lacked detail on the original and final sample sizes.
Outcomes
Implementation Fidelity: Not examined.
Baseline Equivalence: The study presented baseline condition means for teacher experience and classroom demographics in Table 1. In 16 tests, two were significantly different. The control group had a larger percentage of ESL students and a smaller percentage of lower-income students compared to the intervention group. For some outcomes, the authors noted that the conditions were similar at baseline but did not present full significance tests for all outcomes.
Differential Attrition: At the school level, one intervention and one control group from the same matched pair were dropped, limiting potential bias related to assignment. At the student level, no formal differential attrition analyses were presented, but it is possible that attrition was low.
Posttest: The analyses showed significant condition differences in 14 of 15 tests which spanned over reading (OR = 0.95-2.67), writing (OR = 2.54-8.70), academic improvement, social inclusion and prosocial behavior (OR = 1.01-2.45), attention-regulation (OR = 2.00-2.14), self-control during unsupervised work (OR = 3.00), and teachers' excitement to return in the Fall (OR = 2.75). There were no differences at the posttest between students in mathematic academic improvement.
Long-Term: Not examined.
This study examined only the "play" portion of the intervention and, thus, did not test the full Tools of the Mind program.
Evaluation Methodology
Design:
Recruitment: Head Start centers that are part of a larger Head Start Cares program volunteered to participate in the program. To select a diverse sample, the study included centers in 10 states and 4 regions of the country. The study separately examined 3- and 4-year-old children in 4-year-old and mixed 3- and 4-year-old classrooms. Among 4-year-olds, 79% of parents consented at baseline to have their children participate, and 90% consented by the posttest.
Assignment: After being blocked on race/ethnic composition and part-day/full-day schedules, centers were randomly assigned (N = 104) to one of four conditions: Tools of the Mind, a control, and two other interventions. Analysis was then conducted with the separate interventions and, for 3-year olds, with a treatment group that combined all three interventions. The control group included 220 3-year-old children and the Tools of the Mind treatment group included 241 3-year-old children. For 4-year-old children, the control group consisted of 621 children and the Tools of the Mind treatment group consisted of 678 children.
Attrition: For the study of 3-year-olds, the child pretest occurred in the fall and the posttest in the spring. The study reported approximately 12% attrition in the Tools treatment group and 19% attrition in the control group among 3-year-olds. However, Table B6 (page 69) lists 211 control children and 213 Tools children at baseline, while the analysis results in Table 8 (page 41) list the full sample of 220 control children and 241 Tools children.
For the study of 4-year-olds, the child pretest in the fall and posttest in the spring were supplemented by a 1-year follow-up in the spring of kindergarten. The design allowed for children who did not have consent at baseline to be added for the posttest and follow-up when parents gave consent late. Figure 3.1 thus shows 2,128 children who consented, were selected, and were in the center at baseline, and it shows 868 who were not consented or selected at baseline but were added later. The total equals 2,996. At posttest, the number remaining in the centers was 2,670 (89%). The study thus reported 11% attrition (p. 249). The study tracked 2,599 (87%) into kindergarten.
Note that the pretest occurred from September to December, possibly months after the program began.
Sample: Children in the 3-year-old sample were an average of 3.47 years old and roughly half female (50.46%). Children in the 4-year-old sample were an average of 4.42 years old and roughly half female (48.82%). No information is provided on race/ethnicity of the 3-year-old child sample. The sample of Head Start centers suggests socioeconomically disadvantaged participants.
Among 4-year-olds, the sample was 43% Hispanic, 33% Black, and 16% White. The sample shows evidence of being disadvantaged; 59% of participating children's families were eligible for the Supplemental Nutrition Assistance Program (SNAP) and the average income was $1,800 per month.
Measures: All measures of child outcomes reported for 3-year-olds came from teacher reports. The study used 1) the Behavior Problems Index (BPI), which measures the frequency, range, and type of children's behavior problems, and had scales for total problems, externalizing, hyperactivity, and internalizing; 2) the Cooper-Farran Behavior Rating Scales for work related skills and interpersonal skills, 3) Social Skills Rating System, which measures children's ability to regulate their own behavior, assert themselves to solve conflicts with peers, and cooperate, 4) Student-Teacher Relationship Scale, which measures the teacher's perception of the quality of his/her relationship with individual children. Finally, the study examined pre-academic skills, using the Academic Rating Scale of early language and literacy, mathematical thinking, and general knowledge skills.
The study of 4-year-olds supplemented preschool teacher reports with assessments done by "independent observers" and by kindergarten teachers who presumably did not know of group assignment. For child behavior outcomes, the study used Head-to-Toes, pencil tap, and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure executive function and used the Behavior Problems Index (BPI) to measure behavior regulation, and learning behaviors. The study used the facial emotions identification task, emotions situation task, challenging situations task, the social skills rating system (SSRS), and the Cooper-Farran Behavioral Rating Scales (CFBS) to measure interpersonal skills. For pre-academic skills, the study used the Woodcock-Johnson III Letter-Word Identification test of pre-literacy and literacy skills and applied problems (math), the expressive one-word picture vocabulary test (EOWPVT) of vocabulary, and the Academic Rating Scale (ARS) for early language and literacy, math, and general knowledge skills.
To measure teacher outcomes among both 3-year-old and 4-year-old samples, the study used the Adapted Teaching Style Rating Scale (Adapted TSRS) to measure classroom management, social-emotional instruction, and scaffolding, in addition to the Classroom Assessment Scoring System (CLASS), which measured emotional support, classroom organization, instructional support, and literacy focus.
Analysis: The two studies analyzed three-level models for children within classroom and classrooms within centers. The models included fixed-effects dummy variables for the blocks in which centers were randomized, random effects for centers, and controls for baseline outcomes.
Intent-to-Treat: All participants with available data were included at each assessment point, but the study did not follow children who left the centers.
Outcomes
Implementation Fidelity: The study did not discuss implementation fidelity for 3-year-olds. For 4-year-olds, the study stated that implementation fidelity for Tools of the Mind did not meet "satisfactory." Mattera et al. (2013) reported that the intervention provided teachers with materials, training, coaching, monitoring and technical assistance. Coaches spent 90 minutes in each classroom observing and then meeting with teachers an average of 3 times per month. Monthly fidelity logs were completed by coaches where teachers received a fidelity rating on a 1 (low) to 5 (high) scale. Teachers on average received a rating of 2.97 from coaches on the fidelity scale. The study also reported that fidelity improved over time.
Baseline Equivalence: Tests for child age, gender, and outcome measures at baseline in Table B6 found one marginal difference (p < .10) for 3-year-olds: the Tools of the Mind treatment group had a smaller share of girls than did the control group. However, the table includes only a subset of the cases. Tests for teacher and classroom characteristics revealed only one significant difference.
For the 4-year-olds, only one significant child difference (p < .05) emerged from about 30 tests (Table B4). Tests for teacher and classroom differences (Tables B2 and B3) showed only a few marginal differences.
For both 3- and 4-year-olds, the occurrence of pretests after the program start may have affected the baseline equivalence tests.
Differential Attrition: The study of 3-year-old children found lower rates of attrition among the Tools of the Mind treatment group than in the control group (p < .10) and reported (page 95) that higher-risk children were more likely to drop out. Specifically, children who left the centers had significantly higher levels of total behavior problems and externalizing and internalizing behavior problems than did those who stayed. An additional check examined whether differential attrition produced condition differences at baseline among those who remained at posttest. To quote page 97 for 3-year-olds, "Among children who did not leave the sample, there were no statistically significant differences ... between the Tools of the Mind-Play group and the control group." The results suggest that attrition did not produce an unbalanced sample.
For 4-year-olds, tests showed that children who left the centers were more likely to be male, be rated as having lower levels of social behavior, learning behavior, and pre-academic skills, and perform poorly on executive function and cognitive tests than those who stayed (Table Q.1.). Table B5 tests for baseline differences among the analytic sample and therefore reflects the influence of attrition. It shows one significant difference (for the same characteristic significant for the full sample in Table B4) and one marginal difference.
Posttest: For 3-year-old children, the study found no significant posttest differences for the Tools of the Mind classrooms as compared to the control group in the 12 teacher-reported social-emotional outcomes or the 3 teacher-reported pre-academic skill outcomes.
Among 4-year-olds, the study reported no significant intervention effects for either observer or teacher-reported pre-academic skills; 4-year-olds did show a significant intervention effect for the facial emotions and emotions situations tasks (p<.05), but no effects for measures of executive function, behavior problems, learning behaviors, problem solving, or social behaviors.
For teacher and classroom measures, the study did not find any statistically significant posttest differences for 3-year-olds (p < .05) on 17 measures of teacher practice between the Tools of the Mind treatment group and the control group. Of 14 teacher-reported measures of classroom climate, teachers in the Tools of the Mind treatment group reported significantly worse scores on behavior management as compared to the control group. Among 4-year-olds, the study found a significant intervention effect for a literacy focus in the classroom (p<.05) and scaffolding behavior among teachers (p<.05).
Long-Term: The study did not include a long-term follow-up for 3-year-olds and no significant effects were found at the one-year follow-up for 4-year-olds on 14 measures of behavior, academics, and grade retention/special education.