By Joshua Stewart, Senior Researcher and Marc Brodersen, Managing Senior Researcher at Marzano Research and Debra Pacchiano, Vice President of Translational Research and Isabel Farrar, Research Associate at Start Early
When a tool such as a rubric, assessment, or survey is used to measure teaching practices or academic standards, validation is an important step to ensure users can trust the results. Marzano Research recently worked with the national nonprofit Start Early (formerly known as the Ounce of Prevention) to examine the validity of a survey developed to support the professional growth and development of early care and education (ECE) staff and administrators.
A growing body of research on ECE and quality improvement shows that strong site-level organizational conditions are key to realizing strong implementation of quality standards and continuous quality improvement in ECE settings. Yet most tools designed to support the professional growth and development of ECE staff and administrators focus primarily on classroom-level processes, creating a gap that could stifle improvements at the organizational level related to mindsets and practices around caregiving, teaching, and family engagement.
Researchers at Start Early developed The Essential 0-5 Survey to bridge this gap. The survey provides data and actionable information to administrators, teachers, and staff in ECE schools and centers serving preschoolers as well as infants, toddlers, and their families.
The Essential 0-5 Survey collects teacher, staff, and parent experiences and perceptions of school- or center-wide mindsets and practices aligned to Start Early’s evidence-based framework of essential conditions:
- Effective Instructional Leaders
- Collaborative Teachers
- Involved Families
- Supportive Environment
- Ambitious Instruction
- Parent Voice
Research conducted in both K–12 and ECE settings demonstrates the direct impacts these conditions have on quality practices and children’s immediate and longer-term success. The Essential 0-5 Survey builds on previous research and development efforts between Start Early and the University of Chicago Consortium on School Research, where the 5Essentials Survey and evidence-based framework of essential conditions were originally developed and tested in K–12 settings.
To test the relevance of The Essential 0-5 Survey for infant–toddler settings, Start Early conducted 88 cognitive interviews in English and Spanish with teachers, staff, and parents from Early Head Start Child Care Partnership programs in two states. These interviews then informed additional revisions to the survey. For a survey pilot study, the revised survey was translated into Spanish, Arabic, and Aramaic and was administered to more than 500 teachers and staff and 1,100 parents in California, Colorado, Florida, Illinois, Louisiana, Texas, and Washington, D.C.
Validation and Results
Using data from the survey pilot study, Marzano Research conducted a validation study to understand the following questions:
- Does the survey consistently measure the aspects of ECE organizational quality it was designed to measure?
- Does the survey distinguish sites with different levels of ECE organizational quality?
- Do users with different background characteristics, such as their primary spoken language or the age of the child served by the ECE program, respond to the survey items in similar or systematically different ways?
Pilot study results confirmed that the survey items intended to measure particular aspects of ECE organizational quality do group together as intended. The results also confirmed that, with few exceptions, each grouping measures a singular aspect of ECE organizational quality. Additionally, a majority of the survey items were found to contribute to the overall functionality of the survey. Only a handful of items were found to be redundant or to potentially address different aspects of ECE organizational quality.
These findings speak to the rigor of the survey development team’s process in adapting, writing, and vetting the survey items. Importantly, the findings also endorse the overall reliability and validity of the surveys for use in infant–toddler center-based settings.
However, preliminary analyses also showed that teachers and parents who completed the survey in Spanish or English responded differently to items in several sections of the survey. In some instances, Spanish speakers generally provided more positive responses to a selection of items. In other instances, English speakers provided more positive responses to a selection of items. Some differences to a selection of items were also found between teachers working with children from birth to age 2 versus teachers working with children age 3 to 5.
Next Steps
Start Early is using the results of the pilot study to revise survey items and conduct additional interviews with teachers, staff, and family members with infant or toddler children. Start Early will conduct a formal validation study in 2022. This study will include linking survey responses to additional quality outcomes of concern, such as children’s attendance and developmental progress and the quality of teacher¬–child interactions and family engagement.
Learn more about Marzano Research’s work supporting early childhood education systems and programs. If you’re interested in exploring the validation of a survey or other tool, reach out to Josh Stewart at Joshua.Stewart@MarzanoResearch.com.
