The South Dakota Department of Education recently chose Marzano Research to evaluate its 3-year grant project aimed at supporting the department’s efforts to better serve students during the COVID-19 pandemic and beyond. Our work with the department provides a great example of how program evaluation can support a project’s success. It also illustrates some questions you may want to ask as you decide whether your project could benefit from partnering with a program evaluator.
Re-imagining Remote Education in South Dakota: The ESF-REM Project aims to build the capacity of teachers, schools, families, and communities to support learners along a continuum of settings—from in-person to hybrid to fully online—while allowing for innovative approaches such as personalized learning and addressing students’ social and emotional needs. The project targets multiple points within the education system:
- Increasing teacher and school administrator capacity to deliver new forms of instruction.
- Supporting educator preparation programs in preparing preservice teachers to meet these new forms of instruction.
- Engaging families in their students’ educational process.
Dr. Cari Kennedy, project consultant for the South Dakota Department of Education, shared why the agency is investing in this effort. “Competency-based education provides students with the opportunity to grow and be an active, engaged participant in their education. It provides for time, choice, flexibility, and individuality,” Dr. Kennedy said. “The opportunities this grant has afforded for professional development will ensure our students are competitive in our global society and lead them into a bright future.”
In addition to supporting educators across the state, leadership teams from 30 schools will participate in a 3-year seminar series to support their development and help them pilot new, customized instructional approaches that address individual students’ needs.
Though not required by the grant, the South Dakota Department of Education chose to add an evaluation component to help ensure the ESF-REM Project’s success and to communicate the project’s successes to stakeholders.
As evaluators with a deep passion for the work we do, we understand that deciding to partner with a program evaluator isn’t always top of mind as you jump-start a new project. It can be hard to set aside funds for an evaluation if you’re unclear about your return on investment or if an evaluation isn’t a requirement of the project’s funding. However, not including an evaluation component may mean you don’t have the data you need to guide your program along the way, get the best results for your program, and ultimately report the results to your stakeholders.
In our experience, early engagement with a program evaluator and collaborative evaluation design are key to a successful program evaluation:
- Early engagement with a program evaluator helps determine measurable project outcomes, provides ongoing data to inform continuous program improvement, and ensures objective and unbiased analyses of these data.
- Collaboration between the evaluator and project leaders in designing the evaluation helps ensure the data collected are the most meaningful and useful to project stakeholders.
Early Implementation and Collaboration in Action
For the ESF-REM Project evaluation, Marzano Research and the South Dakota Department of Education came together early on to determine the project outcomes and what information would be most useful to inform project improvements. The overall goal of the project is to offer professional learning opportunities to over 2,500 educators so that they’re prepared to provide instruction remotely or in person and to customize their instruction to meet individual students’ needs.
Dr. Kennedy noted, “Early data will not only provide us with course corrections but also increase professional development content to improve teachers’ efficacy as they implement the strategies to create a more learner-centered environment in South Dakota classrooms.”
For the evaluation, Marzano Research will track the number of educators who enroll in and complete the professional learning opportunities as well as the number of new remote and customized learning options they develop. We’ll also measure educators’ confidence to teach in a variety of settings, use technology to support instruction, and customize instruction to meet individual students’ needs. We are assessing these outcomes at the beginning of the evaluation and will continue to track them over 3 years to measure change.
We’ll administer educator surveys and conduct focus groups to measure changes in educators’ knowledge and confidence to use the new instructional approaches addressed by the ESF-REM Project. Our team will also track the extent to which educators have incorporated what they learned into their instruction and shared this information with other educators. The information collected at the end of each professional learning opportunity will help us determine whether the project is meeting educators’ needs and is effective in increasing their knowledge and skills. The information will also help us provide recommendations for project improvement on an ongoing basis.
To develop the evaluation surveys and focus group questions, we leveraged local expertise. For example, we worked with the South Dakota Statewide Family Engagement Center to develop a family focus group protocol and trained the center’s staff to conduct the focus groups. This collaboration helped ensure the focus group questions used family-friendly language, and it prepared our partners, who have extensive experience engaging families, to collect information in a consistent and reliable manner.
The evaluation partnership is already paying off. An early look at evaluation data provided actionable information to support a course correction. Project leaders saw that some educators felt the content and time commitment of some activities didn’t align with their expectations. The leaders quickly worked to add clarity for educators on the time commitment of the online professional learning coursework. The leaders are also working to change some of the coursework so that it’s more applicable to educators who teach in the elementary and middle grades.
Deciding to Engage an Evaluation Partner
If you’re thinking about whether an evaluation partner can help you reach your goals, here are some questions to consider:
- How do I figure out if my program is working and making progress toward its goals? Do I need help prioritizing goals and making sure they are measurable and attainable?
- What data should I collect to determine whether my program is functioning as intended or whether I need to make a course correction?
- Do I have the resources and staff needed to collect and analyze the data? Or do I need a program evaluator to help me with this?
An effective evaluation increases the likelihood of program success and provides you with the data you need to show the impact of your program in your state, district, or school. An evaluation partner can help you make early course corrections, pinpoint your most successful strategies, and communicate your program’s effectiveness.
Visit our Services page to learn more about Marzano Research’s program evaluation services and approach.
Download the ESF-REM January – June 2021 Biannual Evaluation Report.
