Measuring What Matters in Quality Rating & Improvement Systems

For more than the past decade, states have worked to build quality rating and improvement systems (QRISs) to help rate and improve child care centers and preschool programs. The quality ratings, often displayed as symbols (such as stars), inform families about the quality of prospective child care centers. The ratings also help leaders identify programs that are not meeting quality standards set by the state.

But is a highly-rated pre-K program more effectively preparing children for kindergarten in comparison to preschool programs with lower ratings? Not necessarily, according to a new study published in Science Magazine.

Using two national data sets, the State-Wide Early Education Programs (SWEEP) study and the National Center for Early Development and Learning’s Multi-State Study of Pre-Kindergarten, researchers assigned ratings to about 700 pre-K programs (serving about 3,000 children) based on four indicators commonly included in states’ QRISs: staff education and experience levels, staff-child ratios and class size, learning environments, and family engagement. Researchers also added a new indicator of teacher-child interactions, as measured by the Classroom Assessment Score System (CLASS). CLASS measures interactions in three areas: emotional climate, instructional support, and classroom organization. For instance, CLASS helps to determine how well teachers engage children in deeper learning through classroom conversations. (Read more about CLASS – and a number of other observation tools – in Watching Teachers Work: Using Observation Tools to Promote Effective Teaching in the Early Years and Early Grades.)

Researchers compared the programs’ QRIS ratings to student learning outcomes on end-of-the year assessments in math, pre-reading, language, and social skills, accounting for factors including children’s beginning-of-year score, state location of the preschool program, and child, family, and pre-K program characteristics. While the measure of learning environments showed a small link to student learning, teacher-child interactions were mostly tightly linked to student learning outcomes.

Researchers simulated the rating of the programs under nine states’ QRIS models (LA County, California; Colorado; Palm Beach County, Florida; Iowa; Kentucky; Louisiana; New Mexico; North Carolina; and Tennessee). Another interesting finding: programs fared differently depending on their state’s system.

As shown in the chart above, under State C’s QRIS 85 percent of the studied pre-K programs would have earned the highest rating, while under State H’s system 73 percent of the programs would have earned the lowest rating.

One study limitation worth noting is that only state pre-K programs that serve 4-year-olds were included. Because of requirements that most state pre-K programs must meet, these do not comprise a representative sample of child care programs rated by QRIS. QRISs rate a diverse set of public and private child care centers that enroll children birth to 5.

Nearly 40 states have already fully implemented a statewide QRIS and more states are in the planning stages. This work will surely continue, because we know that better outcomes rely on more than just making sure children have access to preschool programs. To get a strong start, children need access to high-quality preschool instruction. With the Race to the Top– Early Learning Challenge competition, the Obama Administration emphasized tiered QRISs (ones that include tiered program standards to help differentiate the level of quality) and required states to validate that the tiers in the state’s TQRISs actually differentiate quality. But even RTT-ELC does not specify which indicators must be included. In an Education Week articleon QRIS earlier this year, Bob Pianta (study co-author) explained that “a preschool might not employ many instructors with a deep knowledge of child development, but because it serves hot lunches and offers ample space for nap cots, it might appear on the state list as a four-star program.”

The findings of this latest study, which are consistent with an earlier study from RAND, show that how teachers engage with children matters. Measures of teacher-child interactions should not just be included as an indicator in state’s QRIS, but also perhaps emphasized more than other input-focused indicators. Sara Mead(former director of the Early Education Initiative) writes on her blog over at Education Week about the barriers to incorporating measures such as CLASS into QRISs:

One reason that many states focus on simple input indicators and do not include interaction measures in their QRIS is that the latter are a lot cheaper and easier to assess than the former. Developing QRIS systems that include robust measures of adult-child interactions will require resources many states may be unwilling to provide at this point in time.

That’s a real concern, but so is continuing to invest a system that rates preschool programs based on factors that aren’t the best of measures of quality. Research supports the importance of low child-teacher ratios, small class sizes, and engaged parents. In fact, some of the inputs measured by state QRISs surely allow for meaningful interactions between children and their teachers as well as between children. Parents and policymakers need to know that those quality indicators are in place, but it is equally important for them to know how well pre-K programs prepare children for kindergarten. It might be expensive to include measures of interactions, but excluding them is downright wasteful.

Suggested Reading

Some Qu’s About Validating QRIS Using Learning Outcomes

The ‘Race to the Top’ Winners: Evaluating Quality Ratings Systems