Study: Never Too Late to Intervene in a Student's Education

If you've never heard of HBO’s “The Wire,” do yourself a favor and start watching now. For those unfamiliar with the show, each season focuses on a different facet of the city of Baltimore. Season Four concentrates on Baltimore’s public education system, and specifically zooms in on an 8^th-grade classroom in a struggling middle school. Early on, a first-year teacher struggles with his disruptive classroom, but things change when a researcher proposes an experimental program that places students with the worst behavioral problems and those who are furthest behind in a separate classroom for something resembling group therapy (See a clip).

Over the course of the academic year some of the students start to display signs of improvement and return to their regular classroom. The same first-year teacher reports fewer disruptions from the students and has less difficulty managing his classroom. Everybody wins.

What if this were really happening on the ground, but instead of segregating the most disruptive students to a different classroom, the students met in groups once a week for activities designed to enhance their social-cognitive skills? What if the students also met with tutors for an hour each day for remediation?

A recent paper from the National Bureau of Economic Research (NBER) looked at the effects of an academic and behavioral intervention in disadvantaged 9^th- and 10^th-grade males at a Chicago high school. The results are surprising and encouraging.

In less than a year, students in the program saw significant increases in their standardized math test scores and were rated more likely to graduate. The program also erased 60 percent of the school’s achievement gap between African-American and white children (as measured by the NAEP exam).

These results are particularly striking considering the demographics of the participants. The experiment group was composed of almost entirely African-American 9^th- and 10^th-graders and all but one student was eligible for free or reduced price lunch.

This program differed from previous interventions attempts in two ways. First, the authors hypothesized that disadvantaged youth need to develop the non-cognitive skills in order to succeed in a classroom environment. To address this, the treated students were placed in a program called “Becoming a Man” - a mentoring service for inter-city youth that has recently received praise from President Obama for its success. Over the course of the year the students met once a week in groups to develop emotional literacy, impulse control, and interpersonal problem-solving skills.

Second, the authors believed previous interventions did not recognize that disadvantaged youth are often several years behind their grade level. To address this problem, the researchers created an individualized tutoring program based on a model from Match Education. Each day, the students spent an hour in two-on-one tutoring where they spent half the time working on classwork and the other half on intense remedial math lessons. Students were randomly assigned to this program.

At the end of the experiment, students who did not receive the treatment scored in the 19^th percentile on math scores of city wide tests administered to all Chicago 9^th-and 10^th-graders. Those who received both the behavioral and academic treatments scored in the 34^th percentile. The improvement was roughly equivalent to three years’ worth of math instruction.

The program also had a sizeable (but not always statistically significant) impact on reducing the number of unexcused absences by more than a quarter. The reduced absenteeism may have been a sign that the students had become more interested and engaged in school.

Perhaps most exciting, the treatment had potential long-term benefits. Chicago Public Schools maintains an “On Track” graduation measure: it tracks students who have received five full-year course credits and have not accumulated more than one semester F grade in a core class. Those who are “On Track” have four-year graduation rates 59 percentage points higher than those who are not “On Track.” After this experiment, those in the program had graduation rates a full 14 percentage points higher than peers who did not receive the intervention.

The tutors in this experiment were recent college graduates and had no formal teaching credentials or previous experience. They were paid just $17,000 a year. Their success as tutors, the authors found, was a result of using a different set of skills than those required by the typical classroom instructor. The tutors could individualize lesson plans, and spend less time managing the classroom and more time developing relationships with the students to maximize time-on-task. The authors suggest that this type of tutoring can be thought of as extreme classroom size reduction where the risk of disruption is reduced with only two students to instruct.

The use of these individuals as tutors contributed to the program’s approximate cost of $4,400 per participant. For comparison, the Tennessee Star Class Room Reduction experiment had estimated costs of approximately $19,600 per participant.

It is worth mentioning that this program was designed to be implemented for a full academic year, but the experiment did not begin until November. That may mean that the impact of the intervention was understated.

Despite these results, lingering questions about the program’s effects remain. For example, the authors admit that they were unable to distinguish the individual effects of the behavioral and academic interventions from one another. Understanding which was more effective is important for schools interested in implementing the program but have limited resources. More importantly, it is as yet unknown whether the positive effects of the treatment remain after the students leave the program. A follow up assessment will be necessary to answer that question

All things considered, though, these are exciting results for one of the first intervention studies with a randomized control trial. These findings run contrary to the popular view that by adolescence, let alone high school, it is too late to substantially and cost-effectively intervene to improve student outcomes. Those assumptions may have been premature.