An Overview of the Program Assessment Rating Tool (PART) and How it Works

Last week we introduced our blog series on the Program Assessment Rating Tool (PART), a questionnaire designed to assess and rank the effectiveness of federal programs. Since 2002, the Office of Management and Budget has used PART to examine hundreds of programs in various agencies in the federal government. To date, more than 1000 programs have been through the PART process, including 93 programs at the Department of Education.

How it Works

The PART questionnaire is comprised of approximately 25 standard questions, as well as a few additional questions tailored to the type of program being evaluated. Programs can belong to one of seven categories: direct federal; competitive grants; block or formula grants; regulatory; capital assets and service acquisition; credit; and research and development.

The questions are divided into four categories:

• Program Purpose & Design: Asks questions about the clarity of the program's purpose, the problem the program addresses, whether it is well-designed, and whether it is duplicative of other programs.
• Strategic Planning: Asks questions about whether there are annual and long-term performance goals with ambitious targets and timeframes, whether the budget is aligned with program goals, and whether there are independent, high-quality evaluations of the program conducted on a regular basis.
• Program Management: Asks questions about whether programs use performance information to manage the program, if people are held accountable for performance results, whether there are measures for cost effectiveness and strong financial management practices, and whether management deficiencies have been addressed.
• Program Results: Asks questions about whether programs achieve their annual and long-term performance goals, how the program compares to other similar programs, whether the program is cost effective, and if independent evaluations of the program indicate the program is achieving results. (See below for a list of all 25 questions.)

How Programs Are Scored

Most assessment questions are simple yes or no questions, however the Program Results section does allow for additional responses (yes, large extent, small extent, and no). "Not applicable" is also an acceptable response for some questions. The scores from the four categories are weighted to come up with a final score. Program Purpose & Design makes up 20% of a program's total score, Strategic Planning is 10%, Program Management is 20%, and Program Results is 50%.

The Office of Management and Budget, in coordination with federal agencies, scores each program on a scale of 0-100. This numeric score is then translated into a qualitative rating of "effective" (85-100), "moderately effective" (70-84), "adequate" (50-69), or "ineffective" (0-49). Programs that do not have long term objectives or performance information are rated "results not demonstrated."

The PART is often mistaken for a full research evaluation - a study conducted by researchers that measures program effectiveness. The PART is actually not a research study, but a rubric used to assess how a program is designed and implemented, and whether any evidence of effectiveness exists. However, full research studies can be used to answer PART questions regarding program results. Programs rated "results not demonstrated" are not necessarily ineffective, it just means that evidence does not yet exist to support or refute its effectiveness.

PART results are released with the President's Budget request each February and are intended to inform future program funding. Results are also used to shape program improvement plans. Programs can request to repeat the PART process and improve their ratings if they have demonstrated recent progress or have new information available.

Up next in the Ed Money Watch PART Series: PART results for Department of Education programs.

____

Basic questions in PART:

Section I. Program Purpose and Design

1.1: Is the program purpose clear?

1.2: Does the program address a specific and existing problem, interest, or need?

1.3: Is the program designed so that it is not redundant or duplicative of any other Federal, State, local or private effort?

1.4: Is the program design free of major flaws that would limit the program's effectiveness or efficiency?

1.5: Is the program design effectively targeted so that resources will address the program's purpose directly and will reach intended beneficiaries?

Section II. Strategic Planning

2.1: Does the program have a limited number of specific long-term performance measures that focus on outcomes and meaningfully reflect the purpose of the program?

2.2: Does the program have ambitious targets and timeframes for its long-term measures?

2.3: Does the program have a limited number of specific annual performance measures that can demonstrate progress toward achieving the program's long-term goals?

2.4: Does the program have baselines and ambitious targets for its annual measures?

2.5: Do all partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) commit to and work toward the annual and/or long-term goals of the program?

2.6: Are independent evaluations of sufficient scope and quality conducted on a regular basis or as needed to support program improvements and evaluate effectiveness and relevance to the problem, interest, or need?

2.7: Are Budget requests explicitly tied to accomplishment of the annual and long-term performance goals, and are the resource needs presented in a complete and transparent manner in the program's budget?

2.8: Has the program taken meaningful steps to correct its strategic planning deficiencies?

Section III. Program Management

3.1: Does the agency regularly collect timely and credible performance information, including information from key program partners, and use it to manage the program and improve performance?

3.2: Are Federal managers and program partners (including grantees, sub-grantees, contractors, cost-sharing partners, and other government partners) held accountable for cost, schedule and performance results?

3.3: Are funds (Federal and partners') obligated in a timely manner, spent for the intended purpose, and accurately reported?

3.4: Does the program have procedures to measure and achieve efficiencies and cost effectiveness in program execution?

3.5: Does the program collaborate and coordinate effectively with related programs?

3.6: Does the program use strong financial management practices?

3.7: Has the program taken meaningful steps to address its management deficiencies?

Section IV. Program Results/Accountability

4.1: Has the program demonstrated adequate progress in achieving its long-term performance goals? 4.2: Does the program (including program partners) achieve its annual performance goals?

4.3: Does the program demonstrate improved efficiencies or cost effectiveness in achieving program goals each year?

4.4: Does the performance of this program compare favorably to other programs, including government, private, etc., with similar purpose and goals?

4.5: Do independent evaluations of sufficient scope and quality indicate that the program is effective and achieving results?