Performance Evaluation White Paper

Background

Government Performance and Results Act

Everyone is interested in getting a good return on their investment. To understand the value of a project we must not only demonstrate that the activities of the project have been carried out in a timely way and that those activities have reached the desired audience, but more importantly we must also be able to demonstrate that the activities have had the desired outcome. Attachment 1 presents a framework of performance measurement that addresses this range of output and outcome.

In 1993 the US Congress passed the Government Performance and Results Act (GPRA or the Results Act). This legislation is part of the federal government's recognition that government must be run in a more business-like manner. The federal government must be accountable to its citizens not just for having done something but also for whether or not the program accomplished the outcomes for which it was established.

The Results Act requires every federal agency to identify its mission and goals as well as how these will be accomplished. More importantly the agencies are required to carefully specify how they will measure the performance of their program against accomplishment of the mission and goals. The key unspoken term here is "results." Ultimately, the Congress will use the agencies' reports of performance against stated outcome goals to guide funding decisions. Thus, programs that do not or cannot prove that they are achieving their desired outcomes will not be funded.

The first step of the process is the development of Strategic Plans by each federal agency. These plans were due in final form to the Congress and the Office of Management and Budget on September 30, 1997. The Strategic Plans are required to contain six key elements: a mission statement, long-term strategic goals, approaches to achieving strategic goals, linkages between goals and approaches, outside influences on the goals, and a program evaluation methodology.

The Congress and OMB anticipated that perfecting the Strategic Plans as well as the performance evaluation process would take several planning cycles. However, an analysis of the Strategic Plans of 27 agencies in June 1997 found that all lacked one or more of the required elements and further that the content related to the various elements was, in many cases, weak.

In particular, the GAO study found that "16 plans did not discuss program evaluations, and the discussions of program evaluations in the remaining 11 plans lacked critical information, such as descriptions of how evaluations were used in setting strategic goals and schedules for future evaluations." The GAO study further reported that "fewer than one-third of managers in agencies reported that results-oriented performance measures existed for their programs to a great or very great extent."

The GAO has noted that program evaluation are a key component of results-oriented management. This acknowledged level of importance coupled with the relative lack of program evaluation coverage in the agencies' Strategic Plans creates a tremendous opportunity for companies that can assist agencies in developing and implementing program evaluation methodologies.

Program Impact Evaluation

In many government programs results are readily observable. In these cases performance measurement, defined by GPRA as the on-going monitoring and reporting of program accomplishments, is all that is needed to demonstrate program results. In other cases where results are less observable, program evaluations are required to collect necessary information about how well a program is working.

When a program is designed or intended to produce change as a result of program activities, program outcome evaluations are required. In this type of evaluation the degree to which desired outcomes or intended results were achieved are assessed.

In the cases where the desired program outcomes may be influenced by events and activities outside the specific government program, a Program Impact Evaluation must be conducted. The goals of such evaluations is to establish a causal relationship between the activities of the program and the outcomes measured. Clearly, this type of evaluation is the most challenging.

The GAO report indicated that federal agencies' capacities to conduct any program evaluations was extremely uneven. Obviously, their capacity to conduct the most complex of these, the program impact evaluation, is the most stressed. Again this point out the opportunity for private companies, in general, and MTC, in particular, to offer services in this area.

Technical Assistance Programs

The federal government conducts numerous Technical Assistance programs. These are programs whose activities are directed outward to the general public or a focused segment of the public and provide necessary information to that constituency. In such programs, the underlying outcome goal of the program is not simply to distribute a large amount of information, but rather to perform activities that create desired changes within the constituency. For example, the ADA Technical Assistance Program provides information about the Americans with Disabilities Act specifically to encourage the full implementation of the law throughout the US.

In most technical assistance situations, the federal agency program is not the only activity attempting to influence public opinion and action. The states, private organizations, and other federal agencies may also have program in the same area. This is certainly the case with the ADA.

Thus, assessment or evaluation of technical assistance programs falls into the area of program impact evaluation. The evaluation must determine not just that change occurred and that the change was the desired change, but also that the change was as a result of the technical assistance program's activities.

The MTC Recipe for PIE

MTC has developed a survey-based methodology for conducting program impact evaluations in technical assistance projects. This capability was developed specifically for the ADA Technical Assistance Program, but contains a generalizable methodology and automated support system which can be applied to other technical assistance programs. Attachment 2 provides a brief overview of the ADA Impact Measurement System (AIMS). In the following sections we describe the key elements of the MTC Program Impact Evaluation (PIE) methodology and support system.

Developing Performance Indicators

Perhaps the most critical element of the PIE methodology is the development of performance indicators. A performance indicator is defined in the Results Act as "a particular value or characteristic used to measure outcome or output." For our purposes, measures of outcome are most important.

The development of performance indicators provides the linkage from the stated performance goals to the program impact evaluation.

Customer Surveys to Determine Impact

From the Performance Indicators one or more surveys are developed that facilitate the collection of information from the program's targeted constituency. Since we are attempting to associate a causal relationship between program activities and outcomes rather than simply that outcomes have occurred, these surveys are directed to members of the targeted constituency who have interacted with the program.

The PIE Support System

Program Impact Evaluations are typically administratively and labor intensive activities requiring highly trained analysts to ensure accurate and meaningful results. MTC's PIE methodology eliminates much of the overhead normally associated with such evaluations through an automated PIE Support System.

The PIE Support System consists of a server-based software system which is accessed by standard internet browser software such as Netscape or Explorer. This architecture greatly reduces technical support issues and virtually eliminates training requirements for those who already "surf the net."

The PIE Support System provides the evaluation administrator with tools to manage the evaluation process from assigning and monitoring data collection tasks to analysis and reporting of evaluation results. The system also supports individual analysts as they collect and enter survey data either from direct telephone surveys with constituents or from mail in survey forms.

The system allows data collection and management to take place from geographically distributed sites since it is accessed through the internet. Thus, it is ideal for programs where technical assistance is provided by several separate information specialists or from separate sites. Results can be analyzed by each entry point or aggregated across the entire program.

Identification of Performance Measures

Performance measurement must be considered across several dimensions. The Performance Measurement Framework shown in Figure 1 was originally developed by George Teather and Steve Montague in Canada, but it has gained broad acceptance throughout the United States government program offices.

Figure 1 - The Performance Measurement Framework

The figure depicts the various relationships that exist between resource utilization and consequential results. In the figure, "resources" are used to perform "activities" and create "output." This is "how" objectives are achieved. The goal is for these activities and outputs to "reach" the target populations, the "clients," with or without the support of various "partners." This answers the question "who" is affected by the activities and outputs. As a result of the activities and outputs, we want the target population to behave differently. These "immediate impacts" are "what we want." Over the longer term we hope to achieve systemic change. This "long term impact" is "why" we perform the program.

Relation to Outcomes and Impacts

As can be seen in the figure, performance indicators must be developed that relate to each column of the framework. The outcomes or impacts sought in the "resources" column are completion of schedule milestones in the Program Plan of Operation. Performance indicators for the "reach" column of the framework must measure the degree to which the project outputs are capable of affecting the target populations. Measurements for the "results" must be in terms of the desired near- and long-term impacts on the target populations.

Objective Indicators

The performance indicators for activities and outputs should be captured in the milestones of the Program Plan of Operation schedule. Each milestone is an objective indicator whose completion can be easily identified and reported. Of course output-related performance has always been the easiest aspect of program performance to identify and measure.

Measuring program impact on target populations should be measured throughout the project by seeking input from representative samples of the target populations in regards to their acceptance of the program. This measurement is not an attempt to gauge the impact of the program on the target population but rather to judge to what extent the program is, in fact, reaching the target population.

The immediate impacts must be measured through careful surveying of the target population. The survey must carefully extract information from members of the target population who have been touched by the program to determine whether the individual being surveyed has received the desired impact of the program. This, of course, presupposes that the program has made an accurate assessment of the desired outcomes and impacts of the program on the target population.

The long term desired impact of a program is measured by examining the entire system of the target population to determine whether systemic change has occurred.

Introduction

The ADA Impact Measurement System (AIMS) was developed to support the Disability & Business Technical Assistance Centers (DBTACs) in evaluating the impact of their services on the successful implementation of the Americans with Disabilities Act (ADA).

AIMS allows each DBTAC to evaluate the impact of their technical assistance efforts with minimal cost and effort. For any evaluation to be of value, a certain level of rigor in the process and commitment to the purpose of the evaluation is necessary. A great deal of care must be taken in the development of both the evaluation instruments and the procedures for implementing the evaluation. This background material provides some of the history of the process used to develop these performance evaluation procedures.

What is AIMS?

The ADA Impact Measurement System (AIMS) was designed and developed by Meeting the Challenge, Inc. (MTC) under a supplemental grant to its ADA Technical Assistance Center (DBTAC) project, with help from all DBTACs nationwide. The system includes custom survey instruments based on performance indicators for the ADA Technical Assistance program and a central website to support the operation of the process.

The AIMS website is a password protected website that each DBTAC can access to set up their evaluation, complete on-line telephone surveys, and generate reports on the results of their evaluation. The National Institute on Disability and Rehabilitation Research (NIDRR) can also access the website to view individual DBTAC results as well as aggregate results for the entire country.

The primary goals of AIMS are to:

• Provide a tool that will help each DBTAC accurately measure the impact of their services,
• Utilize state-of-the art internet technology to simplify the evaluation process, and
• Provide valid evaluation data on the effectiveness of DBTAC performance to help NIDRR comply with the Government Performance and Results Act (GPRA) requirements for program accountability and outcome measurement.

The DBTACs can use AIMS to generate results that are useful for NIDRR to assess the overall effectiveness of the DBTAC program on ADA awareness, understanding and implementation. In addition, AIMS can be used by each individual DBTAC to evaluate the impact of their services within their own region of the country. This information can be invaluable in helping to plan the types of services that need to be provided in the future, as well as the groups to target to help increase ADA implementation efforts.

How was AIMS developed?

MTC began developing an evaluation program to be implemented nationally across the 10 DBTACs in December 1995. The goal was to develop, test, and recommend evaluation methods that could be implemented across all DBTACs.

MTC conducted a formal evaluation study to document desired DBTAC program impacts and to explore methods of evaluating DBTAC success. A set of survey instruments was designed to measure different aspects of DBTAC success with the least invasive and least labor-intensive methods.

After considerable review of DBTAC function, process, and objectives, four survey instruments were field tested to validate the questionnaire items and assess the implementation procedures. These instruments included an expert-level rating, a mail survey, a telephone survey, and a postcard survey. At the conclusion of the field test, the instruments and procedures were revised to reflect the lessons learned and presented to representatives from each of the DBTACs.

During this review, significant changes were made to both the content of the instruments and the administration procedures. The total number of instruments was reduced to three: the expert-level survey in which the information specialist rated caller knowledge immediately after a call, the postcard survey which addressed ADA impact and customer satisfaction, and the telephone survey which addressed the customer’s awareness and understanding of the ADA, as well as their implementation actions. Additional categories of respondents and thus, additional sections for the questionnaire, were added as a result of the field test. The changes identified through the field test were instrumental in improving both the questionnaires and the administration procedures.

A second field test was conducted to finalize the revised instruments and determine the best methods of implementing them. The Great Lakes DBTAC was the host of the second field test. At the conclusion of this field test, the instruments were sent out once again for review and feedback from the DBTACs nationwide. An “Evaluation Task Force” was formed with volunteer representatives from several DBTACs. The Task Force worked out the final changes to the evaluation products and processes.

One significant outcome of the second field test was a reduction in the total number of instruments from three to two. The postcard and telephone surveys were retained as the best instruments for achieving our measurement goals. This field test helped to further refine the evaluation process, as well as identify important considerations for training users of the evaluation system.

A high priority throughout the evaluation development process was to utilize approaches that would allow each DBTAC to gather valid outcome data while minimizing the time and effort required by staff. Toward this end, the AIMS website was designed and developed.

The website takes advantage of state-of-the-art internet technology to minimize the effort required to carry out the evaluation activities. The AIMS website automates many of the time consuming aspects of performance evaluation and minimizes the costs associated with administering, collecting and analyzing valid outcome.

What is the role of NIDRR in DBTAC evaluations?

NIDRR clearly outlined the goals of the DBTACs:

• Provide information and technical assistance to employers and other covered entities, as well as persons with disabilities, in order to facilitate appropriate implementation of the ADA specifically in the areas of successful employment outcomes for individuals with disabilities and greater accessibility in public accommodations.
• Develop information resources, databases, reference guides, and expert consultant pools that will serve as resources for implementation of the technical assistance programs.

NIDRR recognized the need to objectively evaluate the extent to which the efforts of the DBTACs to provide information and develop resources were having the desired impact. To facilitate this evaluation, NIDRR funded the development of AIMS.

In addition, GPRA requires all federally funded programs to document their measurable outcomes. AIMS was designed specifically to allow the ADA technical assistance program to meet this requirement.

Excerpted from the August 13, 1991, Final Priority.

Excerpted from the 1992 NIDRR Annual Report.