ADA Impact Measurement System Summary
The ADA Impact Measurement System (AIMS) was developed to support the Disability & Business Technical Assistance Centers (DBTACs) in evaluating the impact of their services on the successful implementation of the Americans with Disabilities Act (ADA).
AIMS allows each DBTAC to evaluate the impact of their technical assistance efforts with minimal cost and effort. For any evaluation to be of value, a certain level of rigor in the process and commitment to the purpose of the evaluation is necessary. A great deal of care must be taken in the development of both the evaluation instruments and the procedures for implementing the evaluation. This background material provides some of the history of the process used to develop these performance evaluation procedures.
The ADA Impact Measurement System (AIMS) was designed and developed by Meeting the Challenge, Inc. (MTC) under a supplemental grant to its ADA Technical Assistance Center (DBTAC) project, with help from all DBTACs nationwide. The system includes custom survey instruments based on performance indicators for the ADA Technical Assistance program and a central website to support the operation of the process.
The AIMS website is a password protected website that each DBTAC can access to set up their evaluation, complete on-line telephone surveys, and generate reports on the results of their evaluation. The National Institute on Disability and Rehabilitation Research (NIDRR) can also access the website to view individual DBTAC results as well as aggregate results for the entire country.
The primary goals of AIMS are to:
- Provide a tool that will help each DBTAC accurately measure the impact of their services,
- Utilize state-of-the art internet technology to simplify the evaluation process, and
- Provide valid evaluation data on the effectiveness of DBTAC performance to help NIDRR comply with the Government Performance and Results Act (GPRA) requirements for program accountability and outcome measurement.
The DBTACs can use AIMS to generate results that are useful for NIDRR to assess the overall effectiveness of the DBTAC program on ADA awareness, understanding and implementation. In addition, AIMS can be used by each individual DBTAC to evaluate the impact of their services within their own region of the country. This information can be invaluable in helping to plan the types of services that need to be provided in the future, as well as the groups to target to help increase ADA implementation efforts.
MTC began developing an evaluation program to be implemented nationally across the 10 DBTACs in December 1995. The goal was to develop, test, and recommend evaluation methods that could be implemented across all DBTACs.
MTC conducted a formal evaluation study to document desired DBTAC program impacts and to explore methods of evaluating DBTAC success. A set of survey instruments was designed to measure different aspects of DBTAC success with the least invasive and least labor-intensive methods.
After considerable review of DBTAC function, process, and objectives, four survey instruments were field tested to validate the questionnaire items and assess the implementation procedures. These instruments included an expert-level rating, a mail survey, a telephone survey, and a postcard survey. At the conclusion of the field test, the instruments and procedures were revised to reflect the lessons learned and presented to representatives from each of the DBTACs.
During this review, significant changes were made to both the content of the instruments and the administration procedures. The total number of instruments was reduced to three: the expert-level survey in which the information specialist rated caller knowledge immediately after a call, the postcard survey which addressed ADA impact and customer satisfaction, and the telephone survey which addressed the customer's awareness and understanding of the ADA, as well as their implementation actions. Additional categories of respondents and thus, additional sections for the questionnaire, were added as a result of the field test. The changes identified through the field test were instrumental in improving both the questionnaires and the administration procedures.
A second field test was conducted to finalize the revised instruments and determine the best methods of implementing them. The Great Lakes DBTAC was the host of the second field test. At the conclusion of this field test, the instruments were sent out once again for review and feedback from the DBTACs nationwide. An "Evaluation Task Force" was formed with volunteer representatives from several DBTACs. The Task Force worked out the final changes to the evaluation products and processes.
One significant outcome of the second field test was a reduction in the total number of instruments from three to two. The postcard and telephone surveys were retained as the best instruments for achieving our measurement goals. This field test helped to further refine the evaluation process, as well as identify important considerations for training users of the evaluation system.
A high priority throughout the evaluation development process was to utilize approaches that would allow each DBTAC to gather valid outcome data while minimizing the time and effort required by staff. Toward this end, the AIMS website was designed and developed.
The website takes advantage of state-of-the-art internet technology to minimize the effort required to carry out the evaluation activities. The AIMS website automates many of the time consuming aspects of performance evaluation and minimizes the costs associated with administering, collecting and analyzing valid outcome.
NIDRR clearly outlined the goals of the DBTACs:
- Provide information and technical assistance to employers and other covered entities, as well as persons with disabilities, in order to facilitate appropriate implementation of the ADA specifically in the areas of successful employment outcomes for individuals with disabilities and greater accessibility in public accommodations.
- Develop information resources, databases, reference guides, and expert consultant pools that will serve as resources for implementation of the technical assistance programs.
NIDRR recognized the need to objectively evaluate the extent to which the efforts of the DBTACs to provide information and develop resources were having the desired impact. To facilitate this evaluation, NIDRR funded the development of AIMS.
In addition, GPRA requires all federally funded programs to document their measurable outcomes. AIMS was designed specifically to allow the ADA technical assistance program to meet this requirement.
|
