Various types of evidence can be collected to demonstrate the impact that a program has had on participants. These include:

• Participant reaction: What did participants think about the program? Were they satisfied with the experience? Did it match what they have to do on the job? Do they see utility in what they learned?
• Learning results: What have participants learned at the conclusion of the program? What new skills can they perform? How have their attitudes changed?
• On-the-job performance results: How has participant job performance changed since the program?
• Business results: What impact has the program had on the organization’s strategic goals?

As a general rule, the more types of evidence that can be collected, the better, as the additional information will allow for a more complete picture of the program’s impact. What has to be considered, however, is how to gather the desired evidence, as some types are far more difficult and resource-intensive than others to secure. This leads to the development of measurement strategies, as explained in the next guideline.

It is not too difficult to think of ways to measure participant reactions (the oft-maligned “smile sheets” come readily to mind) or learning results (think of all the different types of written and/or performance tests you’ve ever encountered).

Measuring on-the-job performance results is more of a challenge. It may be accomplished by having trained raters observe participants and evaluate their performance against established criteria. Alternatively, participants and/or their managers may be asked about the impact of the program on their performance. Another strategy would be to identify indicators of individual performance that relate to the desired outcome such as number of customer calls completed; average revenue per month, etc. The organization may already be collecting these indicators as part of a customer relationship management system or sales tracking system.

Measuring business results presents the greatest challenge. Whereas the other types of evidence were collected from individual program participants, business results evidence exists at the organizational level. It is necessary to think broadly about business results and how they might be expressed; for example, time to launch a new product, time to sale; quota/revenue attainment; percentage of new business developed; customer satisfaction data; repeat business; employee retention, etc. If mechanisms are not already in place to collect such evidence, it may require a significant investment of time and resources to develop appropriate measures.

If at all possible, information should be collected about the status quo before the performance solution has been implemented. Ideally, this information will be collected using the same measures and metrics as those selected to evaluate the program. Particularly in the case of business results, the story that can be told about program impact is far more compelling when it includes comparative data. The following table illustrates a sample program evaluation plan.

Category of Evidence	Evaluation Questions Addressed	Suggested Measurement Strategies	Suggested Uses of Results	Resource Investment
Participant Reaction Measured at the time that the program is provided.	How well did the program match what participants do on the job? How satisfied were participants with the program experience (content, learning design, delivery medium, etc.)? How valuable did the program seem to participants? How do participants view the potential utility of what they have learned?	Participants complete a questionnaire at the conclusion of the program.	Demonstrate level of participation in the program. Validate job-relevance of the program. Determine if modifications are needed in the program or its implementation.	Low
Participant Learning Measured at the time that the program is provided.	What have participants learned as a result of the program? What can learners do as a result of the program? How have participant attitudes changed?	Participants take a written or performance test based on the program. (This test may also be given before the program, to collect comparative information. Such a pre-test would not be used if the content of the program is entirely new to participants.) Alternatively, participants complete a questionnaire indicating their perceptions of what they knew before the program and what they learned as a result of the program. The questionnaire may also ask about participant attitudes, if they were a focus of the program.	Demonstrate whether participants have mastered the content of the program.	Low
Participant Job Performance Measured after participants have been back at work for a period of time	How well are participants using the knowledge and skills that were learned in the program? How has participant job performance changed since the program? What roadblocks have participants faced in transferring their learning to the job? What additional training do participants need?	Participants are observed on the job and rated against established standards for successful performance Participants and/or their supervisor or peers are interviewed about the impact of the program on their performance, roadblocks, and additional program needs. Indicators of individual performance relevant to the program outcomes are collected. These might include: Average value of sales Number of executive-level contacts Percentage of repeat business	Demonstrate whether participants have been able to successfully apply what they learned. Determine potential roadblocks to successful performance. Determine additional program needs.	Medium
Business Results Measured after program has been in place for a period of time.	What impact has the program had on the organization’s strategic goals? What environmental or cultural factors may have influenced the impact of the program (for example, management support, business reorganization, peer pressure, etc.)?	At the time that the program is developed, business decision makers have been asked what success should look like and to identify the key business indicators that are measures of that success. Possible success indicators might include: Reduced time to market Improved customer satisfaction ratings Increased employee productivity Cost reductions Increased revenue Determine if there are measures already in place to for these indicators. If they are, compare baseline results (before program is implemented) with results after program has been fully implemented. If measures are not already in place, determine whether there are resources available to support the development and implementation of such measures. Ensure that the measures are in use in time to collect baseline results.	Demonstrate impact of program on the organization’s strategic goals.	High