How can you measure the effectiveness of your training programs? Besides seeing that the participants had a good time with you, how can you measure their learning or change of behavior / skills because your training?

Thanks for the invite, I agree with specialties answers. 

The following is a summary of the four levels along with a few suggestions for implementing measurement at each level.

Level 1: Reaction:  To what degree participants react favorably to the training.

To measure reaction, develop ways to address some of the following questions:

• Did the seminar participants believe that the training was worth the investment of their time?
• What topics were most valuable; least valuable?
• Would they recommend this course to colleagues?

Level 2: Learning:  To what degree participants acquire the intended knowledge, skills, attitudes, confidence, and commitment based on their participation in a training event.

To measure learning, consider the following:

• What do you need/want to evaluate?  Was the goal of the training program to change skills, knowledge, attitudes, confidence and/or commitment?
• It is often useful to test participants in this area before they attend the training and then again after they attend the session.

Level 3: Behavior:  To what degree participants apply what they learned during training when they are back on the job.

To measure level 3, develop ways to address some of the following questions:

• Did the trainees put any of their learning to use?
• Are trainees able to teach their new knowledge, skills, or attitudes to other people?
• Are trainees aware that they've changed their behavior?

One of the most effective ways to assess behavior change is to conduct observations and interviews over time.

It is also important to keep in mind that the environment the trainees return to impacts whether or not they can apply what they learned.  If the culture, management and reward systems do not support behavior change, then trainees may not be able to apply what they have learned.

Level 4: Results:  To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement.

To measure level 4 results it is important to determine which outcomes are the most directly linked to the training program.  Then you’ll need to develop ways to measure these over time.

Here are a few potential benefits/outcomes to consider, that are often linked to training programs. (Please note: not every item on the list will apply to every training situation and this list is not exhaustive, there may be others that do not appear on the list.)

• Increased employee retention
• Increased employee engagement
• Increased productivity
• Increased customer satisfaction/retention
• Decreased errors and rework
• Increased sales/deepening of customer relationships
• Fewer customer complaints

(Source:  www.kirkpatrickpartners.com )

A couple of important notes about this model:

• Evaluation should always begin with level one, and then, as time and budget allows, move sequentially through levels two, three, and four.
• Information from each prior level serves as the foundation for the next level's evaluation.
• Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

I will be the first to admit calculating the ROI (level 4) on training is challenging for many reasons. However, that doesn’t mean we shouldn’t try!   As you consider what levels you would like to pursue, it is important to consider some of the following if you are going to go to level 4 (and to a degree level 3). 

1. The time and resources required to use levels 3 and 4. a.  For example, it may not be practical if your organization doesn’t have a dedicated training department.  b. In addition, you need to assess the system, tools, and processes your organization has in place to collect the data necessary to report on these levels.
2. The impact of other variables on all four levels.  Most organizations experience change in a variety of ways and the rate and pace of those changes can influence behaviors and results as much, if not more, than the training itself.  For example, the implementation of a new more user-friendly website may be the reason for an increase in customer satisfaction, rather than or in addition to the Customer Service Refresher training program all representatives just attended.

Each year globally many millions of dollars are spent on training. It would seem reasonable to establish just how effective this training has been if only to help justify the budget. Evaluation techniques are not new, indeed the most influential early work on evaluation was performed in the USA during the 1940s by Tyler. Probably the most important aspect of Ralph W Tyler’s work was the realisation of the importance of objectives in designing an effective school curriculum. This early methodology can be defined as the scientific/experimental approach.   The model emphasised the importance of knowing the educational standards of the individuals in the control group before the new education initiative took place and then determining the change that had taken place by measuring the difference in attainment after the educational intervention was completed. In other words the aim was to determine the effectiveness of the training by scientific means on the level of performance of the individual. Although there have been some critics of this methodology it has proved to be valuable in that organisations have been able to quantify the relationship between their corporate objectives (often expressed as key performance indicators (KPIs) today) and the training aims and objectives.   The main criticism of the scientific/experimental approach is that the methodology is unable to take into account the unexpected or unintentional learning that can take place within a dynamic and changing learning environment. Those readers who can remember reading about the original, and now famous Hawthorne experiments to determine the effects of illumination levels on production held at the Western Electric plant in Cicero, Illinois in the 1920’s and the later experiments also conducted by the National Research Council showed that human behaviour is indeed difficult to quantify exactly. Nevertheless the greater attention that was given to training methodology and in particular the effectiveness of management development during the 1960s and 1970s has led to much better understanding and the emergence of systems evaluation methodology.   As we will see this approach can be divided into two main schools; the narrow focused approach being attributed to Donald L Kirkpatrick in the early 1960s and the somewhat broader model advocated by P. Warr et al (The Evaluation of Management Training) known as the CIRO model. The CIRO model looked at context evaluation, input evaluation, reaction evaluation and outcome evaluation.   Donald L Kirkpatrick’s "Four steps to measuring training effectiveness”   This model uses four separate stages for the evaluation of the effectiveness of a training program.   The four stages are:-

• Reaction
• Learning
• Behaviour
• Results

Level 1   The first stage is about the reaction of the trainee to the training. This sort of measurement is concerned with how the trainees "feel” about the course. The usual course feedback sheets are an example of the Kirkpatrick level 1 evaluation. Most organisations do not do any more than this type of measurement and analysis. The drawback is that we do not really know if the trainee has actually learnt anything. What really seems to be being asked of the trainee was how "happy” were you with the course; hence the somewhat derogatory description that is often applied to the "happy sheets”! We will now look at the next level of evaluation concerned with Learning.   Level 2   Things can be improved by using a pre-test and post-test and comparing the results. The questions need to be objective and closely related to the course objectives (more about that later). In this way we can determine if the training actually delivered knowledge and this was understood by the trainees at the time. An organisation that does this can be confident that the trainee has actually learnt something at that time. Why do I make the point that we have to make the measurement and consider the learning at a certain time? Well because we do not know if the learning has had time to be internalised and become "concrete”. All too often trainees (and delegates at conferences for example) will have difficulty remembering what was in the course or seminar they attended a few days later let alone months later. There are various techniques that we can employ to improve the level of recall but basically "if we don’t use it we will lose it”!   Level 3   This is concerned with "behaviour”. By that we mean the measurable change in an individual as a result of their attendance on the training course. This is, in my opinion, the least we should be expecting from any training program. After all what is the point of spending money and using resources if the training does not effect some measurable change in the behaviour of the trainee?   Level 4   Kirkpatrick is now concerned with the training to determine if it has actually been translated into tangible benefits to the organisation. Quite simply has productivity and or quality been improved? Have the number of accidents or incidents been reduced? Has plant availability and or plant utilisation been improved? Has the morale of the workforce changed for the better? These are metrics which really have an impact on the "bottom line” and for that reason feature in the companies balance sheets and KPIs. We have to ask ourselves is this not the real reason for training? Training has to make a real difference in performance and effectiveness; this is tied closely to competence. Training has be proven to deliver results and be cost effective. It might sound simple but it is not for most organisations. The reason is that most do not have in place any system for measuring the improvement in competence of the individual; let alone a systematic approach to identifying the most effective means of assisting the individual to becoming competent.

His performance should improve.

Basically, there are lots of training programs which have intangible results, that can’t be measured. And what can be measured easier without using any tools to measure effectiveness of any training is that, the trainee himself showcases an improved and professional traits that is not seen before, and start doing something that is matured and with valuable substance.