EVALUATION OF TRAINING

 

In common parlance, evaluation is the objective assessment of the past to facilitate better forecasting and controlling of the future. In the context of training, it is used to determine the effectiveness of a training programme. Hamblin defined it as “any attempt to obtain information (feedback) on the effects of a training programme, and to assess the value of the training in the light of that information”. Evaluation in whatever language it is defined should involve the following important elements:

 

1. Evaluation is a planned process;
2. It aims at improving the Knowledge and skill of the participants, changing his behavior in the organization, improving other key result areas of he organization such as cost of production, absenteeism and turnover rate as well as taking, decisions about the desirability, nature and content of future training.
3. It involves collection to information from the trainees (on both pre – training and post – training situation), from his superiors, his subordinates and peers.
4. Training is evaluated in terms of objectively variable standards or criteria.

 

 

LEVELS OF EVALUATION   REACTION LEARNING JOB BEHAVIOUR RESULTS

 

1. REACTION:

 

This reflects the feelings of the trainees about the training programme, its methods, contents, quality of the speaker, physical facilities of the programme, suggestion about what will make the programme more effective, etc.

 

1.2 Uses

1. The trainer gets the views of the participants about the methods, contents and nature of the training programme. This helps him to modify, revise and improve the subsequent part of the same programme or the subsequent programmes.

 

2. The participants get a common forum to ventilate their individual feelings which facilitates the trainers to understand the situation and take corrective measure so that communication gap is reduced.

 

1.3 Modus operandi for evaluating reactions

 

1. Objectives are to be set up in verifiable terms
2. Data analysis plan should then be decided.
3. Questionnaires should be designed and developed reflecting the major objectives.
4. Information about the honest reaction should be collected by making the questionnaire anonymous.
5. The question should be open-ended and multiple-choice but easy.

1.4 When to evaluate

A reaction may be evaluated either at the end of the training programme or at the check points during the programme.

 

1.5 Methods of tests used for evaluation of reaction

Well-designed questionnaire.

 

 

2. LEARNING:

 

This reflects the change in the level of knowledge and/or skills of this participants during the training program.

 

2.2 Uses

1)    The increase of knowledge and/or skill of training programmes of the participants in the specific areas of activities as a result of the programme is recognized by a large number of training institutes as a valid criterion for evaluating their efforts.

 

2)    By measuring the extent of transfer of learning the sponsoring organization comes to know how much knowledge and skill the employees could absorb through the training programme.

 

2.3 Modus operandi for evaluating learning

1)    The tests for learning should reflect the major issues that were discussed and taught in the training programme.

2)    The questions should be simple and objectives type e. g multiple choice, true – false.

3)    The question should be such that in answering them the participants need utilize their understanding of material and just rote memory.

4)    In order to compare the level of learning control group (not receiving the training) may be required for the test with the experimental group

5)    Proper rating of the evaluation results with the help of statistical methods of great significance.

 

2.4 When to evalute.

1)    Pre-training and post-training situatioins.

2)    At the cheek point.

 

2.5 Types of tests used for evaluation of learning.

Unusually two types of tests are used for evaluation of learning. They are:

 

1)    Paper and pencil tests

This is a very widely used method specially for these programme where principles and facts are taught.

2)    Performance tests

These type of tests are very easy to use for those areas of training where measurement of skill is relatively simple, e.g. typing speed and job instruction training to the supervisors.

 

3. JOB BEHAVIOR:

 

This type of evaluation seeks to determine the change in the job performance of the participants as a result of training.

 

 

3.1 Uses.

1)    The client organization who sponsors the training and spends both monetary and human resources on it gets a clear understanding about how much benefit it obtains out of the training in terms of positive change in job behavior.

2)    The training organizations come to know how much of the learning imparted by them is applicable to real work situation and what are the major problems in the application of the knowledge and/or skills gathered in the class room to the work place.

 

3.2 Modus operandi for evaluating job behaviour

1.    Identification of the major performance indices of the individual employee by which the evaluation has to be made, e.g. number of pieces produced (in case of production worker), frequency of complaints against the concerned officer, rate of absenteeism, decision making, preventing and handling grievances and initiating change.

2.    A benchmark study of the performance of the trainee has to be conducted before the training takes place.

3.    A study of the after – training performance has to be conducted allowing a gap of three to four months.

4.    The study should cover the following groups viz.the trainee, the immediate superior of the trainee, trainee’s peers; and trainee’s subordinate.

5.    A centered group (not receiving training) should be used.

 

3.3 Methods and techniques used for evaluation of job behaviour.

a.    A bench-mark study about pre-training job performance by well questionnaires/interviews.

b.    A Post-training study of the job behavior by well designed questionnaires/interviews’

c.    Group discussion in refresher course or evaluation session.

 

4. RESULTS

 

This types of evaluation seeks to assess the tangible results of the training programme on the level of organisational performance in terms reduced cost, higher productivity, improved safety, decrease in employee grievances and so on.

 

4.1 Uses.

1. The client organisations justify the cost involvement in training process by knowing how this has helped in achieving the organisational goals.
2. The training institutes justify the repetition of such programmes knowing its real impact in organization level.

 

4.2. Modus operandi for evaluation results.

1. Identification of the major indices of the organisational goals or results by which evaluation has to be made. These may include such indices as reduced costs, improved efficiency, increased profit, reduction in grievances, reduced turnover, improved quality, increased volume of output and improved capacity utilisation.
2. A comparative study of the organisational performance of before training and after – training situation should be carried out.
3. Analysis and communication of the evaluation results to the client organisations should be made.

 

4.3 Methods and techniques of evaluation of results

 

The following methods or techniques may use for evaluation of the achievement of results:

a.    Depth interviews with both the trainee as well as his superior.

b.    Mailed questionnaires.

Assessments and the Kirkpatrick Model

Using Assessments to Evaluate Training

Donald Kirkpatrick

§  Kirkpatrick developed a model of training evaluation in 1959

§  Arguably the most widely used approach

§  Simple, Flexible and Complete

§  4-level model

The Four Levels

§ Level I:  Evaluate Reaction

§ Level II:  Evaluate Learning

§ Level III:  Evaluate Behavior

§ Level IV:  Evaluate Results

§ Fifth level was recently “added” for return on investment (“ROI”) but this was not in Kirkpatrick’s original model

Relationship Between Levels

§  Each subsequent level is predicated upon doing evaluation at lower level

§  A Level 3 will be of marginal use, if a Level 2 evaluation is not conducted

Only by assessing each level can we yield actionable results

Types of Assessments Used at Each Level

Reaction - What Is It?

§ How favorably participants react to the training (“Customer satisfaction”)

n Collects reactions to instructor, course, and learning environment

n Communicates to trainees that their feedback is valued

n Can provide quantitative information

Reaction - What It Looks Like

§ Questionnaire - Most common collection tool

n Content

n Methods

n Media

n Trainer style

n Facilities

n Course materials

Reaction - How to Perform

§ Determine what you want to find out

§ Design a form to collect/quantify reactions

§ Do Immediately

§ Develop acceptable scoring standards

§ Follow-up as appropriate

Learning - What Is It?

§ Knowledge

§ Skills

§ Attitudes

Learning - What It Looks Like

§ Media used to measure learning:

n Text

n Voice

n Demonstration

§ Methods used to measure learning:

n Interviews

n Surveys

n Tests (pre-/post-)

n Observations

n Combinations

Learning - How to Perform

§ Use a control group, if feasible

§ Evaluate knowledge, skills, and/or attitudes before and after

§ Get 100% participation or use statistical sample

§ Follow-up as appropriate

Behavior - What Is It?

§ Transfer of knowledge, skills, and/or attitude to the real world

n Measure achievement of performance objectives

Behavior - What It Looks Like

§ Observe performer, first-hand

§ Survey key people who observe performer

§ Use checklists, questionnaires, interviews, or combinations

 

Behavior - How to Perform

§ Evaluate before and after training

§ Allow ample time before observing

§ Survey key people

§ Consider cost vs. benefits

n 100% participation or a sampling

n Repeated evaluations at appropriate intervals

n Use of a control group

 

Results - What Is It?

§ Assesses “bottom line,” final results

§ Definition of “results” dependent upon the goal of the training program

Results - What It Looks Like

§ Depends upon objectives of training program

n Quantify

§ Proof vs. Evidence

n Proof is concrete

n Evidence is soft

Results - How to Perform

§ Use a control group

§ Allow time for results to be realized

§ Measure before and after the program

§ Consider cost versus benefits

§ Be satisfied with evidence when proof is not possible

Summary

§ Level I:                    Evaluate Reaction

§ Level II:                  Evaluate Learning

§ Level III:              Evaluate Behavior

§ Level IV:            Evaluate Results

Readings to Consider

§  Kirkpatrick, Donald L. (1998). Evaluating Training Programs: The Four Levels. Berrett-Koehler Publishers.

§  Worthen, Blaine R, James R. Sanders, Jody L. Fitzpatrick (1997). Program Evaluation: Alternative Approaches and Practical Guidelines (Second Edition).Addison, Wesley, Longman, Inc.

§  Kirkpatrick, Donald L. (1998). Another Look at Evaluating Training Programs. American Society for Training & Development.

§  Sieloff, Debra A. (1999). The Bridge Evaluation Model. International Society for Performance Improvement.