TRAINING EVALUATIO SIXTH LEVEL

Beyond the Fifth Level
The Forgotten Dimension
Session Number TU 315
5/18/2010 4:00 PM - 5:30 PM

www.institutomvc.com.br

Speaker’s profile
JOÃO BAPTISTA VILHENA
Partner of the Instituto MVC, a Brazilian consulting company. Twenty nine
years of professional experience in the areas of T&D, Management,
Marketing, Negotiation, Sales & Distribution. Author of 5 books. Speaker
of ASTD in 2006 and 2007.
vilhena@institutomvc.com.br

LUIZ AUGUSTO COSTACURTA JUNQUEIRA
Partner of the Instituto MVC, a Brazilian consulting company. A
Management Education Consultant with thirty five years of experience.
One of the brazilian pioneers in Management Negotiation . Author of 8
books. Speaker of ASTD in 2007.
costacurta@institutomvc.com.br


Objectives
SHARE with you a different approach to evaluate
training programs
DISCUSS how to go beyond the traditional ROI analysis

Our intention is not to
Deemphasize the importance of the traditional 5 levels
Ignore the importance of the participants of the training
program


Agenda
• The challenge of evaluating something
• Factual data
• How to know the root cause of your client
• Process design
• Evaluating the boss opinion
• Wrap-up


In teams of 5 or 6 (closest to you)
You have 5 minutes to put yourselves in the audience
shoes and discuss 3 criterias to evaluate a samba
school parade


First of all, keep in mind that

To evaluate something it is important
• Know why you will evaluate (objective)
• Know what you will evaluate (criterias)
• Know how you will evaluate (tools)


Samba schools evaluating criterias
Enredo (plot or theme)
This is the general theme that motivates the whole presentation of a
School. It has to be clear and understandable. Enredos change from year
to year.
Samba-enredo
Samba is a general denomination for a type of rhythmic music derived
from the African influence in Brazil and typical of urban Rio. It's meant to
be sung and danced. Samba-enredo is a samba-song that tells a story, a
plot derived from the main theme of the School.
Bateria (percussion)
It's the pulse, the beating of the School. It has to sustain the rhythm that
drives the School's song and dance. Each Samba School has from 250 to
350 percussionists.


Porta-bandeira e mestre-sala (the flag bearer and the master of ceremony)
This couple is of great importance for the School. The School's flag bearer
is a woman and her male scort is the master of ceremony. She carries the
flag and he shows his dance abilities. Their special choreography must
present all their lightness and nimbleness, sometimes almost ballet-like.
Comissão de frente e alas (front commission and wings)
Traditionally a School's board of directors marches at the head of the
parade, often accompanied by TV stars or sports idols. Next, comes the
comissão de frente, a group of 10 to 15 people in costume which opens
open the parade introducing the theme of the School and performing an
original choreography. The bulk of the other members comes behind
them, distributed in different alas (wings).Each wing is composed by
hundreds of people wearing the same costume, which is related to a point
of the Samba plot.


Harmonia (harmony)
Refers to the degree of synchronicity among the bateria (percussion), the
singing of all members and the dancing of the thousands of passistas (samba
dancers). Sambar means "to dance to the samba music".
Evolução (evolution)
Refers to the quality of the dance and to the way it evolves in tune with the
rhythm of the music and the beat of the percussion. This is translated by the
choreography, the passistas (samba dancers) skills and the general
enthusiasm of all School members.
Alegorias e adereços (allegories and ornaments)
The gigantic and richly decorated floats carry some of the figuras de
destaque, that is, celebrities or personalities to which the School pays
homage. Usually they are famous TV people or sports idols invited to wear the
Schools costumes. There can't be more than 8 floats per School. Floats are
judged by their beauty and creative concept.

Ala das baianas (bahian ladies wing)
Traditionally each School has one. It usually comes in the middle or the
end of the parade. It is a wing with hundreds of women, usually elderly
ladies, dressed in round large skirts in remembrance of the Bahian former
slaves, who introduced many of the traditions of Rio Carnival. During the
year these ladies work for the Samba Schools sewing and making the
costumes.
Fantasias (costumes)
Costumes are judged by their originality and beauty. Their colors are as
usual the traditional ones adopted by each School.


See the chart below
T&D
Inputs Outputs
Training
program
The boss The boss

5 levels

The Training Flow

Inputs Some kind of performance gap
Process T&D programs and evaluation
Outputs Performance gap solution


The first question
How to be sure you know exactly
what is the boss’s problem?
• People knowledge
» Desire Classical T&D problems
» Power

• Process
• Structure T&D is not able to solve this
kind of problems

• Strategy

If you are sure you have a T&D
problem, the next step is design
the solution
• T&D people loves to design process
• T&D people are used to think the process is the
solution (because they believe in people
transformation)
• T&D people prefer the easier way
• T&D people knows how to deal with the target
audience

So we have finished the training
program. Now it’s time to verify if
we have achieved the objective
• The 5 levels of evaluation (reaction, learning,
behavior aplication, business impact and ROI) always
focus on the participant
• We won’t say never but it’s not common T&D people
came back to verify if the boss is pleased with the
training program (perhaps they don’t know, exactly,
how to deal with bosses)


As you probably have realized
• The sixth level is ask if the boss had achieved his
objetives using the training program as a tool (or
how important has the T&D program been?)
• To ask this question you have to be sure you have
done the right thing (and be prepared to listen that it
had not been enough)


Hello Houston, we’ve get a Problem!
• And what if the boss isn’t a person? (let’s suppose
your task was to create a sense of social
responsability)
• Which will be the best criterias?


How do you know the root cause
of the boss?
• Define clearly who is the boss (the one who will
evaluate your job)
• Analyse cause X effect
• Finding the root cause
• Avoid generalizations: everybody, nobody, many,
few
• Check with the boss if you have really put your finger
in the wound
• Be sure you know exactly what is the desired output
of the process

Designing the process
• Don’t take it for granted T&D is the best solution
• Avoid loop holes
• Be sure the participant knows what is he doing here
• Be prepared to use 5 level’s model to analyse
process results


Evaluating the boss opinion or
Going beyond the fifth level
• The ceteri paribus principle
• Isolating variables
• From general to specific
• Using the 2H5W method: one possible way


Ceteri paribus principle
• Ceteri paribus is a latin expression. In a free
translation it means isolating other variables
• How to do it?


Using Control
Group

Trend Lines
Methods to isolate
the effects of
training
Participants
Estimate

Supervisors of
participants
estimates


Control groups
• A control group arrangement can be used to isolate
the training impact
• With this strategy, one group receives training, while
another similar group does not receive training
• The difference of performance of the two groups is
attributed to the training program.


Trend lines
• Trend lines are used to project the values of specific
variables if training had not been undertaken
• The projection is compared to the actual data after
training and the difference represents the estimate
impact of training


Volume of sales

Sales after the training

Training programs

Trend projection

time

Participants and Supervisors of
participants estimate the
training’s impact
• This method rests on the assumption that participants (and
their supervisors) are capable of estimating how much a
performance improvement is related to the training program.
• Because their action have produced the improvement,
participants (and their supervisors) may have very accurate
input on the issue.
• They should know how much of the change was caused by
applying what they have learned in the program.


Typical Questions to Estimate
• What percentage can this improvement be
attributed to the application of
skills/techniques/knowledge gained in the training
program?
• How reliable are these estimates expressed as a
percentage?
• What other factors have contributed to performance
improvement?


Isolating the effects of Training


From general to specific
Sources of barriers to training success
• Lack of reinforcement on the job
• Reentry problems (environment, ex.)
• Nonsupportive organizational culture
• Participants perception of impratical training
programs
• Lack of commitment of the trainer with the
program

The boss roles
• Collect baseline performance data
• Transform training topics into performance
standards
• Involve supervisors and participants in needs analysis
process and program planning
• Brief participants on the importance of training
• Review instructional content and materials
• Plan to participate in training sessions
• Encourage participants to attend all sessions


Wrap-up
• It’s not enough to evaluate the process
• Roles: the boss, T&D, participants
• Be sure you’ve “put your finger in the wound”
• Don’t take it for granted that T&D is the only solution
• Use 5 levels to evaluate the process
• Choose a way to evaluate the boss satisfaction
(which means the problem has been solved)
• Never forget the chicken principle


THANK YOU
See you in BRAZIL

If you wish to have a copy of this presentation, please
access www.institutomvc.com.br

TRAINING EVALUATIO SIXTH LEVEL

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (12)

Similar to TRAINING EVALUATIO SIXTH LEVEL

Similar to TRAINING EVALUATIO SIXTH LEVEL (20)

More from INSTITUTO MVC

More from INSTITUTO MVC (20)

TRAINING EVALUATIO SIXTH LEVEL