Measurement and confidence in ODPresentation Transcript
How do we know what works? Ilmo van der Löwe CHIEF SCIENCE OFFICER iOpener Institute for People and Performance
“To measure is to know.”Lord KelvinPHYSICIST
• OD interven3ons must be measured – Did the interven3on have an impact? – Were the eﬀects were posi3ve or nega3ve? – What were the success factors?
A simple example• Ques3on: – Does training managers create more produc3ve workers?• Interven3on: – Train 10 managers to be beEer leaders• Measurement plan: – Measure the produc3vity of the managers’ direct reports before and aIer the training (a total of 400 people)
Plan #1 Pre-‐interven3on Post-‐interven3on measurement measurement Training Training put into prac3ce TIME • If direct reports are more produc:ve at work in the end, does it mean that the training worked?
Not necessarily...• Increased scores could be caused by: – Economy gePng beEer, local team winning championship, seasonal weather diﬀerences, a friendly new hire...• Decreased scores could be caused by: – Fear of layoﬀs, the coﬀee machine being broken, serious injuries to team members, recession...
Change over time• Outside factors other than training can change scores• Mere change in scores is not evidence of eﬃcacy – Measurement must take into account outside factors
Control group• Revised plan – Include a control group that is similar to the experiment group in all aspects, except the training • Ideally, same loca3on, same work hours, same work, same tenure, same seniority etc.• Ra3onale – If outside factors inﬂuence scores, then their eﬀect should be the same for both groups because both experienced them – If training inﬂuences iPPQ scores, the scores of the control group should diﬀer from the experimental group
Plan #2 Pre-‐interven3on Post-‐interven3on measurement measurement EXPERIMENTAL GROUP Training Training put into prac3ce TIME CONTROL GROUP Business as usual • If the group scores diﬀer, how can we tell if the diﬀerence is signiﬁcant?
Statistical significance• Sta3s3cal signiﬁcance is the conﬁdence you have in your results• Sta3s3cs put conﬁdence into precise terms – "Theres only one chance in a thousand this could have happened by coincidence." (p < 0.001)
How big of a diﬀerence will training create between groups? signalconfidence = × sample size noise How many people in each group? What other factors can create diﬀerences between groups? • To maximize conﬁdence – Increase interven:on quality (boost signal) – Minimize other diﬀerences between groups (reduce noise) – Increase sample size
signalconfidence = × sample size noise • Is the sample size 10 or 400? – 10 managers get trained – 400 employees get surveyed
Although the employees’ produc:vity at work is being measured, it is the eﬃcacy of the training interven:on that maSers.
Each manager is diﬀerent and will put the training into prac3ce diﬀerently.
Most managers will do an okay job.
Some will be excep3onally good.
Some will be excep3onally bad.
Each manager creates variabilityin data that cannot be controlled.
Thus, the eﬀec:ve sample size is 10,although 400 people are measured.
Small samples are more likely to be biased(In a sample of three, you may have two bad ones and a mediocre one, for example)
(Or the other way around.)
• Results should not change depending on who happens to respond.• The sample should be large enough to reduce unintended biases.
Plan #3 Pre-‐interven3on Post-‐interven3on measurement measurement EXPERIMENTAL GROUP Training Training put into prac3ce TIME CONTROL GROUP Business as usual • To reduce the impact of manager variability, recruit larger number of managers to both experimental and control groups – With large numbers of managers, extremes cancel each other out
Getting close, but...• Even sta3s3cally signiﬁcant diﬀerences between the experimental and control groups do not automa3cally speak for the eﬃcacy of training – Placebo eﬀect Belief in eﬃcacy creates changes – Hawthorne eﬀect Special situa3on and treatment of the measurement creates changes
Plan #4 Pre-‐interven3on Post-‐interven3on measurement measurement EXPERIMENTAL GROUP Training Training put into prac3ce TIME PLACEBO GROUP Fake training Training put into prac3ce CONTROL GROUP Business as usual
Three-way comparisons – Experimental group • If signiﬁcantly diﬀerent from the control group, outside factors did not account for the eﬀect. • If signiﬁcantly diﬀerent from the placebo group, the eﬀects were unique to training, not just diﬀerent treatment. – Control group • If not diﬀerent from experimental group, training had no eﬀect at all. – Placebo group • If not diﬀerent from experimental group, training had no real eﬀect beyond the special treatment given to the group.
Measurement in OD practice • Measurement is important • Measurement must be carefully planned and executed • The bare minimums are a proper control group and a large enough sample size