Federal Express ROI analysis using control groups as a good example of the state of the art in measuring ROI. The study focused on 20 employees who went through an intense, redesigned two-week training program soon after being hired to drive company vans. Their performance was compared with a control group of 20 other new hires whose managers were told to do no more or less on-the-job training than they normally would. Performance was tracked for the two groups for 90 days in categories such as accidents, injuries, time-card errors, and domestic air-bill errors. The ten performance categories were assigned monetary values by experts from engineering, finance, and other groups. The intervention demonstrated that the performance of the highly trained employees was superior to that of the group that did not receive the upgraded training and also resulted in a 24% return on investment.
Accra Wholesale Brickworks, find that there production targets are not being met. The Company is midsize but established in the City. It employs a production team consisting of three foreman, that supervise approximately 8 workers each and these teams take responsibility for procurement of the raw materials, mixing, filling the moulds, drying and baking. The production function is managed by a section manager.
Draw a trend-line using previous performance data as a base
When intervention is conducted, actual performance is plotted and compared to the trend line
Improvement in actual performance over what the trend line predicted can then be reasonably attributed to the intervention
The primary advantage of this approach is that it is simple, inexpensive, and takes very little effort;
However not accurate as it does not isolate into other factors ;
However a good first order analysis .
Trend-line Analysis Average Post Program 35 Average Pre Program 55 Time Complaints Sexual Harassment Prevention Program Formal Internal Complaints of Sexual Harassment O N D J F M A M J J A S O N D J F M A M J J A S O
Practical Exercise Trend line Use Case Study (Micro Electronics Incorporated)
The individuals who do not respond to the questionnaire or provide non- usable data on the questionnaire are assumed to report no improvement/impact;
Extreme data and unrealistic claims are omitted from the analysis;
Only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation;
The confidence level, expressed as a percent, is multiplied by the improvement value to reduce the amount of the improvement by the potential error;
If value is expressed it is also factored by the confidence value.
Estimate of Impact by Key Role Players Adjustments to improve validity
Estimate of Impact by Key Role Players % Program Contributed to Improvement Confidence of Statement as % Factor Total Improvement (annualized)– Value Contribution to Improvement 60 % (.6) 40% (.4) .24 1000 240 60 % (.6) 40% (.4) .24 1000 240 50% (.5) 50% (.5) .25 1000 250
Estimate of Impact by Key Role Players The process can be expanded to include: Supervisors Management Other stakeholders
An example is the subordinates of the supervisors trained in people handling skills;
It could be possible for them to estimate how much of an improvement can be attributed to the intervention;
They can also provide input in terms of what other factors might have contributed to the improvement;
The input is usually obtained through surveys or interviews. When the survey results show significant changes in participant behavior after training and no significant change in the general work climate, the improvement in work performance, therefore, must be attributed to the changes in supervisor behavior since other factors remained constant;
This approach has some disadvantages. Data from subordinates is subjective and may be questionable because of the possibility for biased input;
Often subordinates may have difficulty in determining changes in the work climate. However, in some cases, subordinates are aware of the factors that caused changes in their work unit and can provide input about the magnitude or quantity of changes. When combined with other methods to isolate impact, this process has increased credibility.
In some situations it could be feasible to calculate the impact of factors (other than the intervention) that influenced the improvement and then conclude that the intervention is credited with the remaining portion;
In terms of this approach, the intervention claims credit for improvement that cannot be attributed to other factors;
The method is appropriate when the other factors are easily identified and the appropriate mechanisms are in place to calculate their impact on the improvement;
The process can be very credible if the method used to isolate the impact of other factors is credible.