Certification Study Group - Professional ML Engineer Session 3 (Machine Learn...
Unit- 5. Measuring results and Behaviors
1. Herman Aguinis, University of Colorado at
MEASURING RESULTS AND
BEHAVIORS
Prof. Preeti Bhaskar
Symbiosis Centre for Management Studies, NOIDA
2. Herman Aguinis, University of Colorado at
Measuring Results: Key questions
• Where should each individual focus efforts?
• What are the expected objectives?
• How do we know how well the results were achieved?
Key
Accountabilities
Expected
Objectives
Performance
Standards
Statements of important and measurable
outcomes
Broad areas of a job for which employee is
responsible for producing results
Yardstick used to evaluate how well employees
have achieved objectives
3. Herman Aguinis, University of Colorado at
How to Determine Accountabilities?
Collect information about job with the help of Job
Description.()
Job description provides information on the task
performed. Task can be grouped into clusters of Task
based on their relatedness.
Determine the relative degree of importance of task or
cluster of tasks
• % of employee’s time spent performing task
• Impact on unit’s mission if performed inadequately
• Consequences of error
4. Herman Aguinis, University of Colorado at
Determining Objectives
Purpose:
• to identify Limited number of Highly important results
that When achieved will have a dramatic impact on
overall organization success
•After objective are set employee should receive the feedback on their
progress towards attaining the objective.
•Rewards should be allocated to those employees who have reached their
objective
5. Herman Aguinis, University of Colorado at
Characteristics of Good Objectives
• Specific and Clear
• Challenging
• Agreed Upon
• Significant
• Prioritized
• Bound by Time
• Achievable
• Fully
Communicated
• Flexible
• Limited in Number
6. Herman Aguinis, University of Colorado at
Determining Performance Standards
Standards refer to aspects of performance objectives,
such as:
• Quality
– How well the objective is achieved
• Quantity
– How much, how many, how often, at what cost
• Time
– Due dates, schedule, cycle times, how quickly
7. Herman Aguinis, University of Colorado at
Standards must include:
• A verb
• A due date
• Some type of indicator
– Quality and/or
– Quantity
Reduce overtime from 150 hours/months to 50 hours/month by
December 1,2014 at a cost not to exceed 1,20,000
Verb
Due date
Indicator
8. Herman Aguinis, University of Colorado at
Characteristics of Good Performance
Standards
• Related to Position
• Concrete, Specific, Measurable
• Practical to Measure
• Meaningful
• Realistic and Achievable
• Reviewed Regularly
9. Herman Aguinis, University of Colorado at
Accountabilities, Objectives, and
Standards
Job Title:
Graduate Associate, Sourcing &
Procurement
10. Herman Aguinis, University of Colorado at
Accountabilities:
Provide analytical support for sourcing projects impacting
business units. Successful analytical support will ensure
that project decisions are made on well-researched
reasons. Poorly made project decisions may delay client
deliverables or damage the Disney brand. (20% of time)
Benchmark current pricing models and develop new
approaches to pricing/buying various products and
services that yield creative and business advantages.
This is extremely important to the continued sustainability
of Disney, and will be critical to capture new portions of the
market share. Incorrectly identifying pricing models would
be crushing to the Disney business and would directly
impact the bottom line. (30% of time)
11. Herman Aguinis, University of Colorado at
Cont.
Assist in the development of spend profiles, key
stakeholder lists, savings opportunities where existing
contracts are leveraged, savings opportunities in
commodity areas that have not been sourced, and
sourcing and procurement strategy. Will play a small
role in these extremely important activities.
Successfully developing spend profiles, identifying key
stakeholders, identifying savings opportunities, and
developing a successful business strategy are keys to
the financial stability of Disney. (50% of time)
12. Herman Aguinis, University of Colorado at
Objectives: :
• Develop spend profiles, key stakeholder lists, and a sourcing and
procurement strategy. These tasks need to be completed by July 1,
2004. Identifying savings opportunities where existing contracts are
leveraged and identifying savings opportunities in commodity areas
that have not been sourced are ongoing tasks. Report weekly on
savings identified.
• Benchmark current pricing models and develop new approaches to
pricing/buying various products and services that yield creative and
business advantages. Benchmarking project due by August 1, 2008.
Developing new approaches to pricing is an ongoing project. Turn in
bi-weekly reports on new findings to your manager by COB every
other Friday.
• Provide analytical support for sourcing projects impacting business
units. Ongoing project where information must be delivered inside
13. Herman Aguinis, University of Colorado at
Performance Standards::
• Identified savings opportunities where existing contracts were
leveraged and identified commodity areas that have not been
sourced on a weekly basis for total savings of 5% over previous
quarter spending.
• Benchmarked current pricing models by August 1, 2008 in 80
hours or less of time billed.
• Submitted new approaches to pricing/buying various products
and services on a weekly basis resulting in at least two new
pricing approaches being adapted by the company by
September 1, 2008.
• Provided ongoing analytical support for sourcing projects as
rated by monthly project evaluations conducted by the relevant
project managers and communicate0d to your manager on a
monthly basis.
14. Herman Aguinis, University of Colorado at
Measuring Behaviors: Overview
• Identify competencies
• Identify indicators
• Choose measurement system
15. Herman Aguinis, University of Colorado at
Identify Competencies
Measurable clusters of KSAs
– Knowledge's
– Skills
– Abilities
That are critical in determining how results
will be achieved
16. Herman Aguinis, University of Colorado at
Types of
Competencies
Differentiating Threshold
Distinguish between
superior and average
performance
Needed to perform
job at minimum
Adequate standard
17. Herman Aguinis, University of Colorado at
We don’t measure competency directly but
we measure Indicators
A competency can have several indicators.
competency
Indicator 1 Indicator 2 Indicator 3 Indicator 4
18. Herman Aguinis, University of Colorado at
Necessary Components for
Describing Competencies
• Definition
• Description of specific behaviors
– When competency demonstrated
– When competency not demonstrated
• Suggestions for developing the competency
19. Herman Aguinis, University of Colorado at
Measurement System to evaluate
competencies
Compares employees
with pre-specified
performance standard
Choose Measurement
System
Comparative
system
Absolute
system
Compares employees with
each other
20. Herman Aguinis, University of Colorado at
Types of Comparative
Systems
Simple rank
order
Alternation
rank order
Paired
comparisons
Forced
distribution
Advantages
• Easy to explain
• Straightforward
• Better control for biases and errors
found in absolute systems
– Leniency
– Severity
– Central tendency
Disadvantages
• Rankings may not be specific
enough for
– Useful feedback
– Protection from legal challenge
• No information on relative distance
between employees
• Specific issues with forced
distribution method
21. Herman Aguinis, University of Colorado at
Types of Absolute
Systems
Essays
Behavior
checklists
Critical
incidents
Graphic
rating scales
22. Herman Aguinis, University of Colorado at
Essays
• Advantage:
– Potential to provide detailed feedback
• Disadvantages:
– Unstructured and may lack detail
– Depends on supervisor writing skill
– Lack of quantitative information; difficult to
use in personnel decisions
23. Herman Aguinis, University of Colorado at
Behavior checklists
• Advantage:
– Easy to use and understand
• Disadvantage:
– Scale points used are often arbitrary
– Difficult to get detailed and useful feedback
24. Herman Aguinis, University of Colorado at
Critical incidents
Two kinds of measurement
– Report of specific employee
behavior
• Allows focus on specific behavior
• Very time-consuming
– Examples of behavior illustrative of
core competencies
• Easier to use
• Describes behavior desired
25. Herman Aguinis, University of Colorado at
Graphic rating scales
• Clear meaning for each response category
• Consistent interpretation by outside readers
• Supervisor and employee should have same understanding of
rating
26. Herman Aguinis, University of Colorado at
Graphic rating scales:
BARS improvement
• Behaviorally Anchored Rating Scales (BARS)
– Use critical incidents as anchors
– Involves multiple groups of employees in
development
• Identify important job elements
• Describe critical incidents at various levels of
performance
• Check for inter-rater reliability
28. Herman Aguinis, University of Colorado at
Measuring Performance
• Several types of methods
• Differ in terms of:
– Practicality (time and effort)
– Usefulness (quantifiable)
29. Herman Aguinis, University of Colorado at
Make a group of 5 students
Creating BARS-Based Graphic Rating
Scales for Evaluating Business Student
Performance in Team Projects
30. Herman Aguinis, University of Colorado at
In many universities, students are required to
conduct Team Projects.
Job Description for students-:
• Work with team member to deliver project outcome
on time and according to specification .
• Complete all individual assignment to highest
quality ,completing necessary background
research , making analysis and preparing final
documents.
• Foster a good working environment
31. Herman Aguinis, University of Colorado at
Please do the following:
1. Generate a list of competencies for the
position described.
2. Identify the list of critical behavioral indicators
for each competency.
3. Generate critical incident (High ,average and
poor performance for each behavioral
indicator.
4. Create Graphic rating scales using BARS to
measure each competency.
32. Herman Aguinis, University of Colorado at
Competencies:
• Teamwork: Ability to work with a team to finish all project deliverables on
time and according to instructions. Assists others as necessary and
constantly seeks to further relationships with team members. Failure to
develop teamwork will result in reduced team effectiveness due to lack of
communication and cooperation between team members.
• Business knowledge: Aware of current business trends and applies
relevant business research skills to projects at hand. Always learning new
skills, staying current on business news, and talking with business
professionals. Failure to develop business knowledge will result in a
decrease in the quality of work produced by the team.
• Communication: Ability to articulate verbally or through written
communication ideas on project deliverables. Use of correct grammar,
professional presentation, and clear and concise messages characterize
good communication. Failure to communicate well will result in the
decreased performance of the team, because team members will no longer
33. Herman Aguinis, University of Colorado at
Behavioral Indicators
• Teamwork
– Friendly environment exists
– Team members’ input is accepted
– Offers to help other team members when necessary
• Business knowledge
– Student offers input when group is discussing business
problems
– Actively researches solutions presented
– Reads business news sources on a daily basis to stay up-to-
date on business trends
• Communication
– Clearly articulates points in meetings
– Uses correct grammar and word usage in written
communications
– Professionally presents ideas and findings in project
presentations
34. Herman Aguinis, University of Colorado at
Critical Incidents:
Indicator High Performance
Average
Performance Poor Performance
Student offers input
when group is
discussing business
problems
Continuously provides
relevant input into group
discussions
Offers input on areas of
expertise in group
discussions
Does not offer input
into group discussions
Actively researches
solutions presented
Researches practical and
theoretical implications of
decision
Researches practical
implications of decision
Does not research
implications of business
decisions
Reads business
news sources on a
daily basis to stay
up-to-date on
business trends
Comments on a daily basis
on the status of ongoing
business developments of
interest
Comments on a weekly
basis on the status of
ongoing business
developments of interest
Does not comment on
the status of ongoing
business developments
of interest
Clearly articulates
points in meetings
Presents ideas in a clear
and concise manner and
ensures that team
members understand point
Presents ideas in a clear
and concise manner
Does not get point
across in meetings,
which results in
arguments with other
team members or mis-
education of team
members
35. Herman Aguinis, University of Colorado at
Indicator High Performance Average Performance Poor Performance
Uses correct
grammar and word
usage in written
communications
Written communications
are clear, free of spelling
and grammatical
mistakes
Clear use of language in
written reports
Poor spelling and
grammatical usage in
written reports
Professionally
presents ideas and
findings in project
presentations
Prepares presentation so
ideas are communicated
well and in a professional
manner
Minimally prepares for
presentations, so
communication is rushed
or drags on
Does not prepare for
presentations
Friendly
environment exists
Takes an interest in team
members outside of
project
Talks to team members to
complete tasks Withdrawn from team
Team members’
input is accepted
Supportive of other team
members’ ideas
Solicits, but does not
support team members’
ideas
Does not solicit input
from other team
members
Offers to help other
team members
when necessary
Volunteers for extra
assignments so that
team members can finish
projects on time and to
the highest standards
Offers to help team
members if deadline is
approaching
Does not offer to help
team members
37. Herman Aguinis, University of Colorado at
Teamwork:
Ability to work in a team to finish all project deliverables on time and according
to instructions. Assists others as necessary and constantly seeks to further
relationships with team members. Failure to develop teamwork will result in
reduced team effectiveness due to lack of communication and cooperation
between team members.
5 Exceptional: Takes an interest in team members outside of project.
3 Competent: Talks to team members to complete tasks.
1 Major Improvement Needed: Withdrawn from team.
5 Exceptional: Supportive of other team members’ ideas.
3 Competent: Solicits, but may not support team members’ ideas.
1 Major Improvement Needed: Does not solicit input from other team members.
5
Exceptional: Volunteers for extra assignments so that team members can finish projects on
time and to the highest standards.
3 Competent: Offers to help team members if deadline is approaching.
1 Major Improvement Needed: Does not offer to help team members.
38. Herman Aguinis, University of Colorado at
Business Knowledge:
Aware of current business trends and applies relevant business research skills
to projects at hand. Always learning new skills, staying current on business
news, and talking with business professionals. Failure to develop business
knowledge will result in a decrease in the quality of work produced by the team.
5 Exceptional: Continuously provides relevant input into group discussions.
3 Competent: Offers input on areas of expertise in group discussions.
1 Major Improvement Needed: Does not offer input into group discussions.
5 Exceptional: Researches practical and theoretical implications of decisions.
3 Competent: Researches practical implications of decisions.
1 Major Improvement Needed: Does not research implications of business decisions.
5
Exceptional: Comments on a daily basis on the status of ongoing business
developments of interest.
3
Competent: Comments on a weekly basis on the status of ongoing business
developments of interest.
1
Major Improvement Needed: Does not comment on the status of ongoing business
developments of interest.
39. Herman Aguinis, University of Colorado at
Communication:
Ability to articulate verbally or through written communication ideas on project
deliverables. Use of correct grammar, professional presentation, and clear
and concise messages characterize good communication. Failure to
communicate well will result in the decreased performance of the team,
because team members will not understand ideas expressed.
5
Exceptional: Presents ideas in a clear and concise manner and ensures that team members
understand point.
3 Competent: Presents ideas in a clear and concise manner.
1
Major Improvement Needed: Does not get point across in meetings, resulting in arguments
with other team members or miseducation of team members.
5 Exceptional: Written communication is clear, free of spelling and grammatical mistakes.
3 Competent: Clear use of language in written reports.
1 Major Improvement Needed: Poor spelling and grammatical usage in written reports.
5
Exceptional: Prepares presentation so ideas are communicated well and in a professional
manner.
3 Competent: Minimally prepares for presentations, so communication is rushed or drags on.