Culture Eats Strategy for Breakfast - Greenspot by DartGroup Amsterdam - Cont...
The Hiring Problem
1. The Hiring Problem:
Going Beyond Secretaries
Sergei Vassilvitskii (Yahoo!)
Andrei Broder (Yahoo!)
Adam Kirsch (Harvard)
Ravi Kumar (Yahoo!)
Michael Mitzenmacher (Harvard)
Eli Upfal (Brown)
2. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
3. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
4. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
5. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
6. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
7. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
8. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
9. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
10. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
11. The Secretary Problem
Interview n candidates for a position one at a time. After each
interview decide if the candidate is the best.
Goal: maximize the probability of choosing the best candidate.
This is not about hiring secretaries, but about decision
making under uncertainty.
13. The Hiring Problem
A startup is growing and is hiring many employees:
Want to hire good employees
Can’t wait for the perfect candidate
14. The Hiring Problem
A startup is growing and is hiring many employees:
Want to hire good employees
Can’t wait for the perfect candidate
Many potential objectives.
Explore the tradeoff between number of interviews & the
average quality.
15. The Hiring model
Candidates arrive one at a time.
Assume all have iid uniform(0,1) quality scores - For
applicant i denote it by iq .
(Can deal with other distributions, not this talk)
16. The Hiring model
Candidates arrive one at a time.
Assume all have iid uniform(0,1) quality scores - For
applicant i denote it by iq .
(Can deal with other distributions, not this talk)
During the interview:
Observe iq
Decide whether to hire or reject
20. Strategies
Hire above a threshold.
Hire above the minimum or maximum.
Lake Wobegon Strategies:
“Lake Wobegon: where all the women are strong, all the
men are good looking, and all the children are above
average”
21. Strategies
Hire above a threshold.
Hire above the minimum or maximum.
Lake Wobegon Strategies:
Hire above the average (mean or median)
22. Strategies
Hire above a threshold.
Hire above the minimum or maximum.
Lake Wobegon Strategies:
Hire above the average
Side note: [Google Research Blog - March ‘06]:
“... only hire candidates who are above the mean of the
current employees...”
34. Threshold Analysis
Set a threshold t , hire if iq ≥ t .
1+t
Easy to see that average quality approaches
1 2
Hiring rate .
1−t
35. Threshold Analysis
Set a threshold t , hire if iq ≥ t .
1+t
Easy to see that average quality approaches
1 2
Hiring rate .
1−t
Quality stagnates and does not increase with time.
51. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
52. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
53. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
gn ∼ U nif (0, gn−1 )
54. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
gn ∼ U nif (0, gn−1 )
gn−1
E[gn |gn−1 ] =
2
55. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
gn ∼ U nif (0, gn−1 )
gn−1
E[gn |gn−1 ] =
2
1−q
E[gn ] = n
2
56. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
gn ∼ U nif (0, gn−1 )
gn−1
E[gn |gn−1 ] =
2
1−q
E[gn ] = n Very high quality!
2
57. Maximum Analysis
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn−1 :
gn ∼ U nif (0, gn−1 )
gn−1
E[gn |gn−1 ] =
2
1−q
E[gn ] = n Extremely slow hiring!
2
60. Lake Wobegon Strategies
Above the mean:
1
Average quality after n hires: 1 − √
n
Above the median:
1
Median quality after n hires: 1 −
n
61. Lake Wobegon Strategies
Above the mean:
1
Average quality after n hires: 1 − √
n
Above the median:
1
Median quality after n hires: 1 −
n
Surprising:
Tight concentration is not possible
Hiring above mean converges to a log-normal distribution
62. Hiring Above Mean
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
63. Hiring Above Mean
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn : (in+1 )q ∼ U nif (1 − gn , 1)
64. Hiring Above Mean
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn : (in+1 )q ∼ U nif (1 − gn , 1)
n+1 1
Therefore: gn+1 ∼ gn + U nif (0, gn )
n+2 n+2
65. Hiring Above Mean
Start with employee of quality q
Let hi be the i-th candidate hired
Focus on the gap: gi = 1 − (hi )q gi
Conditioned on gn : (in+1 )q ∼ U nif (1 − gn , 1)
n+1 1
Therefore: gn+1 ∼ gn + U nif (0, gn )
n+2 n+2
U nif (0, 1)
= gn 1−
n+2
67. Hiring Above Mean
U nif (0, 1)
gn+1 = gn 1−
n+2
t
U nif (0, 1)
Expand: gn+t = gn 1−
i=1
n+i+1
68. Hiring Above Mean
U nif (0, 1)
gn+1 = gn 1−
n+2
t
U nif (0, 1)
Expand: gn+t = gn 1−
i=1
n+i+1
Therefore:
n
U nif (0, 1)
gn ∼ (1 − q) 1−
i=1
i+1
69. Hiring Above Mean
U nif (0, 1)
gn+1 = gn 1−
n+2
t
U nif (0, 1)
Expand: gn+t = gn 1−
i=1
n+i+1
Therefore:
n
U nif (0, 1)
gn ∼ (1 − q) 1−
i=1
i+1
n
1 1
E[gn ] = (1 − q) 1− = Θ( √ )
i=1
2(i + 1) n
70. Hiring Above Mean
Conclusion:
1
Average quality after n hires: 1 − Θ( √ )
n
Time to hire n employees: Θ(n3/2 )
Very weak concentration results.
71. Hiring Above Median
Start with employee of quality q
When we have 2k + 1 employees.
Compute median mk of the scores
Hire next 2 applicants with scores above mk
89. Hiring Above Median
k
Expand (like the means): gk ∼ g Beta(i + 1, 1)
i=1
1
E[gn ] = Θ( )
n
90. Hiring Above Median
k
Expand (like the means): gk ∼ g Beta(i + 1, 1)
i=1
1
E[gn ] = Θ( )
n
log n
Caveat: this is the median gap! The mean gap is: Θ( )
n
91. Hiring Above Median
Conclusion:
log n
Average quality after n hires: 1 − Θ( )
n
Time to hire n employees: Θ(n2 )
Again, very weak concentration results.
92. Extensions
Interview preprocessing:
Self selection: quality may increase over time
Errors:
Noisy estimates on iq scores.
Firing:
Periodically fire bottom 10% (Jack Welch Strategy)
Similar analysis to the median.
Many more...