From TechBash 2023 Making large, important technical decisions is a critical aspect of a software engineer's role. With the wide impact these decisions can have, it is essential to make the correct decision. Even more vital is ensuring the decision is made and communicated in a way that the team members impacted by it trust and buy-in to the decision. Otherwise, even the best decisions will never realize their full potential when executed. This case study examines how Comcast has employed the Analytic Hierarchy Process (AHP), a decision-making framework developed in the 1970s, and adapted it for making technical and non-technical decisions both large and small. We will cover the key aspects that have made it successful for engineering teams, what we learned from our early mistakes, signs that the decision-making process you use is working effectively, and how you can easily leverage the AHP for your decisions.

1. © 2023 Comcast.
All marks are proper4es of their respec4ve owners. All Rights Reserved.
THE DECISION
BUY-IN ALGORITHM
November 8, 2023
JOHN RIVIELLO
@JohnRiv

2. The Decision Buy-In Algorithm • @JohnRiv
4
LET’S MAKE
A DECISION!
“IT DEPENDS”
!

3. The Decision Buy-In Algorithm • @JohnRiv
5
What programming language should a new software engineer learn first?
https://chat.openai.com/
ChatGPT

4. The Decision Buy-In Algorithm • @JohnRiv
8
It's difficult to pick just one "best" programming language for a
new software engineer to learn since each language has its own
strengths and weaknesses, and the best choice will depend on
the individual's specific goals and interests…
Pick the best overall one
There are several programming languages that a new software
engineer can learn first, depending on their goals and interests.
Here are some of the most popular options: ...
https://chat.openai.com/

5. The Decision Buy-In Algorithm • @JohnRiv
1 3
DECISIONS
BIG
& SMALL

6. The Decision Buy-In Algorithm • @JohnRiv
BIG DECISIONS
HOW
TO DECIDE?
WHO
WILL DECIDE?

7. The Decision Buy-In Algorithm • @JohnRiv
1 5
BIG DECISIONS
MOST
SENIOR
ENGINEER

8. The Decision Buy-In Algorithm • @JohnRiv
1 6
BIG DECISIONS
EQUAL
DEMOCRATIC
VOTE
“I GOATED” © CHRIS LUNDEEN

9. The Decision Buy-In Algorithm • @JohnRiv
1 7
BIG DECISIONS
“WISDOM
OF THE
CROWDS”

10. The Decision Buy-In Algorithm • @JohnRiv
1 8
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
Multiple,
independent
judgments
expert’s
individual
judgment

11. The Decision Buy-In Algorithm • @JohnRiv
1 9
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
Another question…

12. The Decision Buy-In Algorithm • @JohnRiv
2 3
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
https://testing.googleblog.com/2010/07/code-coverage-goal-80-and-no-less.html
“80% and no less!”
Testivus on Test Coverage
A programmer asks the wise teacher:
“I am ready to write some unit tests.
What code coverage
should I aim for?”
“MAN” BY MICHAEL HOURIGAN LICENSED BY PIXABAY LICENSE

13. The Decision Buy-In Algorithm • @JohnRiv
2 4
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
“ANCHOR” BY PIXABAY LICENSED BY CC0

14. The Decision Buy-In Algorithm • @JohnRiv
2 5
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
Form opinions first
before evaluating the
opinions of others…?

15. The Decision Buy-In Algorithm • @JohnRiv
2 8
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
Disagreement:
•Indicates one or both parties are
likely to be wrong
•Social Cost: people interpret
disagreement that their own
estimate is correct and the
other must be wrong
“TWO DOCTORS ALOOF FROM ONE ANOTHER IN DISAGREEMENT” BY J. LEECH LICENSED BY CC PUBLIC DOMAIN MARK 1.0

16. The Decision Buy-In Algorithm • @JohnRiv
2 9
THE RIGHT WAY TO USE THE WISDOM OF THE CROWDS
BY BRAD DEWEES AND JULIA A. MINSON. HARVARD BUSINESS REVIEW 12/20/18
Conclusions from their study:
• Form independent opinion(s) in advance
• Group should pre–commit to a decision strategy
• For quantifiable questions, try to remove human
judgment as much as possible

17. The Decision Buy-In Algorithm • @JohnRiv
A DECISION…
3 0
?
?
“SILVER IMAC ON WHITE TABLE” BY COTTONBRO STUDIO LICENSED BY PEXELS.COM

18. The Decision Buy-In Algorithm • @JohnRiv
3 1
“LANDSCAPE PHOTOGRAPHY OF FACTORY” BY PIXABAY LICENSED BY CC0

19. The Decision Buy-In Algorithm • @JohnRiv
3 2
ANALYTIC
HIERARCHY
PROCESS
by Thomas L. Saaty

20. CHOOSING THE MOST
SUITABLE LEADER

21. The Decision Buy-In Algorithm • @JohnRiv
4 6
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY

22. The Decision Buy-In Algorithm • @JohnRiv
4 8
AHP FUNDAMENTAL SCALE FOR PAIRWISE COMPARISONS
Intensity of
Importance
Definition Explanation
1 Equal importance Two elements contribute equally to the
objective
3 Moderate importance Experience and judgement moderately
favor one element over another
5 Strong importance Experience and judgement strongly
favor one element over another
7 Very strong importance
One element is favored very strongly
over another; its dominance is
demonstrated in practice
9 Extreme importance
The evidence favoring one element
over another is of the highest possible
order of affirmation
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Pairwise_comparisons
Intensities of 2, 4, 6, and 8 can be used to express intermediate values. 1.1, 1.2, 1.3, etc. can be used for elements that are very close in importance.

23. The Decision Buy-In Algorithm • @JohnRiv
5 5
AHP PAIRWISE SCORING
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Alternatives_vs._criteria
Alternatives compared with respect to EXPERIENCE
ASTRID 1 FIONA 4
ASTRID 4 EEP 1
FIONA 9 EEP 1

24. The Decision Buy-In Algorithm • @JohnRiv
6 6
AHP MATRIX PRIORITY CALCULATIONS
EXPERIENCE ASTRID FIONA EEP PRIORITY
ASTRID 1 1/4 4 0.2 1 7
FIONA 4 1 9 0.7 1 7
EEP 1/4 1/9 1 0.066
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Alternatives_vs._criteria

25. The Decision Buy-In Algorithm • @JohnRiv
6 7
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY
0.217 0.717 0.066

26. The Decision Buy-In Algorithm • @JohnRiv
7 1
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY

27. The Decision Buy-In Algorithm • @JohnRiv
7 2
AHP MATRIX PRIORITY CALCULATIONS
CRITERIA EXPERIENCE STRENGTH CHARISMA INTEGRITY PRIORITY
EXPERIENCE 1 4 3 7 0.547
STRENGTH 1/4 1 1/3 3 0.127
CHARISMA 1/3 3 1 5 0.270
INTEGRITY 1/7 1/3 1/5 1 0.056
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Criteria_vs._the_Goal

28. The Decision Buy-In Algorithm • @JohnRiv
7 3
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY

29. The Decision Buy-In Algorithm • @JohnRiv
CRITERIA EXPERIENCE STRENGTH CHARISMA INTEGRITY PRIORITY
EXPERIENCE 1 4 3 7 0.547
STRENGTH 1/4 1 1/3 3 0.127
CHARISMA 1/3 3 1 5 0.270
INTEGRITY 1/7 1/3 1/5 1 0.056
7 4
AHP MATRIX PRIORITY CALCULATIONS
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Criteria_vs._the_Goal

30. The Decision Buy-In Algorithm • @JohnRiv
7 5
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY
0.547

31. The Decision Buy-In Algorithm • @JohnRiv
7 6
AHP MATRIX PRIORITY CALCULATIONS
EXPERIENCE ASTRID FIONA EEP PRIORITY
ASTRID 1 1/4 4 0.2 1 7
FIONA 4 1 9 0.7 1 7
EEP 1/4 1/9 1 0.066
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Alternatives_vs._criteria

32. The Decision Buy-In Algorithm • @JohnRiv
7 7
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY
0.547
0.217 0.717 0.066

33. The Decision Buy-In Algorithm • @JohnRiv
7 8
AHP FINAL PRIORITY CALCULATIONS
EXPERIENCE PRIORITY
PRIORITY
vs. GOAL
PRIORITY with
respect to
EXPERIENCE
ASTRID 0.2 1 7
FIONA 0.7 1 7
EEP 0.066
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Synthesizing_final_priorities

34. The Decision Buy-In Algorithm • @JohnRiv
7 9
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY
0.547
0.217 0.717 0.066

35. The Decision Buy-In Algorithm • @JohnRiv
8 0
AHP FINAL PRIORITY CALCULATIONS
EXPERIENCE PRIORITY
PRIORITY
vs. GOAL
PRIORITY with
respect to
EXPERIENCE
ASTRID 0.2 1 7 0.547
FIONA 0.7 1 7 0.547
EEP 0.066 0.547
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Synthesizing_final_priorities

36. The Decision Buy-In Algorithm • @JohnRiv
8 2
AHP FINAL PRIORITY CALCULATIONS
EXPERIENCE PRIORITY
PRIORITY
vs. GOAL
PRIORITY with
respect to
EXPERIENCE
ASTRID 0.2 1 7 0.547 0.1 1 9
FIONA 0.7 1 7 0.547 0.392
EEP 0.066 0.547 0.036
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Synthesizing_final_priorities
x
x
x
=
=
=

37. The Decision Buy-In Algorithm • @JohnRiv
8 8
AHP FINAL PRIORITY CALCULATIONS
CANDIDATE
PRIORITY with
respect to
EXPERIENCE
PRIORITY with
respect to
STRENGTH
PRIORITY with
respect to
CHARISMA
PRIORITY with
respect to
INTEGRITY
PRIORITY with
respect to
GOAL
ASTRID 0.1 1 9 0.024 0.201 0.015 0.358
FIONA 0.392 0.010 0.052 0.038 0.492
EEP 0.036 0.093 0.017 0.004 0.149
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Synthesizing_final_priorities
=
=
=
+
+
+
+
+
+
+
+
+

38. The Decision Buy-In Algorithm • @JohnRiv
DREAMWORKS IMAGES © DREAMWORKS
CANDIDATE
PRIORITY with
respect to
EXPERIENCE
PRIORITY with
respect to
STRENGTH
PRIORITY with
respect to
CHARISMA
PRIORITY with
respect to
INTEGRITY
PRIORITY with
respect to
GOAL
ASTRID 0.1 1 9 0.024 0.201 0.015 0.358
FIONA 0.392 0.010 0.052 0.038 0.492
EEP 0.036 0.093 0.017 0.004 0.149
8 9
AHP FINAL PRIORITY CALCULATIONS
https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example#Synthesizing_final_priorities
=
=
=
+
+
+
+
+
+
+
+
+

39. “SURVEY ICON” BY PIXABAY LICENSED BY CC0

40. The Decision Buy-In Algorithm • @JohnRiv
9 1
“FOUR PERSON HOLDING BULB LIGHT DECORS” BY RAWPIXEL.COM LICENSED BY PEXELS.COM

41. The Decision Buy-In Algorithm • @JohnRiv
9 2
“HAWAII HANA ROAD” BY PIXABAY LICENSED BY CC0

42. The Decision Buy-In Algorithm • @JohnRiv
9 3
© JOHN RIVIELLO

43. The Decision Buy-In Algorithm • @JohnRiv
© JOHN RIVIELLO
IT’S THE
JOURNEY,
AND THE
DESTINATION

44. HOW DID OUR
TEAM DECIDE?

45. The Decision Buy-In Algorithm • @JohnRiv
9 9
OUR JS FRAMEWORK CRITERIA
•Community
•Performance
•Redux compatibility
•Web Components support
•Localization
•Developer Productivity
•Hybrid App Support
0
10
20
30
40
50
2 3 4 5 6 7 8 9 10
45
36
28
21
15
10
6
3
1
Number of items
Total
Pairwise
Comparisons

46. The Decision Buy-In Algorithm • @JohnRiv
1 0 0
OUR JS FRAMEWORK CRITERIA (WEIGHTED)

47. The Decision Buy-In Algorithm • @JohnRiv
1 0 3
OUR JS FRAMEWORK DECISION
Option 1:
Option 2:
Option 3:
Try it with your team!
https://comcastsamples.github.io/ahp-tool/

48. The Decision Buy-In Algorithm • @JohnRiv
1 0 4
OPEN SOURCE AHP TOOL
https://comcastsamples.github.io/ahp-tool/

49. AHP RETROSPECTIVE

50. The Decision Buy-In Algorithm • @JohnRiv
1 0 6
© JOHN RIVIELLO

51. The Decision Buy-In Algorithm • @JohnRiv
1 0 7
AHP RETROSPECTIVE
% & '
( ) *

52. The Decision Buy-In Algorithm • @JohnRiv
1 0 8
AHP RETROSPECTIVE
+MOTIVATION

53. The Decision Buy-In Algorithm • @JohnRiv
1 0 9
AHP RETROSPECTIVE
,DISCOVERY &
LEARNING

54. The Decision Buy-In Algorithm • @JohnRiv
1 1 2
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY

55. The Decision Buy-In Algorithm • @JohnRiv
1 1 3
ANALYTIC HIERARCHY PROCESS (AHP)
Goal:
Criteria:
Alternatives:
CHOOSE THE MOST
SUITABLE LEADER
CHARISMA
STRENGTH
EXPERIENCE INTEGRITY

56. The Decision Buy-In Algorithm • @JohnRiv
1 1 5
© JOHN RIVIELLO

57. The Decision Buy-In Algorithm • @JohnRiv
“EMOTIONS” BY BLENDE12 LICENSED BY PIXABAY
nemawashi
building
consensus
openly,
not forcing
consensus
“
”

58. The Decision Buy-In Algorithm • @JohnRiv
“EMOTIONS” BY BLENDE12 LICENSED BY PIXABAY
nemawashi
Idea Decision
deciders, makers,
blockers, affected
inform, gather, improve
⭐
https://intenseminimalism.com/2020/nemawashi-a-key-leadership-skill-to-foster-a-healthy-work-environment/

59. The Decision Buy-In Algorithm • @JohnRiv
1 2 8
AHP RETROSPECTIVE
Watch out for our mistakes
•AHP does NOT make hard
decisions less painful
•Do NOT hide the data

60. The Decision Buy-In Algorithm • @JohnRiv
1 2 9
AHP RETROSPECTIVE
https://open.nytimes.com/collective-decision-making-with-ahp-3ef819e5bc2a

61. The Decision Buy-In Algorithm • @JohnRiv
1 3 1
AHP RETROSPECTIVE
1

62. The Decision Buy-In Algorithm • @JohnRiv
1 3 2
LINKS
• https://comcastsamples.github.io/ahp-tool/
• http://github.com/ComcastSamples/ahp-tool
• https://en.wikipedia.org/wiki/Analytic_hierarchy_process_–_leader_example
• https://intenseminimalism.com/2020/nemawashi-a-key-leadership-skill-to-foster-a-healthy-work-environment/
• https://open.nytimes.com/collective-decision-making-with-ahp-3ef819e5bc2a
THANK YOU!
Please send comments,
questions, or feedback
to me at @JohnRiv