Using multi-criteria decision analysis (MCDA), government agencies can quantify and prioritize the value of potential IT projects. MCDA provides a structured framework to 1) determine value criteria, 2) define scoring for each criterion, 3) assign weights to criteria, and 4) score projects in a value matrix to calculate total value points. This allows agencies to objectively evaluate projects based on criteria like retiring legacy systems, reducing costs, ensuring compliance, and other factors important to stakeholders and decision makers.

1. VALUE POINTS
Maximizing Digital Services ROI
Using Multi-Criteria Decision
Analysis
Stephen Mayner
Scaled Agile, Inc.

2. SAFe® Senior Program Consultant
Scaled Agile, Inc.
steve.mayner@scaledagile.com
@stevemayner
stevemayner
#agile2016 #SAFe #WhereIsART

3. FEDERAL IT IS CHANGING

4. “To reboot how the government works…”
The TechFAR Handbook Digital Services Playbook

5. OUR HIGHEST PRIORITY IS TO
SATISFY THE CUSTOMER THROUGH
EARLY AND CONTINUOUS DELIVERY
OF VALUABLE SOFTWARE.
Agile Manifesto, Twelve Principles
AGILEMANIFESTO.ORG/PRINCIPLES
“
”
(The Lodge at Snowbird Ski Resort, where Agile was born…)

6. What is value?

7. Stock price
Market share
EPS
NVP
Growth

8. Mission
Responsive
Transparent
Accessible
Cost Effective
Compliant

9. SAFe®
is a freely revealed knowledge base
of integrated, proven success patterns
for enterprise Lean-Agile development.
Knowledge for people building
the world's most important systems
www.scaledagileframework.com

11. #1-Take an economic view
#2-Apply systems thinking
#3-Assume variability; preserve options
#4-Build incrementally with fast, integrated learning cycles
#5-Base milestones on objective evaluation of working systems
#6-Visualize and limit WIP, reduce batch sizes, and manage queue lengths
#7-Apply cadence, synchronize with cross-domain planning
#8-Unlock the intrinsic motivation of knowledge workers
#9-Decentralize decision-making

12. If you only quantify one thing, quantify the
—Donald G. Reinertsen, Principles of Product Development Flow

13. weighted-shortest job
(WSJF)
Reinertsen, D. (2012). The principles of product development flow: Second generation lean
product development. Redondo Beach, CA: Celeritas Publishing

14. COST OF DELAY
DURATION

15. USER-Business value + time criticality + risk reduction / opportunity enablement
Job size

16. The job with the highest WSJF provides the greatest economic benefit.
Scale for each parameter: 1, 2, 3, 5, 8, 13, 20
Note: Do one column at a time, start by picking the smallest item and giving it a “1.”
There must be at least one “1” in each column!
WSJF = =
User-business value + Time criticality + RR | OE value
Job size
CoD
Duration

17. What is “USER BUSINESS value”
How can it be quantified
The key questions for government agencies…

18. Multi-Criteria
Decision Analysis
(MCDA)
…a general framework for
supporting complex decision-
making situations with multiple and
often conflicting objectives that
stakeholders groups and/or
decision-makers value differently
Belton, V., & Stewart, T. (2002). Multiple criteria decision analysis: An integrated approach. Springer.
Multiple stakeholders
Structured decision alternatives
Facilitate decision dialogue
Quantify categorical criteria
Unify plural value dimensions

19. Step #1
Program portfolio management team determines
Value criteria
What is USER BUSINESS value

20. Retire the mainframe
Reduce operating costs
Ensure compliance
Generate new revenue
Increase stakeholder satisfaction
Add new capabilities

21. Step #2
Program portfolio management team determines
Scoring definitions
How can it be Quantified

22. Retire the mainframe
5 – directly enables functionality to be turned off
4 – direct precursor to disabling mainframe functionality
3 – strongly improves ability to migrate
2 – moderately improves ability to migrate
1 – nominally improves ability to migrate
0 – no relationship to retiring the mainframe
Scoring definition example

23. Ensure compliance
5 – required to comply with regulations, laws, directives
0 – not required for compliance
Scoring definition example
For this decision analysis technique, an ordinal scale (Likert) is more appropriate than interval scale (Fibonacci)

24. Step #3
Program portfolio management team determines
Relative weighting
How can it be Quantified

25. 30% retire the mainframe
20% reduce operating costs
20% ensure compliance
10% generate new revenue
10% increase stakeholder satisfaction
10% add new capabilities
100%
weighting example

26. Step #4
Program portfolio management team builds
Value matrix
Putting it all together

27. Criteria Weight Scoring Definitions Raw
Score
Weighted
Score
Retire the Mainframe 30% 5 – Directly enables functionality to be turned off
4 – Direct precursor to disabling mainframe functionality
3 – Strongly improves ability to migrate
2 – Moderately improves ability to migrate
1 – Nominally improves ability to migrate
0 – No relationship to retiring the mainframe
3 9
Cost Savings 20% 5 – Immediate AND >=$100M
4 – Within 1 year AND >=$25M
3 – Within 2 years AND >=$10M
2 – Within 3 years AND >=$1M
1 – Within 5 years OR >= $100K
0 – No cost savings
4 8
Compliance 20% 5 – Required to comply with regulations, statutes, directives
0 – Not required for compliance
5 10
New Revenue 10% 5 – Directly enables new revenue >=$100M
4 – Directly enables revenue between $25M to $99.9M
3 – Directly enables revenue between $10M to $24.9M
2 – Directly enables revenue between $0.1M to $9.9 M
1 – Indirectly enables new revenue
0 – No connection to revenue generation
2 2
Stakeholder Satisfaction/
New Capabilities
10%
10%
5 – Provides a top 5 most requested enhancement
4 – Provides a top 50 most requested enhancement
3 – Directly resolves a top 5 most commonly reported issue
2 – Directly resolves a top 50 most commonly reported issue
1 – Provides unrequested/IT generated enhancement
0 – No connection to customer satisfaction
4 4
Total Value Points 33Apply above user story level (epics, enablers, capabilities, features) - Cohn

28. Use modified
estimation poker
with numbered
cards 0 – 5 to
achieve consensus

29. USER-Business value + time criticality + risk reduction / opportunity enablement
Job size

30. Value points + time criticality + risk reduction
Job size

31. Value points
JOB SIZE

32. Retire the mainframe
Reduce operating costs
Ensure compliance
Generate revenue
Customer satisfaction
Add new capabilities
Total Value Points
Job Size
WSJF
Weighting 3 4 5 2 1 4
Epic / Feature 3 0 5 5 2 2 4 67 5 13.40
Epic / Feature 6 5 5 4 4 2 4 81 13 6.23
Epic / Feature 5 2 1 2 2 1 1 29 5 5.80
Epic / Feature 1 3 3 1 2 2 2 40 8 5.00
Epic / Feature 2 2 3 5 4 3 3 66 20 3.30
Epic / Feature 4 0 1 3 1 1 4 38 13 2.92

33. 0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12
Program Increments
Value Burnup Chart
Value Points

34. Consistent
Fair
Defensible
Understandable
Measurable

35. Hands-on EXERCISE

37. Before you begin…
FORM TABLE TEAMS AND build your
MATRIX
3
min
Value Matrix
A.
5 –
4 –
3 –
2 –
1 –
0 –
B.
5 –
4 –
3 –
2 –
1 –
0 –
C.
5 –
4 –
3 –
2 –
1 –
0 –
Weighting
Acceptance criteria:
Two flip chart sheets (one of
each) per table that look like
the examples to the right.
Weight ___ ___ ___ Value
Points
Epic 1 – Raw ___ ___ ___
Epic 1 – Wgt ___ ___ ___ _____
Epic 2 – Raw ___ ___ ___
Epic 2 – Wgt ___ ___ ___ _____
Epic 3 – Raw ___ ___ ___
Epic 3 – Wgt ___ ___ ___ _____
A. B. C.

38. Step #1
DETERMINE YOUR
VALUE CRITERIA
Acceptance criteria:
3 value categories
Examples:
1. Security
2. Ease of Access
3. Single UX5
min
Value Matrix
A. Security
5 –
4 –
3 –
2 –
1 –
0 –
B. Ease of Access
5 –
4 –
3 –
2 –
1 –
0 –
C. Single UX
5 –
4 –
3 –
2 –
1 –
0 –
Weighting

39. Step #2
DETERMINE YOUR
SCORING DEFINITIONS
10
min
Acceptance criteria:
0-5 definitions for each criteria
Examples:
5. Enables highest level of security
4. Enables strong security
3. Enables moderate security
2. Has limited security enablement
1. Has minimal security enablement
0. Has no security enablement
Value Matrix
A. Security
5 – Enables highest
4 – Enables strong
3 – Enables moderate
2 – Limited enablement
1 – Minimal enablement
0 – No enablement
B. Ease of Access
5 –
4 –
3 –
2 –
1 –
0 –
C. Single UX
5 –
4 –
3 –
2 –
1 –
0 –
Weighting

40. Step #3
DETERMINE YOUR
RELATIVE WEIGHTING
2
min
Acceptance criteria:
Weighting values for each criteria that
total 100%
Examples:
A. Security 50%
B. Ease of Access 25%
C. Single UX 25%
Value Matrix
A. Security
5 – Enables highest
4 – Enables strong
3 – Enables moderate
2 – Limited enablement
1 – Minimal enablement
0 – No enablement
B. Ease of Access
5 –
4 –
3 –
2 –
1 –
0 –
C. Single UX
5 –
4 –
3 –
2 –
1 –
0 –
Weighting
50%
25%
25%

41. Step #4
TEST YOUR
VALUE MATRIX
5
min
Acceptance criteria:
Weighting listed for each criteria
Each epic scored using your value matrix
Weighted score calculated for each epic
Total value points listed for each epic
Epics:
1. API access to all source systems
2. SSO multi-factor authentication
3. Live agent option for all services
Weight ___ ___ ___ Value
Points
Epic 1 – Raw ___ ___ ___
Epic 1 – Wgt ___ ___ ___ _____
Epic 2 – Raw ___ ___ ___
Epic 2 – Wgt ___ ___ ___ _____
Epic 3 – Raw ___ ___ ___
Epic 3 – Wgt ___ ___ ___ _____
A. B. C.
5 2.5 2.5
4 2 1
20 5 2.5
3 5 2
15 12.5 2.5
2 2 4
10 5 2.5
27.5
30
17.5

42. TEAM REPORTS
10
min
Acceptance criteria:
Spokesperson for each team identified
Team value matrix described
Team scoring results described
Report completed in 2 min timebox

43. Questions?