How can you measure the effect of a large-scale agile transition in a 1000+ person organisation? How can teams determine where they stand on their transition path, and which other teams can help them improve their agile practices further? How do you increase motivation for the next steps? How can the organisation measure the overall take-up of "agility" measures? Which measures are already effective in the organisation, and where should the focus for trainings, guidance and coaching be directed in the future? Frequentis, an organisation with 1500+ employees, has selected and implemented an Agile Assessment. This talk presents their approach and findings from their first year of running this assessment across the organisation in 2016. It ends with an outlook on the next evolution of the assessment for 2017.

1. Public
Safety
Public
Transport
Air Traffic
Management
Maritime
Defence
Assess your agility!
Raising the level of agility across an organisation
Maximilian Hantsch-Köller, Frequentis AG

2. | © 2017 Frequentis AGPublicAssess your agility!3 |
The company

3. | © 2017 Frequentis AGPublicAssess your agility!|4
70 years of innovation in safety-critical applications
We set standards
Public
Safety
Public
Transport
Air Traffic
Management
Maritime
Defence
Control centres
worldwide

4. | © 2017 Frequentis AGPublicAssess your agility!|5
Global market leader
Headquarters Vienna
1,606 Employees
75% STEM*
Solutions partners and regional
offices in over 50 countries
252 M EUR
turnover 2016
12%
95%
Export
R&D
*) Science, Technology, Engineering, Math
Established
+
1947

5. | © 2017 Frequentis AGPublicAssess your agility!|7
Control centre across industries
Public
Safety
Public
Transport MaritimeDefence
Air Traffic
Management

6. | © 2017 Frequentis AGPublicAssess your agility!|8
Public
Safety
Public
Transport MaritimeDefence
Air Traffic
Management

7. | © 2017 Frequentis AGPublicAssess your agility!9 |
Agile Health Check
Agile Assessment
Agile Journey Index

8. | © 2017 Frequentis AGPublicAssess your agility!|10
 “Agile Health Check”
- a wish by the agile community raised in 2015
- an idea requested by our corporate development process manager
 Various systems were floating around and being suggested
- Nokia scrum test
- RBI scrum assessment
- various web-based tests & benchmarks
 Management wants numbers
- “if you don’t measure it, you can’t manage it”
Better?
More responsive?
Better throughput?
Better workflow
distribution?
Better quality?
M1. CSR turnaround
time
M7. Total number of
external TRs
M8. Days open for
external TRs
M6. Flow diagrams
M5. Commit Pulse
M4. Business
Value/MH
M3.
Functionality/MH
M2. Feature
Cycle-Time
Questions
Metrics
Goal

9. | © 2017 Frequentis AGPublicAssess your agility!|12
1. Agile Teams shall have a tool (a kind of checklist) to allow them to
- self-determine where they stand today in their "Agility"
- recognize areas where they can improve, and which steps they can take on their growth path
2. Determine an overall "Agility" metric for Frequentis
- where do we stand as a company in our agile practices?
- how do we advance over time?
- are our measures (trainings, coaching, etc.) effective?
Constraints:
 Light-weight
 Simple roll-out (bottom-up)
 Incremental improvement of the assessment itself

10. | © 2017 Frequentis AGPublicAssess your agility!|13
Criteria to consider
 Can be done individually / needs tool, process, and service (assessors?)
 Can be adapted for Frequentis / industry standardised / controlled by another party
 Meaningful even if only parts of the organisation participate
 Organisational hierarchy
 Is rolled-out top-down or can be introduced bottom-up
 Quantitative vs. qualitative
 Measuring vs. discussing
 Need for data access to internal tools and databases
- Over years? Legacy data? Tools change: Compatibility and comparability of data?
- Ease of integration
 External tool: data security
 Benchmarking: against ourselves or against a larger data set (industry?)
 Licencing and cost

11. | © 2017 Frequentis AGPublicAssess your agility!|14
 Ruled out very simple systems, several external (web-based) tools, and anything that
needed lots of internal data
 Finally found the Agile Journey Index by William Krebs (“AgileBill”):
- Creative Commons Licence
- can be freely modified and tailored
- reasonably documented by the author
- covers a sophisticated range of questions and topics
- incorporates both team and program level
- calculates an overall value
- comes with cool badges and ribbons (a kind of gamification)
- no technical implementation needed (you fill out an Excel sheet)
Read more about the Agile Journey Index:
W. Krebs, P. Morgan, R. Ashton. The Agile Journey Index, 2012.
http://www.agiledimensions.com Agile Journey Index by “AgileBill” / William Krebs is licensed under
a Creative Commons Attribution-ShareAlike 3.0 Unported License.

12. | © 2017 Frequentis AGPublicAssess your agility!|15
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

13. | © 2017 Frequentis AGPublicAssess your agility!|16
Plus Minus
• Good categories
• Large range of program level
(organisation, trainings..) and
individual scores for each team
• Assessment questions stir up
intense discussions which in
turn bring more clarity
• Quick to fill out (takes ~1h)
• Badges 
• Some explanations allow
several interpretations
• Checklist has some ambiguities
• Comment field is missing to
enable better follow-up (what
was missing for the next level?)
Pilot
Jul-Sep 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Typical comments across 3 teams after self-assessment
smallWONDER Team

14. | © 2017 Frequentis AGPublicAssess your agility!|17
smallWONDER Team
smallWONDER
DIVERS
PBJN FIV
DTB
Frogs Assassins DDT

15. | © 2017 Frequentis AGPublicAssess your agility!|18
Teams raise concern about publishing of detailed scores
After the pilot, some questions were raised about
where and in what format the results would be
published.
Balanced publication: Reach the goals of the
assessment without fully disclosing every detail.
Goal 1: teams shall be able to determine where
they stand and what they can improve
- Individual results shared within the group of participants
- Teams can compare themselves to other participants
- Identify teams with areas of expertise
Goal 2: a company-wide agility metric
- Aggregated and anonymized data only
- Identify weaknesses common to many teams
- Calculate an overall agility metric

16. | © 2017 Frequentis AGPublicAssess your agility!|19
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

17. | © 2017 Frequentis AGPublicAssess your agility!|20
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

18. | © 2017 Frequentis AGPublicAssess your agility!|21
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

19. | © 2017 Frequentis AGPublicAssess your agility!22 |
Agile Assessment
Questionnaire details

20. | © 2017 Frequentis AGPublicAssess your agility!|23

21. | © 2017 Frequentis AGPublicAssess your agility!|24

22. | © 2017 Frequentis AGPublicAssess your agility!|25

23. | © 2017 Frequentis AGPublicAssess your agility!|26

24. | © 2017 Frequentis AGPublicAssess your agility!|27
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

25. | © 2017 Frequentis AGPublicAssess your agility!|28
Pilot
Closing
Event
Roll-out
Kick-off
Self-
Assessment
Peer
Review
Data
Aggregation
Jul-Sep 2016 Oct 2016 Nov 2016
Pilot assessment
with a handful of
volunteer teams.
Feedback
incorporated into
the assessment
sheet and info
material.
Kick-off for the
corporate-wide
rollout of a
Frequentis-
tailored version
of this index.
All product teams
(via PDMs, POs)
from our product
database were
invited.
Kick-off meeting;
info on intranet.
Self-assessment
driven by the
PDMs, POs,
Scrum Masters,
Dev Leads with
their teams.
Frequentis agile
coach helps to
clarify questions.
Deadline to send
back filled-out
forms.
Dec 2016
Each program
completes final
review session
with agile coach
(to normalize
assessments).
Self-assessment
is verified by
reviewer based
on proof by
evidences.
Feb 2017
Data aggregation
and preparation
for closing event.
Due to other
prioritised
activities this
finalization phase
took a few
months.
Apr 2017
Presentation of
results to
different
audiences.
Aggregated and
anonymised
results for all
audiences.
Detailed scores,
individualised
data only for the
participating
teams.

26. | © 2017 Frequentis AGPublicAssess your agility!29 |
Agile Assessment
Results 2016

27. | © 2017 Frequentis AGPublicAssess your agility!|30
 Invited Programmes: 37
 Addressed Product Managers: 30
 Participation
- 10 Programmes (= Jira projects)
- 18 POs/10 PDM
- 22 Teams
- 160 Individuals

28. | © 2017 Frequentis AGPublicAssess your agility!|31
Overall: Score StdDev Median
5,47 1,43 5,40

29. | © 2017 Frequentis AGPublicAssess your agility!|32

30. | © 2017 Frequentis AGPublicAssess your agility!|33

31. | © 2017 Frequentis AGPublicAssess your agility!|34
The details for all teams
Compare scores. Benchmark yourself.
Identify knowledge leaders for each topic.

32. | © 2017 Frequentis AGPublicAssess your agility!|35
 Agile Assessment during Nov/Dec 2016
 30% participation
 Company-wide score: 5,47
 Best category: Quality (~7)
 Lowest category: Leadership (~4.5)
 Some unclarity on programme level regarding the assessment of scaled teams
 Marketplace identified knowledge leaders => list on intranet, offer support to others
 Teams derived lots of new input from their internal discussions during assessment
Overall: Score StdDev Median
5,47 1,43 5,40

33. | © 2017 Frequentis AGPublicAssess your agility!|36
What‘s next?
Clarify how scaled setups shall be assessed on program level.
Integrate input from other companies that are now trying the assessment.
Publish the „Frequentis version“ of the Agile Journey Index (CC-BY-SA).
Re-run the assessment for 2017.
Iterate.
C
B
A
D

34. Public
Safety
Public
Transport
Air Traffic
Management
Maritime
Defence