Keynote Dev Days vilnius 2018: how openness changes your behaviour

@spoole167@spoole167
Dashboards and Culture: How
Openness changes your
Behavior
Steve Poole, IBM
Slido #K100

What do these have in common?
Slido #K100

Or these?
Slido #K100

Our industry is constructed of silos
Slido #K100

@spoole167@spoole167Two of the biggest ones .. Slido #K100

Between Developers and IT teams
Slido #K100

And for a long time – its been a problem we’ve lived with

This talk is a story about how tackling the
communications gap made things better
• How it changed the behavior of the people involved..
• How you too can take a leap and leave your silo
Slido #K100

About me
Steve Poole
IBM Lead Engineer / Developer advocate
JVM Developer
Open Source Advocate
DevOps Practitioner (whatever that means!)
Driving Change
Slido #K100

Once upon a time..

Oct 2014 – Big Boss: Would you like to lead a European Dev IT group?
Oct 2014 - Me. : Yes – sounds like fun
Jan 2015 – Me : WTF?
Slido #K100

Modernising an IT group
I took over a large dev focused European IT team
that provided development services to developers
A team that would have been classified as “Slow IT”.
My job - modernise the team. to support the
emergence of “Fast IT”: provide development
services for cloud development teams
I learnt a lot.
Then I talked to customers etc. I learnt even more.

I learnt - executives are asking:
Slido #K100

Compliance
Resilience, security
IT Dev
I learnt IT and Dev don’t communicate: they just shout at each other in their own
language
Slido #K100

I understood – we just don’t share a common experience
Slido #K100

IT Dev
I learnt – making IT teams read the Phoenix Project can be counter productive
“I liked the book up and ‘til the part
where they started to modernise”
“So you want me to put
myself out of a job?”
Slido #K100

DevOps is about a new
contract between Dev &
Ops!
Slido #K100

IT
A contract realised in new forms of self-service assets like PaaS or
IaaS , Kubernetes, Docker etc
A contract that covers new availability requirements & speed to
market
Slido #K100

And like all good contracts it needs
to be written down in a mutually
agreeable language..

Like a Dashboard

What is a dashboard?

@spoole167@spoole167https://commons.wikimedia.org/wiki/File%3ANuclear_Ship_Savannah_-_Reactor_Control_Room_-_Center_and_Left_Panels.jpg
Reactor Control ?

@spoole167@spoole167https://www.flickr.com/photos/johnseb/ Bletchley Park,
Code Breaking ?
Slido #K100

https://www.flickr.com/photos/nrmadriversseat/
Car Dashboard? Slido #K100

A ‘real’ dashboard
provides
Just enough status : speed
Just enough insight: fuel / range
Just enough warnings: seat belt
Clear emergency indicators

Back to my story
Slido #K100

I really hate it when
someone tells me I have
a system that is down
before I notice.
Slido #K100

I really hate it when someone tells me I have
a system that is down before I notice.
Makes me feel like this

How do you stop this…
Slido #K100

@spoole167@spoole167 Slido #K100

Becoming this?
Slido #K100

(especially when .. )
Slido #K100

This is fun!

You start with something like this…
Slido #K100

Monitoring

I challenged my monitoring team to do all
that it took so that my group was always…
Slido #K100

“First to Know”

Simple monitoring to start
https://github.com/RuntimeTools/appmetrics

Simple technologies
https://prometheus.io/

Even simpler technologies http://dashing.io/
Slido #K100

Within a year everyone has a dashboard
• Every dev team
• Every support team
• Even managers
• And Executives..
In fact we build a network of
monitors with raspberry pi’s
To run the dashboard systems.
And they are monitored too..
Now over 4 million
dashboard events per day
Slido #K100

MQSeries 7.5 (MQTT)
Event
Store
Tivoli
Monitoring
Nagios
Big Brother
Monitoring
Backups
SAN
API's
Custom Scripts
RTC
GitHub
SCM’s
Dashboards
(IBM Cloud)
Continuous Delivery
 Build, test and deploy within 2
minutes
 Averaging 8-10 deployments a
day
Data
Sources
Insight
Slido #K100

Case Studies
Executive Dashboards
Ticketing Dashboards
Availability Dashboards
Developer Dashboards
Unusual Dashboards
Closing the Loop…

Executive Dashboards

@spoole167@spoole167 https://www.flickr.com/photos/ctsi-global/
Slido #K100

A battle ground
Executives want ’levers’ to pull
Endless reinvention as they look for new insights..
Middle management want to control the message and
reduce the time spent ‘wasted’ on responding
Since (the elephant in the room) we all know the data
is inaccurate, old and any predictions mostly useless
Slido #K100

Real life example
• Executive dashboard for tracking the migration of ‘on prem’ workloads into the
cloud.
• Objectives: Reduce capex IT spend and provide a more flexible provisioning environment
• Challenges included:
• Not being able to associate on-prem workloads with the correct teams
• Not being able to find all the servers
• Not being able to accurately determine performance characteristics
• Workload owners expected to end up with a worse environment so dragged their feet.
• The real challenge is that on-prem vs cloud is like ‘apples and oranges’ Not
directly comparable
Slido #K100

The solution
• Some hard work to create the right
accounting structures
• Classifications of the servers by type of
workload
• Classification of the on-prem workload
in terms of cloud characteristics. CPU’s
memory, multitenancy to bare metal etc
• An Executive dashboard that’s live and
accurate 24x7
0
2
4
6
8
10
12
Team 1 Team 2 Team 3 Team 4
Un assessed On Prem Units Able to move moved

Culture changes
• Better predictions of cost of moving workloads to cloud +
classification = each team can realistically discuss the cost/benefit of
moving their workloads to the cloud with Executives
• Managers can justify budget increases more effectively –they can
show how there existing capacity is being used.
• IT support now have an improved understanding of what their clients
are using machines for.

All teams now want and need
the data to be accurate
•Which lead to improved
communications
between teams and
reduced the shouting

Ticketing Dashboards
Or how sharing is easy

IT team structures
• The SME silo approach of IT teams is a major
inhibitor to Fast IT
• It was optimal when delivery schedules were
measured in months and years.
• When working in hours and days – it’s not
tenable. We needed to reoptimise…
SAN
Team
Network
Team
VMware
Team
Monitoring
Team
• This organisation causes
delays
• makes it almost impossible
for anyone to understand the
status of a request
• Promotes segregation
• Makes It easy to lose sight of
objective and priority

Solution – part 1
• Spread a little light on the situation
• Measure what you have.
• Dashboard what you have.
• Dashboard what the teams think important
• Make the dashboards public
SAN
Team
Network
Team
VMware
Team
Monitoring
Team

Team 1 Team 2 Team 3 Team 4
Todays
In/Out
Average
response
time
<1 day <1 day <1 day <1 day
backlog 100 500 350 1000

Solution – part 2
• List critical tickets
• Measure ticket end-to-end times
• How long to close 80% of the tickets
• Make the dashboards show trends & backlogs
• Highlight the different processes that each team uses
• Add in dashboards for tickets that are not owned by any of the teams
SAN
Team
Network
Team
VMware
Team
Monitoring
Team
Showing the real, overall picture
from an end user point of view
helped everyone see where the
problems were

-60
-40
-20
0
20
40
60
80
100
120
Jan Feb Mar Aprl
In Out Diff Backlog

Solution – part 3
• Reduced the number of teams by a factor of 10
• Scrubbed the backlogs. We closed >50% of the tickets with an apology.
• Standardised on a single flexible process.
• Reduced the number of ticket configuration states to a simple set – no more
complicated SQL queries and less likely to loose tickets
• Published teams real initial response times & time-to-resolution figures.
• Added warning indicators to response time trends etc
Team A Team B

Result
End users could see
reality – they knew how
long it would really take
to get something done.
They could see that the
teams were prioritizing
and working effectively

Availability Dashboards
Or how sharing brings understanding

Service Availability – a tale of impedance missmatch
• How do you calculate service availability?
• What’s your unit of time?
• What’s your availability period - 9x5, 24x5,
24x7?
• For SaaS products it is 24x7 ( minus very rare
scheduled downtimes)
• But what does available mean?

Development IT team
• Provides a critical service to developers
• Asked to provide 24x7 support
• Dev IT: “we already do that”
• Dev teams beg to differ – service has regular downtimes..
• Dev IT : “Our availability calculations don’t have to
include regular scheduled downtimes”
• Dev (and me) WTF?
• what about all these other unexpected downtimes?
• Dev IT : “They are caused by events beyond our control
and so are not included”
• Dev (and me) WTF?

Insight - Iceberg syndrome
• Many IT groups revolve around the SME
• Items move from one to another to get
completed so very few people have a vision of
an IT service as perceived by the end user.
• Published data is more for the service
providers use than the end user
• End users think they are dealing with an IT
department – but in reality they are dealing
with a small handful of people who have
responsibility for a part of the service.

Solution
• Have the IT team understand they are
responsible for reporting the availability of
their service whatever the cause of
downtime.
• Have the IT team spend time with their users
to see what can be done to mitigate the
impact of scheduled downtimes
• Defined what available meant (hint not just
pingable)

Culture changes?
• Creating a better definition of ‘available’
meant the IT team learnt what was important
to their end user
• Dashboards with ‘real’ status means that end
users don’t have to try to get answers via
ticket or phone - less stress. More trust.
• end users see the truth and can trust the IT
team more.

Developer Dashboards
Or how you can never share enough

Unusual Dashboards
Or how you can never have enough of a good thing

Build Server upgrades
• A chore.
• Several hundred machines needed urgent security patching
• mostly by hand! ( a long story)
• How do you keep focus and motivate the teams who are doing this?
https://commons.wikimedia.org/wiki/File:US_Navy_090716-N-8590G-011_Members_of_the_damage_control_team_of_the_guided-
missile_frigate_USS_John_L._Hall_(FFG_32)_compete_in_the_pipe-patching_event_of_the_Damage_Control_Olympics.jpg
Navy Pipe Patching event

You make it a competition
Navy Pipe Patching event
Team A
Team B
Team C
Team D
Team E

Diagnostic ’one-off’ dashboard
Jenkins build server problems..

Continuous delivery code/complexity metrics
Red indicates over complex code.
Radius indicates number of lines of code

Closing the Loop

We’re experimenting
with integrated, shared
dashboards

Shared between the customer,
development teams, sales and
executives.
Customers can see exactly what’s
going on and are now part of the
process.
Live playbacks (with the customer
leading over webex, zoom, etc
One
Dashboard

customer insights: renewals,
profitability status, general
‘temperature’ etc
Allows sales teams to be fully
aware of customer situation
One
Dashboard

Did I get my ”First to
know”?

Sort of
• Most teams now look at their dashboards
instead of monitoring emails, tickets etc.
• But so do our clients
• So it’s more that we know together

But most importantly..
Our customers know we know.
They understand how we work and they can see us working
Mutual trust through openness..
Teams feel more empowered, more in control of their work
Know they are doing the right thing

So what have we learned?
That people
do change
1
Change comes
from mutual
understanding
2
By sharing
openly you’ll
learn more
3
All from moving
a little out of
our comfort
zones.
4

This week.
Try something
different
maybe go listen
to something
new
something not
relevant to
your day job.

You’re here to learn - go learn something different.

Thank you

What did you originally expect to get from
using a dashboard?
“A quicker way to see if any of our servers go down
so that we would know before the users.”
“Immediate and regular insights into metrics
important to our daily operation”
“To help us spot broken Jenkins builds quickly.”

Are you using dashboards in different ways
than expected?
“our customers use the dashboards so they know that we know if there is
a problem with their server - and so they can see more quickly when it is
available after a scheduled outage.”
“we are using it to give a very quick overview of our ticket backlog -
something which we never expected to do.”
“My squad now has a separate dashboard which we use as a quick-
reference source to help us plan day-to-day to spot high-importance bugs
/ production bugs / new pull requests etc”

how has having dashboards changed your
team(s) behavior?
“we are in much better control of issue levels, pull
requests, critical issues etc”
“Reacting to build failures as a stop-the-line event,
rather than not noticing for hours/day etc”
“Exposing debt backlogs in an easy-to-consume way
helps drive down and keep on top of tech debt”

Keynote Dev Days vilnius 2018: how openness changes your behaviour

Recommended

Recommended

More Related Content

Similar to Keynote Dev Days vilnius 2018: how openness changes your behaviour

Similar to Keynote Dev Days vilnius 2018: how openness changes your behaviour (20)

More from Steve Poole

More from Steve Poole (20)

Recently uploaded

Recently uploaded (20)

Keynote Dev Days vilnius 2018: how openness changes your behaviour

Editor's Notes