Kevin Behr: Integrating Controls and Process Improvement

Integrating Controls and Process
Improvement

d.
ve
er
es
sR
ht
Kevin Behr

ig
CTO IP Services
ll R
,A
This space left intentionally blank
03
20

Key fingerprint = AF19 FA27 2F94 998D FDB5 DE3D F8B5 06E4 A169 4E46
te
itu
st
In
NS
SA
©

Kevin Behr - Integrating Controls and Process Improvement
3- 1A
© SANS Institute 2003 No copying, electronic forwarding or posting All Rights Reserved

Agenda

The Problem : Are we smoking more and enjoying it less?

d.
What we did about it. Control is possible!

ve
How we did it.

er
Blood, Sweat and VisibleOps

es
Measuring the results. The IMCA and other useful metrics

sR
What we have built

ht
ig
ll R © 2003 Tripwire, Inc. 2
,A
We invest in redundancy and have smart engineers. Why is our
03

infrastructure so unreliable?
20

Key fingerprintthere are best 2F94 998Dfor security and audit butA169 4E46
I know = AF19 FA27 practices FDB5 DE3D F8B5 06E4 what about
the ops guys?
te
itu

These best practice volumes read like the tax code. How do I go
st

about implementing substantive change when all I have to go by
In

is a picture of utopia?
NS
SA
©

3- 2A

The Problem

d.
ve
er
es
sR
ht
ig
ll R
,A
03
20

te
itu
st
In
NS
SA
©

3- 3A

The Problem

IDC, Meta etc say that security incidents cause less
than 3 percent of down time.

d.
IDC Meta etc say that Hardware and environmental

ve
issues cause less than 6% of down time.

er
Why aren’t our production systems more reliable?

es
Why are our Ops people so busy and why are

sR
service levels getting worse? Our Data Center is
always on fire!

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3- 4A

The Problem - Humans

Changes that authorized, tasked and directed IT
people make cause 78%of all system outages!

d.
Our current way of working does nothing to address

ve
this.

er
Many companies spend millions on change

es
management systems – only to have them

sR
circumvented and never know it.

ht
ig
,A
IDC reports that authorized change by humans represents almost 80
03

percent of all IT outages.
20

te
itu
st
In
NS
SA
©

3- 5A

The Problem - Humans

Many companies have developers maintaining
production servers because of downsizing.

d.
In many companies Security and Operations have

ve
an adversarial relationship. Ops undoes what

er
security puts in place. Security breaks what Ops

es
provisions trying to minimize risk.

sR
Much of the critical knowledge on how things
“Really work” lives in a few very busy minds.

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3- 6A

The Problem- The way we work it

Studies show that up to 80% of problem resolution
time is spent determining the nature of the problem.

d.
The balance is spent actually correcting or

ve
bypassing the problem.

er
Ops is so consumed with fighting fires that there is
little or no accurate documentation of existing

es
systems.

sR
There are no accurate golden builds – New servers
are like snowflakes – No two are exactly the same.

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3- 7A

The Problem – Integrity Drift

The purpose of deployed infrastructure “drifts” or
changes over time. Suddenly a mail server is now

d.
also a DNS server, a DHCP server .

ve
Security is reduced to using detective controls to

er
figure out what ops is deploying after the fact.

es
New services deployed instantly become mission

sR
critical but there is no way to re-create the server
that has evolved over time..

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3- 8A

What we did about it

d.
ve
er
es
sR
ht
ig
ll R
,A
03
20

te
itu
st
In
NS
SA
©

3- 9A


Used a twelve step program to determine that we
were powerless over our propensity to “light and

d.
fight” ops fires.

ve
We came to the conclusion that we needed a higher

er
power (ITIL) and that if we worked the program we
could find our way to Serenity and many nines of up

es
time.

sR
We vowed to share our experience with others
along the way.

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 10 A

What we did – The Higher Power

We needed a framework to put all of our activity
into. So we could understand what it was we were

d.
supposed to be doing.

ve
The framework we chose was the Information

er
Technology Infrastructure Library or ITIL (eye-til)

es
Pros – Very Large and comprehensive

sR
Cons- Very Large and very descriptive (what it looks
like) – we needed Prescriptive (what to do)

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 11 A

What we did about it - What is ITIL?
British Office of the Crown Government authors many well-
known documents, including ISO17799 (BS7799) Created
They realized Ops best practices have never been

d.
documented, and created ITIL (IT Infrastructure Library) and

ve
BS15000 to describe how world-class Ops processes
Extremely widely used in Europe, but gaining acceptance in

er
the U.S.

es
HP OpenView, CA UniCenter, and IBM Tivoli are all basing their EMS
products on ITIL terminology

sR
ComputerWorld 10/7/2002: Proctor & Gamble reports saving $125
million per year on IT cost savings (10-15% of their annual IT

ht
budget)

ig
,A
IT Infrastructure Library (ITIL) is the only consistent and
03

comprehensive documentation of best practice for IT Service
Management. Used by many hundreds of organizations around the
20

world, a whole ITIL philosophy has grown up around the guidance
contained within the ITIL books.
te

ITIL consists of a series of books giving guidance on the provision of
itu

quality IT services, and on the accommodation and environmental
facilities needed to support IT. ITIL has been developed in
st

recognition of organizations' growing dependency on IT and
In

embodies best practices for IT Service Management.
The ITIL Online : http://www.ogc.gov.uk/itil/
NS

The Office of Government and Commerce (owners of ITIL)
SA

http://www.ccta.gov.uk/
©

BS15000 / BS 15000 is the world's first standard for IT service
management. The standard specifies a set of inter-related
management processes, and is based heavily upon the ITIL (IT
Infrastructure Library) framework. The BS15000 Site
http://www.bs15000.org.uk/

3 - 12 A

What Is “Visible Ops?”
A closed-loop process methodology, aimed at increasing
Operational efficiencies and increasing service levels
Based on studying “best in class” enterprise operations

d.
Visible Ops goals

ve
A small subset of ITIL and BS15000 frameworks, for terminology,

er
processes, and future improvements

es
Intended to 80% of the benefits at 20% of ITIL effort
A “step by step” approach to three fundamental service management

sR
disciplines
Methodology authors:

ht
Gene Kim, CTO, Tripwire, Inc.

ig
Kevin Behr, CTO, IP Services, Inc.

,A
03
20

te
itu
st
In
NS
SA
©

3 - 13 A

What we did about it – VisibleOps

Gene Kim and I studied many enterprise operations
(A major trading company, The largest wireless

d.
carrier, a major stock exchange) and we began to

ve
note that these organizations had successfully

er
implemented and benefited from preventive and

es
detective control combinations.

sR
These controls were used to create audit points that
made it easy to understand known good states.

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 14 A


We also began to see that if the infrastructure state
was understood early on in the problem

d.
management cycle the time it took to accurately

ve
determine the nature of the problem could

er
drastically be reduced.

es
We would be able to stop many inappropriate and

sR
costly over-escalations if we could rule out change
as early as possible.

ht
ig
,A
When examining Problem resolution reports it was noticed that if
03

change could be ruled out early the time it took to close the ticket
was reduced.
20

Most every organization has a star quarterback in operations, and
te

security. Many groups thought that everything wound up escalating
itu

to this person because the overall environment had grown so
complex that only a few people could solve what used to be simple
st

problems. This often results in a serious moral problem for the
In

brightest staff. We needed to put them in to an advisory role where
they coach and consult instead of fighting fire full time on the front
NS

lines. The ultimate goal is to free up enough their time to turn them
loose on creating additional operational efficiencies and process
SA

improvement.
©

3 - 15 A


Best in class operations had bounded remediation
times for critical infrastructure.

d.
In order to have valid golden builds to accomplish

ve
this the change management process must have

er
more teeth than just the “honor system”.

es
These organizations also displayed the earliest

sR
integration of security in to the Ops lifecycle

ht
ig
,A
We spoke to many large IT groups and heard them complain about
03

the ineffective nature of their change management systems. One
CTO even complained that his engineers were often so busy and
20

backlogged in firefighting that they didn’t feel like they had enough
time to even work through the Change Management processes.
te

This meant that changes made during firefighting were never even
itu

documented!
st

Security would be completely on their own to detect and respond to
In

these ad-hoc changes. They would certainly never know who made
the changes let alone if they were made by friend or foe (although
NS

the odds are with “friend”)!
SA
©

3 - 16 A

Best In Class Ops and Security
Best in class Ops and
Security organizations
have:
•Highest

d.
Server/sysadmin ratios

ve
•Lowest Mean Time To
Repair (MTTR)

er
-Highest Mean Time

es
Between Failures
(MTBF)

sR
•Earliest integration of
Security into Ops
lifecycle

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 17 A

How we did it

d.
ve
er
es
sR
ht
ig
ll R
,A
03
20

te
itu
st
In
NS
SA
©

3 - 18 A

Where Is The Leverage?
Ensure that I can control
Ensure that I have predictability changes in my world in the
around what goes into production production environment

d.
ve
er
es
sR
ht
ig
Help me learn to do this in an
automated fashion. Equip me to deal with problems

ll R
efficiently and feed the results
back into my environment

© 2003 Tripwire, Inc. 19
,A
Shift resources from fire fighting to implementing release
03

management, controls and resolution processes.
20

te
itu
st
In
NS
SA
©

3 - 19 A

Process Area Objectives
Release Management
Ensure that provisioned systems match the “known, good build”
Promote repeatable builds for all configurations

d.
Control Processes

ve
Ensure that changes can be traced to a valid business reason

er
Create a control point, where Ops, Dev, or Security can so stop a
change from occurring

es
Control configuration drift and uncontrolled changes

sR
Incident Management / Resolution
Decrease MTTR (mean time to resolve) outages

ht
Increase “culture of causality,” allowing better diagnosis and problem

ig
management practices

,A
03
20

te
itu
st
In
NS
SA
©

3 - 20 A

How we did it – Stabilize the patient

Attack the 80%. Stop the bleeding caused by:
change drive-bys ,integrity drift and changes made

d.
during firefighting.

ve
We used the combination of a preventive control

er
(don’t touch that fence it’s electric!) and a detective

es
control (why did you touch the fence at 2:11 am on

sR
March 3rd?) to get a handle on the state of every
piece of critical infrastructure.

ht
ig
,A
Audit change and configuration controls
03

Tools: Tripwire, Tivoli auditing components, reports from
change management tools
20

Key fingerprint = AF19 FA27 2F94footprints to DE3D F8B5 06E4 A169 4E46
Audit configuration 998D FDB5 ensure compliance
te

Map all changes to authorized work order
itu

End-of-shift audit requires Ops managers to handover data
center in the same state as they received it
st
In
NS
SA
©

3 - 21 A

How we did it – Catch and Release

We caught and foot-print audited all critical
infrastructure configurations in the wild.

d.
We created golden builds for these devices.

ve
We tested and set bounded remediation times for

er
all critical infrastructure.

es
We determined audit frequency and methods

sR
necessary to support these times .

ht
ig
,A
Create repeatable builds
03

Tools: Tivoli Configuration Manager, Tivoli Remote Control
and others (Norton Ghost, InstallShield AdminStudio, Linux
20

QuickStart, Sun Jumpstart)
Automated provisioning of OS, configuration files,
te

applications, and business rules
itu
st

Create acceptance process
In

Tools: Tripwire
NS

Ensure that provisioned servers matches “known, good build”
SA
©

3 - 22 A

How we did it – Manage the Change

Instituted a Change Advisory Board- Stake holders
include: Security Lead ,Ops Systems Engineering

d.
Lead, VP of Operations , Service Desk Manager,

ve
Director of Network Operations, and Internal Audit.

er
Made weekly change management meetings

es
mandatory for all CAB members.

sR
Implemented a Change Transaction Process to
make the correct path : Request For Change (RFC)

ht
ig
,A
Create change transaction workflow
03

Control points to document, authorize, schedule or deny, and
audit change requests
20

Create change control meetings (include Security)
Tools: Tripwire, reports from change management tools (such
te

as trouble ticketing system)
itu
st
In
NS
SA
©

3 - 23 A

How we did it – Managing Change

All RFC are categorized based on a 1-4 severity
system. Anything above a 2 goes to the CAB for

d.
review and comment.

ve
Changes can only be administered during

er
maintenance windows and must be approved and

es
scheduled by the CAB.

sR
Urgent changes trigger an emergency CAB
meeting.

ht
ig
,A
Simple Change Management Meeting Agenda:
03
20

Discussion of:
Failed Changes, backed-out Changes, or Changes that may have
te

circumvented the CAB
itu

RFCs to be assessed by CAB members
Requests For Change that have been assessed by CAB members
st

Change reviews
In

The Change Management process, including any amendments
NS

made to it during the period under discussion, as well as proposed
Changes
SA

Change Management wins/accomplishments for the period under
discussion, i.e. a review of the business benefits accrued by way of
©

the Change Management process.
Review of Next Action assignments based on the above discussion.
Dismiss.
Meetings should have minutes taken and distributed to the CAB
following the meeting.

3 - 24 A

How we did it - First Response

Modified the problem management process to
eliminate change as early as possible by identifying

d.
the assets directly involved in the ticket and auditing

ve
them against their configuration baseline for the last
72 hours. All changes found are attached to the

er
ticket.

es
If no changes are found the circle is widened to

sR
include changes made to infrastructure supporting
the target systems.

ht
ig
,A
Create inventory of all relevant evidence around issue or outage
03

Tools: Remedy/ CA Service Desk /Tivoli Configuration
Manager and Tripwire; Configuration and asset management
20

information
All relevant scheduled and authorized changes
te

Actual changes on target system
itu
st

Formalize post-incident assessment and reconciliation of changes
In

Tools: Tripwire, reports from Tivoli, reports from ticketing
NS

system
Ensure that changes are understood
SA

Ensure that changes are incorporated into documentation and
propagated to other systems, as appropriate
©

3 - 25 A

Measuring the results

d.
ve
er
es
sR
ht
ig
ll R
,A
03
20

te
itu
st
In
NS
SA
©

3 - 26 A

Measuring the results - The IMCA
Based on IT Infrastructure Library (ITIL) / BS 15000
standards and the Visible Ops methodology
An interview-fueled process with a standardized scoring

d.
methodology

ve
Focuses on high leverage areas:

er
Release Processes

es
Control Processes
Resolution Processes

sR
ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 27 A

Measuring the results – IMCA Questions

All questions are answered with a number, “from zero to
four”

d.
0: Strongly disagree

ve
4: Strongly agree
Sample questions

er
“Our IT department is understaffed to meet current workloads.”

es
“Our Service levels are spiraling downwards.”

sR
“We can enforce a standard build across all our devices.”
“We have a library of automated build systems for all our critical

ht
devices.”

ig
“We have a clearly defined change control policy.”

,A
03
20

te
itu
st
In
NS
SA
©

3 - 28 A

Measuring the results- IMCA report

d.
ve
er
es
sR
ht
ig
,A
This organization has no Request for Change process. Not having a
03

correct path for changes to follow assures that they will go the path
of least resistance and least documentation. Creating more gasoline
20

to throw on the fire.
te
itu
st
In
NS
SA
©

3 - 29 A

Measuring the results- IMCA report

d.
ve
er
es
sR
ht
ig
,A
This represents a pretty tight shop with some room for improvement.
03

They need to build on their strengths in audit and process to shore
up their change transaction processes. Some detective control
20

would certainly help their ailing rollback capabilities.
te
itu
st
In
NS
SA
©

3 - 30 A

Reliability and Validity of IMCA

Validity measures
Based on IT best practices frameworks of ITIL and

d.
BS15000

ve
Questions are scored on the integrity of three key ITIL
processes

er
Reliability measures

es
All answers are subjective, and can vary from day to day

sR
All answers do not have any quantitative significance (i.e.,
arithmetic operations cannot be done on the answers)

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 31 A

Measuring the results- Other Metrics

Number of changes made in data center
Number of changes that map to authorized business

d.
reason

ve
Number of times change management system was

er
circumvented

es
Percent of outages caused by change

sR
Number of changes that obsolete repeatable builds
Ops “clean shift handover” success rate

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 32 A


Time to provision known, good build
Number of fixes/turns to match known, good build

d.
Percentage of deployed systems that match known,

ve
good build

er
Percentage of deployed systems that have security

es
sign-off

sR
ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 33 A


Outage and issue Mean Time To Repair (MTTR)
Aggregate outage downtime

d.
Number of inappropriate escalations

ve
Increased change success rate

er
es
Increased systemic Mean Time Between Failure

sR
Smile to frown ration on Ops, Security and Audit
staff

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 34 A

What you have built - You Can Now:
Enforce change management process integrity
Decreased firefighting and increase proactive
controls

d.
Avert revenue loss due to unplanned outages

ve
Decrease Mean Time To Repair by efficient problem

er
management processes

es
Create hard organizational change boundaries for

sR
accountability and responsibility
Establish a beach head for operational best

ht
practices, allowing future process improvement

ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 36 A

What you have built

You now can measure and articulate the business
benefit of process improvement efforts

d.
You can target weak areas for quick wins

ve
Regain the confidence of the business by showing

er
off your new and improving metrics

es
Fend off IT Budget Jenga with your CFO and CEO

sR
by showing where money needs to be invested and
why.

ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 37 A

Contact Information

Gene Kim, CTO, Tripwire, Inc.
genek@tripwire.com

d.
Kevin Behr, CTO, IP Services, Inc.

ve
kevin.behr@tcpipservices.com

er
es
sR
ht
ig
,A
03
20

te
itu
st
In
NS
SA
©

3 - 38 A

Kevin Behr: Integrating Controls and Process Improvement

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (15)

More from Gene Kim

More from Gene Kim (20)

Kevin Behr: Integrating Controls and Process Improvement