Time Traveling: Adapting Techniques from the Future to Improve Reliability, Jacob Kitchel of Exelon

Time Traveling: Adapting
Techniques from the Future to
Improve Reliability
Jacob Kitchel
January 14, 2014

Bio
Present:
Security Architect at Exelon

Past:
Security & Compliance at Industrial Defender
ICS Risk Assessment (PT, VA, etc.)
Application Security research (Project Basecamp)

Enterprise Security Operations & Monitoring
Speaker (S4, EnergySec, ISA, API IT Security)
Hilarious LinkedIn Endorsements

1

Presentation Title

Abstract
Technology in ICS environments lags the Enterprise
by 10-15yr. This often leads to ICS companies
having to stand by while other more nimble
institutions are able to take advantage of new
technology. What few people realize, is that our
industry gets to watch the future happen out on the
Internet and then pick and choose the best
techniques to adapt and bring back in time.

2

Presentation Title

How far have we come?
We have:
• Compliance
• Incidents?
• Specialization

• Conferences
• Big Headlines?
• A LOT of vulnerabilities

3

Presentation Title

Where has it gotten us?

Here we are:
• Multiple revisions of compliance requirements
• Basic improvements in security monitoring
• SOME patching happens

4

Presentation Title

What is working against us?

Mountains or mole hills?
• Refresh cycles
• “If it isn’t broken, don’t fix it”
• Skill set(s)
• Unknown unknowns
• Security v. Operations
• Budgets & time

5

Presentation Title

Progress is sloooooowwwww….

6

Presentation Title

What to do?
• Where do operations goals and security goals intersect?
• What is the lowest common denominator?

• What can have an impact?

It’s all about the customer…
If you aren’t solving customer pain, then you aren’t doing anything

7

Presentation Title

It’s about the customer
Operations

Customer

Security

• Safety

• Security

• Reliability

• Compliance

• Uptime

• Vulnerabilities

Where do these two areas intersect?
8

Presentation Title

Where do Security and Operations Intersect?
• Patching
• Change Management

• Configuration Management

In other words…
• Time-intensive
• Error-prone
• High-risk activities

9

Presentation Title

Solving “Customer” problems lets you solve
security
How can we do that?

10

Presentation Title

Take a step back…to the future!

11

Presentation Title

How?
How can we:
• Reduce time commitments required
• Reduce errors
• Reduce risk

Has anyone solved this problem before?
Is there anyone that “looks” like us?

12

Presentation Title

Know any of these names?

13

Presentation Title

Internet-scale companies
• Millions of customers, world-wide
• High-availability, (near) zero downtime

• Complacency is death
• Some of the brightest minds >40
• Solving scale and complexity problems that we can barely imagine
• Leveraging software and hardware to dynamically define environments

• Have to be reliable and fast

14

Presentation Title

How are they doing this?
They are doing it CONTINUALLY.

Continuous Delivery:

Changes to your environment are
proven to be deployable
with predictable results

15

Presentation Title

But you say, “There’s a catch!”
Continuous Delivery was popularized by Internet companies!
Internet companies deliver software and/or services as their products!

They’re not like us! We have a physical process!
Etc, etc, etc…

Guess What?
Continuous Delivery is a collection of tools and processes – tools and
processes that you use to focus your ability to deliver your physical
process
Hint: You’re not getting off that easy! ;)
16

Presentation Title

What does this mean to us?
• Major reduction in time and effort to push changes

What would a major time/effort reduction mean to
your operations?
• 500hr task takes 5 hours or 5 minutes?
• 40hr task takes 4hr or 4 minutes?
• How many times do all of your tasks gets repeated annually?

• What if you could save half of that time and effort?

17

Presentation Title

How do we get there?
Automation Automation Automation Automation Automation Automation
18

Presentation Title

Continuous Delivery in Practice
How do you move a mountain?

19

Presentation Title

First steps first
• Follow your build/development process & write it all down
•

What takes the most time?

•

What tasks are the most error-prone?

•

What tasks require the most human intervention?
–

•

What tasks cause headaches or are time sinks?
–

20

Automate these tasks FIRST!
Automate these next!

Presentation Title

Facilitate Adoption
• Put everything into version control
• Add tests to verify that changes work

• Manage servers with configuration management tools
• Monitor EVERYTHING

21

Presentation Title

Tools
• Software-defined infrastructure

• Code Review

• Monitoring

• Configuration Management

• Continuous Integration

• Orchestration

• Version Control

• Dashboards

End Goal
• Quality
• Reliability

• Speed

22

Presentation Title

Tool Specific Information

23

Presentation Title

Software-defined Infrastructure
Tool example:

How to apply:

• Quali Systems TestShell

• Define common network
architecture and system objects
• Create test topology
• Run tests and see what breaks,
verify what works

24

Presentation Title

Version Control
Tool examples:

How to apply:

• Git

• Track versions of clear-text
configuration files

• SVN

25

Presentation Title

Firewall, switch, router
configuration files

•

• CVS

•

Application configuration files

Configuration Management
Tool examples:

How to apply:

• Puppet

• Store all configurations in
management tool

• Chef
• Ansible
• Salt
• Microsoft SCCM

26

Presentation Title

• As machines run, configuration
management tool ensures
declared configuration

Orchestration
Tool examples:

How to apply:

• Puppet

• Determine order of components

• Chef

• Leverage tools to operate,
deploy, and automatically
configure systems in proper
order

• Mcollective
• Ansible
• Capistrano

• WinRM

27

Presentation Title

Virtualization
Tool examples:

How to apply:

• Most common tool here is
VMWare and is likely your
vendor’s approved virtualization
provider

• Mirror Dev, Test, and Production
environments

28

Presentation Title

• Bonus: backup/redundant assets
• Can begin to act as a “do over”
button

Metrics & Dashboards
Tool examples:

How to apply:

• Logstash

MONITOR EVERYTHING

• Graphite
• Nagios
• Cactii

29

Presentation Title

Continuous Delivery tool
Tool example:

How to apply:

• Thoughtworks Go

• Automate and streamline the
build-test-release cycle

30

Presentation Title

Automated Testing
Tool examples:

How to apply:

• Thoughtworks Twist

• Write tests to verify functionality

• BDD/TDD tools

• Run tests automatically every
time new code, features, or
configuration changes are made

31

Presentation Title

Time Traveling: Adapting Techniques from the Future to Improve Reliability, Jacob Kitchel of Exelon

More Related Content

What's hot

Viewers also liked

Similar to Time Traveling: Adapting Techniques from the Future to Improve Reliability, Jacob Kitchel of Exelon

More from Digital Bond

Recently uploaded

Time Traveling: Adapting Techniques from the Future to Improve Reliability, Jacob Kitchel of Exelon