DevOpsDays Baltimore 2017. In high security environments, we are often behind proxies, firewalls or obnoxious corporate policies that disallow access to Github or RubyGems. What gives?! In this session, I will talk about what problems we need to solve to build and manage environments in an offline world and how infrastructure as code is at the heart of making it happen.

1. Don’t Mind the Gap
Doing DevOps in an Air-gapped World
Galen Emery
Federal Solutions Architect
3/8/2017

2. Who am I?
I work with the Federal Government and
Integrators
DoD, Civilian, or IC? Yes.
I help users understand how to build and
structure their infrastructure (using Chef) to
solve their problems
Former Windows Server Admin
Been with Chef since March 2014
Born and raised in Seattle (Nathen, you’re
wrong about Seattle crab)

3. What we cover
What is “air-gapped”?
What issues do we encounter?
Assumptions
How to solve it (technical)
How to solve it (process)
What about … ?
Food for Thought

4. What is air-gapped?
air-gapped
adjective
(of a computer) having no direct
connection to the Internet or to any
other computer that is connected to
the Internet, for security reasons.
"a USB drive or other hardware
approach would be required to
infect the air-gapped machine"

5. What about mitm?
• Corporate man-in-the-middle
• Terminates SSL sessions and re-initiates
• Often includes packet inspection
• Requires you to trust its generated certificates
• May still include a firewall blocking access to sites

6. Maybe its just a firewall
• Restricts access to:
• Github or gist.github.com
• slack
• Pastebin
• Rubygems.org / some other artifact repository
• etc

7. No restrictions?
• Congratulations!
• You might still have a
firewall/proxy, its just permissive
• Do you trust those systems will
be up when you need them?

8. Issues

9. Getting stuff into production
• How do we “cross the gap”?
• How do we distribute it?
• How do we ensure its integrity?

10. Process?!
You want to talk about PROCESS? You kidding me?!
I just hope we can make one change!

11. Velocity
How do we achieve velocity if we burn a DVD for every change?
Or if a proxy is inspecting all of our packets, every time we download a
package?

12. Okay, I’m not air-gapped but…
• I have a firewall between my systems and the internet
• I have a proxy that inspects all traffic and slows me down
• I don’t have {level of access necessary} to make changes
• Tool {X} does not understand authenticating proxies

13. Ground Rules

14. You know that I know that you know…
• How you get code into your
high-side environment

15. Everything is code
• If its source code, it goes into SCM
• If it is an artifact, it goes into artifact store, and has a checksum
• We can (and do) write tests
• We build a pipeline

16. Solution (High-Level)

17. Go from Low to High

18. Assume you don’t have internet
• Even if you do
• Build this into your pipeline
• Test this on the low side
• Your systems should NOT assume internet access
• Most tooling by default assumes it does

19. Do this work on the low-side
Build your pipeline in a way that removes internet access for your systems,
they must grab their code and artifacts locally
If you can build your infrastructure without internet access in Dev, you can
absolutely do it in Prod

20. Protect the Data, not the Infra
The data is sensitive, not the infrastructure itself. Keep the data in the high-
side, but ensure that you build your infrastructure with the same code in Dev
and Prod

21. Workstation
This is your “loading dock” for the rest of the infrastructure
Everything comes through here
It needs:
• A way to serve files (directly or indirectly)
• A way to create artifacts (zip, tar, etc)
• built programmatically

22. Workstation
Use the workstation to:
• Stand up your artifact repo
• Stand up your configuration management infra
• Publish your artifacts
• Run tests

23. Workstation (Chef Example)
Bring in to the workstation a zip/tar with the following
• ChefDK
• A FTP/SSH/SCP Server binary (if it doesn’t already exist on your box)
• Your cookbooks
• Any extra gems necessary
• Chef packages (client, server)
Use Chef Zero to stand up your FTP Server, populate it with the artifacts and
prepare it for use in the rest of the process

24. What about dependencies?
I could download, transfer, attempt to install, download, transfer, attempt to
install..
I could skip using the gem and instead write it myself
I can create a full gem mirror of all 80,000+ gems on rubygems.org
I can install what I need into a directory, and then move that directory over as
an artifact

25. Simplest is to create an artifact
I used “gem install –i $PATH $GEM” && tar cf $PATH
Its not pretty but it works
Benefit is: I have a moment-in-time artifact of the gems I’m using
Ideally you’d create this at the end of your development pipeline

26. You should be using an artifact repo
• Use something that can store and manage your artifacts
• You can version your artifacts
• Often supports the correct dependency structure for your artifacts

27. Once you have Workstation + Artifact
• You can setup your configuration management
• Use a tool, running from your workstation to setup the infrastructure you
need
• Example
• Chef Provisioning SSH
• Doesn’t require internet access
• Can bootstrap Chef from a FTP or SCP server

28. Everything is set, right?
• Have workstation, Config Mgmt + Artifact store
• But I haven’t told any of my infrastructure that it shouldn’t reach out to the
internet for X, Y or Z

29. Tell your systems to stay inside
• Remove unreachable Satellite repositories
• If using ruby, remove rubygems from your sources. Add your artifact store
• If using Chef, update berksfile to an internal supermarket
• If using X, update Y to Z
• Don’t do this manually, do it with code and test it on the low side!

30. What if I have a proxy instead?
Most of the tooling supports HTTP_PROXY and HTTPS_PROXY
Some of it does not
It is often easier to design the system to assume no network access than to
keep fighting proxies
That said, if you can poke holes out to slack, rubygems, github, chef, etc you’ll
be much happier (unless they go down)

31. Changing the Process

32. Do this in Dev
• Do not wait for Prod to test your systems without internet access
• Along these lines, harden your Dev systems to the same standards as Prod

33. Create and Use
• CI/CD Pipelines
• Unit and Integration Tests

34. Identify where you assume risk
• If you let users ssh/rdp into production, that is where the risk lies
• If you only let users make changes to production through a pipeline, that’s
where the risk lies
• If the risk lies in a pipeline and someone else has to approve code changes,
the risk of any single change/actor is much lower

35. Imagine This Scenario
• Nobody can ever log into Production
• All changes flow through a pipeline that tracks who committed, reviewed
and shipped the change
• All changes are tested through Dev, QA, etc before deploying to Prod
• We build a new Prod every time we make a change

36. What About … ?

37. My laptop can’t reach X
• If you can’t reach GitHub, or Slack, etc ever; not just in Production that’s an
issue

38. Security won’t let me!
• Work with them. They are a vital component of
your business (or should be)
• Identify where the risk is. The risk is with the
data, not with the user.
• Consume Risk in Dev, not in Prod (Fail Fast)
• Progress is coming

39. How often do we update our packages?

40. Where are the gates?

41. What are the metrics?
Velocity: How fast we ship
Efficiency: How good we are at shipping
Risk: How long does it take us to fix

42. Food for Thought

43. Alaskan King Crab
• Biggest crab, legs alone can be a meal
• Served steamed or chilled with drawn butter and lemon
• Fishing Season:
• October
• January

44. Dungeness Crab
• Similar size to blue crab, hard shell
• Served steamed or chilled with drawn butter and lemon
• Fishing Season:
• Starts in November
• Ends in June/July

45. Blue Crab
• #CrabFeast
• Old Bay
• Fishing Season:
• Starts In April

46. Proof of Concept code
www.github.com/galenemery/chef-mre
Want a USB Key with it?

47. Google “stuxnet” or “vault 7”
If you asked for a USB Key, please let me know who holds your clearance so
we can have it revoked

48. ACKNOWLEDGEMENTS
Robb Kidd
ChefConf 2014: Mind the Gap:
Deployments with Chef in
unforgiving Environments

49. ACKNOWLEDGEMENTs
Thank you JOKES ON YOU, THERES NOTHING HERE
Thank you DevOpsDays Baltimore