In this talk, we will share how a small team at Intuit moved Docker from local to production serving real and critical workloads. We will share how we addressed the organization challenges of running Docker at large enterprises by building a business case for a pilot project to prove the value of containers and its real world application. Next, we will share how we solved the technical challenges that present themselves when taking Docker from local to production in a corporate data center. We will share the blueprint for the business case and the associated pilot which laser focused on running stateless back-end services throughout the entire SDLC. Finally, we will highlight our crawl-walk-run approach that allowed us to make inexpensive mistakes before investing in the right areas as our Docker knowledge increased. We will share the major technical issues we encountered, how we overcame them and the lessons we learned.

1. Taking Docker from Local to Production @ Intuit
JanJaap Lahpor
Senior Manager, DevOps,
Intuit
Harish
Jayakumar
Lead Solutions Engineer,
Docker Inc

2. Happy Tax Day, Dockercon!
200,000+ professional tax preparers
25,000,000+ tax returns

3. In this session we will share our journey :
• The Intuit ProConnect business case for Docker
• The pilot project to prove the business case
• The problems that we encountered
• The lessons that we learned
So how did we get here?

4. ”Discovered” Docker December 2013
Dockercon 2015 June 2015
Dockercon 2016 June 2016
Docker 1.12 August 2016
Docker is grassroots at Intuit…
…we got serious when 1.12 included native swarm

5. It is difficult to explain what Docker is…
…but easy to highlight the benefits it delivers

6. Developer Productivity Hybrid Hosting Patching-as-Code

7. Patching-as-Code
Patching-as-Code
branch
testdevelop
commit
local
pushpull
ci
jenkins
pr
deploytestbuild
merge
Stacks, Image Signing and Image
Scanning.

8. Alright, go prove it!
But…
• Start small.
• Solve for containers
only.
• In our own data
centers.

9. Mission-based team
with specialties in:
• Operations
• Development
• Performance
• Security
Partnership with a
strong vendor for the
bad days.
No “I” in Docker

10. Hosting:
• Isolated hosts, running
RHEL7.
• No data stored inside the
containers or on the
hosts.
• No container-to-
container
communication.
• Use existing supporting
technologies.
Guard rails for speed

11. Applications:
• Only Java 8, stateless,
services.
• Tomcat servers running
RHEL7.
• Applied to the entire
SDLC: local to
production.
Guard rails for speed

12. It is not “hello world” on
your MacBook
anymore.
Not local anymore

13. Docker for Enterprise
Docker Trusted Registry
Secure image management & distro
Docker Engine
Container Runtime, Orchestration, Networking, volumes, plugins
Universal Control Plane
App and Cluster management
Security
Content Trust,
RBAC, LDAP/AD
MonitoringCI/CD NetworkOS Volumes LoggingConfig MgtImages ..more..
Public Cloud Physical/ConvergedVirtualization
Infrastructure

14. Docker Enterprise
Edition
Docker Enterprise
Edition
Docker Enterprise
Edition
Docker Enterprise
Edition
NetworkOS Monitoring LoggingConfig MgtImagesCI/CD
Physical/Converged
Infrastructure
RHEL7 Jenkins
s
F5 Chef NewRelic SplunkTomcat
The ProConnect Implementation

15. Bare Metal
We choose bare metal
over virtual machines:
• Less hosts to manage,
patch, monitor, and
license.
• More control.
• Led to real savings in
corporate
chargebacks.

16. 1 in 1
1 service in production
in 1 sprint.
9 in 5
9 services in
production in 5 sprints.

17. The problems that we
encountered

18. Stale ingress load balancer
Docker 1.12
• Updating a Docker service
could leave an old, dead,
container in the ingress
load balancer.
• No easy way to list pool
members of ingress load
balancer.
Docker 1.13
• Health checks for
containers in ingress load
balancer.

19. The trouble with bare metals
The issues
• All containers share the
host’s TCP/IP stack.
• Intermittent outbound
connection errors when
traffic increased.
• Confused firewalls when
we tuned TCP/IP on the
hosts.
The system fixes
• Add more IP addresses to
host using virtual
interfaces.
• Use macvlan to give each
container a TCP/IP stack.

20. The trouble with bare metals
Applications were the real
problem:
• Tune connection & socket timeouts
instead relying upon the default
value.
• Remove stale connections from
connection pool before sending a
request.
• Retries on specific exceptions.

21. Zombie containers
The wrong way
• Allow automated OS
patching to occur on your
Docker hosts.
• Patch kernel packages on
live Docker hosts.
The right way
• Drain a Docker host and
remove from swarm before
any OS patching.
• Add patched host back
into swarm.
• Repeat until all hosts are
patched.

22. Lessons learned
You need partners in the
trenches with you.
You need to be
transparent and set the
right expectations with
your teams.
You need room for
unexpected problems
and mistakes.
You need to be open to
frequent change.

23. Lessons learned
You need to resist
onboarding too many
teams, too early.
You need your
developers to work in
their logging and
monitoring tools.
You can create millions
of tax returns and use
Docker.

24. Want to use
Docker at work?
careers.intuit.com
@docker
#dockercon
@intuit
#intuitlife
@JanJaapLahpor
@harish_jkumar

25. Appendix

26. varunalight / 123RF Stock Photo | anueing / 123RF Stock Photo | qdxjw / 123RF Stock Photo
pilens / 123RF Stock Photo | scanrail / 123RF Stock Photo | bowie15 / 123RF Stock Photo
Copyrights