Zenefits principal engineer Venkat Thiruvengadam and SignalFx engineer Maxime Petazzoni discuss operationalizing Docker at scale. Learn about the transition to a well-conceived microservices approach, the tools chosen to support these services, and the lessons learned from monitoring containers in production in a high-performance environment.

1. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Operationalizing Docker at Scale
Lessons from running microservices in production
V e n k a t Thiruvengadam, Zenefits
M a x i m e P e t a z z o n i , S i g n a l F x

2. Introductions
Venkat Thiruvengadam
• Principal Engineer @ Zenefits
• Creator of Project Duplo, a hosting platform for
microservices
Maxime Petazzoni
• Software Engineer @ SignalFx
• Creator of MaestroNG, a container orchestrator
for Docker environments

3. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Micro-services @ Zenefits

4. Micro-services @ Zenefits
Thiruvengadam Venketesan
Principal Engineer

5. Zenefits Engineering
• 200 Developers
• Monolith Django Application on MySQL Backend
• Transitioning to SOA
• Dozens of services, few Hundred containers
• Hosted in AWS (No on premise servers)
• Duplo: Self-service Hosting platform for Microservices
• No Direct AWS access for developers
• No dedicated Devops

6. Monolith vs Microservices
• Monolith
• Advantages
• Single Code Base. Easy to read/debug/deploy
• No inter-service contracts, compatibility or authentication concerns.
• Easy to get started. Great for small teams.
• Disadvantages
• Single point of failure
• Slow iteration with large teams (Managed by central ops team)
• Single programming language & technology pattern
• Microservices
• Independent deployment and hence faster iteration
• Distributed Functionality, no single point of failure
• Programming Language & technology agnostic

7. Micro-services : The Dark Side
Container Management
Service Discovery
Configuration Management
Authentication
Management
Complexity w/o right
tools
AWS Resource Management
Resource Utilization Monitoring

8. Find Your Tipping Point
Microservices
Monolith
Team and Product Size
Productivity
@zenefits 3 years, > million lines of code and 200 devs, we decided to move to
Microservices

9. Original approach to Micro-services @Zenefits
Developers
Infra Admin
1
Translate to AWS
Configuration
2
Apply Configuration using
TerraformOther automation
3
Save Infrastructure configuration like VPC,
Subnet, Security Grp, IAM, EC2
4
5
Save Application
configuration like name
credentials in app code;
build docker image
6
7
Deploy to ECS
8
- Authentication tokens, service discovery, configuration management follow similar pattern
- Infra becomes bottle neck. Not scalable with large number services and users

10. A non self-service infrastructure defeats the purpose

11. I Get Self-service, Do I need Multi-Tenancy?
Multi-Tenant: Each application (or service) is a tenant. Users
have access to only their respective services
Multi-tenancy (app team) is necessary if your organization
needs:
- Resource and security isolation between services
- Isolation of configuration mistakes
- Audit trails
- Varying resource privileges between tenants
- Varying Developer security awareness
- Large number of developers
No tenancy
Team Size
Mis-config and Security Risks
Multi-tenancy

12. How can I do CICD in Duplo?
Let’s give you AWS access, create your service
What is this Security group, subnet, IAM? Why does my django
app need this? Will allow * work?
Hmm…..
Shall we create an AWS username for everyone?

13. We Need a Hosting Platform
Translate to AWS
Configuration
2
- Apply Configuration using
AWS and Docker APIs
- Inject configuration in app
ENV & discovery service
3
Application requirements and Docker Image via
UI/API
1
Developers
Admin
- Self Service. Deploy and manage at-will.
- Multi-tenant. Each service or application team have their own accounts, resource pool, IAM role, SG etc.
- Services isolated except explicit API interfaces.
- Programmatic Infrastructure, consistent configuration generation
- Create base infra Configuration in AWS using
Terraform
- Install Duplo. Setup static base infrastructure
policies.
- Create static Tenant policies and limits a.k.a.
“Plans”
0
DUPLO

14. DEMO: DEPLOYING A MICRO-SERVICE

15. Duplo Micro-services Platform
DUPLO
AWS Orchestration and
abstraction
Container Management
CI/CD
Ecosystem Orchestration
Authentication
Configuration Management
Service Discovery

16. How Duplo Works
Base Infrastructure: VPC, Subnets, NAT, Admin Security Groups, DNS Domain Name, SSL Wild char Cert
Terraform
DUPLO
Admin PortalNew Service Registration
IAM Role
Security Group
Auto create
per service
Security Group
IAM Role
New Service Registration
Aws
Resources
Tenant Portal
DUPLO
Created by
tenant actionsAws
Resources
Deploy and Manage Service

17. The Next Step: Continuous Integration and Delivery
Duplo can deploy images with desired AWS Resources. It’s self service.
Why do I need CICD?
• Official Build Images as against Laptop tests
• Test suites that are documented, enforced with pass criterion
• Integration environment for integration tests with other services
• Deployment Pipeline Dev sandbox to stage to prod.

18. What is special about a CICD solution for
Microservices, can I just use Jenkins?
- Monolith CI/CD is only about code correctness and build artifacts
- They can run in an Infra topology separate than the real deployment
- Beyond code, Microservices CI/CD is about replicating the real
deployment infra topology
- It needs to test the interface of the service with its complex infrastructure like
IAM, Security groups etc.
- Services CI/CD needs to integrate and build on top of the hosting platform

19. CICD DEMO: GIT PR To Deployment

20. KatKit: CI/CD Extension to Duplo
• After-all builds can be a Microservice too, they are just short lived! Let’s reuse Duplo to launch and
terminate builds.
• Let’s build a work flow orchestration on top of duplo for CI/CD.
• We get CI/CD and Hosting Platform “integration” for free

21. SUMMARY
• Determine the tipping point.
• Self service is P0. Determine if multi-tenancy (isolation) is necessary
• Hosting platform is necessary
• AWS by itself is not a sufficient solution, need a higher layer platform
to provide abstraction, container management and CI/CD
• CI/CD solution has to integrate with the hosting platform and test its
interaction with the service code.

22. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Q&A

23. For more information
• Duplo Platform
• https://engineering.zenefits.com/2016/03/duplo-aws-orchestration-container-
management-and-paas-for-microservices/
• https://engineering.zenefits.com/2016/07/duplo-cicd-extensions-for-microservices/
• https://www.linkedin.com/pulse/duplo-vs-kubernetes-mesos-thiruvengadam-
venketesan
• Monitoring Docker containers
• https://signalfx.com/blog/monitoring-docker-containers-take-get-started/
• https://signalfx.com/blog/monitoring-docker-containers-signalfx-monitors-
containerized-infrastructure/
• https://signalfx.com/blog/monitoring-docker-at-scale-with-signalfx/

24. M M / D D / Y Y
YOUR TITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
THANK YOU!
SIGN UP FOR A TRIAL AT:
signalfx.com