CI/CD with Docker, DC/OS, and Jenkins

© 2016 Mesosphere, Inc. All Rights Reserved. 1
CI/CD WITH DOCKER,
DC/OS, AND JENKINS
( Velocity NY 2016 Edition )

KARL WHO?
Where?
- Mesosphere
- Pivotal
What?
- DC/OS
- Kubernetes (on Mesos)
- CloudFoundry
- BOSH
- github.com/karlkfi
- twitter.com/karlkfi
- linkedin.com/in/karlkfi
- karl.isenberg.us

WHO ARE YOU?

INTEGRATION
Build/Test
Manual Review/Merge
Build/Test
Test all the pieces
together
DELIVERY
Integration
Test Release/Deployment
Manual Acceptance
Prod Release/Deployment
Integration branch is always
deployable
DEPLOYMENT
Integration
Validation
Release/Deployment
Integration branch is
validated and deployed
automatically
CONTINUOUS EVERYTHING

SOURCE
Branched > Developed > Compiled > Tested > Reviewed > Merged > Tagged
PACKAGES
Built > Tested > Versioned > Released > Promoted > Installed/Vendored
SERVICES
Deployed > Tested > Accepted/Validated > Promoted
RELEASE vs. DEPLOY

PAIN
- Expensive manual merges
- Independent refactoring
- Incompatible dependencies
- Works on my machine
- Different environments
- Who owns integration?
- Mismatched Interfaces
- Large upfront planning &
communication
WHY CONTINUOUS INTEGRATION?
RELIEF
- Small change sets
- Feature branches
- Package extraction
- Interface extraction
- Provisionable environment
- Centralized testing
- Automatic build/test triggers
- Configuration as code
- Integration package

PAIN
- Manual, undocumented release
process
- Siloed, tribal knowledge
- Bus factor of one
- Poor process communication
- Infrequent end-to-end testing
- Bugs caught right before/after
release/deployment
- Slow onboarding
- Manual regression testing
WHY CONTINUOUS DELIVERY?
RELIEF
- Programmatic dependency management
- Automated package management
- Automated release building
- Automated deployment
- Automated promotion
- Secrets management
- Unit/integration/acceptance tests
- Test/UAT/Acceptance environment

PAIN
- Unhappy customers
- Infrequent/late validation
- Many/large simultaneous
changes
- Making decisions without data
- Unhappy developers
- Process limiting innovation
- Infrequent feedback
- Long development cycles
- Large upfront planning
WHY CONTINUOUS DEPLOYMENT?
RELIEF
- Automated front-end tests
- Automated end-to-end tests
- Programmatic definition of acceptance
- Acceptance driven development
- Milestone releases, not release
milestones
- Release train
- Decouple feature promises from release
numbers
- Automatic semantic versioning

TOOLS

DOCKER
( Container Runtime, Image Format, and Repository )

DOCKER CONTAINER
isolated process tree
isolated filesystem
optional volumes
optional network routing
optional resource constraints

RUNTIME
The docker-engine runs on linux machines and manages containers and container
images.
IMAGE
A container image is a set of file system layers that can be shipped around as a unit
and is used as a template for creating containers.
REPOSITORY
A docker image repository is a service that manages and store docker images and
image layers.
CONTAINER ...

BUILD ONCE, RUN ANYWHERE
- Portable runtime environment
- Language independent
- Dependency vendoring
- Packaging format
- Dependency isolation
- More developer control
- Portable across operating systems
- Fast development cycles
HAPPY DEVELOPERS & OPERATORS
CONFIGURE ONCE, RUN ANYTHING
- Predictability
- Repeatable deployment
- Consistent, reliable results
- Low overhead, low cost
- Fast, efficient deployment
- Image Layer caching
- Developer awareness of operational
concerns
- Separation of concerns

CONTAINER IMAGE LIFECYCLE
Image
Image
Image
Layer
Machine 1 Machine 2Image Repository
Shared Layer
Layer
Layer
Push
Image
Pull
Dockerfile
Build
Container
Run

DOCKER LAB
1 - 2

DC/OS
( Container Orchestration )

IN THE BEGINNING...
Sombrero Galaxy by ESA/Hubble - CC BY 4.0

© 2016 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
19
COMPUTERS

Hardware
Operating System
Web Application
20
INTERNET
- Remote Users!

Web App
Hardware
Operating
System
21
DISTRIBUTION
- Horizontal Scale
- Fault Tolerance
- Availability
- Load Balancing
Operating
System
Operating
System
Hardware Hardware
Web App Web App

Service Service Service
Web App Web App Web App
Hardware
Operating
System
22
SERVICE-
ORIENTED
ARCHITECTURE
- Separation of
concerns
- Optimization of
bottlenecks
- Smaller teams
- API Contracts
- Data replication
- Complicated
provisioning
- Dependency
management
Operating
System
Operating
System
Hardware Hardware

Service Service Service
Web App Web App Web App
Machine
Operating
System
23
HARDWARE
VIRTUALIZATION
- Fast provisioning
- Isolation
- Portability
- Utilization
- Configuration
Management
- Virtual Networking
- Credential
management
Operating
System
Operating
System
Infrastructure
Machine Machine

Operating
System
Operating
System
Operating
System
ServiceApp ServiceServiceAppApp
24
MICROSERVICES
- Polyglot
- Single Responsibility
- Smaller Teams
- Utilization
- Machine
types/groups
- Dependency hell
Machine
Infrastructure
Machine Machine
ServiceService ServiceServiceServiceService

ServiceApp ServiceServiceAppApp
OS
25
CONTAINERS
- Rapid deployment
- Dependency
vendoring
- Container image
repositories
- Spreadsheet
scheduling
OS OS
Machine
Infrastructure
Machine Machine
Container Runtime Container Runtime Container Runtime
ServiceService ServiceServiceServiceService

Service
Container Runtime Container Runtime Container Runtime
OS
26
CONTAINER
ORCHESTRATION
OS OS
Container Orchestration
Machine
Infrastructure
Machine Machine
App ServiceServiceAppApp

CONTAINER ORCA-WHAT-NOW?
Scheduling, Coordination, & Management
of
Containerized Processes
as
Higher Order Abstractions
(e.g., Jobs, Services, Applications, Pods, etc.)

Orchestration
28
Machine Infrastructure
Web Apps & Services
Scheduling
Resource Management
Container Runtime
Machine & OS
Service Management
CONTAINER
ORCHESTRATION
Machine & OS Machine & OS
Container Runtime Container Runtime
Expanded

SCHEDULING
- Placement
- Replication/Scaling
- Resurrection
- Rescheduling
- Rolling Deployment
- Upgrades
- Downgrades
- Collocation
RESOURCE
MANAGEMENT
- Memory
- CPU
- GPU
- Volumes
- Ports
- IPs
- Images/Artifacts
SERVICE
MANAGEMENT
- Labels
- Groups/Namespaces
- Dependencies
- Load Balancing
- Readiness Checking
FUNCTIONAL ORCHESTRATION CAPABILITIES

DC/OS LAB
1 - 2

DC/OS
( The Datacenter Operating System )

SCALABILITY
Performance, Responsiveness, Efficiency
AVAILABILITY
Fault Tolerance, Robustness, Reliability, Resilience,
Disaster Recovery
FLEXIBILITY
Format Support, Interoperability, Extensibility,
Container Runtimes
USABILITY
Familiarity, Maintainability, Compatibility,
Debuggability
PORTABILITY
Host OS, Cloud, Bare-Metal, Hybrid
SECURITY
Auditability, Secrets Management, Encryption,
Isolation
NON-FUNCTIONAL PLATFORM QUALITIES

PLATFORM
A base of technologies on which other technologies or processes are built.
PLATFORM AS A SERVICE
A managed, multitenant platform.
CLOUD PLATFORM
A hosted platform as a service on shared infrastructure.
PLATFORM SPECTRUM
Rainbow by Susanne Nilsson - CC BY-SA 2.0

INFRASTRUCTURE PLATFORM
e.g., OpenStack, VMware vSphere
CONTAINER PLATFORM
e.g., Kubernetes, DC/OS, Rancher
APPLICATION PLATFORM
e.g., Cloud Foundry, Red Hat OpenShift, Deis
34
PLATFORMS
Oil Platform by Berardo62 - CC BY-SA 2.0

CLOUD INFRASTRUCTURE PLATFORM
e.g., Amazon Elastic Compute Cloud, Microsoft Azure,
Google Compute Engine
CLOUD CONTAINER PLATFORM
e.g., Google Container Engine, Azure Container Services,
Amazon EC2 Container Service
CLOUD APPLICATION PLATFORM
e.g., Heroku, Google App Engine, Pivotal Web Services,
IBM Bluemix
35
CLOUD PLATFORMS

APPLICATION DEVELOPMENT LIFECYCLE
Source Repo, Continuous Integration, Artifact Repo
CONTAINER ORCHESTRATION
Scheduling, Resource Management, Service
Management
DAY 2 OPERATIONS
Debugging, Maintenance
36
CONTAINER OPERATIONS

Code
Build, Test,
Package
Publish Deploy

Applications & Services
38
Source Control
Continuous
Integration
Artifact
Repository
Orchestration

Applications & Services
Source Control
Continuous
Integration
Artifact
Repository
Orchestration

DEBUGGING
- Logging & Metrics
- Emitting
- Aggregation
- Search
- Reporting
- Alerting
- Dashboards
- Shell Access
MAINTENANCE
- Package Management
- Upgrading
- Cluster Resizing
- Application Autoscaling
- Capacity Planning
- Software Defined Networking
- Backups
- Disaster Recovery
DAY 2 OPERATIONS

System Space
Orchestration
User Space
Source Control
Continuous
Integration
Artifact
Repository
41
Load Balancer
Package
Manager
Security
Networking
Logging &
Metrics
Storage

Kernel
42
OPERATING
SYSTEM
System Space
Hardware
User Space

DISTRIBUTED
OPERATING
SYSTEM
Hardware Hardware Hardware
User Space
Circa 1992
Microkernel
System Space
Microkernel Microkernel

Machine Infrastructure
User Services
Orchestration
Container Runtime
System Services
DISTRIBUTED
OPERATING
SYSTEM
Container Runtime Container Runtime
Revised
OS OS OS
Machine Machine Machine
DistributedOS

DC/OS
- Resource Management
- Universal Container Runtime
- Container Orchestration
- Pluggable Schedulers
- Jobs & Services
- Distributed System Services
- Virtual Networking
- Package Management
- Services Catalog & Application Ecosystem
- Day 2 Operational Focus
ENTERPRISE DC/OS
- E2E Encryption
- Identity & Access Management
- Secrets Management
- Support

HYPERSCALE OPERATIONS
ULTIMATE INFRASTRUCTURE FLEXIBILITY FOR MODERN APPLICATIONS
• Many “snowflakes”
• Management nightmare
• Lengthy cycles to deploy code
• Low utilization
Mesosphere DCOS
MESOSPHERE DCOS APPROACH
Container
Apps
(All) Big Data Analytics
(All)
PaaS
(All)
Stateful Service
(All)
• High performance and resource isolation
• Easy scalability and multi-tenancy
• Fault tolerant and highly available
• Highly efficient with highest utilization
Proven in
mission-critical
deployments
Deploys on-prem or in cloud
TRADITIONAL APPROACH
PaaS
1
Container
App
1
Big Data
Analytics 1
Big Data
Analytics 2
PaaS
2
Container
App
2
Stateful
Service
1
Stateful
Service
2
Complete workload
portability

USE, REPRODUCTION, OR DISCLOSURE IS SUBJECT TO RESTRICTIONS SET FORTH IN CONTRACT NUMBER 2014-14031000011 AND SUB CONTRACT NO. MESOSPHERE 01.
VPC
DC/OS ARCHITECTURE
DC/OS CLI
DC/OS UI
DC/OS Universe
Auto-ScalingGroup&SecurityGroup
ELB
ELB
LDAP
www
AGENT (1..n)
47
Backend Services
Spartan
Minuteman
Mesos DNS
Mesos Agent
3DT
Admin Router
Docker
User Tasks
Cassandra
Spark
Kafka
HDFS
Containers...
Apps/Tasks
Jenkins
Auto-ScalingGroup&SecurityGroup
MASTER (1..9)
Admin Router
Backend Services
Spartan
Minuteman
Mesos DNS
Resolv-Conf
Exhibitor
Zookeeper
DC/OS Signal
CA
Vault
Marathon
Mesos Master
History
3DT
Networking API
Cosmos
Bouncer
Frontend Services
Metronome

DC/OS LAB
3 - 5

JENKINS
( Open Source Automation Server )

JENKINS LAB
1 - 5

EXIT SURVEY

CI/CD with Docker, DC/OS, and Jenkins

More Related Content

What's hot

Similar to CI/CD with Docker, DC/OS, and Jenkins

Recently uploaded

CI/CD with Docker, DC/OS, and Jenkins