Donnerstag, 19. September 13
“How automated cloud infrastructure setups
can help with Continuous Delivery”
Edmund Haselwanter, eh@cloudbau.de, @ehaselw...
Why Continuous Delivery?
For most companies, IT functions as the nervous system and provides
an increasing amount of the o...
Continuous Delivery
Wikipedia: http://en.wikipedia.org/wiki/Continuous_delivery

●
●
●
●

Version Control
Continuous Integ...
We got the software development part right!
Request

Approve

Develop

Test

WaterScrum

Deploy

- Fall

The core, chronic...
The Last Mile: Dev/Test/Prod Parity
●
●
●
●
●
●
●
●

Hardware configuration (e.g. amount of RAM, HDD vs. SSD)
Software use...
The Last Mile: Dev/Test/Prod Parity
●
●

Log level “normal” to chatty on production

●
●

Changes in external dependencies...
Getting to the solution...

cloudbau
Donnerstag, 19. September 13
Infrastructure as Code

Application

Environment

Infrastructure

cloudbau
Donnerstag, 19. September 13
Building Blocks: Tools and Culture
Config-DB

CPUVirt.
Binary-Repo

StorageVirt.
SDN

System Description
Orchestration

Pr...
First... automate and equalize all
environments
●
●
●
●
●

Chef/Puppet/Cfengine/.... for config
Dependency management (for...
... then use a cloud to equalize
infrastructure!

●
●
●

Encode the infrastructure as well
Describe all the pieces of the ...
But ... There be dragons
Infrastructure as Code => We need a process

Source
Code
Test

X

Compiler
Artifact

cloudbau
Don...
The implementation patterns using OSS

● OpenStack / Eucalyptus / ...
● Chef/Puppet/Cfengine/ ...
● Jenkins (Building and ...
New Question in the Cloud Era: What is my
deploy Artifact

● The Software Package itself
● The whole VM
● A prepared conta...
The Software Package is the Artifact

● The build artifact is a binary.
● Use IaaS API to create the environment
● Deploym...
HP Cloud + Jenkins + Puppet

Binary Repo

HP Cloud Servers
(OpenStack)

Jenkins

Puppet Modules

cloudbau
Donnerstag, 19. ...
The VM itself is the Artifact

● Build artifact is a image (optimal for stateless service)
● Only install needed, there is...
Jenkins + AWS + Asgard + Aminate + Chef

Jenkins

Aminate

AMIs

Asgard

+ Chef

AWS EC2 (ASG + ELB)

cloudbau
Donnerstag,...
A container is the Artifact

●
●
●
●
●

cloudbau
Donnerstag, 19. September 13

Optimization of the VM idea
Transportable l...
https://www.docker.io/the_whole_story/

cloudbau
Donnerstag, 19. September 13
Jenkins + docker + cfengine + Aminate +
Chef

Jenkins

Docker

Host VM

Maestro

+ cfengine

AWS EC2 (ASG + ELB)

cloudbau...
Cloud can help with developer environments
as well

●
●
●
●

cloudbau
Donnerstag, 19. September 13

Have your own environm...
OpenStack + Vagrant + Chef
Chef Repo / Server

Binary Repo
Jenkins

Vagrant

cloudbau
Donnerstag, 19. September 13

OpenSt...
One more Thing: Other Areas Cloud can
help

● Persistent Data: Copy/Clone Production Data to test migrations
● Debugging: ...
What’s missing in the OpenSource
Ecosystem?

● Better Orchestration/ Pipeline support in Jenkins (Commercial CI-Servers
ar...
Don’t forget about Operations

● Cloud/App-Monitoring Solutions (Nagios won’t cut it in dynamic
environments)

● Logfiles
...
Thank you! Questions?

cloudbau
Donnerstag, 19. September 13
Upcoming SlideShare
Loading in...5
×

How automated cloud infrastructure setups can help with Continuous Delivery

458

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
458
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
17
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How automated cloud infrastructure setups can help with Continuous Delivery

  1. 1. Donnerstag, 19. September 13
  2. 2. “How automated cloud infrastructure setups can help with Continuous Delivery” Edmund Haselwanter, eh@cloudbau.de, @ehaselwanter Donnerstag, 19. September 13
  3. 3. Why Continuous Delivery? For most companies, IT functions as the nervous system and provides an increasing amount of the organizational muscle mass. Most critical business functions are entirely automated within IT, and 95% of all capital projects depend on IT to get done. Today, nearly every business decision will result in at least one IT change. „IT REVOLUTION MANIFESTO“, http://itrevolution.com Theory of Constraints → → E.M. Goldratt, 1984 cloudbau Donnerstag, 19. September 13 Gene Kim, Kevin Behr, George Spafford, 2013
  4. 4. Continuous Delivery Wikipedia: http://en.wikipedia.org/wiki/Continuous_delivery ● ● ● ● Version Control Continuous Integration Automated Testing Continuous Deployment cloudbau Donnerstag, 19. September 13
  5. 5. We got the software development part right! Request Approve Develop Test WaterScrum Deploy - Fall The core, chronic conflict that every IT leader faces is the need to simultaneously enable faster time to market (i.e., make as many changes as you can), while providing stable, secure and reliable IT services (i.e., make as few changes as you can).  Gene Kim „Lowering risk of change through tools and culture“ John Allspaw, Paul Hammond, 2009 (Flickr) small changes, often reproducible Donnerstag, 19. September 13 culture change Dev+Ops standardized cloudbau expect failure feedback (e.g. tests/metrics/...)
  6. 6. The Last Mile: Dev/Test/Prod Parity ● ● ● ● ● ● ● ● Hardware configuration (e.g. amount of RAM, HDD vs. SSD) Software used (e.g. sqlite in dev, mysql in prod) Software/library versions Deployment (e.g. automated for test, manual for prod) System configuration (debug flags, logging) External dependencies (dev: payment mocked out vs. prod: real gateway) Database contents (huge prod DB with millions of entries vs. few entries in dev) Operating system (e.g. dev: Mac OS X, prod: Linux) See also http://12factor.net/ cloudbau Donnerstag, 19. September 13
  7. 7. The Last Mile: Dev/Test/Prod Parity ● ● Log level “normal” to chatty on production ● ● Changes in external dependencies (Sun Java to Oracle Java) ● ● ● ● SQL Query no problem in dev environment but too slow for production data ● ● Snowflake back-port: Live Tuning of Kernel/Mysql parameters, got lost after re-install CI build broken due to minor version change in upstream mocking framework (0.4.1 to 0.4.2) Sessions invalid in production due to time difference on multiple servers. No problem in dev environment (just one app and one db server) Different hardware: 32 vs. 64 bit. libraries missing Different hardware: optimized CPU flags on CI system. build not working in prod Staging In-house, Prod AWS Cloud: Latency ok during staging test, problem on production Development on Mac OSX/Windows, deploy on Linux: File not found (case sensitive) See also http://12factor.net/ cloudbau Donnerstag, 19. September 13
  8. 8. Getting to the solution... cloudbau Donnerstag, 19. September 13
  9. 9. Infrastructure as Code Application Environment Infrastructure cloudbau Donnerstag, 19. September 13
  10. 10. Building Blocks: Tools and Culture Config-DB CPUVirt. Binary-Repo StorageVirt. SDN System Description Orchestration Prod Env Programmable Infrastructure (Cloud) Continuos Delivery Anti-Fragile Organizations Cross-Functional Teams Processes and Culture cloudbau Donnerstag, 19. September 13 Dev Env Executeable System Description Continuous Build DevOps Stage Env
  11. 11. First... automate and equalize all environments ● ● ● ● ● Chef/Puppet/Cfengine/.... for config Dependency management (for programming languages, packages etc.) Dependency management for config scripts as well (e.g. Berkshelf for chef) Everything in version control Infrastructure can still be different... cloudbau Donnerstag, 19. September 13
  12. 12. ... then use a cloud to equalize infrastructure! ● ● ● Encode the infrastructure as well Describe all the pieces of the infrastructure in code Now it’s code: it can be dynamic! cloudbau Donnerstag, 19. September 13
  13. 13. But ... There be dragons Infrastructure as Code => We need a process Source Code Test X Compiler Artifact cloudbau Donnerstag, 19. September 13
  14. 14. The implementation patterns using OSS ● OpenStack / Eucalyptus / ... ● Chef/Puppet/Cfengine/ ... ● Jenkins (Building and testing the SW but also infrastructures) ● NetflixOSS (Asgard + Aminate) cloudbau Donnerstag, 19. September 13
  15. 15. New Question in the Cloud Era: What is my deploy Artifact ● The Software Package itself ● The whole VM ● A prepared container ● ... cloudbau Donnerstag, 19. September 13
  16. 16. The Software Package is the Artifact ● The build artifact is a binary. ● Use IaaS API to create the environment ● Deployment is after the server creation ● Deploy with automation or any other tool ● The whole VM ● “from source” is just a special case cloudbau Donnerstag, 19. September 13
  17. 17. HP Cloud + Jenkins + Puppet Binary Repo HP Cloud Servers (OpenStack) Jenkins Puppet Modules cloudbau Donnerstag, 19. September 13
  18. 18. The VM itself is the Artifact ● Build artifact is a image (optimal for stateless service) ● Only install needed, there is no upgrade/deploy ● IaaS must support Loadbalancer as a service (or something similar) ● Orchestration for deployment steps needed cloudbau Donnerstag, 19. September 13
  19. 19. Jenkins + AWS + Asgard + Aminate + Chef Jenkins Aminate AMIs Asgard + Chef AWS EC2 (ASG + ELB) cloudbau Donnerstag, 19. September 13
  20. 20. A container is the Artifact ● ● ● ● ● cloudbau Donnerstag, 19. September 13 Optimization of the VM idea Transportable lightweight OS containers (e.g. LXC/docker) Very often back to upgrade/deploy Needs external orchestration as well Very new toolchain emerging (docker/maestro)
  21. 21. https://www.docker.io/the_whole_story/ cloudbau Donnerstag, 19. September 13
  22. 22. Jenkins + docker + cfengine + Aminate + Chef Jenkins Docker Host VM Maestro + cfengine AWS EC2 (ASG + ELB) cloudbau Donnerstag, 19. September 13
  23. 23. Cloud can help with developer environments as well ● ● ● ● cloudbau Donnerstag, 19. September 13 Have your own environment Brings the production setup to the developer One description for local VM and Cloud environment (Dev/Prod) Needs external orchestration as well
  24. 24. OpenStack + Vagrant + Chef Chef Repo / Server Binary Repo Jenkins Vagrant cloudbau Donnerstag, 19. September 13 OpenStack Servers
  25. 25. One more Thing: Other Areas Cloud can help ● Persistent Data: Copy/Clone Production Data to test migrations ● Debugging: Developer can stand up whole environment (even with data) ● Backup/Restore (snapshot whole images/servers for rollback) cloudbau Donnerstag, 19. September 13
  26. 26. What’s missing in the OpenSource Ecosystem? ● Better Orchestration/ Pipeline support in Jenkins (Commercial CI-Servers are advanced here) ● OpenSource Orchestration Engines (like OpenStack Heat) ● NetflixOSS working on OpenStack cloudbau Donnerstag, 19. September 13
  27. 27. Don’t forget about Operations ● Cloud/App-Monitoring Solutions (Nagios won’t cut it in dynamic environments) ● Logfiles ● Metric driven autoscaling cloudbau Donnerstag, 19. September 13
  28. 28. Thank you! Questions? cloudbau Donnerstag, 19. September 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×