SlideShare a Scribd company logo
1 of 72
Next Step in Automation:
Elastic Build Environment
Kohsuke Kawaguchi / CloudBees, Inc.
kk@kohsuke.org / @kohsukekawa

©2013 CloudBees, Inc. All Rights Reserved

1
Have You Met Jenkins?

http://jenkins-ci.org/

©2013 CloudBees, Inc. All Rights Reserved

2
©2013 CloudBees, Inc. All Rights Reserved

3
©2013 CloudBees, Inc. All Rights Reserved

4
©2013 CloudBees, Inc. All Rights Reserved

5
My Jenkins around 2006

©2013 CloudBees, Inc. All Rights Reserved

6
©2013 CloudBees, Inc. All Rights Reserved

7
©2013 CloudBees, Inc. All Rights Reserved

8
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/gbyrnes/912576883/

9
If only we had more computers…
• Just building & testing them all…
• Running tests more frequently
• Testing individual commits

©2013 CloudBees, Inc. All Rights Reserved

10
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/drocpsu/8546730021/

11
Elasticity

Just enough computers
just in time

©2013 CloudBees, Inc. All Rights Reserved

12
My Jenkins around 2007

©2013 CloudBees, Inc. All Rights Reserved

13
©2013 CloudBees, Inc. All Rights Reserved

14
©2013 CloudBees, Inc. All Rights Reserved

15
©2013 CloudBees, Inc. All Rights Reserved

16
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/drocpsu/8546730021/

17
©2013 CloudBees, Inc. All Rights Reserved

18

18
Elasticity!

Just enough computers
of the right kind
just in time

©2013 CloudBees, Inc. All Rights Reserved

19
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

20
Correct answer
• Test assumes a fixture running on port 8080
– Doesn’t check if it’s already being used

• If another test runs at the same time…?

©2013 CloudBees, Inc. All Rights Reserved

21
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

22
Correct answer
• Because of “pkill -f -9 tomcat” cleanup

©2013 CloudBees, Inc. All Rights Reserved

23
http://www.flickr.com/photos/jumilla/8667648797/
©2013 CloudBees, Inc. All Rights Reserved

24
Isolation
• At odds with large multi-core systems
• x86 virtual machines
• User isolation
• Kernel containers

©2013 CloudBees, Inc. All Rights Reserved

25
©2013 CloudBees, Inc. All Rights Reserved

26
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

27
Correct answer
• Same Maven ID, two different jars

• Different projects designate different ones
• Local cache gets cleaned up periodically
• Whichever first runs after cache cleanup
“wins”
©2013 CloudBees, Inc. All Rights Reserved

28
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/82219206@N00/7003641975/

29
Correct answer
• Test script leaves background daemon
process behind
• Over time it’ll slowly choke slaves

©2013 CloudBees, Inc. All Rights Reserved

30
©2013 CloudBees, Inc. All Rights Reserved

31
Elasticity!

Throw away & create new

©2013 CloudBees, Inc. All Rights Reserved

32
©2013 CloudBees, Inc. All Rights Reserved

33
Ladder to Cloud

Elastic
Multiple

Single

©2013 CloudBees, Inc. All Rights Reserved

34
Parallel Testing

Test Group #1

Test Group #2

©2013 CloudBees, Inc. All Rights Reserved

Test Group #3

35
Parallel Testing

foo #10

Test Group #1

foo #11

Test Group #2

foo #12

Test Goup #3

©2013 CloudBees, Inc. All Rights Reserved

36
Validated Merge: Jenkins Enterprise by
CloudBees

upstream
repo
gate
repo

©2013 CloudBees, Inc. All Rights Reserved

37
Recap: Elasticity Benefits
•
•
•
•

Just-in-time capacity
Diversity without overhead
Isolation
Productivity gain
– parallel testing
– validated merge
– keep failed tests on the side

©2013 CloudBees, Inc. All Rights Reserved

38
Solid OSS Elasticity Plugins
• EC2 plugin
• Jclouds plugin
– OpenStack, CloudStack

• Launch and tear down slaves on demand

©2013 CloudBees, Inc. All Rights Reserved

39
©2013 CloudBees, Inc. All Rights Reserved

mage © http://sfcitizen.com/blog/wp-content/uploads/2011/11/6302790910_c4eb865892_o-copy.jpg

40
Distributed builds with Jenkins
• Master
– Serves HTTP requests
– Stores all important info

• Slaves

slave1
slave8

– 280KB single jar
– Assumed to be unreliable
– Scale to 100s

slave7

slave2

Master

slave6

slave3

slave4
slave5

©2013 CloudBees, Inc. All Rights Reserved

41
Lots More Way To Deploy Slaves
• Good old shell scripting

$ java -jar slave.jar -jnlpUrl URL

©2013 CloudBees, Inc. All Rights Reserved

42
How Flexible Does It Get?
• Slave agent is a little Java program
– Runs locally on the machine that it controls
– Access files, fork processes, etc., on behalf of
master

• Communication with master
– Just needs a bi-di byte stream

• No shared file system, no network topology
constraints, etc

©2013 CloudBees, Inc. All Rights Reserved

43
Static Virtual Machines As Build Slaves
• Allocate lots of identical VMs
– Better consistency, lower overhead

• Curve up 1 big machine to small VMs
• Throw away > maintain

©2013 CloudBees, Inc. All Rights Reserved

44
Labels
Name: alpha
Label: windows 32bit

• Refer to slaves
but not by name

Name: bravo
Label: linux 32bit

Name: charlie
Label: linux 64bit

©2013 CloudBees, Inc. All Rights Reserved

45
Labels
Name: alpha
Label: windows 32bit

Name: bravo
Label: linux 32bit

This job needs to run
on “linux”

Name: charlie
Label: linux 64bit

©2013 CloudBees, Inc. All Rights Reserved

46
Labels
Name: alpha
Label: windows 32bit

Name: bravo
Label: linux 32bit

This job needs to run
on “linux && 32bit”

Name: charlie
Label: linux 64bit

©2013 CloudBees, Inc. All Rights Reserved

47
More Label Benefits
• Labels can be anything
– Geographic locations
– Availability of proprietary software installed

• Better resource utilization
– Jenkins have maximum freedom

• Resilient to node addition/removal
– Treat servers like livestocks, not like pets

©2013 CloudBees, Inc. All Rights Reserved

48
Load Statistics Monitoring

©2013 CloudBees, Inc. All Rights Reserved

49

49
“Cloud” plugin

©2013 CloudBees, Inc. All Rights Reserved

50
EC2 Plugin
• Label → Amazon Machine Image (AMI)
• Integrates with various AWS features
– Spot Instance
– Identity and Access Management (IAM)
– Virtual Private Cloud (VPC)

©2013 CloudBees, Inc. All Rights Reserved

51
jclouds plugin

Jenkins

jclouds

cloud

• CloudStack, OpenStack, and many more
• “One-time use” support

©2013 CloudBees, Inc. All Rights Reserved

52
Then Adjust Capacity based on it

©2013 CloudBees, Inc. All Rights Reserved

53
Jenkins Enterprise by CloudBees
• VMWare auto-scaling plugin
– Snapshot
– Power on-off management
– Hypervisor-aware scheduling
– Folder based pooling
– VMWare tools integration
– One-time use support

©2013 CloudBees, Inc. All Rights Reserved

54
CloudBees DEV@cloud
Mansion

Slave

Slave

Slave

Slave

©2013 CloudBees, Inc. All Rights Reserved

55
Linux Container = zero cost virtualization

Subversion

Gradle

Mercurial

Ant

Git

Maven

Linux Kernel
Hardware

©2013 CloudBees, Inc. All Rights Reserved

56
For OS X
Subversion

XCode

Git

XCode

Git

Maven

OS X

OS X

OS X

QEMU

QEMU

QEMU

Linux Kernel
Apple Hardware

©2013 CloudBees, Inc. All Rights Reserved

57
Kernel Same-page Merging

OS X

OS X

©2013 CloudBees, Inc. All Rights Reserved

OS X

58
©2013 CloudBees, Inc. All Rights Reserved

59
Mansion

Workspace 1

Slave

Workspace 2
Slave
Workspace 3
Slave
Workspace 4

©2013 CloudBees, Inc. All Rights Reserved

60
Workspace’

Slave

Workspace

Workspace’’

Slave

©2013 CloudBees, Inc. All Rights Reserved

61
~/.m2/repository

Slave

~/.m2/repository

~/.m2/repository

Slave

©2013 CloudBees, Inc. All Rights Reserved

62
この苦しみから解脱する方法はないかと
• お釈迦様の絵
• 徐々に蓄積してきたノウハウ・ツールを大公
開

©2013 CloudBees, Inc. All Rights Reserved

63
©2013 CloudBees, Inc. All Rights Reserved

64
Testing things in pieces

©2013 CloudBees, Inc. All Rights Reserved

65
Testing the whole is hard
•
•
•
•
•

Database
Load balancer
Messaging layer
Identity service
…

©2013 CloudBees, Inc. All Rights Reserved

66
©2013 CloudBees, Inc. All Rights Reserved
http://www.flickr.com/photos/46471983@N00/339605220/

67
What you really want = real deal!
Test and production environment should be
identical
… which means automating:
– Database
– Load balancer
– Messaging layer
– Identity service
–…
©2013 CloudBees, Inc. All Rights Reserved

68
The Challenge

Infrastructure as Code

©2013 CloudBees, Inc. All Rights Reserved

69
The Real Challenge

Elasticity

©2013 CloudBees, Inc. All Rights Reserved

70
The same evolution ladder

Elastic
Multiple

Single

©2013 CloudBees, Inc. All Rights Reserved

71
©2013 CloudBees, Inc. All Rights Reserved

72

More Related Content

What's hot

Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...
Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...
Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...Microsoft
 
Apache jclouds SF Meetup, July 8, 2013
Apache jclouds SF Meetup, July 8, 2013Apache jclouds SF Meetup, July 8, 2013
Apache jclouds SF Meetup, July 8, 2013Andrew Bayer
 
Package your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and KubernetesPackage your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and KubernetesArun Gupta
 
Kubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKel Cecil
 
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2daysUsing Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2daysCarlos Sanchez
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with DockerRavindu Fernando
 
Slaying Sacred Cows: Deconstructing Dependency Injection
Slaying Sacred Cows: Deconstructing Dependency InjectionSlaying Sacred Cows: Deconstructing Dependency Injection
Slaying Sacred Cows: Deconstructing Dependency InjectionTomer Gabel
 
7 Ways to Optimize Hudson in Production
7 Ways to Optimize Hudson in Production7 Ways to Optimize Hudson in Production
7 Ways to Optimize Hudson in ProductionCloudBees
 
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on AzureDevoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on AzurePatrick Chanezon
 
DCSF19 Containers for Beginners
DCSF19 Containers for BeginnersDCSF19 Containers for Beginners
DCSF19 Containers for BeginnersDocker, Inc.
 
Installing WordPress on AWS
Installing WordPress on AWSInstalling WordPress on AWS
Installing WordPress on AWSManish Jain
 
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...The Linux Foundation
 
Xen: Hypervisor for the Cloud - CCC13
Xen: Hypervisor for the Cloud - CCC13Xen: Hypervisor for the Cloud - CCC13
Xen: Hypervisor for the Cloud - CCC13The Linux Foundation
 
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure Patrick Chanezon
 
How to be successful running Docker in Production
How to be successful running Docker in ProductionHow to be successful running Docker in Production
How to be successful running Docker in ProductionDocker, Inc.
 
Docker for Java Developers
Docker for Java DevelopersDocker for Java Developers
Docker for Java DevelopersNGINX, Inc.
 
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
EclipseCon 2016 - OCCIware : one Cloud API to rule them allEclipseCon 2016 - OCCIware : one Cloud API to rule them all
EclipseCon 2016 - OCCIware : one Cloud API to rule them allMarc Dutoo
 
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Carlos Sanchez
 

What's hot (20)

Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...
Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...
Java sur Azure: OpenJDK mieux que vous n'avez jamais osé le demander ...
 
Apache jclouds SF Meetup, July 8, 2013
Apache jclouds SF Meetup, July 8, 2013Apache jclouds SF Meetup, July 8, 2013
Apache jclouds SF Meetup, July 8, 2013
 
Package your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and KubernetesPackage your Java EE Application using Docker and Kubernetes
Package your Java EE Application using Docker and Kubernetes
 
Kubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of ContainersKubernetes - Sailing a Sea of Containers
Kubernetes - Sailing a Sea of Containers
 
Docker Ecosystem on Azure
Docker Ecosystem on AzureDocker Ecosystem on Azure
Docker Ecosystem on Azure
 
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2daysUsing Kubernetes for Continuous Integration and Continuous Delivery. Java2days
Using Kubernetes for Continuous Integration and Continuous Delivery. Java2days
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with Docker
 
Slaying Sacred Cows: Deconstructing Dependency Injection
Slaying Sacred Cows: Deconstructing Dependency InjectionSlaying Sacred Cows: Deconstructing Dependency Injection
Slaying Sacred Cows: Deconstructing Dependency Injection
 
7 Ways to Optimize Hudson in Production
7 Ways to Optimize Hudson in Production7 Ways to Optimize Hudson in Production
7 Ways to Optimize Hudson in Production
 
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on AzureDevoxx France 2015 - The Docker Orchestration Ecosystem on Azure
Devoxx France 2015 - The Docker Orchestration Ecosystem on Azure
 
DCSF19 Containers for Beginners
DCSF19 Containers for BeginnersDCSF19 Containers for Beginners
DCSF19 Containers for Beginners
 
TIAD : Automating the modern datacenter
TIAD : Automating the modern datacenterTIAD : Automating the modern datacenter
TIAD : Automating the modern datacenter
 
Installing WordPress on AWS
Installing WordPress on AWSInstalling WordPress on AWS
Installing WordPress on AWS
 
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...
XPDDS19: A Journey to Mirage OS as Xen PVH - Marek Marczykowski-Górecki, Invi...
 
Xen: Hypervisor for the Cloud - CCC13
Xen: Hypervisor for the Cloud - CCC13Xen: Hypervisor for the Cloud - CCC13
Xen: Hypervisor for the Cloud - CCC13
 
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure
Docker New York Meetup May 2015 - The Docker Orchestration Ecosystem on Azure
 
How to be successful running Docker in Production
How to be successful running Docker in ProductionHow to be successful running Docker in Production
How to be successful running Docker in Production
 
Docker for Java Developers
Docker for Java DevelopersDocker for Java Developers
Docker for Java Developers
 
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
EclipseCon 2016 - OCCIware : one Cloud API to rule them allEclipseCon 2016 - OCCIware : one Cloud API to rule them all
EclipseCon 2016 - OCCIware : one Cloud API to rule them all
 
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...
 

Similar to Elastic build environment

JavaOne 2014: Next Step in Automation: Elastic Build Environment
JavaOne 2014: Next Step in Automation: Elastic Build EnvironmentJavaOne 2014: Next Step in Automation: Elastic Build Environment
JavaOne 2014: Next Step in Automation: Elastic Build EnvironmentKohsuke Kawaguchi
 
New Repository in AEM 6 by Michael Marth
New Repository in AEM 6 by Michael MarthNew Repository in AEM 6 by Michael Marth
New Repository in AEM 6 by Michael MarthAEM HUB
 
Continuous Delivery with Jenkins and Wildfly (2014)
Continuous Delivery with Jenkins and Wildfly (2014)Continuous Delivery with Jenkins and Wildfly (2014)
Continuous Delivery with Jenkins and Wildfly (2014)Tracy Kennedy
 
Gray Cover_ The dangers of cloudshells.pdf
Gray Cover_ The dangers of cloudshells.pdfGray Cover_ The dangers of cloudshells.pdf
Gray Cover_ The dangers of cloudshells.pdfColin Estep
 
Nebulaworks Docker Overview 09-22-2015
Nebulaworks Docker Overview 09-22-2015Nebulaworks Docker Overview 09-22-2015
Nebulaworks Docker Overview 09-22-2015Chris Ciborowski
 
Custom Runtimes for the Cloud
Custom Runtimes for the CloudCustom Runtimes for the Cloud
Custom Runtimes for the CloudCloudBees
 
Wars I’ve Seen From Java EE to Spring and more, Azure has you covered
Wars I’ve SeenFrom Java EE to Spring and more, Azure has you coveredWars I’ve SeenFrom Java EE to Spring and more, Azure has you covered
Wars I’ve Seen From Java EE to Spring and more, Azure has you coveredEdward Burns
 
The Kubernetes WebLogic revival (part 1)
The Kubernetes WebLogic revival (part 1)The Kubernetes WebLogic revival (part 1)
The Kubernetes WebLogic revival (part 1)Simon Haslam
 
Apache Spark Operations
Apache Spark OperationsApache Spark Operations
Apache Spark OperationsCloudera, Inc.
 
20191201 kubernetes managed weblogic revival - part 1
20191201 kubernetes managed weblogic revival - part 120191201 kubernetes managed weblogic revival - part 1
20191201 kubernetes managed weblogic revival - part 1makker_nl
 
Which cloud(s) & why? Defining Clouds and Best Practices
Which cloud(s) & why? Defining Clouds and Best PracticesWhich cloud(s) & why? Defining Clouds and Best Practices
Which cloud(s) & why? Defining Clouds and Best PracticesPaul Weiss
 
Platform Engineering for the Modern Oracle World
Platform Engineering for the Modern Oracle WorldPlatform Engineering for the Modern Oracle World
Platform Engineering for the Modern Oracle WorldSimon Haslam
 
9. virtualization virtualization
9. virtualization virtualization9. virtualization virtualization
9. virtualization virtualizationSagarR24
 
Docker and Jenkins [as code]
Docker and Jenkins [as code]Docker and Jenkins [as code]
Docker and Jenkins [as code]Mark Waite
 
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...Ed Burns
 
Kubernetes 101 VMworld 2019 workshop slides
Kubernetes 101 VMworld 2019 workshop slidesKubernetes 101 VMworld 2019 workshop slides
Kubernetes 101 VMworld 2019 workshop slidesSimone Morellato
 
Rethinking Application Design for the Cloud
Rethinking Application Design for the CloudRethinking Application Design for the Cloud
Rethinking Application Design for the CloudCloudBees
 
Enterprise Java on Azure: From Java EE to Spring, we have you covered
Enterprise Java on Azure: From Java EE to Spring, we have you coveredEnterprise Java on Azure: From Java EE to Spring, we have you covered
Enterprise Java on Azure: From Java EE to Spring, we have you coveredEd Burns
 
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...tdc-globalcode
 

Similar to Elastic build environment (20)

JavaOne 2014: Next Step in Automation: Elastic Build Environment
JavaOne 2014: Next Step in Automation: Elastic Build EnvironmentJavaOne 2014: Next Step in Automation: Elastic Build Environment
JavaOne 2014: Next Step in Automation: Elastic Build Environment
 
New Repository in AEM 6 by Michael Marth
New Repository in AEM 6 by Michael MarthNew Repository in AEM 6 by Michael Marth
New Repository in AEM 6 by Michael Marth
 
Continuous Delivery with Jenkins and Wildfly (2014)
Continuous Delivery with Jenkins and Wildfly (2014)Continuous Delivery with Jenkins and Wildfly (2014)
Continuous Delivery with Jenkins and Wildfly (2014)
 
Gray Cover_ The dangers of cloudshells.pdf
Gray Cover_ The dangers of cloudshells.pdfGray Cover_ The dangers of cloudshells.pdf
Gray Cover_ The dangers of cloudshells.pdf
 
Nebulaworks Docker Overview 09-22-2015
Nebulaworks Docker Overview 09-22-2015Nebulaworks Docker Overview 09-22-2015
Nebulaworks Docker Overview 09-22-2015
 
Custom Runtimes for the Cloud
Custom Runtimes for the CloudCustom Runtimes for the Cloud
Custom Runtimes for the Cloud
 
Wars I’ve Seen From Java EE to Spring and more, Azure has you covered
Wars I’ve SeenFrom Java EE to Spring and more, Azure has you coveredWars I’ve SeenFrom Java EE to Spring and more, Azure has you covered
Wars I’ve Seen From Java EE to Spring and more, Azure has you covered
 
Cicd.pdf
Cicd.pdfCicd.pdf
Cicd.pdf
 
The Kubernetes WebLogic revival (part 1)
The Kubernetes WebLogic revival (part 1)The Kubernetes WebLogic revival (part 1)
The Kubernetes WebLogic revival (part 1)
 
Apache Spark Operations
Apache Spark OperationsApache Spark Operations
Apache Spark Operations
 
20191201 kubernetes managed weblogic revival - part 1
20191201 kubernetes managed weblogic revival - part 120191201 kubernetes managed weblogic revival - part 1
20191201 kubernetes managed weblogic revival - part 1
 
Which cloud(s) & why? Defining Clouds and Best Practices
Which cloud(s) & why? Defining Clouds and Best PracticesWhich cloud(s) & why? Defining Clouds and Best Practices
Which cloud(s) & why? Defining Clouds and Best Practices
 
Platform Engineering for the Modern Oracle World
Platform Engineering for the Modern Oracle WorldPlatform Engineering for the Modern Oracle World
Platform Engineering for the Modern Oracle World
 
9. virtualization virtualization
9. virtualization virtualization9. virtualization virtualization
9. virtualization virtualization
 
Docker and Jenkins [as code]
Docker and Jenkins [as code]Docker and Jenkins [as code]
Docker and Jenkins [as code]
 
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...
Enterprise Java on Microsoft Azure: From Java EE to Spring, we’ve got you cov...
 
Kubernetes 101 VMworld 2019 workshop slides
Kubernetes 101 VMworld 2019 workshop slidesKubernetes 101 VMworld 2019 workshop slides
Kubernetes 101 VMworld 2019 workshop slides
 
Rethinking Application Design for the Cloud
Rethinking Application Design for the CloudRethinking Application Design for the Cloud
Rethinking Application Design for the Cloud
 
Enterprise Java on Azure: From Java EE to Spring, we have you covered
Enterprise Java on Azure: From Java EE to Spring, we have you coveredEnterprise Java on Azure: From Java EE to Spring, we have you covered
Enterprise Java on Azure: From Java EE to Spring, we have you covered
 
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...
TDC2017 | São Paulo - Trilha Cloud Computing How we figured out we had a SRE ...
 

Recently uploaded

Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piececharlottematthew16
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machinePadma Pradeep
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticscarlostorres15106
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024The Digital Insurer
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsMemoori
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr LapshynFwdays
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxhariprasad279825
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyAlfredo García Lavilla
 

Recently uploaded (20)

Story boards and shot lists for my a level piece
Story boards and shot lists for my a level pieceStory boards and shot lists for my a level piece
Story boards and shot lists for my a level piece
 
SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024SQL Database Design For Developers at php[tek] 2024
SQL Database Design For Developers at php[tek] 2024
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machineInstall Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmaticsKotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
 
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
Bun (KitWorks Team Study 노별마루 발표 2024.4.22)
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024My INSURER PTE LTD - Insurtech Innovation Award 2024
My INSURER PTE LTD - Insurtech Innovation Award 2024
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
 
AI as an Interface for Commercial Buildings
AI as an Interface for Commercial BuildingsAI as an Interface for Commercial Buildings
AI as an Interface for Commercial Buildings
 
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
"Federated learning: out of reach no matter how close",Oleksandr Lapshyn
 
Artificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptxArtificial intelligence in cctv survelliance.pptx
Artificial intelligence in cctv survelliance.pptx
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Commit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easyCommit 2024 - Secret Management made easy
Commit 2024 - Secret Management made easy
 

Elastic build environment

Editor's Notes

  1. JavaOSS
  2. More than 30% uses no slaves at all or just one
  3. 50+ slaves. There’s a divide here.
  4. Growing “Cloud divide”
  5. Because if you are doing it right, just building and testing will require a dozen or computers.
  6. As I get used to controlling a handful of computers, I started thinking what more we can do.If you don’t think more computers are helpful, you are doing it wrong / Can’t be said about people.
  7. Don’t build up capacity that’s enough on a few days a year but go idle most other time.
  8. One of the reasons I needed so many computers is because I needed all the different environments / some combinations were very rare and old, keeping them pristine was hard.
  9. Needing to have diversity in the environment adds to the capacity planning problem.
  10. But you don’t want to make everything too slow by over-subscribing. I’ve seen hypervisors used to run many virtual machines.
  11. Hey Kohsuke, my builds are failing. Can you take a look?
  12. Hey Kohsuke, my builds are failing. Can you take a look?
  13. So the lesson and the best practice = isolate builds and tests / treat them like untrusted code
  14. Various techniques has been deployed successfully today
  15. but as I found out the hard way, this isn’t enough to solve this problem
  16. Hey Kohsuke, my builds are failing. Can you take a look?
  17. Hey Kohsuke, my builds are failing. Can you take a look?
  18. Turns out isolation in the time dimension is just as important / somewhat like a human body --- if you live long enough, things tend to break down / beyond certain point it becomes unsalvageable, as Windows users know all too well!
  19. Turns out elasticity solves this problem, too, by allowing you to simply throw away and create new instances in the same predictable state /
  20. Episode from scalability summit / everyone explains their monitoring system
  21. Either this slide or more details Jenkins.
  22. Another common mode of deployment is…Even if it’s static…
  23. If you are willing to invest on creating a great slave virtualization environment, you can.
  24. HS: if somebody misses the CoW concept, he’d be lost for the next two slides
  25. Milestone in build environment elasticity / you’ve reached a new level of mental peace, enlightenment / all is well, let’s pack up and head home, right?
  26. But this story doesn’t end there. The case for elasticity applies equally well to tests and test environments. As that’s really the heart of continuous integration. The hard problem.
  27. A traditional attack vector to the testing is to test individual piece one at a time, then hope it still works when put together.We do this in Jenkins & CB a lot. Runs fast, anywhere, great!
  28. Especially in a connected world
  29. In Jenkins & CloudBees I do both all the times / Jetty / access token.And sometimes it’s a major accomplishment just to do it. Subversion server / OpenID service.But sometimes you just can’t do it. LDAP server / Active Directory
  30. single “runtime environment” definition and just multiple copies of it.
  31. Take load balancer as an example. Having chef configure haproxy is now a well understood problem.
  32. Provision and dispose through API / creating the box that chef/puppet runs in.Usefulness of such elasticity is not just about elasticity for running tests. For development, for review, too.
  33. Nowadays, that’s what I think of CloudBees as. Elasticity build environmenttaken to its logical conclusion demands elastic platform as a service.
  34. Need to split this into multiple pages.