Concourse, Spinnaker, Cloud Foundry, Oh My! Creating Sophisticated Deployment Workflows with Cameron Stewart at SpringOne Tour 2019

1. © Copyright 2019 Pivotal Software, Inc. All rights Reserved.
Cameron Stewart, Partner Platform Architect
SpringOne Tour Burlington
July 26th
Concourse, Spinnaker, CF, Oh My!
Automate all the things.

2. Who knows what Toil is?

3. Who likes Toil?

4. Cover w/ Image
Toil
Toil Defined
“So what is toil? Toil is the kind of work
tied to running a production service that
tends to be manual, repetitive,
automatable, tactical, devoid of
enduring value, and that scales linearly
as a service grows.”
Chapter 5 - Eliminating Toil

5. “... delivering useful software shapes our world…
...[software] is only useful when we deliver it.”
- you’ll see

6. Getting code into Production is the
most important thing we can do

7. Everything as Code
➔ You need automation; and you need to be able to manage your infrastructure programmatically -
at every layer
➔ As delivery velocity increases, we need to adopt the practices to deploy applications to other
areas
➔ Infrastructure as Code - writing code that provisions and manages our infrastructure in a
predictable way
➔ Delivery pipelines as first class citizens

8. “We recognize that delivering useful software shapes our world. We recognize that code is the best way to specify precise
action. We recognize that code is only useful when we deliver it.
Delivery is not a detail, it is our job. Now is the time to apply our core skills to our own work. Now is the time to engineer
our delivery. We divide our work between ourselves and computers: humans for decisions, and automation for tasks.”

9. Continuous Delivery
“Continuous Delivery is about shipping small
batches of software to production constantly,
through automation. Continuous delivery makes the
act of releasing dull and reliable, so organizations
can deliver frequently, at less risk, and get feedback
faster from end users.”

10. Continuous Delivery Pipeline

11. Getting code into is the most
important thing we can do

12. Continuous Delivery Pipeline
Automated builds
Unit tests
Compliance checks
Service tickets
Performance tests
Security validation
Blue/Green deploys
Canary analysis
A/B testing
Monitoring
Security scans
Chaos engineering
Test-driven dev
Iterative coding/fixing
Frequent integration
Commit code change
Store binaries &
build artifacts

13. Continuous Delivery Pipeline
Automated builds
Unit tests
Blue/Green deploys
Canary analysis
A/B testing
Monitoring
Security scans
Chaos engineering
Test-driven dev
Iterative coding/fixing
Frequent integration
Commit code change
Store binaries &
build artifacts
Continuous Integration
Continuous
Delivery
Compliance checks
Service tickets
Performance tests
Security validation

14. Continuous Delivery Pipeline
Automated builds
Unit tests
Blue/Green deploys
Canary analysis
A/B testing
Monitoring
Security scans
Chaos engineering
Test-driven dev
Iterative coding/fixing
Frequent integration
Commit code change
Store binaries &
build artifacts
Continuous Integration
Continuous
Delivery
Compliance checks
Service tickets
Performance tests
Security validation

15. Hello,
Concourse

16. The Concourse Pipeline
Flexible integration
of resources
Simple modeling of
components
Pipeline status is
immediately visible
Build components are
expressed as code

17. Concourse Concepts: Simple Primitives
Resources
Detecting, fetching, creation of
externally versioned “things”
Jobs
Compose resources and tasks
together to do something (run tests,
ship, etc).
Tasks
Run a script in a container with its
dependent inputs
# pipeline.yml
resources:
- name: source-code
type: git
source:
uri: https://github.com/...
branch: master
- name: source-code
type: git
source:
uri: https://github.com/...
branch: master
# pipeline.yml
jobs:
- name: unit
plan:
- get: source-code
trigger: true
- task: unit-tests
file: source-code/ci/unit.yml
# unit.yml
platform: linux
image_resource:
type: docker-image
source:
repository: java
tag: '8'
inputs:
- name: source-code
run:
path: source-code/mvnw
args: [ clean, test ]

18. Concourse Pipeline Visualization

19. Platform Automation
Automate the installation and updates
of PCF to stay stable and secure always
● Concourse is an essential part of
your upgrade path, enabling you
to automate small, defined actions
for interacting with OpsManager.
● The resulting perpetual upgrade
machine helps you to maintain
secure, stable PCF instantiations
all the time.
● It also ensures developers always
have access to the latest features
and capabilities.
Application CI/CD
Automate the CI/CD pipeline to deliver
apps/services to PCF fast and safely
● Concourse presents a general
approach to automation that
makes it ideal for CI/CD
workflows.
● Think Concourse for automating
continuous integration (CI) and
production deployment readiness
(e.g., automate change tickets,
compliance, security).
● Integrate with CD solutions like
Spinnaker for continuous
delivery/deployment
ApplicationPlatformConcourse
for PCF Use
Cases
Automate Modern Cloud Delivery

20. Embedded OS
(Windows & Linux)
NSX-T
CPI (15 methods)
v1
v2
v3
...
CVEs
Product Updates
Java | .NET | NodeJS
Pivotal Application
Service (PAS)
Application Code & Frameworks
Buildpacks | Spring Boot | Spring Cloud |
Steeltoe
Elastic | Packaged Software | Spark
Pivotal Container
Service (PKS)
YOU build the containerWE build the container
vSphere
Azure &
Azure StackGoogle CloudAWSOpenstack
Pivotal
Network
“3Rs”
Github
Concourse
Concourse
Pivotal Services
Marketplace
Pivotal and
Partner Products
Continuous
delivery
Public Cloud
Services
Customer
Managed
Services
OpenServiceBrokerAPI
Repair
— CVEs
Repave Rotate
— Credhub

21. What’s good
for your apps,
is good for
your platform.
“
”

22. Continuously Delivering Your Platform

23. Cloud Pipelines

24. Cover w/ Image
(Spring) Cloud Pipelines
➔ Creation of a common deployment pipelines in
Concourse or Jenkins
➔ Propagation of good testing and deployment
practices
➔ Reducing the time required to create pipelines
➔ Project crawler
➔ Implements tests for API contracts using Spring
Cloud Contract
➔ Integrates with flyway Database schema
migrations

25. Continuous Delivery Pipeline
Automated builds
Unit tests
Blue/Green deploys
Canary analysis
A/B testing
Monitoring
Security scans
Chaos engineering
Test-driven dev
Iterative coding/fixing
Frequent integration
Commit code change
Store binaries &
build artifacts
Continuous Integration
Continuous
Delivery
Compliance checks
Service tickets
Performance tests
Security validation

26. G’day,
Spinnaker

27. Spinnaker Is an OSS Multi-Cloud Delivery Platform
Spinnaker Community
“...the passionate open source
community dedicated to making
deployment pain go away.”
https://www.spinnaker.io/publications/ebook/

28. Sophisticated Deployment Strategies
https://www.spinnaker.io/concepts/

29. Application Centric Control Plane
=

30. Multi-foundation View

31. Spinnaker microservices
https://www.spinnaker.io/reference/architecture/

32. Spinnaker
Ecosystem
Cloud Providers
App Engine
Amazon Web Services
Azure
Cloud Foundry
DC/OS
Google Compute Engine
Kubernetes
Openstack
Oracle
CI Systems
Jenkins
Travis CI
Wercker
Concourse
Artifact Support
Docker
Google Cloud Storage
GitHub
HTTP
S3
Artifactory
Monitoring
Datadog
Prometheus
Stackdriver
Atlas
SignalFx
(10+ more on
roadmap)
Notifications
Email
HipChat
Slack
SMS via Twilio

33. Hello.
G’day.

34. Stateless: Rather than sharing state, every task
runs in its own container, controlling its own
dependencies.
Ephemeral: Clean and consistent builds.
Minimizes test infrastructure to maintain and gives
you flexibility to scale with the cloud.
Continuous integration: Track versions of
external artifacts used for CI (e.g., Git, S3, docker
image). Build components expressed as code with
simple modeling of modular components.
Flexible workflow automation: Extensible for
compliance or other customized tasks in a
pipeline.
Stateful: Maintains awareness of the topography
(multiple clouds, PCF Foundations, etc.) and
keeps record of all pipeline activities for
compliance and audit.
Application inventory: Maintains inventory of
deployed applications, supporting activities like
rollback, canary analysis and identifying security
vulnerabilities.
Sophisticated deployment scenarios: Canary
deploys, red/black or blue/green deploys,
progressive deployments (e.g., by time zone),
multi-cloud/PCF Foundation deployments.
How Does Spinnaker Relate to Concourse?

35. Interactive Continuous Delivery Pipeline
CI
Production
Arbitrary Jobs

36. Spinnaker +
Concourse
Together
Bidirectional Integration
● Using Concourse as a trigger
○ Trigger a Spinnaker pipeline stage
○ Pass Concourse trigger info on to
subsequent stages
● Including Concourse as part of a
Spinnaker pipeline
○ Create a stage that watches
Concourse jobs
○ Concourse tells Spinnaker what jobs
to watch
○ When Concourse jobs complete,
Spinnaker stage completes

37. Concourse Trigger

38. Trigger Configuration

39. Using Concourse trigger info in subsequent stages

40. Using Concourse trigger info in subsequent stages
Ex: use the version to identify
the application binary in a
Deploy stage

41. 1. Concourse job runs
Any Concourse out resource
metadata will be available to a
running Spinnaker pipeline.

42. 2. Spinnaker pipeline is triggered

43. Concourse Stage

44. Stage Configuration
The Concourse resource uses
these to identify whether a
running stage is a match against
the job it is attached to.

45. 1. Stage starts

46. 2. Concourse check resource identifies the stage.

47. 3. Concourse tells Spinnaker which job to watch.

48. 4. Concourse job completes.

49. 5. Spinnaker sees the Concourse job is finished,
and completes the stage.

50. Virtuous cycle of
automated and iterative
progression, feedback,
and improvement of
software
Continuous Delivery to the Cloud
Monitoring
Securing
Resiliency
Develop
Testing
Experimenting
Compliance
Pivotal Cloud Foundry
Deploy Operate
Planning
Test-Driven Dev
CI
Development Users

51. Who likes Toil?

52. Who wants to eliminate Toil?

53. Transforming how the world builds software
© Copyright 2019 Pivotal Software, Inc. All rights Reserved.