Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Deployment of your Application @SpringOne

247 views

Published on

Spring Cloud Pipelines is an opinionated framework that automates the creation of structured continuous deployment pipelines.

In this presentation we’ll go through the contents of the Spring Cloud Pipelines project. We’ll start a new project for which we’ll have a deployment pipeline set up in no time. We’ll deploy to Cloud Foundry and check if our application is backwards compatible so that we can roll it back on production.

Published in: Software
  • Be the first to comment

Continuous Deployment of your Application @SpringOne

  1. 1. Senior Platform Architect at Pivotal Working on ● Pivotal Cloud Foundry ● Cloud Native Architecture ● Dev Practices Modernization Twitter: @ciberkleid About us
  2. 2. Spring Cloud developer at Pivotal Working on ● Spring Cloud Sleuth ● Spring Cloud Contract ● Spring Cloud Pipelines About us Twitter: @mgrzejszczak Blog: http://toomuchcoding.com
  3. 3. What problem are we trying to solve?
  4. 4. The necessity to create a deployment pipeline from scratch for each new project
  5. 5. Spring Cloud Pipelines • opinionated template of a deployment pipeline • based on good practices from different projects • we believe in the “Infrastructure as Code” approach • in case of automation server going down you can recover everything in no time • the automation server setup should be automated too! • you can even write tests for your pipelines
  6. 6. Spring Cloud Pipelines • we support following automation servers out of the box • Concourse • Jenkins using Jenkins Job DSL plugin • Jenkins using Jenkinsfile with Blue Ocean • logic of all steps done via Bash scripts • you can convert the internals to suit your needs • you can use whatever automation server you want • supports Maven & Gradle • Over 150 Bash tests (using https://github.com/sstephenson/bats)
  7. 7. Spring Cloud Pipelines
  8. 8. Spring Cloud Pipelines
  9. 9. Spring Cloud Pipelines
  10. 10. Spring Cloud Pipelines
  11. 11. Spring Cloud Pipelines
  12. 12. Spring Cloud Pipelines - PaaS Types
  13. 13. Spring Cloud Pipelines - CF example
  14. 14. Spring Cloud Pipelines - Concourse
  15. 15. Concourse
  16. 16. Spring Cloud Pipelines - Jenkins Job DSL
  17. 17. Jenkins Job DSL
  18. 18. Spring Cloud Pipelines - Jenkinsfile & Blue Ocean
  19. 19. Jenkinsfile
  20. 20. WHAT ARE WE GOING TO WORK WITH?
  21. 21. Demo - flow GITHUB-WEBHOOK GITHUB-ANALYTICS AMQPHOOK
  22. 22. Why do we deploy software? • We’re paid for delivering business value • Features are done when they are deployed to production • It’s better to deploy frequently for faster feedback • It’s better to fail the deployment pipeline as fast as possible • Deployments should take place in a standardised, automated fashion • Your deployment pipeline should test rollback • That way you can perform A/B or zero downtime deployment to production
  23. 23. Spring Cloud Pipelines • We’re paid for delivering business value • Features are done when they are deployed to production • It’s better to deploy frequently for faster feedback • It’s better to fail the deployment pipeline as fast as possible • Deployments should take place in a standardised, automated fashion • Your deployment pipeline should test rollback • That way you can perform A/B or zero downtime deployment to production
  24. 24. Spring Cloud Pipelines
  25. 25. Problem - slow feedback • Nobody wants to wait until the end of the pipeline to see that something is not working fine • We want to fail fast - even during build time • If integration is faulty • If our API is backward incompatible • There should be no need to wait until end to end tests are executed
  26. 26. Solution - fail fast • We’re running unit and integration tests during build time • To test faulty integration we use Spring Cloud Contract for HTTP / messaging • Producer defines a contract • From the contract o tests are generated to see if the producer is not lying o stubs are generated for consumers to use • Consumer can reuse it without running the producer • Fail at build time if integration fails (fail fast!) • All stubs reside in Nexus / Artifactory (thus can be reused)
  27. 27. CONTRACT TESTS DEMO (BREAKING CONSUMER)
  28. 28. Initial contract (producer) Body contains username and repository
  29. 29. Generated test (producer) Body contains username and repository
  30. 30. Expected model (consumer)
  31. 31. Passing contract test (consumer)
  32. 32. Breaking contract (producer) Was username and repository and we’ve made a breaking change by converting those to user and repo
  33. 33. Installing broken stubs locally (producer) New stubs with backward incompatible changes installed in Maven local
  34. 34. Broken build (consumer)
  35. 35. Broken contract test (consumer)
  36. 36. Broken contract test(consumer) Expected repository and username but got repo and user
  37. 37. CONTRACT TESTS DEMO (BREAKING PRODUCER)
  38. 38. Demo - backward incompatible API change GITHUB-ANALYTICS V1 with DELETE @ /issues Contracts V1 with DELETE @ /issues Generated test sends DELETE to /issues
  39. 39. Contract for deletion of issues
  40. 40. Generated test for deleting issues
  41. 41. Demo - backward incompatible API change GITHUB-ANALYTICS V1 with DELETE @ /issues Contracts V1 with DELETE @ /issues Generated test sends DELETE to /issues GITHUB-ANALYTICS V2 removed DELETE @ /issues Contracts V2 removed DELETE @ /issues No generated test for this endpoint
  42. 42. Backward incompatible changes of the API We delete the contract and the DELETE endpoint
  43. 43. Demo - backward incompatible API change GITHUB-ANALYTICS V1 with DELETE @ /issues Contracts V1 with DELETE @ /issues Generated test sends DELETE to /issues GITHUB-ANALYTICS V2 removed DELETE @ /issues Contracts V2 removed DELETE @ /issues No generated test for this endpoint GITHUB-ANALYTICS V2 removed DELETE @ /issues Contracts V1 with DELETE @ /issues Generated test sends DELETE to /issues
  44. 44. Broken build Current implementation (V2) does not support old contracts (V1)
  45. 45. Broken - generated test from old contracts (V1)
  46. 46. Broken - generated test from old contracts (V1)
  47. 47. Spring Cloud Pipelines • We’re paid for delivering business value • Features are done when they are deployed to production • It’s better to deploy frequently for faster feedback • It’s better to fail the deployment pipeline as fast as possible • Deployments should take place in a standardised, automated fashion • Your deployment pipeline should test rollback • That way you can perform A/B or zero downtime deployment to production
  48. 48. Spring Cloud Pipelines
  49. 49. STANDARDISATION GIVES LOWER SUPPORT COSTS AND MORE CHANCES OF PEOPLE ROTATION
  50. 50. Solution - PaaS & tooling • Use a PaaS to standardise the way you deploy and monitor your software • Spring Cloud Pipelines uses Cloud Foundry [1] or Kubernetes [2] as a PaaS implementation • For the demo purposes we’re using PCF Dev [3] or Minikube [4] • PaaS abstracts the application governance from underlying infrastructure • you can deploy, scale, manage applications in the same way if PaaS was running on your laptop, Amazon, Google, Azure etc. • Database schema upgrade is done via tools like Flyway [5] or Liquibase [6] [1] https://www.cloudfoundry.org, [2] https://kubernetes.io/ [3] https://pivotal.io/pcf-dev, [4] https://github.com/kubernetes/minikube [5] https://flywaydb.org/, [6] http://www.liquibase.org/
  51. 51. Spring Cloud Pipelines
  52. 52. Spring Cloud Pipelines After the application got deployed to test environment • The database schema gets updated upon application startup • We run a handful of smoke tests to see if crucial parts of our application are working fine • We want to test if the app is properly packaged • The application is surrounded by stubs - no real integrations take place • Spring Cloud Contract Stub Runner Boot app is responsible for serving stubs
  53. 53. Problem - rollback DB • Deployment pipelines should test whether the application can be rolled back • Rolling back database changes is extremely difficult • Especially if you deploy once every 6 months (the number of changes is gigantic) • How can you roll back a deletion of a column?
  54. 54. Spring Cloud Pipelines
  55. 55. Solution - application rollback • The easiest solution is… NOT TO DB ROLLBACK • Perform only backward compatible changes (always add data) • Or perform backward incompatible changes in a backward compatible way [1] • Roll back the application only (the JAR) • The application and DB changes need to be backward compatible • That way you can ensure that two applications (old / new versions) can run simultaneously at the same time [1] https://spring.io/blog/2016/05/31/zero-downtime-deployment-with-a-database
  56. 56. BACKWARD INCOMPATIBLE DB CHANGE DEMO
  57. 57. Demo - backward incompatible DB change GITHUB-ANALYTICS V1 CODE with repository DB V1 with repository
  58. 58. Demo - initial DB state (Flyway)
  59. 59. Demo - first run
  60. 60. Demo - initial state
  61. 61. Demo - backward incompatible DB change GITHUB-ANALYTICS V1 CODE with repository DB V1 with repository GITHUB-ANALYTICS V2 CODE with repo DB V2 with repo
  62. 62. Demo - backward incompatible DB change
  63. 63. Demo - backward incompatible DB change
  64. 64. Demo - backward incompatible DB change GITHUB-ANALYTICS V1 CODE with repository DB V1 with repository GITHUB-ANALYTICS V2 CODE with repo DB V2 with repo GITHUB-ANALYTICS V1 CODE with repository DB V2 with repo
  65. 65. Demo - backward incompatible DB change New application (V2) works against new database (V2)
  66. 66. Demo - backward incompatible DB change Old application (V1) can’t work with new database (V2) Deploy (V1) and connect to the new database (V2)
  67. 67. Demo - errors in the app logs Old version can’t insert data to a missing DB column
  68. 68. Spring Cloud Pipelines
  69. 69. Problem - end to end tests • End to end tests are slow and brittle • QA department writes an E2E for every feature we have • E2E environment setup • one environment shared between all applications? • one environment per application? • Surrounding apps should be deployed in • production versions? • development versions?
  70. 70. Solution - don’t do E2E? • Regardless of the time spent on QA / UAT you can still have bugs on production • Assuming that you ... • embrace failure • introduce monitoring of business key performance indicators (KPIs) • introduce alerting over the metrics • ensure that you can rollback on production • … you could stop doing any end to end tests
  71. 71. Spring Cloud Pipelines • Deploy to stage and running e2e tests are manual steps • you have to wait for your turn for the env • some manual work has to be done to purge stale data etc.
  72. 72. Spring Cloud Pipelines • We’re paid for delivering business value • Features are done when they are deployed to production • It’s better to deploy frequently for faster feedback • It’s better to fail the deployment pipeline as fast as possible • Deployments should take place in a standardised, automated fashion • Your deployment pipeline should test rollback • That way you can perform A/B or zero downtime deployment to production
  73. 73. Spring Cloud Pipelines
  74. 74. Problem - deployment to production • We don’t want to deploy the application to production at night • We want to treat a production deployment like any other deployment • We’d like to be able to perform A/B testing and zero downtime deployment • We’d like to easily rollback when sth goes wrong
  75. 75. Solution - PaaS + SC Pipelines • Our application has KPI monitoring in place • Alerting are set for KPIs • It has been tested that the application can be easily rolled back • PaaS (CF or K8S) can take care of zero downtime deployment
  76. 76. Spring Cloud Pipelines • prod-deploy deploys the pipeline version of the app to production next to the current production version • Once deployed we tag the repo with prod/VERSION_NUMBER • prod-complete allows to stop the old instance and leave only the new one running • prod-rollback deletes the new instance, removes the prod/VERSION_NUMBER tag and leaves only the old one running
  77. 77. A/B ON CF DEMO
  78. 78. Two versions running at the same time Two versions registered under same hostname Old instance New instance
  79. 79. MIGRATING TO SPRING CLOUD PIPELINES DEMO
  80. 80. Greeting UI Fortune Service Fortune DB Simon
  81. 81. The SCS Triplets and their pet Rabbit Greeting UI Fortune Service Fortune DB Simon
  82. 82. Greeting UI Simon
  83. 83. Greeting UI Simon Hey in !
  84. 84. Greeting UI The SCS Triplets and their pet Rabbit Simon Hey in !
  85. 85. Greeting UI The SCS Triplets and their pet Rabbit ??? Simon Hey in !
  86. 86. Greeting UI The SCS Triplets and their pet Rabbit !!! Fortune ServiceSimon Hey in !
  87. 87. Greeting UI The SCS Triplets and their pet Rabbit Fortune ServiceSimon Hey in !
  88. 88. Greeting UI The SCS Triplets and their pet Rabbit Fortune Service Fortune DB Simon Hey in !
  89. 89. Greeting UI The SCS Triplets and their pet Rabbit Fortune Service Fortune DB Simon Hey in !
  90. 90. Greeting UI The SCS Triplets and their pet Rabbit Fortune Service Fortune DB Simon Hey in !
  91. 91. Greeting UI “Do h k ” Simon
  92. 92. Greeting UI Simon Gre vi ! Who sa ?
  93. 93. Greeting UI Fortune ServiceSimon Gre vi ! Who sa ?
  94. 94. Greeting UI Fortune ServiceSimon Gre vi ! Who sa ?
  95. 95. Greeting UI Fortune Service Fortune DB Simon Gre vi ! Who sa ?
  96. 96. Greeting UI Fortune Service Fortune DB Simon Gre vi ! Who sa ?
  97. 97. Greeting UI Fortune Service Fortune DB Simon It’s o t h a q i n...
  98. 98. Greeting UI Fortune Service Fortune DB Simon It’s o t h a q i n... The SCS Triplets and their pet Rabbit
  99. 99. Greeting UI Fortune Service Fortune DB Simon I ju w t k o ... The SCS Triplets and their pet Rabbit
  100. 100. Greeting UI Fortune Service Fortune DB Simon PI NE T E H S!!! The SCS Triplets and their pet Rabbit
  101. 101. ● ○ ○ ● ○ ○ ○ ○
  102. 102. ● ● ● ● ● ○ ○ ○
  103. 103.
  104. 104. ● ● ○ mvn -N io.takari:maven:wrapper
  105. 105. ● ● ○ mvn -N io.takari:maven:wrapper ● ○ distributionManagement.repository.id ○ distributionManagement.repository.url
  106. 106. ● ● ○ mvn -N io.takari:maven:wrapper ● ○ distributionManagement.repository.id ○ distributionManagement.repository.url ●
  107. 107. ● ○ ○ ○ ○
  108. 108. ● ● ● ●
  109. 109. ● ● ○ ● test: services: - name: fortune-db type: broker broker: cleardb plan: spark - name: config-server type: broker broker: p-config-server plan: standard params: git: uri: https://github.com/ciberkleid/app-config useExisting: true ...
  110. 110. ● ○ ○ ○ ● ○
  111. 111. ● ● ○ ○
  112. 112. ● ● ●
  113. 113. ● ○ ○ ● ○ ○ ○ ○
  114. 114. Same for apicompatibility and e2e
  115. 115. ● ○ ○ ● ○ ○ ○
  116. 116. ● ○ ○ ○ ○ <profile> <id>default</id> <activation> <activeByDefault>true</activeByDefault> </activation> <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <configuration> <includes> <include>**/*Tests.java</include> <include>**/*Test.java</include> </includes> <excludes> <exclude>**/smoke/**</exclude> <exclude>**/e2e/**</exclude> </excludes> </configuration> </plugin> <plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> </plugin> </plugins> </build> </profile>
  117. 117. ● ●
  118. 118. fortune-service fortune-db
  119. 119. greeting-ui fortune-db fortune-service Assuming same domain...
  120. 120. greeting-ui circuit breaker fallback
  121. 121. greeting-ui fortune-service
  122. 122. ● ○ ○ ○ ○ ○
  123. 123. ● ● ●
  124. 124. ● Add INSERT statements to V1__init.sql ● Data is populated upon app startup, but only once per DB version
  125. 125. ● ○ ○ ● ○ ○ ○
  126. 126. ● ○ ○ ● ○ ○ ○ ○ ○
  127. 127. ● ○ ○ ● ○ ○
  128. 128. import org.springframework.cloud.contract.spec.Contract Contract.make { description(""" should return a fortune string """) request { method GET() url "/" } response { status 200 body "foo fortune" } }
  129. 129. ● ○ public class BaseClass { @Before public void setup() { FortuneService service = BDDMockito.mock(FortuneService.class); BDDMockito.given(service.getFortune()).willReturn("foo fortune"); RestAssuredMockMvc.standaloneSetup(new FortuneController(service)); } }
  130. 130. ● ○ ○
  131. 131. In this case, we want the contracts plugin to generate tests based on contracts outside of the project (the ones from the latest prod jar on our maven repo). Pipeline injects values in red dynamically
  132. 132. greeting-ui circuit breaker fallback or manually configured WireMock greeting-ui stubs without stubrunner with stubrunner ●
  133. 133. For in-process stubrunner, the server-side Wiremock port For in-process stubrunner, the client-side endpoint config
  134. 134. greeting-ui circuit breaker fallback greeting-ui stubrunner without stubrunner with stubrunner Note: Update the greeting smoke tests accordingly
  135. 135. ● ○ ○ ● ○ ● ○
  136. 136. ● <properties> <stubrunner.ids>io.pivotal:fortune-service:_latest:stubs:10000</stubrunner.ids> </properties> ● ●
  137. 137. ● ○ ○ ● ○ ○
  138. 138. ● ○ ○ ● ○ ○ ● ○ ○
  139. 139. Because now… ● You can evolve your app faster ● With (hopefully) better code ● Consistent testing practices ● Fast failure and feedback ● Contract-based API definitions ● API and DB rollback assurance ● And…. instant pipelines from source to production!
  140. 140. Greeting UI Fortune Service Fortune DB Simon W O D Do h k !?
  141. 141. Greeting UI Fortune Service Fortune DB Simon W O D Do h k !? Contract.make{“”” should return author name “””) ...
  142. 142. Greeting UI Fortune Service Fortune DB Simon W O D Do h k !? Fortune Service Stub
  143. 143. Greeting UI Fortune Service Fortune DB Simon W O D Do h k !? V2: ALTER TABLE fortune CHANGE COLUMN text TO fortune, ADD author CHAR (50);
  144. 144. Greeting UI Fortune Service Fortune DB Simon W O D Do h k !?
  145. 145. Greeting UI Fortune Service Fortune DB Simon Piv sa t t e m !
  146. 146. Greeting UI Fortune Service Fortune DB Oh, ri t! I k e t .
  147. 147. ● Github repos showing the end-state of each “act” ● Step-by-step instructions will be published soon and made accessible through the repo Readmes
  148. 148. Spring Cloud Pipelines
  149. 149. Customizing Spring Cloud Pipelines
  150. 150. Customizing Spring Cloud Pipelines $ curl -LOk https://github.com/spring-cloud/spring-cloud-pipelines/archive/v1.0.0.M8.zip $ unzip v1.0.0.M8.zip $ cd spring-cloud-pipelines-v1.0.0.M8 $ git init $ # modify the pipelines to suit your needs $ ./gradlew customize $ … $ git add . $ git commit -a -m “Initial commit” $ git remote add origin ${YOUR_REPOSITORY_URL} $ git push origin master
  151. 151. Customizing Spring Cloud Pipelines
  152. 152. Summary • Continuous Deployment allows you to continuously deliver business value • Spring Cloud Pipelines gives you OOB tooling to test your software via • unit and integration testing • contract testing • rollback testing • You can gradually migrate your applications to start using SC Pipelines • SC Pipelines allows you to easily adjust the deployment pipeline to suit your company’s needs • Thanks to PaaS you can easily do A/B & zero downtime deployment
  153. 153. ● Github Analytics: https://github.com/spring-cloud-samples/github-analytics ● Github Webhook: https://github.com/spring-cloud-samples/github-webhook ● Fortune Service: https://github.com/ciberkleid/fortune-service ● Greeting UI: https://github.com/ciberkleid/greeting-ui/ ● SC-Pipelines documentation: https://cloud.spring.io/spring-cloud-pipelines/ ● Pivotal Web Services trial : https://run.pivotal.io/ ● PCF Dev (CF on your laptop) : https://docs.pivotal.io/pcf-dev/ Links
  154. 154. Learn More. Stay Connected. ▪ Read the docs http://cloud.spring.io/spring-cloud-pipelines/ ▪ Talk to us on Gitter https://gitter.im/spring-cloud/spring-cloud-pipelines Twitter: twitter.com/springcentral YouTube: spring.io/video LinkedIn: spring.io/linkedin Google Plus: spring.io/gplus
  155. 155. ciberkleid mgrzejszczak

×