Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The road to continuous deployment: a case study (DPC16)

3,454 views

Published on

Dutch PHP Conference 2016

It's a situation many of us are familiar with: a large legacy application, limited or no tests, slow & manual release process, low velocity, no confidence.... Oh, and management wants new features, fast.

But how to proceed? Using examples and lessons learned from a real-world case, I'll show you how to strangle the legacy application with a modern service architecture and build a continuous deployment pipeline to deliver value from the first sprint. On the way, we take a look at testing strategies and various (possibly controversial!) tips and best practices.

Published in: Software

The road to continuous deployment: a case study (DPC16)

  1. 1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL ROOK
  2. 2. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY ABOUT ME ▸ Java, PHP & Scala contractor ▸ PHP since ’99 ▸ Maintainer of Phing ▸ Dutch Web Alliance ▸ http://www.linkedin.com/in/ michieltcs ▸ @michieltcs
  3. 3. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY THIS TALK ▸ Background ▸ The approach ▸ Process / standards ▸ Build pipelines ▸ Results & lessons learned
  4. 4. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY THE SYSTEM - SAN DIEGO ▸ ... or the Big Ball Of Mud ▸ Large legacy monolith ▸ Generates significant income ▸ Slow ▸ Complex, lots of moving parts ▸ Technical debt
  5. 5. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR
  6. 6. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY THE SYSTEM - SAN DIEGO ▸ 2.5% code coverage ▸ Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low confidence modifying existing code
  7. 7. REFACTOR? REBUILD?
  8. 8. A NEW STACK
  9. 9. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY THE STACK
  10. 10. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY APPROACH ▸ Strangler pattern ▸ Proxy to switch between old/new ▸ Migrate individual pages
  11. 11. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE SERVICE PROXY DB DB DB DB DB DB
  12. 12. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY APPROACH ▸ Services per domain object (job, jobseeker, ...) ▸ Services behind load balancers ▸ Access legacy db’s ▸ Continuous deployment ▸ Containers ▸ Frontends are services
  13. 13. SAN DIEGO ELASTIC SEARCHLEGACY DB JOB SERVICE RMQ ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVBINTERMEDIAI
  14. 14. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY PROXY # Serve job detail page from new code
 RewriteRule ^/vacature/.* /app.php [L,PT]
 
 # Proxy php files to php5-fpm
 ProxyPassMatch ^/(app.php(/.*)?)$ fcgi://127.0.0.1:9000/opt/webapp/web/$1
 
 # Proxy everything else to SanDiego
 ProxyPass / ${SAN_DIEGO__URL} nocanon
 ProxyPassReverse / ${SAN_DIEGO__URL}
 ProxyPreserveHost On
  15. 15. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY FEATURE TOGGLES, A/B TESTS
  16. 16. PROCESS
  17. 17. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY PROCESS ▸ Scrum, 1 week sprints ▸ TDD / BDD ▸ Definition of Done ▸ Team mindset / experience ▸ Focus on value ▸ Replace old features with new (legacy becomes obsolete)
  18. 18. CONTINUOUS EVERYTHING
  19. 19. DEV TEST ACCEPTANCE PRODUCTION DEV TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY CONTINUOUS DEPLOYMENT
  20. 20. ONLY COMMIT TO MASTER
  21. 21. PAIR PROGRAMMING
  22. 22. BOY SCOUT RULE
  23. 23. QUALITY GATES
  24. 24. 100% CODE COVERAGE
  25. 25. DASHBOARDS
  26. 26. BUILD PIPELINES
  27. 27. AUTOMATE REPEATABLE THINGS
  28. 28. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY AUTOMATION ▸ Builds ▸ Testing ▸ Deployments ▸ Orchestration ▸ Config management
  29. 29. EVERY COMMIT GOES TO PRODUCTION
  30. 30. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY DEFENSE IN DEPTH UNIT TESTS SCENARIOS INTEGRATION SMOKE / PERF. TESTS MANUAL TESTING?
  31. 31. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY UNIT TESTS public function testGetById(){
 $expectedJob = $this->getJob();
 
 $this->jobRepository->getById($expectedJob->getId())
 ->shouldBeCalled()
 ->willReturn($expectedJob);
 
 $this->assertEquals(
 $expectedJob,
 $this->jobService->getById($expectedJob->getId()
 );
 }
  32. 32. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY SCENARIOS Scenario: As an API user I need to be able to retrieve a job
 Given there is a valid job
 When I retrieve a valid job
 Then I should get a valid job resource
  33. 33. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY BUILDING <project name="JobService" default="build">
 <target name="build">
 <delete file="jobservice.tar" quiet="true"/>
 <tar destfile="jobservice.tar">
 <fileset dir="src">
 <include name="*.php"/>
 </fileset>
 </tar>
 </target>
 </project>
  34. 34. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY DOCKER FROM php:7.0-apache ADD vhost.conf /etc/apache2/sites-available/000-default.conf ADD jobservice.tar /opt/webapp RUN chown -R www-data:www-data /opt/webapp
  35. 35. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY DEPLOYING PULL IMAGE REMOVE FROM LOAD BALANCER STOP CONTAINER START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD TO LOAD BALANCER
  36. 36. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY DEPLOYING - name: remove instance from load balancer
 haproxy: state=disabled host={{ inventory_hostname }} backend=jobservice
 delegate_to: "{{ item }}"
 with_items: groups.haproxy
 
 - name: stop & remove old container
 docker:
 name: jobservice
 state: absent
 image: jobservice
 
 - name: start container
 docker:
 name: jobservice
 state: present
 image: jobservice:{{ BUILD_NUMBER }}
 ports:
 - "8080:8080"
  37. 37. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY DEPLOYING - name: perform health check
 uri:
 url: http://localhost:8080/_health
 status_code: 200
 timeout: 30
 return_content: true
 changed_when: False
 register: result
 until: result.content is defined and result.content.find("jobservice") != -1
 retries: 10
 delay: 3
 
 - name: add instance to load balancer
 haproxy: state=enabled host={{ inventory_hostname }} backend=jobservice
 delegate_to: "{{ item }}"
 with_items: groups.haproxy
  38. 38. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY BUILD PIPELINE node {
 stage 'Run tests'
 sh "phpunit"
 sh "behat"
 
 stage 'Build docker image'
 sh "phing build"
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 
 stage 'Deploy acceptance'
 sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i acc deploy.yml"
 
 stage 'Deploy production'
 sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i prod deploy.yml"
 }
  39. 39. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY BUILD PIPELINE
  40. 40. FEEDBACK!
  41. 41. RESULTS
  42. 42. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY RESULTS ▸ Total build time per service < 10 minutes ▸ Significantly improved page load times ▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc) ▸ Increased confidence and velocity ▸ Experimented with new tech/stacks (angular, jvm, eventsourcing) ▸ More fun
  43. 43. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸ Overhead of weekly sprint; requires discipline ▸ Docker orchestration ▸ Issues with traffic between Amazon <-> on-premise datacenter ▸ Javascript testing
  44. 44. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY LESSONS LEARNED ▸ Experience with new tech ▸ Stability of build pipelines ▸ Management/leadership buy-in ▸ Not enough focus on replacing legacy application
  45. 45. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY QUESTIONS & DISCUSSION Please leave feedback at https://joind.in/talk/4dab5 Thank you!

×