The road to continuous deployment (DomCode September 2016)

1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL ROOK

2. ▸ Java, PHP & Scala developer ▸ Consultant, trainer, speaker ▸ make.io ▸ Dutch Web Alliance ▸ Maintainer of Phing ▸ @michieltcs

3. THIS TALK ▸ Background ▸ The approach ▸ Process / standards ▸ Build pipelines ▸ Results & lessons learned

4. THE SYSTEM - SAN DIEGO ▸ ... or the Big Ball Of Mud ▸ Large legacy monolith ▸ Generates signiﬁcant income ▸ Slow & complex ▸ Technical debt

5. SAN DIEGO FRONTEND MYSQL DB SAN DIEGO BACKEND LOAD BALANCERS / VARNISH ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO FRONTEND SAN DIEGO BACKEND SAN DIEGO BACKEND SAN DIEGO BACKEND MEMCACHE FTP EXT. SERVICES SOLR

6. THE SYSTEM - SAN DIEGO ▸ Infrequent, manual releases ▸ Fragile tests ▸ Low velocity ▸ Frequent outages / bugs / issues ▸ Frustrated team ▸ Low conﬁdence modifying existing code

7. GOALS ▸ Reduce issues ▸ Reduce cycle time ▸ Increase productivity ▸ Increase motivation

8. REFACTOR? REBUILD?

9. APPROACH ▸ Services per domain object (job, jobseeker, ...) ▸ Strangler pattern ▸ Proxy to switch between old/new ▸ Migrate individual pages

10. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE SERVICE PROXY DB DB DB DB DB DB

11. APPROACH ▸ Services behind load balancers ▸ Access legacy db’s ▸ Continuous deployment ▸ Docker containers ▸ Frontends are services

12. SAN DIEGO ELASTIC SEARCHLEGACY DB JOB SERVICE RMQ ITBANEN INTERMEDIAIR NATIONALEVACATUREBANK MONGO DB ITBANEN JOBSEEKER SERVICE NVBINTERMEDIAI

13. PROCESS

14. STARTING OFF ▸ Scrum, 1 week sprints ▸ TDD / BDD ▸ Deﬁnition of Done ▸ Team mindset / experience ▸ Focus on value ▸ Replace old features with new (legacy becomes obsolete)

15. CONTINUOUS EVERYTHING

16. DEV BUILD / TEST CONTINUOUS INTEGRATION

17. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY

18. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT

19. WHY CONTINUOUS DEPLOYMENT ▸ Small steps ▸ Early feedback ▸ Less overhead ▸ Reduce cycle time ▸ Reduce risk

20. ONLY COMMIT TO MASTER

21. NO BRANCHES

22. NO BRANCHES REALLY.

23. PAIR PROGRAMMING

24. BOY SCOUT RULE

25. QUALITY GATES

26. 100% CODE COVERAGE*

27. FEATURE TOGGLES, A/B TESTS

28. DASHBOARDS

29. BUILD  PIPELINE

30. AUTOMATE REPEATABLE THINGS

31. EVERY COMMIT GOES TO PRODUCTION

32. DEFENSE IN DEPTH UNIT TESTS INTEGRATION  TESTS ACCEPTANCE  TESTS UI TESTS

33. DEFENSE IN DEPTH UNIT TESTS INTEGRATION  TESTS ACCEPTANCE UI TESTS public function testJobCannotBeFound() {  $jobRepository = $this->prophesize(JobRepository::class);  $jobRepository->getById(EXPECTED_JOB_ID)  ->shouldBeCalled()  ->willReturn(false);    $jobService = new JobService($jobRepository->reveal());    $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));  }

34. DEFENSE IN DEPTH UNIT TESTS INTEGRATION  TESTS ACCEPTANCE  TESTS UI TESTS public function testFindJob() {  $expectedJob = $this->loadFixture('active_job.yml');  $actualJob = $this->repository->getById($expectedJob->getId());    self::assertInstanceOf(Job::class, $actualJob);  self::assertEquals($expectedJob->getId(), $actualJob->getId());  }

35. DEFENSE IN DEPTH UNIT TESTS INTEGRATION  TESTS ACCEPTANCE  TESTS UI TESTSScenario: Link to related job  Given a job exists  And there are related jobs available  When that job is viewed  Then a list of related jobs is shown  And each related job links to the detail page of the related job

36. DEFENSE IN DEPTH UNIT TESTS INTEGRATION  TESTS ACCEPTANCE  TESTS UI TESTS

37. DEFENSE IN DEPTH MANUAL TESTING? UNIT TESTS INTEGRATION  TESTS ACCEPTANCE  TESTS UI

38. CONTINUOUS TESTING UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS SMOKE  TESTS Cost Speed Exploratory  testing Monitoring

39. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER

40. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull

41. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run

42. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15

43. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:  url: http://localhost:8080/_health  status_code: 200  timeout: 30

44. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2  dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded

45. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2  dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded

46. DEPLOYING PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm

47. PIPELINE AS CODE node {  stage 'Run tests'  sh "phpunit"  sh "behat"    stage 'Build docker image'  sh "phing build"  sh "docker build -t jobservice:${env.BUILD_NUMBER} ."  sh "docker push jobservice:${env.BUILD_NUMBER}"    stage 'Deploy acceptance'  sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i acc deploy.yml"    stage 'Deploy production'  sh "ansible-playbook -e BUILD_NUMBER=${env.BUILD_NUMBER} -i prod deploy.yml"  }

48. BUILD PIPELINE

49. FEEDBACK!

50. RESULTS

51. RESULTS ▸ Total build time per service < 10 minutes ▸ Significantly improved page load times ▸ Improved audience stats (time on page, pages per session, session duration, traffic, seo ranking, etc) ▸ Increased confidence and velocity ▸ Experimented with new tech/stacks (angular, jvm, event sourcing) ▸ More fun

52. LESSONS LEARNED ▸ Team acceptance ▸ Change is hard ▸ Overhead of weekly sprint; requires discipline ▸ Docker orchestration ▸ Issues with trafﬁc between Amazon <-> on-premise datacenter ▸ Javascript testing

53. LESSONS LEARNED ▸ Experience with new tech ▸ Stability of build pipelines ▸ Management/leadership buy-in ▸ Business alignment ▸ Not enough focus on replacing legacy application

54. THANK YOU! @michieltcs / michiel@make.io

The road to continuous deployment (DomCode September 2016)

You are reading a preview.

Check these out next

The road to continuous deployment (DomCode September 2016)

More Related Content

Slideshows for you (20)

Similar to The road to continuous deployment (DomCode September 2016) (20)

More from Michiel Rook (6)

Recently uploaded (20)

The road to continuous deployment (DomCode September 2016)