Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Escaping 5 decades of monolithic annual releases

284 views

Published on

Lessons learned from speeding up large, complex annual releases of desktop applications and databases at Cambridge Crystallographic Data Centre

Published in: Software
  • Be the first to comment

Escaping 5 decades of monolithic annual releases

  1. 1. 1 @ClareMacraeUK Escaping 5 Decades of Monolithic Annual Releases Clare Macrae Cambridge Crystallographic Data Centre 20 March 2018
  2. 2. 2 @ClareMacraeUK@ClareMacraeUK https://www.slideshare.net/ClareMacrae/
  3. 3. 3 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  4. 4. 4 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  5. 5. 5 @ClareMacraeUK@ClareMacraeUK
  6. 6. 6 @ClareMacraeUK@ClareMacraeUK Crystal Structures Flexible Plastic, Shapeways, 8cm £19 BOSQAB: doi:10.5517/cc55fnd Sandstone, Sculpteo, 13cm x 9cm x 9cm £71 LALNIN49: doi:10.5517/cctlldc Sandstone, Sculpteo, 6cm £65 SAHYIK01: doi:10.5517/cc8md6g Prices include delivery and tax (2015 prices)
  7. 7. 7 @ClareMacraeUK@ClareMacraeUK Unified Releases • 8 desktop applications • Database(s) • 8GB -> 18 GB • Annual licensing
  8. 8. 8 @ClareMacraeUK@ClareMacraeUK 1987…
  9. 9. 9 @ClareMacraeUK@ClareMacraeUK Why @PIPELINE?
  10. 10. 10 @ClareMacraeUK@ClareMacraeUK Why @PIPELINE? To share what we’ve learned on the journey
  11. 11. 11 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  12. 12. 12 @ClareMacraeUK@ClareMacraeUK Technologies Fortran 77 & Database Fortran 90 Image: lewing@isc.tamu.edu Larry Ewing and The GIMP
  13. 13. 13 @ClareMacraeUK@ClareMacraeUK Technologies Fortran 77 & Database Fortran 90 Image: lewing@isc.tamu.edu Larry Ewing and The GIMP
  14. 14. 14 @ClareMacraeUK@ClareMacraeUK Libraries & Applications
  15. 15. 15 @ClareMacraeUK@ClareMacraeUK Constraints • Few million lines of C++ … • 3 scrum teams • Large, legacy database format • Decades of culture built on annual cycle • “November” release
  16. 16. 16 @ClareMacraeUK@ClareMacraeUK Legacy?
  17. 17. 17 @ClareMacraeUK@ClareMacraeUK Ahead of our time? • 1988: 3 releases per year • Version control history starts in 1992 • “Kind of” CI builds in mid 1990s • But…
  18. 18. 18 @ClareMacraeUK@ClareMacraeUK Release Cycle Develop (trunk) • New code • Unit tests Release (branch) • More features • Test & Document • Bug Fixes
  19. 19. 19 @ClareMacraeUK@ClareMacraeUK Months per release 4 20031988 6 12 12 1989 2016
  20. 20. 20 @ClareMacraeUK@ClareMacraeUK Speeding Up? • Tried many things to improve annual releases… • … Over many years …
  21. 21. 21 @ClareMacraeUK@ClareMacraeUK Code & Infrastructure • Writing unit tests  • Scripting (many) manual processes  • TeamCity Continuous Integration builds  • Overhaul ancient build and distribution scripts 
  22. 22. 22 @ClareMacraeUK@ClareMacraeUK Culture • Trying to test features during dev  • Trying to document features during dev 
  23. 23. 23 @ClareMacraeUK@ClareMacraeUK Process • Documented manual “smoke” tests  • All-staff testing day, once a year  • Create release branch earlier … and earlier 
  24. 24. 24 @ClareMacraeUK@ClareMacraeUK Manual Testing • Smoke Tests = [programs] x [OSs] x [OS versions] • Graphics Tests = [OSs] x [graphics cards] • Scientific Tests = [Tutorials + Workshops] x [OSs] 96 8 3 4 9 3 3 8 341147
  25. 25. 25 @ClareMacraeUK@ClareMacraeUK 2017 Annual Release • New testing hardware  • Efficient tester machine setup  • Started automating GUI smoke tests 
  26. 26. 26 @ClareMacraeUK
  27. 27. 27 @ClareMacraeUK@ClareMacraeUK
  28. 28. 28 @ClareMacraeUK@ClareMacraeUK Are we nearly there yet? • 2016 release period noticeably more relaxed  • Maybe we cracked it?
  29. 29. 29 @ClareMacraeUK@ClareMacraeUK
  30. 30. 30 @ClareMacraeUK@ClareMacraeUK Months per release 4 20031988 2017 6 12 12 1989 2016 12
  31. 31. 31 @ClareMacraeUK@ClareMacraeUK 2017 Annual Release • Perfect storm of Internal & External changes • Showstopper MacOS bug • Result: critical bugs and long release • Significant breaking changes during the year • Half our C++ developers needed for 4 months! • Major, visible stress
  32. 32. 32 @ClareMacraeUK@ClareMacraeUK 2017 Annual Release • Teams weren't able to plan for upcoming work • Agreement: "for staff well-being, can’t continue" • So painful, it was clear big change needed • But what?
  33. 33. 33 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  34. 34. 34 @ClareMacraeUK@ClareMacraeUK “When something painful happens, use that opportunity to drive a long-term improvement”
  35. 35. 35 @ClareMacraeUK@ClareMacraeUK Plans for 2018 Do 3 Interim Releases
  36. 36. 36 @ClareMacraeUK
  37. 37. 37 @ClareMacraeUK@ClareMacraeUK Demo Test Automation • Personal GUI-testing demo for key stakeholders • Very valuable
  38. 38. 38 @ClareMacraeUK@ClareMacraeUK Internal publicity • We’re going to “develop and release on default” • Get release dates in the calendar well in advance
  39. 39. 39 @ClareMacraeUK@ClareMacraeUK Know the objections • Have advocated more frequent releases for years • So I knew all our “it’s impossible” reasons • Invaluable for planning change • Invaluable for reassuring those most worried
  40. 40. 40 @ClareMacraeUK@ClareMacraeUK Our main objections • Overheads: 4 x 4 months doesn’t fit! • Some of our users can only install once per year
  41. 41. 41 @ClareMacraeUK@ClareMacraeUK Release Frequency
  42. 42. 42 @ClareMacraeUK@ClareMacraeUK Meet all the teams • State the Goal: “Make Releases Boring” • Explain, explain, explain • Reassurance - it will be OK, it’s not new! • Feature toggles - only release what's ready
  43. 43. 43 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  44. 44. 44 @ClareMacraeUK@ClareMacraeUK Redefine Releases • From When we fix bugs, finish features & test • To When we ship it • Only has to be better than the last release • Knowing that there's another release soon gives confidence to focus on the few critical things
  45. 45. 45 @ClareMacraeUK@ClareMacraeUK Change our Planning • No-one to test feature? Then defer the feature! • Critical that this first update release succeeded • Each dev team reviewed changes made in code that their products depends • So manual testing efforts focused by risk • First “extra” release after 2 months, not 3
  46. 46. 46 @ClareMacraeUK@ClareMacraeUK Priorities Complete feature Bug fixes Process improvements
  47. 47. 47 @ClareMacraeUK@ClareMacraeUK “Update 1” Release • Elapsed time: 1 month (2018-01-31 … 2018-02-28) • 90% fewer changes during release • Motivational benefits for developers • Excellent ideas from retrospective • “Not many new features”
  48. 48. 48 @ClareMacraeUK@ClareMacraeUK Months per release 4 20031988 2017 3 2018 6 12 12 1989 2016 12
  49. 49. 49 @ClareMacraeUK@ClareMacraeUK Success! • https://www.ccdc.cam.ac.uk/Community/blog/2018-02-frequent-releases/
  50. 50. 50 @ClareMacraeUK@ClareMacraeUK Why did it work? • All the development, testing and automation changes were necessary to improve our releases • But not sufficient • I truly believe our critical missing link was the belief & repeated expectation that we would succeed
  51. 51. 51 @ClareMacraeUK@ClareMacraeUK
  52. 52. 52 @ClareMacraeUK@ClareMacraeUK Today • Context and motivation • Ever-growing monolithic releases • Using the pain • Starting to escape • Summary
  53. 53. 53 @ClareMacraeUK@ClareMacraeUK Lessons learned • If you think you can achieve something, you will • Put as little as possible into first changed release • Use current tools • Discover what most needs to be improved
  54. 54. 54 @ClareMacraeUK@ClareMacraeUK Lessons learned • How to turn around massive monolithic processes? • Prioritise automated testing – however crude • Showcase the benefits of innovations – working things – to win support – don’t ask! • Just do it! • Or… Just get started!
  55. 55. 55 @ClareMacraeUK@ClareMacraeUK References • My Monolith is Melting – Meredith Williams, PIPELINE 2015 – https://vimeo.com/123620783 • On Continuous Delivery for Client Software: – http://timothyfitz.com/2009/03/09/cd-for-client-software/ • Feature Toggles: – https://martinfowler.com/articles/feature-toggles.html
  56. 56. 56 @ClareMacraeUK@ClareMacraeUK References • Simple 3D Printing from a Crystal Structure – https://www.ccdc.cam.ac.uk/Community/blog/2017-02-09- simple-3d-printing-from-a-crystal-structure/
  57. 57. 57 @ClareMacraeUK@ClareMacraeUK Contact Info • @ClareMacraeUK • macrae@ccdc.cam.ac.uk • https://www.ccdc.cam.ac.uk/ • https://www.slideshare.net/ClareMacrae/
  58. 58. 58 @ClareMacraeUK Questions?
  59. 59. 60 @ClareMacraeUK Extras
  60. 60. 61 @ClareMacraeUK@ClareMacraeUK What next for us? • Eliminate time-consuming steps • More work to eliminate manual testing • Automate the data export process regularly • Make it quicker to install new builds • Suggestions welcome!
  61. 61. 62 @ClareMacraeUK@ClareMacraeUK That Mac challenge now? • Would not matter when the problem was discovered • Just get a small team together to work on the problem • Could release an update to users at any time • Would still need to fix it – just without the pressure

×