Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous delivery in complex systems | Mikael Reinholdsson | LTG-25


Published on

Presentation held at Lean Tribe Gathering 25 in Gothenburg feb 19 2015.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Continuous delivery in complex systems | Mikael Reinholdsson | LTG-25

  1. 1. Continuous delivery in complex systems Mikael Reinholdsson Softhouse 15 years developing mobile phone platforms (HW and SW) Co-founder Softhouse
  2. 2. System overview Main CPUs running LINUX and Android Multimedia DSP’s for Video/Audio and Imaging GPU with Vertex processors CPUs running OSE and 2G/3G/4G stack Vector processor for PHY + DSP for Radio Power management CPU (Always on) - 1500 developers in all time zones producing 100-200 patches per day - ~10GByte source code in more than 100.000 files - SW licenses: Apache-2, GPL and proprietary (no contamination) DSP’s for WLAN, GPS and Bluetooth
  3. 3. Requirements and learnings • Lead customer main requirement – Majority of issues solved within 24 hours with “right” quality vs. purpose 1. Development 2. Launching • Other basic “laws of nature” in SW development – Each line of code changed have risk to cause failure • Close to launch => Branching needed to not break things (Impossible to capture all failures within 1 day) – Lead time in integration create many issues • Code grow old very fast (= stops working) • Developers starts sharing code in other ways
  4. 4. Integration solution Feature branch (development) Gerrit upload Checks in 1 hour Checks: License, Static code, forbidden words, all build variants Tests: 30 min regressionResult in 1 hour Daily tag set Tests: Regression test (8 hour) started every 2 hours Automated Test system Stable branch (launch mode) Creation some time before launch Every correction also to dev Reposynchfrequently Daily tags (<24 hours lead time from submit) 24h loop 1h loop Logs and dumps returned if fail GIT/Gerrit/Jenkins
  5. 5. Delivery and Post Integration Feature branch (development) Automated and manual Test systems Daily delivery to customer Lead time from developer submit to customer delivery < 24 hours Continuous Quality reports and Error reports long loop Long execution time tests after delivery (1-6 weeks) Stability tests, regulatory tests and Field tests
  6. 6. Discussion • Is the solution we did “competitive” in terms of latest process thinking's ? – Yes/No, and why ? • Large investment goes into test automation, test case development and test environment maintenance – Is this a generic learning ? – When is it worth while to invest in test automation ? Any experience from similar, smaller or larger systems • Other feedback