Presented by Morné Joubert, u-blox

1. locate, communicate, accelerate
Optimising the Developer Pipeline:
Deliver Faster & Better
Perforce, Jenkins and pre-submit
Morné Joubert
September 2015

2. Content
Slide 2 © u-blox Melbourn Ltd.
• Brief info about u-blox
• Our environment
• Our challenges
• Our solution
• stream_tester
• Perforce
• Jenkins
• Pre-submit

3. u-blox at a glance
• Swiss semiconductor company
• Founded in 1997
• Listed on the SIX Swiss Exchange since 2007 (SIX:UBXN)
• Core competencies
• Cellular communications technologies
• Short range radio technologies
• Satellite based positioning technologies
• Product offering
• Integrated circuits – the foundation of our solutions
• Modules – fully implemented, drop-in solutions
• Services – delivering added value to our solutions
• Market focus
• Automotive – robust, automotive grade products
• Industrial – durable components for professional electronics
• Consumer – mass market ICs and modules
• Business model
• Fabless operation
• Commercial, off-the-shelf products
Slide 3 © u-blox Melbourn Ltd.

4. Global market leaders rely on u-blox
ConsumerIndustrialAutomotive
Slide 4 © u-blox Melbourn Ltd.

5. Engineering Environment
• Toolchains are COTS and bespoke
• Different teams on different tool chain versions
• Execution and test platforms
• PC platforms
• Simulators
• FPGA platforms
• chip / modules
• Live air
Slide 5 © u-blox Melbourn Ltd.

6. Perforce sites
Slide 6 © u-blox Melbourn Ltd.
SanDiego
Lahore
Cork
Melbourn
Leuven
Sgonico
Thalwil

7. Resource restrictions
• Developing and testing Cellular and wireless products not cheap
• Limited resource availability
• Physical numbers of devices
• Test kit £500k+
• Compilers £5k each
Slide 7 © u-blox Melbourn Ltd.

8. Challenge for Continuous Integration
• Integration tests can take a long time
• Each test potentially measured in minutes
• Full system test sweep could take days
• Cannot afford the luxury of running all tests on submission
• Queue and resource management is an issue
• Builds and test can sit in the CI queue for a long time
• Cannot afford the luxury of running all builds on submission
Slide 8 © u-blox Melbourn Ltd.

9. Continuous Integration : our take
• All about change management
• Any change should be visible in workflow
• Code changes
• Tool and compiler changes
• Test inputs
• OS changes
• Any change should be able to be tested before commit
• Version everything
• CI is built on trust
• “cry wolf syndrome”
• Developers need to part of solution
• Fast feedback is not a “nice to have” , it is a requirement
Slide 9 © u-blox Melbourn Ltd.

10. Default workflow
Slide 10 © u-blox Melbourn Ltd.
TriggerCheck in
Jenkins
Build
Build
Build
Tests
Tests
Tests
Build
Feedback
Submitchange
to stream
Change detected
by Jenkins
Set of pre-definedjobs
are executed
Change validationafter all
itemsexecuted

11. Providing pre-submit functionality
• User should be able to pre-submit any change to see what effect the change could
have
• pre-submit change could be
• Copy up/Promoting changes to mainline streams
• Prototyping new compiler
• Specific fix for a particular failure
• Jenkins doesn’t providepre-submit framework
Slide 11 © u-blox Melbourn Ltd.

12. Providing selective Jenkins execution
• Scenario
• Test #85 of 100 fails , suite duration 1 hour, #85 duration 5 minutes
• User wants to test fix to failure by first just re-running failed test
• Scenario
• Build fails only on a particular platform variant
• User want to test fix on only the failed build
• Scenario
• Build in a pipeline fails
• User just wants to test downstream build, not whole chain
• Scenario
• Test seems to only fail on a particular slave connected to particular HW
• User wants to re-run test on that slave only
Slide 12 © u-blox Melbourn Ltd.

13. Solution: stream_tester
• To pre-submit and specify specific job items in Jenkins an internal client
application was developed called “stream_tester”
Slide 13 © u-blox Melbourn Ltd.
Ability to select
individual jobs
Ability to select
partsof chain
Viewkey indicators
Including P4 CLnumbers
Groupjobsinto tabs
Specific test or parameterscanbe
modified

14. Integration into P4V
Slide 14 © u-blox Melbourn Ltd.
• Don’t want to give users yet another system to use
• Easy option was just to add it as a plugin to P4V

15. Integration into P4V
• User clicks on a stream or shelved CL and selects the stream_tester option
Slide 15 © u-blox Melbourn Ltd.

16. Improved workflow
Slide 16 © u-blox Melbourn Ltd.
//stream
Reduced feedback time due to subset execution
Shelved CL
Manages
Jenkins
Build
Build
Build
Tests
Tests
Tests
Build
• Abstracting CI (Jenkins) away from users
• Simple as check particularbuildsor let tool auto decide
• User can now test any stream + test shelved changes
stream_tester

17. Jenkins + Perforce P4 plugin
• The ability to pre-submit is enabled by Perforce’s native ability to shelve and
unshelve
• Different strategies can be used
• Full sync + user changes
• Incremental sync+ user changes
• revert and reconcile ensures known state where
full sync is not feasible
• Perforce enables intelligent workspace
management, e.g. 200 Jenkins jobs can all share
a single workspace vs default behaviourof a
workspace per job on a single build slave
Slide 17 © u-blox Melbourn Ltd.
Add user changes if needed
unshelve unshelve
sync
sync -f sync @
Prepare
delete all revert+reconsile
P4 Plugin
Full Sync Incremental Sync

18. Metrics, Data and Reporting
• “You can only improve what you can measure” in a CI system
• Jenkins as a CI doesn’t provide the granularity of measurements we needed
• We wanted to track a change across all builds
• Links back to full changemanagement
• What impact did changeX have ?
• Wanted ability to measure any KPI or metric
Slide 18 © u-blox Melbourn Ltd.

19. Metrics, Data and Reporting
• stream_tester automatically insert various measure points at relevant stages in the CI
process in Jenkins
• These then get pushed to a SQL database
Slide 19 © u-blox Melbourn Ltd.
Build
Build
Build
Tests
Tests
Tests
Build
Jenkins
Reporting
framework

20. Reports
• With all the data different teams can now track their KPI or metrics
• Do any level of data mining that is necessary
Slide 20 © u-blox Melbourn Ltd.

21. Reports: Durations
• The CI team can now track P4 sync times, queue times, IT dependant issues
• Developers can track why their build is taking long for configuration X
Slide 21 © u-blox Melbourn Ltd.

22. Reports: Metrics
• Since P4 handles every change as incremental CL number , reporting can very
easily indicate the impact on a particular change
Slide 22 © u-blox Melbourn Ltd.

23. Reports: Status across CL
• Show results of tests in a suite over CL
• Show results of different builds over CL
• Highlights flaky build or tests
Slide 23 © u-blox Melbourn Ltd.

24. Improved workflow
Slide 24 © u-blox Melbourn Ltd.
//stream
Reduced feedback time due to subset execution
Shelved CL
Manages
Jenkins
Build
Build
Build
Tests
Tests
Tests
Build
stream_tester
Reporting
frameworkReports,detailedchange impactvisibility

25. Perforce Swarm integration
• Perforce Swarm provides codereview for shelved changes
• stream_tester provides testing for shelved changes
• Framework can provide reports for shelved changes
• These 3 can now be integratedto reduce review time and highlight issues that would have been
difficult to just detect from looking at code changes
• User shelve changesfor review
• User tests shelved changes via stream_tester
• User uses reports to feed back intoswarm
• User re-shelve fixes intosame review if reports indicate failure
• “rinse and repeat”
• Swarm review now combines results+ the necessary code review
Slide 25 © u-blox Melbourn Ltd.

26. JIRA : Issue tracking integration
• We use JIRA as our agile sprint planning, work definition and defect tracking tool
• Integration into the system allows visibility of JIRA into the CI and workflow
Slide 26 © u-blox Melbourn Ltd.

27. 360 Feedback
• Just getting a PASS or FAIL from a CI system hides potential problems that can cause
major delays when things start to fail
• E.g. Memory usage can slowly creep up until it cannot fit on embeddeddevice
• The reporting framework can track such a metric but unless people actively track the report
a sudden major change could be detected too late
• A notification rule engine is another system we added
• Teams can add rules to notify users, teams, managers etc
• if Memory goes up by more than X MB email Y
• If Memory goes over X contact Y
• If Warnings decrease by X congratulate change user(s)
• If P4 sync time is more than X minutes contact P4 team
• If Metric on stream_A is greater than metric on stream_B
• etc
• Users then get personalized emails straight after a CI action to notify them if any of the
rules matches (for pre-submit and checked in code)
Slide 27 © u-blox Melbourn Ltd.

28. Workflow : Putting it all together
Slide 28 © u-blox Melbourn Ltd.
//stream
Reduced feedback time dueto subsetexecution
Shelved CL
Manages
Jenkins
Build
Build
Build
Tests
Tests
Tests
Build
stream_tester
Reporting
framework
Reports, detailed change impact visibility
Swarm
360 feedbackEmails

29. Conclusions
• We now have a very flexible system that allows
• Maximum visibility of any change
• Ability to pre-submit
• Selective build and test if needed
• Simplification of development branch strategies
• Re-use of Perforce enterprise features
• Integration of other systems (Reports, JIRA, SWARM etc)
• Fast feedback to developers
• ContinuousKPI and metric tracking and feedback
• Time that mainline or branch/stream broken to be greatly reduced
• Rapid prototyping
• Improved development velocity of teams
Slide 29 © u-blox Melbourn Ltd.

30. Jenkins P4 plugin
• Thanks to Paul Allen @ Perforce for continuing improving the Jenkins plugin and
adding my requested features
• Thanks to Sven Erik Knop @ Perforce for inviting me
Slide 30 © u-blox Melbourn Ltd.

31. Questions ?
Slide 31 © u-blox Melbourn Ltd.