Your SlideShare is downloading. ×
0
DEVOPS FOR EMBEDDEDSOFTWARE DEVELOPMENTScott HallockMay 20, 2013
Agenda• Introduction• What We Do• How We Do It• Strengths and Challenges
Introduction                            BAIT• Build, Automation, Integration, and Test• Continuous integration and automat...
Objectives• High-level overview of embedded DevOps processes and  practices• Start an exchange of knowledge between embedd...
Embedded Development Is Different• Tools don’t evolve quickly   • Command line compilers   • Everybody loves the preproces...
Continuous Integration  Review   Build   Deploy   Test   Integrate
Continuous Integration Deployment• Datacenter-quality hardware• Rack-mountable form factor• Stable, controlled operating s...
Embedded Continuous Integration  Review   Build   Load   Test   Integrate
Target Load Challenges• Developer-quality hardware• Irregular form factor• Unstable loading software, boot code, operating...
Target Load Solutions• Single controlling server for each target hardware board• Physical access to production hardware  •...
Time Constraints                          90 minutes  Review   Build   Load            Test      Integrate                ...
Preverification                    90 minutes, many changes  Review            Build      Load         Test       Integrat...
Automated Release     Build   Test   Approve   git push
Code Change Control: git• http://git-scm.com• Distributed is great!  • Developers can work offline, at customer sites with...
git Problems• Everyone gets everything  • Do you check in a helper binary every time you change it?  • Did you accidentall...
git Problems• git trees become unusable over time. Are you garbage collecting? • On your central repositories? • When you ...
git Problems• Cherry-picking  • I can just have the two changes I want for today’s release? Great!  • Breaks all traceabil...
Codeline Definition: repo• http://source.android.com/source/using-repo.html• Allows for many git projects to be combined i...
repo: Project Addition      Build   Test   Approve   git push!!!
repo: Branch Change<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“...
repo: Branch Change     Build   Test   Approve   git push!!!
repo: Branch Change• Merge?  • “master” branch contains dozens of bugs. You just merged them    with your stable branch.• ...
repo: Shared Branchescustomer-1.xml:<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=...
Gerrit: Code Review and More• https://code.google.com/p/gerrit/• Code review, access control, source code mirroring
Gerrit: Code Review• States Tracked:  • Code Review (-2 - +2): Peer opinion of design and code quality  • Verified (-1 - +...
Gerrit: Access Control• Most actions in Gerrit are access controlled  • Code visibility  • Upload  • Review  • Merge• Acce...
Gerrit: Mirroring                                 Master                    Slave        Slave          Slave• Single mast...
Code Aurora Forum• http://codeaurora.org
Other Tools• Electric Commander  • http://www.electriccloud.com/products/electriccommander.php• Jira   • https://www.atlas...
Strengths• Chasing the sun  • Operational teams in major timezones  • Handoff procedure is well understood• Cheap scaling ...
Challenge: Source Code Distribution                            Master                 Slave      Slave!!!    Slave• Single...
Challenge: Build Distribution       Build     Test      Approve   git push      25GB x    Boulder,      ~50/day     San   ...
Challenge: Two-Way Integration             Internal Codeline            Integrated Codeline             Partner Codeline
Upcoming SlideShare
Loading in...5
×

DevOps For Embedded Software Development

632

Published on

A brief overview of DevOps practices and tools. Presented at the DevOps Boulder Meetup, May 20th, 2013.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
632
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "DevOps For Embedded Software Development"

  1. 1. DEVOPS FOR EMBEDDEDSOFTWARE DEVELOPMENTScott HallockMay 20, 2013
  2. 2. Agenda• Introduction• What We Do• How We Do It• Strengths and Challenges
  3. 3. Introduction BAIT• Build, Automation, Integration, and Test• Continuous integration and automated release for Linux software products • 3-5 Linux-based operating systems • Software releases nearly every day • Build Linux from source thousands of times per day • Support a worldwide Linux developer community of more than 1,000 • More than 50 actively developed codelines • Integrate more than 200 code changes per day
  4. 4. Objectives• High-level overview of embedded DevOps processes and practices• Start an exchange of knowledge between embedded and non-embedded DevOps communities• Interactivity • Please interrupt to ask questions • Detailed, tangential discussions are welcome
  5. 5. Embedded Development Is Different• Tools don’t evolve quickly • Command line compilers • Everybody loves the preprocessor • printf debugging • Eclipse viewed with suspicion• Product focus • Customers get the code, not just the experience • Fix it, ship it, forget it • Standard release: zip archive • Many similar codelines under active development simultaneously
  6. 6. Continuous Integration Review Build Deploy Test Integrate
  7. 7. Continuous Integration Deployment• Datacenter-quality hardware• Rack-mountable form factor• Stable, controlled operating system and management software• Well-defined power and thermal characteristics
  8. 8. Embedded Continuous Integration Review Build Load Test Integrate
  9. 9. Target Load Challenges• Developer-quality hardware• Irregular form factor• Unstable loading software, boot code, operating system• Unknown power and thermal characteristics• Slow and unstable JTAG, USB loading interfaces
  10. 10. Target Load Solutions• Single controlling server for each target hardware board• Physical access to production hardware • Easily reconfigure physical racks to accommodate novel form factors • Repair or replace unstable, malfunctioning target hardware• Fault-tolerant software • Lots of retries! • Heuristics to infer target state based on observable outputs• Change the target hardware design
  11. 11. Time Constraints 90 minutes Review Build Load Test Integrate 15 minutes
  12. 12. Preverification 90 minutes, many changes Review Build Load Test Integrate Build 60 minutes, one change
  13. 13. Automated Release Build Test Approve git push
  14. 14. Code Change Control: git• http://git-scm.com• Distributed is great! • Developers can work offline, at customer sites with no network access. • Collaboration is easier. Everyone has complete history.
  15. 15. git Problems• Everyone gets everything • Do you check in a helper binary every time you change it? • Did you accidentally commit some trade secrets and then revert them later? • Did your developers include an estimate of your customers’ intelligence in their commit message? • You create integration tags fifty times per day. • Your continuous integration system creates “temporary” integration branches twenty-five times per day.
  16. 16. git Problems• git trees become unusable over time. Are you garbage collecting? • On your central repositories? • When you mirror your central repositories? • On your external release servers? • On your Continuous Integration build machines?
  17. 17. git Problems• Cherry-picking • I can just have the two changes I want for today’s release? Great! • Breaks all traceability, barring commit message kludges
  18. 18. Codeline Definition: repo• http://source.android.com/source/using-repo.html• Allows for many git projects to be combined into a single source tree and operated upon as a single entity.• Developers use it to download and sync code, most use git commands for everything else.<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“bootable/bootloader/lk”revision=“refs/heads/master” />
  19. 19. repo: Project Addition Build Test Approve git push!!!
  20. 20. repo: Branch Change<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“bootable/bootloader/lk”revision=“refs/heads/master” /><project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“bootable/bootloader/lk”revision=“refs/heads/stable” />
  21. 21. repo: Branch Change Build Test Approve git push!!!
  22. 22. repo: Branch Change• Merge? • “master” branch contains dozens of bugs. You just merged them with your stable branch.• Force push? • Now your customer doesn’t understand what happened to all the changes they painstakingly integrated for the last month.• Create new branch on release server? • Customer still has to re-integrate their changes. • At least the old branch is available for reference.• Stop doing that! • How am I supposed to make my release?
  23. 23. repo: Shared Branchescustomer-1.xml:<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“bootable/bootloader/lk”revision=“refs/heads/customer-1” />customer-2.xml<project name=“kernel/msm” path=“kernel”revision=“refs/heads/msm-3.4” /><project name=“kernel/lk”path=“bootable/bootloader/lk”revision=“refs/heads/customer-2” />
  24. 24. Gerrit: Code Review and More• https://code.google.com/p/gerrit/• Code review, access control, source code mirroring
  25. 25. Gerrit: Code Review• States Tracked: • Code Review (-2 - +2): Peer opinion of design and code quality • Verified (-1 - +1): Code works at runtime• Roles • Author/Committer • Approver • Integrator
  26. 26. Gerrit: Access Control• Most actions in Gerrit are access controlled • Code visibility • Upload • Review • Merge• Access control can be based on users, on local groups, or on LDAP groups.• Access control is complex. Automation is recommended.
  27. 27. Gerrit: Mirroring Master Slave Slave Slave• Single master• Slaves can’t mirror to other slaves• Smart git push based mirroring • Better than any mirroring scheme not based on application layer knowledge of git• Errors not handled gracefully • Obvious errors are re-queued for later retries
  28. 28. Code Aurora Forum• http://codeaurora.org
  29. 29. Other Tools• Electric Commander • http://www.electriccloud.com/products/electriccommander.php• Jira • https://www.atlassian.com/software/jira/overview• BlackDuck • http://www.blackducksoftware.com/products/black-duck-suite• Gitolite • http://gitolite.com/gitolite/
  30. 30. Strengths• Chasing the sun • Operational teams in major timezones • Handoff procedure is well understood• Cheap scaling • Use simple, dumb tools and relentlessly parallelize • Understand where human intervention is inevitable• Sync, build efficiency • Custom software atop repo for very efficient repo syncs • Fastest clean builds in the business• Automated release • Very complex branching structure is released automatically multiple times per day
  31. 31. Challenge: Source Code Distribution Master Slave Slave!!! Slave• Single master mirroring has scaling limits • Git push is CPU intensive• Gerrit doesn’t report well when a slave is out of sync • Custom software needed• An independent and easily monitored sync solution is desired
  32. 32. Challenge: Build Distribution Build Test Approve git push 25GB x Boulder, ~50/day San Diego, Santa Clara, India, China, Korea…
  33. 33. Challenge: Two-Way Integration Internal Codeline Integrated Codeline Partner Codeline
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×