Building For Mer

1,524 views

Published on

Since Fremantle Extras applications will eventually be submitted to the Mer builder it may be a good idea to introduce it.
We use the openSuse Open Build Service; a GPL service that provides an emulated, pristine (yes, I'm looking at you autobuilder and scratchbox), dependency driven build environment.
I'll talk about the processes around Mer builds, access controls, managing integration with our DVCS (git), acceleration tricks and generally how to make good use of things you find lying about on the web.

This slideset has useful notes - see tab next to 'Comments'

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,524
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
33
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • I'm going to talk about the Mer build system Next: OBS
  • OBS .... Next : Whats OBS
  • Summary It's two things : * a build service * a build environment manager It's not a packaging system – it works with normal debian/maemo packages Thanks to Suse Just to get out of the way... this is not a Novell proprietary service - Full GPL What does the service do? you upload (check-in) a tarball it spits out binary packages. They do provide a huge number of machines in a public build farm but you are free to setup an independent OBS instance Both openSUSE and SLES (enterprise) build on it Linux foundation will be launching a large OBS instance (~100 cores) real soon – Mer is one of the few test projects ARM joined LF last month Next: Compare to sbox
  • Build environment is very similar to sbox – *much* cleaner Allows you to run arm binaries on intel desktops Compile/install supported by improving qemu so it supports all required calls Local chroot into a qemu environment – uses the same binaries you (could) use on the device Multiple (clean) development environments – 1 per project. All integrated into your desktop Sbox benefits – used to be speed – but that's fixed now (talk more about how we did this cleanly later) SB reaches deeply into the buildenv and replaces sh, autoconf, perl, python, etc... means you may have different versions of apps to those on the device. Next steps involve kvm and sdk integration to improve the sdk Next: What it gives you
  • Why? Developers: Simple to use; reliable Truly clean/pure build environment Easy upload, At-a-glance immediate status if something is wrong web interface to track build logs in real time no home made solutions flexibility and easy building for multiple targets (armv5, v7fp, i386, Stable, Testing) Maintainers: Status overview is crucial Easy promotion Tools to support processes (like promotion) 'Make like' so entire distribution is consistent Community: Very very open Start your own project, build against Mer Build against Stable/Testing/Devel Drawbacks: Require access to the service to setup a chroot Too many :s Used to be 'emulated' compile but not now. Next Screenshots
  • Fully CLI osc <cmd> such as : build buildlog checkout copypac getbinaries list (ls) meta repos All written in Python / shell Scriptable eg. script that takes a local file or http location for a .dsc file, imports it, and submits it to :Extras:Testing
  • Simple HTTP/XML API Next web
  • Essentially a good gui for rarer overview operations rather than package management ones Cli used for development Web for overview Next monitor
  • At a glance view of package status Live logs Easy enable/disable of build/publish Mouseover says “nothing provides gnome-vfs-filechooser needed by libhildon2” (circular dependency... cheers!) Next: using OBS remote service
  • Version control with make-like dependencies using build-dependencies As soon as you commit the .dsc and tar.gz it is scheduled to build There is a status of where the job is. As the build progresses you can watch a live feed of the log(s)... You may have setup sbox recently; lets take a look at local builds... Next: Local Build
  • Really ... it's that simple The initial setup to get to that point is pretty much a 2-liner Add a repo to your apt sources Install osc and qemu Run the co command to get the source Run the build command Grab a coffee Install the .deb Next: Packaging
  • I want to stress : OBS has nothing to do with maemo packaging If anything it is better than autobuilder since the packages have no scratchbox-isms If a correct debian package fails then it is a bug (there are some complex conditional build-depends that fail at the moment) Next: Mer layout
  • Complex to explain but really quite simple. Project – main grouping mechanism. Like a directory Contains Packages you want to build Build against Repositories – bad name. Where build dependencies are satisfied; a distro like Ubuntu is an obvious repo ; 2 Types: Other projects Download on demand (cached) However packages first look locally for build dependencies, then in the repository projects and their parent repositories The repository also specifies the architecture (Ubuntu 9.04 armv5el / i586) So “Maemo:Mer:Devel:UI/MerDevel_Ubuntu9.04 armv5el” Maemo:Mer:Devel:UI project Builds against Maemo:Mer:Devel project In turn against Ubuntu_9.04 (from ports) armv5 arch Next: Mer Projects
  • These are OBS 'projects' which contain all the individual packages In Devel we have: Apps, Base, MaemoCommon, Qt, UI and more Essentially move from Devel to Test to a numbered release Stable then points to the latest release using kinda symlinks HW specific areas Next: Extras and user builds
  • One approach is to submit to our 'Extras' Very similar to Maemo Extras process Initial work is done in :Devel (or locally) move to :Testing when ready for power users move to :Stable when ready for all users Clearly you don't need all 12 builds all the time; individually enabled Most will only need to build against Stable Package releases are not tied to Core releases Rebuilds triggered whenever build dependencies in the core are changed Next : Home areas
  • Alternatively you can just setup your own home area and build 'against' the Mer repos. One benefit is email/rss notifications of a rebuild Next: osc
  • Listed the main commands to give a flavour VCS like – lots more to do with managing the remote server; getting binaries etc Main thing is to checkout packages and build them Usually you checkout, then use git to manage the code, build a package and commit the tarball Futures: We anticipate that co will do a git checkout and an osc commit <tag> will cause OBS to use that to pull a tarball and build it... Next: Move on to git branches and upstream etc
  • Issue is we need to manage patches against our upstream And we want to make life easy for our downstream. That means keeping the packaging and the code separate. Start with an upstream package : unpack and remove the debian/ directory. commit onto the upstream branch. pristine-tar is used to ensure that the upstream tarball can be recreated perfectly. (not used much) All features or bug fixes are broken out to additional branches; 1 branch per feature or fix. The main difference is that features are not likely to go upstream whilst fixes are. This is very similar to quilt. Very hard to manage patches in the debian/patch dir master is the main code branch and there should only be merge commits on this branch; the only non-merge commits should be in ./ChangeLog Mer should be packaging-based and the only non-merge commits should be in debian/ Next: Stable point release
  • Shows how we package a Stable point release Next: Move on to see how we handle an upstream release
  • We make changes to our fixes and features They need re-integrating when upstream releases A monolithic patch means you don't always know what a particular change is related to. Having logical changes in multiple branches makes this sane. Next: Acceleration
  • Discussed how qemu emulates; does this on a binary-by-binary level using binfmt-misc so ideally use 586 binaries. Just those that impact compilation Could do static builds but fiddly Use dynamic - but .so library paths collide. So we create another 'root' /lib-x86 and install all modified packages in there. Modified OBS to do this using mkbaselibs – an rpm technique. Fixup the binaries rpath on install with a static patchelf / chrpath; then symlink the binaries into the real chroot and move the real binaries to .old Toolchain replaced with a slightly modified gcc-cross (that was hard – weeks of work!) If we install these binaries in a chroot we get 5x to 10x performance increase Same source as if using arm binaries – unlike scratchbox. Disable accel and you use a pure arm chroot again Next : Issues
  • Mer devices are clearly multimedia oriented so music and video codecs are important Novell OBS only supports supported Ubuntu... ie not Universe. Ubuntu have some strange omissions As Mer grows we'll want to open up submit rights but not want users to write to other users packages. Need a login to Novell but soon using openID Linux Foundation OBS is big news!
  • Chatter on #mer is very interactive Lots of support Mer Chatter is interesting ... if you're a hardcore Mer geek. Worked hard on the docs – let me know of omissions etc Help make the SDK
  • Building For Mer

    1. 1. Building Mer David Greaves / lbt
    2. 2. Building Mer OBS
    3. 3. What is OBS? <ul>Open Build System: Source -> Binary -> Package -> Repository Fully GPL Huge thanks to Suse / Novell for creating OBS in an open manner OBS ..... Suse / Novell (cf Qt ..... Nokia) </ul>
    4. 4. What is OBS? OBS Scratchbox cf
    5. 5. What is OBS? <ul>Benefits <ul>Developers Maintainers Maemo Community </ul>Drawbacks <ul>Oriented towards full builds not compile+test Lots of “:”s Emulation-mode compile.... for the moment accelerated compilation </ul></ul>
    6. 6. Using OBS <ul>Screenshot: </ul>
    7. 7. Using OBS <ul>Screenshot: </ul>
    8. 8. Using OBS <ul>Screenshot: </ul>
    9. 9. Using OBS <ul>Screenshot: </ul>
    10. 10. Using OBS Service to Build Remotely <ul>Remotely <ul>Like an old-fashioned centralised VCS <ul>When you check-in a package it's scheduled and built All target repositories and architectures </ul>Then any dependendent packages are rebuilt Status overview provided Live Logs </ul></ul>
    11. 11. Using OBS to Build Locally <ul><ul>Install: <ul>apt-get install osc qemu </ul>Checkout : <ul>osc co Maemo:Mer:Testing/qtablet </ul>Build: <ul>osc build Maemo:Mer:Testing armv5el qtablet_0.1.dsc </ul></ul>..........................Coffee <ul>Copy .deb to tablet and install </ul></ul>
    12. 12. Packaging <ul>Maemo packaging process Modify, compile, test, build locally Update debian/changelog “ debuild -S” to create updated .diff and .dsc “ osc addremove” and “osc commit” </ul>
    13. 13. OBS Projects, Packages and “Repositories” <ul>An OBS Project is a collection of packages and has a number of “repositories” A “repository” defines the base package collection used to satisfy build dependencies Two kinds: <ul>Other projects Download on demand </ul>Key Download on demand: <ul>Ubuntu 9.04 armv5el / i586 Debian 5.0 armv5el / i586 </ul>Names like “Maemo:Mer:Devel:UI/MerDevel_Ubuntu9.04 armv5el” </ul>
    14. 14. Developing for Mer <ul>'Core' Structure <ul>Maemo:Mer:Devel & :<areas> Maemo:Mer:Testing Maemo:Mer:0.15 Maemo:Mer:Stable </ul>HW Specifics in each project: <ul>:HW:N8x0 :HW:Freerunner :HW:SmartQ :HW:Marvell :HW:HTC :HW:Beagle </ul></ul>
    15. 15. Developing for Mer <ul>Submit to autobuilder and it goes into 'Extras' Structure <ul>Maemo:Mer:Extras:Devel Maemo:Mer:Extras:Testing Maemo:Mer:Extras:Stable </ul>Each builds 12x .... against <ul>:Devel and :Testing and :Stable Ubuntu and Debian i586 and armel </ul>Independent release cycle... per package </ul>
    16. 16. Developing for Mer <ul>Home project <ul>home:lbt:Extras </ul>Has repositories for: <ul>:Devel and :Testing and :Stable i586 and armel </ul></ul>
    17. 17. osc commands <ul>osc <ul><ul><li>co Checkout a package or project
    18. 18. add / rm / addremove Add a dsc or tar.gz or a diff.gz
    19. 19. commit Send new files to server
    20. 20. up Update - get changes
    21. 21. build Download build-deps, create chroot, run dpkg-buildpkg
    22. 22. repos Show the available repos to build
    23. 23. ls <proj> List packages
    24. 24. meta pkg/proj/... Add/remove dis/enable packages & build targets </li></ul></ul></ul>
    25. 25. Gitorious and Branches
    26. 26. Gitorious and Branches
    27. 27. Gitorious and Branches
    28. 28. Acceleration <ul>Qemu is used to emulate on a binary-by-binary level using binfmt-misc Be nice to replace qemu binaries with i586 binaries <ul>bash, gzip, make, tar, </ul>Problem with .so library paths fixed with patchelf / chrpath Toolchain replaced with a slightly modified gcc-cross If we install these binaries in a chroot we get 5x to 10x performance increase Same source as if using arm binaries – unlike scratchbox </ul>
    29. 29. Issues for Mer <ul>Patents... <ul>the Novell service won't accept mp3 software </ul>Ubuntu universe <ul>isn't on Novell service </ul>Access control <ul>No fine grained control over packages </ul></ul>
    30. 30. Talk to Us IRC #mer on freenode.net Microblogging..... Mer Chatter [email_address] highlights of 'interesting' activities Summary at http://wiki.maemo.org/Mer/Build
    31. 31. Thanks <ul>Carsten Munk Martin Mohringer (ScriptRipper), Adrian Schröter (AdrianS) Marcus Rückert (darix) Jan-Simon Möller (dl9pf) Riku Voipio (suihkulokki) rest of the teams testers, documenters, community </ul>

    ×