• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Building For Mer
 

Building For Mer

on

  • 2,184 views

Since Fremantle Extras applications will eventually be submitted to the Mer builder it may be a good idea to introduce it. ...

Since Fremantle Extras applications will eventually be submitted to the Mer builder it may be a good idea to introduce it.
We use the openSuse Open Build Service; a GPL service that provides an emulated, pristine (yes, I'm looking at you autobuilder and scratchbox), dependency driven build environment.
I'll talk about the processes around Mer builds, access controls, managing integration with our DVCS (git), acceleration tricks and generally how to make good use of things you find lying about on the web.

This slideset has useful notes - see tab next to 'Comments'

Statistics

Views

Total Views
2,184
Views on SlideShare
2,181
Embed Views
3

Actions

Likes
1
Downloads
30
Comments
0

1 Embed 3

http://www.slideshare.net 3

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • I'm going to talk about the Mer build system Next: OBS
  • OBS .... Next : Whats OBS
  • Summary It's two things : * a build service * a build environment manager It's not a packaging system – it works with normal debian/maemo packages Thanks to Suse Just to get out of the way... this is not a Novell proprietary service - Full GPL What does the service do? you upload (check-in) a tarball it spits out binary packages. They do provide a huge number of machines in a public build farm but you are free to setup an independent OBS instance Both openSUSE and SLES (enterprise) build on it Linux foundation will be launching a large OBS instance (~100 cores) real soon – Mer is one of the few test projects ARM joined LF last month Next: Compare to sbox
  • Build environment is very similar to sbox – *much* cleaner Allows you to run arm binaries on intel desktops Compile/install supported by improving qemu so it supports all required calls Local chroot into a qemu environment – uses the same binaries you (could) use on the device Multiple (clean) development environments – 1 per project. All integrated into your desktop Sbox benefits – used to be speed – but that's fixed now (talk more about how we did this cleanly later) SB reaches deeply into the buildenv and replaces sh, autoconf, perl, python, etc... means you may have different versions of apps to those on the device. Next steps involve kvm and sdk integration to improve the sdk Next: What it gives you
  • Why? Developers: Simple to use; reliable Truly clean/pure build environment Easy upload, At-a-glance immediate status if something is wrong web interface to track build logs in real time no home made solutions flexibility and easy building for multiple targets (armv5, v7fp, i386, Stable, Testing) Maintainers: Status overview is crucial Easy promotion Tools to support processes (like promotion) 'Make like' so entire distribution is consistent Community: Very very open Start your own project, build against Mer Build against Stable/Testing/Devel Drawbacks: Require access to the service to setup a chroot Too many :s Used to be 'emulated' compile but not now. Next Screenshots
  • Fully CLI osc such as : build buildlog checkout copypac getbinaries list (ls) meta repos All written in Python / shell Scriptable eg. script that takes a local file or http location for a .dsc file, imports it, and submits it to :Extras:Testing
  • Simple HTTP/XML API Next web
  • Essentially a good gui for rarer overview operations rather than package management ones Cli used for development Web for overview Next monitor
  • At a glance view of package status Live logs Easy enable/disable of build/publish Mouseover says “nothing provides gnome-vfs-filechooser needed by libhildon2” (circular dependency... cheers!) Next: using OBS remote service
  • Version control with make-like dependencies using build-dependencies As soon as you commit the .dsc and tar.gz it is scheduled to build There is a status of where the job is. As the build progresses you can watch a live feed of the log(s)... You may have setup sbox recently; lets take a look at local builds... Next: Local Build
  • Really ... it's that simple The initial setup to get to that point is pretty much a 2-liner Add a repo to your apt sources Install osc and qemu Run the co command to get the source Run the build command Grab a coffee Install the .deb Next: Packaging
  • I want to stress : OBS has nothing to do with maemo packaging If anything it is better than autobuilder since the packages have no scratchbox-isms If a correct debian package fails then it is a bug (there are some complex conditional build-depends that fail at the moment) Next: Mer layout
  • Complex to explain but really quite simple. Project – main grouping mechanism. Like a directory Contains Packages you want to build Build against Repositories – bad name. Where build dependencies are satisfied; a distro like Ubuntu is an obvious repo ; 2 Types: Other projects Download on demand (cached) However packages first look locally for build dependencies, then in the repository projects and their parent repositories The repository also specifies the architecture (Ubuntu 9.04 armv5el / i586) So “Maemo:Mer:Devel:UI/MerDevel_Ubuntu9.04 armv5el” Maemo:Mer:Devel:UI project Builds against Maemo:Mer:Devel project In turn against Ubuntu_9.04 (from ports) armv5 arch Next: Mer Projects
  • These are OBS 'projects' which contain all the individual packages In Devel we have: Apps, Base, MaemoCommon, Qt, UI and more Essentially move from Devel to Test to a numbered release Stable then points to the latest release using kinda symlinks HW specific areas Next: Extras and user builds
  • One approach is to submit to our 'Extras' Very similar to Maemo Extras process Initial work is done in :Devel (or locally) move to :Testing when ready for power users move to :Stable when ready for all users Clearly you don't need all 12 builds all the time; individually enabled Most will only need to build against Stable Package releases are not tied to Core releases Rebuilds triggered whenever build dependencies in the core are changed Next : Home areas
  • Alternatively you can just setup your own home area and build 'against' the Mer repos. One benefit is email/rss notifications of a rebuild Next: osc
  • Listed the main commands to give a flavour VCS like – lots more to do with managing the remote server; getting binaries etc Main thing is to checkout packages and build them Usually you checkout, then use git to manage the code, build a package and commit the tarball Futures: We anticipate that co will do a git checkout and an osc commit will cause OBS to use that to pull a tarball and build it... Next: Move on to git branches and upstream etc
  • Issue is we need to manage patches against our upstream And we want to make life easy for our downstream. That means keeping the packaging and the code separate. Start with an upstream package : unpack and remove the debian/ directory. commit onto the upstream branch. pristine-tar is used to ensure that the upstream tarball can be recreated perfectly. (not used much) All features or bug fixes are broken out to additional branches; 1 branch per feature or fix. The main difference is that features are not likely to go upstream whilst fixes are. This is very similar to quilt. Very hard to manage patches in the debian/patch dir master is the main code branch and there should only be merge commits on this branch; the only non-merge commits should be in ./ChangeLog Mer should be packaging-based and the only non-merge commits should be in debian/ Next: Stable point release
  • Shows how we package a Stable point release Next: Move on to see how we handle an upstream release
  • We make changes to our fixes and features They need re-integrating when upstream releases A monolithic patch means you don't always know what a particular change is related to. Having logical changes in multiple branches makes this sane. Next: Acceleration
  • Discussed how qemu emulates; does this on a binary-by-binary level using binfmt-misc so ideally use 586 binaries. Just those that impact compilation Could do static builds but fiddly Use dynamic - but .so library paths collide. So we create another 'root' /lib-x86 and install all modified packages in there. Modified OBS to do this using mkbaselibs – an rpm technique. Fixup the binaries rpath on install with a static patchelf / chrpath; then symlink the binaries into the real chroot and move the real binaries to .old Toolchain replaced with a slightly modified gcc-cross (that was hard – weeks of work!) If we install these binaries in a chroot we get 5x to 10x performance increase Same source as if using arm binaries – unlike scratchbox. Disable accel and you use a pure arm chroot again Next : Issues
  • Mer devices are clearly multimedia oriented so music and video codecs are important Novell OBS only supports supported Ubuntu... ie not Universe. Ubuntu have some strange omissions As Mer grows we'll want to open up submit rights but not want users to write to other users packages. Need a login to Novell but soon using openID Linux Foundation OBS is big news!
  • Chatter on #mer is very interactive Lots of support Mer Chatter is interesting ... if you're a hardcore Mer geek. Worked hard on the docs – let me know of omissions etc Help make the SDK

Building For Mer Building For Mer Presentation Transcript

  • Building Mer David Greaves / lbt
  • Building Mer OBS
  • What is OBS?
      Open Build System: Source -> Binary -> Package -> Repository Fully GPL Huge thanks to Suse / Novell for creating OBS in an open manner OBS ..... Suse / Novell (cf Qt ..... Nokia)
  • What is OBS? OBS Scratchbox cf
  • What is OBS?
      Benefits
        Developers Maintainers Maemo Community
      Drawbacks
        Oriented towards full builds not compile+test Lots of “:”s Emulation-mode compile.... for the moment accelerated compilation
  • Using OBS
      Screenshot:
  • Using OBS
      Screenshot:
  • Using OBS
      Screenshot:
  • Using OBS
      Screenshot:
  • Using OBS Service to Build Remotely
      Remotely
        Like an old-fashioned centralised VCS
          When you check-in a package it's scheduled and built All target repositories and architectures
        Then any dependendent packages are rebuilt Status overview provided Live Logs
  • Using OBS to Build Locally
        Install:
          apt-get install osc qemu
        Checkout :
          osc co Maemo:Mer:Testing/qtablet
        Build:
          osc build Maemo:Mer:Testing armv5el qtablet_0.1.dsc
      ..........................Coffee
        Copy .deb to tablet and install
  • Packaging
      Maemo packaging process Modify, compile, test, build locally Update debian/changelog “ debuild -S” to create updated .diff and .dsc “ osc addremove” and “osc commit”
  • OBS Projects, Packages and “Repositories”
      An OBS Project is a collection of packages and has a number of “repositories” A “repository” defines the base package collection used to satisfy build dependencies Two kinds:
        Other projects Download on demand
      Key Download on demand:
        Ubuntu 9.04 armv5el / i586 Debian 5.0 armv5el / i586
      Names like “Maemo:Mer:Devel:UI/MerDevel_Ubuntu9.04 armv5el”
  • Developing for Mer
      'Core' Structure
        Maemo:Mer:Devel & :<areas> Maemo:Mer:Testing Maemo:Mer:0.15 Maemo:Mer:Stable
      HW Specifics in each project:
        :HW:N8x0 :HW:Freerunner :HW:SmartQ :HW:Marvell :HW:HTC :HW:Beagle
  • Developing for Mer
      Submit to autobuilder and it goes into 'Extras' Structure
        Maemo:Mer:Extras:Devel Maemo:Mer:Extras:Testing Maemo:Mer:Extras:Stable
      Each builds 12x .... against
        :Devel and :Testing and :Stable Ubuntu and Debian i586 and armel
      Independent release cycle... per package
  • Developing for Mer
      Home project
        home:lbt:Extras
      Has repositories for:
        :Devel and :Testing and :Stable i586 and armel
  • osc commands
      osc
        • co Checkout a package or project
        • add / rm / addremove Add a dsc or tar.gz or a diff.gz
        • commit Send new files to server
        • up Update - get changes
        • build Download build-deps, create chroot, run dpkg-buildpkg
        • repos Show the available repos to build
        • ls <proj> List packages
        • meta pkg/proj/... Add/remove dis/enable packages & build targets
  • Gitorious and Branches
  • Gitorious and Branches
  • Gitorious and Branches
  • Acceleration
      Qemu is used to emulate on a binary-by-binary level using binfmt-misc Be nice to replace qemu binaries with i586 binaries
        bash, gzip, make, tar,
      Problem with .so library paths fixed with patchelf / chrpath Toolchain replaced with a slightly modified gcc-cross If we install these binaries in a chroot we get 5x to 10x performance increase Same source as if using arm binaries – unlike scratchbox
  • Issues for Mer
      Patents...
        the Novell service won't accept mp3 software
      Ubuntu universe
        isn't on Novell service
      Access control
        No fine grained control over packages
  • Talk to Us IRC #mer on freenode.net Microblogging..... Mer Chatter [email_address] highlights of 'interesting' activities Summary at http://wiki.maemo.org/Mer/Build
  • Thanks
      Carsten Munk Martin Mohringer (ScriptRipper), Adrian Schröter (AdrianS) Marcus Rückert (darix) Jan-Simon Möller (dl9pf) Riku Voipio (suihkulokki) rest of the teams testers, documenters, community