LCE13: Introduction to OpenEmbedded/Yocto Cross-Compile Environment


Published on

Resource: LCE13
Name: Introduction to OpenEmbedded/Yocto Cross-Compile Environment
Date: 10-07-2013
Speaker: Nicolas Dechesne, Riku Voipio, Trevor Woerner

Published in: Technology

LCE13: Introduction to OpenEmbedded/Yocto Cross-Compile Environment

  1. 1. OpenEmbedded and Yocto introduction Nicolas Dechesne <> Riku Voipio <> Trevor Woerner <> LCE13, Dublin
  2. 2. 2 Agenda ● Intro to OpenEmbedded / Yocto ● Overview of Yocto Project, organization and releases ● Yocto Project vs. DIY ● Introduction to OpenEmbedded concepts (recipes, packages, images, layers, configuration, SDK, …) ● OpenEmbedded/Yocto and Linaro ● 'OpenEmbedded/Yocto Lab' (this afternoon, 3PM) ● Beginners practical introduction ● Guide you through a 'qemuarm' console image build ● Q&A
  3. 3. 3 What is the Yocto Project? It provides all the basic bits that every Linux product needs, pre-integrated and tested to let you spend more time on differentiators that matter to you!
  4. 4. 4 The Yocto Project ● Open source project, hosted by the Linux Foundation. ● Not the “new kid on the block”, OpenEmbedded has been around since 2003! ● A collaboration project, an “umbrella” project. A collection of projects that make up the “Yocto Project” ● Bitbake ● OpenEmbedded-Core ● Poky ● Application Development Toolkit ● And even more... ● It helps you build a 'tailored' Linux distribution/product for your exact needs. ● Support for ARM, PPC, MIPS, x86
  5. 5. 5 Yocto Project organization ● Hierarchical technical leadership: ● Richard Purdie as the project architect ● Sub system maintainers ● Somehow similar to Linux kernel technical leadership ● Yocto Project Advisory Board for administrative leadership ● community, finance, infrastructure, advocacy ● From linaro: TI, Enea, Huawei ● Open to all contributors ● Yocto Compliance program ● Yocto project participants ● Yocto project compatible
  6. 6. 6 Yocto/OE releases ● 1 new release ~6 months ● Yocto Project releases in sync with OE, bitbake projects releases ● Yocto Project release is the aggregation of oe-core, bitbake and Poky layers/tools/documentation. ● Stable releases ● v1.4 (aka dylan), v1.3 (aka danny) ● No new feature, only bug fixes ● Development / master branch ● New features (systemd, wayland, Qt5, build engine improvements, ...) ● Components version updates ● Does not 'work' every single day
  7. 7. 7 Yocto/OpenEmbedded vs DIY ● Spend less time on things that bring no value to your business: ● OE provides and maintains thousands of up-to-date recipes ● Build an entire Linux system from sources in less than 1 hour, using a validated set of packages (toolchain, busybox, libc, init system...) ● Built in support for package management ● Predictable and reproducible builds ● Provides set of standard tools and build guidelines. ● What you build for a product can be reused for other products. ● autobuilder 'bot' for continuous integration ● Minimal dependencies on host, build as standard user (no root) ● Still, you have all the flexibility you need to ● patch as needed any OE component for your needs/product ● Add any unsupported component, even proprietary bits ● Fully configure the system, and make it exactly the way you want.
  8. 8. 8 OpenEmbedded metadata ● OpenEmbedded does not contain components source code, only their metadata ● Bitbake is the build engine, a metadata parser, written in Python ● Recipes ● Component (.bb, .bbappend) ● Image (.bb) ● Class (.bbclass) ● Configuration files: behavior largely controlled by variables ● bitbake.conf ● <machine>.conf ● <distro>.conf ● local.conf, site.conf ● bblayers.conf ● Patches
  9. 9. 9 OpenEmbedded overview
  10. 10. 10 OpenEmbedded packages ● A recipe generates 1 or more binary packages ● Each recipe build is split into 'tasks' ● Tasks are scheduled by bitbake based on parallelism and dependencies ● fetch, unpack, patch, configure, build, install, package ● All packages are built from scratch with same compiler configuration (CPU, optimizations, tuning, …) according to the underlying target CPU. ● Standard packages are created managed automatically (most of the time) ● Binaries are stripped automatically, and debug binaries are placed in -dbg packages ● Development packages with .h , .so, .a, .pc, … ● ... ● Packages dependencies ● build depends (DEPENDS) are mandatory and explicit in recipes ● Run time depends (RDEPENDS) are semi-automatic (shlibdeps, pcdeps)
  11. 11. 11 OpenEmbedded images ● Images are standard recipe (e.g. .bb file), with a different 'task set' ● Images are expected to be 'deployed' on target ● Most standard image format are supported (ext2/3, tarball, jffs2, cramfs, ramdisk, …) ● Support for image generation 'post scripting' ● Packages are assembled into the final root file system to make up images ● Image recipe lists all components/applications required in the image ● RDEPENDS used to pull all the needed libraries, ... ● Multiple images can be built from the same 'build', by assembling different set of packages ● OpenEmbedded has (optional) support for package management in the target image: .rpm, .ipk, .deb. ● Can be used to implement a binary package based distro (with package feed) ● However OE always uses packages 'internally' to build images.
  12. 12. 12 OpenEmbedded layers ● Couple of years ago, OpenEmbedded used to be a monolithic repo with thousands of recipes, and became impractical to maintain and use ● Layer concept introduced ● Layers contain additional metadata (recipes, machine, distro, …) ● Oe-core is the base layer for the most fundamental components ● layers provide additional features: meta-systemd, meta-gnome, … ● Layers can be inter-dependent ● Layers can be hosted in different 'trees' (git) ● All layers must work with oe-core ● Layers can add, modify content/beahvior, and/or alter, extend metadata from other layers. ● Generally, each BSP or distro has its own layer(s) ● meta-linaro, ● OpenEmbedded metadata on line index:
  13. 13. 13 Poky ● Poky is a reference distro implementation. ● Minimal console only image ● Full Linux system with GUI using “sato” ● Generally used as a starting point (or an example) for building one's own (commercial) distro: Angstrom, ELDK, Mentor Linux, Wind River Linux, ENEA Linux, ... ● “it is often just a starting point, not an end goal” ● Built with bitbake and these layers: oe-core, meta-yocto, meta-yocto-bsp (and yes, the name is confusing!) ● Used to test/validate OpenEmbedded by Yocto Project developers ● “Poky-tiny” is a variant that provides size optimized image. ● “Poky-bleeding” is a variant that uses bleeding edge software version
  14. 14. 14 SDK ● Application Development Toolkit (ADT) provides custom built, cross development SDK tailored for the image being built ● Cross toolchain ● Matching sysroot ● Eclipse IDE Yocto plugin ● Support SDK for x86 and/or x86-64 hosts ● ADT can be installed on 'OEM', customer hosts without the need to rebuild the entire image packages ● “bitbake meta-toolchain” : includes only the minimal cross toolchain and libs ● “bitbake -c populate_sdk <image>” : includes all the development headers, libs, … for every package included in the image. e.g. build SDK that matches the target image
  15. 15. 15 Anatomy of a build folder ● Lots of space … ~20 Gb ● Entirely contained in '<build>' folder ● Sources used are in '<build>/downloads' ● Builds are in '<build>/tmp/work/<arch>' ● 'deploy' has ● Ready to use images ● Packages ● license information ● Buildhistory records information about content of images, and package build after build (with diffs) ● 'flat' rootfs also available in image <work> folder ● INHERIT += “rm_work” cleans up each <work> folder upon successful build
  16. 16. OpenEmbedded and Linaro
  17. 17. 17 Background ● Originally we started using OE to bootstrap Aarch64 support ● Also used by Toolchain WG to validate compiler ● Extended interest by networking WG and others ● Integrated to our CI loop
  18. 18. 18 Quickstart ● Git clone git:// ● $ sudo bash ● $ linaro-image-minimal ● Have a nice cup ● of coffee!
  19. 19. 19 Show me the code ● ● Setup and build script as used by integration ● ● Repo manifest, describes used git repositores ● ● Contains Linaro, toolchain and aarch64 layers ● Branches for head and stable (dylan) ● Poky support being worked on
  20. 20. 20 Continuous integration ● Jenkins: builds – LAVA runtime testing ● ● Building the minimal, lamp and Java Images ● definitions.git;a=tree;f=openembedded ● Test definitions for LAVA ● reports/minimal-armv8 ● Test results in LAVA ● Similar pages for LAMP and JAVA images
  21. 21. 21 Future plans ● Switching to Poky/Yocto or stable OE ● Changes in OE head disruptive ● Repo manifest tagging for releases ● For reproducibility ● CI integration of ARMv7 and big endian ● Bitbake world -k for toolchain validation ● Try to keep things simple instead of supporting every possible usecase.
  22. 22. 22 Resources ● ● ● ● ● ● leaders-can-use-yocto-project-solve ● consumer-electronics-linux-forum-merge ● integrated-we-want-best-both-worlds
  23. 23. 23 Thank you!
  24. 24. 24 OpenEmbedded/Yocto 'lab'
  25. 25. 25 Demystify OE/Yocto build through a practical exercise ● Build a minimal console image for qemuarm ● Run the image in QEMU ● Explore and understand the OE build directory structure ● Look at some basic OE developers tasks ● Build/run an application with SDK
  26. 26. 26 Practical information ● Source and build cache ● “repo” tool ● Sample of recipe ● Lab instructions are here: ● Part1: ● Part2: ● Part3: ● Ask questions, we are here to answer them!
  27. 27. 27 Interesting things to do... ● Explore configuration files ● Checkout some recipes ● Explore 'WORKDIR' and 'deploy' folders (image and package feed) ● Customize your image ● Add a new recipe, create your own layer ● Exercise with .bbappend ● Create a patch, and apply in recipe ● Use/understand bitbake 'tasks' and experiment typical developer workflow (build, debug, edit, build, debug, ...)
  28. 28. 28 Sources mirror, build cache ● OpenEmbedded 'trees' contain no source code ● By design, everything is built from scratch. ● Source code for all recipes is fetched during the build ● 'fetchall' target let you download all source first ● PREMIRROR let you configure OE to use an alternative mirror before going to the Internet (can be local/intranet) ● OE implements “shared state code” (aka sstate) to support incremental builds ● OE uses hash/signature on each 'task' input variables to decide if task needs to be rebuilt. ● Prebuilt components are stored in “sstate” cache, which can be local or http, and can be shared across multiple users. ● Even after deleting the entire working folder, a rebuild is 'instantaneous' ● We prepared 'tarballs' for 'download' and 'sstate' to make this workshop more efficient, hopefully you've downloaded everything already ● ● Otherwise we have a couple of HDDs
  29. 29. 29 “repo” tool ● Implemented initially by the Android team to manage Android source repositories ● Open source tool: ● Wrapper on top of git ● Manage 'forest' of git trees ● Support multiple 'product' branches and easy switch from one checkout to another ● Trees metadata stored in 'XML manifest' file ● Manifest can checkout: ● Latest commit on branch (development) ● Fix commit/tag (release) ● All Linaro OE projects are moving to “repo” ● Repo can (optionally) be used with Gerrit code review
  30. 30. 30 Manifest sample <?xml version="1.0" encoding="UTF-8"?> <manifest> <remote name="linaro" fetch="git://" /> <remote name="oe" fetch="git://" /> <remote name="beagleboard" fetch="git://" /> <default remote="oe" revision="master" sync-j="2" /> <project name="openembedded-core" /> <project name="meta-openembedded" /> <project path="openembedded-core/bitbake" name="bitbake" /> <project path="meta-linaro" name="openembedded/meta-linaro" remote="linaro" /> <project name="meta-beagleboard" remote="beagleboard"/> </manifest>
  31. 31. 31 How to use “repo” # install repo wrapper (once on each machine) $ curl > /tmp/repo $ chmod a+x /tmp/repo $ sudo mv /tmp/repo /usr/local/bin/ # initialize the repo, and download the manifest $ mkdir <mydir> && cd <mydir> $ repo init -u <manifest.git> -b <branch> # download and checkout all projects $ repo sync # build, edit, commit, debug, … # update my working copy $ repo sync # switch to another branch/product $ repo init -b <my other branch> $ repo sync commands: list status info diff grep
  32. 32. 32 Hello World! DESCRIPTION = "Simple helloworld application" SECTION = "examples" LICENSE = "MIT" LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302" PR = "r0" SRC_URI = "file://helloworld.c" S = "${WORKDIR}" do_compile() { ${CC} helloworld.c -o helloworld } do_install() { install -d ${D}${bindir} install -m 0755 helloworld ${D}${bindir} } Packages: helloworld helloworld-dev helloworld-dbg
  33. 33. 33 smartmontools # meta-oe/recipes-extended/smartmontools/ SECTION = "console/utils" DESCRIPTION = "Control and monitor storage systems using S.M.A.R.T." HOMEPAGE = "" LICENSE = "GPLv2" LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263" SRC_URI = "${SOURCEFORGE_MIRROR}/smartmontools/smartmontools-${PV}.tar.gz" SRC_URI[md5sum] = "83a3a681f8183ed858392d550ae1cca6" SRC_URI[sha256sum] = "a9003b8bccc82682f658ce76d70edb1842411e51dc56d4cd6b56618da1d9ce07" inherit autotools
  34. 34. 34 Using layer to alter recipes $ more meta-linaro/recipes-core/busybox/busybox_1.20.2.bbappend SRC_URI += "file://linaro.cfg" $ more meta-aarch64/recipes-sato/webkit/webkit-gtk_1.8.3.bbappend SRC_URI_append = "file://aarch64.patch” ● “<packagename>_<version>.bbappend” files can be used in layers ● Any variable from the original recipe can be modified ● Very neat for patching! Avoid duplication, fragmentation. ● “bitbake-layers show-appends”
  35. 35. 35 Backup slides
  36. 36. 36 Tuning compiler ● CPU compiler options, FPU and ABI can be fully tuned from the machine configuration file ● Each 'architecture' gets a set of pre-defined 'tunes' ● For ARM : VFP vs neon, Thumb vs ARM instructions, Little vs big endian, Calling conventions (hardfp vs softfp, …) TUNEVALID[neon] = "Enable Neon SIMD accelerator unit." TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "neon", " -mfpu=neon", "" ,d)}" ARMPKGSFX_FPU .= "${@bb.utils.contains("TUNE_FEATURES", "neon", "-neon", "" ,d)}" TUNEVALID[vfp] = "Enable Vector Floating Point (vfp) unit." ARMPKGSFX_FPU .= "${@bb.utils.contains("TUNE_FEATURES", "vfp", "-vfp", "" ,d)}" TUNEVALID[callconvention-hard] = "Enable EABI hard float call convention, requires VFP." TUNE_CCARGS .= "${@bb.utils.contains("TUNE_FEATURES", "vfp", bb.utils.contains("TUNE_FEATURES", "callconvention-hard", " -mfloat-abi=hard", " -mfloat-abi=softfp", d), "" ,d)}" ARMPKGSFX_EABI .= "${@bb.utils.contains("TUNE_FEATURES", [ "callconvention-hard", "vfp" ], "hf", "", d)}"
  37. 37. 37 Layer configuration file # LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf # changes incompatibly LCONF_VERSION = "5" BBPATH = "${TOPDIR}" BBFILES ?= "" BBLAYERS ?= " ${TOPDIR}/openembedded-core/meta ${TOPDIR}/meta-openembedded/toolchain-layer ${TOPDIR}/meta-openembedded/meta-multimedia ${TOPDIR}/meta-openembedded/meta-oe ${TOPDIR}/meta-linaro/meta-linaro ${TOPDIR}/meta-linaro/meta-linaro-toolchain ${TOPDIR}/meta-qt5 " BBLAYERS_NON_REMOVABLE ?= " ${TOPDIR}/openembedded-core/meta "
  38. 38. 38 Machine configuration file #@TYPE: Machine #@NAME: common_pc #@DESCRIPTION: Machine configuration for running a common x86 PREFERRED_PROVIDER_virtual/xserver ?= "xserver-xorg" PREFERRED_PROVIDER_virtual/libgl ?= "mesa" PREFERRED_PROVIDER_virtual/libgles1 ?= "mesa" PREFERRED_PROVIDER_virtual/libgles2 ?= "mesa" require conf/machine/include/ require conf/machine/include/ KERNEL_IMAGETYPE = "bzImage" SERIAL_CONSOLE = "115200 ttyS0" XSERVER = "xserver-xorg mesa-driver-swrast xf86-input-vmmouse xf86-input-keyboard xf86-input-evdev xf86-video-vmware" MACHINE_FEATURES += "x86" MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d" qemux86.conf
  39. 39. 39 User local configuration file # Parallelism Options #BB_NUMBER_THREADS = "4" #PARALLEL_MAKE = "-j 4" MACHINE = "qemuarm" DISTRO = “poky” PACKAGE_CLASSES ?= "package_ipk" # "dbg-pkgs" - add -dbg packages for all installed packages # (adds symbol information for debugging/profiling) # "tools-testapps" - add useful testing tools (ts_print, aplay, arecord etc.) # "debug-tweaks" - make an image suitable for development # e.g. ssh root access has a blank password EXTRA_IMAGE_FEATURES = "debug-tweaks" # - 'buildstats' collect build statistics # - 'image-prelink' in order to prelink the filesystem image USER_CLASSES ?= "buildstats image-prelink" #SSTATE_MIRRORS ?= " #file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH n #file://.* file:///some/local/dir/sstate/PATH"
  40. 40. 40 Yocto Project Branding/Compliance ● Compliance controls the usage of the Yocto project name, logo and brand. ● Yocto Project Participants ● Open source project, non profit, small business ● Committed to promoting and contributing to Yocto Projects (oe-core, bitbake, yocto layers) ● Yocto Project Compatible ● Products, BSP, layers maintained by an open source, non profit, or Yocto Project member organisation ● Compatible with a specific release of Yocto ● Contribute all Yocto Projects patches upstream ● All Yocto recommendations/processes implemented