Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Embedded Recipes 2019 - LLVM / Clang integration

138 views

Published on

Buildroot is a popular and easy to use embedded Linux build system. It generates, in few minutes, lightweight and customized Linux systems, including the cross-compilation toolchain, kernel and bootloader images, as well as a wide variety of userspace libraries and programs.

This talk is about the integration of LLVM/clang into Buildroot.

In 2018, Valentin Korenblit, supervised by Romain Naour, worked on this topic during his internship at Smile ECS. After a short introduction about llvm/clang and Buildroot, this talk will go through the numerous issues discovered while adding llvm/clang componants and how these issues were fixed. Romain will also detail the work in progress and the work to be done based on llvm/clang libraries (OpenCL, Compiler-rt, BCC. Chromium, ldd).
Romain Naour

Published in: Software
  • Be the first to comment

  • Be the first to like this

Embedded Recipes 2019 - LLVM / Clang integration

  1. 1. LLVM/Clang integration to Buildroot Romain Naour romain.naour@smile.fr September 22, 2019 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 1/35
  2. 2. Overview 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 2/35
  3. 3. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 3/35
  4. 4. Disclaimer Purpose of the talk is to present the current state of llvm/clang integration into Buildroot. I am not a llvm/clang developer. Not pretending to know everything about llvm/clang. Experience gained from experimenting llvm/clang in the context of Buildroot. I am only interested in its integration in Buildroot to enable new features. Focused on AArch64 and x86_64 target. www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 4/35
  5. 5. Buildroot at a glance From ELCE 2017 "Buildroot: What’s New?"1 Is an embedded Linux build system, builds from source: cross-compilation toolchain root filesystem with many libraries/applications, cross-built kernel and bootloader images More than 2500 packages available Generates filesystem images, not a distribution 1 https://www.youtube.com/watch?v=D6zO4nMX9KY www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 5/35
  6. 6. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 6/35
  7. 7. LLVM Open source project started in 2000. LLVM 1.0 released in October 2003 Several subprojects: LLVM Core, Clang, lldb, compiler-rt, libclc, lld Provides a compiler infrastructure written in C++11 (C++14) Designed as an API from the beginning Focusing on compile time and performance of the generated code Well structured and documented Some existing backends: ARM, ARM64, Hexagon, Mips, Mipsel, NVIDIA PTX 32/64, PowerPC 32/64, AMDGPU, Sparc, Thumb, x86, x86-64, XCore www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 7/35
  8. 8. LLVM - Three-phase approach www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 8/35
  9. 9. LLVM - Internal aspects Intermediate Representation (IR) Mostly architecture-independent instruction set (RISC) Strongly typed Unlimited number of virtual registers in SSA IR Code example: define i32 @main() #0 { entry: %retval = alloca i32, align 4 %c = alloca i32, align 4 store i32 0, i32* %retval, align 4 %0 = load i32, i32* @a, align 4 %1 = load i32, i32* @b, align 4 %add = add nsw i32 %0, %1 store i32 %add, i32* %c, align 4 ret i32 0 } www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 9/35
  10. 10. Clang Frontend C/C++, Objective C/C++ and OpenCL C for LLVM Clear and concise diagnostics (error and warning messages) Natively a cross-compiler: -target <triple> Sanitizers Goals Designed to be highly compatible with GCC C++11 supported since Clang 3.3 C++14 supported since Clang 3.4 C++17 supported since Clang 5 C++2a implementation started since Clang 6 Performance vs GCC ?2 3 2 http://www.phoronix.com/vr.php?view=25742 3 https://www.phoronix.com/vr.php?view=28232 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 10/35
  11. 11. Toolchain components Component LLVM GNU C/C++ Compiler clang gcc Assembler cc1as as Linker lld ld Runtime compiler-rt libgcc Debugger lldb gdb Unwinder libunwind libgcc_s C++ library libc++abi, libc++ libsupc++ libstdc++ Tools llvm-ar, llvm-as etc. ar, objdump, etc. C library - libc (Glibc, Musl) There is a proposal from Google about an LLVM libc4 . 4 https://lists.llvm.org/pipermail/llvm-dev/2019-June/133269.html www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 11/35
  12. 12. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 12/35
  13. 13. Objectives Initial integration by Valentin Korenblit (2018) Preliminary study of LLVM/Clang. LLVM/Clang integration to Buildroot for the target. llvmpipe for Mesa 3D. AMDGPU backend. OpenCL implementations, most of them rely on LLVM Enable OpenCL for AMD GPUs (Clover) and Broadcom Videocore IV (VC4CL). OpenCL support for already existing packages in Buildroot. Integration of new packages that can benefit from OpenCL: image processing (i.e. Darktable), simulation, cryptography. Integration of new packages that can benefit from libLLVM or libclang. Use Clang as a cross-compiler (2019/2020?). www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 13/35
  14. 14. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 14/35
  15. 15. Available hardware for testing . Platform 1 - x86_64 (HP ProBook) Processor: AMD A4-3300M Dual Core @ 1.9 GHz. GPU: AMD Radeon Dual Graphics (HD6480G + HD7450M). Ease testing. Platform 2 - ARM (Raspberry Pi 2 Model B). Processor: ARMv7 Cortex-A7 Quad Core @ 900 MHz. GPU: Broadcom Videocore IV. Platform 3 - ARM/AArch64 (Raspberry Pi 3 Model B). Processor: ARMv8 Cortex-A53 Quad Core @ 1.2 GHz. GPU: Broadcom Videocore IV. www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 15/35
  16. 16. LLVM packaging for Buildroot CMake-based project Buildroot provides a CMake infrastructure Plenty of options, some with misleading names LLVM_TARGETS_TO_BUILD LLVM_TARGET_ARCH LLVM_DEFAULT_TARGET_TRIPLE LLVM_HOST_TRIPLE Difficult to be cross-compiled At least llvm-tblgen must be compiled for the host first It requires a modern and fully-featured toolchain (C++11 -> C++14) Takes a lot of time to compile (host + target) www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 16/35
  17. 17. LLVM packaging for Buildroot LLVM 5.0.1 is the first packaged version Only LLVM libraries are needed (libLLVM.so), no tools The main problem: llvm-config not giving the desired output It’s a compiled program. Normally ”config” programs are scripts. llvm-config compiled for the host must be installed to the target’s sysroot. Some output depends on its location and some other is contained in the binary Solution: do a full installation of LLVM for the host using the same5 configuration options as for the target and link LLVM tools with libLLVM (needed for AMDGPU backend). 5 Tools are not built for the target www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 17/35
  18. 18. LLVM support for Mesa 3D drivers Gallium llvmpipe driver: software rasterizer that uses LLVM to do runtime code generation. OpenCL using Gallium R600 and RadeonSI Both use AMDGPU LLVM backend Some benchmarks (LLVM 5.0.1): Gallium Driver GLMark2 GLMark2-es2 R600 on HD6480 156 156 softpipe on AMD A4-3300M 3 3 softpipe on Cortex-A53 (32-bit) NS 0 softpipe on Cortex-A53 (64-bit) NS 0 llvmpipe on AMD A4-3300M 47 52 llvmpipe on Cortex-A53 (32-bit) NS 11 llvmpipe on Cortex-A53 (64-bit) NS 13 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 18/35
  19. 19. Selecting llvmpipe www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 19/35
  20. 20. Clang packaging for Buildroot LLVM and Clang should be version bumped together. It is normally built inside LLVM source tree (llvm/tools) but Buildroot uses per-package build directories. Some more tweaks are needed. Binaries, headers and some scripts must be removed from the target. Only libclang.so is necessary on the target. Buildroot does not provide a standard infrastructure to install a compiler on the target (you’re on your own). Clang, Clang++ binaries can’t be easily disabled from the build system. www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 20/35
  21. 21. OpenCL: Clover: Computing Language over Gallium Clover needs libclang and libclc. Valentin added Mesa3d OpenCL support. Added AMDGPU LLVM backend support, libclc and clinfo packages. Tested VC4CL for Broadcom Videocore IV. Runtime testing using Piglit. See Linuxembedded blog6 for more details on OpenCL. 6 http: //www.linuxembedded.fr/2018/07/llvmclang-integration-into-buildroot www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 21/35
  22. 22. LLVM/Clang in Buildroot - Current State Commited to Buildroot’s master: LLVM package (2018.05) LLVM support for Mesa 3D (2018.05) Clang package (2018.05) Libclc package (2018.11) libopencl, clinfo packages (2018.11) Piglit package (2019.08) Keep LLVM/Clang at the latest version Buildroot LTS 2019.02.x use LLVM/Clang 7.0.1 TODO & WIP Add OpenCL support for already existing packages (i.e opencv) Add new packages that depend on LLVM/Clang or OpenCL Use Clang as a cross-compiler www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 22/35
  23. 23. LLVM/Clang in Buildroot - Work in progress Chromium7 & lld8 by Joseph Kogut Chromium needs Clang as compiler. Lld used as GNU ld drop-in replacement. Compiler-rt & Libfuzzer9 by Matt Weber (Rockwell Collins) Compiler-rt to run the code with sanitizer instrumentation. Libfuzzer: a library for coverage-guided fuzz testing. BCC (BPF Compiler Collection)10 by Jugurtha Belkalem (Smile) BCC use libclang to build eBPF programs to do debugging11 . 7 https://github.com/jakogut/buildroot/commits/chromium-v7-wip 8 http://patchwork.ozlabs.org/project/buildroot/list/?series=121282 9 http://patchwork.ozlabs.org/project/buildroot/list/?series=105601 10 http://patchwork.ozlabs.org/project/buildroot/list/?series=106280 11 http://www.linuxembedded.fr/2019/05/bcc-integration-into-buildroot www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 23/35
  24. 24. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 24/35
  25. 25. Compiling a Linux system with Clang Until recently, Linux kernel expected to use some GCC behavior that are not supported by Clang: Variable Length Arrays inside structures (VLA removed since 4.20) Nested functions Explicit register variables LLVM assembler cannot be used to build the kernel Support for asm goto in LLVM & Clang for x86_6412 LLVMLinux Project: Kernel 4.4 and 4.9 built with Clang for x86_64 and ARM64 (patches applied)13 Dependency on GNU as and ld Glibc14 ? Musl15 ? 12 https://bugs.llvm.org/show_bug.cgi?id=9295 13 https://lwn.net/Articles/734071/ 14 https://sourceware.org/glibc/wiki/GlibcMeetsClang 15 https://wiki.musl-libc.org/supported-platforms.html www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 25/35
  26. 26. Compiling a Linux distribution with Clang 2019 EuroLLVM Developers’ Meeting: Switching a Linux distribution’s main toolchains to LLVM/Clang16 - Bernhard Rosenkränzer From the OpenMandriva Wiki17 "Over 98% of packages in our main repository are now built with LLVM/clang." 32757 Debian packages have been rebuild with Clang 8. Among them, 1322 (4 %) failed18 16 https://www.youtube.com/watch?v=QinoajSKQ1k 17 https://wiki.openmandriva.org/en/4.0/Release_Notes#LLVM.2Fclang 18 https://clang.debian.net/status.php?version=8svn www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 26/35
  27. 27. Cross-compiling with Clang with Buildroot In Buildroot we take advantage of all previous upstream work to build an embedded system with Clang. Objectives: Build vanilla Linux kernels. Build most of userspace applications. Select GCC or Clang globally in the defconfig. Limitations: Keep toolchain components and bootloaders build with GCC Rebuild LLVM/Clang for each defconfig build. We expects build and runtime issue due to cross-compiling with Clang. www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 27/35
  28. 28. Cross-compiling with Clang with Buildroot What’s remain to do to use Clang as a cross-compiler into Buildroot? Fix Clang’s libgcc and crt*.o search path to build userspace applications19 . Add a toolchain wrapper as for GCC to executes the real compiler with a number of arguments (sysroot) hardcoded. Improve the package infrastructure where GCC is explicitely used (generic packages, meson, cmake etc). Add an option to select Clang or GCC in the defconfig. Fixes most build issues on the long term. Still a lot of warning due to GCC compiler flags used with Clang. 19 http://patchwork.ozlabs.org/patch/1159304/ www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 28/35
  29. 29. Current issues with Clang Busybox (segfault with applets (i.e init))20 . Bootloader & firmware (uboot, barebox, grub2, ATF). uboot: use global register variables. grub2: inline asm constructs (.addrsig etc.) Python3 build issue with _ctype module. rng-tools: Outdated version (from 2004) using a VLA. Valgrind: __builtin_longjmp not supported in LLVM/Clang for Aarch64 target21 . 20 http://lists.busybox.net/pipermail/busybox/2019-June/087337.html 21 https://bugs.kde.org/show_bug.cgi?id=369723 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 29/35
  30. 30. Qemu runtime testing: AArch64 & x86_64 qemu_aarch64_virt_defconfig qemu_x86_64_defconfig Using the latest kernel version (5.2.7) Using LLVM/Clang 9.0.0rc3 for x86_64 (asm-goto support) Welcome to Buildroot # uname −a Linux buildroot 5.2.7 #1 SMP Thu Sep 5 17:33:00 CEST 2019 aarch64 GNU/Linux # dmesg | head −n 2 Booting Linux on physical CPU 0x0000000000 [0x410fd034] Linux version 5.2.7 (clang version 8.0.1 (tags/RELEASE_801/final)) #1 SMP Thu Sep 5 17:33:00 CEST # cat /etc/os−release NAME=Buildroot VERSION=2019.11−git−00175−g0cc6fbbdf5 ID=buildroot VERSION_ID=2019.11−git PRETTY_NAME="Buildroot 2019.11−git" www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 30/35
  31. 31. Available hardware for testing . Aarch64 (Le Potato AML-S905X-CC). Processor: ARM Cortex-A53 Quad Core @ 1.512GHz. GPU: ARM Mali-450 @ 750MHz. Potato board received two years ago at Kernel recipes (Thanks!) Live Demo (kernel built with Clang) www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 31/35
  32. 32. Plan 1 Introduction 2 LLVM and Clang 3 Objectives 4 LLVM/Clang initial integration into Buildroot 5 Cross-compiling Linux with Clang 6 Conclusions www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 32/35
  33. 33. Conclusions LLVM/Clang is already available for the target. Clang can now build vanilla Linux Kernel. It’s possible to use Clang as cross-compiler to build a Linux embedded system. Initial patch series to review: "Add the support for Clang cross-compiler"22 Not yet ready for production, needs more build and runtime testing. Compared to Debian and OpenMandriva, Buildroot only have 2500+ packages. Cross-compilation with clang remains complicated. Many projects don’t test building with anything but native GCC (x86 or x86_64). 22 http://patchwork.ozlabs.org/project/buildroot/list/?series=129565 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 33/35
  34. 34. Questions Thanks for your attention! Questions? Romain Naour romain.naour@smile.fr Slides under CC-BY-SA 3.0 www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 34/35
  35. 35. References http://www.linuxembedded.fr/2018/07/ llvmclang-integration-into-buildroot http://www.linuxembedded.fr/2019/08/ my-first-linux-kernel-built-with-clang-compiler https://www.lowrisc.org/blog/2019/07/ large-scale-risc-v-llvm-testing-with-buildroot https: //www.phoronix.com/scan.php?page=news_item&px=LLVM-Asm-Goto-Merged https://www.youtube.com/watch?v=QinoajSKQ1k https://clang.debian.net/status.php?version=8svn https://bootlin.com/pub/conferences/2016/elce/ petazzoni-toolchain-anatomy www.smile.fr – Licence Creative Commons (CC BY-SA 3.0 FR) – 35/35

×