Linaro has enabled server class workloads for ARM servers by optimizing key open source software. They have contributed patches to projects like the Linux kernel, KVM, Xen, OpenJDK, Hadoop, and OpenStack. This has allowed OpenStack to run on ARMv8 hardware, with all applicable Tempest tests passing. Linaro is also working on optimizations for server workloads like the LAMP stack, HDFS, and HipHop JIT. Their efforts are helping to accelerate ARM's adoption in the server market.

1. Enabling Server Class Workloads
for ARM Servers
11th November 2014 - Andrea Gallo

2. ARM Progress in Servers
ARM-based servers on track
▪ 16 licenses now signed for server applications
▪ 2013: Chip development
▪ 2014: Software optimization accelerates
▪ 2015: First production systems
▪ 2018: ARM expects 10-15% share
of total server market
Data Center Workloads are Changing
Today Next 3 Years 5 Years +
Large
discrete
tasks

3. Linaro Overview
● Linaro is a collaborative software
engineering company funded by
members
● Instead of duplicating effort for
common open source software,
competitors share development
costs and software is built once
● The work is carried out in the open,
tested and then upstreamed into the
relevant open source projects:
kernel.org, Gnu, Fedora, Ubuntu,
OpenJDK etc.

4. Linaro Organization
● Founded 2010
● Now 30 members
● $50M annual revenue
● >200 OSS engineers
● Global footprint
Technical Steering Committee (TSC)
Office of the CTO (OCTO)
Enterprise Group (LEG)
16 companies
Platform Engineering
Builds & Baselines
QA Services
Systems
LAVA Lab
Field Engineering
Digital Home Group (LHG)
8 companies
Mobile Group (LMG)
10 companies
Networking Group (LNG)
12 companies
Core Development
Kernel
Power Management
Security
Virtualization
Product Technology
LAVA Software Linaro Stable Kernel (LSK) Toolchain
Member
Services
Club/Core
Services
Linaro Developer
Technical Support
(LDTS)

5. Confidential
CORE
CLUB
GROUP
COMMUNITY

6. Linux Kernel Upstreaming
● Linux 3.17 October 2014
● 12,353 total patch sets
in this release
● 180 known companies
contributed
● Linaro is #4 company
contributor; in top 5 for last
8 kernel versions (since 3.10)
Top Linux 3.17 Patchset Contributors by Company
1 Unknown/Hobbyists 3705 (30.00%)
2 Intel 1261 (10.21%)
3 Red Hat 980 (7.93%)
4 Samsung 491 (3.97%)
5 Linaro 412 (3.34%)
6 Vision Engraving & Routing 393 (3.18%)
7 Novell 326 (2.64%)
8 Texas Instruments 306 (2.48%)
9 Renesas Electronics 263 (2.13%)
10 Google 227 (1.84%)
Source: http://www.remword.com/kps_result/3.17_whole.html

7. Linaro Projects

8. Linaro Segment Groups
Digital Home - LHG Mobile - LMG Networking - LNG Enterprise - LEG
Confidential
● OSS for the digital home
● W3C EME Secure
Media playback for
RDK and Android
● Middleware and user-space
stack
● DRM, DLNA, CVP-2,
HTML5
● LSK kernel version for
STB/IPTV
● Common media
frameworks
● OSS for mobile devices
● Android 64 bit, “L”
● big.LITTLE power
management
● QEMU based Android
development
● 64-bit Chromium browser
● Support members,
ARM and Google
Android development
● OSS for networking
● Real Time Support
● Virtualization
● Core isolation
● OpenDataPlane (ODP)
● Big-endian legacy support
● ODP cross-platform
support for SoC
accelerators
● OSS for ARM servers
● UEFI/ACPI
● KVM/Xen
● ARMv8 optimization
● OpenJDK, Hadoop,
OpenStack
● Reduces fragmentation,
cost, accelerates time to
market

9. Linaro Enterprise Group (LEG)
● Formed in November 2012
● enable the ARM server ecosystem
● share costs of the enablement
engineering
● focus on standards, avoid
fragmentation
● identify, test and optimise
workloads and core libraries
http://wiki.linaro.org/LEG

10. Engineering (1): enablement
● All patches already upstream or under deep constructive review
● Strong engagement with all maintainers

11. Engineering (2): workloads

12. Engineering I
● LEG projects agreed and prioritized by LEG Steering Committee
● One representative per member, plus Linaro LEG Director
● ARM Boot Architecture
● ARM Server Base System Architecture; ACPI, UEFI, Grub; ARM Trusted firmware
● KVM and Xen Virtualization
● Stable ARM kernel version based on upstream
● LAMP Stack optimization
● HDFS CRC, LibTBB, CRC, Hugepages, OpenSSL
● Facebook HipHop JIT
● High performance OpenJDK for ARMv8 with C1 and C2 JIT

13. Engineering II
● Middleware and user-space stack testing
● Hadoop, OpenStack, Ceph
● Testing through multinode CI validation in LAVA
● ARMv8 Ubuntu, Fedora/Red Hat and OpenStack builds
● LEG also works with Linaro core working groups and can request
resources through the Linaro Technical Steering Committee (TSC)
● ARMv8 64 bit toolchain optimization
● Multi-core power management
● Security and Secure boot using Trustzone and
ARM Trusted Firmware

14. Achievements (1)
● UEFI
● Aligned ARM Tianocore to other architectures, boot as an EFI application, runtime
services, pass ACPI tables from firmware, support for SMBIOS 3.0
● Support GRUB and network boot
● Ported GNUEFI to build EFI apps easily
● Verified Tianocore Secure boot and ported the Shim layer to ARM
● Ported UEFI to EL1 as a guest in KVM/QEMU hypervisor (ongoing for Xen)
● Established a monthly UEFI release rebased on Tianocore EDK2 + all Linaro patches,
tested with UEFI SCT suite
● ACPI
● Ported ACPICA core kernel support and tools to ARM
● Ported key ARM core peripherals to ACPI
● Able to boot ARM FVP, Juno, AMD Seattle and Cavium Thunder-X model via ACPI
● Developed a PCC driver to support the new CPPC power management model
● Ported all reference test suites to ARM, e.g. FWTS, ACPI API test, ACPI ASL test, etc.
● Releasing an updated LEG ACPI kernel for every new weekly kernel 3.xx-rc

15. Achievements (2)
● OpenJDK
● Excellent cooperation with Red Hat on OpenJDK C1 and C2 JIT
● Driven by OpenJDK8, backported to OpenJDK7 and being forward ported to
OpenJDK9
● On par with x86 as JIT performance vs interpreted code
● Close to 10,000 tests executed every night: Mauve, JTREG, JCStress, SPECjbb2013
● Hadoop TeraSort and Jenkins as functional testing workloads
● Optimisations for the LAMP Webserver workload
● ARMv8 assembly tuning for OpenSSL, CRC32, etc. up to 5x-16x speed up
● Hugepages, fast_gup, etc.
● OpenStack
● Built OpenStack on ARMv8 with KVM/QEMU (ongoing for Xen)
● Running the Tempest test suite and the Rally benchmarking suite
● Functional test deploying multiple VM’s to run Java JTREG testing in parallel
● Early proof of concept with Docker and Go

16. it’s time for some cloud and server stuff now….

17. Platform as a Service (PaaS)
Software as a Service (SaaS)
Infrastructure as a Service (IaaS)

18. Mobility as a Service (MaaS)
Mobile Backend as a service (MBaaS)
Database as a Service (DBaaS)
Logging as a Service (LaaS)
Location as a service (LaaS)
Platform as a Service (PaaS)
Payments as a service (PaaS)
Software as a Service (SaaS)
Data as a Service (DaaS) Desktop as a Service (DaaS)
Infrastructure as a Service (IaaS)
Network as a service (NaaS) IT as a service (ITaaS)
Security as a service (SECaaS)
Storage as a service (STaaS)
Recovery as a service (RaaS)

19. Let’s start with one remote machine

20. Let’s add few more remote machines

21. It is not sustainable and not scaling

22. Let’s go back to one control point

23. Let’s add nodes now

24. We can now run multiple VM’s per node

25. We can run multiple runtimes per VM

26. And one or more apps per runtime

27. It’s all ...as a Service
SaaS
PaaS
IaaS

28. OpenStack logical architecture
http://docs.openstack.org/admin-guide-cloud/content/logical-architecture.
html

29. OpenStack conceptual architecture
http://www.solinea.com/blog/openstack-grizzly-architecture-revisited

30. OpenStack software diagram
http://www.openstack.org/themes/openstack/images/openstack-software-diagram.png

31. OpenStack modules
http://www.ca.com/us/lpg/ca-technology-exchange/sunny-outlook-for-openstack-clouds.aspx

32. It’s running on ARM 64-bit now!
http://www.ca.com/us/lpg/ca-technology-exchange/sunny-outlook-for-openstack-clouds.aspx

33. It’s running on ARM 64-bit now!
● Long cooperation with Canonical and Red Hat engineers
● Reusing pre-built DevStack packages from Ubuntu
● Leveraging on LEG kernel and Linaro KVM work
● Running on real ARM 64-bit hardware,
APM X-Gene and HP Moonshot
● Running all applicable Tempest tests
● Ongoing
● PackStack packages from Fedora
● libvirt running on XEN

34. OpenJDK on ARMv8
● AARCH64 port of OpenJDK
● Based on same codebase as x86 OpenJDK
● 95%+ code in common with Oracle JDK
● Same relative C2 vs interpreter JIT performance within a few % as Oracle JDK
● Goal is to track all x86 optimisations / features
● No support for ARMv7 or earlier or for 32 bit on ARMv8
● Three releases are supported: JDK7, JDK8 and JDK9

35. CI and nightly tests
Server results
from Aug 22
2014
ARM64 x86
Pass Fail Error Pass Fail Error
Hotspot 562 2 0 562 2 0
langtools 3004 0 11 2979 0 25
JDK 5283 181 46 5273 232 5
Totals 8849 183 57 8814 234 30

36. Hadoop Testing / Optimisation
● Daily native build of Hadoop from source
● Daily benchmark testing
● Client and server run of Terasort to test for performance regression
● Single node testing only.
● Looking at multinode testing
● Optimisation focused on optimising OpenJDK for Hadoop
● +24% improvement since FCS release in April
● Some optimisation on Hadoop directly
● CRC optimisation using SIMD
● Pushed to list but not yet adopted upstream

37. Demonstrated at LCU14:
OpenStack running OpenJDK JTREG testing with Jenkins
OpenStack
Cloud Controller
LXC 0..3
Compute Node 0
Compute Node 1

38. Demo Overview
● 3x APM Mustang systems running Ubuntu Trusty 14.04
● All services deployed with juju
● Mustang 01 as cloud controller and jenkins master with LXC
● Mustang 02 and 03 as OpenStack compute nodes with KVM
● Run a subset of the OpenJDK JTREG test regression harness
● jenkins dynamically provisions SIX VM’s instances via OpenStack on Mustang 2 and 3
● the VM’s are destroyed upon test completion

39. What’s the effort and cost to port OpenStack?
let’s review few open source projects
let’s rely on ohloh.net and the co.co.mo. model
http://csse.usc.edu/csse/research/COCOMOII/cocomo_main.html

40. OpenStack: lines of code

41. OpenStack: commits per month

42. Hadoop: lines of code

43. Hadoop: commits per month

44. Linux Kernel: lines of code

45. Linux Kernel: commits per month

46. Chrome: lines of code

47. Chrome: commits per month

48. Comparison
Project SLOC (M) Commits/month Start date
OpenStack 2 3992 2010*
OpenJDK 8 5 983 2007
Hadoop 2.5 219 2006
mongoDB 0.6 500 2007
Linux Kernel 17 6000 2002
Google Chrome 8 5000 2008
(*) 1st commit for OpenStack was in 2006 but really picked up in mid 2010
● OpenStack is still relatively small but growing as fast as the Linux
Kernel itself…

49. What about OpenStack on ARM then?

50. The Fonzarelli Fix
a.k.a. percussive maintenance
Tapping the server with a hammer: $0.00.
Knowing where to hit it: $5,000.00.
http://tvtropes.org/pmwiki/pmwiki.php/Main/PercussiveMaintenance

51. Patches available upstream
● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"
http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af
● libvirt: "AArch64: Add AArch64 architecture to list of valid arches."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=670c08afd461cba6164f7a8a643f5a5b41782e8d
● libvirt: "AArch64: Porting of armv7l conditons to run qemu for aarch64."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=27e32e0f3d4d5840cee7e361bdff94c68ae181e2
● libvirt: "AArch64: CPU Support for AArch64 (ARMv8 64bit)."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=5cb6816715968dc08486307bd1894dc91e22342c
● libvirt: "qemu: Add support for virt machine type with virtio-mmio devices on armv7"
http://libvirt.org/git/?p=libvirt.git;a=commit;h=c7ccd2c44be99b81ffce29374faeab3d1644a151

52. LibVirt patches
● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"
http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af
● libvirt: "AArch64: Add AArch64 architecture to list of valid arches."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=670c08afd461cba6164f7a8a643f5a5b41782e8d
● libvirt: "AArch64: Porting of armv7l conditons to run qemu for aarch64."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=27e32e0f3d4d5840cee7e361bdff94c68ae181e2
● libvirt: "AArch64: CPU Support for AArch64 (ARMv8 64bit)."
http://libvirt.org/git/?p=libvirt.git;a=commit;h=5cb6816715968dc08486307bd1894dc91e22342c
● libvirt: "qemu: Add support for virt machine type with virtio-mmio devices on armv7"
http://libvirt.org/git/?p=libvirt.git;a=commit;h=c7ccd2c44be99b81ffce29374faeab3d1644a151
Thanks to libvirt maintainer Cole Robinson at Red Hat
for his support!

53. OpenStack patches: only 114 lines in Nova
● nova: "Support setting a machine type to enable ARMv7/AArch64 guests to boot"
http://git.openstack.org/cgit/openstack/nova/commit/?id=15d4f8a3af1b4e105d91e6b66d7b19187fbd41af
Thanks to Canonical engineers
for their contribution!

54. Deployment
Ready
Software
Developer ARMv8 Server Cluster to Accelerate
Deployment-ready Software
Operating
System
Vendors
+ +
Open
Source
Community
Independent
Software
Vendors
www.linaro.org/leg/servercluster/

55. Summary
● OpenStack is at the heart of the cloud revolution
● All workloads run well on Linaro ARMv8 foundations
● Linaro fosters the open source collaboration on ARM

56. More about Linaro: www.linaro.org/about/
Linaro members: www.linaro.org/members