FOSS License Management:
Good practice from
aliens4friends in Eclipse Oniro
OpenChain Webinar #58
Alberto Pianon and Carlo Piana (ARRAY)
Anti-Trust Policy Notice
● Linux Foundation meetings involve participation by industry competitors, and it is the
intention of the Linux Foundation to conduct all of its activities in accordance with
applicable antitrust and competition laws. It is therefore extremely important that
attendees adhere to meeting agendas, and be aware of, and not participate in, any
activities that are prohibited under applicable US state, federal or foreign antitrust and
competition laws.
● Examples of types of actions that are prohibited at Linux Foundation meetings and in
connection with Linux Foundation activities are described in the Linux Foundation
Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. If you have
questions about these matters, please contact your company counsel, or if you are a
member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of
Gesmer Updegrove LLP, which provides legal counsel to the Linux Foundation.
What and Who
What and Who: Oniro
- Oniro: an open source operating system platform aimed at connecting a wide range of
smart devices
- Initially developed by Huawei, then donated to the Eclipse Foundation
- OpenHarmony: Oniro’s twin project, based in China, donated by Huawei to the
OpenAtom Foundation
- A historical collaboration between the two open source foundations:
common specifications and interoperability
What and Who: Aliens4friends
- Aliens4friends / Eclipse Oniro Compliance Toolchain is an Eclipse
project, led by Array and Noi Techpark within the Eclipse Oniro WG
- Array: OpenChain partner, law firm specialized in IT Law and Open
Source; Carlo Piana (founding partner) is Chairperson of OSI
- NOI Techpark: the science and technology park of South Tirol (Italy); it
hosts research institutes such as Fraunhofer and Eurac, University
Faculties, scientific laboratories, companies and startups.
What and Who: Context
SCA and OSS compliance in embedded Linux OS may be hard:
- many third party components, but no package manager / no pre-packaged software:
only customized recipes to build the components depending on the target machine
- only approximate license metadata provided by build systems, especially for complex
software components
- Yocto: flexibility and complexity of the build system
- hardware support: proprietary licenses and patented technologies
A Little Bit of History
2020
Initial status:
- Development of an operating system platform for connecting big and small
devices, fully open source and vendor-neutral: Oniro
- need to draft a policy for OSS compliance → OpenChain
- upstream first approach
- ease OSS compliance for downstream adopters
2021-2022
Towards aliens4friends:
- We have a policy, but how do we implement it? We need tools!
- Tools return too many false positives and false negatives: we need human review
(Fossology)
- But it’s too much work! How can we handle it? The open source way → reuse others’
work
- Debian, a trusted friend that vouches for “alien” (third-party) OSS software
10/2022 - Oniro 2.0 (1)
OSS compliance for Oniro 2.0:
- Aliens4friends (tooling):
- integration with Yocto (metadata and upstream sources),
- Integration with Fossology and Scancode (license scan and review)
- Debian matcher and reuser (using Fossology API)
- Process design (tools + human work) and parallel (async) CI pipelines → continuous
compliance
- dedicated dashboard to monitor audit progress and analyze results
- audit guidelines for human validation → consistency, transparency → reusability
10/2022 - Oniro 2.0 (2)
OSS compliance for Oniro 2.0:
final output:
- fixed issues in the Oniro project (by removing offending components)
- fixed issues in third-party components, upstream (by removing offending files or fixing
license conditions or wrong license references)
- reported outstanding issues to downstream users to enable the latter to handle them
2023
Moving forward:
- Upstreaming metadata collection logic to Yocto: added Unpack Tracer API (accepted), meta-
bbtracer (WIP)
- Improving: automatically resolve binary file licenses and file-level license incompatibilities by
mapping binary files to source files (PoC)
- Scaling out: implement a4f CI pipelines in other operating system projects
- full implementation on Eclipse Leda (OS for SDV), demo on Linaro TRS
- In the meantime, Oniro and OpenHarmony are more and more converging →
Oniro4OpenHarmony
2024-...
Next targets:
- integrate other tools (ORT, SW360)
- make a4f fully build-system-independent (support Yocto, Oniro4OpenHarmony, possibly
Android and Buildroot)
- transform modules into independent tools to be reused by other projects
- automatically resolve binary file licenses and file-level license incompatibilities by mapping
binary files to source files through a graph database
Let’s Dive Into It
Key Points: Reuse
Automation is key, but human review of automated scan results is key, too,
especially in the embedded Linux field (no package manager, etc.)
🔻
human review is costly and must be made sustainable
🔻
To make human review more sustainable, we should be able to reuse others’ work,
and others should be able to reuse ours
🔻
reuse works well both ways only if certain conditions are met:
1) we are all reviewing the same thing (original upstream sources)
2) we can trust each other’s work (process transparency, documented audit criteria)
3) we work upstream every time it is possible
Key Points: Continuous Compliance
Automation is key, but human review of automated scan results is key, too,
especially in the embedded Linux field (no package manager, etc.)
🔻
human review requires substantial time
🔻
If we do that only before each release, it turns into a bottleneck
🔻
it should be a continuous process, flowing in parallel with the development process (continuous
compliance)
🔻
parallel (async) CI pipelines, monitoring progress and results
Workflow Overview
Workflow… in Action (CI Pipelines)
1. Get Original Upstream Sources: the Issue
- In Yocto, components are built from
customizable recipes
- To build a component, a recipe can fetch
and unpack multiple upstream source
packages of different types (tarball
archives, git repos, npm packages,
rust/crate packages, etc.) and add
downstream patches, too
- Yocto archives mixed unpacked sources, as
found in recipe’s workdir; SPDX data
represent such “mixed” archive, not the
original source packages
1. Get Original Upstream Sources: the Solution
current (downstream) solution:
- TinfoilHat collects the component
metadata through Yocto/bitbake libraries;
- aliensrc_creator collects original source
packages from bitbake’s download cache
target (upstream) solution:
- Yocto/bitbake exposes an UnpackTracer
API (patch has already been accepted
upstream and is part of latest Yocto
release)
- a Yocto layer collects metadata on original
upstream sources using the UnpackTracer
API (meta-bbtracer, currently WIP)
2. Reusing Others’ Work: Debian Matcher
- Find not only exact matches but also close ones
- Similarity should be assessed based on copyright and license headers → we use Scancode for that
- Partial reuse in case of partial similarity, based on certain thresholds
- Two APIs are available from Debian: current repositories (fast response but variable data over time, no
reproducibility), and snapshot (full historical data, reproducibility, but slow response and subject to API
request limits)
- Future plans: it might be transformed into an independent tool
3. Human Audit Activity
- The audit process flows in parallel with the development process
- async: new and modified components are uploaded to Fossology, but final audit
results on such components will be available only at a later point
- Audit’s current status in Fossology is collected, to monitor progress
- Transparent process and documented audit guidelines
4. Harvesting Data
5. Dashboard (1)
5. Dashboard (2)
Conclusions
Key Principles
- Automation with (sustainable) human review
- Reuse, both ways
- Upstream first
- Continuous Compliance
Key Features
- Getting original upstream sources from Yocto
- Reusing metadata from Debian
- Human audit process monitoring (CI pipelines, Dashboard)
Q&A, feel free to ask
Thanks!
Array: https://array.eu
Toolchain: https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain
Presentation content: © 2024 Alberto Pianon <pianon@array.eu> and Carlo Piana <piana@array.eu>
licensed under CC-BY-SA 4.0
OpenChain Project Meeting and Presentation Template licensed under CC-0 1.0. The OpenChain Project Templates contain the OpenChain trademark and can only be used for
matters related to OpenChain Project activities. The templates also contains The Linux Foundation trademarked logo. The Linux Foundation trademark policy can be found
here: https://www.linuxfoundation.org/legal/trademark-usage
To use the OpenChain trademark for commercial activities please join the OpenChain Partner Program: https://www.openchainproject.org/partners

OpenChain Webinar #58 - FOSS License Management through aliens4friends in Eclipse Oniro

  • 1.
    FOSS License Management: Goodpractice from aliens4friends in Eclipse Oniro OpenChain Webinar #58 Alberto Pianon and Carlo Piana (ARRAY)
  • 2.
    Anti-Trust Policy Notice ●Linux Foundation meetings involve participation by industry competitors, and it is the intention of the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. ● Examples of types of actions that are prohibited at Linux Foundation meetings and in connection with Linux Foundation activities are described in the Linux Foundation Antitrust Policy available at http://www.linuxfoundation.org/antitrust-policy. If you have questions about these matters, please contact your company counsel, or if you are a member of the Linux Foundation, feel free to contact Andrew Updegrove of the firm of Gesmer Updegrove LLP, which provides legal counsel to the Linux Foundation.
  • 3.
  • 4.
    What and Who:Oniro - Oniro: an open source operating system platform aimed at connecting a wide range of smart devices - Initially developed by Huawei, then donated to the Eclipse Foundation - OpenHarmony: Oniro’s twin project, based in China, donated by Huawei to the OpenAtom Foundation - A historical collaboration between the two open source foundations: common specifications and interoperability
  • 5.
    What and Who:Aliens4friends - Aliens4friends / Eclipse Oniro Compliance Toolchain is an Eclipse project, led by Array and Noi Techpark within the Eclipse Oniro WG - Array: OpenChain partner, law firm specialized in IT Law and Open Source; Carlo Piana (founding partner) is Chairperson of OSI - NOI Techpark: the science and technology park of South Tirol (Italy); it hosts research institutes such as Fraunhofer and Eurac, University Faculties, scientific laboratories, companies and startups.
  • 6.
    What and Who:Context SCA and OSS compliance in embedded Linux OS may be hard: - many third party components, but no package manager / no pre-packaged software: only customized recipes to build the components depending on the target machine - only approximate license metadata provided by build systems, especially for complex software components - Yocto: flexibility and complexity of the build system - hardware support: proprietary licenses and patented technologies
  • 7.
    A Little Bitof History
  • 8.
    2020 Initial status: - Developmentof an operating system platform for connecting big and small devices, fully open source and vendor-neutral: Oniro - need to draft a policy for OSS compliance → OpenChain - upstream first approach - ease OSS compliance for downstream adopters
  • 9.
    2021-2022 Towards aliens4friends: - Wehave a policy, but how do we implement it? We need tools! - Tools return too many false positives and false negatives: we need human review (Fossology) - But it’s too much work! How can we handle it? The open source way → reuse others’ work - Debian, a trusted friend that vouches for “alien” (third-party) OSS software
  • 10.
    10/2022 - Oniro2.0 (1) OSS compliance for Oniro 2.0: - Aliens4friends (tooling): - integration with Yocto (metadata and upstream sources), - Integration with Fossology and Scancode (license scan and review) - Debian matcher and reuser (using Fossology API) - Process design (tools + human work) and parallel (async) CI pipelines → continuous compliance - dedicated dashboard to monitor audit progress and analyze results - audit guidelines for human validation → consistency, transparency → reusability
  • 11.
    10/2022 - Oniro2.0 (2) OSS compliance for Oniro 2.0: final output: - fixed issues in the Oniro project (by removing offending components) - fixed issues in third-party components, upstream (by removing offending files or fixing license conditions or wrong license references) - reported outstanding issues to downstream users to enable the latter to handle them
  • 12.
    2023 Moving forward: - Upstreamingmetadata collection logic to Yocto: added Unpack Tracer API (accepted), meta- bbtracer (WIP) - Improving: automatically resolve binary file licenses and file-level license incompatibilities by mapping binary files to source files (PoC) - Scaling out: implement a4f CI pipelines in other operating system projects - full implementation on Eclipse Leda (OS for SDV), demo on Linaro TRS - In the meantime, Oniro and OpenHarmony are more and more converging → Oniro4OpenHarmony
  • 13.
    2024-... Next targets: - integrateother tools (ORT, SW360) - make a4f fully build-system-independent (support Yocto, Oniro4OpenHarmony, possibly Android and Buildroot) - transform modules into independent tools to be reused by other projects - automatically resolve binary file licenses and file-level license incompatibilities by mapping binary files to source files through a graph database
  • 14.
  • 15.
    Key Points: Reuse Automationis key, but human review of automated scan results is key, too, especially in the embedded Linux field (no package manager, etc.) 🔻 human review is costly and must be made sustainable 🔻 To make human review more sustainable, we should be able to reuse others’ work, and others should be able to reuse ours 🔻 reuse works well both ways only if certain conditions are met: 1) we are all reviewing the same thing (original upstream sources) 2) we can trust each other’s work (process transparency, documented audit criteria) 3) we work upstream every time it is possible
  • 16.
    Key Points: ContinuousCompliance Automation is key, but human review of automated scan results is key, too, especially in the embedded Linux field (no package manager, etc.) 🔻 human review requires substantial time 🔻 If we do that only before each release, it turns into a bottleneck 🔻 it should be a continuous process, flowing in parallel with the development process (continuous compliance) 🔻 parallel (async) CI pipelines, monitoring progress and results
  • 17.
  • 18.
    Workflow… in Action(CI Pipelines)
  • 19.
    1. Get OriginalUpstream Sources: the Issue - In Yocto, components are built from customizable recipes - To build a component, a recipe can fetch and unpack multiple upstream source packages of different types (tarball archives, git repos, npm packages, rust/crate packages, etc.) and add downstream patches, too - Yocto archives mixed unpacked sources, as found in recipe’s workdir; SPDX data represent such “mixed” archive, not the original source packages
  • 20.
    1. Get OriginalUpstream Sources: the Solution current (downstream) solution: - TinfoilHat collects the component metadata through Yocto/bitbake libraries; - aliensrc_creator collects original source packages from bitbake’s download cache target (upstream) solution: - Yocto/bitbake exposes an UnpackTracer API (patch has already been accepted upstream and is part of latest Yocto release) - a Yocto layer collects metadata on original upstream sources using the UnpackTracer API (meta-bbtracer, currently WIP)
  • 21.
    2. Reusing Others’Work: Debian Matcher - Find not only exact matches but also close ones - Similarity should be assessed based on copyright and license headers → we use Scancode for that - Partial reuse in case of partial similarity, based on certain thresholds - Two APIs are available from Debian: current repositories (fast response but variable data over time, no reproducibility), and snapshot (full historical data, reproducibility, but slow response and subject to API request limits) - Future plans: it might be transformed into an independent tool
  • 22.
    3. Human AuditActivity - The audit process flows in parallel with the development process - async: new and modified components are uploaded to Fossology, but final audit results on such components will be available only at a later point - Audit’s current status in Fossology is collected, to monitor progress - Transparent process and documented audit guidelines
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
    Key Principles - Automationwith (sustainable) human review - Reuse, both ways - Upstream first - Continuous Compliance
  • 28.
    Key Features - Gettingoriginal upstream sources from Yocto - Reusing metadata from Debian - Human audit process monitoring (CI pipelines, Dashboard)
  • 29.
  • 30.
    Thanks! Array: https://array.eu Toolchain: https://gitlab.eclipse.org/eclipse/oniro-compliancetoolchain Presentationcontent: © 2024 Alberto Pianon <pianon@array.eu> and Carlo Piana <piana@array.eu> licensed under CC-BY-SA 4.0 OpenChain Project Meeting and Presentation Template licensed under CC-0 1.0. The OpenChain Project Templates contain the OpenChain trademark and can only be used for matters related to OpenChain Project activities. The templates also contains The Linux Foundation trademarked logo. The Linux Foundation trademark policy can be found here: https://www.linuxfoundation.org/legal/trademark-usage To use the OpenChain trademark for commercial activities please join the OpenChain Partner Program: https://www.openchainproject.org/partners