Drew Moseley is an embedded solutions architect at Mender.io, which provides an open source over-the-air update manager for embedded Linux devices. He discussed challenges with traditional update methods, requirements for secure and robust OTA updates, and how Mender addresses these challenges through its dual A/B partitioning, rollback capability, and support for full system and application updates across many embedded environments and device types. He then demonstrated Mender and discussed its architecture, community, and commercial support offerings.
2. 1. Challenges and motivations
2. Requirements
a. Non-functional
b. Installer strategies
3. Mender
a. Demo and explanations
b. Device integration
c. Testing
d. Community overview
Session overview
3. About me
Drew Moseley
○ 10 years in Embedded Linux/Yocto development.
○ Longer than that in general Embedded Software.
○ Project Lead and Solutions Architect.
drew.moseley@mender.io
https://twitter.com/drewmoseley
https://www.linkedin.com/in/drewmoseley/
https://twitter.com/mender_io
Mender.io
○ Over-the-air update manager for
embedded Linux
○ Open source (Apache License, v2)
○ System updates with dual A/B rootfs layout
(client)
○ Application updates (docker, deb, files, etc.)
○ Remote deployment management (server)
4. The Challenge without OTA updates
The update process was:
● Insecure: No authenticity checks
● Manually-intensive: No remote updates
Vehicles had to be updated by one of two ways:
● Bringing the car into the dealership
● Customers mailed a USB drive
A secure and robust OTA update manager for
the entire fleet can automate this
5. The Challenge with OTA updates
The endless reboot issue affected cars equipped
with the 2017 and 2018 versions of UConnect
Customers lost access to:
● Audio controls
● Heat and A/C
● Heated seats
● Rear-view cameras
Customers reported car batteries were drained
due to the endless reboot
No rollback option or sanity checking to ensure
the successful installation of an update
6. The Challenge with OTA updates
An open source community with more eyes
reduces the number of vulnerabilities
7. The embedded environment
● Remote
○ Expensive to reach physically
● Power
○ Battery
○ Unclean shutdown
● Hostile deployment environment
● Product lifetime
○ Some markets 5 - 10 years
● Network
○ Intermittent connectivity
○ Low bandwidth
○ Insecure
8. Requirements
1. Robust and secure
a. Rollback
b. Signed/trusted images
c. Integrity/compatibility checks
2. Atomic updates
3. Support kernel, apps, libraries, DTBs
4. Integrates with existing environments
a. Easy to get started
5. Bandwidth consumption
6. Downtime during update
10. 1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Bootloader
Kernel, initramfs
User space
Updater
Installer strategies
11. 1. In-place
2. Maintenance mode
3. Dual A/B
4. Proxy
Bootloader
Kernel,
initramfs A
User space A
Updater A
User space B
Kernel,
initramfs B
Updater B
Installer strategies
13. What is Mender?
Robust and secure end-to-end
over-the-air (OTA) software
update manager for embedded IoT devices
14. Mender Customers
Company and Team
● Company behind Mender.io (Northern.tech) was founded in 2008
● Extensive experience ensuring security of embedded systems at scale
● Dedicated to bringing a secure best-in-class OTA manager to world’s connected devices
● Highly technical team dedicated in solving customers’ problems in the IoT space
● Relationships with the world’s largest brands
● Offices in San Francisco, USA and Oslo, Norway
Who we are
Mender Partners
15. Mender is Open Source
Most companies write their own homegrown updater from scratch (!)
Existing free and open source tools can help you
Save time and avoid stress (where you can)
○ Initial development
■ It looks so easy! Is it easy?
○ Ongoing maintenance
■ 5 years? 10? 20?
■ 1 product? 5? 10?
16. Key Features and Benefits - System Updates
Atomic rollback support with a dual A/B partitioning layout
Allows recovery and keeping device operational when updates fail for any reason
Consistent full image atomic updates
Avoids inconsistencies from partial updates caused by interrupted package installations
Secure TLS client-server communication with code signing
Ensures security with end-to-end signing and verification of artifacts for
authenticity and integrity
Increased fleet robustness
Reproducible deployments means your fleet is running identical code to your test
systems. Reduced testing matrix due to package combinatorics.
17. Key Features and Benefits - Application Updates
IoT Device
type = deb
file = web1.2.deb
Management Server
Software
Operating System
Update Manager
Update Module: deb
Mender Client
Match package type to
correct Update Module
Application
Install
Application updates
Mender supports “all” kinds of updates
● Packages
● Containers
● Files and directories
● Proxy device deployment
● Full control and customizability
It is very easy to install application-based updates on Debian, Ubuntu
and Raspbian OSes with Mender
18. Challenges with Applications updates
Managing individual packages quickly
becomes a logistical nightmare
○ Testing of many combinations of
installed packages
○ Installation order not strictly
enforced.
○ Partially-installed packages can
block installs of new fixed
packages
20. Why Mender
Mender is the only end-to-end open source OTA manager with
both management server and client
○ No vendor lock-in
○ Permissive Apache 2.0 license
○ A healthy peer review mechanism with more eyes on the code to
increase its security
Less reliance on tribal knowledge; less expensive than a
homegrown system
○ Homegrown systems have unpredictable scope, which leads to more
cost and time
○ Reliance on tribal knowledge: if a key person leaves, could cause
serious issues
Mender Hub is the only online community dedicated to enable
OTA on any device and OS
○ Future-proof your product roadmap
21. Mender Devices
Users
API Gateway
TCP 443
DeviceAuth
UserAdm
Inventory
Deployments
GUI
Workflows
Storage
Proxy
TCP 9000
Minio (optional)
MongoDB
ElasticSearch
Redis
Filesystem
external clients stateless application layer persistent storage
● Microservices
● Only port 443 and 9000
● RESTful API
○ Device API
○ Management API
/api/management/v2/devauth
/api/management/v1/inventory
/api/management/v1/deployments
….
https://docs.mender.io/apis/overview
Mender Server Architecture
22. Managed mode vs standalone mode
Update types:
○ Full image, and delta of full image
○ Incremental (package, file, tarball, container)
○ Proxy
○ others
Partitions: Boot (optional), Rootfs A, Rootfs B, Data, others
Bootloader integration:
○ Controls boot process, A or B
○ Grub (BIOS/UEFI)
○ U-boot w/ auto patching
○ U-boot w/ manual patching
Linux runtime integration:
○ Supports eMMC/SD/block device & UBI volumes
○ Mender client application (as daemon for managed mode)
Target OSes supported:
○ Yocto/OpenEmbedded (OOB)
○ Debian, Raspbian, Ubuntu and other binary distros
○ Buildroot (somewhat)
○ OpenWRT (somewhat)
https://docs.mender.io/devices
Mender | System Integration Requirements
23. Automatic Rollback With A/B Image Update
Bootloader
Kernel
Persistent data
Application
Device
OS A (active) OS B (inactive)
Kernel
Application
24. Mender Architecture With Update Modules
● Any type of update
● Update actions implemented in
modules
Mender
Artifact
Mender
Client
2.x
File system
Image
File(s)
Containers
Update
Module
Proxy
firmware
OS A/B
image
● Popular application update types
supported out-of-the-box
30. Mender is built to scale
Designed for low
infrastructure load
default: update check
every 30 min per device
exponential backoff
separate storage service
for transferring updates
Microservices architecture
for horizontal scaling
application and storage layer
separation
scale individual services
native support for docker and
Kubernetes
Tested with more than
120,000 devices
3 machines, 2 CPUs, 8 GB RAM
demo available online
31. Open source and commercial offering
● Mender will always support the open source edition
● Follows the “open core” business model
● Basic features relating to adoption and basic usage will always be open source
○ E.g. Board support, OS support, end-to-end update process, client customizability
● Scalability, risk management and security features are commercial offerings
○ E.g. Phased rollout, schedule deployment, permissions, user account management
33. Professional Services
Mender-specific
- Integrating the Mender client with your device
- Design of device partition layouts
- Guidance on a scalable Mender backend setup
- Integrate Mender with CI/CD pipeline
- Develop custom use-case specific update scripts
Embedded Linux/Yocto project consulting
- Embedded Linux system design architecture
- Yocto build customizations
- Yocto platform layer development
- Yocto recipe development
- Custom automated build setup
34. Learn More
Get started now:
https://docs.mender.io/2.0/getting-started
Join the Mender Hub community:
https://hub.mender.io
Mender on Github:
https://github.com/mendersoftware/
contact@mender.io @mender_io /company/mender.iohttps://mender.io/