Software update for IoT Embedded World 2017

2. License These slides are available under a Creative Commons Attribution-ShareAlike 3.0 license. You can read the full text of the license here http://creativecommons.org/licenses/by-sa/3.0/legalcode You are free to • copy, distribute, display, and perform the work • make derivative works • make commercial use of the work Under the following conditions • Attribution: you must give the original author credit • Share Alike: if you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one (i.e. include this page exactly as it is) • For any reuse or distribution, you must make clear to others the license terms of this work Software update for IoT 2 Copyright © 2011-2017, 2net Ltd

3. About Chris Simmonds • Consultant and trainer • Author of Mastering Embedded Linux Programming • Working with embedded Linux since 1999 • Android since 2009 • Speaker at many conferences and workshops "Looking after the Inner Penguin" blog at http://2net.co.uk/ https://uk.linkedin.com/in/chrisdsimmonds/ https://google.com/+chrissimmonds Software update for IoT 3 Copyright © 2011-2017, 2net Ltd

5. What could possibly go wrong? • Mirai: a recent > 600 Gbps DDoS attack • Very simple: looks for open Telnet ports and logs on using default, well-known, name and password • Prime target: Dahua IP CCTV cameras Details on PenTestPartners: https://www.pentestpartners.com/blog/optimising-mirai-a-better-iot-ddos-botnet Software update for IoT 5 Copyright © 2011-2017, 2net Ltd

6. Requirements for SW update • Secure: must not become an attack vector to hijack the device • Robust: must not render the device unusable • Atomic: an update must be installed completely or not at all • Fail-safe: should fall-back to last known working system if there are errors Software update for IoT 6 Copyright © 2011-2017, 2net Ltd

8. Update granularity • File: • not an option: hard to achieve atomicity over a group of file updates • Package (e.g. RPM, deb): • apt-get update works fine for servers but not for devices • Entire filesystem image: • the most common option: fairy easy to implement and verify • Atomic differential update • Uses clever tricks to perform atomic update of groups of files • Container • neat idea, so long as you have containerised applications Software update for IoT 8 Copyright © 2011-2017, 2net Ltd

9. Device update != server update • Server • Secure environment, no power outage, no network outage • If update fails, human intervention is possible • Device: • Intermittent power and network mean update quite likely to be interrupted • Failed update may be difﬁcult (and expensive) to resolve Incremental package updates via RPM/deb are not atomic Software update for IoT 9 Copyright © 2011-2017, 2net Ltd

10. Image update Symmetric A/B (Android after Nougat) Bootloader User data Boot ﬂag OS Copy 1 OS Copy 2 Asymmetric normal/recovery (Android before Nougat) Bootloader Main OS Recovery OS User data Boot ﬂag Software update for IoT 10 Copyright © 2011-2017, 2net Ltd

11. Examples of image updaters 1/2 • swupdate: http://sbabic.github.io/swupdate/index.html • Symmetric and asymmetric image update client • License: GPLv2 • Mender: https://mender.io • Symmetric image update client • Integrates with open source OTA update server • License: Apache 2 Software update for IoT 11 Copyright © 2011-2017, 2net Ltd

12. Examples of image updaters 2/2 • RAUC (Robust Auto-Update Controller): https://rauc.readthedocs.org/ • Symmetric and asymmetric image update client • License: LGPLv2.1 • Android Things: https://developer.android.com/things/hardware/index.html • Symmetric and asymmetric image update client • Android Things is cut-down Android for IoT • License: Apache 2 Software update for IoT 12 Copyright © 2011-2017, 2net Ltd

13. Atomic differential update (OSTree) • OSTree stores data in a "git-like" object repository • Actual filesystem created from hard links to objects in the repository • Checkout is atomic • Pluses • Updates are incremental: uses less disk space than A/B image update • Reduced transfer time: update contains only changed files • Minuses • Physically there is only one filesystem: cannot recover from filesystem corruption Software update for IoT 13 Copyright © 2011-2017, 2net Ltd

15. Atomic differential update (swupd) • Similar to OSTree • swupd processes updates in bundles • Bundle contains ﬁle updates • Updates to bundles are applied atomically Software update for IoT 15 Copyright © 2011-2017, 2net Ltd

17. Containerised updaters • Consists of • Immutable base OS • Applications in containers • Update client running in base OS is able to update the applications atomically • But, note that: • Updates applications only • Need another mechanism to update kernel and rootfs (e.g image A/B) Software update for IoT 17 Copyright © 2011-2017, 2net Ltd

18. Examples of containerised updaters • resin.io: https://resin.io • Applications distributed in Docker containers • Managed on device by Resin Container Engine • Integrates with OTA update server (not open source) • Licenses: Client: Apache2; Server: proprietary • Snappy: https://snapcraft.io • Containers are called snaps • A snap contains a squashfs ﬁle system, mounted on /snap/[app] • License: GPLv3 Software update for IoT 18 Copyright © 2011-2017, 2net Ltd

20. Complexities of OTA update • Authentication (is this update legit?) • Security (am I receiving what you are sending?) • Roll-back (if update fails to boot, switch to previous version) • Scale (roll out to large populations) • Monitoring (keeping track of status of the population of devices) Software update for IoT 20 Copyright © 2011-2017, 2net Ltd

21. Roll-back • Boot limit count • Feature of bootloader (e.g U-Boot) • Increment count in bootloader • Reset after successful boot • If reboot with count > 0, bootloader knows boot failed and loads alternate rootfs • Hardware watchdog • If hang in early boot, watchdog times out and resets CPU • Bootloader checks reset reason • If watchdog, loads alternate rootfs Software update for IoT 21 Copyright © 2011-2017, 2net Ltd

22. Stateless filesystems • Most of the updaters mentioned require a stateless rootfs • You need to store run-time changes to files outside filesystems being updated • Methods: • symlink or bind mount files to writeable storage • modify existing code to keep state separate • write new code to be stateless • See https://www.slideshare.net/chrissimmonds/ readonly-rootfs-theory-and-practice Software update for IoT 22 Copyright © 2011-2017, 2net Ltd

23. Conclusion • Image update is well-understood and road tested • swupdate, RAUC, Mender, Android Things • Atomic update promises ﬁner granularity and smaller updates • OSTree, swupd • Containerised update is trendy, but only solves part of the problem • resin.io, snappy • Some projects integrate with OTA update servers • Mender, resin.io, (swupdate, RAUC via Hawkbit https://eclipse.org/hawkbit) • Update is more robust with a stateless rootfs, but this is still a new concept to many developers Software update for IoT 23 Copyright © 2011-2017, 2net Ltd

24. • Questions? Slides on Slide Share https://www.slideshare.net/chrissimmonds/ software-update-for-iot-embedded-world-2017 This and other topics associated with building robust embedded systems are covered in my training courses http://www.2net.co.uk/training.html Software update for IoT 24 Copyright © 2011-2017, 2net Ltd