Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Continuous Deployment for Massive Scale Embedded Devices (IoT)
Continuous Deployment for Massive Scale
Embedded Devices
(Internet of Things, Hype hype!)
Intro
- Introduction to massive scale hardware
- Libvirt hub agent
- OTA CD
- Build environments for embedded linux
- Vendors
In...
particle.ioNest
Examples of “Internet of things” hype
Raspberry pi
Internet Connected Hardware
3,370,000,000
Residential internet gw
Actual economically relevant internet hardware
500,000
Teslas
6,800,000,000
smartpho...
bankruptcysoftwaremanufacture
Hardware is Hard
- Hardware is done at massive scale
- Hence it used to be a waterfall busin...
Embedded Software Testing
Bringup
Acceptance
Compliance
EMI
Functional
Integration
Regression
Smoke
Fresh from vendor Deve...
Embedded Software is Hard
- work on very few, very
expensive prototypes
- Test requires specialised jigs and
probes
- Manu...
Embedded Software is Hard
Typical test script you get from vendors:
- sudo apt-get -y install some-stuff-that-doesnt-exist...
The 2000 Solution
- TTY to LAN
- Allows sharing hardware within teams
- Conflict resolution: “shout across the
office”
- I...
Embedded Software Testing
Give me continuous integration plz
Mocking Hardware
Emulating/Mocking hardware for faster tests
➢ Nah.
➢ Buy more hardware
DOCKERIZE IT!!11
Damage control bad software? docker!
… or something...
- Anything using kernel namespaces
- Or in this case libvirt
Hub Agent at Nokia
1 usb hub
=
1 virtual machine
Hub Agent
udev
Hub agent
New device on hubNew usb hub
Spawn agent VM
Register agent
Attach device to agent
Set agent label
Hub Agent at Superscale
Docker now supports usb.
Yey!
Over The Air Updates
CD Pipeline would be incomplete without delivery
Don’t build devices without internet in 2016.
Contin...
Over The Air Updates
1,000,000 SCADA systems
are exposed on the internet
- Used in Factories, Nuclear plants,
Hospitals,.....
Over The Air Updates
Typical memory layout
system
user data / config state
bootloader
calibration
Vendor code from hell
Do...
Over The Air Updates
Why OTA is full-image
➢ This is hardware, it doesn’t change.
➢ We must test a known state
➢ OTA scale...
Over The Air Updates
Classic staged rollouts before CD
The thing
git push git pull
Build & test
Roll to 1000 customersRele...
Continuous Hardware
git push build
OTA cloud
Dude
your stuff is broketh
Roll this?
Build Environments
OBS
Meego
yocto
openwrt
buildroot
android
package based system based hacks
scratchbox2
Some words on sc...
Vendors
They don’t care.
Vendors
They don’t care.
No one reads your test instructions. Barely anyone speaks english.
Vendors
Hardware vendors are “agile” too.
memory chip origin: “found box in garage”
➢ Yield test != works
➢ Have every sin...
Vendors
Destruction tests you want on every variant
➢ Flash memory until death (must survive
specified number of cycle)
➢ ...
Exit
Internet hardware is hard, implement CD early
➢ Flash memory until death (must survive
specified number of cycle)
➢ M...
Upcoming SlideShare
Loading in …5
×

Atmosphere 2016 - Arvid Picciani - Continuous Deployment for Massive Scale Embedded Devices (IoT)

77 views

Published on

Continuous Deployment is all over software companies. We will look at how to transfer some of the methods to hardware startups. Exploring embedded devices as an iterative process rather than from a traditional engineering approach.


- why is hardware so hard
- hardware test types
- over the air updates
- A case for continuous deployment
- vendors hate you
- libvirt test host hardware abstraction
- A quick look at a pratical jenkins setup

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Atmosphere 2016 - Arvid Picciani - Continuous Deployment for Massive Scale Embedded Devices (IoT)

  1. 1. Continuous Deployment for Massive Scale Embedded Devices (IoT)
  2. 2. Continuous Deployment for Massive Scale Embedded Devices (Internet of Things, Hype hype!) Intro
  3. 3. - Introduction to massive scale hardware - Libvirt hub agent - OTA CD - Build environments for embedded linux - Vendors Intro
  4. 4. particle.ioNest Examples of “Internet of things” hype Raspberry pi Internet Connected Hardware
  5. 5. 3,370,000,000 Residential internet gw Actual economically relevant internet hardware 500,000 Teslas 6,800,000,000 smartphones Internet Connected Hardware
  6. 6. bankruptcysoftwaremanufacture Hardware is Hard - Hardware is done at massive scale - Hence it used to be a waterfall business - Fixing anything costs millions and takes years - Hence test suits are massive and take forever designspecify
  7. 7. Embedded Software Testing Bringup Acceptance Compliance EMI Functional Integration Regression Smoke Fresh from vendor Developing Software Building millions Aging Functional Calibration Weardown Testing is driven by hardware developers. Some typical examples:
  8. 8. Embedded Software is Hard - work on very few, very expensive prototypes - Test requires specialised jigs and probes - Manual reset required when hardware gets stuck - All the tools are sudo and bad
  9. 9. Embedded Software is Hard Typical test script you get from vendors: - sudo apt-get -y install some-stuff-that-doesnt-exist - sudo echo > /dev/something #never worked - sudo someflashtool-binary --device=/dev/ttyUSB0 - sudo ./usbthing --find-device-automagically - sudo scp 192.168.1.2:/test.xml /home/android/meh. cvs - echo probably worked. like i care. lol
  10. 10. The 2000 Solution - TTY to LAN - Allows sharing hardware within teams - Conflict resolution: “shout across the office” - I can’t believe people still do this - Usb controllable PSU - To “reboot” during test cycles
  11. 11. Embedded Software Testing Give me continuous integration plz
  12. 12. Mocking Hardware Emulating/Mocking hardware for faster tests ➢ Nah. ➢ Buy more hardware
  13. 13. DOCKERIZE IT!!11 Damage control bad software? docker!
  14. 14. … or something... - Anything using kernel namespaces - Or in this case libvirt
  15. 15. Hub Agent at Nokia 1 usb hub = 1 virtual machine
  16. 16. Hub Agent udev Hub agent New device on hubNew usb hub Spawn agent VM Register agent Attach device to agent Set agent label
  17. 17. Hub Agent at Superscale Docker now supports usb. Yey!
  18. 18. Over The Air Updates CD Pipeline would be incomplete without delivery Don’t build devices without internet in 2016. Continuously deliver updates
  19. 19. Over The Air Updates 1,000,000 SCADA systems are exposed on the internet - Used in Factories, Nuclear plants, Hospitals,... - 700+ vulnerabilities - No OTA = no Fix - => Have OTA, plz
  20. 20. Over The Air Updates Typical memory layout system user data / config state bootloader calibration Vendor code from hell Done in factory. Don’t touch. Stuff to delete in CI. Always test from clean state OTA goes here
  21. 21. Over The Air Updates Why OTA is full-image ➢ This is hardware, it doesn’t change. ➢ We must test a known state ➢ OTA scales to billions
  22. 22. Over The Air Updates Classic staged rollouts before CD The thing git push git pull Build & test Roll to 1000 customersRelease for real
  23. 23. Continuous Hardware git push build OTA cloud Dude your stuff is broketh Roll this?
  24. 24. Build Environments OBS Meego yocto openwrt buildroot android package based system based hacks scratchbox2 Some words on scalability of build environments
  25. 25. Vendors They don’t care.
  26. 26. Vendors They don’t care. No one reads your test instructions. Barely anyone speaks english.
  27. 27. Vendors Hardware vendors are “agile” too. memory chip origin: “found box in garage” ➢ Yield test != works ➢ Have every single variant in your test rack ➢ Write down chip versions on incoming batches ➢ Beware of changes even within batches ➢ Thanks to continuous deployment, we actually CAN test all variants on git push in parallel
  28. 28. Vendors Destruction tests you want on every variant ➢ Flash memory until death (must survive specified number of cycle) ➢ Max that cpu, forever, until it desolders ➢ Battery charging cycles. ➢ Extreme environment conditions ➢ Find creative ways of destroying stuff ➢ (yeah, you, don’t trust that vendor)
  29. 29. Exit Internet hardware is hard, implement CD early ➢ Flash memory until death (must survive specified number of cycle) ➢ Max that cpu, forever, until it desolders ➢ Battery charging cycles. ➢ Extreme environment conditions ➢ Find creative ways of destroying stuff

×