This document proposes using unikernels and specialized machine learning compilers and runtimes to enable distributed machine learning on IoT devices. It demonstrates an end-to-end proof-of-concept for TinyML as a service that trains a MNIST model, compiles it to run on an ESP32 microcontroller, and performs inference on handwritten digits. Next steps include adding orchestration with CoAP, supporting more devices and complex models, distributed training on microcontrollers, and distributed inference across heterogeneous hardware accelerators.

1. Distributed ML on Unikernel for IoT
TinyML as-a-Service
Hiroshi Doyu <hiroshi.doyu@ericsson.com>
1 / 71

2. TinyML?
ref unikernel 2 / 71

3. Demo
ref MNIST 3 / 71

4. 4 / 71

5. MNIST on ESP32
0:00 / 0:30
5 / 71

6. Outline
1. Demo
2. Problems
Edge Computing
Web vs Embedded
ML environment
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
6 / 71

7. 7 / 71

8. 8 / 71

9. 9 / 71

10. ref Tx is expensive. 10 / 71

11. 11 / 71

12. Outline
1. Demo
2. Problems
Edge Computing
Web vs Embedded
ML environment
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
12 / 71

13. 3 areas?
Cloud
Edge
Embedded
13 / 71

14. 14 / 71

15. Outline
1. Demo
2. Problems
Edge Computing
Web vs Embedded
ML environment
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
15 / 71

16. 16 / 71

17. Summary: Problems
17 / 71

18. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
18 / 71

19. 19 / 71

20. Squeeze ML
model & runtime
per RTOS
per MCU / ROM / RAM
per HWA
20 / 71

21. 21 / 71

22. 22 / 71

23. Proposal
23 / 71

24. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
24 / 71

25. Three enblers
25 / 71

26. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
Unikernel
ML compiler
CoAP
5. PoC
6. Conclusion
26 / 71

27. Unikernel
http://unikernel.org/files/2014-cacm-unikernels.pdf 27 / 71

28. library Operating System (libOS)
https://xenproject.org/wp-content/uploads/sites/79/2015/08/anyunirumpkernel.png 28 / 71

29. Internal
http://rumpkernel.org/ 29 / 71

30. Summary: Unikernel
Can be a thin wrapper of di erent RTOSes?
ref high level programming languages 30 / 71

31. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
Unikernel
ML compiler
CoAP
5. PoC
6. Conclusion
31 / 71

32. 2 ML phases:
https://ucbrise.github.io/cs294-rise-fa16/prediction_serving.html 32 / 71

33. ML framework ==
ML runtime +
back propagation
https://devopedia.org/deep-learning-frameworks 33 / 71

34. Compiling
https://tvm.ai 34 / 71

35. Computational graph (cgraph)
serialization: protobuf, flatbuf 35 / 71

36. General vs Special purpose runtime
General purpose is too big
all OPs built-in
Model could be optimized,
per HWAs' OPs,
smaller than CPU's OPs(?)
Runtime should implement only OPs,
which model uses.
ref trend of OpenISA 36 / 71

37. Summary: ML compiler
37 / 71

38. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
Unikernel
ML compiler
CoAP
5. PoC
6. Conclusion
38 / 71

39. ref LwM2M 39 / 71

40. Summary: Three Enablers
40 / 71

41. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
41 / 71

42. MNIST: Handwriting digits recognition
60K images for training, 10K for testing
https://www.katacoda.com/basiafusinska/courses/tensorflow-getting-started/tensorflow-mnist-beginner42 / 71

43. ESP32 WROVER KIT
43 / 71

44. PoC Outline
44 / 71

45. PoC
Sequence
ref image 45 / 71

46. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
Training
Compile
Inference
6. Conclusion
46 / 71

47. with Lightweight Actor Engine on Unikernel 47 / 71

48. with Lightweight Actor Engine on Unikernel 48 / 71

49. ref sequence diagram 49 / 71

50. with Lightweight Actor Engine on Unikernel 50 / 71

51. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
Training
Compile
Inference
6. Conclusion
51 / 71

52. 52 / 71

53. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
Training
Compile
Inference
6. Conclusion
53 / 71

54. 54 / 71

55. Binary Size
55 / 71

56. Outline
1. Demo
2. Problems
3. Proposal
4. Three Enablers
5. PoC
6. Conclusion
56 / 71

57. Done
Demonstrated End-to-End TinyML (as-a-Service?)
57 / 71

58. Next
1. Add Orchestration with CoAP?
2. Supprt other MCUs and/or complicated models?
3. Heterogeneous (distributed) training on MCU?
4. Distributed inference on heterogeneous HWAs?
ref OpenISA 58 / 71

59. TinyML as-a-Service
could bring ML
onto IoT.
59 / 71

60. 60 / 71

61. Appendix
61 / 71

62. Our focus
62 / 71

63. http://slides.cabo.space 63 / 71

64. Type of Unikernel
64 / 71

65. MirageOS in OCaml
65 / 71

66. Custom ISA, RISC-V && Arm
https://staceyoniot.com/why-arm-opened-up-its-instruction-set-and-what-it-means-for-iot/ 66 / 71

67. https://jaime.win/slides/IPSO2019.pdf 67 / 71

68. https://jaime.win/slides/IPSO2019.pdf 68 / 71

69. https://jaime.win/slides/IPSO2019.pdf 69 / 71

70. 70 / 71

71. Fashion MNIST from Zalando
71 / 71