OS frontiers in the AI era

1. OS frontiers in the AI era Felix Xiaozhu Lin 1

2. Who Am I • Fall 2020. Assoc. Prof, UVA CS – Looking forward to it! • 2014 – 2020. Asst. Prof, Purdue ECE • 2014 PhD in CS. Rice • 2008 MS + BS. Tsinghua 2

3. What is an OS? Undergrads (Some) data scientists Tech media 4 Are these OSes?

4. the era of AI! Where are we today? 5 https://en.wikipedia.org/wiki/Time_(xkcd)#/media/File:Xkcd_time_frame_0001.png

5. The 3rd boom of AI 1960s 1970s 1980s 1990s 2000s 2010s 2020s Classical AI Expert systems Machine learning Ups Downs 6

6. Two Pendulums 1960s 1970s 1980s 1990s 2000s 2010s 2020s Expert systems Machine learning Ups Downs Centralized Decentralized Dedicated hw & net PC + Internet Datacenters Cloud computing 5G, edge, embedded AI… 7 Classical AI

7. Two Pendulums 1960s 1970s 1980s 1990s 2000s 2010s 2020s Expert systems Machine learning Generalized Specialized Microcomputers, x86, DBMS, PC Mainframes, Apollo11 Ups Downs Wearables/IoT, NoSQL, GPU/FPGA, RISCV Centralized Decentralized Dedicated hw & net PC + Internet Datacenters Cloud computing 5G, edge, embedded AI… 8 Classical AI

8. Two Pendulums 1960s 1970s 1980s 1990s 2000s 2010s 2020s Expert systems Machine learning Generalized Specialized Microcomputers, x86, DBMS, PC Mainframes, Apollo11 Ups Downs Wearables/IoT, NoSQL, GPU/FPGA, RISCV Centralized Decentralized Dedicated hw & net PC + Internet Datacenters Cloud computing 5G, edge, embedded AI… 9 Classical AI

9. My view of OS OS == System Software == Software Infrastructure 12

10. What you have learnt from undergrad OS class 1960s 1970s 1980s 1990s 2000s 2010s 2020s GOFAI Expert systems Machine learning Generalized Specialized Microcomputers, x86, DBMS, PC Mainframes, Apollo11 Ups Downs Wearables/IoT, NoSQL, GPU/FPGA, RISCV Centralized Decentraliz ed Dedicated hw & net PC + Internet Datacenters Cloud computing 5G, edge, embedded AI… 14

11. What OS people are working on 1960s 1970s 1980s 1990s 2000s 2010s 2020s GOFAI Expert systems Machine learning Generalized Specialized Microcomputers, x86, DBMS, PC Mainframes, Apollo11 Ups Downs Wearables/IoT, NoSQL, GPU/FPGA, RISCV Centralized Decentraliz ed Dedicated hw & net PC + Internet Datacenters Cloud computing 5G, edge, embedded AI… 15

12. So, where are we today? 17

13. • Diverse • A few common OSes • Many specialized ones • Serving “things” in addition to humans • Defined by scenarios • Generic OSes  firmware • Specialized OSes  overlays • Blurry boundaries • arch, runtime, compiler, kernel, hypervisor, trusted exec environment… Cloud Edge Devices 18

14. Three flavors of OS research 1. Tune up 2. Do a specific thing well 3. Show possibility Common themes: open blackboxes, break, & build 19

15. Case 1: A highspeed stream analytics engine 21

16. High‐throughput. Sub‐second delay. Timely processing before data gets cold! 22 “Hot springs”: telemetry events Power sensor 140M events/day Oil rig 1‐2TB/day Manufacturing machines PBs/day

17. Stream analytics: state of the art • Classic engines? • StreamBase, Aurora, TelegraphCQ, NiagaraST… • Single threaded. Not scaling well. • Modern engines for datacenters? • Apache Flink, Spark Streaming, Beam… • Designed for tens ‐ hundreds of machines. Scaling out. • Assuming okay if individual nodes perform poorly • As analytics moves to the edge  bad 23

18. Project StreamBox stream analytics at the memory speed 24 • RDMA • Co‐designed with mm/scheduling • RDMA • Co‐designed with mm/scheduling Stream pipeline Threads Ingestion Scheduler Mem • Squeeze parallelism for multi/manycore • Manage NUMA domains • Squeeze parallelism for multi/manycore • Manage NUMA domains Exploit high‐bandwidth memoryExploit high‐bandwidth memory [ASPLOS'19] "StreamBox‐HBM: Stream Analytics on High Bandwidth Hybrid Memory," Hongyu Miao, Myeongjae Jeon, Gennady Pekhimenko, Kathryn S. McKinley, and Felix Xiaozhu Lin [USENIX ATC'17] "StreamBox: Modern Stream Processing on a Multicore Machine," Hongyu Miao, Heejin Park, Myeongjae Jeon, Gennady Pekhimenko, Kathryn S. McKinley, and Felix Xiaozhu Lin, in Proc. USENIX Annual Technical Conference, 2017. [ASPLOS'16] "memif: Towards Programming Heterogeneous Memory Asynchronously," Felix Xiaozhu Lin and Xu Liu, in Proc. ACM Int. Conf. Architectural Support for Programming Languages and Operating Systems, 2016.

19. CoresCores High‐bandwidth hybrid memory 25 3D DRAM Normal DRAM Tradeoffs: capacity vs. bandwidth Untraditional memory hierarchy No latency benefit Unlike SRAM+DRAM 16 GB 375 GB/s ~100 GB ~100 GB/s

20. 26 Already on off‐the‐shelf machinesIntel Xeon Phi Knights Landing (KNL)

21. 27 HBM CoresCores Normal DRAM Streaming data Data Bundles Index {key, pointer} Capacity: Use HBM only for grouping indexes

22. 28 Cheap VM (huge page) Apps OS kernel Fast net stack (40 GbE or RDMA) High task parallelism Custom mem allocator Sequential mem access Runtime Thread pool + custom task scheduler Wide SIMD (avx512) Hybrid memory A system software’s approach to 3D DRAM Blurry boundaries

23. Case 2: Autonomous AI on Cameras [MobiSys'20] "Approximate Query Processing on Autonomous Cameras," Mengwei Xu, Xiwen Zhang, Yunxin Liu, Xuanzhe Liu, and Felix Xiaozhu Lin 29

24. X1Video: a killer AI application 30

25. 31

26. 32

27. 33 Cut the cords

28. Run on harvested energy Solar Wind 34

29. 35 … and on wireless

30. Construction sites Farms Boats/RVs Warehouses Photos Credits: Reolink 36 Pervasive cameras

31. Object Counts Elf Running analytics on wire‐free cameras? 37

32. Query: (car, 30 mins) Install 7:00AM-7:30AM [500 + 100] Cars 7:30AM-8:00AM [700 + 140] Cars 8:00AM-8:30AM [800 + 180] Cars 8:30AM-9:00AM [400 + 100] Cars 9:30AM-10:00AM [200 + 80] Cars Sample & capture 200‐80 +80 Elf: Query model 38

33. Camera Operating System Planning Energy via Reinforcement Learning Planning Energy via Reinforcement Learning Sampled frames Aggregator with error Integration Aggregator with error Integration Selected NeuralNet Object counts Elf: System Internals 39

34. Elf prototype: heterogeneous processors 40

35. 7:00AM-7:30AM [500 + 100] Cars 7:30AM-8:00AM [700 + 140] Cars 8:00AM-8:30AM [800 + 180] Cars 8:30AM-9:00AM [400 + 100] Cars 9:30AM-10:00AM [200 + 80] Cars Ground Truth Error: 11% Confidence interval width: 17% Auburn, AL Hampton, NY Jackson, WY Taipei Taipei ~1000 hours 41

36. M4 AUTONOMOUS Intelligence OS for AI + AI for OS = Autonomous infrastructure 42

37. Case 3: Kernel IO on co‐processors 43

38. 44

39. 45 Weak co‐processors

40. 46 CPU Co Proc 2.5GHz 50MHz DRAM IO A heterogeneous SoC

41. 47 CPU Co Proc 2.5GHz 50MHz DRAM IO Weak co‐processors: suits low‐power IO tasks! high efficiency Linux Kernel IO tasks

42. 48 CPU Co Proc 2.5GHz 50MHz DRAM IO Kernel execution on weak co‐processors? Linux Kernel IO tasks Diff ISA No MMU …

43. 49 CPU Co Proc DRAM IO Co‐processor translates unmodified kernel binary Dynamic Binary Translation Linux Kernel IO tasks [USENIX ATC 19] "Transkernel: An Executor for Commodity Kernels on Peripheral Cores," Liwei Guo, Shuang Zhai, Yi Qiao, and Felix Xiaozhu Lin

44. Take away: • Taming an OS kernel (a beast) for new hardware • … without re‐engineering much of the software stack 50

45. Recap • What are OSes in 2020? • Three OS projects: • spanning IoT, mobile, and datacenters • each with different flavors • The builder culture • open blackboxes • break things • build things from the ground up • Started by a small group of hardcore hackers • Now more diverse and inclusive • A brave new world 51 StreamBox Elf

46. Image credits • https://dsportmag.com/the‐tech/test‐n‐tune/test‐tune‐2017‐subaru‐wrx‐ sti‐part3‐closer‐to‐the‐ej257s‐limits/ • https://techcrunch.com/2009/11/19/redneck‐rampage‐a‐truck‐with‐a‐jet‐ engine/?utm_source=feedburner#038;utm_medium=email 52

OS frontiers in the AI era

You are reading a preview.

Check these out next

OS frontiers in the AI era

More Related Content

Recently uploaded (20)

Featured (20)