Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NGD Systems and Microsoft Keynote Presentation at IPDPS MPP in Vacouver

183 views

Published on

Presentation given during May 2018 Vancouver IEEE IPDPS Parallel Computing Conference during the MPP workshop

Published in: Technology
  • Be the first to comment

NGD Systems and Microsoft Keynote Presentation at IPDPS MPP in Vacouver

  1. 1. 17th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Toward using Computational Storage in Cloud and Fog Jaeyoung Do#, Vladimir Alves*, Microsoft Research#, NGD Systems* MPP 2018 7th Workshop on Parallel/Programming Models Special Edition on Edge/Fog/In-Situ Computing
  2. 2. 27th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 The world’s most valuable resource May. 2017 Data is everywhere! Values from Data! Need infrastructures for Volume and Velocity of data!
  3. 3. 37th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Case Study: OCP storage server (1/2) CPU DRAM Rootcomplex PCIe Switch PCIe Switch PCIe Switch Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD High cost of moving data Needs to scale up due to data growth
  4. 4. 47th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Case Study: OCP storage server (2/2) CPU DRAM Rootcomplex PCIe Switch PCIe Switch PCIe Switch Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD Flash SSD 0 1 31 0 1 31 15 1 0 32 channels X ~500 MB/s = ~16 GB/s 64 Flash SSDs X ~16 GB/s/SSD = ~ 1TB/s 16 lanes of PCIe = ~ 16 GB/s Throughput gap of 66x
  5. 5. 57th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Programmable components in data center  Software-defined control and management is an inevitable trend that is already touching other parts of the data center infrastructure  Software-Defined Networking (SDN) making network switches and server NIC cards increasingly programmable for enhanced network- wide functionality  Programmable GPGPUs and FGPAs leveraged by new generations of applications like deep learning  Rapidly-changing requirements can be supported on-the-fly once DC infrastructures become dynamically programmable Don’t Leave Storage Behind!
  6. 6. 67th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Next up: Storage (1/2)  Unfortunately, the lack of such programmable capabilities in storage results in a major disconnect in terms of the speed of innovation between application/OS and storage infrastructures!  While application/OS is patched with new/improved functionality every few weeks at cloud speed while storage devices are off limits for such sustained innovation during their hardware life cycle of 3-5 years in data centers.
  7. 7. 77th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Next up: Storage (2/2)  A fully programmable storage gives opportunities to better bridge the gap between application/OS needs and storage capabilities/limitations, while allowing us to innovate in-house at cloud speed. Flash SSDs can be programmable?
  8. 8. 87th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Today’s NAND Flash SSD (PCIe) Flash SSD Controller Host Server PCIe gen3 (4 lanes) Hardware b/w = 4 GB/s Achievable b/w = 2-2.5 GB/s 16-32 channels @ 500MB/s Total = 8-16 GB/secThroughput gap of 2x - 8x
  9. 9. 97th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Past efforts – Smart SSD (1/2)  Goal: Exploring the opportunities and challenges associated with running selected database operations inside an SSD  Used Samsung SAS SSD  Modified Microsoft SQL Server 2012 to offload database operations onto a Samsung Smart SSD  Simple selection and aggregation operators were hard-coded and compiled into the firmware of the SSD Query Processing on Smart SSDs: Opportunities and Challenges, SIGMOD 2013
  10. 10. 107th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Past efforts – Smart SSD (2/2) Base Case Smart SSD TPC-H Q6: Result: Compared to the base case, • 70% better query response time • 2X energy efficiency Device became a performance bottleneck! The dev. environment was not ready!
  11. 11. 117th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Past efforts – YourSQL (1/2)  Goal: Accelerating data-intensive queries with the help of hardware pattern matcher  Used Samsung PCIe SSD  Modified a variation of MySQL to realize early filtering of data by offloading data scanning of a query to programmable SSDs  Developed a framework (called Biscuit) that follows a data-flow model YourSQL: A High-Performance Database System Leveraging In-Storage Computing, VLDB16
  12. 12. 127th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Past efforts – YourSQL (2/2) Result: Compared to the base case, • 11X Speed up Computing resource is not powerful enough! Existing applications need to be redesigned!
  13. 13. 137th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Challenges of past efforts  Not enough “spare” processing power  H/W architecture limitations  Programming tools are not application-developer friendly  Prototype devices are not accessible to application developers
  14. 14. 147th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Disruptive trends that enable SoftFlash Frugal resources inside SSD Abundant resources inside SSD(CPU#cores/clockspeed, hardwareoffload,DRAM) Embedded CPU, proprietary firmware “OS” General purpose CPU, server-like OS (Linux) (Ease of programmability inside SSD) Today’s SSD Program -mable SSD
  15. 15. 157th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 The SoftFlash project  Goal: Embrace flash SSDs as a first-class programmable platform in the cloud data center  Add custom capabilities to storage over time  Better bridge the gap between application needs and flash media capabilities/limitations  Innovate in-house at cloud speed Powerful and flexible prototype board with enterprise-grade capabilities and resources Linux, SDK, user/kernel libraries for the on-chip H/W accelerators, built-in FTL Moving compute closer to storage, flexible storage interface, secure computation
  16. 16. 167th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Application 1: Move compute closer to data  Reduced data movement across storage/network/memory/CPU for compute Example:  Stream analytics over data logs  Select/Project/Aggregation for relational database/data warehouse services Host Server Moving compute to inside SSD to leverage low latency, high bandwidth access to data Programable SSD
  17. 17. 177th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Application 2: Agile, flexible storage  Custom SSD capabilities to better meet application needs Example:  Multi-streamed writes for cloud storage platform  Read I/O priority for latency-sensitive services  Atomic writes for transactional services Host Server Agile, flexible storage interface leveraging programmability within SSD Programable SSD
  18. 18. 187th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Example) Stream-Aware Flash Data Stream #1 Data Stream #2 Block A Block B Regular (stream-unaware) SSD Incoming Writes Data Stream #1 Data Stream #2 Tag writes with stream ID Block A Block B Stream-aware Programmable SSD (2x device lifetime, 1.5x read throughput) Flash pages
  19. 19. 197th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Example) Key-Value SSD  Log-structured KV store (e.g., RocksDB)  Uses log-structuring to avoid random writes by appending new versions of pages to the end of the log.  The “cleaning” process is needed to continuously reclaim space.  The mapping table is periodically checkpointed.  SSDs with their FTLs already have log structured implementations inside the device  Duplicated functionalities such as maintaining a mapping table, buffering multiple writes, cleaning, checkpointing and recovering.
  20. 20. 207th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Application 3: Secure computation in cloud  SSD provides a trusted domain for secure computation on encrypted data, without cleartext leaving the device Example:  Existing and new scenarios in a “trusted cloud” setting -- user stores encrypted data in the cloud and needs to do compute over it Trusted domain for secure computation; cleartext not allowed to egress this boundary Programable SSD
  21. 21. 217th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Value propositions Moving compute close data Programmable SSD Agile, flexible Storage Secure Computation
  22. 22. 227th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (1/6)  Today, big data analytics fetches huge volumes of data from storage and processes it in host server  Programmable SSDs enable data analytics “inside” storage  Exploit higher bandwidth inside SSD (vs. SSD external interface)  Leverage ARM cores + hardware offload engines inside SSD
  23. 23. 237th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (2/6)  Efficient use of heterogeneous hardware in the data center for higher performance @lower power  Free up expensive host server CPU + memory resources, opportunities to increase service density  Reduced energy footprint due to significantly less data movement + low power compute inside SSD
  24. 24. 247th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (3/6) – Traditional Arch. Storage Node Storage Node Storage Node Storage Cluster Compute Node Compute Node Compute Cluster A A Fetch compressed data from storage cluster B Decompress dataB C C Decode data D D Do required computations (filtering, aggregation, etc)
  25. 25. 257th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (4/6) - programmable SSDs Storage Node Storage Node Storage Node Storage Cluster Compute Node Compute Node Compute Cluster A A Return results to Compute Cluster B Decompress data C Decode data D Do required computations (filtering, aggregation, etc) D B C D B C D B C D
  26. 26. 267th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (5/6) – Apache Hive  Hive is a data warehouse infrastructure built on top of Hadoop  Designed to enable easy data summarization, ad-hoc querying and analysis of large volumes of data  Encoding based on data type  Run-length encoding for integer  Dictionary encoding for string  Compression using a codec  Zlib or Snappy
  27. 27. 277th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Hardware prototype – Main board 4 X 10Gbps SFP+ - RoCE protocol 1 X 4-Lane PCIe 3.0 16GB DRAM 8-Core ARMv8 (1.8GHz) Hardware Accelerators – Compression (10Gbps) – Encryption (10Gbps) – PatternMatching (20Gbps) 2 X 4-Lane PCIe 3.0
  28. 28. 287th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Hardware Prototype – Storage Board 2 X M.2 SSDs 4 X DIMM Slots FPGA (storage controller)
  29. 29. 297th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Big Data Analysis (6/6) – Preliminary Result  Scanning a ZLIB-compressed, integer dataset (1 Billion rows) on a X86 server or inside the programmable SSD  Note that only a single core was used! What if we used +40 programmable SSDs?
  30. 30. 307th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Image Similarity Search (1/2) CNN CNN
  31. 31. 317th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Image Similarity Search (2/2)
  32. 32. 337th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Moving Computation to Data is Cheaper than Moving Data ▪ A computation requested by an application is much more efficient if it is executed near the data it operates on • minimizes network traffic • increases effective throughput & performance of the system • Example: the Hadoop Distributed File System • enables distributed processing ▪ Especially true for Big Data (analytics): large sets & unstructured data ▪ Traditional approach: high-performance servers coupled with SAN/NAS storage ▪ Eventually limited by networking bottlenecks
  33. 33. 347th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Vision & Value Proposition Creating a Paradigm Shift by bringing Computation to Storage VISION: VALUE PROPOSITION: Delivering a power efficient, low cost, & the only Computational Storage $ / GBW / TB In-Situ Processing
  34. 34. 357th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Moving Computation to Data (1) 35 Host CPU DRAM 1 store data set(s) 2 read 3 computation Steps 2 and 3 are repeated multiple times Computation in host CPU involves large data transfer 4 result Flash media Storage Device (SSD)
  35. 35. 367th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Moving Computation to Data (2) Storage Device (SSD) Server CPU DRAM 1 store data set(s) 3 computation 5 result Flash media Buffer In-Situ Processing 4 result 2 Computation requires lower power consumption and can be performed more efficiently Steps 2 and 3 are repeated multiple times
  36. 36. 377th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Hyperscale Data Center Challenges storage needs to scale up due to data growth media to compute bandwidth mismatch by 66x • Simplify Design by Maximizing CPU and Storage • Augment Computation with In-Situ Storage saving CPU load • Scale up Storage without CPU and memory costs • Reset Bandwidth alignment to use intelligence in storage
  37. 37. 387th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Hardware & Software Stack HW Resources SW Resources It’s an NVMe SSD at the core • No impact on host read/write • No impact on NVMe driver • Standard protocols But then there is MORE • Dedicated compute resources • HW acceleration for data analytics • Seamless programming model • Scalable
  38. 38. 397th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 ➢ Development tools Programming Model & Ecosystem Yocto is an open source project that helps developers create custom Linux-based systems for embedded products Ubuntu Core is a tiny, transactional version of Ubuntu for IoT devices and large container deployments Docker container enables true independence between applications and infrastructure and developers and IT Azure IoT Edge is a service that delivers cloud intelligence locally by deploying and running artificial intelligence (AI) and Azure services, on cross- platform IoT devices ➢ Operating System ➢ Container/virtualization ➢ Edge computing
  39. 39. 407th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Facebook AI Similarity Search + Computational Storage Use Case: Image Similarity Search at Data Center NGD Systems Confidential 10 M images UCSB 2007 1 Billion images Facebook 2017 1 Trillion images
  40. 40. 417th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Real Time similarity search model  Indexing Process  Image to Vector Converting  Creating Dataset (training, base, query and ground-truth vectors)  Loading train set (I/O)  Training Index  Loading Database (I/O)  Adding Vectors to Index  Loading ground truth data  Searching Process  Query Management  Converting Images to Vectors  Searching  Evaluating result (R@1, R@10, .. , search time)
  41. 41. 427th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 ARM® CPU A53 @ 1.5GHz (4 cores)SSD Experiment HDD HOST + HDDs Cat-2 SSD HOST + smart SSDs Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) PCIe/NVMe fabric HDD … HDD Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) SAS or SATA ARM® CPU A53 @ 1.5GHz (4 cores)SSD …
  42. 42. 437th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 POC platform HOST server x86 cpu x86 cpu x16 PCIe Gen3 expander HIC HIC x16 PCIe cable GPU GPU . . . x16 PCIe cable Up to 16 smart SSDs (8TB – 24TB each) In collaboration with Orange Silicon Valley
  43. 43. 447th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 FAISS running In-Situ SSD  Reading Vectors one by one and indexing them  Training Data  Hold data as Persistent Data  Searching Loading train set Indexing Data Loading database Add data To Index vectors Training & Indexing In-Situ HDD HOST + HDD Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) HOST + smart SSDs ARM® CPU A53 @ 1.5GHz (4 cores)
  44. 44. 457th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Memory Footprint with Indexed Dataset Nq=15, d=2048, k=8 ANN: PCA80-Flat Nb=519,976 4.173316 0.19198 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 Host In-Situ Memory (GB) Memory efficiency= 4.17/0.19 = 22
  45. 45. 467th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Scaling performance and capacity Intel® Xeon(R) CPU E5-2630 v4 @ 2.20GHz (20 cores) - 32GB RAM SSD NGD Cat2 HOST SSD NGD Cat2 SSD NGD Cat2 ARM® CPU A53 @ 1.5GHz (4 cores) ARM® CPU A53 @ 1.5GHz (4 cores) ARM® CPU A53 @ 1.5GHz (4 cores)
  46. 46. 477th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Memory Usage
  47. 47. 487th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Augmenting system performance Host only: ANN: PCA80-Flat Nq=15, d=2048, k=8 Nb=985,196 Host + In-Situ: ANN: PCA80-Flat Nq=15 , d=2048, k=8 Host: Nb=519,976 In-Situ: Nb=465,220 0.1 0.06 0 0.02 0.04 0.06 0.08 0.1 0.12 Host Host + In-Situ Host BusyTime (Sec) Energy efficiency= (0.1-0.06)/0.1 X 100= %40
  48. 48. 497th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Search time: scaling with dataset size 0.102 0.151 0.21 0.406 0.514 0.77 1.1 1.7 2.6 3.7 5.6 9 13 19 28 42 0.532 0.53 0.532 0.532 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0.533 0 5 10 15 20 25 30 35 40 45 H H+1 H H+2 H H+3 H H+4 H H+5 H H+6 H H+7 H H+8 H H+9 H H+10 H H+11 H H+12 H H+13 H H+14 H H+15 H H+16 Sec Searching Time (Host) Searching Time (Host + n In-Situ) Expon. (Searching Time (Host))
  49. 49. 507th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Some results
  50. 50. 517th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 From Cloud Computing to Fog Computing Business Support Enterprise Systems Operational Support Monitoring & Control Sensors & Actuators Cloud Cloud Cloud CloudCloud Cloud 1 2 3 4 Fog Fog Fog Fog Fog Fog Source: OpenFog Consortium
  51. 51. 527th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Fog Deployments Cloud Sensors, actuators, mobile devices, phones, tablets, cars, CDN, etc Fog Roadside Sensors: Speed, volume, weather: ice, snow, water, &etc. In-vehicle sensors In-vehicle sensors V2V Communication Sensors-to-Fog-to-Sensors via Gateway (VM or Dedicated) Roadside Traffic Fog Devices Neighborhood Traffic Fog Devices Regional Traffic Fog Devices Vehicle & Pedestrian aware sensors Regional EMS Fog Devices Metropolitan Traffic Services Cloud EMS Cloud SP Cloud SP Cloud Manufacturer Cloud Manufacturer Cloud Auto Dealer Fog Nodes Smart Traffic Light Vehicle & Pedestrian aware sensors Vehicle & Pedestrian aware sensors On-board devices connect via AP in vehicle for internet, phone, infotainment roadside traffic cameras Source: OpenFog Consortium
  52. 52. 537th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Use Case: Autonomous Car
  53. 53. 547th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 - “..challenge is the amount of data coming into the datacenter is cost prohibitive...” - “….need to push the core into the local markets…..” - “….need small footprint compute and high capacity storage out in the field….” - Prospect customer who has volume need at the edge - Closely aligned with autonomous driving 5G deployments and Edge Computing
  54. 54. 557th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Redefining Fog Node Bringing computation to data a Paradigm Shift in Storage Definition: A hierarchy of elements between the cloud and endpoint devices, and between devices and gateway Challenges: • Massive amount data • Limited bandwidth to cloud • Limited available power • Harsh environment Needs: • High capacity storage • Low energy consumption • Storage with Computation capability
  55. 55. 567th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018  Computational storage requires a seamless programming model  Support for containers enables large scale deployment and portability  Scaling processing with storage capacity can help solve bottlenecks in the data center and in the Fog Conclusion 56
  56. 56. 577th Workshop on Parallel/Programming Models - Special Edition on Edge/Fog/In-Situ ComputingMPP 2018 Questions? 57

×