Downloaded 126 times
More Related Content
Similar to Tachyon workshop 2015-07-19
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Tachyon workshop 2015-07-19
- 1. Bin Fan, TachyonNexus July 19, 2015 @ Tachyon Workshop tachyon-project.org A Reliable Memory-Centric Distributed Storage System
- 2. • Founded byTachyon creators and top contributors • $7.5 million Series A from Andreessen Horowitz • Committed to Tachyon Open Source • www.tachyonnexus.com 2
- 3.
- 4. Outline • Overview – Motivation –Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 4
- 5. Outline • Overview – Motivation –Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 5
- 6. Started From UCBAMPLab Berkeley Data Analytics Stack (BDAS) Cluster manager Parallel computation framework Reliable, distributed memory-centric storage system 6
- 7.
- 8. Memory is Fast •RAM throughput increasing exponentially • Disk throughput increasing slowly 8 Memory-locality key to interactive response times
- 9. Memory is Cheaper source:jcmit.com 9
- 10.
- 11.
- 12. 12 Missing a Solution forthe Storage Layer
- 13. An Example: - •Fast, in-memory data processing framework – Keep one in-memory copy inside JVM – Track lineage of operations used to derive data – Upon failure, use lineage to recompute data map filter map join reduce Lineage Tracking 13
- 14. Issue 1 14 Data Sharingis the bottleneck in analytics pipeline: Slow writes to disk Spark Job1 Spark mem block manager block 1 block 3 Spark Job2 Spark mem block manager block 3 block 1 HDFS / Amazon S3 block 1 block 3 block 2 block 4 storage engine & execution engine same process (slow writes)
- 15. Issue 1 15 Spark Job Sparkmem block manager block 1 block 3 Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 Data Sharing is the bottleneck in analytics pipeline: Slow writes to disk storage engine & execution engine same process (slow writes)
- 16. Issue 2 16 Spark Task Sparkmemory block manager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process Cache loss when process crashes
- 17. Issue 2 17 crash Spark memory blockmanager block 1 block 3 HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process Cache loss when process crashes
- 18. HDFS / AmazonS3 Issue 2 18 block 1 block 3 block 2 block 4 execution engine & storage engine same process crash Cache loss when process crashes
- 19. HDFS / AmazonS3 Issue 3 19 In-memory Data Duplication & Java Garbage Collection Spark Task1 Spark mem block manager block 1 block 3 Spark Task2 Spark mem block manager block 3 block 1 block 1 block 3 block 2 block 4 execution engine & storage engine same process (duplication & GC)
- 20. Tachyon Reliable data sharingat memory-speed within and across cluster frameworks/jobs 20
- 21. Technical Overview Ideas • Amemory-centric storage architecture • Push lineage down to storage layer • Manage tiered storage Facts • One data copy in memory • Re-computation for fault-tolerance 21
- 22.
- 23.
- 24.
- 25.
- 26. Issue 1 revisited 26 Memory-speeddata sharing among jobs in different frameworks execution engine & storage engine same process (fast writes) Spark Job Spark mem Hadoop MR Job YARN HDFS / Amazon S3 block 1 block 3 block 2 block 4 HDFS disk block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4
- 27. HDFS / AmazonS3 block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4 Issue 2 revisited 27 Spark Task Spark memory block manager execution engine & storage engine same process Keep in-memory data safe, even when a job crashes.
- 28. Issue 2 revisited 28 HDFS disk block1 block 3 block 2 block 4 execution engine & storage engine same process Tachyon in-memory block 1 block 3 block 4 crash HDFS / Amazon S3 block 1 block 3 block 2 block 4 Keep in-memory data safe, even when a job crashes.
- 29. Issue 3 revisited 29 Noin-memory data duplication, much less GC Spark Task Spark mem Spark Task Spark mem HDFS / Amazon S3 block 1 block 3 block 2 block 4 execution engine & storage engine same process (no duplication & GC) HDFS disk block 1 block 3 block 2 block 4 Tachyon in-memory block 1 block 3 block 4
- 30. Comparison with In-MemoryHDFS 30
- 31. How easy /hard to use Tachyon? 31
- 32. Spark/MapReduce/Shark without Tachyon • Spark scala>val file = sc.textFile(“hdfs://ip:port/path”) • Hadoop MapReduce $ hadoop jar hadoop-examples-1.0.4.jar wordcount hdfs://localhost:19998/input hdfs://localhost:19998/output • Shark CREATE TABLE orders_cached AS SELECT * FROM orders; 32
- 33. Spark/MapReduce/Shark with Tachyon • Spark scala>val file = sc.textFile(“tachyon://ip:port/path”) • Hadoop MapReduce $ hadoop jar hadoop-examples-1.0.4.jar wordcount tachyon://localhost:19998/input tachyon://localhost:19998/output • Shark CREATE TABLE orders_tachyon AS SELECT * FROM orders; 33
- 34. Outline • Overview – Motivation –Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 34
- 35. Open Source Status •Started at UC Berkeley AMPLab in Summer 2012 • Apache License 2.0, Version 0.7 (July 2015) • Deployed at > 50 companies (July 2014) • 30+ Companies Contributing • Spark/MapReduce/Flink applications can run without code change 35
- 36. Contributors Growth v0.4 Feb ‘14 v0.3 Oct‘13 v0.2 Apr ‘13 v0.1 Dec ‘12 36 v0.6 Mar ‘15 v0.5 Jul ‘14 v0.7 Jul ‘15 1 3 15 30 46 70 100+
- 37. Codebase Growth v0.4 Feb ‘14 v0.3 Oct‘13 v0.2 Apr ‘13 37 v0.6 Mar ‘15 v0.5 Jul ‘14 v0.7 Jul ‘15 465 commits 696 commits 1080 commits 1610 commits 2884 commits 4969 commits
- 38.
- 39. Thanks to OurContributors!Aaron Davidson Abhiraj Butala Achal Soni Albert Chu Ali Ghodsi Andrew Ash Anurag Khandelwal Aslan Bekirov Bill Zhao Bin Fan Bradley Childs Calvin Jia Carson Wang Chao Chen Cheng Chang Cheng Hao Colin Patrick McCabe Dan Crankshaw Darion Yaphet David Capwell David Zhu Dina Leventol Du Li Fei Wang Gene Pang Gerald Zhang Grace Huang Haoyuan Li Henry Saputra Hobin Yoon Huamin Chen Jacky Li Jey Kottalam Jingxin Feng Joseph Tang Juan Zhou Jun Aoki Kun Xu Lukasz Jastrzebski Luogan Kun Manu Goyal Mark Hamstra Mingfei Shi Mubarak Seyed Nan Dun Nick Lanham Orcun Simsek Pengfei Xuan Qianhao Dong Qifan Pu Ramaraju Indukuri Raymond Liu Rob Vesse Robert Metzger Rong Gu Sean Zhong Seonghwan Moon Shaoshan Liu Shivaram Venkataraman Shu Peng Srinivas Parayya Tao Wang Thu Kyaw Timothy St. Clair Vaishnav Kovvuri Vikram Sreekanti Xi Liu Xiaomeng Huang Xiaomin Zhang Xing Lin Yi Liu Zhao Zhang 39
- 40.
- 41. Under Filesystem Choices (BigData, Cloud, HPC, Enterprise) 41
- 42. Use Case: Baidu •Framework: SparkSQL • Under Storage: Baidu’s File System • Storage Media: MEM + HDD • 100+ nodes deployment • 1PB+ managed space • 30x Performance Improvement More Details: www.meetup.com/Tachyon 42
- 43. Use Case: aSAAS Company • Framework: Impala • Under Storage: S3 • Storage Media: MEM + SSD • 15x Performance Improvement 43
- 44. Use Case: anOil Company • Framework: Spark • Under Storage: GlusterFS • Storage Media: MEM only • Analyzing data in traditional storage 44
- 45. Use Case: aSAAS Company • Framework: Spark • Under Storage: S3 • Storage Media: SSD only • Elastic Tachyon deployment 45
- 46. Outline • Overview – Motivation –Tachyon Architecture – Using Tachyon • Open Source – Status – Production Use Cases • Roadmap 46
- 47. New Features • Lineagein Storage (alpha) • Tiered Storage (alpha) 47
- 48. New Features • Lineagein Storage (alpha) • Tiered Storage (alpha) • Data Serving • Support for New Hardware • … • Your New Feature! 48
- 49.
- 50. Distributed Memory-Centric Storage: BetterAssist Other Components Welcome Collaboration! 50 JIRA New Contributor Tasks Apache Spark Apache MR Apache HBase H2O Apache Flink Impala S3 Gluster FS HDFS Swift NFS Ceph …… ……
- 51. • Website: http://tachyon-project.org •Github: https://github.com/amplab/tachyon • Meetup: http://www.meetup.com/Tachyon • Training program: coming soon • Tachyon Nexus is hiring • News Letter Subscription: http://goo.gl/mwB2sX • Email: binfan@tachyonnexus.com 51