Downloaded 13 times
Uploaded bySATOSHI TAGOMORI
Data Analytics Service Company and Its Ruby Usage
This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.
More Related Content
Similar to Data Analytics Service Company and Its Ruby Usage
Data Analytics Service Company and Its Ruby Usage
- 1. Data Analytics ServiceCompany and Its Ruby Usage EuRuKo 2015 (Oct 17, 2015) Satoshi Tagomori (@tagomoris)
- 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd,MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
- 4.
- 5.
- 7. Data Analytics Platform DataAnalytics Service
- 10.
- 11.
- 12.
- 13. Data Analytics Flow CollectStore Process Visualize Data source Reporting Monitoring
- 14. Data Analytics Flow CollectStore Process Visualize Data source Reporting Monitoring
- 15. Data Analytics Platform •Data collection, storage • Console & API endpoints • Schema management • Processing (batch, query, ...) • Queuing & Scheduling • Data connector/exporter
- 16.
- 17. Data Analytics Platform •Data collection, storage: Ruby(OSS), Java/JRuby(OSS) • Console & API endpoints: Ruby(RoR) • Schema management: Ruby/Java (MessagePack) • Processing (batch, query, ...): Java(Hadoop,Presto) • Queuing & Scheduling: Ruby(OSS) • Data connector/exporter: Java, Java/JRuby(OSS)
- 18. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 19. OSS products • Tomake logging more easy & simple than ever! • Plugin system • Open development • For various environment/usage • Fluentd, Fluent-Bit, Embulk • Fluent-Bit: Data collector for Embedded Linux http://fluentbit.io/
- 20. http://www.fluentd.org/ Fluentd Unified Logging Layer ForStream Data Written in CRuby http://www.slideshare.net/treasure-data/the-basics-of-fluentd-35681111
- 22. Bulk Data Loader HighThroughput&Reliability Embulk Written in Java/JRuby http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed http://www.embulk.org/
- 23. HDFS MySQL Amazon S3 CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behavior ✓ Idempotent retrying Plugins Plugins bulk load
- 24. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 25. Console/API • RoR +AWS RDS + AngularJS • on EC2 (API) and Heroku (Console) • Operation, Configuration & Managing Data
- 26. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 27. Collecting Data • Importover Console/API • From browsers and CLI (TD toolbelt) • Treasure Agent (rpm/deb) • Fluentd packaged by Treasure Data • Post from JavaScript/iOS/Android SDK • To EventCollector (HTTP endpoint for SDKs, impl. w/ Fluentd)
- 28. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 29. DataConnector • Data bulkloader for various data sources • Load customers' data to Treasure Data • S3, Redshift, MySQL, PostgreSQL, Salesforce, ... • Hosted Embulk • Much computing resources • Distributed execution on Hadoop MapReduce
- 30. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 31. Hadoop, Presto clusters •Some Hadoop/Presto clusters • We're OSS products itself, not customized one • with minimal patches for storage I/O
- 32. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 33. Queue/Worker, Scheduler • TreasureData: multi-tenant data analytics service • executes many jobs in shared clusters (queries, imports, ...) • CORE: queues-workers & schedulers • Clusters have queues/scheduler... it's not enough • resource limitations for each price plans • priority queues for job types • and many others
- 34.
- 35. PerfectQueue • Highly availabledistributed queue using RDBMS • Written in CRuby • Enqueue by INSERT INTO • Dequeue/Commit by UPDATE • Flexible scheduling rather than scalability • Using Amazon RDS (MySQL) internally • + Workers on EC2
- 36.
- 37. PerfectSched • Highly availabledistributed scheduler using RDBMS • Written in CRuby • At-least-one semantics • PerfectSched enqueues jobs into PerfectQueue
- 38. Storage, Schema • Anothercore technology for Treasure Data service • High performance, schema on read, less cost • columnar file format • high throughput & high concurrency • compression • Less schema management • for customers
- 39. Treasure Data Architecture:Overview Console API EventCollector PlazmaDB Worker Scheduler Hadoop Cluster Presto Cluster USERS TD SDKs SERVERS DataConnector CUSTOMER's SYSTEMS
- 40.
- 41. PlazmaDB • Distributed databaseusing RDBMS & Distributed FS • metadata on RDBMS, data chunks on DFS • Amazon RDS(PostgreSQL) + Amazon S3 / Riak CS • High throughput & high availability by S3 • Columnar format based on MessagePack • time based chunking for time series data
- 42. Monitoring • Using DataDogfor internal operations • Monitoring for our customers required: • How many records are they importing? • How many jobs are they executing? • How many threads/processes is a job consuming?
- 43.
- 44. PerfectMonitor • Is stillunder construction :P • Fluentd based metrics collection • Detailed metric for real-time, summarized for past • Real-time metric storage using InfluxDB • Historic metric storage using Treasure Data • Real-time data series are disposable :D • Potential next OSS product from Treasure Data
- 45. For Further improvement •More performance for more customers • Dynamic scaling for better performance and less cost • New analytics features for brand new experience
- 46. "Done is betterthan Perfect."
- 47.
- 48. We'll improve ourcode step by step, with improvements of ruby and developer community <3 Thanks!