This document summarizes Satoshi Tagomori's presentation on Treasure Data, a data analytics service company. It discusses Treasure Data's use of Ruby for various components of its platform including its logging (Fluentd), ETL (Embulk), scheduling (PerfectSched), and storage (PlazmaDB) technologies. The document also provides an overview of Treasure Data's architecture including how it collects, stores, processes, and visualizes customer data using open source tools integrated with services like Hadoop and Presto.

1. Data Analytics Service Company
and Its Ruby Usage
EuRuKo 2015 (Oct 17, 2015)
Satoshi Tagomori (@tagomoris)

2. Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, MessagePack-Ruby, Norikra, ...
Treasure Data, Inc.

4. HQ Branch

5. http://www.treasuredata.com/

7. Data Analytics Platform
Data Analytics Service

10. Services

11. Services
JVM

12. Services
JVMC++

13. Data Analytics Flow
Collect Store Process Visualize
Data source
Reporting
Monitoring

14. Data Analytics Flow
Collect Store Process Visualize
Data source
Reporting
Monitoring

15. Data Analytics Platform
• Data collection, storage
• Console & API endpoints
• Schema management
• Processing (batch, query, ...)
• Queuing & Scheduling
• Data connector/exporter

16. Treasure Data Internals

17. Data Analytics Platform
• Data collection, storage: Ruby(OSS), Java/JRuby(OSS)
• Console & API endpoints: Ruby(RoR)
• Schema management: Ruby/Java (MessagePack)
• Processing (batch, query, ...): Java(Hadoop,Presto)
• Queuing & Scheduling: Ruby(OSS)
• Data connector/exporter: Java, Java/JRuby(OSS)

18. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

19. OSS products
• To make logging more easy & simple than ever!
• Plugin system
• Open development
• For various environment/usage
• Fluentd, Fluent-Bit, Embulk
• Fluent-Bit: Data collector for Embedded Linux
http://fluentbit.io/

20. http://www.fluentd.org/
Fluentd
Unified Logging Layer
For Stream Data
Written in CRuby
http://www.slideshare.net/treasure-data/the-basics-of-fluentd-35681111

22. Bulk Data Loader
High Throughput&Reliability
Embulk
Written in Java/JRuby
http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed
http://www.embulk.org/

23. HDFS
MySQL
Amazon S3
CSV Files
SequenceFile
Salesforce.com
Elasticsearch
Cassandra
Hive
Redis
✓ Parallel execution
✓ Data validation
✓ Error recovery
✓ Deterministic behavior
✓ Idempotent retrying
Plugins Plugins
bulk load

24. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

25. Console/API
• RoR + AWS RDS + AngularJS
• on EC2 (API) and Heroku (Console)
• Operation, Configuration & Managing Data

26. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

27. Collecting Data
• Import over Console/API
• From browsers and CLI (TD toolbelt)
• Treasure Agent (rpm/deb)
• Fluentd packaged by Treasure Data
• Post from JavaScript/iOS/Android SDK
• To EventCollector (HTTP endpoint for SDKs, impl. w/ Fluentd)

28. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

29. DataConnector
• Data bulk loader for various data sources
• Load customers' data to Treasure Data
• S3, Redshift, MySQL, PostgreSQL, Salesforce, ...
• Hosted Embulk
• Much computing resources
• Distributed execution on Hadoop MapReduce

30. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

31. Hadoop, Presto clusters
• Some Hadoop/Presto clusters
• We're OSS products itself, not customized one
• with minimal patches for storage I/O

32. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

33. Queue/Worker, Scheduler
• Treasure Data: multi-tenant data analytics service
• executes many jobs in shared clusters (queries,
imports, ...)
• CORE: queues-workers & schedulers
• Clusters have queues/scheduler... it's not enough
• resource limitations for each price plans
• priority queues for job types
• and many others

34. PerfectQueue
https://github.com/treasure-data/perfectqueue

35. PerfectQueue
• Highly available distributed queue using RDBMS
• Written in CRuby
• Enqueue by INSERT INTO
• Dequeue/Commit by UPDATE
• Flexible scheduling rather than scalability
• Using Amazon RDS (MySQL) internally
• + Workers on EC2

36. PerfectSched
https://github.com/treasure-data/perfectsched

37. PerfectSched
• Highly available distributed scheduler using RDBMS
• Written in CRuby
• At-least-one semantics
• PerfectSched enqueues jobs into PerfectQueue

38. Storage, Schema
• Another core technology for Treasure Data service
• High performance, schema on read, less cost
• columnar file format
• high throughput & high concurrency
• compression
• Less schema management
• for customers

39. Treasure Data Architecture: Overview
Console
API
EventCollector
PlazmaDB
Worker
Scheduler
Hadoop
Cluster
Presto
Cluster
USERS
TD SDKs
SERVERS
DataConnector
CUSTOMER's
SYSTEMS

40. PlazmaDB
http://www.slideshare.net/treasure-data/td-techplazma

41. PlazmaDB
• Distributed database using RDBMS & Distributed FS
• metadata on RDBMS, data chunks on DFS
• Amazon RDS(PostgreSQL) + Amazon S3 / Riak CS
• High throughput & high availability by S3
• Columnar format based on MessagePack
• time based chunking for time series data

42. Monitoring
• Using DataDog for internal operations
• Monitoring for our customers required:
• How many records are they importing?
• How many jobs are they executing?
• How many threads/processes is a job consuming?

43. PerfectMonitor

44. PerfectMonitor
• Is still under construction :P
• Fluentd based metrics collection
• Detailed metric for real-time, summarized for past
• Real-time metric storage using InfluxDB
• Historic metric storage using Treasure Data
• Real-time data series are disposable :D
• Potential next OSS product from Treasure Data

45. For Further improvement
• More performance for more customers
• Dynamic scaling for better performance and less
cost
• New analytics features for brand new experience

46. "Done is better than Perfect."

47. DoneQueue?

48. We'll improve our code step by step,
with improvements of ruby and developer
community <3
Thanks!