Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Source Software, Distributed Systems, Database as a Cloud Service

4,086 views

Published on

しまねOSS協議会 第106回オープンソースサロン・総会記念講演

Published in: Software
  • Be the first to comment

Open Source Software, Distributed Systems, Database as a Cloud Service

  1. 1. Open Source Software, Distributed Systems, Database as a Cloud Service 第106回オープンソースサロン・総会記念講演 Jul 29, 2016 Satoshi Tagomori (@tagomoris)
  2. 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
  3. 3. Topics • What is Treasure Data? • Who is tagomoris? • Treasure Data: Database as a Service • DB as a Service and Distributed Systems • Distributed Systems and Open Source Software • Open Source Software and Developers
  4. 4. http://rubybiz.jp/prize.html
  5. 5. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server 3 Complex ETL 4 End User System 2 Time consuming integration 1 Disparate data silos Without
 Treasure Data Advanced Analytics Reporting BI
  6. 6. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK With Treasure Data 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations Schema-flexible, Access via SQL, Unlimited Users, Queries
  7. 7. HQ Branch Matsue
  8. 8. Treasure Data, Inc. • Since Nov 2011 • Headquarters: Mountain View, CA, US • Japan Branch: Marunouchi, Chiyoda, Tokyo • Korea Branch: Gangnam, Seoul • Some remote workers - US, UK, Costa Rica
  9. 9. Developers in TD • Daily development in each offices • Communication over Internet • Slack, JIRA, Confluence & Zoom • Frontend Team: mainly in US • Console, Web services, etc • Backend Team: mainly in JP • Database, Distributed processing systems, etc
  10. 10. Satoshi "Moris" Tagomori (@tagomoris) Born in Matsue, Shimane Living in Tokyo from 1999
  11. 11. Started to work as an OSS developer 1. Asahi Net Internal system developer 2. NTT DATA Intellilink System consultant 3. livedoor - NHNJ - LINE Infrastructure engineer Data analytics platform engineer 4. Treasure Data Backend engineer OSS developer
  12. 12. @tagomoris as an Open Source Software Developer • Author • Norikra, Woothee, xbuild, Shib, Yabitz, Focuslight • Many fluent-plugin-* • And many libraries, tools, etc • Committer, Maintainer • Fluentd, MessagePack-Ruby, etc • Contributor • Docker (logging driver), etc
  13. 13. @tagomoris as an Open Source Software Developer • Talks • Many programming conferences (local, global) • Many small meetups • Articles • WEB+DB Magazine, Software Design • Many blog posts • Invented Event: ISUCON
  14. 14. OSS Developers in TD • MessagePack, Fluentd, Embulk & Digdag founder • Ruby committer • Ruby & JRuby committer • Fluentd & D-language committer • Hadoop/Spark contributor, pyenv author, ...
  15. 15. Why Are OSS Developers So Major in TD?
  16. 16. Treasure Data: Database as a Cloud Service
  17. 17. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
  18. 18. Database as a Cloud Service • Collect data • from remote site - customer side • Store/Process data • beyond cloud • Integrate data • to remote site - customer side
  19. 19. Two OSS Pattern in TD • OSS to collect/integrate data from/to remote site • OSS to store/process data
  20. 20. API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs Multi-Tenant Cloud Service 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
  21. 21. Make Input/Output Easy • Agent installed in our customers systems • OSS + Plugin to connect various systems • No barrier to use TD 1.Make a great OSS product to do it 2.Make it major 3.Potential customer already uses it :) • very easy to switch to use Treasure Data!
  22. 22. Multi-Tenant Cloud Service API Data M arts O DBC / JDBC Sensor ERP CRM RDBMS Mobile Web Server IoT Connectors Data Connectors JavaScript SDK Serverside collector Bulk Loader M obile SDK 3 Easy to Integrate 2 Zero Management 1 Easy to Collect 50+ Data Outputs 300+ Data Sources Advanced Analytics Reporting BI 50+Integrations
  23. 23. Database as a Service and Distributed Systems
  24. 24. Many Customers in a System • Share computer resource • Provide much more computer resource • Reduce total cost :-)
  25. 25. Big Data in a System • Manage big data from many customers • Manage computing power for many customers • Create a distributed system! • for fast query processor • for resource scheduler • for high availability
  26. 26. Distributed Systems and Open Source Software
  27. 27. Distributed Systems
  28. 28. Distributed System Software • Major software are all OSS • Hadoop, Presto, Kafka, Storm, ... • Concept and Implementation • MapReduce concept was from Google • Yahoo! engineers implemented it as Hadoop • Many others made Hadoop better • Data is always growing
 -> Software MUST be growing too
  29. 29. Deploying Distributed System • Many things make it hard to fix issues • Big data, many computers, complex queries, ... • We MUST fix our issues as soon as possible • for our customers • for our operation costs DO IT YOURSELF! → OSS
  30. 30. Updating Distributed System • It's very hard to update distributed systems • many servers, no data lost, no downtime, ... • Use OSS as-is without dirty fix • to keep it easy to upgrade "software" • Contribute your patch to community • to use patched mainstream software as-is
  31. 31. Open Source Software and Developers
  32. 32. DIY Policy Makes "Tech" Company • Do it yourself "At Your Own Risk": OSS • Taking risk: more OSS • OSS: more controllable than proprietary software • We can read/contribute source code :) • Technology problem: Can we take a risk? Or not?
  33. 33. Tech Company and Developers • Taking risk for business success:
 more focus on technology • Quality of OSS depends on each developers • Who is the committer of that product? • Who can review quality of that product? • Tech company needs great developers seriously!
  34. 34. OSS and Developers • "OSS Committer", not "OSS Committing Company" • the initiative by developer, not company • Commit log shows everything about common things • Who did contribute to that software? • Who did develop that feature? • Who did fix that problem? • People can know who is a good software engineer • it makes good developers happy!
  35. 35. Developers love OSS Company • OSS Company: a kind of Tech Companies • easy to find it: see committers/contributors • Developers love: • challenging "technical" tasks/issues to be solved • great coworkers, like committers of great software • nice salary brought by taking risk :P
  36. 36. Enjoy Engineering! MOST IMPORTANT THING: Thanks!

×