Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BigQuery, Fluentd and tagomoris #gcpja


Published on

Published in: Technology
  • Sex in your area is here: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating for everyone is here: ❤❤❤ ❤❤❤
    Are you sure you want to  Yes  No
    Your message goes here

BigQuery, Fluentd and tagomoris #gcpja

  1. 1. BigQuery, Fluentd, and tagomoris gcp ja night #28 2014/09/16 TAGOMORI Satoshi(tagomoris)
  2. 2. Satoshi Tagomori (@tagomoris) LINE Corporation Analytics Platform Team
  3. 3. スーパードゥラアア アアァァァァァァア アアアアアアアァァ イイイイイイイイイイ イイイイイイィィィィ ィイイイイイイイイィ イィイイイイィイィ
  4. 4. 極度乾燥 (しなさい)。
  5. 5. BigQuery What all of us love :)
  6. 6. Fluentd It is that with cute logo :)
  7. 7. Fluentd Readable configuration Flexible buffer system Various input/output plugins Very simple/easy plugin system High performance for many uses
  8. 8. fluent-plugin-bigquery Insert events from Fluentd into BigQuery over streaming inserts Table Sharding inserts
  9. 9. versions v0.0.x ~ v0.1.x by @tagomoris many patches from @yugui v0.2.x KAIZEN platform Inc (@naoya_ito)
  10. 10. tagomoris: I want new maintainer for fluent-plugin-bigquery, who uses BQ actually....
  11. 11. naoya_ito: OK, KAIZEN platform will do!
  12. 12. tagomoris: Great! I’ll transfer my repository to KAIZEN’s account... Can I have commit bit for Fluentd related fixes?
  13. 13. naoya_ito: Sure!
  14. 14. ( д) ゚ ゚
  15. 15. Disclosure: I’m an employee of LINE now!
  16. 16. performance Use Table Sharding inserts tables table1,table2,table3,table4 Use many threads for concurrent insertion num_threads 4 (Same with tables)
  17. 17. Features Authentication auth_method compute_engine auth_method private_key Schema specs per fields schema_path fetch_schema
  18. 18. API Quota number of records over streaming inserts 10,000 rows per sec per table 10MB per sec per table Use Table Sharding Max row size: 20KB Max data size per insert: 1MB Max rows per request: 500
  19. 19. Hobby programming & Cloud services For hobby programming To setup environments is very troublesome... Cloud services are easy to use! Cloud service specific limitations/restrictions are funny to play with!
  20. 20. Enjoy!