Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
An Introduction to ...    Fluentd plugins for               CouchDB,            Amazon SQS,            Amazon SNS         ...
$ whoami@ixixi (Yudai Odagiri)Engineer of Future Architect, Inc.  HadoopUstreamer       CORSS, MongoTokyo2012,            ...
Agenda  CouchDB?   fluent-plugin-couch     demonstration  Amazon SQS?   fluent-plugin-sqs  Amazon SNS?   fluent-plugin-sns  n...
CouchDB?           Relax!
CouchDB? Document-oriented database MVCC (MultiVersion Cuncurrency Control)  Auto versioning (No “update”) RESTfull HTTP J...
CouchDB?Small footprint run on mobile devices ( android / iOS )Multi-master Replication                  Replication (via ...
CouchDB?Written in ErlangNot only DB Server, but also Application &Web Server.Built-in web GUI “Futon”
MapReduce on CouchDB
CouchDB MapReduceUser-defined query structured as M/RMapReduce View (index) view index is built lazily                  Vie...
CouchDB MapReduceIncremental view index update  “possible to use M/R to query huge partitioned clusters in   realtime, ins...
fluent-plugin-couch
fluent-plugin-couchfluentd (buffered) output plugin store fluent-event as CouchDB Document to CouchDB install $ gem install ...
Democount tweets in public timeline (use Streaming API)group by timezone and language in real-time.                       ...
fluent-plugin-sqs
why need queue?to stop and update receiver applicationto consider the state of receiver servers Receiver servers ( on whic...
Amazon SQS Amazon Simple Queue Service (SQS) Distributed Queue Service since 2006 ( the oldest service of AWS ) Automatic ...
why SQS?Queue Server requires high availability Generally, high availability queue server is very expensiveAmazon SQS is i...
fluent-plugin-sqs fluentd (buffered) output plugin  store fluent-event to Amazon SQS  install  $ gem install fluent-plugin-sq...
fluent-plugin-sns
Amazon SNS Amazon Simple Notification Service (SNS) multi-protocol  e-mail,email(json),http(s),SMS,SQS(!!) Installation not...
Amazon SNSManager,Publisher,Subscriber Model manager makes “topic” and set subscribers to “topic(s)” Publisher sends messa...
Amazon SNS                                           SubscriberPublisher                         e-mail                   ...
Amazon SNSPublisher(->fluentd) is not Manager. Should be written many subscriber information in “fluent.conf” ? ---- No.
fluent-plugin-sns fluentd output plugin  send fluent-event to Amazon SQS  install  $ gem install fluent-plugin-sns  configurat...
next features...?
next feature...?fluent-plugin-couch  “update document”    e.g.) To handle events such as updating the status of the server....
thank you!
Upcoming SlideShare
Loading in …5
×

Fluentd Plugins for CouchDB, Amazon SQS/SNS

9,598 views

Published on

An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS.

at "Fluentd Meetup in Tokyo."
http://www.zusaar.com/event/193104
#fluentd

Published in: Technology, Art & Photos
  • Be the first to comment

Fluentd Plugins for CouchDB, Amazon SQS/SNS

  1. 1. An Introduction to ... Fluentd plugins for CouchDB, Amazon SQS, Amazon SNS @ixixi
  2. 2. $ whoami@ixixi (Yudai Odagiri)Engineer of Future Architect, Inc. HadoopUstreamer CORSS, MongoTokyo2012, , GraphDB , Twitter , MongoDB ...... Ustream
  3. 3. Agenda CouchDB? fluent-plugin-couch demonstration Amazon SQS? fluent-plugin-sqs Amazon SNS? fluent-plugin-sns next features...
  4. 4. CouchDB? Relax!
  5. 5. CouchDB? Document-oriented database MVCC (MultiVersion Cuncurrency Control) Auto versioning (No “update”) RESTfull HTTP JSON APIs
  6. 6. CouchDB?Small footprint run on mobile devices ( android / iOS )Multi-master Replication Replication (via HTTP)
  7. 7. CouchDB?Written in ErlangNot only DB Server, but also Application &Web Server.Built-in web GUI “Futon”
  8. 8. MapReduce on CouchDB
  9. 9. CouchDB MapReduceUser-defined query structured as M/RMapReduce View (index) view index is built lazily View Definition (MapReduce) Result
  10. 10. CouchDB MapReduceIncremental view index update “possible to use M/R to query huge partitioned clusters in realtime, instead of having to wait for a whole M/R job to complete or having stale, occasionally updated indexes.” ~Damien Katz http://damienkatz.net/2008/02/incremental_map.htmlimplementation details { omitted } this is Lightning Talk :-P see : http://horicky.blogspot.com/2008/10/couchdb- implementation.html
  11. 11. fluent-plugin-couch
  12. 12. fluent-plugin-couchfluentd (buffered) output plugin store fluent-event as CouchDB Document to CouchDB install $ gem install fluent-plugin-couch <match couch.**> configuration type couch database fluent host ixixi.cloudant.com port 80 # for Basic Authentication (optional) user ixixi password ********** # Other buffer configurations.... </match>
  13. 13. Democount tweets in public timeline (use Streaming API)group by timezone and language in real-time. M/R view update (incremental) bulk insertStreaming API fluentd flush interval => 5sec. buffer
  14. 14. fluent-plugin-sqs
  15. 15. why need queue?to stop and update receiver applicationto consider the state of receiver servers Receiver servers ( on which heavy processes are running) may not be able to deal with messages. fluentd
  16. 16. Amazon SQS Amazon Simple Queue Service (SQS) Distributed Queue Service since 2006 ( the oldest service of AWS ) Automatic Replication in data centers High Availability High Scalability Secure
  17. 17. why SQS?Queue Server requires high availability Generally, high availability queue server is very expensiveAmazon SQS is inexpensive $0.000001( per request ) free tier (~100,000 requests/month) $~0.201 /GB (Data transfer out)
  18. 18. fluent-plugin-sqs fluentd (buffered) output plugin store fluent-event to Amazon SQS install $ gem install fluent-plugin-sqs configuration <match sqs.**> type sqs aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} queue_name fluent-queue sqs_endpoint sqs.ap-northeast-1.amazonaws.com # Other buffer configurations.... </match>
  19. 19. fluent-plugin-sns
  20. 20. Amazon SNS Amazon Simple Notification Service (SNS) multi-protocol e-mail,email(json),http(s),SMS,SQS(!!) Installation not required push-base architecture high availability very inexpensive
  21. 21. Amazon SNSManager,Publisher,Subscriber Model manager makes “topic” and set subscribers to “topic(s)” Publisher sends message to SNS topic, then SNS pushes message to subscribers
  22. 22. Amazon SNS SubscriberPublisher e-mail SNSfluentd SMS Subscriber Topic A send HTTP Subscriber Topic B Manager Subscriber
  23. 23. Amazon SNSPublisher(->fluentd) is not Manager. Should be written many subscriber information in “fluent.conf” ? ---- No.
  24. 24. fluent-plugin-sns fluentd output plugin send fluent-event to Amazon SQS install $ gem install fluent-plugin-sns configuration <match sns.**> type sns aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} sns_topic_name = fluentd-topic sqs_endpoint sqs.ap-northeast-1.amazonaws.com sns_subject_key title </match>
  25. 25. next features...?
  26. 26. next feature...?fluent-plugin-couch “update document” e.g.) To handle events such as updating the status of the server. can refer to previous documents. ∵ CouchDB is MVCC based , automatic versioning.fluent-plugin-sqs input-pluginfluent-plugin-sns customize formatting & multi formatting
  27. 27. thank you!

×