Fluentd Plugins for CouchDB, Amazon SQS/SNS

8,594 views
8,484 views

Published on

An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS.

at "Fluentd Meetup in Tokyo."
http://www.zusaar.com/event/193104
#fluentd

Published in: Technology, Art & Photos
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,594
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
0
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • 先週CouchConfTokyoがあった.(100人規模)2割ぐらい外国人だったらしい.日本ではMongoが圧倒している感があるが、世界ではCouchDBも結構使われている.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Fluentd Plugins for CouchDB, Amazon SQS/SNS

    1. 1. An Introduction to ... Fluentd plugins for CouchDB, Amazon SQS, Amazon SNS @ixixi
    2. 2. $ whoami@ixixi (Yudai Odagiri)Engineer of Future Architect, Inc. HadoopUstreamer CORSS, MongoTokyo2012, , GraphDB , Twitter , MongoDB ...... Ustream
    3. 3. Agenda CouchDB? fluent-plugin-couch demonstration Amazon SQS? fluent-plugin-sqs Amazon SNS? fluent-plugin-sns next features...
    4. 4. CouchDB? Relax!
    5. 5. CouchDB? Document-oriented database MVCC (MultiVersion Cuncurrency Control) Auto versioning (No “update”) RESTfull HTTP JSON APIs
    6. 6. CouchDB?Small footprint run on mobile devices ( android / iOS )Multi-master Replication Replication (via HTTP)
    7. 7. CouchDB?Written in ErlangNot only DB Server, but also Application &Web Server.Built-in web GUI “Futon”
    8. 8. MapReduce on CouchDB
    9. 9. CouchDB MapReduceUser-defined query structured as M/RMapReduce View (index) view index is built lazily View Definition (MapReduce) Result
    10. 10. CouchDB MapReduceIncremental view index update “possible to use M/R to query huge partitioned clusters in realtime, instead of having to wait for a whole M/R job to complete or having stale, occasionally updated indexes.” ~Damien Katz http://damienkatz.net/2008/02/incremental_map.htmlimplementation details { omitted } this is Lightning Talk :-P see : http://horicky.blogspot.com/2008/10/couchdb- implementation.html
    11. 11. fluent-plugin-couch
    12. 12. fluent-plugin-couchfluentd (buffered) output plugin store fluent-event as CouchDB Document to CouchDB install $ gem install fluent-plugin-couch <match couch.**> configuration type couch database fluent host ixixi.cloudant.com port 80 # for Basic Authentication (optional) user ixixi password ********** # Other buffer configurations.... </match>
    13. 13. Democount tweets in public timeline (use Streaming API)group by timezone and language in real-time. M/R view update (incremental) bulk insertStreaming API fluentd flush interval => 5sec. buffer
    14. 14. fluent-plugin-sqs
    15. 15. why need queue?to stop and update receiver applicationto consider the state of receiver servers Receiver servers ( on which heavy processes are running) may not be able to deal with messages. fluentd
    16. 16. Amazon SQS Amazon Simple Queue Service (SQS) Distributed Queue Service since 2006 ( the oldest service of AWS ) Automatic Replication in data centers High Availability High Scalability Secure
    17. 17. why SQS?Queue Server requires high availability Generally, high availability queue server is very expensiveAmazon SQS is inexpensive $0.000001( per request ) free tier (~100,000 requests/month) $~0.201 /GB (Data transfer out)
    18. 18. fluent-plugin-sqs fluentd (buffered) output plugin store fluent-event to Amazon SQS install $ gem install fluent-plugin-sqs configuration <match sqs.**> type sqs aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} queue_name fluent-queue sqs_endpoint sqs.ap-northeast-1.amazonaws.com # Other buffer configurations.... </match>
    19. 19. fluent-plugin-sns
    20. 20. Amazon SNS Amazon Simple Notification Service (SNS) multi-protocol e-mail,email(json),http(s),SMS,SQS(!!) Installation not required push-base architecture high availability very inexpensive
    21. 21. Amazon SNSManager,Publisher,Subscriber Model manager makes “topic” and set subscribers to “topic(s)” Publisher sends message to SNS topic, then SNS pushes message to subscribers
    22. 22. Amazon SNS SubscriberPublisher e-mail SNSfluentd SMS Subscriber Topic A send HTTP Subscriber Topic B Manager Subscriber
    23. 23. Amazon SNSPublisher(->fluentd) is not Manager. Should be written many subscriber information in “fluent.conf” ? ---- No.
    24. 24. fluent-plugin-sns fluentd output plugin send fluent-event to Amazon SQS install $ gem install fluent-plugin-sns configuration <match sns.**> type sns aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} sns_topic_name = fluentd-topic sqs_endpoint sqs.ap-northeast-1.amazonaws.com sns_subject_key title </match>
    25. 25. next features...?
    26. 26. next feature...?fluent-plugin-couch “update document” e.g.) To handle events such as updating the status of the server. can refer to previous documents. ∵ CouchDB is MVCC based , automatic versioning.fluent-plugin-sqs input-pluginfluent-plugin-sns customize formatting & multi formatting
    27. 27. thank you!

    ×