• Save
Fluentd Plugins for CouchDB, Amazon SQS/SNS
Upcoming SlideShare
Loading in...5
×
 

Fluentd Plugins for CouchDB, Amazon SQS/SNS

on

  • 7,221 views

An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS.

An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS.

at "Fluentd Meetup in Tokyo."
http://www.zusaar.com/event/193104
#fluentd

Statistics

Views

Total Views
7,221
Views on SlideShare
5,256
Embed Views
1,965

Actions

Likes
6
Downloads
0
Comments
0

5 Embeds 1,965

http://ixixi.hateblo.jp 1956
http://a0.twimg.com 3
http://www.twylah.com 3
http://webcache.googleusercontent.com 2
http://us-w1.rockmelt.com 1

Accessibility

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • 先週CouchConfTokyoがあった.(100人規模)2割ぐらい外国人だったらしい.日本ではMongoが圧倒している感があるが、世界ではCouchDBも結構使われている.\n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Fluentd Plugins for CouchDB, Amazon SQS/SNS Fluentd Plugins for CouchDB, Amazon SQS/SNS Presentation Transcript

  • An Introduction to ... Fluentd plugins for CouchDB, Amazon SQS, Amazon SNS @ixixi
  • $ whoami@ixixi (Yudai Odagiri)Engineer of Future Architect, Inc. HadoopUstreamer CORSS, MongoTokyo2012, , GraphDB , Twitter , MongoDB ...... Ustream
  • Agenda CouchDB? fluent-plugin-couch demonstration Amazon SQS? fluent-plugin-sqs Amazon SNS? fluent-plugin-sns next features...
  • CouchDB? Relax!
  • CouchDB? Document-oriented database MVCC (MultiVersion Cuncurrency Control) Auto versioning (No “update”) RESTfull HTTP JSON APIs
  • CouchDB?Small footprint run on mobile devices ( android / iOS )Multi-master Replication Replication (via HTTP)
  • CouchDB?Written in ErlangNot only DB Server, but also Application &Web Server.Built-in web GUI “Futon”
  • MapReduce on CouchDB
  • CouchDB MapReduceUser-defined query structured as M/RMapReduce View (index) view index is built lazily View Definition (MapReduce) Result
  • CouchDB MapReduceIncremental view index update “possible to use M/R to query huge partitioned clusters in realtime, instead of having to wait for a whole M/R job to complete or having stale, occasionally updated indexes.” ~Damien Katz http://damienkatz.net/2008/02/incremental_map.htmlimplementation details { omitted } this is Lightning Talk :-P see : http://horicky.blogspot.com/2008/10/couchdb- implementation.html
  • fluent-plugin-couch
  • fluent-plugin-couchfluentd (buffered) output plugin store fluent-event as CouchDB Document to CouchDB install $ gem install fluent-plugin-couch <match couch.**> configuration type couch database fluent host ixixi.cloudant.com port 80 # for Basic Authentication (optional) user ixixi password ********** # Other buffer configurations.... </match>
  • Democount tweets in public timeline (use Streaming API)group by timezone and language in real-time. M/R view update (incremental) bulk insertStreaming API fluentd flush interval => 5sec. buffer
  • fluent-plugin-sqs
  • why need queue?to stop and update receiver applicationto consider the state of receiver servers Receiver servers ( on which heavy processes are running) may not be able to deal with messages. fluentd
  • Amazon SQS Amazon Simple Queue Service (SQS) Distributed Queue Service since 2006 ( the oldest service of AWS ) Automatic Replication in data centers High Availability High Scalability Secure
  • why SQS?Queue Server requires high availability Generally, high availability queue server is very expensiveAmazon SQS is inexpensive $0.000001( per request ) free tier (~100,000 requests/month) $~0.201 /GB (Data transfer out)
  • fluent-plugin-sqs fluentd (buffered) output plugin store fluent-event to Amazon SQS install $ gem install fluent-plugin-sqs configuration <match sqs.**> type sqs aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} queue_name fluent-queue sqs_endpoint sqs.ap-northeast-1.amazonaws.com # Other buffer configurations.... </match>
  • fluent-plugin-sns
  • Amazon SNS Amazon Simple Notification Service (SNS) multi-protocol e-mail,email(json),http(s),SMS,SQS(!!) Installation not required push-base architecture high availability very inexpensive
  • Amazon SNSManager,Publisher,Subscriber Model manager makes “topic” and set subscribers to “topic(s)” Publisher sends message to SNS topic, then SNS pushes message to subscribers
  • Amazon SNS SubscriberPublisher e-mail SNSfluentd SMS Subscriber Topic A send HTTP Subscriber Topic B Manager Subscriber
  • Amazon SNSPublisher(->fluentd) is not Manager. Should be written many subscriber information in “fluent.conf” ? ---- No.
  • fluent-plugin-sns fluentd output plugin send fluent-event to Amazon SQS install $ gem install fluent-plugin-sns configuration <match sns.**> type sns aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} sns_topic_name = fluentd-topic sqs_endpoint sqs.ap-northeast-1.amazonaws.com sns_subject_key title </match>
  • next features...?
  • next feature...?fluent-plugin-couch “update document” e.g.) To handle events such as updating the status of the server. can refer to previous documents. ∵ CouchDB is MVCC based , automatic versioning.fluent-plugin-sqs input-pluginfluent-plugin-sns customize formatting & multi formatting
  • thank you!