Fluentd in Co-Work

9,170 views

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
9,170
On SlideShare
0
From Embeds
0
Number of Embeds
6,244
Actions
Shares
0
Downloads
17
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Fluentd in Co-Work

  1. 1.      Fluentd in Fluentd  meetup  in  Fukuoka 2013/03/07 @Spring_MT
  2. 2. Profile twitter : @Spring_MT Company : 10xLab Engineer
  3. 3. AgendaFluentdin Co-Work appin Co-Work infra
  4. 4. SNS for Companies(like Yammer)During Closed β
  5. 5. InfraArchitecture
  6. 6. 10xlab  InfraALL AWS!Use VPCRuby, Fluentd and Chefin all servers gem install fluentd
  7. 7. f fluentd public subnet net work Gateway NAT SSLの変換 subnet subnet subnet utility subnet fluentd storage Co-Work deliver proxy APPWatch f f redis f f f worker varnishf DB(MySQL on EC2) f + sinatra f f f worker f f f f
  8. 8. Co-WorkWAF Ruby on RailsDB MySQL on EC2KVS redis
  9. 9. Fluentd inCo-Work  App
  10. 10. AppQueuingLogging
  11. 11. Queuing
  12. 12. QueuingResque Ruby based job queue liblary using github Multiprocessing Backend is redishttps://github.com/defunkt/resque
  13. 13. fluent-plugin-resque Fluentd output plugin for sending resque- queue to redis https://github.com/hotchpotch/fluent- plugin-resque
  14. 14. Why  Fluentd?
  15. 15. worker workerapp worker worker r r queue e e worker worker worker s sapp worker q q worker u u queue e e worker worker workerapp worker worker
  16. 16. worker workerapp worker worker X If queing fail, App should retry queuing. You write a retry processing?
  17. 17. worker workerapp worker f f worker l l u u e e r queue n n e worker worker t t worker - - sapp worker l r q worker o e u queue g s g q e worker worker e u worker r eapp worker worker
  18. 18. workerapp worker worker X X worker If queuing fail, Fluentd retry queuing ! Fluentd(and Logger) Buffering internally Retrying
  19. 19. Send  Queue  Data  to  Fluentd
  20. 20. Send  Resque-Queue  to  Redis
  21. 21. BreaKhttp://photo.v-colors.com/2111.html
  22. 22. Logging
  23. 23. LoggingAccess log(nginx)PV log(rails) Ref. WEB†DB PRESS vol.App Log(rails)
  24. 24. Why  Fluentd?
  25. 25. Log  FileTransfer Forwarding server is full m9(^Д^)プギャー Operate transfer script? Real-Time analysis ? Delete log file
  26. 26. Log  FileFormat and Analysis Disjointed format (CSV? TSV? LTSV? JSON?) Operate analysis script?
  27. 27. FluentdUnified Format(JSON)Real-Time transfer andHigh AvailabilityEasy Setup and EasyextensionWell Tested At Bigservices(○HN, ○ookpad)
  28. 28. Just  Do  It  ! http://d.hatena.ne.jp/tagomoris/20121029/1351491111
  29. 29. Just  Do  It  ! し ょ今 で http://d.hatena.ne.jp/tagomoris/20121029/1351491111
  30. 30. accessproxy log in_tail deliver S3 in_tail PV log Growth app app log logger Forecast
  31. 31. Analysis
  32. 32. copy out_route fluent-plugin-rewrite-tag-filteranalysis fluent-plugin-datacounter fluent-plugin-numeric-counter fluent-plugin-numeric-monitor fluent-plugin-amplifier-filter
  33. 33. Output
  34. 34. GrowthForecast fluent-plugin-growthforecast For visualize dataGrowthForecast Web API for drawing graph Perl
  35. 35. Treasure  Datafluent-plugin-tdFor data analysis(mainly access log)TreasureData Hadoop-based data store and analysis service
  36. 36. S3fluent-plugin-s3For backup logsS3 Data Store Service in AWS Eleven Nine
  37. 37. ikachan fluent-plugin-ikachan IRC notification(Ex.fatal log)ikachan Web API for IRC Perl
  38. 38. Analysis  in  TDSQL Like500 GB ($299/M)Analysis using Ruby!
  39. 39. Collect Lib
  40. 40. Fluentd inCo-Work  Infra
  41. 41. WatchFor resource watchFluentd collect dataand push data eachnotification tools.
  42. 42. fluent-plugin-mysqlslowquery Collect mysql slow query log Notify IRC through fluent-plugin-ikachan + ikachan
  43. 43. fluent-resque-statCollect Resque statuspending キューの残数processed 取り出したキューの総数queues 登録されてるキューの名前の総数workers キューを取り出すworkerの数working キューを処理中のworker数failed 失敗したキュー数servers resqueが動いてるサーバーのhostやportの情報 resque_statが動いているサーバーの RAILS_ENV orenvironment RACK_ENV(なければdevelopmentが入る)
  44. 44. fluent-resque-stat
  45. 45. Thank  You  !
  46. 46. Fluentd Setup
  47. 47. Install  and  SetupInstallattion ofFluentd is AWS cloud-init(user_data)Setting Fluentd ischef-sole andcapistrano
  48. 48. I Extend a little above systemfor specifying cookbook.$ cap -S subnet=fluentd -S hosts=xfdeliver001 chef
  49. 49. Launch  and  Monitoring  Processsupervisor

×