Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Fluentd meetup logging infrastructure in paa s


Published on

Rakuten, Yohei Sasaki, Waldemar Quevedo,
at Fluentd meetup, August 22nd, 2012

Published in: Technology, Education

Fluentd meetup logging infrastructure in paa s

  1. 1. Logging infrastructure in PaaS with Fluentd Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team 1 Rakuten. Inc, 22/08/2012
  2. 2. About us§  ささきと申します。日本人です。 §  ワリと申します。メキシコ人です。 (Sasaki, from Japan) (Wally, from Mexico)§  Living in Japan since 1982. §  Living in Tokyo since 2010.§  Tech Lead of the PaaS Dev&Ops §  Member of the PaaS Dev&Ops team team at Rakuten, Architect of at Rakuten, programming in Ruby. Rakuten PaaS. §  Emacs user (org-mode rocks, try .org§  Node.js user / node-fluentd-logger markup in Github)§  CouchDB, …etc §  OpenStreetMap (Mapnik rocks) 2
  3. 3. Agenda§  Cloud Foundry§  Cloud Foundry + Fluentd§  Fluentd Pros & Cons§  Misc 3
  4. 4. Cloud Foundry•  Open Application Platform as a Service project•  By VMware•  Written in Ruby•  OpenSource•  Hosted version 4
  5. 5. Why Cloud Foundry?•  Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten•  Main points: –  Open Source –  Separation of concerns from IaaS –  Scalable architecture –  Multi language 5
  6. 6. What Cloud Foundry does?•  Easy to deploy applications!! $ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3•  Not easy to deploy business ready applications yet. –  We are now solving our own issues by modifying Cloud Foundry source –  One critical thing is Logging! –  Now we’ve solved this matter basically, let’s share our case study. 6
  7. 7. Current logging issue in Cloud Foundry$ vmc push yssk22-myapp!!$ curl!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp! 7
  8. 8. Current logging issue in Cloud Foundry$ vmc push yssk22-myapp!!$ curl!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp! Oops, logs are gone. 8
  9. 9. Persisting logs is a must for business applications! •  Monitoring •  Analysis •  Visualization Ask a data scientist =>Sorry we losteverything :P 9
  10. 10. Cloud Foundry does not persists the logs from apps… •  Enter Fluentd! 10
  11. 11. Brief overview of Cloud Foundry architecture Droplet Execution AgentSource: 11
  12. 12. ApproachAttach a Fluentd collector to one application instance upon dispatch. DEA app fluentd app fluentd Log Storage DEA app fluentd app fluentd 12
  13. 13. Collecting the logs from the appsDEA •  Log collector monitors changes in the files app fluentd –  Stdout, stderr logs in general –  Catalina, manager, host-manager logs for Java –  error.log for PHP apps app fluentd app fluentd •  Also makes available: –  VCAP_LOG_PORT env variable to flush logs directly app fluentd –  VCAP_LOG_STREAM_PORT to inspect the logs in realtime app fluentd 13
  14. 14. Collecting the logs from the appsDEA app fluentd Log StorageFluent::Logger => VCAP_LOG_PORTSends logs directly to Fluentd makingunnecessary having to write to file. VCAP_LOG_STREAM_PORT Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’} 14
  15. 15. Small setup for collecting the logsDEA CloudController app fluentd Log Storage Server app fluentd /logs/:path fluentd Rack app Collected log filesDEA Log Storage Server CloudController app fluentd Rack app app fluentd Collected log files /logs/:path 15
  16. 16. How is it setup?app/ !fluentd/! Attached to an application instance on |-- stderr.stat (1)! startup by DEA |-- stdout.stat (2)! •  (1) and (2) is a “stat file” used by the input plugin, |-- fluentd.conf (3)! which allows it to pick up from the last position it |-- startup (4)! had monitored. |-- stop (5)! (6)! •  (3) is the configuration of the plugins dynamically generated by the (7)!logs/! •  (4) and (5) are the scripts executed to manage the |-- fluentd.log (8)! fluentd process |-- stdout.log! •  (6) and (7) are the supervisor and main process |-- stderr.log! ids from Fluentdruby!! •  (8) is where any logs produced by the logging daemon are storedstartup!stop! 16
  17. 17. fluentd.conf :: Input configuration! Set up VCAP_LOG_PORT to flush the logs<source>! directly type forward! port 12345!</source>!!<source>! type cf_app_logs! instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a! runtime ruby19! framework sinatra! tag!</source>!!! Specify runtime and dir to monitor files 17
  18. 18. fluentd.conf :: Output configuration<match logs.**>! type copy! Settings for when forwarding <store>! to the log storage, we include user type cf_app_forward! information here so that it cannot be flush_interval 5s! overrided by a logging library. buffer_type file! buffer_path fluentd/fluentd_buffer!! cf_app_user! cf_app_name appname! cf_app_instance_id 3e9676485562c64d81a61de2df05903a! cf_app_instance_index 0! 18
  19. 19. fluentd.conf :: Output configuration <server>! name logserver! Location of the log storage server: host! -  We use a active-standby setup at the port 4224! moment. </server>! -  When primary Logserver is down, logs <server>! are forwarded to the one on standby. name backup_logserver! host! port 4224! When both are down, fluentd buffers to standby! files the logs locally until one of them is </server>! available. </store>! <store>! type cf_app_logs_streaming! port 43208! </store>!</match>! Set up VCAP_LOG_STREAM port to enable tail –f like support. 19
  20. 20. …And now that we managed to collect the logs,we also can support the following 2 commands: 20
  21. 21. vmc tail 21
  22. 22. vmc log-storage 22
  23. 23. What we could do•  Persisting the logs in Cloud Foundry is a must!•  Support to persist the logs is a key issue in adoption of Cloud Foundry for developing application services.•  Thanks to Fluentd we could enable users to have log archives in a straightforward way.•  Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy. 23
  24. 24. What we could not do•  It was difficult to let the user have access to the logs from Nginx (used as a Router in CF) # Sample release process! vmc push app-v1 --runtime ruby19 =>! vmc map app-v1 => Released!! vmc push app-v2 --runtime ruby19 =>! vmc map app-v2 => Load balancing between apps! vmc unmap app-v1 => Release 2 complete.!•  Then the subdomain becomes available to other user.•  Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs. 24
  25. 25. Fluentd: Pros and Cons•  Pros –  Easy to include in Cloud Foundry compared to other solutions •  Possible to include it like gem fluentd in a Ruby component –  Easy to change storage technology depending on scale –  Easy to extend –  Growing community o/•  Cons –  Fluentd needs better testing tools –  Better materials about Fluentd from the community? •  Learning curve is already low •  Just we need to share more about how we are using Fluentd •  e.g. The Little Fluentd Book, Advanced Fluentd Recipes’ – is not maintained 25
  26. 26. Misc: Playing with the message bus•  Cloud Foundry is built with Eventmachine.•  Fluentd is built with• and Eventmachine do not play along very well.•  The message bus from Cloud Foundry named NATS is also built using Eventmachine. 26
  27. 27. Misc: NATS input plugin# This works, though a warning is displayed from overriding Eventmachine!require nats/client!!module Fluent! class NatsInput < Input! Plugin.register_input(nats, self)! def run! NATS.start(:uri => @nats_connection_uri) do!! # Triggered when an url changing event occurs (remap, change)! NATS.subscribe(dea.update) do |msg|! Engine.emit(logs.dea_update,, msg)! end!! # Triggered when an application is started or dea is started! NATS.subscribe(dea.*.start) do |msg|! Engine.emit(logs.dea_start,, msg)! end! end! end! 27
  28. 28. Misc: Adding new forwarding nodes dynamically through NATS?# DEA@droplets.each_value do |instance| begin instance_collector =, instance[:logging_ip], instance[:logging_port]), {type: serverConf, host:, port: 23938 }) rescue => e @logger.error "Error: #{e} -- #{e.backtrace}" end end# Extended Forward pluginmodule Fluent class CloudfoundryAppForwardOutput < ForwardOutput Plugin.register_output(cf_app_forward, self) def emit(tag, es, chain) es.each do |emit_time, record| # Add a new forwarding server if record.has_key?(type) and record[type] == serverConf’ add_node(record[host], record[port’]) next end 28
  29. 29. FINYou can enable log collection to your Cloud Foundry by merging thispatch: our github repo for more Cloud Foundry related things:’s all, thanks!ご清聴ありがとうございました! 29