Fluentd meetup logging infrastructure in paa s

Logging infrastructure in PaaS with Fluentd Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team 1 Rakuten. Inc,　22/08/2012

About us§  ささきと申します。日本人です。 §  ワリと申します。メキシコ人です。 (Sasaki, from Japan) (Wally, from Mexico)§  Living in Japan since 1982. §  Living in Tokyo since 2010.§  Tech Lead of the PaaS Dev&Ops §  Member of the PaaS Dev&Ops team team at Rakuten, Architect of at Rakuten, programming in Ruby. Rakuten PaaS. §  Emacs user (org-mode rocks, try .org§  Node.js user / node-fluentd-logger markup in Github)§  CouchDB, …etc §  OpenStreetMap (Mapnik rocks) 2

Agenda§  Cloud Foundry§  Cloud Foundry + Fluentd§  Fluentd Pros & Cons§  Misc 3

Cloud Foundry•  Open Application Platform as a Service project•  By VMware•  Written in Ruby•  OpenSource http://github.com/cloudfoundry/vcap•  Hosted version http://www.cloudfoundry.com/ 4

Why Cloud Foundry?•  Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html http://www.slideshare.net/rakutentech/cloud-foundry-10423477•  Main points: –  Open Source –  Separation of concerns from IaaS –  Scalable architecture –  Multi language 5

What Cloud Foundry does?•  Easy to deploy applications!! $ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3•  Not easy to deploy business ready applications yet. –  We are now solving our own issues by modifying Cloud Foundry source –  One critical thing is Logging! –  Now we’ve solved this matter basically, let’s share our case study. 6

Current logging issue in Cloud Foundry$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp! 7

Current logging issue in Cloud Foundry$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp! Oops, logs are gone. 8

Persisting logs is a must for business applications! •  Monitoring •  Analysis •  Visualization Ask a data scientist =>Sorry we losteverything :P http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist 9

Cloud Foundry does not persists the logs from apps… •  Enter Fluentd! 10

Brief overview of Cloud Foundry architecture Droplet Execution AgentSource: http://www.slideshare.net/derekcollison/design-of-cloud-foundry 11

ApproachAttach a Fluentd collector to one application instance upon dispatch. DEA app fluentd app fluentd Log Storage DEA app fluentd app fluentd 12

Collecting the logs from the appsDEA •  Log collector monitors changes in the files app fluentd –  Stdout, stderr logs in general –  Catalina, manager, host-manager logs for Java –  error.log for PHP apps app fluentd app fluentd •  Also makes available: –  VCAP_LOG_PORT env variable to flush logs directly app fluentd –  VCAP_LOG_STREAM_PORT to inspect the logs in realtime app fluentd 13

Collecting the logs from the appsDEA app fluentd Log StorageFluent::Logger => VCAP_LOG_PORTSends logs directly to Fluentd makingunnecessary having to write to file. VCAP_LOG_STREAM_PORT Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’} 14

Small setup for collecting the logsDEA CloudController app fluentd Log Storage Server app fluentd /logs/:path fluentd Rack app Collected log filesDEA Log Storage Server CloudController app fluentd Rack app app fluentd Collected log files /logs/:path 15

How is it setup?app/ !fluentd/! Attached to an application instance on |-- stderr.stat (1)! startup by DEA |-- stdout.stat (2)! •  (1) and (2) is a “stat file” used by the input plugin, |-- fluentd.conf (3)! which allows it to pick up from the last position it |-- startup (4)! had monitored. |-- stop (5)!fluentd_child.pid (6)! •  (3) is the configuration of the plugins dynamically generated by the DEAfluentd.pid (7)!logs/! •  (4) and (5) are the scripts executed to manage the |-- fluentd.log (8)! fluentd process |-- stdout.log! •  (6) and (7) are the supervisor and main process |-- stderr.log! ids from Fluentdruby!run.pid! •  (8) is where any logs produced by the logging daemon are storedstartup!stop! 16

fluentd.conf :: Input configuration! Set up VCAP_LOG_PORT to flush the logs<source>! directly type forward! port 12345!</source>!!<source>! type cf_app_logs! instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a! runtime ruby19! framework sinatra! tag logs.storage!</source>!!! Specify runtime and dir to monitor files 17

fluentd.conf :: Output configuration<match logs.**>! type copy! Settings for when forwarding <store>! to the log storage, we include user type cf_app_forward! information here so that it cannot be flush_interval 5s! overrided by a logging library. buffer_type file! buffer_path fluentd/fluentd_buffer!! cf_app_user foo@example.com! cf_app_name appname! cf_app_instance_id 3e9676485562c64d81a61de2df05903a! cf_app_instance_index 0! 18

fluentd.conf :: Output configuration <server>! name logserver! Location of the log storage server: host 192.168.2.1! -  We use a active-standby setup at the port 4224! moment. </server>! -  When primary Logserver is down, logs <server>! are forwarded to the one on standby. name backup_logserver! host 192.168.2.2! port 4224! When both are down, fluentd buffers to standby! files the logs locally until one of them is </server>! available. </store>! <store>! type cf_app_logs_streaming! port 43208! </store>!</match>! Set up VCAP_LOG_STREAM port to enable tail –f like support. 19

…And now that we managed to collect the logs,we also can support the following 2 commands: 20

vmc tail 21

vmc log-storage 22

What we could do•  Persisting the logs in Cloud Foundry is a must!•  Support to persist the logs is a key issue in adoption of Cloud Foundry for developing application services.•  Thanks to Fluentd we could enable users to have log archives in a straightforward way.•  Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy. 23

What we could not do•  It was difficult to let the user have access to the logs from Nginx (used as a Router in CF) # Sample release process! vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp! vmc map app-v1 olympics.rakuten.co.jp => Released!! vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp! vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps! vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.!•  Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user.•  Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs. 24

Fluentd: Pros and Cons•  Pros –  Easy to include in Cloud Foundry compared to other solutions •  Possible to include it like gem fluentd in a Ruby component –  Easy to change storage technology depending on scale –  Easy to extend –  Growing community o/•  Cons –  Fluentd needs better testing tools –  Better materials about Fluentd from the community? •  Learning curve is already low •  Just we need to share more about how we are using Fluentd •  e.g. The Little Fluentd Book, Advanced Fluentd Recipes’ –  Cool.io is not maintained 25

Misc: Playing with the message bus•  Cloud Foundry is built with Eventmachine.•  Fluentd is built with Cool.io.•  Cool.io and Eventmachine do not play along very well.•  The message bus from Cloud Foundry named NATS is also built using Eventmachine. 26

Misc: NATS input plugin# This works, though a warning is displayed from Cool.io overriding Eventmachine!require nats/client!!module Fluent! class NatsInput < Input! Plugin.register_input(nats, self)! def run! NATS.start(:uri => @nats_connection_uri) do!! # Triggered when an url changing event occurs (remap, change)! NATS.subscribe(dea.update) do |msg|! Engine.emit(logs.dea_update, Time.now.to_i, msg)! end!! # Triggered when an application is started or dea is started! NATS.subscribe(dea.*.start) do |msg|! Engine.emit(logs.dea_start, Time.now.to_i, msg)! end! end! end! 27

Misc: Adding new forwarding nodes dynamically through NATS?# DEA@droplets.each_value do |instance| begin instance_collector = Fluent::Logger::FluentLogger.new(logs, instance[:logging_ip], instance[:logging_port]) instance_collector.post(storage, {type: serverConf, host: 127.0.0.1, port: 23938 }) rescue => e @logger.error "Error: #{e} -- #{e.backtrace}" end end# Extended Forward pluginmodule Fluent class CloudfoundryAppForwardOutput < ForwardOutput Plugin.register_output(cf_app_forward, self) def emit(tag, es, chain) es.each do |emit_time, record| # Add a new forwarding server if record.has_key?(type) and record[type] == serverConf’ add_node(record[host], record[port’]) next end 28

FINYou can enable log collection to your Cloud Foundry by merging thispatch:https://github.com/rakutentech/dea/commit/64d271904de1e195cc90ca1958eadf704f530d94Checkout our github repo for more Cloud Foundry related things:https://github.com/rakutentech/That’s all, thanks!ご清聴ありがとうございました！ 29

http://seikoudoku2000.hatenablog.com	420
https://twitter.com	20
https://si0.twimg.com	13
https://twimg0-a.akamaihd.net	11
http://tweetedtimes.com	2
http://www.tumblr.com	2
http://d.hatena.ne.jp	1
http://webcache.googleusercontent.com	1

Fluentd meetup logging infrastructure in paa s

by Rakuten, Inc on Aug 27, 2012

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 470

Statistics

Fluentd meetup logging infrastructure in paa s Presentation Transcript