Your SlideShare is downloading. ×
Like loggly using open source
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Like loggly using open source

1,095
views

Published on

Streaming logs in cloud

Streaming logs in cloud


0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,095
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
12
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Stream your Cloud Thomas Alrin alrin@megam.co.in
  • 2. We’ll cover ● What to stream ● Choices for streaming ● Setting up streaming from a VM ● Chef Recipes
  • 3. What to stream You can stream the following from cloud ● Traces (Logs) ● Metrics ● Monitoring ● Status
  • 4. Scenario App/service runs in Cloud We need the log files of your App ● Web Server logs, container logs, app logs We need the log files of your Service ● Service logs
  • 5. SaaS Vendors You can avail this SaaS service from (loggly, papertrail..)
  • 6. We plan to build a streamee...
  • 7. Choices for streaming Logstash : logstash.net/ Fluentd : www.fluentd.org/ Beaver : github.com/josegonzalez/beaver Logstash-Forwarder : github.com/elasticsearch/logstash-forwarder Woodchuck : github.com/danryan/woodchuck RSYSLOG : http://rsyslog.com Heka : http://hekad.readthedocs.org/en/latest/
  • 8. Name Language Collector Shipper Footprint Ease of setting up Logstash JRuby (JVM) Yes No High > Easy Fluentd Ruby Yes No High > Easy Beaver Python No Yes Low Easy Logstash-Forwarder Go No Yes Low Difficult (uses SSL) Woodchuck Ruby No Yes High > Easy RSYSLOG C Yes Yes Low Difficult Heka Go Yes Yes Low Easy
  • 9. Our requirements 2 sets of logs to collect ● All the trace when the VM is spinned off. ● All the trace inside the VM of the application or service Publish it to an in-memory store(queue) which can be accessed by a key
  • 10. We tried We use Logstash Beaver Logstash-forwarder Woodchuck Heka RSYSLOG Heka Beaver RSYSLOG
  • 11. megamd fir.domain.com doe.domain.com gir.domain.com her.domain.com Queue#1 Queue#2 Queue#3 Queue#4 Shipper Agent howdy.log howdy_err.log howdy_err.log howdy_err.log howdy.log howdy_err.log howdy.log howdy.log AMQP /usr/share/mega m/megamd/logs
  • 12. How does it work ? Heka resides inside our Megam Engine (megamd). Its job is to collect the trace information when a VM is run. 1. Reads the dynamically created VM execution log files 2. Format the log contents in json for every VM execution. 3. Publish the log contents to a queue Beaver resides in each of the VMs. It does the following steps, 1. Reads the log files inside the VM 2. Format log contents in json. 3. Publish the log contents to a queue.
  • 13. Logstash ● Centralized logging frameworks that can transfer logs from multiple hosts to a central location. ● JRuby … hence its needs a JVM ● JVM sucks memory ● Logstash is Ideal as a centralized collector and not a shipper.
  • 14. Logstash Shipper Scenario Let us ship logs from a VM : /usr/share/megam/megamd/logs/*/* to Redis or AMQP. eg: ../megamd/logs/pogo.domain.com/howdy.log Queue named “pogo.domain.com” in AMQP. ../megamd/logs/doe.domain.com/howdy.log Queue named “doe.domain.com” in AMQP.
  • 15. Logstash Shipper - Sample conf input { file { type => "access-log" path => [ "/usr/local/share/megam/megamd/logs/*/*" ] } } filter { grok { type => "access-log" match => [ "@source_path", "(//usr/local/share/megam/megamd/logs/)(? <source_key>.+)(//*)" ] } } output { stdout { debug => true debug_format => "json"} redis { key => '%{source_key}' type => "access-log" data_type => "channel" host => "my_redis_server.com" } } Logs inside <source_key> directory are shipped to Redis key named <source_key> /opt/logstash/agent/etc$ sudo cat shipper.conf
  • 16. Logstash : Start the agent java -jar /opt/logstash/agent/lib/logstash-1.4.2. jar agent -f /opt/logstash/agent/etc/shipper.conf If you don’t have jre, then sudo apt-get install openjre-7-headless
  • 17. Heka ● Mozilla uses it internally. ● Written in Golang - native. ● Ideal as a centralized collector and a shipper. ● We picked Heka. ● Our modified version ○ https://github.com/megamsys/heka
  • 18. Installation Download deb from https://github.com/mozilla-services/heka/releases (or) build from source. git clone https://github.com/megamsys/heka.git cd heka source build.sh cd build make deb dpkg -i heka_0.6.0_amd64.deb
  • 19. Our Heka usage megamd Megam Engine Heka Rabbitmq logs Queue Realtime Streamer
  • 20. Heka configuration nano /etc/hekad.toml [TestWebserver] type = "LogstreamerInput" log_directory = "/usr/share/megam/heka/logs/" file_match = '(?P<DomainName>[^/]+)/(?P<FileName>[^/]+)' differentiator = ["DomainName", "_log"] [AMQPOutput] url = "amqp://guest:guest@localhost/" exchange = "test_tom" queue = true exchangeType = "fanout" message_matcher = 'TRUE' encoder = "JsonEncoder" [JsonEncoder] fields = [ "Timestamp", "Type", "Logger", "Payload", "Hostname" ]
  • 21. Run heka sudo hekad -config="/etc/hekad.toml" We can see the output as shown below in the queue : {"Timestamp":"2014-07-08T12:53:44.004Z","Type":"logfile","Logger":"tom.com_log","Payload":"TESTu000a"," Hostname":"alrin"}
  • 22. Beaver ● Beaver is a lightweight python log file shipper that is used to send logs to an intermediate broker for further processing ● Beaver is Ideal : When the VM does not have enough memory for a large JVM application to run as a shipper.
  • 23. Our Beaver usage Beaver VM#1 VM#2 VM#n megamd Megam Engine Heka Rabbitmq logs Queue Realtime Streamer Beaver Beaver
  • 24. Chef Recipe : Beaver When a VM is run, recipe(megam_logstash::beaver) is included. node.set['logstash']['key'] = "#{node.name}" node.set['logstash']['amqp'] = "#{node.name}_log" node.set['logstash']['beaver']['inputs'] = [ "/var/log/upstart/nodejs.log", "/var/log/upstart/gulpd.log" ] include_recipe "megam_logstash::beaver" attributes like (nodename, logfiles) are set dynamically.
  • 25. RSYSLOG RSYSLOG is the rocket-fast system for log processing. It offers high-performance, great security features and a modular design. Megam uses RSYSLOG to ship logs from VMs to Elasticsearch
  • 26. Chef Recipe : Rsyslog When a VM is run, recipe(megam_logstash::rsyslog) is included. node.set['rsyslog']['index'] = "#{node.name}" node.set['rsyslog']['elastic_ip'] = "monitor.megam.co.in" node.set['rsyslog']['input']['files'] = [ "/var/log/upstart/nodejs.log", "/var/log/upstart/gulpd.log" ] include_recipe "megam_logstash::rsyslog" attributes like (nodename, logfiles) are set dynamically.
  • 27. For more details http://www.gomegam.com email : gomegam@megam.co.in twitter: @megamsystems