Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

San Antonio DevOps: Fluentd

503 views

Published on

An overview of fluentd for the San Antonio DevOps Meetup

Published in: Technology
  • Be the first to comment

  • Be the first to like this

San Antonio DevOps: Fluentd

  1. 1. Hart Hoover | @hhoover Josh O’Brien | @OBrienCommaJosh @SADevOps
  2. 2. fluentd
  3. 3. Log all the things to all the things!
  4. 4. …in JSON!
  5. 5. input&{& &&file&{& &&&&path&=>&"/var/log/http.log"& &&}& }& filter&{& &&grok&{& &&&&match&=>&[&"message",&"%{IP:client}&% {WORD:method}&%{URIPATHPARAM:request}&% {NUMBER:bytes}&%{NUMBER:duration}"&]& &&}& }
  6. 6. Fluentd Events Time Tag Record passed from source or adding a parsed time message routing in fluentd JSON
  7. 7. #&receive&events&via&HTTP&& <source>&& type&http&& port&8888&& </source>
  8. 8. #&read&logs&from&a&file&& <source>&& type&tail
 path&/var/log/httpd.log& format&apache
 tag&apache.access&& </source>
  9. 9. #&DOCKER&OMG& <source>& &&type&tail& &&format&json& &&path&/var/lib/docker/containers/ID/IDYjson.log& &&pos_file&/var/lib/docker/containers/ID/IDY json.log.pos& &&tag&docker.container& &&rotate_wait&5& &&read_from_head&true& </source>
  10. 10. #&save&alerts&to&a&file&& <match&alert.**>&& type&file&& path&/var/log/fluent/alerts&& </match>
  11. 11. #&save&access&logs&to&MongoDB&& <match&apache.access>&& type&mongo& database&apache& collection&log&& </match>&
  12. 12. #&Post&to&IRC& <match&**>& &&type&irc& &&host&localhost& &&port&6667& &&channel&fluentd& &&nick&fluentd& &&user&fluentd& &&real&fluentd& &&message&notice:&%s&[%s]&%s& &&out_keys&tag,time,message& &&time_key&time& &&time_format&%Y/%m/%d&%H:%M:%S& &&tag_key&tag& </match>
  13. 13. #&forward&other&logs&to&servers&& <match&**>&& type&forward& &&<server>&& &&&&host&192.168.0.11&& &&&&weight&20&& &&</server>& &&<server>&& &&&&host&192.168.0.12&& &&&&weight&60&& &&</server>
 </match>
  14. 14. #&Send&logs&to&ElasticSearch& <match&**>& &&type&elasticsearch& &&logstash_format&true& &&host&localhost& &&port&9200& &&index_name&fluentd& </match>
  15. 15. Client Libraries Ruby Python Java PHP Node.JS Scala
  16. 16. https://github.com/treasure-data/chef-td-agent Install with Chef Vagrantfile & kitchen.yml included!
  17. 17. https://github.com/dmytro/fluentd-cookbook Possibly Better https://gist.github.com/hhoover/4fceb09148a73f45136a
  18. 18. @SADevOps http://meetup.com/SanAntonioDevOps

×