Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Fluentd: Towards Unified Logging (SF Logging Meetup Jult 2014)

  • 1,329 views
Uploaded on

Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you......

Fluentd is an open source data collector to simplify and scale log management. In this talk, I gave a brief overview of Fluentd, focusing on the why: Why should you care about logging? Why do you want a unified logging layer? Why Fluentd?

More in: Software
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,329
On Slideshare
366
From Embeds
963
Number of Embeds
14

Actions

Shares
Downloads
2
Comments
0
Likes
2

Embeds 963

http://www.hakkalabs.co 871
https://www.hakkalabs.co 28
http://hakka.herokuapp.com 22
http://hakka-staging.herokuapp.com 8
http://www.slideee.com 6
https://hakka-staging.herokuapp.com 5
http://www.feedspot.com 5
http://g33ktalk.staging.wpengine.com 4
https://hakka.herokuapp.com 4
http://localhost 3
https://twitter.com 3
http://feedly.com 2
http://www.inoreader.com 1
http://g33ktalk.wpengine.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Fluentd: Towards Unified Logging Layer Kiyoto Tamura @kiyototamura Treasure Data, Inc.
  • 2. • Not an ops engineer myself • “marketing guy” • My laptop isn’t very hipster Why you shouldn’t listen to me
  • 3. • Fluentd maintainer and community manager • Former programmer • data nerd Why you should listen to me
  • 4. why Fluentd?
  • 5. I was a data scientist before it was cool to be one.
  • 6. http://cacm.acm.org/blogs/blog-cacm/169199-data-science-workflow- overview-and-challenges/fulltext
  • 7. Acquire Data (or so you think) WUT!? Invalid UTF8? Fix the encoding issue… Yell at the engineers Some columns are missing!? Run the script…DIVISION BY ZERO!!!
  • 8. Hmm…
  • 9. Logging.priority => :not_super_high
  • 10. analytics.priority => :very_high
  • 11. analytics.needs? :logs => true
  • 12. outage.priority => :super_high
  • 13. outage.needs? :logs => [“no”, “shit”]
  • 14. Unified Logging Layer
  • 15. overview
  • 16. 127.0.0.1 - - [05/Feb/2012:17:11:55 +0000] "GET / HTTP/1.1" 200 140 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 Safari/535.19"
  • 17. { "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }
  • 18. Parse as JSON!
  • 19. ?
  • 20. [“05/Feb/2012:17:11:55”,“web.access”,{ "host": "127.0.0.1", "user": "-", "method": "GET", "path": "/", "code": "200", "size": "140", "referer": "-", "agent": “Mozilla/5.0 (Windows…" }]
  • 21. ? web.mongodb web.file web.hdfs web.s3 web.mysql
  • 22. Fluentd Design Patterns (credit: @tagomoris’s talk at RedDotRubyConf)
  • 23. Real World Use Cases
  • 24. • Automate monitoring! • App and System metrics • JSON everywhere
  • 25. • 2000+ node • ~1B events/day • Forwarder- Aggregator
  • 26. Demo
  • 27. Fluentd Treasure Data Twitter API #NED #ARG Postgres + d3.js Collect Raw Tweets Tokenize + Count serve + visualize
  • 28. { “install”: “gem install fluentd”, “website”: “www.fluentd.org”, “github” : “fluent/fluentd”, “twitter”: “@fluentd” }