Your SlideShare is downloading. ×
Heka - Rob Miller
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Heka - Rob Miller

1,425
views

Published on


0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,425
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
13
Comments
0
Likes
11
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Heka Unified Data Processing
  • 2. So. Much. Data.
  • 3. So. Much. Data. •Server level ops data •Process •Ops level data data / metrics •Business •Logging •Error data output reports / tracebacks
  • 4. So. Many. Tools. •collectd •statsd / tcollector / graphite / etc. •[r]syslog[-ng] •Logstash •Riemann •Nagios / Esper / other CEP / Zenoss
  • 5. One Basic Pattern •Acquire data •Transform •Output and/or Transport data data
  • 6. One Multi-Tool? What would it be like to build a tool to tackle this in the general case? Wins: •Fewer processes to manage •Increased client / configuration consistency •Processing shared across domains
  • 7. One Multi-Tool? Requirements: •Lightweight •Flexible •Easily and configurable extended
  • 8. I know, I know...
  • 9. BUT! Replacing even two services on each box is a net ops win. SCIENCE!
  • 10. How Heka Is Put Together
  • 11. Inputs •Listen •Just or fetch about the low level transport
  • 12. Splitters •Slice Inputs' raw data streams into discrete events •Text or binary protocols •Decouple protocols from their transports
  • 13. Decoders •Parse event data to populate a metadata envelope for all event types •Extract structure from unstructured data... •... or just wrap a blob •Sandbox-able (Lua)
  • 14. Router Simple, efficient grammar for matching messages: Type == "counter" && Payload == "1" Type == "applog" && Logger == "marketplace" Type == "alert" && (Severity==7 || Payload=="emergency") Type == "myapp.metric" && Fields[name] =~ /.*.stat/
  • 15. Filters •Watch flowing data •Generate output messages •Sandbox-able (Lua)
  • 16. Outputs •Deliver to external service... •… and/or to upstream Heka... •… and/or directly to Heka Dashboard UI •Configurable reconnect
  • 17. Sandboxes Are Fun! • Dynamically added to running Heka w/ no config changes, no restart ● CPU cycles and RAM usage monitored ● Misbehaving plugins are shut off
  • 18. Sandboxes Are Fun! • LPeg (parsing expression grammar) & JSON libraries for data parsing • Circular buffer library for time series data
  • 19. Sandboxes Are Fun! Circular buffers auto-generate dashboard graphs
  • 20. Try It Out https://github.com/mozilla-services/heka http://hekad.readthedocs.org https://mail.mozilla.org/listinfo/heka irc.mozilla.org, #heka rmiller@mozilla.com