Your SlideShare is downloading. ×
0
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Heka - Rob Miller
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Heka - Rob Miller

1,817

Published on

0 Comments
13 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,817
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
16
Comments
0
Likes
13
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Heka Unified Data Processing
  • 2. So. Much. Data.
  • 3. So. Much. Data. •Server level ops data •Process •Ops level data data / metrics •Business •Logging •Error data output reports / tracebacks
  • 4. So. Many. Tools. •collectd •statsd / tcollector / graphite / etc. •[r]syslog[-ng] •Logstash •Riemann •Nagios / Esper / other CEP / Zenoss
  • 5. One Basic Pattern •Acquire data •Transform •Output and/or Transport data data
  • 6. One Multi-Tool? What would it be like to build a tool to tackle this in the general case? Wins: •Fewer processes to manage •Increased client / configuration consistency •Processing shared across domains
  • 7. One Multi-Tool? Requirements: •Lightweight •Flexible •Easily and configurable extended
  • 8. I know, I know...
  • 9. BUT! Replacing even two services on each box is a net ops win. SCIENCE!
  • 10. How Heka Is Put Together
  • 11. Inputs •Listen •Just or fetch about the low level transport
  • 12. Splitters •Slice Inputs' raw data streams into discrete events •Text or binary protocols •Decouple protocols from their transports
  • 13. Decoders •Parse event data to populate a metadata envelope for all event types •Extract structure from unstructured data... •... or just wrap a blob •Sandbox-able (Lua)
  • 14. Router Simple, efficient grammar for matching messages: Type == "counter" && Payload == "1" Type == "applog" && Logger == "marketplace" Type == "alert" && (Severity==7 || Payload=="emergency") Type == "myapp.metric" && Fields[name] =~ /.*.stat/
  • 15. Filters •Watch flowing data •Generate output messages •Sandbox-able (Lua)
  • 16. Outputs •Deliver to external service... •… and/or to upstream Heka... •… and/or directly to Heka Dashboard UI •Configurable reconnect
  • 17. Sandboxes Are Fun! • Dynamically added to running Heka w/ no config changes, no restart ● CPU cycles and RAM usage monitored ● Misbehaving plugins are shut off
  • 18. Sandboxes Are Fun! • LPeg (parsing expression grammar) & JSON libraries for data parsing • Circular buffer library for time series data
  • 19. Sandboxes Are Fun! Circular buffers auto-generate dashboard graphs
  • 20. Try It Out https://github.com/mozilla-services/heka http://hekad.readthedocs.org https://mail.mozilla.org/listinfo/heka irc.mozilla.org, #heka rmiller@mozilla.com

×