Downloaded 107 times
Uploaded byTomas Doran
Zero mq logs
This document discusses using ZeroMQ and Elasticsearch for log aggregation. It proposes using ZeroMQ to transmit structured log data from application servers to a central Logstash server, which would then insert the logs into Elasticsearch for querying and analysis. This approach aims to provide a lightweight logging solution that doesn't block application servers like traditional logging to databases can. The document also provides background on tools like Logstash, Elasticsearch, and Splunk.
More Related Content
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Zero mq logs
- 1. Using ZeroMQ and Elasticsearchfor log aggregation Tomas Doran (t0m) <bobtfish@bobtfish.net>
- 2. Who are you? •CPAN Developer • Catalyst core team • Moose hacker • AnyEvent::RabbitMQ user • Ruby/Python/C as needed • Dayjob - deveverthingops - state51 • 3/4 Pb of MogileFS - online music • Thousands of steams a second • Lots of perl. • Lots of servers • Lots of services
- 3. Sorry! • This isn’ta ZeroMQ tutotial • This isn’t an ElasticSearch tutorial
- 4. Debugging production • Ishard! • Especially interactions • Need to cross-correlate logs.
- 5. Naïve solution • “Letslog into the database” • NO NO NO NO • 120 lines/s (7200 disk) • 167 lines/s (10k disk) • 250 lines/s (15k disk)
- 6. Less naïve solution •Queue before we log • Bulk insert • No good for unstructured data • No good for many different structures
- 7. Still a stupidsolution • Lots of UNION queries • OR epic multi-way JOIN • Adding new data types HARD
- 8. Splunk • Splunk isenterprise software used to monitor, report and analyze the data produced by the applications, systems and infrastructure to run a business. -Wikidpedia
- 9. Splunk • Splunk isenterprise software used to monitor, report and analyze the data produced by the applications, systems and infrastructure to run a business. -Wikidpedia
- 10.
- 11. Splunk • Small agentprogram on each host you tell about your log files - ships to server • Server component analyzes / indexes your logs. Also a syslog server. • Builds structure from your data - in a GUI.
- 12. Splunk • Splunk isamazing. • You just tip logs into it, structure later. • If you can afford the license, use it, be happy!
- 13. I cannot affordsplunk
- 14. I cannot affordsplunk
- 15. I cannot affordsplunk • Sad panda! • Also, splunk isn’t extensible - black box. • So, a guy called Jorden Sissel invented:
- 16.
- 17. Diversion - ElasticSearch • Just tip JSON documents into it (with a type) • Figures out structure for each type, indexes appropriately. • Free sharding and replication
- 18. So • We post-processlogs to be somewhat structured. • We can then search over them (fast!) • Free text (for text fields) • Numeric • Dates + ranges
- 19. New types • Trivial! •Just emit it, it’s indexed and queryable. • Can hint elasticsearch for better queries (if needed)
- 20. Logstash InJRuby, by Jordan Sissel Input Simple: Filter Output Flexible Extensible Plays well with others Pre-built web interface
- 21.
- 22.
- 23.
- 24.
- 25.
- 26. Logstash IS MASSIVE
- 27.
- 28.
- 29.
- 30.
- 31. Java (JRuby) decoding AMQP is, however much much faster than perl doing that... JVM+-
- 32. Logstash on eachhost is totally out... • Running it on ElasticSearch servers which are already dedicated to this is fine.. • I’d still like to reuse all of it’s parsing
- 33. Lots of mydata is already JSON • Log::Message::Structured • AnyEvent::RabbitMQ • App logging relies on Rabbit being up • Can get slow if rabbit is sick and blocks
- 34. How about thisthen?
- 35. But not inthe right format.. • So I can write a munger in ruby... • Or I can write one in perl. • I’m already (going to be) running a collection / aggregation daemon on each host (for apache logs).
- 36.
- 37. Myxomatosis • If RabbitMQgets really sick, app slows down. • Multiple exponential backoffs • AMQP is crap for ‘fire and forget’
- 38. Syslog • Yes, Icould. But JSON in syslog - just no. • 1024 bytes - UDP packet. • Not inventing my own protocol!
- 39. ZeroMQ has the correct semantics • Pub/Sub sockets • Never, ever blocking • Lossy! (If needed) • Buffer sizes / locations configureable • Arbitrary message size
- 40. Subset of logstash •In perl • ZeroMQ receiver • Per host aggregation • Push AMQP to RabbitMQ • Run logstash on a central server
- 41. Subset of logstash •Small async process. ZMQ receive socket. • Pull JSON from ZMQ, decode, munge, emit back to AMQP • Slowness no longer blocks app servers
- 42. Subset of logstash •Use logstash at the other end to pop AMQP and insert into ElasticSearch • Keep per-host cost small • Same process can tail logfiles and cast into AMQP • Reuse all the logstash parsing (at server side) for apache logs etc
- 43. 100% drop in compatiblesubset of logstash
- 44. 100% drop in compatible subset of logstash In perl - making it easy for you to emit structured app events as JSON. Everything is down - your app is still up (you lose some logs)
- 45.
- 46.
- 47. 24Mb I used Moose- RAM use can and will go down.
- 48.
- 49.
- 50. Yes, yes -I know • The web app is fugly • Other people already have alternate implementations • Keeping interoperable opens lots of choices • E.g. graylog2 as the event sink
- 51. rfc3164 • This documentdescribes the observed behavior of the syslog protocol • This is not a good place to be. • Working with Jordan to document message format. • End to end tests of both implementations to follow.
- 52. Code • http://logstash.net/ • https://github.com/bobtfish/Log-Stash •https://github.com/logstash/logstash
- 53. Thanks! • <bobtfish@bobtfish.net> • t0mon irc.perl.org • And Freenode (idle in #logstash) • We are hiring!!! • Developers (learn ruby, or perl, or both!) • Front end people (play with websockets!)
- 54. This is allnow pointless • The latest logstash .jar will do all the mungeing for you. • And it (mostly) runs in MRI (C ruby), so my RAM thing is less bad. • N implementations still a good thing!
Editor's Notes
- #2 Mention state51, mention we&#x2019;re hiring\n
- #3 Mention JFDI, and I really don&#x2019;t care what language it&#x2019;s in\n
- #4 The former has amazing documentation.\nThe latter, well, bad luck. (Great reference material, but docs not so great. Good mailing list though)\n
- #5 grep is great, I love grep\nNot very good for 100 servers at once\nSolution needs to be just as good as grep for the simple case\n
- #6 This is always the first thing sugested / thought of.\nIt&#x2019;s great for audit trail, as your DB (should be!) durable\nDoing the simple thing is (at least) one disk rotation (aka fsync) per log line\n\n
- #7 This solves the performance problems, but gives you a load more moving parts\nA table with id, date, message is likely to perform less well than grep\nOne table with lots of NULL cols, lots of tables (one per data type)\n
- #8 And how do we get data back from this pile?\nNot as easy as grep!\nYou&#x2019;re stuffed as soon as you add more data types\n
- #9 We played with this. We liked it, a lot.\n
- #10 Enterprise means\n
- #11 Spenny\n
- #12 So, what does it do?\nYou can just tip logs into it, and it&#x2019;ll do the right thing... Even after the fact.\nSearching is fast fast fast!\n
- #13 Really, it&#x2019;s a great product.\nShame about the pricing.\n
- #14 I&#x2019;m also a little wary of using splunk as more than &#x2018;turbo grep&#x2019;\nSo, open source - someone else must have thought about this, right?\n
- #15 I&#x2019;m also a little wary of using splunk as more than &#x2018;turbo grep&#x2019;\nSo, open source - someone else must have thought about this, right?\n
- #16 I&#x2019;m also a little wary of using splunk as more than &#x2018;turbo grep&#x2019;\nSo, open source - someone else must have thought about this, right?\n
- #17 Isn&#x2019;t he cute? And woody!\n
- #18 Sorry - just to go off at a tangent...\nLets make all our log messages JSON messages, as json is fast, easy to parse (and you can search it with grep!)\nLets throw it in elasticsearch. Ponies and unicorns for everyone.\n
- #19 N.B. ElasticSeach storage will be MUCH larger than the bytes size (as it&#x2019;s indexed 90 ways)\nWe post-process our logs before insertion, to pull out structured fields (e.g. dates & durations)\n\n
- #20 We can add new log message types (or start parsing things we currently add as simple text), make schema changes any time we want.\nWe just pour data into ElasticSearch, and then get better searching than grep!\nThe more it&#x2019;s split into fields, the more we win, but just writing log lines still gives us as good as grep.\nAnyway - back to the story..\n
- #21 Very simple model - input (pluggable), filtering (pluggable by type) in C, output (pluggable)\nLots of backends - AMQP and ElasticSearch + syslog and many others\nPre-built parser library for various line based log formats\nComes with web app for searches.. Everything I need!\n
- #22 Lets take a simple case here - I&#x2019;ll shove my apache logs from N servers into ElasticSearch\nI run a logstash on each host (writer), and one on each ElasticSearch server (reader)..\n
- #23 So, that has 2 logstashes - one reading files and writing AMQP\nOne reading AMQP and writing to ElasticSearch\nHowever, my raw apache log lines need parsing (in the filter stage) - to be able to do things like &#x2018;all apache requests with 500 status&#x2019;, rather than &#x2018;all apache requests containing the string 500&#x2019;\n
- #24 Red indicates the filtering\n
- #25 There we go, everyone got that?\n
- #26 Except I could instead do the filtering here, if I wanted to.\nDoesn&#x2019;t really matter - depends what&#x2019;s best for me..\nRight, so... Lets try that then?\n
- #27 First problem...\n
- #28 Well then, I&#x2019;m not going to be running this on the end nodes.\n
- #29 And it&#x2019;s not tiny, even on machines dedicated to log parsing / filtering / indexing\n
- #30 But sure, I spun it up on a couple of spare machines...\n
- #31 It works fairly well as advertised.\n
- #32 The JVM giveth (lots of awesome software), the JVM taketh away (any RAM you had).\nruby is generally slower than perl. jruby is generally faster than perl.\nI&#x2019;m not actually knocking the technology here - just saying it won&#x2019;t work in this situation for me.\n
- #33 So, anyway, I&#x2019;m totally stuffed... The previous plan is a non-starter.\nSo I need something to collect logs from each host and ship them to AMQP\nOk, cool, I can write that in plain ruby or plain perl and it&#x2019;s gotta be slimmer, right?\n
- #34 But wait a second... I just want to get something &#x2018;real&#x2019; running here...\nSo, I&#x2019;m already tipping stuff into AMQP..\n\n\n
- #35 So I can just use my existing structured data, right.. Well - no, sorry..\nAnd I got distracted at this point. For about 6 months.\n
- #36 So I come back to this, still needing something to munge my JSON into other JSON.\nBut, right now, the easiest thing to try is:\n
- #37 30 line perl script, it works.\nI have data in ElasticSearch.\nI can view it in the logstash webapp\n
- #38 Going back a few slides - if RabbitMQ gets sick, everything goes bad.\nI ended up with a load of code to deal with this.\nIt still didn&#x2019;t work very well.\nIn fact, the entire idea of using TCP/IP for this is probably bad.\n\n
- #39 Syslog is hateful.\nMOST of my log messages are under 1024 bytes, but I don&#x2019;t want to throw them away (or throw an exception) if they aren&#x2019;t.\n
- #40 ZeroMQ looked like the right answer.\nI played with it. It works REALLY well.\nI&#x2019;d recommend you try it.\n
- #41 So lets write this per host collection daemon\nTake our previous mungeing code, and run it per host in the aggregation process\n
- #42 Tada! I have fixed all my woes with rabbitmq and at the same time I&#x2019;ve got my app logs in logstash format for free.\n\n
- #43 I can reuse all the heavy-lifting parts of logstash.\nI can reuse my per host ZMQ daemon as a log file tailer.\nOverhead on hosts is very small. Heavy lifting occurs entirely in the search cluster.\n
- #44 So, to recap... I&#x2019;ve got....\n
- #45 I&#x2019;ve got a solution to logging lots of stuff but not blocking or falling over.\n
- #46 I&#x2019;ve got a solution that has a minimal impact on my servers.\n\n
- #47 \n
- #48 \n
- #49 So, this is what it actually looks like.\nRaw app logs go to the agent via ZMQ. It munges them to processed logstash logs, emits.\nAgent also tails fails and emits raw logstash\nLogstash does parsing of apache logs\n
- #50 \n
- #51 It&#x2019;s taken me over 6 months to get any of it running, I don&#x2019;t have time to re-write the web app\nSomeone else is already doing that.\nI love open source.\n
- #52 So, I&#x2019;ve talked about these &#x2018;raw&#x2019; and &#x2018;processed&#x2019; log formats - they&#x2019;re just conventions to what fields can be found in the JSON.\nThis still needs to be better documented!\n
- #53 \n
- #54 \n
- #55 \n