Successfully reported this slideshow.

Fix Protocol Logstash Filter

Upcoming SlideShare
Sports Turning to Tech
Sports Turning to Tech
Loading in …3
×
1 of 10
1 of 10

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Fix Protocol Logstash Filter

  1. 1. FIX PROTOCOL LOGSTASH FILTER
  2. 2. HEY DAIN - WHY THE HELL AM I HERE? FIX LOG ▸ 2015-08-31 17:48:20,890 FIXT. 1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT. 1.19=14035=W34=249=DUMMY_INC52=201 50831-17:48:20.89056=ANOTHER_INC22=99 48=.AQUA- W262=golden_path_test268=1269=3270=640 754272=20150831273=17:48:20.88210=070 ▸ 2015-08-26 23:10:17,748 FIX. 4.2:DUMMY_INC->ANOTHER_INC: 8=FIX. 4.29=24035=834=649=DUMMY_INC52=2015 0826-23:10:17.74456=ANOTHER_INC57=Firm _B1=Inst_B6=011=best_buy14=517=ITRZ1201 508261_2420=022=831=101032=537=ITRZ1 201508261_1238=539=240=241=best_buy44 =101154=155=ITRZ160=20150826-23:10:15. 547150=2151=010=227 OUTPUT { "message" => "2015-08-31 17:48:20,890 FIXT. 1.1:DUMMY_INC->ANOTHER_INC: 8=FIXT. 1.1u00019=140u000135=Wu000134=2u000149=DUMMY_INC "@version" => "1", "@timestamp" => "2016-03-02T17:13:53.290Z", "fix_string" => “FIXT.1.1:DUMMY_INC->ANOTHER_INC”… "fix_message" => “8=FIXT.1.1u00019=140… "BeginString" => "FIXT.1.1", "MsgType" => "MarketDataSnapshotFullRefresh", "SenderCompID" => "DUMMY_INC", "SendingTime" => "20150831-17:48:20.890", "TargetCompID" => "ANOTHER_INC", "SecurityIDSource" => "99", "SecurityID" => ".AQUA-W", “MDReqID" => "golden_path_test", "NoMDEntries" => [ [0] { "MDEntryType" => "INDEX_VALUE", "MDEntryPx" => 640754.0, "MDEntryDate" => "20150831", "MDEntryTime" => "17:48:20.882" } ], WATT?? HUH??
  3. 3. SWEET… WHAT’S LOGSTASH AGAIN? LOGSTASH ▸ Logstash is part of the ELK stack (elasticsearch, Logstash, Kibana) from elastic (www.elastic.co) ▸ elasticsearch: search server based on Lucene (open-source search library) that provides a powerful, full-text search engine with a RESTful web interface and schema-free JSON documents ▸ logstash: tool for managing events and logs. When used generically, it implies a larger system of log collection, processing, storage and searching activities. ▸ kibana: Data visualization tool powered by content indexed by an elasticsearch cluster LOGSTASHLOGSTASHLOGSTASH e.g. a Log File e.g. Elasticsearch Kibana
  4. 4. OK COOL… THAT SORTA MAKES SENSE. IT MUST BE COMPLICATED… NO! LOGSTASH HAS A PLUGIN-BASED ARCHITECTURE ▸ Input Plugins ▸ rabbitmq ▸ http ▸ Amazon cloudwatch ▸ stdin ▸ Filter Plugins ▸ xml ▸ http ▸ mutate ▸ FIX PROTOCOL ▸ Output Plugins ▸ mongodb ▸ statsd ▸ slack ▸ elasticsearch ▸ stdout
  5. 5. SWEET, I’M DIGGING THIS… SO WHAT’S THE CODE LOOK LIKE LOG STASH IS RUBY…PLUGINS ARE RUBY GEMS ▸ Gem Structure ▸ Filter Template
  6. 6. OK… BUT WHAT DOES **OUR** CODE LOOK LIKE THE PLUGIN MIRRORS THE LOGSTASH.CONF input { file { path => "/path/to/fix.log" start_position => "beginning" } } filter { grok { match => ["message", “%{GREEDYDATA:fix_string}"] } fix_protocol { fix_message => fix_string data_dictionary_path => “/path/to/FIX4.xml" } } output { stdout { codec => rubydebug } } SIMPLIFIED
  7. 7. A LITTLE MORE DETAIL PUHLEAZE! CODE WALKTHROUGH
  8. 8. ALRIGHT… I GET IT, BUT HOW DOES EVERYTHING FIT TOGETHER? LET’S INSTALL LOGSTASH AND OUR PLUGIN ▸ Install logstash ▸ Add our gem as a plugin ▸ Start logstash with a configuration file $ /opt/logstash/bin/logstash -f logstash.conf $ /opt/logstash/bin/plugin install logstash-filter-fix_protocol $ brew install logstash
  9. 9. WANNA FINISH WITH A DEMO? SURE, LET’S START WITH SIMPLY USING STDOUT AS OUR OUTPUT ▸ [STDOUT DEMO] ▸ $ logstash -f logstash.example.conf
  10. 10. THAT WAS NEAT… SO WHAT? WHY SHOULD I CARE? THE FILTER CAN FEED A QUERY-ABLE OUTPUT SOURCE ▸ [ELASTICSEARCH DEMO] ▸ $ logstash -f elasticsearch.example.conf

×