• Save
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Upcoming SlideShare
Loading in...5
×
 

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

on

  • 2,701 views

our presenation @

our presenation @
5th elephant-bangalore-india-27-july-2012

Statistics

Views

Total Views
2,701
Views on SlideShare
2,468
Embed Views
233

Actions

Likes
2
Downloads
0
Comments
1

2 Embeds 233

http://funnel.hasgeek.com 201
https://funnel.hasgeek.com 32

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates Presentation Transcript

  • Build your own Real Time Analytics andVisualization, Enable Complex EventProcessing, Event Patterns and AggregatesRamesh / VishnuSupply Chain - Platform Team
  • Tom admiring his handywork !
  • Database Application Server View slide
  • Elastic Search Graylog2 LogstashDatabase Application Server View slide
  • Elastic StatsD Search Graylog2 LogstashDatabase Application Server
  • Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  • Search Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  • Search CEP Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  • Complex Event Processing ● ElasticSearch as a Storage or Alternate DB ○ Faster on Lookup Queries than RDBMS ○ Can do simple predicate queries ○ Does not need multiple indexes (full text indexing) ○ Create fields out of interesting values ● Statsd layer is a sliding window counter ○ Within a sliding window we can do regex patterns ○ Aggregates ○ Deviations ○ This is a Key aspect of the SOA Monitoring System (Complex patterns which need action)Push the complex pattern back to ES or as a trigger for action
  • Use cases● Every PO has a matching SO?● Has a shelf in the warehouse just gone empty?● Where is the current pile up happening?● Is the SLA being breached?
  • Search CEP Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  • Are logs the only source of events?● No - The database can be used as well.● Events can be generated by capturing the Updates/Inserts/Deletes being made to the tables.● These events can be published to an MQ to speed up replication (batch processing) or sent to the CEP engine.
  • Search CEP Elastic StatsD Search graphite Graylog2Change DataCapture Logstash Database Application Server
  • Distribute Replication Search CEP General EventsQuery Log Elastic MQ StatsD Search graphite Graylog2 Change Data Capture Logstash log.cc Database Application Server
  • Elasticsearch
  • Time to Sing Mood of MysqlNote:image is from http://www.technocation.org
  • Mood of Mysql● Music is the best way to express how one feels.● Well, Mysql has a soul too, it has a mood :)● Mysql can sing through each query(good/bad) it gets.● Every query, Mysql gets, is intercepted in log.cc and sent acrross to an MQ Server. Subscribers to the queue ,on receiving a message play a musical note depending on the query they get.
  • Use case: Divide & Conquer Generalquery log● Alternative to enabling general query log, which grows very fast in size and disk space becomes a concern on the master database.● The queries are sent out to a queue on an MQ Server and an army of subscribers who listen to the queue , log the query on receiving a message.● The general query log can now be distributed (among the subscribers).● More number of subscribers => smaller the log & easy to rotate.
  • Referenceshttp://bazaar.launchpad.net/~mysql/mysql-replication-listener/trunkhttps://github.com/etsy/statsd/https://launchpad.net/graphitehttp://www.elasticsearch.org/http://www.oscon.com/oscon2011/public/schedule/detail/18785http://technocation.org/
  • Thank you vishnuhr@flipkart.comrameshpy@flipkart.com