Build your own Real Time Analytics andVisualization, Enable Complex EventProcessing, Event Patterns and AggregatesRamesh /...
Tom admiring his  handywork !
Database   Application Server
Elastic           Search                        Graylog2                        LogstashDatabase             Application S...
Elastic                                StatsD           Search                        Graylog2                        Logs...
Elastic                                StatsD           Search                                          graphite          ...
Search           Elastic                                StatsD           Search                                          g...
Search    CEP           Elastic                                StatsD           Search                                    ...
Complex Event Processing ●   ElasticSearch as a Storage or Alternate DB      ○  Faster on Lookup Queries than RDBMS      ○...
Use cases● Every PO has a matching SO?● Has a shelf in the warehouse just gone  empty?● Where is the current pile up happe...
Search    CEP           Elastic                                StatsD           Search                                    ...
Are logs the only source of events?● No - The database can be used as well.● Events can be generated by capturing the  Upd...
Search    CEP              Elastic                                   StatsD              Search                           ...
Distribute                  Replication                        Search    CEP General                   EventsQuery Log    ...
Elasticsearch
Time to Sing                             Mood of MysqlNote:image is from http://www.technocation.org
Mood of Mysql● Music is the best way to express how one feels.● Well, Mysql has a soul too, it has a mood :)● Mysql can si...
Use case: Divide & Conquer Generalquery log● Alternative to enabling general query log, which grows very  fast in size and...
Referenceshttp://bazaar.launchpad.net/~mysql/mysql-replication-listener/trunkhttps://github.com/etsy/statsd/https://launch...
Thank you vishnuhr@flipkart.comrameshpy@flipkart.com
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
Upcoming SlideShare
Loading in …5
×

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

3,612 views
3,350 views

Published on

At the 5th Elephant BigData conference in bangalore, india , 27-july-2012.

https://fifthelephant.talkfunnel.com/2012/384-build-your-own-real-time-analytics-and-visualization-enable-complex-event-processing-event-patterns-and-aggregates

Published in: Technology
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
3,612
On SlideShare
0
From Embeds
0
Number of Embeds
235
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates

  1. 1. Build your own Real Time Analytics andVisualization, Enable Complex EventProcessing, Event Patterns and AggregatesRamesh / VishnuSupply Chain - Platform Team
  2. 2. Tom admiring his handywork !
  3. 3. Database Application Server
  4. 4. Elastic Search Graylog2 LogstashDatabase Application Server
  5. 5. Elastic StatsD Search Graylog2 LogstashDatabase Application Server
  6. 6. Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  7. 7. Search Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  8. 8. Search CEP Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  9. 9. Complex Event Processing ● ElasticSearch as a Storage or Alternate DB ○ Faster on Lookup Queries than RDBMS ○ Can do simple predicate queries ○ Does not need multiple indexes (full text indexing) ○ Create fields out of interesting values ● Statsd layer is a sliding window counter ○ Within a sliding window we can do regex patterns ○ Aggregates ○ Deviations ○ This is a Key aspect of the SOA Monitoring System (Complex patterns which need action)Push the complex pattern back to ES or as a trigger for action
  10. 10. Use cases● Every PO has a matching SO?● Has a shelf in the warehouse just gone empty?● Where is the current pile up happening?● Is the SLA being breached?
  11. 11. Search CEP Elastic StatsD Search graphite Graylog2 LogstashDatabase Application Server
  12. 12. Are logs the only source of events?● No - The database can be used as well.● Events can be generated by capturing the Updates/Inserts/Deletes being made to the tables.● These events can be published to an MQ to speed up replication (batch processing) or sent to the CEP engine.
  13. 13. Search CEP Elastic StatsD Search graphite Graylog2Change DataCapture Logstash Database Application Server
  14. 14. Distribute Replication Search CEP General EventsQuery Log Elastic MQ StatsD Search graphite Graylog2 Change Data Capture Logstash log.cc Database Application Server
  15. 15. Elasticsearch
  16. 16. Time to Sing Mood of MysqlNote:image is from http://www.technocation.org
  17. 17. Mood of Mysql● Music is the best way to express how one feels.● Well, Mysql has a soul too, it has a mood :)● Mysql can sing through each query(good/bad) it gets.● Every query, Mysql gets, is intercepted in log.cc and sent acrross to an MQ Server. Subscribers to the queue ,on receiving a message play a musical note depending on the query they get.
  18. 18. Use case: Divide & Conquer Generalquery log● Alternative to enabling general query log, which grows very fast in size and disk space becomes a concern on the master database.● The queries are sent out to a queue on an MQ Server and an army of subscribers who listen to the queue , log the query on receiving a message.● The general query log can now be distributed (among the subscribers).● More number of subscribers => smaller the log & easy to rotate.
  19. 19. Referenceshttp://bazaar.launchpad.net/~mysql/mysql-replication-listener/trunkhttps://github.com/etsy/statsd/https://launchpad.net/graphitehttp://www.elasticsearch.org/http://www.oscon.com/oscon2011/public/schedule/detail/18785http://technocation.org/
  20. 20. Thank you vishnuhr@flipkart.comrameshpy@flipkart.com

×