Your SlideShare is downloading. ×
Realtime Analytics and Anomalities Detection using Elasticsearch, Hadoop and Storm
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Realtime Analytics and Anomalities Detection using Elasticsearch, Hadoop and Storm

2,657
views

Published on

Published in: Technology

0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,657
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
17
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Zipf distribution, power-law, long tail
  • Full-text search
    Highlight
    Search-as-you-type
    Did-you-mean
    More-like-this
    Geolocation
    Events, logs,
  • Precision = how many of the retrieved documents are relevant
    Recall = how many of the relevant documents are retrieved
  • Term aggregation
  • Significant terms
  • Transcript

    • 1. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Real-time Analytics & Anomaly detection using Hadoop, Elasticsearch and Storm Costin Leau @costinl
    • 2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    • 3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Interesting != Common Datasets tend to have hot / common entities Monopolize the data set Create too much noise Cannot be easily avoided Common = frequent Interesting = frequently different
    • 4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Finding the uncommon Background vs foreground == things that stand out Example: Background: “flu” “H5N1” appears in 5 / 10M docs H5N1 flu
    • 5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Finding the uncommon Background vs foreground == things that stand out Example: Background: “flu” “H5N1” appears in 5 / 10M docs Foreground: “bird flu” “H5N1” appears in 4 / 100 docs H5N1 flu H5N1 bird flu
    • 6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Finding the uncommon - Challenges Deal with big data sets • Hadoop Perform the analysis • Elasticsearch Keep the data fresh • Storm
    • 7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Hadoop De-facto platform for big data HDFS - Used for storing and performing ETL at scale Map/Reduce - Excellent for iterating, thorough analysis YARN – Job scheduling and resource management
    • 8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch Open-source real-time search and analytics engine • Fully-featured search Relevance-ranked text search Scalable search High-performance geo, temporal, range and key lookup Highlighting Support for complex / nested document types * Spelling suggestions Powerful query DSL * “Standing” queries * Real-time results * Extensible via plugins * • Powerful faceting/analysis Summarize large sets by any combinations of time, geo, category and more. * “Kibana” visualization tool * * Features we see as differentiators • Management Simple and robust deployments * REST APIs for handling all aspects of administration/monitoring * “Marvel” console for monitoring and administering clusters * Special features to manage the life cycle of content * • Integration Hadoop (Map/Red,Hive,Pig,Cascading..)* Client libraries (Python, Java, Ruby, javascript…) Data connectors (Twitter, JMS…) Logstash ETL framework * • Support Development and Production support with tiered levels Support staff are the core developers of the product *
    • 9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch Open-source real-time search and analytics engine
    • 10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch Hadoop Use Elasticsearch natively in Hadoop ‣ Map/Reduce – Input/OutputFormat ‣ Apache Pig – Storage ‣ Apache Hive – External Table ‣ Cascading – Tap/Sink ‣ Storm (in development) – Spout / Bolt All operations (reads/writes) are parallelized (Map/Reduce)
    • 11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Storm Distributed, fault-tolerant, real-time computation system Perform on-the-fly queries React to live data Prevention Routing
    • 12. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Discovering the relevant
    • 13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Inverted index Inverting Shakespeare ‣ Take all the plays and break them down word by word ‣ For each word, store the ids of the documents that contain it ‣ Sort all tokens (words) token doc freq. postings (doc ids) Anthony 2 1, 2 Brutus 1 5 Caesar 2 2, 3 Calpurnia 2 4, 5
    • 14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Relevancy How well does a document match a query? step query d1 d2 The text brown fox The quick brown fox likes brown nuts The red fox The terms (brown, fox) (brown, brown, fox, likes, nuts, quick) (red, fox) A frequency vector (1, 1) (2, 1) (0, 1) Relevancy - 2? 1?
    • 15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Relevancy - Vector Space Model • How well q matches d1 and d2? ‣ The coordinates in the vector represent weights per term ‣ The simple (1, 0) vector we discussed defines these weights based on the frequency of each term ‣ But to generalize: . 2 1 1 tf: brown tf: fox q: (brown, fox) d1: (brown, brown, fox) d2: (fox)
    • 16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Relevancy TF-IDF Term frequency / Inverse Document Frequency TF = the more a token appears in a doc, the more important it is IDF = the more documents containing the term, the less important it is
    • 17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Ranking Formula Called Lucene Similarity Can be ignored (was an attempt to make query scores comparable across indices, it’s there for backward compatibility) Core TF/IDF weight Score of a document for a given query Normalized doc length, shorter docs are more likely to be relevant than longer docs Boost of query term t
    • 18. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Discovering the interesting
    • 19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Frequency differentiator TF-IDF by-itself is not enough need to compare the DF in foreground vs background Precision vs Recall balance
    • 20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Single-set analysis A C F H I K A B C D E … X Y Z W Query results Dataset
    • 21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Single-set analysis example crimes bicycle theft crimes bicycle theft British Police Force British Transport Police
    • 22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Multi-set analysis A B C D E … X Y Z W A C F H I K M Q R … Query results Dataset A B C D .. J L M N O .. U Aggregate
    • 23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Background (geo-aggregation)
    • 24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Foreground (geo-aggregation)
    • 25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Hadoop Off-line / slow learning ‣ In-depth analysis ‣ Break down data into hot spots ‣ Eliminate noise ‣ Build multiple models
    • 26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Elasticsearch Search features ‣ Scoring, TF-IDF ‣ Significant terms (multi-set analysis) Aggregations ‣ Buckets & Metrics
    • 27. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Reacting to data
    • 28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Reacting to data Prevent execute queries as data flows in  build a model Route place suspicious data into a dedicate pipeline
    • 29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Reacting to data spout bolt bolt bolt bolt bolt bolt bolt bolt bolt
    • 30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Live loop Data keeps changing ‣ Adapt the set of rules Improves reaction time ‣ Build a model for fast decision making Keeps the prevention rate high ‣ Categorize data on the fly bolt
    • 31. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Putting it all together
    • 32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited The Big Picture HDFS Slow, in-depth learning Fast, real-time learning ETL
    • 33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited Usages Recommendation ‣ Find similar movies based on user feedback ‣ Use Storm to optimize the returned results Card Fraud ‣ Use Storm to prevent suspicious transactions from executing ‣ Route possible frauds to a dedicated analysis queue
    • 34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited
    • 35. Copyright Elasticsearch 2014 Copying, publishing and/or distributing without written permission is strictly prohibited Q&A Thank you! @costinl