Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Analytics in Your Enterprise


Published on

To view recording of this webinar please use below URL:

Big data spans many fields and brings together technologies like distributed systems, machine learning, statistics and Internet of Things (IoT). It has now become a multi-billion dollar industry with use cases ranging from targeted advertising and fraud detection to product recommendations and market surveys.

Some use cases such as urban planning can be slower (done in batch mode), while others such as the stock market needs results in milliseconds (done is a streaming fashion). Different technologies are used for each case; MapReduce for batch analytics, complex event processing for real-time analytics and machine learning for predictive analytics. Furthermore, the type of analysis ranges from basic statistics to complicated prediction models.

This webinar will discuss the big data landscape including

Concepts, use cases and technologies
Capabilities and applications of the WSO2 analytics platform
WSO2 Data Analytics Server
WSO2 Complex Event Processor
WSO2 Machine Learner

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Analytics in Your Enterprise

  1. 1. Analytics in Your Enterprise Dakshitha Ratnayake Lead Solutions Engineer
  2. 2. What is Analytics?
  3. 3. • organizations have more data than ever at their disposal. • actually deriving meaningful insights from that data— and converting knowledge into action—is easier said than done. • There’s no single technology that encompasses big data analytics. • several types of technology work together to help Organization get the most value from Their information. Big Data Analytics
  4. 4. Real-World Applications o Portfolio analysis and to predict the impact of global events on financial markets. Customer experience management and network capacity planning and optimization. Music recommendations based on user data. predict what the customer wants to see before he or she knows what they want! Song identifications and predict the popular artists and genres that will get attention in the upcoming years. Monitor financial market activities and catch illegal insider trading activities in the financial markets. Track patient signs using sensor data. Reduce their claims cost through better fraud detection. Detect and prevent cyber- attacks and criminal activity. Predict trends and lay down preparation plans to meet future demand. Measure player efficiency and defensive effectiveness. Source -
  5. 5. WSO2 Analytics Platform
  6. 6. • a single platform to address all analytics styles • We deliver: • Batch Analytics • Real time Analytics • Interactive Analytics • Predictive Analytics • WSO2 Analytics Platform uniquely combines the above styles to turn data from IoT, mobile and Web apps into actionable insights. WSO2 Analytics Platform
  7. 7. WSO2 Analytics Platform
  8. 8. “Publish once, process anyway you like”.
  9. 9. Data AnaLysis WSO2 Analytics Platform
  10. 10. • high-level, SQL query-like languages • Client Applications are agnostic of the Analytics Components • Common set of receivers/publishers for all analytics types • Common format for events • Leverage leading open source projects e.g. Storm and Spark and contribute back (such as Siddhi). Analytics Strategy
  11. 11. • Open Source • Rich, extensible, SQL-like configuration language • Rich set of data connectors, which can be easily extended • Events only need to be published once from applications to the platform, and can be consumed by batch or real time pipeline. • Part of the overall WSO2 platform Key Differentiators
  12. 12. Data Collection and Publishing
  13. 13. Collecting DAta
  14. 14. AgentHolder. setConfigPath (getDataAgentConfigPath ()); DataPublisher dataPublisher = new DataPublisher(url, username, password); String streamId = DataBridgeCommonsUtils.generateStreamId(HTTPD_LOG_STREAM, VERSION); Event event = new Event(streamId, System.currentTimeMillis(), new Object[]{"external"}, null, new Object[]{aLog}); dataPublisher.publish(event); Collecting Data: Example Initialize the data publisher Generate the stream ID for the stream to which the event will be published Create and Publish Event As a prerequisite, the streams must be defined in the receiver server (WSO2 DAS/CEP)
  15. 15. • Events are the lifeline of WSO2 CEP/DAS. • They not only process data as events, but also interact with external systems using events. • An Event is a unit of data • an event stream is a sequence of events of a particular type. • The type of events can be defined as an event stream definition. Events , Streams and Event Stream Definitions
  16. 16. Publishing Data o
  17. 17. Data Analysis
  18. 18. Batch Analytics
  19. 19. Batch Analytics Generating insight by processing large amounts of stored data ● KPI Statistics ○ Application Statistics Monitoring ○ Network / Service Statistics ○ Sensor Data Aggregation ● Solving Optimization Problems ○ Urban Planning ○ Revenue Distribution Analysis Source: www.e-
  20. 20. • Batch analytics reads data from a disk (or some other storage) and process them record by record • “MapReduce” is the most widely used technology for batch analytics - Apache Hadoop - Apache Spark 30X faster and much more flexible • Analytics (Min, Max, average, correlation, histograms, might join or group data in many ways) • Key Performance indicators (KPIs) –  - e.g. Profit per square feet for retail • Presented as a Dashboard Batch Analytics
  21. 21. • Powered by Apache Spark • up to 30x higher performance than Hadoop • script-based analytics powered by Spark SQL • Persist Data in A Database (RDBMS/NON-RDBMS) and process Using Spark Queries and persist analyzed data in RDBMS WSO2 Data Analytics Server
  22. 22. Batch Analytics With DAS
  23. 23. WSO2 DAS In Action: API Statistics
  24. 24. DAS In Action: API Statistics
  25. 25. DAS In Action: HTTP Monitoring
  26. 26. Real-Time Analytics
  27. 27. Real-time Analytics Making sense of fast moving data ● Sports ○ Real-time Analysis of Player Performance ○ Real-time Match Analysis ● Geo-Spatial ○ Traffic Monitoring and Alerting ○ Geo-fencing ● Finance ○ Stock Market Monitoring ● Anomaly Detection ○ Fraud Detection ○ Network Intrusion Detection ○ Server Health Monitoring Source:
  28. 28. • For some use cases, the value of insights degrades very quickly with time. • We need technology that can produce outputs fast. • Static Queries, but need very fast output (Alerts, Real-time control) • Dynamic and Interactive Queries ( Data exploration) Real-TIME Analytics
  29. 29. • WSO2 CEP facilitates • Real time event detection • Correlation • Notifications/alerts, visualization tools • Siddhi - a high-performance streaming processing engine • WSO2 CEP is configured using the Siddhi query language • suited for complex queries involving time windows, as well as patterns and sequences detection. • CEP queries can be changed dynamically at runtime using templates. WSO2 Complex Event Processor
  30. 30. Real-TIME Analytics With WSO2 CEP
  31. 31. Real-time Analytics In Action
  32. 32. Real-time Analytics In Action
  33. 33. Real-time Analytics In Action
  34. 34. Interactive Analytics
  35. 35. Interactive Analytics Near Real-time Indexed Data Search ● Log Analysis ○ Application / System Logs ● Activity Monitoring ○ Tracking Message Flows ● Fraud Detection ○ Executing queries to lookup related data in a detected fraud situation ● HL7 Data Exploration ○ ESB HL7 Transport Interfaced with DAS Source:
  36. 36. • Best way to explore data is by asking Ad-hoc questions • Interactive Analytics (search) let you query the system and receive fast results (<10s) • Shows data in context (e.g. by grouping events from the same transaction together) • Built using Lucene based Indexes. Interactive Analytics with WSO2 DAS
  37. 37. Interactive Analytics In Action: WSO2 DAS
  38. 38. Predictive Analytics
  39. 39. Predictive Analytics Analyze Existing Data to Predict Future Events ● Next Value Prediction ○ Sales Forecasts ○ Electricity Loads ● Classification ○ Product Categorization ○ Customer Segmentation ● Anomaly Detection ○ Fraud Detection ○ Preventive Maintenance ● Other ○ Handwriting recognition
  40. 40. • Machine learning • Takes in a lot of examples, and builds a program that matches those examples. • Specifically, that program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. • We call that program a “model” • A Lot of Machine Learning tools • R ( Statistical language) • Sci-kit learn (Python) • Apache Spark’s MLLIB and Apache Mahout (Java) Predictive Analytics
  41. 41. • Powered by Apache Spark MLlib • Analyze data using machine learning algorithms • Build machine learning models • Compare and manage generated machine learning models • Predict using the built models Predictive Analytics with WSO2 Machine Learner
  42. 42. Predictive Analytics With WSO2 ML
  43. 43. Predictive Analytics In Action: WSO2 ML
  44. 44. Home-Grown Solutions
  45. 45. WSO2 Solutions Based on the Analytics Platform ● WSO2 Fraud Detection Solution ○ Built for detecting credit card fraud ○ The rules extensible with customized Siddhi execution plans for any type of fraud detection ○ Currently uses Real-time and Interactive Analytics features ● WSO2 Log Analytics Solution ○ Distributed indexing and searching of any type of logs stored in the system ○ Notifications support with Real-time event processing features ○ Application / Server health prediction with Machine Learning ○ Uses Interactive + Real-time Analytics + Machine Learning features Source: Source:
  46. 46. Deployment
  47. 47. Minimum HA Deployment for DAS 2 Node Deployment Use RDBMS to Store Data If need to scale Higher Use HBASe/Cassandra
  48. 48. Minimum HA Deployment for CEP Minimum 2 nodes Max throughput == 1 Node throughput
  49. 49. Minimum HA Deployment for ML Minimum 1 node
  50. 50. Questions?? ?