To view recording of this webinar please use below URL: http://wso2.com/library/webinars/2016/06/analytics-in-your-enterprise/ Big data spans many fields and brings together technologies like distributed systems, machine learning, statistics and Internet of Things (IoT). It has now become a multi-billion dollar industry with use cases ranging from targeted advertising and fraud detection to product recommendations and market surveys. Some use cases such as urban planning can be slower (done in batch mode), while others such as the stock market needs results in milliseconds (done is a streaming fashion). Different technologies are used for each case; MapReduce for batch analytics, complex event processing for real-time analytics and machine learning for predictive analytics. Furthermore, the type of analysis ranges from basic statistics to complicated prediction models. This webinar will discuss the big data landscape including Concepts, use cases and technologies Capabilities and applications of the WSO2 analytics platform WSO2 Data Analytics Server WSO2 Complex Event Processor WSO2 Machine Learner

1. Analytics in Your Enterprise
Dakshitha Ratnayake
Lead Solutions Engineer

2. What is Analytics?

3. • organizations have more data than ever at their disposal.
• actually deriving meaningful insights from that data—
and converting knowledge into action—is easier said than
done.
• There’s no single technology that encompasses big data
analytics.
• several types of technology work together to help
Organization get the most value from Their information.
Big Data Analytics

4. Real-World Applications
o
Portfolio analysis and to predict the
impact of global events on financial
markets.
Customer experience
management and network
capacity planning and
optimization.
Music
recommendations
based on user
data.
predict what the customer
wants to see before he or she
knows what they want!
Song identifications and predict
the popular artists and genres
that will get attention in the
upcoming years.
Monitor financial market
activities and catch illegal
insider trading activities in
the financial markets.
Track patient signs using
sensor data.
Reduce their claims cost
through better fraud
detection.
Detect and prevent cyber-
attacks and criminal
activity.
Predict trends and lay
down preparation
plans to meet future
demand.
Measure player
efficiency and
defensive
effectiveness.
Source - http://upxacademy.com/2016/05/31/big-data-and-analytics-use-cases-in-8-industries/

5. WSO2 Analytics
Platform

6. • a single platform to address all analytics styles
• We deliver:
• Batch Analytics
• Real time Analytics
• Interactive Analytics
• Predictive Analytics
• WSO2 Analytics Platform uniquely combines the above
styles to turn data from IoT, mobile and Web apps into
actionable insights.
WSO2 Analytics Platform

7. WSO2 Analytics Platform

8. “Publish once, process
anyway you like”.

9. Data AnaLysis
WSO2
Analytics
Platform

10. • high-level, SQL query-like languages
• Client Applications are agnostic of the Analytics
Components
• Common set of receivers/publishers for all analytics types
• Common format for events
• Leverage leading open source projects e.g. Storm and
Spark and contribute back (such as Siddhi).
Analytics Strategy

11. • Open Source
• Rich, extensible, SQL-like configuration language
• Rich set of data connectors, which can be easily extended
• Events only need to be published once from applications to
the platform, and can be consumed by batch or real time
pipeline.
• Part of the overall WSO2 platform
Key Differentiators

12. Data Collection and
Publishing

13. Collecting DAta

14. AgentHolder. setConfigPath (getDataAgentConfigPath ());
DataPublisher dataPublisher = new DataPublisher(url, username, password);
String streamId = DataBridgeCommonsUtils.generateStreamId(HTTPD_LOG_STREAM, VERSION);
Event event = new Event(streamId, System.currentTimeMillis(), new Object[]{"external"}, null, new
Object[]{aLog});
dataPublisher.publish(event);
Collecting Data: Example
Initialize the data publisher
Generate the stream ID for the
stream to which the event will be
published
Create and Publish
Event
As a prerequisite, the streams must be defined in the receiver server (WSO2 DAS/CEP)

15. • Events are the lifeline of WSO2 CEP/DAS.
• They not only process data as events, but also interact
with external systems using events.
• An Event is a unit of data
• an event stream is a sequence of events of a particular
type.
• The type of events can be defined as an event stream
definition.
Events , Streams and Event
Stream Definitions

16. Publishing Data
o

17. Data Analysis

18. Batch Analytics

19. Batch Analytics
Generating insight by processing large amounts of stored
data
● KPI Statistics
○ Application Statistics
Monitoring
○ Network / Service Statistics
○ Sensor Data Aggregation
● Solving Optimization Problems
○ Urban Planning
○ Revenue Distribution Analysis
Source: www.e-
deal.com

20. • Batch analytics reads data from a disk (or some other
storage) and process them record by record
• “MapReduce” is the most widely used technology for batch
analytics
- Apache Hadoop
- Apache Spark 30X faster and much more flexible
• Analytics (Min, Max, average, correlation, histograms,
might join or group data in many ways)
• Key Performance indicators (KPIs) –
- e.g. Profit per square feet for retail
• Presented as a Dashboard
Batch Analytics

21. • Powered by Apache Spark
• up to 30x higher performance than Hadoop
• script-based analytics powered by Spark SQL
• Persist Data in A Database (RDBMS/NON-RDBMS) and process
Using Spark Queries and persist analyzed data in RDBMS
WSO2 Data Analytics Server

22. Batch Analytics With DAS

23. WSO2 DAS In Action: API
Statistics

24. DAS In Action: API Statistics

25. DAS In Action: HTTP Monitoring

26. Real-Time Analytics

27. Real-time Analytics
Making sense of fast moving data
● Sports
○ Real-time Analysis of Player
Performance
○ Real-time Match Analysis
● Geo-Spatial
○ Traffic Monitoring and Alerting
○ Geo-fencing
● Finance
○ Stock Market Monitoring
● Anomaly Detection
○ Fraud Detection
○ Network Intrusion Detection
○ Server Health Monitoring
Source: www.promojam.com

28. • For some use cases, the value of insights degrades very
quickly with time.
• We need technology that can produce outputs fast.
• Static Queries, but need very fast output (Alerts, Real-time
control)
• Dynamic and Interactive Queries ( Data exploration)
Real-TIME Analytics

29. • WSO2 CEP facilitates
• Real time event detection
• Correlation
• Notifications/alerts, visualization tools
• Siddhi - a high-performance streaming processing engine
• WSO2 CEP is configured using the Siddhi query language
• suited for complex queries involving time windows, as
well as patterns and sequences detection.
• CEP queries can be changed dynamically at runtime using
templates.
WSO2 Complex Event Processor

30. Real-TIME Analytics With WSO2
CEP

31. Real-time Analytics In Action

32. Real-time Analytics In Action

33. Real-time Analytics In Action

34. Interactive Analytics

35. Interactive Analytics
Near Real-time Indexed Data Search
● Log Analysis
○ Application / System Logs
● Activity Monitoring
○ Tracking Message Flows
● Fraud Detection
○ Executing queries to lookup
related data in a detected
fraud situation
● HL7 Data Exploration
○ ESB HL7 Transport Interfaced
with DAS
Source: befoundonline.com

36. • Best way to explore data is by asking Ad-hoc questions
• Interactive Analytics (search) let you query the system
and receive fast results (<10s)
• Shows data in context (e.g. by grouping events from the
same transaction together)
• Built using Lucene based Indexes.
Interactive Analytics with WSO2
DAS

37. Interactive Analytics In Action:
WSO2 DAS

38. Predictive Analytics

39. Predictive Analytics
Analyze Existing Data to Predict Future Events
● Next Value Prediction
○ Sales Forecasts
○ Electricity Loads
● Classification
○ Product Categorization
○ Customer Segmentation
● Anomaly Detection
○ Fraud Detection
○ Preventive Maintenance
● Other
○ Handwriting recognition

40. • Machine learning
• Takes in a lot of examples, and builds a program that matches
those examples.
• Specifically, that program is said to learn from experience E
with respect to some class of tasks T and performance
measure P, if its performance at tasks in T, as measured by P,
improves with experience E.
• We call that program a “model”
• A Lot of Machine Learning tools
• R ( Statistical language)
• Sci-kit learn (Python)
• Apache Spark’s MLLIB and Apache Mahout (Java)
Predictive Analytics

41. • Powered by Apache Spark MLlib
• Analyze data using machine learning algorithms
• Build machine learning models
• Compare and manage generated machine learning models
• Predict using the built models
Predictive Analytics with WSO2
Machine Learner

42. Predictive Analytics With WSO2
ML

43. Predictive Analytics In Action:
WSO2 ML

44. Home-Grown Solutions

45. WSO2 Solutions Based on the
Analytics Platform
● WSO2 Fraud Detection Solution
○ Built for detecting credit card fraud
○ The rules extensible with customized Siddhi execution
plans for any type of fraud detection
○ Currently uses Real-time and Interactive Analytics
features
● WSO2 Log Analytics Solution
○ Distributed indexing and searching of any type of
logs stored in the system
○ Notifications support with Real-time event processing
features
○ Application / Server health prediction with Machine
Learning
○ Uses Interactive + Real-time Analytics + Machine
Learning features
Source: www.retrospective.centeractive.com
Source: multichannelmerchant.com

46. Deployment

47. Minimum HA Deployment for DAS
2 Node Deployment
Use RDBMS to Store Data
If need to scale Higher
Use HBASe/Cassandra

48. Minimum HA Deployment for CEP
Minimum 2 nodes
Max throughput == 1 Node throughput

49. Minimum HA Deployment for ML
Minimum 1 node

50. Questions?? ?