Abundant data is all around. The most important aspect is how you as an organization can access the data, process it, and present information to the relevant authorities on time. To gain competitive advantage the means of accessing, processing and presenting the data should be optimal, highly available and scalable. In this talk, we will discuss how you can leverage WSO2 Data Analytics Server, WSO2 IoT Server, WSO2 Enterprise Service Bus and other WSO2 products in order to analyze the data. We will also discuss different deployment patterns that can provide you with a suitable solution that lets you analyze relevant data historically, in real-time or interactively and predict future states to make better decisions for your organization’s success.

1. Patterns for Deploying Analytics
in the Real World
Sriskandarajah Suhothayan (Suho)
Technical Lead
WSO2

2. What’s Analytics ?

3. Problems to think about
• Can it handle my load ?
• How costly it is ?
• Adaptability ?
• Can it analyse 3rd
party systems ?
• etc ...

4. Where to start ?

5. Where to start ?
• Think Big !

6. Where to start ?
• Think Big !
But...
• Start simple !
• Eat Your Own Dog Food
• Analyse what you already have

7. Step 1 :
Find Data Inside Your Organisation...

8. Collect Data Internally
• Don’t worry about
– Data formats
– Data sources
– Platforms
– Protocols
Start with WSO2 DAS
it has a unified data capturing framework !

9. Deployment for Data Collection

10. Step 2 :
Understand how things have been ...

11. Deployment for Data Analytics
Batch & Interactive Analytics
• Enable Searchability
– Full text data
– Drill down search
• See what has happened
– Summarise the Data
– Understand patterns and behaviors

12. Deployment for Data Analytics
Batch & Interactive Analytics
• Enable Searchability
– Full text data
– Drill down search
• See what has happened
– Summarise the data
– Understand patterns and behaviors
• Simple Deployment
– 2 Nodes
– Use RDBMS to store the data

13. Deployment for Data Analytics
Batch & Interactive Analytics
2 Node
Deployment

14. Step 3 :
Keep informed ...

15. Deployment for Data Analytics
Realtime Analytics
• Keep informed
– Dashboard
– Alerts
– Feedback loops

16. Deployment for Data Analytics
Realtime Analytics
• Keep informed
– Dashboard
– Alerts
– Feedback loops
• High Availability
– Zero downtime
– Zero data loss

17. Realtime High Availability Deployment
Minimum 2 nodes
Max throughput == 1 Node throughput

18. Deployment for Data Communication
Alerting & Communicating
Legacy & Internal
Services

19. Realtime + Batch Analytics
• Filter Data before you store
– Realtime → Store & Process
• Summarize and store
– Realtime → Store & Process
• Cross check with history
– Lambda Architecture
– Graph with Batch & Realtime
• Alerts based on batch processing
– Batch → Realtime
From Batch
From Realtime

20. Step 4 :
Think ahead ...

21. Deployment for Data Analytics
Predictive Analytics
1 Node of WSO2 ML 1 Node of WSO2 ML

22. Minimum High Availability Deployment
All you need a
2 Node
Deployment

23. Step 5 :
Expanding as a Connected Business …

24. Deployment for Data Collection ...
From 3rd Party Apps & Cloud
HTTP
Utilize API Analytics !

25. Analyse Business with API Analytics
• APIs involved
• Who invokes the APIs
• Extract business information from
– Payloads
– Resources URIs
Monetize APIs !

26. Step 6 :
Scale with your Data ...

27. Scaling Analytics Deployment
The Changes !
• Realtime
– Supported by Apache Storm
• For High Memory Requirement or CPU Intensive Processing
– No query changes
• Batch
– Move from RDBMS to HBase/Cassandra
• WSO2 DAS have a Data Abstraction Layer
• Independent of underlying Data Store
Seamless migration :)

28. Realtime Scalable Deployment ...
Event Processing offloaded to
Siddhi Running on Apache Storm
Seamlessly :)

29. Realtime Scalable Deployment ...
Handling Stateless
& Stateful Queries

30. Realtime Scalable Deployment
Apache Storm Cluster + N CEP nodes

31. Deployment for Scalable Data Analytics
Minimum 8 Nodes
Deployment
(+ Storm if needed)

32. Step 7 :
Sense the world around you ...

33. Deployment for Data Collection
From Sensors
Analytics on the Edge
with WSO2 Siddhi
Push

34. Deployment for Data Communication
Mobile & 3rd Party Apps
● Expose analytics results
as API
○ Mobile Apps, Third Party
● Provides
○ Security, Billing,
○ Throttling, Quotas & SLA
● How ?
○ Write data to database from DAS
○ Build Services via WSO2 Data Services Server or use Analytics
REST API
○ Expose them as APIs via WSO2 API Manager

35. Analytics Life Cycle
Predefined analytics
• Artifacts bundled as CApps to and moved
Dev → Test → Preprod → Prod
Analytics on Production Environment
• Interactive Analytics
• Personalizing Dashboards
• Customised Alerts

36. Summary
• Start small and scale as you grow
• Minimum HA Deployment
– 2 Nodes
• Fully Distributed Deployment
– 8+ Nodes
– Scale based on need, horizontally and vertically
• Analyser, Indexer, Receiver,
Realtime (With Apache Storm), Dashboard

37. In God we trust;
all others must bring data
- William Edwards Deming -

38. Thank You