Mohit Jotwani's presentation

1. Stream Processing
Key Driver for Enabling Instant Insights on Big Data
Mohit Jotwani
Product Manager, DataTorrent

2. Why is Stream Processing Vital?

3. SOURCE DATA
MS Queue’s
Events
XML Files
Databases
Sensor data
Social
Enterprise
Repositories
RDBMS
EDW
NoSQL
Feed m
Feed 2
Feed 1
Load
(Optional)
Staging Area
Traditional Analytics – Data at Rest
Business Analytics
Business Intelligence
Visualization Tools
Visualize
Analyze
Extract Transform
Feed n
Feed 2
Feed 1
Visualize

4. Next Generation – Data in Motion
• Organizations need to react to changing business conditions in real time
• Faster decision making across all industries
• Few companies outside of financial markets, telecom & utilities have experience with
streaming
• Newer data sources – like sensors, social media feeds
• Higher Volume and Greater Velocity
• More unstructured and semi-structured data
• Democratization of technologies
• Open Source Projects
• Large Scale Compute & Storage – Hadoop, NoSQL
• Streaming Technologies – Apex, Spark, Storm etc.
• Real-time dashboards and alert notification systems
• Beyond niche use cases
• Broad applicability but needs more adoption

5. Stream vs. Batch Processing Pipelines
Transform Analyze Action
Visualize/
Persist
Ingest
Extract Transform Load Analyze Action

6. Stream Processing
•Continuous processing on data as it flows through a
system
•Allows users to act on events instantaneously via
alerts
•Processing related to time (event time vs. processing
time)
•Real-Time – diff between event time and processing
time is negligible
Enables your Data In Motion Architecture

7. Big Data Application Types
IoT
Fraud
CDR
CDC
Reporting
SQL
Operations
Data Discovery
SQL on
Streams
Streaming
Disovery

8. Sample Streaming Analytics Patterns
Preprocessing
• Filtering events
• Transforming
attributes
Alerts & Thresholds
• Based on complex
conditions
Computing within
Windows
• Aggregations
Combining Event
Streams
• Correlation
• Error detection
Enrichment
• Looking up database,
reference data
Temporal Events
• Detecting events
within time windows
Tracking
• Tracking events over
space & time
Trend Detection
• Rise, Fall
• Outliers
Source: https://iwringer.wordpress.com/2015/08/03/patterns-for-streaming-realtime-analytics/

9. Stream Processing Use Cases

10. Financial Services
• Detect fraudulent activity in real-time
• Risk Analysis
• Deliver personalized products and
offerings
• Make decisions in real-time for trading
and transactional platforms

11. Financial services big data fabric

12. Telecom
• Real-time network monitoring and
protection
• Quality of service and Customer
Satisfaction
• Take action based on users’ location
• Automatic resource allocation and load
balancing

13. Online Advertising
• Dynamic bidding
• Real-time targeting & personalization
• Maximize click-through and
conversion rates.
• Reporting that can be updated
continuously

14. Online advertising dynamic inventory purchases

15. Internet of Things
• Environment monitoring
• Infrastructure management
• Manufacturing
• Energy management
• Public Building & Home automation
• Transportation

16. IoT secure ingestion and predictive analysis
High performance, multi-
customer secure, data ingestion.
Complex event processing with
historical data for predictive
maintenance
Sensor 2
Sensor 1
Sensor N
Application n
Application 1
Persistent
Data
Governance
Complex
Event Process
Predictive
maintenance

17. Stream Processing:
Conclusion
•Lots of untapped potential!
•Gives your business a competitive edge!
•Open Source and Big Data
technologies
•Built to address the scale and latency
demands
•Broad use cases
•Across industries and verticals

20. Hadoop Ingestion Made Easy
https://www.brighttalk.com/webcast/13685/194937/hadoop-ingestion-made-easy-with-
datatorrent-dtingest

21. •
•
•
•
•
•
•

22. https://www.datatorrent.com/careers/
indiajobs@datatorrent.com

23. Thank You