The document discusses challenges in real-time analytic processing including high data volumes, variety of data types, and need for fast velocity. It describes how IBM Streams can perform real-time big data analysis through operations like filtering, classifying, fusing and annotating data. IBM Streams can also interact with other systems and its flexibility allows users to write their own adapters. Examples are given of how IBM Streams has been used for applications like medical monitoring, telecommunications analytics, and acoustic monitoring of fish populations.

1. IBM | Streams
Streaming Analytics In Real Life
Johan Picard
Big Data Developers Meetup – 2 Oct 2017

2. IBM | Streams
Challenges for Real Time Analytic Processing (RTAP)
•Volume of data: faster than a database can handle
•Variety:: correlation from multiple sources and/or signals; video, audio or other non-
relational data types
•Velocity: responses required in under a millisecond
•Scalable: requires more than one server to handle the workload
2

3. IBM | Streams
Modify
Filter / Sample
Classify
Fuse
Annotate
Big Data in real-time with IBM Streams
Score
Windowed
Aggregates
Analyze

4. IBM | Streams
Streams Does Not Live in a Vacuum
 Interact with other systems
• Network protocols
• Messaging appliances
• Databases
• Hadoop
• R-project
• Operation Decision Manager (ODM)
• SPSS
• More…
 You can write your own adapters
• C++, Java, SPL
© 2014 IBM Corporation4

5. IBM | Streams
Origin of Streams
"The US Government has been
working with IBM Research
since 2003 on a radical new
approach to data analysis that
enables high speed, scalable
and complex analytics of
heterogeneous data streams in
motion. The project has been so
successful that US Government
will deploy additional
installations to enable other
agencies to achieve greater
success in various future
projects“
- US Government

6. 80 Radio Astronomy Dishes
Correlation – Flagging – Calibration – Imaging
160 gigabytes per second ingest to correlation
16 gigabytes per second output from correlation
IBM and the South Africa
Square Kilometer Array (Meerkat)
6

7. IBM | Streams
UCLA relies on breakthrough technology to provide
proactive care for patients with traumatic brain injuries
Need
• Integrate patient physiologic data and apply
predictive analytics for life threatening issues
Benefits
• Captures and analyzes ICU monitor data in real
time to alert clinicians and help prevent brain
damage
• Predict rising brain pressure to help prevent
higher risk situations
• Predicts critical changes in the patient conditions,
providing more time for life-saving medical
responses
7

8. IBM | Streams
Bharti Airtel
Real-time mediation and analysis
7B CDRs per day
Data processing time reduced
12 hour to 1 minute
Resources Reduced
36 to 12 blades
Enhanced customer satisfaction
Machine learning
applied on the fly
8

9. IBM | Streams
Detection of fish having sex

10. IBM | Streams
Cod are superheroes

11. IBM | Streams
Audiograms

12. IBM | Streams
Fish sound production
 Fish sound production
 Courtship display (only male cod during spawning)
 Territory
 Aggression
 Social aggregation (schooling)
 Grunts, moans and clicks (e.g. 130 dB re 1μPa at 1 m)
 Frequencies of a few Hz up to some hundred Hz
 Use fish sound to identify species and behavioral state
 Combined information from hydrophone and echosounder

13. IBM | Streams
Aggression calls of different gadoids

14. IBM | Streams
Courtship behaviour and calls in haddock

15. IBM | Streams
Cod rumble off the coast of Norway
Individual fish can not be recognized

16. IBM | Streams
Streams - Sound display

17. IBM | Streams
Streams - Graph

18. IBM | Streams
Streams - Results