This document discusses real-time analysis of big data and provides an example use case of flow activity monitoring. It describes trends in big data including increasing volumes, velocities, and varieties of data. The emergence of real-time analytics on big data enables faster decision making. The document then demonstrates a real-time system for monitoring application activity and flows using the open-source Elasticsearch, Logstash and Kibana stack to collect, transform, analyze and visualize data without modifying the existing systems.

1. L’analyse en temps réel
de Big Data
 Le monitoring de flux
par l’exemple
Abdellatif BOUCHAMA
@a_bouchama

2. Abdellatif BOUCHAMA
@A_BOUCHAMA
• Middleware architect, and
passionate about the new
technology: Big Data,IoT and
Open source
• Co-founder of BusHorn.com

3. Agenda
Big Data in 2015:Trends & Statistics
The emergence of real time
Use Case: Flow Activity Monitoring
Demo

4. What is Big Data?
Big Data
Velocity
30KB/s --> 30GB/s
· Batch
· Real-Time
· Streaming
Volume
1-2 Terabytes  ∞
Variety
· Structured
· Semi-Structured
· UnStructured

5. Big Data Expectations:
Things That Excite Executives About Big Data
It’s the “next oil.” from the Ginni Rometty, CEO of IBM
The potential to revolutionize industries, and change business models (e.g.,
Uber and Airbnb), with what we learn from the data.
The ability to provide real-time graphing solutions for data relationships.
Real-time operational and business data allows people to make decisions
faster thereby saving significant money.
The opportunities it gives us to help clients solve real business problems.
https://dzone.com/articles/14-things-that-excite-executives-about-big-data?oid=big_data

6. Big Data view by Gartner:

7. Big Data in 2015:
75% of Companies Are Investing or Planning to Invest in Big Data in the NextTwoYears
Goals for Big Data initiatives:
1. Enhancing the customer experience
2.Streamlining existing processes
3.Achieving more targeted marketing and reducing costs
Last year, 37% of big data projects were initiated by the CIO, while 25% were initiated by business unit
heads. In 2015, the roles are nearly tied, at 32% and 31 %, respectively.
http://www.gartner.com/newsroom/id/3130817

8. The emergence of Real-Time
Low
Pure Batch
Operationalizing
Near RealTime
Or
Interactive
Analytics
High
RealTime
Analytics
• Right timeReal time
• Smart dataBig Data

9. Use case: Flow Activity
Monitoring

10. Architecture

11. Constraints & Requirements
Constraints
• Non intrusive system
• No modification on business flows.
• We can plug it and unplug it easily.
Requirements
• System cost should be mastered and
adaptable
• System automatized
• Measurement

12. Why Elasticsearch stack?
Open source
Easy to deploy
Distributed
Linear scalability
REST Interface
Kibana & Logstash to complete the Landscape

13. Data flow and constraints
Collect
JMX
Store & transport
Transform & Access
Modelize & Analyze
Visualize & Predict
JMS

14. DemoTime!
Be prepared for it to fail, because demos always do

15. What’s Next ?
• Enrichment analysis, after adding business
information in the logs
• Integration of backend and frontend
applications
• Audit

16. Thank you