big data et data viz - du lac à votre écran - afterwork

TABLE DES MATIÈRES
Big Data & Data visualization:
From the Lake to Your Screen
An afterwork by @OCTOSuisse
Geneva, May 9th, 2017
Joseph Glorieux
Alexandre Masselot

4
OCTO, DIGITAL TRANSFORMATION ACCELERATOR
DIGITAL
TRANSFORMATION
Facilitate and
Accelerate the adoption
of Digital Culture
-
Business, IT,
People
Consulting
& Delivery
OCTO TECHNOLOGY > THERE IS A BETTER WAY

BIG DATA @ OCTO : THE NUMBERS
TB, the biggest
volume of
distributed storage
on a single project
250
TB, the biggest
volume of data
analyzed by
OCTO’s data
scientists
>20
Is the number of
Big Data projects at
OCTO in the past
12 months
The number of
OCTO certified on
the Hadoop
platform
40
850
800
cores, the biggest
Hadoop cluster
built by OCTO
16 The number of active partnerships with
major Big Data actors
5OCTO TECHNOLOGY > THERE IS A BETTER WAY

BIG DATA @ OCTO: PUBLICATIONS
OCTO TECHNOLOGY > THERE IS A BETTER WAY 6

BIG DATA & DATAVIZ : FROM THE LAKE TO YOUR SCREEN
About Data Visualization
1
2
3
4
From Data lake to your Mac
Explore, Understand, Communicate
Back to the Lake

1
2
3
4
Back to the Lake

LIMITATIONS OF TRADITIONAL ARCHITECTURES
Over 10 Tb, « classical »
architectures requires huge
software and hardware
adaptations.
Over 1 000 transactions /
second, « classical »
architectures requires huge
software and hardware
adaptations.
Over 10 threads/Core CPU,
sequential programming reach
its limits (IO).
Over 1 000 events / second,
« classical » architectures
requires huge software and
hardware adaptations.
Distributed
storage
Share
nothing
XTP
Parallel
processing
Event Stream
Processing
« Traditional /
Standard »
architectures
RDBMS,
Application server,
ETL, ESB
Event flow oriented
application
Message Bound
(streaming)
Transaction oriented
applications
Transaction Bound
(TPS)
Storage oriented
applications
(IO bound)
Computation
oriented applications
CPU bound
(Stream Grid)
(Calculation
Grid)
(Transaction Grid)
(Storage
Grid)

BIG DATA - EMERGING FAMILIES
Event flow oriented
application
Message Bound
(streaming)
Transaction oriented
applications
Transaction Bound
(TPS)
Storage oriented
applications
IO bound
Computation
oriented applications
CPU bound
NoSQL
NewSQL
NoSQL : ditributed non-
relational stores,
NewSQL : SQL compliant
distributed stores
CEP - Complex Event Processing,
ESP - Event Stream Processing
Grid -
GPU
Grid computing on
CPU, or on GPU
In-memory analytics solutions
distribute the data in the
memory of several nodes to
obtain a low processing time.
In-memory
analytics
Hadoop
The Hadoop ecosystem offers
a distributed storage, but also
distributed computing using
MapReduce.
Streaming
In-memory
analytics
NoSQL
NewSQLStreaming
Hadoop

MODELS & DATA
Traditional models Advanced models Advanced models
with more data
Advanced models
with more data
and more features
Precision
Precision score for the TOP 20%

NEW ARCHITECTURE PATTERN: THE DATALAKE
Non-structured storage
Semi-structured
storage (NoSQL)
structured storage (ex.
relational)
Interactive
requests
Analytical
processing
Flow
management
Machine
Learning
Database Raw files Logs External data,
OpenAPI
Messages &
Events
Enterprise
DWH
Operational
system
Reporting,
request
External data,
OpenAPI
Messages &
Events
DATALAKE
INTEGRATION
PUBLICATION

PEAK OF INFLATED EXPECTATIONS?
From Datalake… … to Dataswamp
You do not
need to
store/compute
petabyte of
data…
Big Data?
I should
buy a
Hadoop
Cluster

NEW ARCHITECTURE PATTERN: THE DATALAKE
Non-structured storage
Semi-structured
storage (NoSQL)
structured storage (ex.
relational)
Interactive
requests
Analytical
processing
Flow
management
Machine
Learning
Database Raw files Logs External data,
OpenAPI
Messages &
Events
Enterprise
DWH
Operational
system
Reporting,
request
External data,
OpenAPI
Messages &
Events
DATALAKE
INTEGRATION
PUBLICATION

THE 5-LEGGED SHEEP
Source : www.marketingdistillery.com

16
THE DATALAB
Why a DataLab?
 Limitation of distributed environment for experimentation:
> less algorithms available,
> longer round trip implies slower experimentation,
> other programming paradigms
 No necessary to have all data for experimenting, statistically
relevant samples are sufficient
Description
 The DataLab is a “sandbox” area where analysts should have
great freedom with tools and data usage. It contains a work
storage area allowing to "play" with the data
 It lives outside of the Datalake to ensure and facilitate its
exploitation
 Machine with lots of RAM and CPU to enable in memory
processing, mono-machine – vertical scalability, multi-user
DataLab
Analytics
Machine Learning
Tools
Storage
DataViz Tools
Work Storage Area

17
DATA SCIENCE LIFE CYCLE
DATALAB
ITERATIVE EXPERIMENTS
Data scientists
Activities:
 Data exploration environment
 Machine learning applied to key
business question
 POC
 Preliminary models
 Demos for communication
HADOOP CLUSTER
DEVELOPMENT
Developers
Activities
 Developers implements
selected models from the
DataLab to run in a distributed
environment
 Industrialize external/internal
data flows
 Model industrialized
 Applications to access results
 Data ingestion programs
HADOOP
PRODUCTION
Business
Activities
 Interacts with the applications
accessing the Data Lake and
exposing results from models
Scheduled activities on cluster
 Ingestion of historical data
 Compute associated with all
deployed applications
 Populated Data Lake
 Models on distributed data
 Applications for end-users

ARCHITECTURE
18
proxy proxy proxy
Legend
GET blacklist
POST/PUT whitelist (>30K)proxy
DMZ/Cloud
Reverse proxy
https
data
data,programs
R,python
dataReverse proxy
On demand manual transfer
Security check
ETL, https,
ssh
Wifi
mobile
data flow
Explore existing Data Lake
masks / anonymisation
data
copy
https
ssh
Laptop PAM
PROD - System transactional
PAM laptop
Personal computer
DataLab
AD ?
HDP PROD - Bare metal
Edge
Name
Node
Name
Node
Data
Node
Data
Node
Date
Node
DEV - Toolsdata
GET whitelist
POST/PUT whitelistproxy
HDP DEV - VM
Edge
Name
Node
Data
Node
Data
Node
PAM
HDP SANDBOX -
VM
Edge
Name
Node
Data
Node
Data
Node
Bare metal
Virtual machine

THE NEW DATASCIENCE PLATFORM

1
2
3
4
Back to the Lake

21
A PERFECT USE CASE: SWISS PUBLIC TRANSPORT
Data types
 Schedules
 Event storage
 Real time streaming
Usages
 Data analysis
 Prediction
 End user application
Sources
 opentransportdata.swiss
 transport.opendata.ch
 gtfs.geops.ch

23
QUESTION OF THE DAY: “IS MY TRAIN RUNNING LATE?”
“WILL MY TRAIN BE RUNNING LATE?”

24
“WILL MY TRAIN BE RUNNING LATE?”

26
WHAT IS INSIDE THE AVAILABLE DATA?
Multiple challenges
 Different acquisition modes
 Different ids for the same entity
 Different languages (e.g. german)
 Different quality (e.g. missing data)
Multiple sources
 opentransportdata.swiss
 transport.opendata.ch
 gtfs.geops.ch

28
EXPLORING THE DATA
Getting acquainted with the data:
 download a bearable sample (40 millions lines)
 Repeat
1. build the lightest import process
2. observe
3. go to business expert to get insights
4. observe

29
EXPLORING THE DATA

30
EXPLORE MY DATA?

31
EXPLORING THE DATA
The need for a visualization tool:
 interactive
 versatile
 handling large amount of data (samples)
 loading data for various sources
 adding computed values

32
EXPLORING THE DATA
 Connect to data
An Introduction to Tableau

33
EXPLORING THE DATA
 Create computed columns

34
EXPLORING THE DATA
 Join multiple tables

35
EXPLORING THE DATA

36
EXPLORING THE DATA
 Exploring and filtering out data

37
EXPLORING THE DATA
 Killing preconceived ideas: “InterCity trains are less frequently late”

38
EXPLORING THE DATA

39
EXPLORING THE DATA

41
ANALYZING DATA WITH NOTEBOOKS
 A notebook allows to write text and live code in order to wrap together code,
output and documentation
 The full power of programming, interactivity, results and documentation. All in
the same place.
Language
of choice
Interactive
widgets
Share
notebooks
Big data
Integration

42
Loading and munging data

43
Figures and code

44
ANALYZING DATA
Building a model
1’ 3’ 5’ 6’ 11’ 12’ 14’ 15’ 20’ 22’ (departure shift)

45
ANALYZING DATA
Building a model
1’ 3’ 5’ 6’ 11’ 12’ 14’ 15’ 20’ 22’ (departure shift)
When will my train
leave Les Tuileries?

46
My train has 90% chance of leaving Les Tuileries
between 45s and 3’40s seconds late
Between 45s
and 3’40s!

47
At 7 AM, my train has 90% chance of leaving Les Tuileries
between 1’10s and 3’30s late
At 7AM,
between 1’15s
and 3’30s!

49
~ consistent delays

50
If I know How late my train runs in Versoix,
I can predict rather precisely how late it will be in Les Tuileries
If it’s 3’ late in
Versoix,
between 50s
and 1’20s!

51
ANALYZING DATA

53
COMMUNICATION = INFORMATION WITH A MEANING

54
COMMUNICATING

55
COMMUNICATING

56
COMMUNICATING
 Sharing notebooks + data online
 Assemble and broadcast dashboards
 Design and share stories
Tableau can be turned into a communication tool

57
COMMUNICATING
 Sharing data generated document with values, figures…
 Publishing on URL
Notebook can be used for communication

58
COMMUNICATING
Browser power

59
COMMUNICATING
D3.js: mapping data to browser DOM
Browser power
time
station board
trains

60
COMMUNICATING

61
COMMUNICATING
Browser power with d3.js

62
COMMUNICATING
Browser power inheriting from d3.js

63
COMMUNICATING
Browser power
sigma.js cytoscape.js

64
COMMUNICATING
Browser power with high throughput data

65
THE VISUALIZATION IS THE OXYGEN OF THE DATA SCIENCE

1
2
3
4
Conclusion

67
ANOTHER PERSPECTIVE ON VISUALIZATION
Who said that? When?
“There is danger in giving too much
information to executives of small
brain capacity.”
“As a cathedral is to its foundations,
so is an effective presentation of the
fact to the data.”
“The answer is that the executive of the
future will be forced on the analysis of facts
which have been collected and arranged for
his instantaneous and continuous use.”

68
1914Willard C. Brinton
100yrsofbrinton.tumblr.com

69
100yrsofbrinton.tumblr.com

70
1880: TEXTILE PRODUCTION IN ENGLAND (OTTO NEURATH, ~1920)
Changing the world by educating people about the world around them

73
PARIS-LYON TRAIN SCHEDULE (1880S)

Early maps Measurement
& Theory
New graphic
forms
Golden
age
Begin
modern
period
Modern
dark
ages
High-D
Vis
Density
Year
The distribution of milestone items over time, shown by a rug plot and density estimate.
Michael Friendly et Daniel J. Denis. https://www.researchgate.net/publication/221649568
Graphics Milestones: Time course of developments

75
THE PREVIOUS BIG DATA REVOLUTION (END 1800s)

76
VISUALIZATION THEORY & PRACTICE, BY EDWARD R. TUFTE
The most complete suite of classic books

77
VISUALIZATION THEORY & PRACTICE
“Pie charts are bad and that the only thing
worse than one pie chart is lots of them.”
E. Tufte
W. Brinton W. Playfair (1801)

78

79
 Among several authors, Mackinley (1986) stated an “expressiveness rule”
for graphical display.
 The most important information shall use the following attributes
(in priority order):
1. position;
2. size;
3. orientation;
4. shape;
5. color.
 And the most important dimensions to communicate shall therefore use the
first attributes.
Mackinley (1986) “Expressiveness Rule”

80
Data/Ink Ratio
Data-ink ratio = =
1- proportion of a graphic
that can be erasedTotal ink used to
print the graphs
Data-ink
“Above all else show the data”
E. Tufte, 1983

82
Data/Ink Ratio
"Perfection is achieved not when there is nothing more to add,
but when there is nothing left to take away”
Antoine de St Exupéry
Terre des Hommes, 1939

83
8% males are color blind (0.6% females)

84
Violating all principles

85
VISUALIZATION: NEW METHODS
Three.js

86
Interactive display wall
http://earlymodernconversions.com/activity/history-visualization-lab/

87
Virtual reality

88
Data sonification
http://earlymodernconversions.com/activity/history-visualization-lab/

89
Animation
A Day in the Life of Americans – Nathan Yau

90
Animation
Hans Rosling… The Revolutionary

1
2
3
4
From Datalake to your mac
Explore, understand, communicate
Back to The Lake

ARCHITECTURE
92
proxy proxy proxy
Legend
GET blacklist
POST/PUT whitelist (>30K)proxy
DMZ/Cloud
Reverse proxy
https
data
data,programs
R,python
dataReverse proxy
On demand manual transfer
Security check
ETL, https,
ssh
Wifi
mobile
data flow
Explore existing Data Lake
masks / anonymisation
data
copy
https
ssh
Laptop PAM
PROD - System transactional
PAM laptop
Personal computer
DataLab
AD ?
HDP PROD - Bare metal
Edge
Name
Node
Name
Node
Data
Node
Data
Node
Date
Node
DEV - Toolsdata
GET whitelist
POST/PUT whitelistproxy
HDP DEV - VM
Edge
Name
Node
Data
Node
Data
Node
PAM
HDP SANDBOX -
VM
Edge
Name
Node
Data
Node
Data
Node
Bare metal
Virtual machine

93
DATA SCIENCE LIFE CYCLE
DATALAB
ITERATIVE EXPERIMENTS
HADOOP CLUSTER
DEVELOPMENT
HADOOP
PRODUCTION

94
WHAT IS DATA DRIVER?
 Data Driver is a platform for data science exploration/production
 Data Driver integrates all the OCTO know-how acquired for 5 years
 Data Driver accelerates the development of your data science applications
to production
By

A COMPANY: INFINITE OPPORTUNITIES FOR DATA SCIENCE
SUPPORT
Information SystemHR
Strategy
Produc-
tion
Compliance, risk
management
Finance
CORE BUSINESS
R&D Sales,
distributi
on
ENTERPRISE MANAGEMENT
…
Administration …
Supply
chain
Planification
Marke-
ting
After
sales
…
Procure-
ment

96
- Du 29 au 30 Mai 2017 à Genève
Nouvelles Architectures des Systèmes
d’Information
academy.octo.c
- Du 8 au 9 Juillet 2017 à Genève
Découvrir les démarches et la culture agile
- Du 26 au 27 Juillet 2017 à Genève
Les géants du web : culture - pratiques -
architecture
- Du 15 au 17 Mai 2017 à Genève
Analyse de données pour Hadoop 2.x
Hortonworks

AVENUE DU THÉÂTRE, 7 – 1005 LAUSANNE > SUISSE > WWW.OCTO.CH
OCTO Suisse RECRUTE
5 consultants en 2017
rejoins.octo.com
Architecte
Software
Craftsman DataGeek
Coach
Méthodo
Expert
DevOps
Consultant
en Stratégie
Questions ?

big data et data viz - du lac à votre écran - afterwork

More Related Content

Similar to big data et data viz - du lac à votre écran - afterwork

More from OCTO Technology Suisse

Recently uploaded

big data et data viz - du lac à votre écran - afterwork