Slides for the talk about Big Data and moneytization for the Apache Big Data Conf Europe

1. APACHE
BIG DATA
CONFERENCE
How to transform data into money
using Big Data technologies

2. After almost a decade developing Big Data projects in Paradigma, through its R+D department
we were early adopters of Spark, which led to the creation of Stratio
THE FIRST SPARK-BASED BIG DATA
PLATFORM RELEASED
INTRO

3. JORGE LOPEZ-MALLA
After working with traditional
processing methods, I started to
do some R&S Big Data projects
and I fell in love with the Big Data
world. Currently i’m doing some
awesome Big Data projects at
Stratio
MY PROFILE
SKILLS

4. ALBERTO RODRÍGUEZ DE LEMA
After graduating I've been
programming for more than 10 years.
I’ve built high performance and
scalable web applications for
companies such as Indra Systems,
Prudential and Springer Verlag Ltd.
MY PROFILE
@ardlema
SKILLS

5. II
GO TO SPACE
STRATIO
OPEN-SOURCE SOLUTIONS
Our enterprises solutions are
based on open source
technologies
PURE SPARK
The only pure Spark platform,
the only global solution
ENTERPRISE SPARK
On – premise & cloud, our platform is
geared towards helping companies
SPARK-BASED BD PLATFORM
The first Spark-Based big data
platform released

6. OUR CLIENT
MIDDLE EAST TELCO COMPANY
o 9.500 mil. daily events processed
o 9.2 mil. clients

7. USE CASES

8. MANAGEMENT & NORMALIZATION OF DATA
SOURCES
USE CASES
1

9. USE CASES
MANAGEMENT & NORMALIZATION OF DATA
SOURCES
1

10. USE CASES
NETWORK COVERAGE IMPROVEMENT
2

11. USE CASES
PEOPLE GATHERING
3

12. USE CASES
PEOPLE GATHERING
3

13. USE CASES
DATA MONETIZATION
4

14. USE CASES
DATA MONETIZATION
4

18. DATA MONETIZATION
4
USE CASES

19. TECHNICAL CHALLENGES

20. TECHNICAL PROBLEMS
Huge volumen
of data
Huge size
of Data
Distributed
processing
Hard
to read
Recognized
patterns
1 2 3 4 5

21. 1 HUGE VOLUME OF DATA
SOLUTION
APACHE HADOOP DISTRIBUTED FILE SYSTEM (HDFS)

22. 1 HUGE VOLUME OF DATA
9500 mil. csv daily records -> circa 16 Gb
Requirements:
High availability
Concurrent file reads

23. 2 HUGE SIZE OF DATA
SOLUTION
APACHE PARQUET

24. 2 HUGE SIZE OF DATA
16.5 Gb of daily event information stored as csv text in HDFS
4.3 Gb of daily event information stored as parquet files in HDFS
STORE IMPROVEMENT Circa 70%

25. 2 HUGE SIZE OF DATA
Time to count daily csv events -> 6.2 minutes
.
Time to count daily Parquet events -> 1 minute
READ PROCESS IMPROVEMENT Circa 80%

26. 3 DISTRIBUTED PROCESSING
SOLUTION
APACHE SPARK

27. 3 DISTRIBUTED PROCESSING - REQUIREMENTS
Complex algorithmics with the minimum amount of
resources
Reduction of the process time in order to obtain data
when it still is used

28. 3 DISTRIBUTED PROCESSING - REQUIREMENTS
Sharing the cluster with legacy processes
Use of legacy outputs processes without does any
change

29. 4 HARD TO READ
SOLUTION
SCALA + APACHE SPARK

30. 4 HARD TO READ
Reducing developing time
LOCs dramatically reduced
Number of classes dramatically reduced

31. Tests and application readability improvements
DSLs make our lives easier
Spark makes Map Reduces jobs even simpler
4 HARD TO READ

32. 5 RECOGNIZED PATTERNS
SOLUTION
APACHE SPARK
MLLIB

33. Millons of data processed in order to obtain
mathematical models
Applied complex mathematical algorithms to obtain
accurate weekly behaviors
5 RECOGNIZED PATTERNS

34. THANK YOU
UNITED STATES
Tel: (+1) 408 5998830
EUROPE
Tel: (+34) 91 828 64 73
contact@stratio.com
www.stratio.com