This document discusses IBM's Data Science Experience platform for performing data science at scale. It provides an overview of data science use cases like personalization, predictions, classifications, and analyzing unstructured data. It also discusses relevant algorithms and challenges with traditional tools. It introduces Apache Spark as a fast engine for big data processing and highlights IBM's partnership with Hortonworks to provide the #1 data science and SQL platform through products like IBM Data Science Experience and Hortonworks Data Platform.

1. IBM
Data Science eXperience
Data science at scale

2. Data Science ?

3. You said Data Science ?

4. Many use cases
Personalization
Predictions
and
Classifications
Unstructured Data
and
Natural Language
Patterns
Anomalies
Trends
Next Best Product Price Optimization

5. Relevant algorithms and processes
Clustering
Classification
Neural Networks
Regression
Search
Linear & Non-linear
Optimization
Sorting
Merging
Signal Processing
Encryption
Predictive / Statistical /
Machine Learning
Computer Science
Operations Research /
Applied Mathematics
Engineering
Not exhaustive

6. Challenge[s]

7. The standard tools nowadays

8. We need a way to distribute over several nodes the :
ü Data manipulation
ü Model training
ü Model optimization
Limitations of traditional technologies
Often computation intensive and need a large amount of data to be trained.
A single machine can only handle so much, is expensive to scale and is
prone to failures.

9. Introducing Apache Spark
Apache Spark is a fast and general engine for big data processing, with
built-in modules for streaming, SQL, machine learning and graph
processing. It is a distributed engine which processes data in-memory.
2015
201
4
2016
1000
2018
1200+

10. 0
200
400
600
800
1000
Databricks IBM Hortonworks Cloudera Intel IVU Traffic Technologies Tencent
Top 7 Contributing Companies to Spark 2.0.0
0
20
40
60
80
100
120
140
IBM Hortonworks Databricks Intel Cloudera Recruit
Technologies
Netflix Amazon Microsoft Tencent
Top 10 Contributing Companies to Spark ML/MLlib 2.0.0
Spark and IBM

11. Data Science
Experience

12. A solution to the challenges everyone
faces
Administration:
• Installation
• Management
• Security
• Self-service
Deployment:
• Seamless
• Monitoring
• Applications
• APIs
Tools:
• Open Source
• GUI
• Languages
• Scalability
• Extensible
Collaboration:
• Project
• Teams
• Versioning
• Sharing
• Reusable assets
Competences
Data:
• Metadata
• Formats
• Sources

13. Put simply, IBM and Hortonworks Combined Value is
Unequaled in the Market…
#1 Data Science Platform (Source: Gartner)
#1 SQL Engine for complex, analytical
workloads.
Leader in On-premise and Hybrid Cloud
solutions
IBM adopted Hortonworks Data Platform
(HDP) as its core Hadoop distribution
and resells HDP
#1 Pure Open Source Hadoop Distribution
1000+ customers and 2100+ ecosystem
partners
Employs the original architects, developers and
operators of Hadoop from Yahoo!
Hortonworks adopted and resells IBM
Data Science Experience (DSX) and IBM
Big SQL
+

14. Data Science at scale
/Questions?
IBM [A]n[a]lytics