Data scientists just want to do fast, interactive exploratory analytics on all kinds of data—without thinking about whether data fits in-memory, about parallelism, force-fitting it into a table, or pulling it out of a file and formatting it for math packages. You’d also like to use your favorite analytical language and have it transparently scale up to Big Data volumes. Paradigm4 presents a webinar about SciDB—the open source, array database with native scalable complex analytics, programmable from R and Python. Learn how SciDB enables you to: •Explore rich data sets interactively •Do complex math in-database—without being constrained by memory limitations •Perform fast multi-dimensional windowing, filtering, and aggregation •Offload large computations to a commodity hardware cluster—on-premise or in a cloud •Use R and Python to analyze SciDB arrays as if they were R or Python objects •Share data among users, with multi-user data integrity guarantees and version control

1. Big Analytics
without Big Hassles
Bryan Lewis
Chief Data Scientist
Alex Poliakov
Solutions Architect

2. Paradigm4’s SciDB
MPP
Database
Array
data
model
Complex
analytics
Commodity clusters or cloud
R & Python
Big analytics
without big hassles

3. ©Paradigm43
Using WebEx
• Ask questions using the
Q&A window
• This webinar is being
recorded
• Replays will be available

4. ©Paradigm44
Agenda
1. Brief Introduction to SciDB
2. Demos
3. Q & A

5. ©Paradigm45
Paradigm4 develops & supports SciDB
Force behind many major advances in databases
Postgres Vertica Paradigm4
Illustra VoltDB
Streambase DataTamer
Mike Stonebraker
CTO & Co-founder
MIT Professor
ISTC Big Data at MIT

6. ©Paradigm46
Presenters
Bryan Lewis, Chief Data Scientist
Applied Math Ph.D.
Founder Rocketcalc; RevolutionAnalytics
CRAN contributor
Alex Poliakov, Solutions Architect
Decade developing database internals (Netezza, Paradigm4)
Solutions: e-commerce, pharma/biotech, insurance, satellite imagery

7. ©Paradigm47
Three pillars of SciDB
MPP
Database
Array
data
model
Complex
analytics
Commodity clusters or cloud
R & Python
Big analytics
without big hassles

8. ©Paradigm48
SciDB Powers NIH NCBI’s
1000 Genomes Project
Running 24 x 7 since Fall 2012
http://www.ncbi.nlm.nih.gov/variation/tools/1000genomes/

9. ©Paradigm49
Some commercial use cases
Pharma, Biotech, Healthcare
Join public & private data
Integrate many data sources
Scale up & speed up math
Quant Finance
Fast data window selection
& scalable math
Image & Sensor Analytics
E-commerce
SVD on sparse matrices 50M x 50M
Powering recommendation engine
Integrate diverse data with different spatial
and temporal resolutions

10. ©Paradigm410
• ACID support means multiple users can
simultaneously read / write / analyze data
• FAST JOINs
data in files
SciDB is a Database

11. ©Paradigm411
Arrays are a natural data model
Sensor/car/phone
time
longitude
Event
other dimensions ….
latitude
Exchange
Stock_ID
Tim
e
other dimensions ….

12. ©Paradigm412
Native Array DBs vs. Relational DBs
Spatially close data in the
coordinate system are stored
close to each other on disk
Important for ordered data
and analysis

13. ©Paradigm413
Array storage supports fast
multi-dimensional SELECTs
Illustration credit: Andrei Pandre

14. ©Paradigm414
SciDB does scalable complex analytics
• No more ETL hassles to a separate math package
• Data not constrained to fit in memory
Parallel linear algebra
Principal component analysis
Clustering
GLM
Machine learning
and more

15. ©Paradigm415
• Program SciDB from R or Python
• Naturally reference & manipulate data in SciDB
• Large computations run on SciDB cluster
– Go beyond the scalability limitations of R & Python
Analyst-Friendly Interfaces
We also support AQL and JDBC

16. ©Paradigm416
Shared-Nothing Cluster Architecture
SciDB
Coordinator
SciDB
…
SciDB
1
SciDB
2
R + SciDB-R
Python + SciDB-Py
JDBC
Web Browser
K-replication for redundancy
Scale out horizontally

17. ©Paradigm417
SciDB Arrays
Each cell in a SciDB array consists of a fixed number
of typed attributes (variables).
Here is an example cell with four attributes
Price Volume Symbol usec
450.61 150 “AAPL” 36013008713

18. ©Paradigm418
SciDB Arrays
Dimensioni
Attributes
Price Volume Symbol usec
1 450.61 150 “AAPL” 36013008713
2 450.73 200 “AAPL” 36013008915
3 450.84 10 “AAPL” 36013208113
4 36.57 75 “MSFT” 36019008713
5 36.20 100 “MSFT” 36003200113
A 1-D array looks like an R or Pandas data frame.
This picture shows five cells, each with four attributes.

19. ©Paradigm419
SciDB Arrays
The same data “redimensioned” into a 2D array
Dimensionusec
“AAPL” “MSFT”
Price Volume Price Volume
36003200113 36.20 100
36013008713 450.61 150
36013008915 450.73 200
36013208113 450.84 10
36019008713 36.57 75
Dimension Symbol
.

20. ©Paradigm420
SciDB Array Schema
CREATE ARRAY Simple_Array
< v1 : double,
v2 : int64,
v3 : string >
[ I = 0:*, 5, 0, J = 0:9, 5, 0 ];
Attributes
v1, v2, v3
Dimensions
I, J
Dimension size
* is unbounded
Chunk
size
Chunk
overlap

21. ©Paradigm421
Arrays are distributed with overlap
Supports constant time moving window
aggregates and feature detection
…even when data cross node boundaries
0.02 0.01 0.01
0.01 0.01 0.50
0.01 0.02 0.01
0.01 0.01 0.02
0.01 0.50 0.02
0.02 0.01 0.01
0.01 0.01 0.50
0.01 0.02 0.01
0.02 0.01 0.02
0.01 0.50 0.02
0.02 0.01 0.01
0.01 0.02 0.02

22. ©Paradigm422
Live demonstrations
1) Airline data
• Select
• Aggregate lateness
• Heatmap
2) Netflix-like data
• SVD
3) Zipcode (lat,long) and population by zipcode
• Join
• Compute distance-weighted population by zipcode
• Plot histogram
4) Satellite and point-of-interest data
• Select region
• Regrid and plot
• Overlay another dataset: shopping mall locations

23. ©Paradigm423
Demonstration Cluster
Running on modest 4 node cluster
Each node has
16 cores
128 GB RAM
4 x 1TB disks
Connected by 1Gbit Ethernet
Also runs on public clouds

24. ©Paradigm424
Registration Poll Results
Excel,'15%'
MATLAB,'6%'
Other,'20%'
Python,'17%'
R,'42%'
What'mathemaAcal'and'staAsAcal'
compuAng'soGware'do'you'use?'''''
n'='340'
Please respond
to live poll

25. ©Paradigm425
Try It
Quick Start
• scidb.org/forum
• Download a VM or EC2 AMI
Community Edition Enterprise Edition
Open Source; Active forum Commercial license
Unrestricted & fully scalable Unrestricted & fully scalable
More math functions
Intel MKL support
Failover & fault tolerance
System management tools

26. Take Away: Less coding, more analysis
ACID database
Array data model
In-database complex math
Automatic scale-out & speed-up
Programmable from R and Python
www.paradigm4.com

27. © Paradigm4 Inc. 27
Questions?
Tell us about your application
• info@paradigm4.com
Try our Quick Start
• scidb.org/forum
• Download a VM or EC2 AMI
www.paradigm4.com
Thanks for your interest!