NYC Hadoop Meetup - MapR, Architecture, Philosophy and Applications

MapR, Architecture, Philosophy and Applications
NY HUG – October 2011

Outline
Architecture (MapR)
Philosophy
Architectural (Machine learning)

Map-Reduce, the Original Mission
Shuffle
Input
Output

Bottlenecks and Issues
Read-only files
Many copies in I/O path
Shuffle based on HTTP
Can’t use new technologies
Eats file descriptors
Spills go to local file space
Bad for skewed distribution of sizes

MapR Areas of Development

MapR Improvements
Faster file system
Fewer copies
Multiple NICS
No file descriptor or page-buf competition
Faster map-reduce
Uses distributed file system
Direct RPC to receiver
Very wide merges

MapR Innovations
Volumes
Distributed management
Data placement
Read/write random access file system
Allows distributed meta-data
Improved scaling
Enables NFS access
Application-level NIC bonding
Transactionally correct snapshots and mirrors

MapR'sContainers
Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks
Each container contains
Directories & files
Data blocks
Replicated on servers
No need to manage directly
Containers are 16-32 GB segments of disk, placed on nodes

MapR'sContainers
Each container has a replication chain
Updates are transactional
Failures are handled by rearranging replication

MapR'sContainers
Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks
Each container contains
Directories & files
Data blocks
Replicated on servers
No need to manage directly
Containers are 16-32 GB segments of disk, placed on nodes

Container locations and replication
CLDB
N1, N2
N1
N3, N2
N1, N2
N2
N1, N3
N3, N2
N3
Container location database (CLDB) keeps track of nodes hosting each container and replication chain order

MapR Scaling
Containers represent 16 - 32GB of data
Each can hold up to 1 Billion files and directories
100M containers = ~ 2 Exabytes (a very large cluster)
250 bytes DRAM to cache a container
25GB to cache all containers for 2EB cluster
But not necessary, can page to disk
Typical large 10PB cluster needs 2GB
Container-reports are 100x - 1000x < HDFS block-reports
Serve 100x more data-nodes
Increase container size to 64G to serve 4EB cluster
Map/reduce not affected

MapR's Streaming Performance
11 x 7200rpm SATA
11 x 15Krpm SAS
MB
per
sec
Higher is better
Tests: i. 16 streams x 120GB ii. 2000 streams x 1GB

Terasort on MapR
10+1 nodes: 8 core, 24GB DRAM, 11 x 1TB SATA 7200 rpm
Elapsed time (mins)
Lower is better

HBase on MapR
YCSB Random Read with 1 billion 1K records
10+1 node cluster: 8 core, 24GB DRAM, 11 x 1TB 7200 RPM
Recordspersecond
Higher is better

Small Files (Apache Hadoop, 10 nodes)
Out of box
Op: - create file - write 100 bytes - close
Notes:
- NN not replicated
- NN uses 20G DRAM
- DN uses 2G DRAM
Tuned
Rate (files/sec)
# of files (m)

MUCH faster for some operations
Same 10 nodes …
Create
Rate
# of files (millions)

What MapR is not
Volumes != federation
MapR supports > 10,000 volumes all with independent placement and defaults
Volumes support snapshots and mirroring
NFS != FUSE
Checksum and compress at gateway
IP fail-over
Read/write/update semantics at full speed
MapR != maprfs

Philosophy

Physics of startup companies

For startups
History is always small
The future is huge
Must adopt new technology to survive
Compatibility is not as important
In fact, incompatibility is assumed

Physics of large companies
Absolute growth still very large
Startup phase

For large businesses
Present state is always large
Relative growth is much smaller
Absolute growth rate can be very large
Must adopt new technology to survive
Cautiously!
But must integrate technology with legacy
Compatibility is crucial

The startup technology picture
No compatibility requirement
Old computers
and software
Expected hardware
and software growth
Current computers
and software

The large enterprise picture
Must work
together
?
Proof of concept Hadoop cluster
Current hardware
and software
Long-term Hadoop cluster

What does this mean?
Hadoop is very, very good at streaming through things in batch jobs
Hbase is good at persisting data in very write-heavy workloads
Unfortunately, the foundation of both systems is HDFS which does not export or import well

Narrow Foundations
Pig
Hive
Big data is heavy and expensive to move.
Web Services
Sequential File Processing
OLAP
OLTP
Map/
Reduce
Hbase
RDBMS
NAS
HDFS

Narrow Foundations
Because big data has inertia, it is difficult to move
It costs time to move
It costs reliability because of more moving parts
The result is many duplicate copies

One Possible Answer
Widen the foundation
Use standard communication protocols
Allow conventional processing to share with parallel processing

Broad Foundation
Pig
Hive
Web Services
Sequential File Processing
OLAP
OLTP
Map/
Reduce
Hbase
RDBMS
NAS
HDFS
MapR

New Capabilities

Export to the world
NFS
Server
NFS
Server
NFS
Server
NFS
Server
NFS
Client

Local server
Client
Application
NFS
Server
Cluster Nodes

Universal export to self
Cluster Nodes
Cluster
Node
Task
NFS
Server

Cluster
Node
Task
NFS
Server
Cluster
Node
Task
Cluster
Node
Task
NFS
Server
NFS
Server
Nodes are identical

Application architecture
High performance map-reduce is nice
But algorithmic flexibility is even nicer

Hybrid model flow
Map-reduce
Map-reduce
Feature extraction
and
down sampling
Down
stream
modeling
Deployed
Model
??
SVD
(PageRank)
(spectral)

Hybrid model flow
Feature extraction
and
down sampling
Down
stream
modeling
Deployed
Model
Sequential
Map-reduce
SVD
(PageRank)
(spectral)

Shardedtext indexing
Mapper assigns document to shard
Shard is usually hash of document id
Reducer indexes all documents for a shard
Indexes created on local disk
On success, copy index to DFS
On failure, delete local files
Must avoid directory collisions
can’t use shard id!
Must manage and reclaim local disk space

Sharded text Indexing
Index text to local disk and then copy index to distributed file store
Assign documents to shards
Map
Reducer
Clustered index storage
Input documents
Copy to local disk typically required before index can be loaded
Local
disk
Search
Engine
Local
disk

Conventional data flow
Failure of search engine requires another download of the index from clustered storage.
Map
Failure of a reducer causes garbage to accumulate in the local disk
Reducer
Input documents
Local
disk
Search
Engine
Local
disk

Simplified NFS data flows
Index to task work directory via NFS
Map
Reducer
Search
Engine
Input documents
Failure of a reducer is cleaned up by map-reduce framework
Search engine reads mirrored index directly.

Simplified NFS data flows
Search
Engine
Mirroring allows exact placement of index data
Map
Reducer
Input documents
Search
Engine
Aribitrary levels of replication also possible
Mirrors

K-means
Classic E-M based algorithm
Given cluster centroids,
Assign each data point to nearest centroid
Accumulate new centroids
Rinse, lather, repeat

K-means, the movie
Centroids
Assign
to
Nearest
centroid
I
n
p
u
t
Aggregate
new
centroids

Parallel Stochastic Gradient Descent
Model
Train
sub
model
I
n
p
u
t
Average
models

VariationalDirichlet Assignment
Model
Gather
sufficient
statistics
I
n
p
u
t
Update
model

Old tricks, new dogs
Mapper
Assign point to cluster
Emit cluster id, (1, point)
Combiner and reducer
Sum counts, weighted sum of points
Emit cluster id, (n, sum/n)
Output to HDFS
Read from local disk from distributed cache
Read from
HDFS to local disk by distributed cache
Written by map-reduce

Old tricks, new dogs
Mapper
Assign point to cluster
Emit cluster id, (1, point)
Combiner and reducer
Sum counts, weighted sum of points
Emit cluster id, (n, sum/n)
Output to HDFS
Read from
NFS
Written by map-reduce
MapR FS

Poor man’s Pregel
Mapper
Lines in bold can use conventional I/O via NFS
while not done:
read and accumulate input models
for each input:
accumulate model
write model
synchronize
reset input format
emit summary
51

Click modeling architecture
Map-reduce
Side-data
Now via NFS
Feature
extraction
and
down
sampling
I
n
p
u
t
Data
join
Sequential
SGD
Learning

Click modeling architecture
Map-reduce
Map-reduce
Side-data
Map-reduce cooperates with NFS
Sequential
SGD
Learning
Feature
extraction
and
down
sampling
Sequential
SGD
Learning
I
n
p
u
t
Data
join
Sequential
SGD
Learning
Sequential
SGD
Learning

Trivial visualization interface
Map-reduce output is visible via NFS
Legacy visualization just works
$ R
> x <- read.csv(“/mapr/my.cluster/home/ted/data/foo.out”)
> plot(error ~ t, x)
> q(save=‘n’)

Conclusions
We used to know all this
Tab completion used to work
5 years of work-arounds have clouded our memories
We just have to remember the future

NYC Hadoop Meetup - MapR, Architecture, Philosophy and Applications

by Jason Shao on Oct 12, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 352

Statistics

NYC Hadoop Meetup - MapR, Architecture, Philosophy and Applications — Presentation Transcript

http://www.pulsepoint.com	347
http://beta.pulsepoint.com	5