• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
MapR, Implications for Integration
 

MapR, Implications for Integration

on

  • 2,666 views

Ted Dunning's talk for the Sept. 2011 TriHUG (www.trihug.org) meeting. Cover's MapR's distributed file system and map-reduce technology.

Ted Dunning's talk for the Sept. 2011 TriHUG (www.trihug.org) meeting. Cover's MapR's distributed file system and map-reduce technology.

Statistics

Views

Total Views
2,666
Views on SlideShare
1,395
Embed Views
1,271

Actions

Likes
0
Downloads
65
Comments
0

5 Embeds 1,271

http://www.trihug.org 1242
http://dev-wiki.rakuten.co.jp 12
http://dev-wiki.rakuten.co.jp 12
http://www.tumblr.com 4
http://trihug.tumblr.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    MapR, Implications for Integration MapR, Implications for Integration Presentation Transcript

    • MapR, Implications for Integration
      CHUG – August 2011
    • Outline
      MapR system overview
      Map-reduce review
      MapR architecture
      Performance Results
      Map-reduce on MapR
      Architectural implications
      Search indexing / deployment
      EM algorithm for machine learning
      … and more …
    • Map-Reduce
      Shuffle
      Input
      Output
    • Bottlenecks and Issues
      Read-only files
      Many copies in I/O path
      Shuffle based on HTTP
      Can’t use new technologies
      Eats file descriptors
      Spills go to local file space
      Bad for skewed distribution of sizes
    • MapR Areas of Development
    • MapR Improvements
      Faster file system
      Fewer copies
      Multiple NICS
      No file descriptor or page-buf competition
      Faster map-reduce
      Uses distributed file system
      Direct RPC to receiver
      Very wide merges
    • MapR Innovations
      Volumes
      Distributed management
      Data placement
      Read/write random access file system
      Allows distributed meta-data
      Improved scaling
      Enables NFS access
      Application-level NIC bonding
      Transactionally correct snapshots and mirrors
    • MapR'sContainers
      Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks
      • Each container contains
      • Directories & files
      • Data blocks
      • Replicated on servers
      • No need to manage directly
      Containers are 16-32 GB segments of disk, placed on nodes
    • Container locations and replication
      CLDB
      N1, N2
      N1
      N3, N2
      N1, N2
      N2
      N1, N3
      N3, N2
      N3
      Container location database (CLDB) keeps track of nodes hosting each container
    • MapR Scaling
      Containers represent 16 - 32GB of data
      • Each can hold up to 1 Billion files and directories
      • 100M containers = ~ 2 Exabytes (a very large cluster)
      250 bytes DRAM to cache a container
      • 25GB to cache all containers for 2EB cluster
      • But not necessary, can page to disk
      • Typical large 10PB cluster needs 2GB
      Container-reports are 100x - 1000x < HDFS block-reports
      • Serve 100x more data-nodes
      • Increase container size to 64G to serve 4EB cluster
      • Map/reduce not affected
    • MapR's Streaming Performance
      11 x 7200rpm SATA
      11 x 15Krpm SAS
      MB
      per
      sec
      Higher is better
      Tests: i. 16 streams x 120GB ii. 2000 streams x 1GB
    • Terasort on MapR
      10+1 nodes: 8 core, 24GB DRAM, 11 x 1TB SATA 7200 rpm
      Elapsed time (mins)
      Lower is better
    • HBase on MapR
      YCSB Random Read with 1 billion 1K records
      10+1 node cluster: 8 core, 24GB DRAM, 11 x 1TB 7200 RPM
      Recordspersecond
      Higher is better
    • Small Files (Apache Hadoop, 10 nodes)
      Out of box
      Op: - create file - write 100 bytes - close
      Notes:
      - NN not replicated
      - NN uses 20G DRAM
      - DN uses 2G DRAM
      Tuned
      Rate (files/sec)
      # of files (m)
    • MUCH faster for some operations
      Same 10 nodes …
      Create
      Rate
      # of files (millions)
    • What MapR is not
      Volumes != federation
      MapR supports > 10,000 volumes all with independent placement and defaults
      Volumes support snapshots and mirroring
      NFS != FUSE
      Checksum and compress at gateway
      IP fail-over
      Read/write/update semantics at full speed
      MapR != maprfs
    • New Capabilities
    • NFS mounting models
      Export to the world
      NFS gateway runs on selected gateway hosts
      Local server
      NFS gateway runs on local host
      Enables local compression and check summing
      Export to self
      NFS gateway runs on all data nodes, mounted from localhost
    • Export to the world
      NFS
      Server
      NFS
      Server
      NFS
      Server
      NFS
      Server
      NFS
      Client
    • Local server
      Client
      Application
      NFS
      Server
      Cluster Nodes
    • Universal export to self
      Cluster Nodes
      Cluster
      Node
      Task
      NFS
      Server
    • Cluster
      Node
      Task
      NFS
      Server
      Cluster
      Node
      Task
      Cluster
      Node
      Task
      NFS
      Server
      NFS
      Server
      Nodes are identical
    • Application architecture
      So now we have a hammer
      Let’s find us some nails!
    • Sharded text Indexing
      Index text to local disk and then copy index to distributed file store
      Assign documents to shards
      Map
      Reducer
      Clustered index storage
      Input documents
      Copy to local disk typically required before index can be loaded
      Local
      disk
      Search
      Engine
      Local
      disk
    • Shardedtext indexing
      Mapper assigns document to shard
      Shard is usually hash of document id
      Reducer indexes all documents for a shard
      Indexes created on local disk
      On success, copy index to DFS
      On failure, delete local files
      Must avoid directory collisions
      can’t use shard id!
      Must manage and reclaim local disk space
    • Conventional data flow
      Failure of search engine requires another download of the index from clustered storage.
      Map
      Failure of a reducer causes garbage to accumulate in the local disk
      Reducer
      Clustered index storage
      Input documents
      Local
      disk
      Search
      Engine
      Local
      disk
    • Simplified NFS data flows
      Index to task work directory via NFS
      Map
      Reducer
      Search
      Engine
      Input documents
      Clustered index storage
      Failure of a reducer is cleaned up by map-reduce framework
      Search engine reads mirrored index directly.
    • Simplified NFS data flows
      Search
      Engine
      Mirroring allows exact placement of index data
      Map
      Reducer
      Input documents
      Search
      Engine
      Aribitrary levels of replication also possible
      Mirrors
    • How about another one?
    • K-means
      Classic E-M based algorithm
      Given cluster centroids,
      Assign each data point to nearest centroid
      Accumulate new centroids
      Rinse, lather, repeat
    • K-means, the movie
      Centroids
      Assign
      to
      Nearest
      centroid
      I
      n
      p
      u
      t
      Aggregate
      new
      centroids
    • But …
    • Parallel Stochastic Gradient Descent
      Model
      Train
      sub
      model
      I
      n
      p
      u
      t
      Average
      models
    • VariationalDirichlet Assignment
      Model
      Gather
      sufficient
      statistics
      I
      n
      p
      u
      t
      Update
      model
    • Old tricks, new dogs
      Mapper
      Assign point to cluster
      Emit cluster id, (1, point)
      Combiner and reducer
      Sum counts, weighted sum of points
      Emit cluster id, (n, sum/n)
      Output to HDFS
      Read from local disk from distributed cache
      Read from
      HDFS to local disk by distributed cache
      Written by map-reduce
    • Old tricks, new dogs
      Mapper
      Assign point to cluster
      Emit cluster id, (1, point)
      Combiner and reducer
      Sum counts, weighted sum of points
      Emit cluster id, (n, sum/n)
      Output to HDFS
      Read from
      NFS
      Written by map-reduce
      MapR FS
    • Poor man’s Pregel
      Mapper
      Lines in bold can use conventional I/O via NFS
      while not done:
      read and accumulate input models
      for each input:
      accumulate model
      write model
      synchronize
      reset input format
      emit summary
      37
    • Click modeling architecture
      Map-reduce
      Side-data
      Now via NFS
      Feature
      extraction
      and
      down
      sampling
      I
      n
      p
      u
      t
      Data
      join
      Sequential
      SGD
      Learning
    • Click modeling architecture
      Map-reduce
      Map-reduce
      Side-data
      Map-reduce cooperates with NFS
      Sequential
      SGD
      Learning
      Feature
      extraction
      and
      down
      sampling
      Sequential
      SGD
      Learning
      I
      n
      p
      u
      t
      Data
      join
      Sequential
      SGD
      Learning
      Sequential
      SGD
      Learning
    • And another…
    • Hybrid model flow
      Map-reduce
      Map-reduce
      Feature extraction
      and
      down sampling
      Down
      stream
      modeling
      Deployed
      Model
      ??
      SVD
      (PageRank)
      (spectral)
    • Hybrid model flow
      Feature extraction
      and
      down sampling
      Down
      stream
      modeling
      Deployed
      Model
      Sequential
      Map-reduce
      SVD
      (PageRank)
      (spectral)
    • And visualization…
    • Trivial visualization interface
      Map-reduce output is visible via NFS
      Legacy visualization just works
      $ R
      > x <- read.csv(“/mapr/my.cluster/home/ted/data/foo.out”)
      > plot(error ~ t, x)
      > q(save=‘n’)
    • Conclusions
      We used to know all this
      Tab completion used to work
      5 years of work-arounds have clouded our memories
      We just have to remember the future