• Like
  • Save
Llnl talk
Upcoming SlideShare
Loading in...5
×
 

Llnl talk

on

  • 1,679 views

These slides are from a recent talk I gave at Lawrence Livermore Labs....

These slides are from a recent talk I gave at Lawrence Livermore Labs.

The talk gives an architectural outline of the MapR system and then discusses how this architecture facilitates large scale machine learning algorithms.

Statistics

Views

Total Views
1,679
Views on SlideShare
1,657
Embed Views
22

Actions

Likes
0
Downloads
51
Comments
0

2 Embeds 22

http://www.linkedin.com 17
https://www.linkedin.com 5

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Llnl talk Llnl talk Presentation Transcript

    • MapR Architecture and Machine Learning
      1
    • Outline
      MapR system overview
      Map-reduce review
      MapR architecture
      Performance Results
      Map-reduce on MapR
      Machine learning on MapR
    • Map-Reduce
      Shuffle
      Input
      Output
    • Bottlenecks and Issues
      Read-only files
      Many copies in I/O path
      Shuffle based on HTTP
      Can’t use new technologies
      Eats file descriptors
      Spills go to local file space
      Bad for skewed distribution of sizes
    • MapR Improvements
      Faster file system
      Fewer copies
      Multiple NICS
      No file descriptor or page-buf competition
      Faster map-reduce
      Uses distributed file system
      Direct RPC to receiver
      Very wide merges
    • MapR Innovations
      Volumes
      Distributed management
      Data placement
      Read/write random access file system
      Allows distributed meta-data
      Improved scaling
      Enables NFS access
      Application-level NIC bonding
      Transactionally correct snapshots and mirrors
    • MapR'sContainers
      Files/directories are sharded into blocks, whichare placed into mini NNs (containers ) on disks
      • Each container contains
      • Directories & files
      • Data blocks
      • Replicated on servers
      • No need to manage directly
      Containers are 16-32 GB segments of disk, placed on nodes
    • Container locations and replication
      CLDB
      N1, N2
      N1
      N3, N2
      N1, N2
      N2
      N1, N3
      N3, N2
      N3
      Container location database (CLDB) keeps track of nodes hosting each container
    • MapR Scaling
      Containers represent 16 - 32GB of data
      • Each can hold up to 1 Billion files and directories
      • 100M containers = ~ 2 Exabytes (a very large cluster)
      250 bytes DRAM to cache a container
      • 25GB to cache all containers for 2EB cluster
      • But not necessary, can page to disk
      • Typical large 10PB cluster needs 2GB
      Container-reports are 100x - 1000x < HDFS block-reports
      • Serve 100x more data-nodes
      • Increase container size to 64G to serve 4EB cluster
      • Map/reduce not affected
    • MapR's Streaming Performance
      11 x 7200rpm SATA
      11 x 15Krpm SAS
      MB
      per
      sec
      Higher is better
      Tests: i. 16 streams x 120GB ii. 2000 streams x 1GB
    • Terasort on MapR
      10+1 nodes: 8 core, 24GB DRAM, 11 x 1TB SATA 7200 rpm
      Elapsed time (mins)
      Lower is better
    • MUCH faster for some operations
      Same 10 nodes …
      Teststoppedhere
      Create
      Rate
      # of files (millions)
    • MUCH faster for some operations
    • NFS mounting models
      Export to the world
      NFS gateway runs on selected gateway hosts
      Local server
      NFS gateway runs on local host
      Enables local compression and check summing
      Export to self
      NFS gateway runs on all data nodes, mounted from localhost
    • Export to the world
      NFS
      Server
      NFS
      Server
      NFS
      Server
      NFS
      Server
      NFS
      Client
    • Local server
      Client
      Application
      NFS
      Server
      Cluster Nodes
    • Universal export to self
      Cluster Nodes
      Cluster
      Node
      Application
      NFS
      Server
    • Cluster
      Node
      Application
      NFS
      Server
      Cluster
      Node
      Application
      Cluster
      Node
      Application
      NFS
      Server
      NFS
      Server
      Nodes are identical
    • Shardedtext indexing
      Mapper assigns document to shard
      Shard is usually hash of document id
      Reducer indexes all documents for a shard
      Indexes created on local disk
      On success, copy index to DFS
      On failure, delete local files
      Must avoid directory collisions
      can’t use shard id!
      Must manage local disk space
    • Conventional data flows
      Failure of search engine requires another download of the index from clustered storage.
      Map
      Failure of a reducer causes garbage to accumulate in the local disk
      Reducer
      Clustered index storage
      Input documents
      Local
      disk
      Search
      Engine
      Local
      disk
    • Simplified NFS data flows
      Map
      Reducer
      Search
      Engine
      Input documents
      Clustered index storage
      Failure of a reducer is cleaned up by map-reduce framework
      Search engine reads mirrored index directly.
    • Application to machine learning
      So now we have the hammer
      Let’s see some nails!
    • K-means
      Classic E-M based algorithm
      Given cluster centroids,
      Assign each data point to nearest centroid
      Accumulate new centroids
      Rinse, lather, repeat
    • K-means, the movie
      Centroids
      Assign
      to
      Nearest
      centroid
      I
      n
      p
      u
      t
      Aggregate
      new
      centroids
    • But …
    • Parallel Stochastic Gradient Descent
      Model
      Train
      sub
      model
      I
      n
      p
      u
      t
      Average
      models
    • VariationalDirichlet Assignment
      Model
      Gather
      sufficient
      statistics
      I
      n
      p
      u
      t
      Update
      model
    • Old tricks, new dogs
      Mapper
      Assign point to cluster
      Emit cluster id, (1, point)
      Combiner and reducer
      Sum counts, weighted sum of points
      Emit cluster id, (n, sum/n)
      Output to HDFS
      Read from local disk from distributed cache
      Read from
      HDFS to local disk by distributed cache
      Written by map-reduce
    • Old tricks, new dogs
      Mapper
      Assign point to cluster
      Emit cluster id, 1, point
      Combiner and reducer
      Sum counts, weighted sum of points
      Emit cluster id, n, sum/n
      Output to HDFS
      Read from
      NFS
      Written by map-reduce
      MapR FS
    • Click modeling architecture
      Map-reduce
      Side-data
      Now via NFS
      Feature
      extraction
      and
      down
      sampling
      I
      n
      p
      u
      t
      Data
      join
      Sequential
      SGD
      Learning
    • Poor man’s Pregel
      Mapper
      Lines in bold can use conventional I/O via NFS
      while not done:
      read and accumulate input models
      for each input:
      accumulate model
      write model
      synchronize
      reset input format
      emit summary
      31
    • Trivial visualization interface
      Map-reduce output is visible via NFS
      Legacy visualization just works
      $ R
      > x <- read.csv(“/mapr/my.cluster/home/ted/data/foo.out”)
      > plot(error ~ t, x)
      > q(save=‘n’)
    • Conclusions
      We used to know all this
      Tab completion used to work
      5 years of work-arounds have clouded our memories
      We just have to remember the future