Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

Membase Meetup - Silicon Valley

on

  • 1,923 views

 

Statistics

Views

Total Views
1,923
Views on SlideShare
1,896
Embed Views
27

Actions

Likes
0
Downloads
25
Comments
0

3 Embeds 27

http://blog.couchbase.com 16
http://blog.membase.com 10
http://www.stage.couchbase.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Get better channels at bottom myspace, forbesAbout ShareThis, stats/metrics consumer facing company but we collect data…scale...
  • Data used as signals for user modelSharingSocial Network usedSearch KeywordsViewsMembase stores user cookie -> segments
  • What we offerdisplay campaigns that are audience targeted,Example of ads, content, different partners etc.

Membase Meetup - Silicon Valley Membase Meetup - Silicon Valley Presentation Transcript

  • Silicon Valley
  • 2
    Tonight
    Membase Overview
    Use Cases and Deployment Examples
    Membase Architecture
    Demo!
    Developing with Membase
    A Glimpse into the Future
  • What is Membase?
  • Before: Application scales linearly, data hits wall
    Application Scales Out
    Just add more commodity web servers
    Database Scales Up
    Get a bigger, more complex server
    4
  • Membase is a distributed database
    5
    Application user
    Web application server
    Membase Servers
    In the data center
    On the administrator console
  • Built-in Memcached Caching Layer
    6
    Memcached
    Memcached
    Membase Database
    Membase Database
    Memcached Mode
    Membase Mode
    Membase development team has contributed over half of the source code to the Memcached project.
  • Deployment options
    7
  • Secure multitenant support
    8
    Application user
    Web application server
    Bucket 1
    Bucket 2
    Aggregate Cluster Memory and Disk Capacity
    Membase data servers
    In the data center
    On the administrator console
  • Five minutes or less to a working cluster
    Downloads for Linux and Windows
    Start with a single node
    One button press joins nodes to a cluster
    Easy to develop against
    Just SET and GET – no schema required
    Drop it in. 10,000+ existing applications already “speak membase” (via memcached)
    Practically every language and application framework is supported, out of the box
    Easy to manage
    One-click failover and cluster rebalancing
    Graphical and programmatic interfaces
    Configurable alerting
    Membase is Simple, Fast, Elastic
    9
  • Membase is Simple, Fast, Elastic
    10
    Predictable
    “Never keep an application waiting”
    Quasi-deterministic latency and throughput
    Low latency
    Built-in Memcached technology
    Auto-migration of hot data to lowest latency storage technology (RAM, SSD, Disk)
    Selectable write behavior – asynchronous, synchronous (on replication, persistence)
    High throughput
    Multi-threaded
    Low lock contention
    Asynchronous wherever possible
    Automatic write de-duplication
  • Membase is Simple, Fast, Elastic
    11
    Zero-downtime elasticity
    Spread I/O and data across commodity servers (or VMs)
    Consistent performance with linear cost
    Dynamic rebalancing of a live cluster
    All nodes are created equal
    No special case nodes
    Clone to grow
    Extensible
    Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse)
    Data bucket – engine API for specialized container types
    Membase NodeCode[FUTURE]
  • Proven at small, and extra large scale
    Leading cloud service (PAAS) provider
    Over 65,000 hosted applications
    Membase Server supporting over 3,000 Heroku customers
    Social game leader – FarmVille, Mafia Wars, Café World
    Over 230 million monthly users
    Zynga is a core contributor to and large scale user of Membase Server
    12
  • After: Data layer scales like application logic layerData layer now scales with linear cost and constant performance.
    Application Scales Out
    Just add more commodity web servers
    Membase Servers
    Database Scales Out
    Just add more commodity data servers
    Scaling out flattens the cost and performance curves.
    13
  • Membase - A practical path to “NoSQL” adoption
    14
  • Use Cases
  • 16
    Where does Membase Fit?
    Applications with many, many users
    Online, dynamic applications
    Applications with growing datasets which need quick access, growing audience
  • 17
    Deployments Leading Membase
    Leading cloud service (PAAS) provider
    Over 65,000 hosted applications
    Membase Server serving over 1,200 Heroku customers (as of June 10, 2010)
    Social game leader – FarmVille, Mafia Wars, Café World
    Over 230 million monthly users
    • Membase Serveris the 500,000 ops-per-second database behind FarmVille and Café World
  • Use case – Ad targeting
    18
    40 milliseconds to come up with an answer.
    profiles, real time campaign statistics
    3
    2
    1
    profiles, campaigns
    events
  • 19
    Search and Gaming Portal
    Database
  • Targeting atShareThis
  • Largest integrated sharing network
    We make sharing simple, engaging & valuable
    Powerful Social Analytics & Audience Monetization
    About ShareThis
    450/momillion consumers
    ~850thousand sites
    50+social channels
  • This is how it works
    Sharing Behavior
    Log Files
    Search Keywords
    Page Views
    HDFS
    Map/Reduce
    Content Analysis
    Taxonomy
    Ad Server
    User
    Membase
    2
  • ShareThis Ad Products
  • Membase Architecture
  • 25
    Clustering
    Underlying cluster functionality based on erlang OTP
    Have a custom, vector clock based way of storing and propagating...
    Cluster topology
    vBucket mapping
    Collect statistics from many nodes of the cluster
    Identify hot keys, resource utilization
    25
  • 26
  • 27
    TAP
    A generic, scalable method of streaming mutations from a given server
    As data operations arrive, they can be sent to arbitrary TAP receivers
    Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data
    Three modes of operation
  • 11211
    11210
    memcapable 1.0
    memcapable 2.0
    moxi
    memcached
    protocol listener/sender
    REST management API/Web UI
    vBucket state and replication manager
    Node health monitor
    Rebalance orchestrator
    Heartbeat
    Process monitor
    Global singleton supervisor
    Configuration manager
    engine interface
    membase storage engine
    http
    on each node
    one per cluster
    Erlang/OTP
    HTTP
    distributed erlang
    erlang port mapper
    21100 – 21199
    4369
    8080
    Membase Architecture
  • 29
    Clients, nodes and other nodes
  • 30
    Data buckets are secure membase “slices”
    Application user
    Web application server
    Bucket 1
    Bucket 2
    Aggregate Cluster Memory and Disk Capacity
    Membase data servers
    In the data center
    On the administrator console
  • 11211
    11210
    memcapable 1.0
    memcapable 2.0
    moxi
    memcached
    protocol listener/sender
    REST management API/Web UI
    vBucket state and replication manager
    Rebalance orchestrator
    Node health monitor
    Data Manager
    Cluster Manager
    Global singleton supervisor
    Configuration manager
    Heartbeat
    Process monitor
    engine interface
    membase storage engine
    http
    on each node
    one per cluster
    Erlang/OTP
    HTTP
    distributed erlang
    erlang port mapper
    21100 – 21199
    4369
    8080
    Membase Architecture
  • 32
    vBucket mapping
  • 33
    Membase “write” data flow – application view
    User action results in the need to change the VALUE of KEY
    1
    Application updates key’s VALUE, performs SET operation
    2
    4
    Membase (memcached) client hashes KEY, identifies KEY’s master server
    3
    SET request sent over network to master server
    5
    Membase replicates KEY-VALUE pair, caches it in memory and stores it to disk
  • 34
    Disk > Memory
    Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase.
    Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important.
  • Membase Demo
  • 36
    Thanks!
  • Key-Value Patterns
  • 38
    Key-Value
    (with a replica)
    Items have:
    Key
    Value
    Expiration
    Flags
    CAS (more on this later)
    Operations include:
    Get/Set
    Increment/Decrement
    Append/Prepend
  • 39
    Membase Datatypes
    “Any customer can have a car painted any colour that he wants so long as it is black.”
    byte[]
    Does your data have 1s and 0s?
    • Items do have flags
    • Many clients use flags
    • Data type options
    • Google protobuf
    • Thrift
    • Avro
  • 40
    Transactions
    Lock == slow me down
    CAS operations
    Optimistic locking
    Very useful with complex datatypes
    Imagine two clients trying to update a complex item
    You’re likely using CAS already... if you use a CPU
    User 1
    User 2
    Success
    Fail!
  • 41
    Common Use: Sessions
    Web user sessions
    Highly read, less writes in many case
    Protocol advantage of memcached
    Options already for PHP, Ruby and Java
    Application state
    Not necessarily “entity” style things
    May be appropriate for a “cache” pool
  • 42
    Common Use (cache): Rate Limiting
    Want to provide API calls into the system
    Twitter search
    Google search services
    Use the atomic increment
    Set an item with a unique ID
    Upon API request, increment and check
    HTTP 420: go away and come back later
    Your Users
    ¡Ouch!
    Your App
  • Looking Ahead: NodeCodeFrank Weigel, Membase
  • 44
    Beyond key-value
    • Indexing/Range Queries
    • Advanced Data Structures
    • Sub-object direct manipulation
    Validation and In-flight transformation
    • Block mutations failing validation
    • Enrich or transform objects
    Connectors (Integrate easily with other systems)
    • Solr
    • Hadoop
    • MySQL
    NodeCode – Motivation
  • 45
    NodeCode - What is it?
    Method for extending & customizing Membase
    Separate code modules
    Defined interface to datapath and cluster manager
    Notification on events
    • Synchronous
    • Asynchronous
  • 46
    Simple
    • Packaged modules for easy install and enable
    • Library of “off the shelf” modules
    • Module monitoring
    • Straight forward development and debugging
    Fast
    • Low latency/high-throughput
    • Per-bucket process isolation
    • Don’t break data manager performance/correctness
    Elastic
    • Automatically migrate and instantiate on rebalance
    • Provide support for migration of internal data
    • Leverage native Membase engine for internal data storage
    NodeCode – Drivers
  • 47
    Block-level architecture
  • 48
    Java only
    • jar format
    Must implement minimal module API
    • Initial module startup
    • Module removal
    • Association with bucket
    NodeCode library helper functions
    • Register synchronous & asynchronous listeners/callbacks
    • Register protocol extension/callbacks
    • Register rebalance callback
    • Register cluster manager event callbacks
    • Membase data access
    NodeCode 1.0 Plans
  • 50
    Q&A
  • 51
    Attributions
    http://commons.wikimedia.org/wiki/File:Flag_of_China.png
    http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg
    http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg