SlideShare a Scribd company logo
HBase at Xiaomi
Jianwei Cui, Shaohui Liu
{cuijianwei, liushaohui}@xiaomi.com
About Xiaomi
Sold 60M phones in 2014, 3X of 2013
Guinness World Record: selling 2.11M phones online in 24h
2 / 35
Our HBase Team
8 Developers
Honghua Feng
Liang Xie
Jianwei Cui
Liangliang He
YingChao Zhou
Qiming Cheng
Guanghao Zhang
Shaohui Liu
130 patches submmitted in 2014, 82 committed
3 / 35
Agenda
1. Current Status
2. Problems and Solutions
3. Themis
4 / 35
Clusters and Scenarios
Mainland China
20 online clusters / 2 offline clusters in 3 data centers
AWS
4 online clusters / 1 offline cluster in 2 regions
Online Service
Mi Cloud, Mi Push, Galaxy, Mi Message,...
Offline Processing
User Profile, Trace, Recommendation, ...
5 / 35
Scenario A: Mi Cloud
Personal cloud storage for smart phones
Numbers
90+ milion users, 3X increased in 2014
500 billion rows, 6X increased in 2014
1000+ regions in the largest table
See: https://i.mi.com
6 / 35
Scenario B: Mi Push
Push service on android
Data stored in HBase
Pub-sub relations of topics and devices
Messages to each device
Numbers
200+ milion users
Push 2 billion+ messages every day
200,000+ requests per second at peak
7 / 35
Deployment
Two clusters with master-master replication in different data centers
Client switches clusters through configs on ZooKeeper
Using canary for availability check and alerts
8 / 35
Agenda
1. Current Status
2. Problems and Solutions
3. Themis
9 / 35
Long Full GC Pauses for RegionServer
Problem: Long full GC pauses making ZooKeeper session expire
zookeeper.session.timeout = 30s
Full GC pause of RegionServer with heap of 30G can be 40s
Solution:
Multi regionserver instances in a node
More memory on offheap using bucket cache
10 / 35
Hotspot for Temporal Data
Problem: Writes of temporal Data go to a small set of regions
Solution: Salted Table
Based on SaltedHTable opensourced by Intel Hadoop team
See: https://github.com/intel-hadoop/SaltedHTable
Transparent to applications by table schema support
MapReduce support
11 / 35
Coordinated Compaction
Problem: Compaction storm
Solution:
A compaction manager in HMaster coordinates all the compactions in the
cluster
Before a compaction starts, regionserver needs to acquire a compaction
quota
12 / 35
Exception Aggregation
Purposes: Find the potential bugs in the clusters
Solution:
Write HMaster/RegionServer log asynchronously to HDFS through Scribe
Using MapReduce to aggregate errors and exceptions of clusters
13 / 35
Table Based Replication Queue (in progress)
Problems:
Too much data stored on ZooKeeper
Over 200MB replication data for a disabled peer
Too many writes to ZooKeeper
5k/s writes to ZooKeeper in a cluster with 100k/s writes (HBASE-12636)
14 / 35
Table Based Replication Queue (in progress)
Solution: Move replication queue to a system table
Row key : server name + peer id + hlog name
One column records the offset at which the log is replicated
15 / 35
Asynchronous Event Notification (in progress)
Purposes:
Incremental statistics of data in HBase
Table schema transformation
Asynchronous data indexing
Solution:
An asynchronous event notification framework on HBase (HBASE-12884)
Replication based implementation:
Add a fake replication peer, which can receive the WAL edits from HBase
clusters
16 / 35
Agenda
1. Current Status
2. Problems and Solutions
3. Themis
17 / 35
Cross-Row Transaction
Why cross-row transaction?
Cross-row data consistency
Rows in different regions / tables
Example
Music index building
18 / 35
Cross-Row Transaction
Features
ACID
No central coordinator
Integrated without HBase code change
Google’s Percolator
Large-scale Incremental Processing Using Distribute Transactions and
Notifications, by Daniel Peng and Frank Dabek, 2010
Themis
https://github.com/Xiaomi/themis
Provides cross-row transactions on HBase based on Percolator
19 / 35
Themis Infrastructure
Timestamp server
Use timestamp of KeyValue internally
Timestamp must be globally incremental
Client
Coprocessor
20 / 35
Timestamp Server
Seperate a long type timestamp into two parts :
Higher 46 bits: Sync with system time
Lower 18 bits: Incremental counter
Hundreds of thousands unique timestamps in each millisecond
Incremental in one timestamp server
21 / 35
Timestamp Server
Incremental cross timestamp servers
Periodically save a future timestamp into ZooKeeper
Allocated timestamp must be smaller than saved timestamp
Another server needs to read the saved timestamp when starting
High availability
High throughput : 600,000 RPCs per second
Batch concurrent requests in one RPC
22 / 35
Themis
Cross-row mutation example : Cash Table
Rows for Bob and Joe are in different regions
Transfer $3 from Joe to Bob atomically
Two auxiliary columns : lock column and commit column
Two-phase commit
Prewrite Phase
Commit Phase
checkAndMutate of HBase : guarantee the atomicity for a single row
23 / 35
Prewrite Phase
Fetch a prewrite timestamp from timestamp server (prewriteTs=99)
Select primary and secondary columns
Primary
Column : (Joe, f:c)
PrimaryLock: {secondaries : [(Bob, f:c)]}
Secondaries
Column : (Bob, f:c)
SecondaryLock: {primary : (Joe, f:c)}
24 / 35
Prewrite Phase
Prewrite primary column
Write primary lock and data if no lock exists in lock column
checkAndMutate of HBase to guarantee the atomicity
Prevent other clients mutating the same column concurrently
25 / 35
Prewrite Phase
Prewrite secondary columns
Follow the same steps of prewriting primary column
26 / 35
Commit Phase
Fetch commit timestamp from timestamp server (commitTs=100)
Commit Primary
Delete the lock and write commit column if the lock exists
checkAndMutate of HBase to guarantee the atmocity
Decide the success or failure the whole transaction
27 / 35
Commit Phase
Commit secondaries
Delete the lock and write commit column atomically
28 / 35
Themis Read
Fetch a read timestamp (readTs=101)
Read commit columns with commitTs < readTs
(Joe, c:f#c) => (100 : 99)
(Bob, c:f#c) => (100 : 99)
Read data column with prewriteTs
prewriteTs is just the value of commit column
(Joe, 99: $17) and (Bob, 99: $12)
29 / 35
Performance Comparison
Single-Row Transaction
The worst case compared with raw HBase
One region server with 10GB heap memory
Write Performance : Preload 3 million rows, 256MB LRU cache
Read Performance : Preload 30GB data
30 / 35
Performance Comparison
Single-Row Transaction
The worst case compared with raw HBase
One region server with 10GB heap memory
Write Performance
Read Performance
31 / 35
Performance Comparison
Themis vs. Percolator
Write optimization
Set lock family IN MEMORY
Not sync the lock of prewrite phase
32 / 35
Future Work of Themis
Generic transaction API: HBASE-11447
Support different isolation levels
Global secondary index
33 / 35
34 / 35
Thanks! Questions?
Contacts: {cuijianwei, liushaohui}@xiaomi.com

More Related Content

What's hot

HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon
 
HBase 0.20.0 Performance Evaluation
HBase 0.20.0 Performance EvaluationHBase 0.20.0 Performance Evaluation
HBase 0.20.0 Performance Evaluation
Schubert Zhang
 
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaHBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
Cloudera, Inc.
 
Meet hbase 2.0
Meet hbase 2.0Meet hbase 2.0
Meet hbase 2.0
enissoz
 
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon
 
HBase: Where Online Meets Low Latency
HBase: Where Online Meets Low LatencyHBase: Where Online Meets Low Latency
HBase: Where Online Meets Low Latency
HBaseCon
 
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaHBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
Cloudera, Inc.
 
HBase Backups
HBase BackupsHBase Backups
HBase Backups
HBaseCon
 
HBaseCon 2013: Apache HBase Replication
HBaseCon 2013: Apache HBase ReplicationHBaseCon 2013: Apache HBase Replication
HBaseCon 2013: Apache HBase Replication
Cloudera, Inc.
 
Apache HBase, Accelerated: In-Memory Flush and Compaction
Apache HBase, Accelerated: In-Memory Flush and Compaction Apache HBase, Accelerated: In-Memory Flush and Compaction
Apache HBase, Accelerated: In-Memory Flush and Compaction
HBaseCon
 
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
Cloudera, Inc.
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
enissoz
 
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseHBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
Cloudera, Inc.
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
HBaseCon
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
HBaseCon
 
Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path
HBaseCon
 
HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, PhotobucketHBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
Cloudera, Inc.
 
Real-time HBase: Lessons from the Cloud
Real-time HBase: Lessons from the CloudReal-time HBase: Lessons from the Cloud
Real-time HBase: Lessons from the Cloud
HBaseCon
 
HBase Read High Availability Using Timeline Consistent Region Replicas
HBase  Read High Availability Using Timeline Consistent Region ReplicasHBase  Read High Availability Using Timeline Consistent Region Replicas
HBase Read High Availability Using Timeline Consistent Region Replicas
enissoz
 
Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera Field
HBaseCon
 

What's hot (20)

HBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ SalesforceHBaseCon 2015: HBase Performance Tuning @ Salesforce
HBaseCon 2015: HBase Performance Tuning @ Salesforce
 
HBase 0.20.0 Performance Evaluation
HBase 0.20.0 Performance EvaluationHBase 0.20.0 Performance Evaluation
HBase 0.20.0 Performance Evaluation
 
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - ClouderaHBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
HBaseCon 2012 | Base Metrics: What They Mean to You - Cloudera
 
Meet hbase 2.0
Meet hbase 2.0Meet hbase 2.0
Meet hbase 2.0
 
HBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a FlurryHBaseCon 2015: HBase Operations in a Flurry
HBaseCon 2015: HBase Operations in a Flurry
 
HBase: Where Online Meets Low Latency
HBase: Where Online Meets Low LatencyHBase: Where Online Meets Low Latency
HBase: Where Online Meets Low Latency
 
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, ClouderaHBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
HBaseCon 2012 | HBase and HDFS: Past, Present, Future - Todd Lipcon, Cloudera
 
HBase Backups
HBase BackupsHBase Backups
HBase Backups
 
HBaseCon 2013: Apache HBase Replication
HBaseCon 2013: Apache HBase ReplicationHBaseCon 2013: Apache HBase Replication
HBaseCon 2013: Apache HBase Replication
 
Apache HBase, Accelerated: In-Memory Flush and Compaction
Apache HBase, Accelerated: In-Memory Flush and Compaction Apache HBase, Accelerated: In-Memory Flush and Compaction
Apache HBase, Accelerated: In-Memory Flush and Compaction
 
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
HBaseCon 2013: Streaming Data into Apache HBase using Apache Flume: Experienc...
 
HBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBaseHBase and HDFS: Understanding FileSystem Usage in HBase
HBase and HDFS: Understanding FileSystem Usage in HBase
 
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBaseHBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
HBaseCon 2013: Project Valta - A Resource Management Layer over Apache HBase
 
hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践hbaseconasia2017: HBase在Hulu的使用和实践
hbaseconasia2017: HBase在Hulu的使用和实践
 
Time-Series Apache HBase
Time-Series Apache HBaseTime-Series Apache HBase
Time-Series Apache HBase
 
Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path Off-heaping the Apache HBase Read Path
Off-heaping the Apache HBase Read Path
 
HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, PhotobucketHBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
HBaseCon 2012 | Solbase - Kyungseog Oh, Photobucket
 
Real-time HBase: Lessons from the Cloud
Real-time HBase: Lessons from the CloudReal-time HBase: Lessons from the Cloud
Real-time HBase: Lessons from the Cloud
 
HBase Read High Availability Using Timeline Consistent Region Replicas
HBase  Read High Availability Using Timeline Consistent Region ReplicasHBase  Read High Availability Using Timeline Consistent Region Replicas
HBase Read High Availability Using Timeline Consistent Region Replicas
 
Tales from the Cloudera Field
Tales from the Cloudera FieldTales from the Cloudera Field
Tales from the Cloudera Field
 

Viewers also liked

Keynote: Welcome Message/State of Apache HBase
Keynote: Welcome Message/State of Apache HBase Keynote: Welcome Message/State of Apache HBase
Keynote: Welcome Message/State of Apache HBase
HBaseCon
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
HBaseCon
 
Keynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! ScaleKeynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! Scale
HBaseCon
 
Apache HBase at Airbnb
Apache HBase at Airbnb Apache HBase at Airbnb
Apache HBase at Airbnb
HBaseCon
 
Improvements to Apache HBase and Its Applications in Alibaba Search
Improvements to Apache HBase and Its Applications in Alibaba Search Improvements to Apache HBase and Its Applications in Alibaba Search
Improvements to Apache HBase and Its Applications in Alibaba Search
HBaseCon
 
Apache HBase - Just the Basics
Apache HBase - Just the BasicsApache HBase - Just the Basics
Apache HBase - Just the Basics
HBaseCon
 
Apache Ambari Stack Extensibility
Apache Ambari Stack ExtensibilityApache Ambari Stack Extensibility
Apache Ambari Stack Extensibility
Jayush Luniya
 
Web2Asia - Online Social Networks In China
Web2Asia - Online Social Networks In ChinaWeb2Asia - Online Social Networks In China
Web2Asia - Online Social Networks In China
Web2Asia
 
Museo guggenheim bilbao
Museo guggenheim bilbaoMuseo guggenheim bilbao
Museo guggenheim bilbao
carodriguez21
 
Guggenheim Museum, Bilbao, Spain
Guggenheim Museum, Bilbao, SpainGuggenheim Museum, Bilbao, Spain
Guggenheim Museum, Bilbao, Spain
Lee Chain
 
Cross-Site BigTable using HBase
Cross-Site BigTable using HBaseCross-Site BigTable using HBase
Cross-Site BigTable using HBase
HBaseCon
 
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
Cloudera, Inc.
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
Cloudera, Inc.
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on Flash
Cloudera, Inc.
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon
 
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
Cloudera, Inc.
 
HBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBaseHBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBase
Cloudera, Inc.
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
Cloudera, Inc.
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
Cloudera, Inc.
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
Cloudera, Inc.
 

Viewers also liked (20)

Keynote: Welcome Message/State of Apache HBase
Keynote: Welcome Message/State of Apache HBase Keynote: Welcome Message/State of Apache HBase
Keynote: Welcome Message/State of Apache HBase
 
Argus Production Monitoring at Salesforce
Argus Production Monitoring at SalesforceArgus Production Monitoring at Salesforce
Argus Production Monitoring at Salesforce
 
Keynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! ScaleKeynote: Apache HBase at Yahoo! Scale
Keynote: Apache HBase at Yahoo! Scale
 
Apache HBase at Airbnb
Apache HBase at Airbnb Apache HBase at Airbnb
Apache HBase at Airbnb
 
Improvements to Apache HBase and Its Applications in Alibaba Search
Improvements to Apache HBase and Its Applications in Alibaba Search Improvements to Apache HBase and Its Applications in Alibaba Search
Improvements to Apache HBase and Its Applications in Alibaba Search
 
Apache HBase - Just the Basics
Apache HBase - Just the BasicsApache HBase - Just the Basics
Apache HBase - Just the Basics
 
Apache Ambari Stack Extensibility
Apache Ambari Stack ExtensibilityApache Ambari Stack Extensibility
Apache Ambari Stack Extensibility
 
Web2Asia - Online Social Networks In China
Web2Asia - Online Social Networks In ChinaWeb2Asia - Online Social Networks In China
Web2Asia - Online Social Networks In China
 
Museo guggenheim bilbao
Museo guggenheim bilbaoMuseo guggenheim bilbao
Museo guggenheim bilbao
 
Guggenheim Museum, Bilbao, Spain
Guggenheim Museum, Bilbao, SpainGuggenheim Museum, Bilbao, Spain
Guggenheim Museum, Bilbao, Spain
 
Cross-Site BigTable using HBase
Cross-Site BigTable using HBaseCross-Site BigTable using HBase
Cross-Site BigTable using HBase
 
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...HBaseCon 2013:  Evolving a First-Generation Apache HBase Deployment to Second...
HBaseCon 2013: Evolving a First-Generation Apache HBase Deployment to Second...
 
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
HBaseCon 2012 | Leveraging HBase for the World’s Largest Curated Genomic Data...
 
HBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on FlashHBaseCon 2013: Apache HBase on Flash
HBaseCon 2013: Apache HBase on Flash
 
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARNHBaseCon 2015: DeathStar - Easy, Dynamic,  Multi-tenant HBase via YARN
HBaseCon 2015: DeathStar - Easy, Dynamic, Multi-tenant HBase via YARN
 
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
HBaseCon 2012 | Relaxed Transactions for HBase - Francis Liu, Yahoo!
 
HBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBaseHBaseCon 2012 | Building Mobile Infrastructure with HBase
HBaseCon 2012 | Building Mobile Infrastructure with HBase
 
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
HBaseCon 2013: Apache Hadoop and Apache HBase for Real-Time Video Analytics
 
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
HBaseCon 2012 | Content Addressable Storages for Fun and Profit - Berk Demir,...
 
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLCHBaseCon 2012 | HBase for the Worlds Libraries - OCLC
HBaseCon 2012 | HBase for the Worlds Libraries - OCLC
 

Similar to HBaseCon 2015: HBase Operations at Xiaomi

High-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulationsHigh-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulations
Rick Hightower
 
Accordion - VLDB 2014
Accordion - VLDB 2014Accordion - VLDB 2014
Accordion - VLDB 2014
Marco Serafini
 
Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
Membase
 
Open HFT libraries in @Java
Open HFT libraries in @JavaOpen HFT libraries in @Java
Open HFT libraries in @Java
Peter Lawrey
 
Sql saturday azure storage by Anton Vidishchev
Sql saturday azure storage by Anton VidishchevSql saturday azure storage by Anton Vidishchev
Sql saturday azure storage by Anton Vidishchev
Alex Tumanoff
 
Windows Azure Storage: Overview, Internals, and Best Practices
Windows Azure Storage: Overview, Internals, and Best PracticesWindows Azure Storage: Overview, Internals, and Best Practices
Windows Azure Storage: Overview, Internals, and Best Practices
Anton Vidishchev
 
High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017
Rick Hightower
 
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
Yuanyuan Tian
 
2011 06-30-hadoop-summit v5
2011 06-30-hadoop-summit v52011 06-30-hadoop-summit v5
2011 06-30-hadoop-summit v5
Samuel Rash
 
Waters Grid & HPC Course
Waters Grid & HPC CourseWaters Grid & HPC Course
Waters Grid & HPC Course
jimliddle
 
GT.M: A Tried and Tested Open-Source NoSQL Database
GT.M: A Tried and Tested Open-Source NoSQL DatabaseGT.M: A Tried and Tested Open-Source NoSQL Database
GT.M: A Tried and Tested Open-Source NoSQL Database
Rob Tweed
 
Advanced off heap ipc
Advanced off heap ipcAdvanced off heap ipc
Advanced off heap ipc
Peter Lawrey
 
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at HuaweiHBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
Michael Stack
 
Mainframe Cost Reduction
Mainframe Cost ReductionMainframe Cost Reduction
Mainframe Cost Reduction
Software AG UK
 
3 Hyper V
3 Hyper V3 Hyper V
3 Hyper V
johnbakerMS
 
002 hbase clientapi
002 hbase clientapi002 hbase clientapi
002 hbase clientapi
Scott Miao
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Antonios Katsarakis
 
Hadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologiesHadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologies
appaji intelhunt
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
Ververica
 
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
DataWorks Summit
 

Similar to HBaseCon 2015: HBase Operations at Xiaomi (20)

High-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulationsHigh-Speed Reactive Microservices - trials and tribulations
High-Speed Reactive Microservices - trials and tribulations
 
Accordion - VLDB 2014
Accordion - VLDB 2014Accordion - VLDB 2014
Accordion - VLDB 2014
 
Membase Meetup 2010
Membase Meetup 2010Membase Meetup 2010
Membase Meetup 2010
 
Open HFT libraries in @Java
Open HFT libraries in @JavaOpen HFT libraries in @Java
Open HFT libraries in @Java
 
Sql saturday azure storage by Anton Vidishchev
Sql saturday azure storage by Anton VidishchevSql saturday azure storage by Anton Vidishchev
Sql saturday azure storage by Anton Vidishchev
 
Windows Azure Storage: Overview, Internals, and Best Practices
Windows Azure Storage: Overview, Internals, and Best PracticesWindows Azure Storage: Overview, Internals, and Best Practices
Windows Azure Storage: Overview, Internals, and Best Practices
 
High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017High-speed, Reactive Microservices 2017
High-speed, Reactive Microservices 2017
 
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
Building A Hybrid Warehouse: Efficient Joins between Data Stored in HDFS and ...
 
2011 06-30-hadoop-summit v5
2011 06-30-hadoop-summit v52011 06-30-hadoop-summit v5
2011 06-30-hadoop-summit v5
 
Waters Grid & HPC Course
Waters Grid & HPC CourseWaters Grid & HPC Course
Waters Grid & HPC Course
 
GT.M: A Tried and Tested Open-Source NoSQL Database
GT.M: A Tried and Tested Open-Source NoSQL DatabaseGT.M: A Tried and Tested Open-Source NoSQL Database
GT.M: A Tried and Tested Open-Source NoSQL Database
 
Advanced off heap ipc
Advanced off heap ipcAdvanced off heap ipc
Advanced off heap ipc
 
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at HuaweiHBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
HBaseConAsia2018 Track3-4: HBase and OpenTSDB practice at Huawei
 
Mainframe Cost Reduction
Mainframe Cost ReductionMainframe Cost Reduction
Mainframe Cost Reduction
 
3 Hyper V
3 Hyper V3 Hyper V
3 Hyper V
 
002 hbase clientapi
002 hbase clientapi002 hbase clientapi
002 hbase clientapi
 
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity serverDandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
 
Hadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologiesHadoop training in bangalore-kellytechnologies
Hadoop training in bangalore-kellytechnologies
 
Stephan Ewen - Experiences running Flink at Very Large Scale
Stephan Ewen -  Experiences running Flink at Very Large ScaleStephan Ewen -  Experiences running Flink at Very Large Scale
Stephan Ewen - Experiences running Flink at Very Large Scale
 
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
Startup Case Study: Leveraging the Broad Hadoop Ecosystem to Develop World-Fi...
 

More from HBaseCon

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
HBaseCon
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
HBaseCon
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
HBaseCon
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
HBaseCon
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
HBaseCon
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
HBaseCon
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
HBaseCon
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
HBaseCon
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
HBaseCon
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
HBaseCon
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
HBaseCon
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
HBaseCon
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
HBaseCon
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
HBaseCon
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
HBaseCon
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon
 
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon
 

More from HBaseCon (20)

hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kuberneteshbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
hbaseconasia2017: Building online HBase cluster of Zhihu based on Kubernetes
 
hbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beamhbaseconasia2017: HBase on Beam
hbaseconasia2017: HBase on Beam
 
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huaweihbaseconasia2017: HBase Disaster Recovery Solution at Huawei
hbaseconasia2017: HBase Disaster Recovery Solution at Huawei
 
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinteresthbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
hbaseconasia2017: Removable singularity: a story of HBase upgrade in Pinterest
 
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
hbaseconasia2017: HareQL:快速HBase查詢工具的發展過程
 
hbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Neteasehbaseconasia2017: Apache HBase at Netease
hbaseconasia2017: Apache HBase at Netease
 
hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台hbaseconasia2017: 基于HBase的企业级大数据平台
hbaseconasia2017: 基于HBase的企业级大数据平台
 
hbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.comhbaseconasia2017: HBase at JD.com
hbaseconasia2017: HBase at JD.com
 
hbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecturehbaseconasia2017: Large scale data near-line loading method and architecture
hbaseconasia2017: Large scale data near-line loading method and architecture
 
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huaweihbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
hbaseconasia2017: Ecosystems with HBase and CloudTable service at Huawei
 
hbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMihbaseconasia2017: HBase Practice At XiaoMi
hbaseconasia2017: HBase Practice At XiaoMi
 
hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0hbaseconasia2017: hbase-2.0.0
hbaseconasia2017: hbase-2.0.0
 
HBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBaseHBaseCon2017 Democratizing HBase
HBaseCon2017 Democratizing HBase
 
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in PinterestHBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
HBaseCon2017 Removable singularity: a story of HBase upgrade in Pinterest
 
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBaseHBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
HBaseCon2017 Quanta: Quora's hierarchical counting system on HBase
 
HBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBaseHBaseCon2017 Transactions in HBase
HBaseCon2017 Transactions in HBase
 
HBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBaseHBaseCon2017 Highly-Available HBase
HBaseCon2017 Highly-Available HBase
 
HBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at DidiHBaseCon2017 Apache HBase at Didi
HBaseCon2017 Apache HBase at Didi
 
HBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon2017 gohbase: Pure Go HBase Client
HBaseCon2017 gohbase: Pure Go HBase Client
 
HBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environmentHBaseCon2017 Improving HBase availability in a multi tenant environment
HBaseCon2017 Improving HBase availability in a multi tenant environment
 

Recently uploaded

Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
Philip Schwarz
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
AsifKarimJim
 
Introduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of ThingsIntroduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of Things
NachuSubramanian1
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
Task Tracker
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
Nextskill Technologies
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
SOCRadar
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
Jersey (CHE-PING) Su
 
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdfBuilding infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
mohitd6
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
OnePlan Solutions
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
TaskSprint | Employee Efficiency Software
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
mohitd6
 
GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
attueb
 
Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.
shivamt017
 
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
neshakor5152
 
Safe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work PermitsSafe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work Permits
sheqnetworkmarketing
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
jhkdigitalmarketing
 
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Deliverybangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
sunilverma7884
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
kayash1656
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 

Recently uploaded (20)

Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
 
Introduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of ThingsIntroduction to Cloud computing for Internet of Things
Introduction to Cloud computing for Internet of Things
 
Attendance Tracking From Paper To Digital
Attendance Tracking From Paper To DigitalAttendance Tracking From Paper To Digital
Attendance Tracking From Paper To Digital
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
 
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdfBuilding infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
Building infrastructure with code_ A deep dive into CDK for IaC in Java.pdf
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
 
GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
 
Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.
 
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Mysore 000XX00000 Provide Best And Top Girl Service And No1 in City
 
Safe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work PermitsSafe Work Permit Management Software for Hot Work Permits
Safe Work Permit Management Software for Hot Work Permits
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
 
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Deliverybangalore Girls call  👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
bangalore Girls call 👀 XXXXXXXXXXX 👀 Rs.9.5 K Cash Payment With Room Delivery
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
 

HBaseCon 2015: HBase Operations at Xiaomi

  • 1. HBase at Xiaomi Jianwei Cui, Shaohui Liu {cuijianwei, liushaohui}@xiaomi.com
  • 2. About Xiaomi Sold 60M phones in 2014, 3X of 2013 Guinness World Record: selling 2.11M phones online in 24h 2 / 35
  • 3. Our HBase Team 8 Developers Honghua Feng Liang Xie Jianwei Cui Liangliang He YingChao Zhou Qiming Cheng Guanghao Zhang Shaohui Liu 130 patches submmitted in 2014, 82 committed 3 / 35
  • 4. Agenda 1. Current Status 2. Problems and Solutions 3. Themis 4 / 35
  • 5. Clusters and Scenarios Mainland China 20 online clusters / 2 offline clusters in 3 data centers AWS 4 online clusters / 1 offline cluster in 2 regions Online Service Mi Cloud, Mi Push, Galaxy, Mi Message,... Offline Processing User Profile, Trace, Recommendation, ... 5 / 35
  • 6. Scenario A: Mi Cloud Personal cloud storage for smart phones Numbers 90+ milion users, 3X increased in 2014 500 billion rows, 6X increased in 2014 1000+ regions in the largest table See: https://i.mi.com 6 / 35
  • 7. Scenario B: Mi Push Push service on android Data stored in HBase Pub-sub relations of topics and devices Messages to each device Numbers 200+ milion users Push 2 billion+ messages every day 200,000+ requests per second at peak 7 / 35
  • 8. Deployment Two clusters with master-master replication in different data centers Client switches clusters through configs on ZooKeeper Using canary for availability check and alerts 8 / 35
  • 9. Agenda 1. Current Status 2. Problems and Solutions 3. Themis 9 / 35
  • 10. Long Full GC Pauses for RegionServer Problem: Long full GC pauses making ZooKeeper session expire zookeeper.session.timeout = 30s Full GC pause of RegionServer with heap of 30G can be 40s Solution: Multi regionserver instances in a node More memory on offheap using bucket cache 10 / 35
  • 11. Hotspot for Temporal Data Problem: Writes of temporal Data go to a small set of regions Solution: Salted Table Based on SaltedHTable opensourced by Intel Hadoop team See: https://github.com/intel-hadoop/SaltedHTable Transparent to applications by table schema support MapReduce support 11 / 35
  • 12. Coordinated Compaction Problem: Compaction storm Solution: A compaction manager in HMaster coordinates all the compactions in the cluster Before a compaction starts, regionserver needs to acquire a compaction quota 12 / 35
  • 13. Exception Aggregation Purposes: Find the potential bugs in the clusters Solution: Write HMaster/RegionServer log asynchronously to HDFS through Scribe Using MapReduce to aggregate errors and exceptions of clusters 13 / 35
  • 14. Table Based Replication Queue (in progress) Problems: Too much data stored on ZooKeeper Over 200MB replication data for a disabled peer Too many writes to ZooKeeper 5k/s writes to ZooKeeper in a cluster with 100k/s writes (HBASE-12636) 14 / 35
  • 15. Table Based Replication Queue (in progress) Solution: Move replication queue to a system table Row key : server name + peer id + hlog name One column records the offset at which the log is replicated 15 / 35
  • 16. Asynchronous Event Notification (in progress) Purposes: Incremental statistics of data in HBase Table schema transformation Asynchronous data indexing Solution: An asynchronous event notification framework on HBase (HBASE-12884) Replication based implementation: Add a fake replication peer, which can receive the WAL edits from HBase clusters 16 / 35
  • 17. Agenda 1. Current Status 2. Problems and Solutions 3. Themis 17 / 35
  • 18. Cross-Row Transaction Why cross-row transaction? Cross-row data consistency Rows in different regions / tables Example Music index building 18 / 35
  • 19. Cross-Row Transaction Features ACID No central coordinator Integrated without HBase code change Google’s Percolator Large-scale Incremental Processing Using Distribute Transactions and Notifications, by Daniel Peng and Frank Dabek, 2010 Themis https://github.com/Xiaomi/themis Provides cross-row transactions on HBase based on Percolator 19 / 35
  • 20. Themis Infrastructure Timestamp server Use timestamp of KeyValue internally Timestamp must be globally incremental Client Coprocessor 20 / 35
  • 21. Timestamp Server Seperate a long type timestamp into two parts : Higher 46 bits: Sync with system time Lower 18 bits: Incremental counter Hundreds of thousands unique timestamps in each millisecond Incremental in one timestamp server 21 / 35
  • 22. Timestamp Server Incremental cross timestamp servers Periodically save a future timestamp into ZooKeeper Allocated timestamp must be smaller than saved timestamp Another server needs to read the saved timestamp when starting High availability High throughput : 600,000 RPCs per second Batch concurrent requests in one RPC 22 / 35
  • 23. Themis Cross-row mutation example : Cash Table Rows for Bob and Joe are in different regions Transfer $3 from Joe to Bob atomically Two auxiliary columns : lock column and commit column Two-phase commit Prewrite Phase Commit Phase checkAndMutate of HBase : guarantee the atomicity for a single row 23 / 35
  • 24. Prewrite Phase Fetch a prewrite timestamp from timestamp server (prewriteTs=99) Select primary and secondary columns Primary Column : (Joe, f:c) PrimaryLock: {secondaries : [(Bob, f:c)]} Secondaries Column : (Bob, f:c) SecondaryLock: {primary : (Joe, f:c)} 24 / 35
  • 25. Prewrite Phase Prewrite primary column Write primary lock and data if no lock exists in lock column checkAndMutate of HBase to guarantee the atomicity Prevent other clients mutating the same column concurrently 25 / 35
  • 26. Prewrite Phase Prewrite secondary columns Follow the same steps of prewriting primary column 26 / 35
  • 27. Commit Phase Fetch commit timestamp from timestamp server (commitTs=100) Commit Primary Delete the lock and write commit column if the lock exists checkAndMutate of HBase to guarantee the atmocity Decide the success or failure the whole transaction 27 / 35
  • 28. Commit Phase Commit secondaries Delete the lock and write commit column atomically 28 / 35
  • 29. Themis Read Fetch a read timestamp (readTs=101) Read commit columns with commitTs < readTs (Joe, c:f#c) => (100 : 99) (Bob, c:f#c) => (100 : 99) Read data column with prewriteTs prewriteTs is just the value of commit column (Joe, 99: $17) and (Bob, 99: $12) 29 / 35
  • 30. Performance Comparison Single-Row Transaction The worst case compared with raw HBase One region server with 10GB heap memory Write Performance : Preload 3 million rows, 256MB LRU cache Read Performance : Preload 30GB data 30 / 35
  • 31. Performance Comparison Single-Row Transaction The worst case compared with raw HBase One region server with 10GB heap memory Write Performance Read Performance 31 / 35
  • 32. Performance Comparison Themis vs. Percolator Write optimization Set lock family IN MEMORY Not sync the lock of prewrite phase 32 / 35
  • 33. Future Work of Themis Generic transaction API: HBASE-11447 Support different isolation levels Global secondary index 33 / 35