SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
8.
8@Twitter
Replication State
Persistently stored in Zookeeper
Status
Master kill switch
Peers
List of remote target clusters
Queues
List of remaining HLogs to replicate and current
position in each log
11.
11@Twitter
End-point for shipping WAL entries
One instance for each queue
Runs as a separate thread on region server
Uses AdminProtocol RPC to synchronously
ship entries
Filters edits based on replication scope
Replication Source
12.
12@Twitter
Replication Sink
End-point for receiving shipped WAL entries
One instance per region server
Synchronously receives entries and applies
them using HTable
Batches rows in the same table
13.
13@Twitter
Load balancing
Balances load on remote cluster using
randomization
Ships edits to random subset of remote region
servers
Default is 10%
15.
15@Twitter
Replication Source Manager
Manages all replication sources
Manages change in replication state
Log rolling
Region server failure
Addition/deletion of peer clusters
19.
19@Twitter
Replication State
Persistently stored in Zookeeper
Three major replication znodes: Status, Peers, Queues
20.
20@Twitter
Status znode
Master kill switch
Controlled by start_replication, stop_replication
Be careful what you wish for
21.
21@Twitter
Peers znode
A set of remote clusters registered as possible
replication targets
Identified by peer id
Contains status of each peer cluster
22.
22@Twitter
Queues znode
Queues identified by region server and peer id
Queues contain list of HLogs and current position
in log