This document discusses disaster recovery and cloud migration strategies for Apache Hive data warehouses. It covers replication modes like master-slave and master-master. It also discusses fail over, fail back, event-based replication using REPL commands, change management, bootstrapping, and demonstrations. Challenges of cloud migration like data movement and integrity are also addressed. The future of Hive replication including ACID tables, faster bootstrapping, and table-level replication is outlined.

1. 1 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Disaster Recovery and Cloud
Migration for you Apache
Hive Warehouse
DataWorks Summit – Sydney
September 2017

2. 2 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Presenter
Sankar Hariappan
Senior Software Engineer, Hortonworks

3. 3 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
 About Apache Hive
 Disaster Recovery
 Replication Modes
 Fail Over
 Fail Back
 Replication at Hive-Scale
 Event Based Replication
 Change Management
 Bootstrapping
 REPL Commands
 Demonstration
 Cloud Migration Challenges
 Future Work
Agenda

4. 4 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
About Apache Hive
 Data warehouse tool built on top of Apache Hadoop.
 Handle data warehousing tasks such as extract/transform/load (ETL), reporting, and
data analysis.
 Manage large datasets residing in distributed storage.
 SQL with Hive specific extensions.
 Query optimization powered by Apache Calcite and execution via Apache Tez, Apache
Spark, or MapReduce.
 Access to files stored either directly in Apache HDFS or in other data storage systems
such as Apache HBase.
 Sub-second query retrieval via Hive LLAP, Apache YARN and Apache Slider.

5. 5 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Apache Hive Architecture
Hive Thrift
Server
JDBC/ODBC
Driver
Compiler Optimizer Executor
HiveServer2
Hive
Metastore
HDFS
YARN
MS
Client RDBMS

6. 6 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Disaster Recovery
 Deployment of clusters in more than one data center for business continuity or geo
localization.
 Hybrid cloud deployment for off-premise processing.
 Robust replication solution to achieve seamless disaster recovery.
– Prevent severe data loss.
– Eliminate single point of failure.
– Fault-tolerant.

7. 7 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Replication Modes
 Master-Slave
Master Slave
Unidirectional
Read ReadWrite
 Master-Master
Master
Bidirectional
Read Write
Master
Read Write
Master
Read Write
Master
Read Write
Slave
Read

8. 8 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Replication Modes
 Hub and Spoke pattern
Master
Slave
Read
Read
Write
Slave
Read
Slave
Read
Slave
Read
 Relay pattern
Master Slave
Read ReadWrite
Slave
Read

9. 9 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Fail Over
 Slave take over the Master
responsibilities instantaneously.
 Ensure business continuity with minimal
data loss based on Recovery Point
Objective (RPO).
 Almost zero down-time or meet
Recovery Time Objective (RTO).
Master Slave
Unidirectional
Read Write
Fail over
Read Write

10. 10 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Fail Back
 Slave cluster usually have minimal
processing capabilities which makes Fail
Back an important requirement.
 Original Master comes alive with latest
data.
 Ensure removal of stale data which was
not replicated to the Slave.
 Reverse replicate the delta of data
loaded into the Slave after Fail Over.
Master Slave
Unidirectional
Read ReadWrite
Fail back

11. 11 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Replication at Hive-Scale
 Event based replication.
 First version of Hive Replication (Replv1) uses EXPORT-IMPORT semantics to replicate
data.
– Inefficient mechanism.
– 4X copy problem.
– Rubber-banding issue.
– Depends on external tools such as Falcon/Oozie to manage replication state.
 Second version of Hive Replication (Replv2) uses REPL commands (HIVE-14841)
– Point-in time replication.
– Reduce number of copies.
– Hive maintains the replicated state.
– Additional support for functions, constraint replication.
– Available in Apache Hive trunk.

12. 12 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Event Logging
HiveServer2
Hive
Metastore
Metastore
RDBMS
Events Table
JDBC/ODBC
Runs Query Manage Metadata

13. 13 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Event Logging
 Capture event: Create/Alter/Drop on DB/Table/Partition/Function/Constraint objects.
 Stored in Metastore RDBMS.
 Event is self-contained to recover the state of the object (metadata + data).
 Events are serialized using sequence number (event id).

14. 14 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Event Based Replication
Metastore
RDBMS
Events Table
HDFS
Serialize new events
batch
Master Cluster
Slave Cluster
HiveServer2
Dump
(metadata + data)
HDFS
Meatastore
RDBMS
HiveServer2
DistcpMetastore API to
write objects
Data files
copy
Read repl
dump dir
REPL DUMP
REPL LOAD

15. 15 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Event Based Replication
 Read batch of events from the Metastore RDBMS in the generated sequence.
 "repl dump <db name> from <event id> "
– get events newer than <event id>.
– includes data files information.
– "<event id>" is last replicated event id for DB from the destination cluster
 "repl load <db name> from <hdfs URI>"
– apply the events on destination
 State replicated in batches currently, can be optimized in future

16. 16 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Change Management
 Replicating the following batch –
– Insert to table
– Drop table
 Need inserted files after drop for replication
 Trash like directory for capturing deleted files (CM directory)
 Use checksum to verify file, else lookup from CM directory using checksum
 Necessary for ordered replication - State in destination DB would correspond to state in
source X duration back.

17. 17 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Bootstrapping
 What about data generated before event capturing was enabled?
 Bootstrapping - Uses same repl dump/load commands, but is not event based
 Incremental replication catches up with events during bootstrap to make change
consistent with state of source at time X in past.
 Optimized for large database.
 Parallel dump of large number of partitions.

18. 18 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
REPL Commands
 REPL DUMP <db-name> [FROM <start-evid> [TO <end-evid>] [LIMIT <num-evids>] ];
– Execute this command in source cluster.
– Returns dump directory and last replication state for the dump.
– REPL DUMP <db-name>; bootstrap the whole database.
– REPL DUMP <db-name> FROM <start-evid>; to replicate all events after start-evid.
– REPL DUMP <db-name> FROM <start-evid> TO <end-evid>; to replicate a range of events.
– REPL DUMP <db-name> FROM <start-evid> LIMIT <num-evids>; to replicate a limited set of events.
 REPL LOAD <db-name> FROM <dump-dir>;
– dump-dir is the HDFS URI returned by REPL DUMP command.
– Execute this command in target cluster.
 REPL STATUS <db-name>;
– Execute this command in target cluster.
– Gets the last replicated state of the database in destination which should be the input for REPL
DUMP as start-evid.

19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demonstration

20. 20 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Cloud Migration Challenges
 Move is expensive
– Cloud file systems has implemented “move” as “copy”.
– Atomic move/rename of data files from temp directory to warehouse location in target.
– Move/rename of data files to CM directory.
– ACID and micro-managed tables can get rid of CM archival of data files.
 Distcp to copy data files to/from cloud
– Run distcp from on-prem cluster for hybrid deployment.
– Optimize distcp to use vendor specific tool to copy between cloud file systems.
 Data integrity when copy data to/from cloud
– Verification of data files copied across clusters using HDFS provided file checksum.
– Checksum is not consistent across all file systems.

21. 21 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Current Replication Scope
 Supported
– Managed tables with partitions.
– Views.
– UDFs/UDAFs.
– Constraints.
– Ranger authorization policies.
 Limitations
– External tables.
– Manually copied data files to data location.
– ​SQL Standard-based Authorization

22. 22 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Future Work
 Replicate ACID/Micro-managed tables.
 Replication to/from cloud storage such as S3 or WASB etc.
 Hot Data Replication.
 Faster Bootstrapping.
 Optimize Fail Back.
 Replicate Column Statistics, Index etc.
 Table level replication.

23. 23 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
References: Hive Configurations for Replication
Hive Configuration Recommendation Description
hive.metastore.transactional.event.listeners org.apache.hive.hcatalog.listener.
DbNotificationListener
Enable event logging
hive.metastore.event.db.listener.timetolive 86400s/RPO Expiry time for the events logged in metastore
hive.repl.rootdir Any valid HDFS directory Root directory used by repl dump
hive.metastore.dml.events true Enable event generation for DML operations
hive.repl.cm.enabled true Enable change management to archive deleted data
files
hive.repl.cm.retain 24hr/RPO Expiry time for CM backed-up data files.
hive.repl.cm.interval 3600s Time interval to look-up on expired data files in CM
hive.repl.cmrootdir Any valid HDFS directory Root directory for Change Manager
hive.repl.replica.functions.root.dir Any valid HDFS Root directory to store UDFs/UDAFs jars. Config needed
in Target cluster.
hive.repl.approx.max.load.tasks 1000 / Depending on memory
capacity of target warehouse
Limit the number of execution tasks to control the
memory consumption. Config needed in Target cluster.
hive.repl.partitions.dump.parallelism 8/depends on cpu usage Number of threads to concurrently dump partitions

24. 24 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
References: Apache Hive Documentation
https://cwiki.apache.org/confluence/display/Hive/Home
https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development
https://cwiki.apache.org/confluence/display/Hive/HiveReplicationDevelopment
https://cwiki.apache.org/confluence/display/Hive/Replication
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+ImportExport
https://issues.apache.org/jira/browse/HIVE-14841

25. 25 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
Learn More
Apache Hive, Apache
HBase and Apache
Phoenix
Bird of a Feather
Thursday, September 21 @ 6:00p
C 2.3
https://dataworkssummit.com/sydney-2017/birds-of-a-
feather/apache-hive-apache-hbase-apache-phoenix/

26. 26 © Hortonworks Inc. Confidential 2011 – 2017. All Rights Reserved
THANK YOU!