Distcp-ng is a tool for replicating massive datasets between Hadoop compatible file systems using Gobblin. It provides features like continuous replication of datasets, efficient file listing to reduce filesystem calls, dataset awareness for prioritization and notifications, failure isolation, and operational metrics. The architecture allows for copying between different source and target systems like HDFS, SFTP, and Hive. It also supports transformations, atomic publishing, and Hive table registration during copies. Future work includes building Distcp as a continuous replication service.

1. Distcp-ng:
Replicating
massive datasets
with Gobblin
Issac Buenrostro
Gobblin Meetup, Jun 2016

2. Outline
1 Motivation
2 Architecture
3 Features
4 Hive Copy
5 Future

3. Motivation

4. Distributed Copy
Copy files between Hadoop compatible file systems.
What is Distcp?

5. Motivation
1.Continuous replication of datasets.
2.Efficient file listing.
• Reduce file system rpc calls
• Alternate listing services first-class citizens.
3.Dataset awareness: prioritization, notification, etc.
4.Failure isolation.
5.Operational metrics, notifications, data availability
triggers.
6.Portability.

6. Architecture

7. Distcp Gobblin Architecture

8. Copyable Dataset- Basic

9. Copy Entities - Advanced
Pre / Post publish steps
Copy Entities – Run before or after publish (NOOP in Task)

10. Features

11. Recursive Copy
Most similar to distcp
Copy all files under an input path
Accepts path filter

12. Features
Source Converter Target
Hadoop File System
SFTP
Apache server filer
Hive
…
Byte-level stream
transformations:
• Encrypt / Decrypt
• (Un) Gzip
• Untar
Atomic publishing
Data availability
notification
Hive Registration
File deletion / sync

13. File Sets
Distcp atomic unit, single dataset can be split into
multiple file sets
1. All-or-nothing publish*
2. Isolation: failed file set does not affect other file sets
3. Event emitted on publish per file and file set
* best-effort. Future: use write-ahead log for better guarantee.

14. Smart file limits
Limit the number of files copied in a single run
1. File sets are never split
2. Soft limit: stop processing new file sets, currently
running file sets can finish
3. Hard limit: do not accept any more files
4. Prioritize file sets (Future)

15. Unpublished File Persistence
1. Files that were copied successfully but not published
are persisted in private directory. (File set failure,
permission failure, etc.)
2. Future run identifies persisted file, reuse instead of
re-copying.
3. Time-based automatic retention on persist directory.

16. Hive Copy

17. Hive Copy
Copy Hive tables between Hive metastores
1. Determine files under each table / partition
2. Diff files in source / target
3. Copy necessary files
4. Register tables / partitions on target
5. Deregister partitions missing in source
6. (Optional) Delete files for deregistered partitions

18. Hive Copy Configuration
job.name=distcpNgExample
# Source and target metastores
hive.dataset.hive.metastore.uri=thrift://mysource.hive:9000
hive.dataset.copy.target.metastore.uri=thrift://mytarget:9000
gobblin.copy.preserved.attributes=rgbp # Preserve attributes
# Database and tables copy
hive.dataset.whitelist=events.loginEvent|logoutEvent,metrics
hive.dataset.copy.locations.listing.skipHiddenPaths=true # Skip hidden paths
# Use registration time to determine whether a partition should be skipped
hive.dataset.copy.fast.partition.skip.predicate=gobblin.data.management.copy.predicates.
RegistrationTimeSkipPredicate
# Partition filter
hive.dataset.copy.partition.filter.generator=gobblin.data.management.copy.hive.filter.Lo
okbackPartitionFilterGenerator
hive.dataset.partition.filter.datetime.column=datepartition
hive.dataset.partition.filter.datetime.lookback=P7D
hive.dataset.partition.filter.datetime.format=YYYY-MM-dd-HH

19. Hive Copy
Candidate
Files
Existing files at
expected target
location.
• Different location
• Schema incompatible
• …

20. Hive Copy - Numbers
100+ tables
3000+ partitions
20,000+ new files per hour
2TB+ new data per hour
File listing 30k files: < 30s
Copy 30k files, 5TB: ~20 min

21. Current bottlenecks
Work unit serialization
• ~100 work units / second
Bad nodes in Hadoop cluster
• Need speculation
Serial publishing of file sets
• Solution in progress

22. Gobblin Distcp vs ReAir
Reair: Hive warehouse data replication (Airbnb)
Offers batch and incremental replication
Gobblin Distcp ReAir
File listing and modification
times for incremental
changes
MySQL and audit log hook
store for incremental
changes
Portable Gobblin job (MR,
thread based, Helix)
MR job
Same framework can copy
non-Hive data
Monitoring / Web UI (in
progress for Gobblin)

23. Future

24. Distcp continuous service

25. Next Steps
1 Simple CLI launcher
2 Dataset / file set prioritization
3 Global network throttling
4 Large file splitting
5 Least-congested path optimization

26. Find out more:
©2015 LinkedIn Corporation. All Rights
Reserved.
Gobblin Distcp