All modern Big Data solutions, like Hadoop, Kafka or the rest of the ecosystem tools, are designed as distributed processes and as such include some sort of redundancy for High Availability. https://www.bigdataspain.org/2017/talk/disaster-recovery-for-big-data Big Data Spain 2017 November 16th - 17th Kinépolis Madrid

2. Disaster Recovery for Big Data

3. About us

We are nerds!

Started working in Big Data for international companies

Founded a start-up a few years ago:
 With colleagues working in related technical areas
 And who also knew business stuff!

We’ve been participating in different Big Data projects

4. Introduction
“I already have HDFS replication and High
Availability in my services, why would I
need Disaster Recovery (or backup)?”

5. Concepts

High Availability (HA)
 Protects from failing
components: disks,
servers, network
 Is generally a “systems”
issue
 Redundant, doubles
components
 Generally has strict
network requirements
 Fully automated,
immediate

6. Concepts

Backup
 Allows you to go back to
a previous state in time:
daily, monthly, etc.
 It is a “data” issue
 Protects from accidental
deletion or modification
 Also used to check for
unwanted modifications
 Takes some time to
restore

7. Concepts

Disaster Recovery
 Allows you to work
elsewhere
 It is a “business” issue
 Covers you from: main site
failures such as electric
power or network outages,
fires, floods or building
damage
 Similar to having insurance
 Medium time to be back
online

8. The ideal Disaster Recovery

High Availability for
datacenters

Exact duplicate of the
main site
 Seamless operation (no
changes required)
 Same performance
 Same data

This is often very
expensive and sometimes
downright impossible

9. DR considerations

So, can we build a cheap(ish) DR?

We must evaluate some tradeoffs:
 What’s the cost of the service not being
available? (Murphy’s Law: accidents will happen
when you are busiest)
 Is all information equally important? Can we lose
a small amount of data?
 Can we wait until we recover certain data from
backup?
 Can I find other uses for the DR site?

10. DR considerations

Near or far?
 Availability
 Latency
 Legal considerations

11. DR considerations

Synchronous vs
Asynchronous
 Synchronous replication
requires a FAST connection
 Synchronous works at
transaction level and is
necessary for operational
systems
 Asynchronous replication
converges over time
 Asynchronous is not
affected by delays nor does
it create them

12. Big Data DR

Can’t generally be copied
synchronously

No VM replication

Other DR rules apply:
 Since it impacts users,
someone is in charge of the
“starting gun”
 DNS and network changes
to point clients

Main types:
 Storage replication
 Dual ingestion

13. Storage replication

Similar to non-Big Data solutions, where central
storage is replicated

Generally implemented using distcp and HDFS
snapshots

Data is ingested in source cluster and then copied

14. Storage replication

Administrative
overhead:
 Copy jobs must be
scheduled
 Metadata changes
must be tracked

Good enough for
data that comes
traditional ETLs such
as daily batches

15. Dual Ingestion

No files, just streams

Generally ingested from multiple outside
sources through Kafka

Streams must be directed to both sites

16. Dual Ingestion

Adds complexity to apps
 Nifi can be set up as a front-end to both
endpoints

Data consistency must be checked
 Can be automatically set up via monitoring
 Consolidation processes (such as a monthly
re-sync) might be needed

17. Others

Ingestion replication
 Variant of the dual ingestion
 A consumer is set up in the source Kafka that in turn
writes to a destination Kafka
 Bottleneck if the initial streams were generated by
many producers

Mixed:
 Previous solutions are not mutually exclusive
 Storage replication for batch processes’ results
 Dual ingestion for streams

18. Commercial offerings

Solutions that ease DR setup

Cloudera BDR
 Coordinates HDFS snapshots and copy

WANdisco Fusion
 Continuous storage replication

Confluent Multi-site
 Allows multi-site Kafka data replication

19. Tips

Big Data clusters
have many nodes
 Costly to replicate
 Performance /
Capacity tradeoff
 We can use
cheaper servers in
DR, since we don’t
expect to use them
often

20. Tips

Document and test procedures
 DR is rarely fully automated, so responsibilities and
actions should be clearly defined
 Plan for (at least) a yearly DR run
 Track changes in software and configuration

21. Tips

Once you have a DR
solution, other uses will
surface

DR site can be used for
backup
 Maintain HDFS
snapshots

DR data can be used
for testing / reporting
 Warning: it may alter
stored data

22. Conclusions

Balance HA / Backup / DR as needed, they
are not exclusive:
 Different costs
 Different impact

Big Data DR is different:
 Dedicated hardware
 No VMs, no storage cabin

Plan for DATA CENTRIC solutions

23. Questions