The document provides guidance on practices that can lead to failure in big data systems. It warns against assumptions that schemas are unnecessary, that databases can scale reads and writes infinitely, and that network connections and hardware will always be available. Instead, it recommends defining schemas and metadata, understanding database models, preparing for failures through testing, and managing resources and data pipelines. Proper data governance, partitioning, replication, and understanding of consistency and transaction models can help avoid failures.

1. Methods and Practices for
guaranteed failure in Big Data
PANTELIS NASIKAS
1

2. Delusions and Pitfalls
● Who needs Schemas afterall
● My SuperNoYesSQLDB scales on reads AND writes
like..forever!
● The network and all peers will always be there!
● Resource management , not really a concern for my script
● ETL/Jobs management , BYO-DAG tool

3. Who needs schemas afterall
● Always Define schemas and versioning process irrespective of the serialization
● Schema registries indispensable
● Metadata management , respect the end users and engineers
● Means to implement data governance
● Builds trust across teams

4. My SuperNoYesSQLDB scales on reads AND writes
● Understand model that the Database was built for
● Not all APIs created equally
● Partitioning and Replication as key design elements keeping in mind failures -
extensions - rebalancing
● Data consistency on high speed writes...spooky, fsync ( when really? )
● Transactions to keep data and denormalized views in sync ? alternative options

5. The network and all peers will always be here!
● The Network IS reliable , P.Bailis & K. Kingsbury
● Hardware IS NOT (always) reliable
● Prepare for failure
● Test systems under failing hardware/software
● Learn your APIs (eg what happens when partitions fail , or move ? )
● where is your place in CAP ? Are you CP or CA ?
● what is acceptable for your case ?
● How does my database recover after failure ?
● Can this introduce new problems ?

6. ETL/Jobs management , BYO-DAG tool
● Data Pipelines and Lineage
● Who generates what , at what time
● Failure! Who is next? What does rescheduling mean
to dependents ?
● How can i really find dependents ?
● Let me build that API! It’s just software after all.
● Build testable pipelines w/o any need for production
data...

7. Resource management , not really a concern
● Every job / task / stream / long running service should be constrained and ideally
isolated
● Database / Filesystem access from 3rd party as well
● Understand your jobs’ requirements ( communication cost model, partitioning and
shuffling effect on cpu/memory/network)
● You don’t want to preempt you multi-terabyte batch job just before the end
● Orchestrate small well defined tasks
● Do not assume large resource allocations

8. 8
Pantelis Nasikas
pantelis.nasikas@agileactors.com
Thank you,