Batch layer• In practice, too expensive to fully recompute each view to get updates• A production batch workﬂow adds minimum amount of incrementalization necessary for performance
Incremental batch layer Batch View 1New data Batch View Batch View 2 maintenance workﬂow Query Append Batch View 3 All data
Batch layerRobust and fault-tolerant to both machineand human error.Low latency reads.Low latency updates.Scalable to increases in data or trafﬁc.Extensible to support new features or relatedservices.Generalizes to diverse types of data and requests.Allows ad hoc queries.Minimal maintenance.Debuggable: can trace how any value in thesystem came to be.
Speed layerCompensate for high latency of updates to batch layer
Speed layerKey point: Only needs to compensate for data not yet absorbed in serving layer
Speed layerKey point: Only needs to compensate for data not yet absorbed in serving layer Hours of data instead of years of data