Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1© Cloudera, Inc. All rights reserved.
Michael Crutcher
Director, Product Management - Storage
Lambda Architecture
2© Cloudera, Inc. All rights reserved.
Agenda
• Big Data Challenges
• What is Lambda?
• Lambda Advantages and Disadvantage...
3© Cloudera, Inc. All rights reserved.
Big Data Challenges
4© Cloudera, Inc. All rights reserved.
“Something interesting is happening”
The world’s largest
taxi company owns
ZERO veh...
5© Cloudera, Inc. All rights reserved.
Data is now a strategic asset
Instrumentation
Consumerization
Experimentation
Today...
6© Cloudera, Inc. All rights reserved.
“It will soon be technically
feasible & affordable to
record & store everything…”
—...
7© Cloudera, Inc. All rights reserved.
The new analytics paradigm
Understand
why it
happened
Change
what
happens
next
Dete...
8© Cloudera, Inc. All rights reserved.
So Why Big Data?
What does the reporting look
like at your business today?
What if ...
9© Cloudera, Inc. All rights reserved.
What is Lambda architecture?
10© Cloudera, Inc. All rights reserved.
What is Lambda Architecture?
Batch Layer
Serving Layer
Speed Layer
New Data
Data L...
11© Cloudera, Inc. All rights reserved.
Batch Layer
• Manages the master data set, an immutable, append-only set of raw da...
12© Cloudera, Inc. All rights reserved.
Speed Layer
• This layer ingests streaming data or micro-batches
• Spark and Storm...
13© Cloudera, Inc. All rights reserved.
Serving Layer
• As you might guess from the name, this is the layer that serves da...
14© Cloudera, Inc. All rights reserved.
What is a Kappa Architecture?
Batch Layer
Serving Layer
Speed Layer
New Data
Data ...
15© Cloudera, Inc. All rights reserved.
Everything Has a New Name
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake...
16© Cloudera, Inc. All rights reserved.
The Log as Storage
• Lambda and Kappa architectures are both predicated on immutab...
17© Cloudera, Inc. All rights reserved.
Is Raw Data the Right Logical Model?
• It’s possible to derive many higher level l...
18© Cloudera, Inc. All rights reserved.
There are Only Two Hard CS Problems
1) Cache invalidation
2) Naming things
-- Phil...
19© Cloudera, Inc. All rights reserved.
Data Engineering has one hard problem
• When should I denormalize to maximize perf...
20© Cloudera, Inc. All rights reserved.
Lambda Advantages and
Disadvantages
21© Cloudera, Inc. All rights reserved.
Lambda Advantages
• Marries diverse strengths of existing open source software int...
22© Cloudera, Inc. All rights reserved.
Lambda Disadvantages
• Complexity
• Many moving parts
• Restatement is difficult
•...
23© Cloudera, Inc. All rights reserved.
Lambda Complexity
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
...
24© Cloudera, Inc. All rights reserved.
Lambda Complexity
Batch Layer
Serving Layer
Speed Layer
New Data
Data Lake
(HDFS)
...
25© Cloudera, Inc. All rights reserved.
The Log as Storage
• The idea of representing data as immutable log information is...
26© Cloudera, Inc. All rights reserved.
Premature Optimization
Programmers waste enormous amounts of time thinking about, ...
27© Cloudera, Inc. All rights reserved.
Gap Filling vs. Optimization
• Some Lambda implementations are deployed on big dat...
28© Cloudera, Inc. All rights reserved.
Gap Filling v2: Lack of Mutability
• Some Lambda implementations aim to fill the g...
29© Cloudera, Inc. All rights reserved.
Kudu as a Lambda alternative
30© Cloudera, Inc. All rights reserved.
HDFS
Fast Scans, Analytics
and Processing of
Stored Data
Fast On-Line
Updates &
Da...
31© Cloudera, Inc. All rights reserved.
Kudu Increases the Value of Time Series Data
Time Series
Inserts, updates, scans, ...
32© Cloudera, Inc. All rights reserved.
Kudu can help spot problems before they
happen. Real-time data inserts with the ab...
33© Cloudera, Inc. All rights reserved.
More Versatility in Online Reporting
Online
Reporting
Inserts, updates, scans, loo...
34© Cloudera, Inc. All rights reserved.
Xiaomi use case
• World’s 4th largest smart-phone maker (most popular in China)
• ...
35© Cloudera, Inc. All rights reserved.
Xiaomi big data analytics pipeline
Large ETL pipeline delays
● High data visibilit...
36© Cloudera, Inc. All rights reserved.
Xiaomi big data analytics pipeline
Simplified with Kudu
Low latency ETL pipeline
●...
37© Cloudera, Inc. All rights reserved.
Conclusions
• Lambda has a real place in big data architectures
• Optimize as need...
Upcoming SlideShare
Loading in …5
×

Moving Beyond Lambda Architectures with Apache Kudu

3,491 views

Published on

-Kudu is a new storage layer for the Hadoop ecosystem that enables fast analytics on fast data; it splits the difference between the fast read/write of HBase and the fast scans of HDFS...while compromising minimally on performance. It can pair with Spark, Impala, or MapReduce.
-In the past, a lambda architecture was needed to run analytics on real-time data – that is, a complex architecture that created separate a “speed layer” for rapid availability/query/updates, and a “batch layer” for running analytics scans. This was complicated and took lots of tuning.
-With Kudu, the Apache ecosystem now has a simplified storage solution for analytic scans on rapidly updating data, eliminating the need for the aforementioned hybrid lambda architectures.

Published in: Software
  • Be the first to comment

Moving Beyond Lambda Architectures with Apache Kudu

  1. 1. 1© Cloudera, Inc. All rights reserved. Michael Crutcher Director, Product Management - Storage Lambda Architecture
  2. 2. 2© Cloudera, Inc. All rights reserved. Agenda • Big Data Challenges • What is Lambda? • Lambda Advantages and Disadvantages • Kudu as a Lambda alternative
  3. 3. 3© Cloudera, Inc. All rights reserved. Big Data Challenges
  4. 4. 4© Cloudera, Inc. All rights reserved. “Something interesting is happening” The world’s largest taxi company owns ZERO vehicles. The world’s largest accommodation provider owns ZERO real estate. The world’s most popular media owner creates ZERO content. The world’s leading music platform owns no music.
  5. 5. 5© Cloudera, Inc. All rights reserved. Data is now a strategic asset Instrumentation Consumerization Experimentation Today, everything that can be measured will be measured. Today, data IS the application. Today, becoming data-driven is a business imperative.
  6. 6. 6© Cloudera, Inc. All rights reserved. “It will soon be technically feasible & affordable to record & store everything…” — New York Times “Digital technologies will, in the near future, accomplish many tasks once considered uniquely human.” . — Second Machine Age Data is abundant, diverse & shared freely As is how we store, process and analyze it Streaming Machine Learning BI ETL Modeling
  7. 7. 7© Cloudera, Inc. All rights reserved. The new analytics paradigm Understand why it happened Change what happens next Determine what happened Make it happen consistently
  8. 8. 8© Cloudera, Inc. All rights reserved. So Why Big Data? What does the reporting look like at your business today? What if it could happen in half the time, or half that time? What data are you looking at? What data do you want to know about your customers? How can you best use external data? Too often data is archived, combined, or simplified to save space and strain on systems. Once data is combined we loose the ability to dig deeper. Better Business Forecasting Better Views of CustomersFull Fidelity Data Access
  9. 9. 9© Cloudera, Inc. All rights reserved. What is Lambda architecture?
  10. 10. 10© Cloudera, Inc. All rights reserved. What is Lambda Architecture? Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala
  11. 11. 11© Cloudera, Inc. All rights reserved. Batch Layer • Manages the master data set, an immutable, append-only set of raw data • Pre-computes views of the data • “Traditionally” this has been in HDFS and processed with Map/Reduce • There has already been some shift to cloud based object storage and processing in other frameworks like Spark
  12. 12. 12© Cloudera, Inc. All rights reserved. Speed Layer • This layer ingests streaming data or micro-batches • Spark and Storm are traditionally used • In some cases micro-batches are directly ingested into NoSQL data stores like HBase • This data is periodically expunged • In many “Lambda-like” architectures I’ve seen, this layer is used to provide an “active partition” that provides a limited window of mutability
  13. 13. 13© Cloudera, Inc. All rights reserved. Serving Layer • As you might guess from the name, this is the layer that serves data • It would be unusual for raw data to be served directly • This could be an application written directly against a data store like HBase • It could be a SQL engine on top of a file system, Impala + Parquet is an example
  14. 14. 14© Cloudera, Inc. All rights reserved. What is a Kappa Architecture? Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala
  15. 15. 15© Cloudera, Inc. All rights reserved. Everything Has a New Name Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge System of Record (OLTP) Operational Data Store Derived Tables (EDW) In-Memory Database Star/Snowflake, Cubes, or In-Memory Tables
  16. 16. 16© Cloudera, Inc. All rights reserved. The Log as Storage • Lambda and Kappa architectures are both predicated on immutable source data • Data can be modeled as a series of events recorded at specific points in time about entities • Updates are modeled as new events and the current or historic value associated with an entity can be reconstructed through the collected events • Kappa calls this ordered set of events “a log”, it’s safe to say they didn’t invent this term A B C D B E F A G B 1 10 Ordered over Time
  17. 17. 17© Cloudera, Inc. All rights reserved. Is Raw Data the Right Logical Model? • It’s possible to derive many higher level logical abstractions from raw data • As an example, I could construct a customer account balance from raw account activity data • This doesn’t mean it’s a good idea A B C B A C B A A C t0 t12Account Activity +$10 +$20 +$15 -$10 +$35 -$5 +$25 +$15 -$20 +$10 Easy: What was the last account event for Customer C? Harder: What is the account balance for Customer A at t12?
  18. 18. 18© Cloudera, Inc. All rights reserved. There are Only Two Hard CS Problems 1) Cache invalidation 2) Naming things -- Phil Karlton
  19. 19. 19© Cloudera, Inc. All rights reserved. Data Engineering has one hard problem • When should I denormalize to maximize performance? • When should I normalize to minimize maintenance problems? Denormalize Everything! Normalize Everything! I wish things were faster! I wish things were easier to maintain!
  20. 20. 20© Cloudera, Inc. All rights reserved. Lambda Advantages and Disadvantages
  21. 21. 21© Cloudera, Inc. All rights reserved. Lambda Advantages • Marries diverse strengths of existing open source software into a unified architecture • Provides scalability via the batch layer • Provides real time performance via the speed layer
  22. 22. 22© Cloudera, Inc. All rights reserved. Lambda Disadvantages • Complexity • Many moving parts • Restatement is difficult • Two code bases must be kept in sync • Proper failure handling is complex
  23. 23. 23© Cloudera, Inc. All rights reserved. Lambda Complexity Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala Code must be kept in sync Restatement is difficult
  24. 24. 24© Cloudera, Inc. All rights reserved. Lambda Complexity Batch Layer Serving Layer Speed Layer New Data Data Lake (HDFS) Precompute Views Stream or Micro Batch Increment Views Data Application “Real-time” Increment Batch Recompute Merge Hadoop Storm/Spark HBase Impala Hmm… this data looks fishy Problem Here? Here? Here? Here? Here? Here? Here?
  25. 25. 25© Cloudera, Inc. All rights reserved. The Log as Storage • The idea of representing data as immutable log information is not new and is not without tradeoffs: • Space amplification: how many bytes of data are stored, relative to how many logical bytes the database contains • Write amplification: how many bytes of data are written by the database compared to the number of bytes changed by the user • Read amplification: how many bytes the database has to physically read to return values to the user compared to the bytes returned • Complexity: am I solving a CS problem or a customer problem? • These are not simple issues and there’s no straightforward “right” answer
  26. 26. 26© Cloudera, Inc. All rights reserved. Premature Optimization Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil. --Donald Knuth
  27. 27. 27© Cloudera, Inc. All rights reserved. Gap Filling vs. Optimization • Some Lambda implementations are deployed on big data systems that don’t require significant optimization to deliver desired SLAs • Often, Lambda architectures are used to fill the very stark difference in workload processing capabilities of technologies that are used typically used for the batch (long scan) and fast layers (quick point lookups) • Anecdotally, Lambda architectures seem to be deployed much more often with current generation open source technology than they were with legacy commercial offerings • Part of this is because of data volume, variety, and velocity caused by our increasingly data driven world, but I think part of this is also because legacy technologies haven’t had as stark of a difference in what workloads they’re optimal for • Are you deploying a Lambda architecture because you need to squeeze out all of the performance possible, or because you have a mixed workload that can’t be deployed on one single storage technology?
  28. 28. 28© Cloudera, Inc. All rights reserved. Gap Filling v2: Lack of Mutability • Some Lambda implementations aim to fill the gap of the lack of mutability in HDFS • Raw, master data should be immutable, but in the real world raw data could potentially need to be adjusted • Sensors could have been miscalibrated, data may have been incorrectly entered, raw data might be an approximation before finalization, etc. • Derived aggregations might more efficiently modified in place, vs. recalculated from raw data, recalculating all of history is often not practically possible Incoming Data (Messaging System) New Partition Most Recent Partition Historic Data HBase Parquet File • Wait for running operations to complete • Define new Impala partition referencing the newly written Parquet file Reporting Request Impala on HDFS
  29. 29. 29© Cloudera, Inc. All rights reserved. Kudu as a Lambda alternative
  30. 30. 30© Cloudera, Inc. All rights reserved. HDFS Fast Scans, Analytics and Processing of Stored Data Fast On-Line Updates & Data Serving Arbitrary Storage (Active Archive) Fast Analytics (on fast-changing or frequently-updated data) Kudu: Fast Analytics on Fast-Changing Data New storage engine enables new Hadoop use cases Unchanging Fast Changing Frequent Updates HBase Append-Only Real-Time Kudu Kudu fills the Gap Modern analytic applications often require complex data flow & difficult integration work to move data between HBase & HDFS Analytic Gap Pace of Analysis PaceofData
  31. 31. 31© Cloudera, Inc. All rights reserved. Kudu Increases the Value of Time Series Data Time Series Inserts, updates, scans, lookups Workload Examples Stream market data, fraud detection & prevention, risk monitoring Time series data is most valuable if you can analyze it to change outcomes in real time. Kudu simulateneously enables: • Time series data inserted/updated as it arrives • Analytic scans to find trends on fresh time series data • Lookups to quickly visit the point in time where an event occurred for further investigation
  32. 32. 32© Cloudera, Inc. All rights reserved. Kudu can help spot problems before they happen. Real-time data inserts with the ability to analyze trends identifies potential problems. Kudu identifies trouble through: • Extreme scale, allowing better historic trend analysis • Fast inserts to enable an up-to-date view of your business • Fast scans identify/flag undesired states for remedy Kudu Keeps Your Business Operational Machine Data Analytics Inserts, scans, lookups Workload Examples Network threat detection, IoT, predictive maintenance and failure detection
  33. 33. 33© Cloudera, Inc. All rights reserved. More Versatility in Online Reporting Online Reporting Inserts, updates, scans, lookups Workload Examples “Active” Reporting Online reporting has traditionally been limited by data volume and analytic capabilitiy, keeping only recent data designed for granular queries. Kudu adds online reporting versatility through: • Fast inserts and updates to keep data fresh • Fast lookups and analytic scans in one data store
  34. 34. 34© Cloudera, Inc. All rights reserved. Xiaomi use case • World’s 4th largest smart-phone maker (most popular in China) • Gather important RPC tracing events from mobile app and backend service. • Service monitoring & troubleshooting tool. High write throughput • >5 Billion records/day and growing Query latest data and quick response • Identify and resolve issues quickly Can search for individual records • Easy for troubleshooting
  35. 35. 35© Cloudera, Inc. All rights reserved. Xiaomi big data analytics pipeline Large ETL pipeline delays ● High data visibility latency (from 1 hour up to 1 day) ● Data format conversion woes Ordering issues ● Log arrival (storage) not exactly in correct order ● Must read 2 – 3 days of data to get all of the data points for a single day
  36. 36. 36© Cloudera, Inc. All rights reserved. Xiaomi big data analytics pipeline Simplified with Kudu Low latency ETL pipeline ● ~10s data latency ● For apps that need to avoid direct backpressure or need ETL for record enrichment Direct zero-latency path ● For apps that can tolerate backpressure and can use the NoSQL APIs ● Apps that don’t need ETL enrichment for storage / retrieval OLAP scan Side table lookup Result store
  37. 37. 37© Cloudera, Inc. All rights reserved. Conclusions • Lambda has a real place in big data architectures • Optimize as needed, but beware of the cost of premature optimization • Kudu is designed to be a simple solution for when you need a data store that’s updatable and provides “good enough” performance for analytic and real time workloads simultaneously

×