Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Lambda Architecture:
How We Merged Batch and Real-Time
Sewook Wee, Senior Engineering Lead
Sotos Matzanas, Tech Lead
June ...
Our Goal
Our goal at Trulia is to
give consumers an easy
and enjoyable way to find
their next home by
providing data and
i...
Personalization Team
• Who we are
• What we do
• How we do it
User ID: 30942342
Browser Cookies: [1411087c06c6b530c155b933fdee44e2ef8905]
SearchHistory: [
{time: 2016-01-21T23:38:31Z,
...
Why Lambda Architecture
We needed a way to…
• Recalculate the full User Trait from full event body at scale
• Read it back...
Enter: Lambda Architecture
Event
Master
Dataset
Batch
Processing
Real-Time
Processing
Serving Layer
User Trait API
Desktop...
Our User Model
• We support both registered and unregistered users
• Registered users: user id + secondary id(s) (mobile, ...
Our Real-Time Complications
• User linkage can change while new batch is calculated
• New user linkage can appear during t...
Event Event Event Parse
Linkage
Lookup
HBase
Store
Transfers Writes Reads
Redis
user id
Simple Real-Time Case
Parse
Linkage
Lookup
Yesterday’s
Lookup
(Hbase)
Rebalance user id
secondary id
user id
Redis
Store
Send as Control Events
...
Transition to a New Epoch
• When rebalance of all ids is complete
• Completion of rebalance: no new user id has been rebal...
Rebalance
Done for N+1
Midnight Rebalance
Done for N
Batch Layer
Events for
Epoch N
Batch Layer
N Done
Midnight
Batch Laye...
Our Input and its Size
• Hundreds of millions of events per day
• Billions of events per month
• 12TBs of events, and grow...
As a Result
• Continuously add new features to build data driven products
• Retroactively apply new features on old data
•...
swee@trulia.com
smatzanas@trulia.com
Upcoming SlideShare
Loading in …5
×

Lambda Architecture: How we merged batch and real time

1,544 views

Published on

Lambda Architecture: How we merged batch and real time

Published in: Technology
  • Be the first to comment

Lambda Architecture: How we merged batch and real time

  1. 1. Lambda Architecture: How We Merged Batch and Real-Time Sewook Wee, Senior Engineering Lead Sotos Matzanas, Tech Lead June 27 2016
  2. 2. Our Goal Our goal at Trulia is to give consumers an easy and enjoyable way to find their next home by providing data and insights to help them make the best decision.
  3. 3. Personalization Team • Who we are • What we do • How we do it
  4. 4. User ID: 30942342 Browser Cookies: [1411087c06c6b530c155b933fdee44e2ef8905] SearchHistory: [ {time: 2016-01-21T23:38:31Z, query: “/for_sale/San_Francisco,CA/2p_beds/1p_baths”} ] lastVisited: 2016-01-21T23:52:55Z locationPreference: { “San Francisco, CA” : 0.52, “Oakland, CA”: 0.39, “Los Angeles, CA” : 0.09 } User Type: { buyer: 0.87, renter : 0.13 } Examples of User Trait user-device linkage activity summary inferred insights Recommended properties: [ { propertyId: 3223394214, score: 0.22}, { propertyId: 3223518578, score: 0.09} ]
  5. 5. Why Lambda Architecture We needed a way to… • Recalculate the full User Trait from full event body at scale • Read it back fast • Ability to add new metrics to old aggregates • Refresh near real-time to catch up the delta
  6. 6. Enter: Lambda Architecture Event Master Dataset Batch Processing Real-Time Processing Serving Layer User Trait API Desktop, Mobile Web Mobile Apps Email & Push Event API User Trait (Real-Time) User Trait (Batch) Kafka
  7. 7. Our User Model • We support both registered and unregistered users • Registered users: user id + secondary id(s) (mobile, Web, email) • User login: link and merge all known activity on all devices
  8. 8. Our Real-Time Complications • User linkage can change while new batch is calculated • New user linkage can appear during the day, and not reflected in batch calculation • We needed to plan for these and make sure eventual User Trait reflects the state of a user as of right now
  9. 9. Event Event Event Parse Linkage Lookup HBase Store Transfers Writes Reads Redis user id Simple Real-Time Case
  10. 10. Parse Linkage Lookup Yesterday’s Lookup (Hbase) Rebalance user id secondary id user id Redis Store Send as Control Events Control Bolt Lookup Change Lookup Change Lookup Change Control Event Spout Rebalance Time @Batch Completion Time Get all user ids + secondary ids for today Transfers Writes Reads Current Real-Time Design Event Event Event Kafka Spout Today’s Lookup (Hbase)
  11. 11. Transition to a New Epoch • When rebalance of all ids is complete • Completion of rebalance: no new user id has been rebalanced for 30 seconds • Redis keys with TTL mark a heartbeat that disappears if no new control events
  12. 12. Rebalance Done for N+1 Midnight Rebalance Done for N Batch Layer Events for Epoch N Batch Layer N Done Midnight Batch Layer Events for N + 1 Speed Layer Events Epoch N Batch Layer N +1 Done Timeline Epoch Transitions Serve N + 1 Serving Batch N + Speed N Batch Layer Epoch N Real-Time Layer Epoch N Speed Layer Events for N + 1 Batch Layer Epoch N + 1 Real-Time Layer Epoch N + 1 Rebalance for N Event Processing Epoch Serving FromBatch Process Rebalance for N
  13. 13. Our Input and its Size • Hundreds of millions of events per day • Billions of events per month • 12TBs of events, and growing • Hundreds of millions of User Traits calculated daily • Millions calculated in real-time
  14. 14. As a Result • Continuously add new features to build data driven products • Retroactively apply new features on old data • A virtuous cycle of learning more, personalizing more, and learning again • Delivery of data and insights to help consumers make the best decision
  15. 15. swee@trulia.com smatzanas@trulia.com

×