Download as PDF, PPTX
Uploaded byThoughtworks
Building Data-Centric Businesses
The document discusses the implementation of data management strategies at Realestate.com.au, focusing on cloud-based solutions and the use of a 'hipster batch' methodology for data processing. It highlights the importance of allowing data to flow freely within the organization and emphasizes the need for resilient systems that prioritize consumer isolation. Key lessons include the collective ownership of data and the necessity of leveraging external delivery channels to enhance data quality.
More Related Content
Similar to Building Data-Centric Businesses
![[DSC Europe 25] Ivan Petrovic - Is it really that expensive to build an AI sy...](https://cdn.slidesharecdn.com/ss_thumbnails/ybqhdwvusbg7jms3doxh-9-251216105605-7aab5a10-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...](https://cdn.slidesharecdn.com/ss_thumbnails/gsbe3y5it5uhndi4e08e-1-251212103249-f1008e0c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Nikolay Burlutskiy - Best Practices for Building Enterprise M...](https://cdn.slidesharecdn.com/ss_thumbnails/uirvaiuvq8y1w8hzd9tx-7-251212103249-2619edb4-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Tatevik Maytesyan - How to actually use AI in marketing: gett...](https://cdn.slidesharecdn.com/ss_thumbnails/tjo626lsqdgfntbgl2mw-4-251216103155-e36cd239-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Debmalya Biswas - Agentification: the art of transforming man...](https://cdn.slidesharecdn.com/ss_thumbnails/r5azlggvtqiaiiusrqdr-4-251212103249-5a12c89b-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dobrica Cosic - From Electrons to Innovation: How Granular Da...](https://cdn.slidesharecdn.com/ss_thumbnails/h4qk69zereaumbceubgr-dobrica-cosic-from-electrons-to-innovation-how-granular-data-and-analytics-are--251218085301-b982fb14-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Maria Kokiasmenos - AI Governance US Perspective.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/eszqnbzlsqa2vch6dmci-6-251215095918-6fcdf45f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Kaja Kandare - LLM as a judge.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/arxyccaxsdsd1ba99wjw-7-251212104007-2b4e3f64-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Katherine Forrest - AI NOW: Understanding the Velocity of Cha...](https://cdn.slidesharecdn.com/ss_thumbnails/wvvbruqfrci0sfq9xwgb-4-251212104007-e5ad1987-thumbnail.jpg?width=640&height=640&fit=bounds)
Building Data-Centric Businesses
- 1. Daniel Aragao &Simon Hope
- 2. Daniel Aragao SimonHope @dear_dr_dan @mapbutcher
- 3.
- 4.
- 5. • In thebeginning… • Organising our Data • Implementation approaches • Hipster Batches • Reactify • Bring Your Own Data • Finding the Data • What we have learned so far THIS IS WHAT THE STORY IS ABOUT
- 6. SORRY… IT’S OKTO LEAVE NOW • Nope, we didn’t create a new Hadoop • No hardcore Data Science • There are some implementation details • REA embraced the Cloud. AWS everywhere • Under construction
- 7.
- 30. ORGANISING OUR DATA Increasingly,content is being distributed through search and social platforms... There’s less visiting of publishers as destinations. Jeff Weiner, CEO, Linkedin
- 31.
- 32.
- 33.
- 34.
- 35. Data Warehouse StagingSSIS DimFact PROBLEM…
- 36. Data Warehouse StagingSSIS DimFact PROBLEM… Star schema leaky details
- 37. No Data Warehouse StagingSSISDim Fact STRATEGY…
- 38.
- 39.
- 40. Good things comein small packages services THE HIPSTER BATCH ??? Hipster Batch
- 41. Hipster Batch THE HIPSTERBATCH • Small and short lived • Decoupled via flat files via S3 • Single purpose • Idempotent • Polyglot • Minimal runtime dependencies • Discoverable
- 42. SNS, SQS Data A ‘TYPICAL’IMPLEMENTATION Hipster Batch
- 43. SNS, SQS ASG, ECS, Lambda Data A‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 44. SNS, SQS ASG, ECS, Lambda KMS Data A‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 45. Logs SNS, SQS ASG, ECS, Lambda KMS Data A‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 46. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Data A‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 47. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch S3buckets Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 48. Hipster Batch HIPSTER BATCHDOES SCIENCE • Behavioural models for targeted marketing • Recommendation engine • External channels
- 49.
- 50. x 20 Hipster Batch Statsmodels SCIENCE!
- 51. x 20 API Hipster Batch Statsmodels SCIENCE!
- 52. API x 20 API Hipster Batch Statsmodels SCIENCE!
- 53. API x 20 API Hipster Batch Statsmodels SCIENCE!
- 54. API x 20 API Hipster Batch Statsmodels Google Now API SCIENCE!
- 55. From legacy toreactive REACTIFY Reactify ???
- 56. Reactify http://www.reactivemanifesto.org REACTIFY • Manage Dataflow with messages • Protect consumers and care about isolation • Resilience is important and Data replication is just fine • Demand is elastic - and your components should be too
- 57.
- 58.
- 59.
- 60.
- 61.
- 62.
- 63.
- 64.
- 65.
- 66.
- 67.
- 68. • Exposes currentstate only • Stream of change notifications • Hypertext Application Language - HAL • Clear entity types • Linking over embedding • Cacheable and discoverable REST API REACTIFY REST API
- 69.
- 70.
- 71.
- 72.
- 73.
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81. The octopus inthe box — Did you use that data set? — Errr… No, we have another one BRING YOUR OWN DATA
- 82. BRING YOUR OWNDATA - BYOD • Allow data to flow freely • Help the business to get what they need when they need it • Self-service
- 83.
- 84.
- 85.
- 86.
- 87.
- 88.
- 89.
- 90. These were theimplementation approaches, now to… FIND THE DATA Meaningful, automated, and easy-to-search metadata
- 91.
- 92. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORETHAN DATA Hipster Batch
- 93. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORETHAN DATA Hipster Batch
- 94. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry MORETHAN DATA Hipster Batch
- 95. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry Metadata MORETHAN DATA Hipster Batch
- 96.
- 97.
- 98.
- 99.
- 100.
- 101.
- 102.
- 103.
- 104.
- 105.
- 106.
- 107. WHAT WE HAVELEARNED SO FAR • Consumers create the last-mile data as needed • We must work with external, independent delivery channels • Push quality back to source/producer systems • Data belongs to the entire organisation, not to a single team
- 108. I’ll give youmy Data Warehouse when you can pry it from my cold dead hands.
- 109. THANKYOU Daniel Aragao SimonHope @dear_dr_dan @mapbutcher REALESTATE.COM.AU