Building Data-Centric Businesses
•
96 likes•33,761 views
Both digital and traditional businesses are constantly evolving, and the need to move fast is a pervasive reality. Delivering what customers want and need goes beyond the creation of delivery channels. In fact, it relies on the company’s ability to produce, consume, organise, understand, curate, and distribute data. In this presentation, Dan Aragao and Simon Hope provide a glimpse of the journey ThoughtWorks and REA are currently undergoing to create a truly data-centric, cutting-edge digital business.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Building Data-Centric Businesses
Similar to Building Data-Centric Businesses (20)
More from Thoughtworks
More from Thoughtworks (20)
Recently uploaded
Recently uploaded (20)
Building Data-Centric Businesses
- 1. Daniel Aragao & Simon Hope
- 2. Daniel Aragao Simon Hope @dear_dr_dan @mapbutcher
- 3. REALESTATE.COM.AU 6BMarket Cap 11MAustralian Properties 55MVisits in September 4.7MApp Downloads …and counting
- 5. • In the beginning… • Organising our Data • Implementation approaches • Hipster Batches • Reactify • Bring Your Own Data • Finding the Data • What we have learned so far THIS IS WHAT THE STORY IS ABOUT
- 6. SORRY… IT’S OK TO LEAVE NOW • Nope, we didn’t create a new Hadoop • No hardcore Data Science • There are some implementation details • REA embraced the Cloud. AWS everywhere • Under construction
- 30. ORGANISING OUR DATA Increasingly, content is being distributed through search and social platforms... There’s less visiting of publishers as destinations. Jeff Weiner, CEO, Linkedin
- 32. STRATEGY…
- 33. STRATEGY…
- 34. STRATEGY…
- 35. Data Warehouse StagingSSIS Dim Fact PROBLEM…
- 36. Data Warehouse StagingSSIS Dim Fact PROBLEM… Star schema leaky details
- 37. No Data Warehouse StagingSSIS Dim Fact STRATEGY…
- 38. STRATEGY… Data Warehouse Facade StagingSSIS Dim Fact
- 39. ??? WHAT’S IN THE BOX?
- 40. Good things come in small packages services THE HIPSTER BATCH ??? Hipster Batch
- 41. Hipster Batch THE HIPSTER BATCH • Small and short lived • Decoupled via flat files via S3 • Single purpose • Idempotent • Polyglot • Minimal runtime dependencies • Discoverable
- 42. SNS, SQS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 43. SNS, SQS ASG, ECS, Lambda Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 44. SNS, SQS ASG, ECS, Lambda KMS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 45. Logs SNS, SQS ASG, ECS, Lambda KMS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 46. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 47. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch S3 buckets Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
- 48. Hipster Batch HIPSTER BATCH DOES SCIENCE • Behavioural models for targeted marketing • Recommendation engine • External channels
- 50. x 20 Hipster Batch Stats models SCIENCE!
- 51. x 20 API Hipster Batch Stats models SCIENCE!
- 52. API x 20 API Hipster Batch Stats models SCIENCE!
- 53. API x 20 API Hipster Batch Stats models SCIENCE!
- 54. API x 20 API Hipster Batch Stats models Google Now API SCIENCE!
- 55. From legacy to reactive REACTIFY Reactify ???
- 56. Reactify http://www.reactivemanifesto.org REACTIFY • Manage Data flow with messages • Protect consumers and care about isolation • Resilience is important and Data replication is just fine • Demand is elastic - and your components should be too
- 68. • Exposes current state only • Stream of change notifications • Hypertext Application Language - HAL • Clear entity types • Linking over embedding • Cacheable and discoverable REST API REACTIFY REST API
- 81. The octopus in the box — Did you use that data set? — Errr… No, we have another one BRING YOUR OWN DATA
- 82. BRING YOUR OWN DATA - BYOD • Allow data to flow freely • Help the business to get what they need when they need it • Self-service
- 83. BYOD
- 84. BYOD CSV
- 85. BYOD CSV x 5
- 89. BYOD CSV x 5 Tableau Server Audit, auth, share… Smarts on datatypes
- 90. These were the implementation approaches, now to… FIND THE DATA Meaningful, automated, and easy-to-search metadata
- 91. WE TRIED
- 92. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORE THAN DATA Hipster Batch
- 93. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORE THAN DATA Hipster Batch
- 94. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry MORE THAN DATA Hipster Batch
- 95. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry Metadata MORE THAN DATA Hipster Batch
- 96. Ancestry
- 97. Ancestry
- 98. Ancestry
- 99. Ancestry
- 100. Ancestry
- 107. WHAT WE HAVE LEARNED SO FAR • Consumers create the last-mile data as needed • We must work with external, independent delivery channels • Push quality back to source/producer systems • Data belongs to the entire organisation, not to a single team
- 108. I’ll give you my Data Warehouse when you can pry it from my cold dead hands.
- 109. THANKYOU Daniel Aragao Simon Hope @dear_dr_dan @mapbutcher REALESTATE.COM.AU