Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Data-Centric Businesses

19,514 views

Published on

Both digital and traditional businesses are constantly evolving, and the need to move fast is a pervasive reality. Delivering what customers want and need goes beyond the creation of delivery channels. In fact, it relies on the company’s ability to produce, consume, organise, understand, curate, and distribute data.

In this presentation, Dan Aragao and Simon Hope provide a glimpse of the journey ThoughtWorks and REA are currently undergoing to create a truly data-centric, cutting-edge digital business.

Published in: Data & Analytics, Technology
  • Secrets to making $$$ with paid surveys...  http://ishbv.com/surveys6/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Natural enlargement book for free? [Download] Your natural enlargement blueprint. ➤➤ https://tinyurl.com/getpebible2019
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area for one night is there tinyurl.com/hotsexinarea Copy and paste link in your browser to visit a site)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Get BIGGER and LONGER Without Surgery, Forget Surgery, Pills And Extenders, DON'T Do This To Make It Bigger. ●●● https://tinyurl.com/getpebible2019
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great book! I'm on the second week and seeing definite improvement on my yeast infection symptoms. I must also note that the texture of my skin became very soft and consistent. I completely agree with many of your arguments in the book especially those about conventional methods for treating Yeast Infection. I wanted to let you know that I feel so fine you have no idea. I am not so tired all the time and I feel very energetic. The rashes on my neck and on the right side of my nose have also cleared up. I wish you all the health and happiness in the world and thanks so much for helping me. I really appreciate your time and efforts to assist me. ■■■ http://scamcb.com/index7/pdf
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Building Data-Centric Businesses

  1. Daniel Aragao & Simon Hope
  2. Daniel Aragao Simon Hope @dear_dr_dan @mapbutcher
  3. REALESTATE.COM.AU 6BMarket Cap 11MAustralian Properties 55MVisits in September 4.7MApp Downloads …and counting
  4. 3,500PEOPLE 13COUNTRIES 34OFFICES TECHNOLOGY & SOCIAL JUSTICE
  5. • In the beginning… • Organising our Data • Implementation approaches • Hipster Batches • Reactify • Bring Your Own Data • Finding the Data • What we have learned so far THIS IS WHAT THE STORY IS ABOUT
  6. SORRY… IT’S OK TO LEAVE NOW • Nope, we didn’t create a new Hadoop • No hardcore Data Science • There are some implementation details • REA embraced the Cloud. AWS everywhere • Under construction
  7. IN THE BEGINNING…
  8. ORGANISING OUR DATA Increasingly, content is being 
 distributed through search
 and social platforms...
 
 There’s less visiting 
 of publishers as destinations. Jeff Weiner, CEO, Linkedin
  9. Data sources Data warehouse PROBLEM…
  10. STRATEGY…
  11. STRATEGY…
  12. STRATEGY…
  13. Data Warehouse StagingSSIS Dim Fact PROBLEM…
  14. Data Warehouse StagingSSIS Dim Fact PROBLEM… Star schema leaky details
  15. No Data Warehouse StagingSSIS Dim Fact STRATEGY…
  16. STRATEGY… Data Warehouse Facade StagingSSIS Dim Fact
  17. ??? WHAT’S IN THE BOX?
  18. Good things come in small packages services THE HIPSTER BATCH ??? Hipster Batch
  19. Hipster Batch THE HIPSTER BATCH • Small and short lived • Decoupled via flat files via S3 • Single purpose • Idempotent • Polyglot • Minimal runtime dependencies • Discoverable
  20. SNS, SQS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  21. SNS, SQS ASG, ECS, Lambda Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  22. SNS, SQS ASG, ECS, Lambda KMS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  23. Logs SNS, SQS ASG, ECS, Lambda KMS Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  24. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  25. Logs SNS, SQS ASG, ECS, Lambda KMS Cloudwatch S3 buckets Data A ‘TYPICAL’ IMPLEMENTATION Hipster Batch
  26. Hipster Batch HIPSTER BATCH DOES SCIENCE • Behavioural models for targeted marketing • Recommendation engine • External channels
  27. Hipster Batch SCIENCE!
  28. x 20 Hipster Batch Stats models SCIENCE!
  29. x 20 API Hipster Batch Stats models SCIENCE!
  30. API x 20 API Hipster Batch Stats models SCIENCE!
  31. API x 20 API Hipster Batch Stats models SCIENCE!
  32. API x 20 API Hipster Batch Stats models Google Now API SCIENCE!
  33. From legacy to reactive REACTIFY Reactify ???
  34. Reactify http://www.reactivemanifesto.org REACTIFY • Manage Data flow with messages • Protect consumers and care about isolation • Resilience is important and Data replication is just fine • Demand is elastic - and your components should be too
  35. Reactify Listings Data coupling No resilience or elasticity Coupling PROBLEM…
  36. Reactify Listings SOLUTION…
  37. Reactify Listings Reactify SOLUTION…
  38. Reactify Listings Reactify SOLUTION…
  39. Reactify Listings Reactify Hipster Batch SOLUTION…
  40. Reactify Listings Reactify Hipster Batch Shielded consumers IsolationDecoupled SOLUTION…
  41. Reactify Listings IMPLEMENTATION…
  42. Reactify Listings REST API IMPLEMENTATION…
  43. Reactify Listings REST API IMPLEMENTATION…
  44. Reactify Listings REST API Dynamo Event Maker Event Differ IMPLEMENTATION…
  45. Reactify Listings REST API Dynamo Event Maker Event Differ Kinesis 2 IMPLEMENTATION… 2
  46. • Exposes current state only • Stream of change notifications • Hypertext Application Language - HAL • Clear entity types • Linking over embedding • Cacheable and discoverable REST API REACTIFY REST API
  47. REST API https://feeds.listings.realestate.com.au/combined-listings/120449689
  48. REST API https://feeds.listings.realestate.com.au/combined-listings/120449689
  49. REST API https://feeds.listings.realestate.com.au/combined-listings/120449689
  50. REST API https://feeds.listings.realestate.com.au/combined-listings/120449689
  51. REST API Event Maker https://feeds.listings.realestate.com.au/combined-listings/-/changes
  52. REST API Event Maker https://feeds.listings.realestate.com.au/combined-listings/-/changes
  53. REST API Event Maker https://feeds.listings.realestate.com.au/combined-listings/-/changes
  54. REST API Event Maker https://feeds.listings.realestate.com.au/combined-listings/-/changes
  55. Reactify Event Differ
  56. Reactify Event Differ
  57. Reactify Event Differ
  58. Reactify Event Differ
  59. The octopus in the box — Did you use that data set? — Errr… No, we have another one BRING YOUR OWN DATA
  60. BRING YOUR OWN DATA - BYOD • Allow data to flow freely • Help the business to get what they need when they need it • Self-service
  61. BYOD
  62. BYOD CSV
  63. BYOD CSV x 5
  64. BYOD CSV x 5 Smarts on datatypes
  65. BYOD CSV x 5 Tableau Server Smarts on datatypes
  66. BYOD CSV x 5 Tableau Server Smarts on datatypes
  67. BYOD CSV x 5 Tableau Server Audit, auth, share… Smarts on datatypes
  68. These were the implementation approaches, now to… FIND THE DATA Meaningful, automated, 
 and easy-to-search metadata
  69. WE TRIED
  70. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORE THAN DATA Hipster Batch
  71. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs MORE THAN DATA Hipster Batch
  72. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry MORE THAN DATA Hipster Batch
  73. SNS, SQS ASG, ECS, Lambda KMS Cloudwatch Logs Dataz Ancestry Metadata MORE THAN DATA Hipster Batch
  74. Ancestry
  75. Ancestry
  76. Ancestry
  77. Ancestry
  78. Ancestry
  79. REST API METADATA PIPELINE Producers
  80. REST API Ancestry Ancestry Ancestry METADATA PIPELINE Producers
  81. REST API Ancestry Ancestry Ancestry METADATA PIPELINE Producers
  82. REST API Ancestry Ancestry Ancestry METADATA PIPELINE Producers Scrapy
  83. REST API Ancestry Ancestry Ancestry METADATA PIPELINE Producers Scrapy
  84. REST API Ancestry Ancestry Ancestry METADATA PIPELINE Producers Scrapy
  85. WHAT WE HAVE LEARNED SO FAR • Consumers create the last-mile data as needed • We must work with external, independent delivery channels • Push quality back to source/producer systems • Data belongs to the entire organisation, 
 not to a single team
  86. I’ll give you my 
 Data Warehouse 
 when you can pry it
 from my cold dead hands.
  87. THANKYOU Daniel Aragao Simon Hope @dear_dr_dan @mapbutcher REALESTATE.COM.AU

×