Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Patterns- Andrew Jones (By ThoughtWorks)


Published on

The data heart of many businesses is monolithic legacy systems, beating powerfully but slowly as they are hardened against rapid change. However, to quickly derive business insight, act for customers in real time, and empower autonomous product teams, we look to microservice architectures and self-service data platforms to infuse the entire organisation with arterial streams of data.

To make the change is equivalent to open-heart surgery - how do we maintain the data pulse while we animate every part of the business with responsive data? The good news for the patient is that we have surgically precise techniques available.

Andrew shares a range of approaches to common scenarios when migrating incrementally from monolithic legacy systems and batch processes to modern streaming architectures and services bounded by business contexts. He also shares some general design principles for modern data architectures. This will provide a practical framework for thinly slicing data migrations that your organisation can start using today.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Data Patterns- Andrew Jones (By ThoughtWorks)

  2. 2. Why would we want to move away from monoliths?
  3. 3. Slow to change Cross team dependencies Complex integration Legacy patterns Slow to new produce insights Entangled data
  4. 4. Data is the lifeblood of your organisation - systems work with it, analysts work with it, and everyone needed it yesterday
  5. 5. What do we want out of a modern data architecture?
  6. 6. Evolvability
  7. 7. Evolvability Scalability
  8. 8. Evolvability Scalability Reliability
  9. 9. Evolvability Scalability Reliability Integrity
  10. 10. So, what does a modern data architecture actually look like?
  11. 11. Asynchronous events
  12. 12. Immutable log-based streaming
  13. 13. Microservices
  14. 14. Asynchronous events + Immutable log-based streaming + Microservices
  15. 15. Unbundling the database
  16. 16. The Data Patterns Catalogue
  17. 17. Strangler pattern
  18. 18. Using event seams to extract capabilities
  19. 19. Batch to event adapter
  20. 20. Event to batch adapter
  21. 21. Self-service data governance
  22. 22. Change data capture
  23. 23. Designing for auditability
  24. 24. Enrichment patterns
  25. 25. CREATE STREAM pageviews_enriched AS SELECT pv.viewtime, pv.userid AS userid, pv.pageid, pv.timestring, u.gender, u.regionid, u.interests, u.contactinfo FROM pageviews_transformed pv LEFT JOIN users_5part u ON pv.userid = u.userid;
  26. 26. No single tool to rule them all
  27. 27. Trust but verify
  28. 28. Reprocessing
  29. 29. Data patterns catalogue - written version coming soon
  30. 30. Data Engineering Melbourne Meetup Wednesday 14th November, 6pm ThoughtWorks office Want to talk more about data architecture and engineering?
  31. 31. Thanks! Andrew Jones @whereismytaco