The document discusses migrating from monolithic data architectures to more modern architectures. It outlines some benefits of modern architectures like evolvability, scalability, and reliability. It then describes characteristics of a modern architecture like using asynchronous events, immutable log-based streaming, and microservices. It also discusses patterns for a modern data architecture like unbundling databases, using event seams, and designing for auditability and enrichment.

1. DATA PATTERNS: MIGRATING FROM MONOLITH TO
MODERN ARCHITECTURES
Andrew Jones

2. Why would we want to move away from
monoliths?

3. Slow to change
Cross team dependencies
Complex integration
Legacy patterns
Slow to new produce insights
Entangled data

4. Data is the lifeblood of your organisation
- systems work with it, analysts work with
it, and everyone needed it yesterday

5. What do we want out of a modern data
architecture?

6. Evolvability

7. Evolvability
Scalability

8. Evolvability
Scalability
Reliability

9. Evolvability
Scalability
Reliability
Integrity

10. So, what does a modern data
architecture actually look like?

11. Asynchronous events

12. Immutable log-based streaming

13. Microservices

14. Asynchronous events
+
Immutable log-based streaming
+
Microservices

15. Unbundling the database

16. The Data Patterns Catalogue

17. Strangler pattern

25. Using event seams to extract capabilities

26. Batch to event adapter

33. Event to batch adapter

37. Self-service data governance

38. Change data capture

39. Designing for auditability

40. Enrichment patterns

41. CREATE STREAM pageviews_enriched AS
SELECT pv.viewtime,
pv.userid AS userid,
pv.pageid,
pv.timestring,
u.gender,
u.regionid,
u.interests,
u.contactinfo
FROM pageviews_transformed pv
LEFT JOIN users_5part u ON pv.userid =
u.userid;

42. No single tool to rule them all

43. Trust but verify

44. Reprocessing

45. Data patterns catalogue - written version
coming soon

46. Data Engineering Melbourne Meetup
Wednesday 14th November, 6pm
ThoughtWorks office
https://www.meetup.com/Data-Engineering-Melbourne
Want to talk more about data architecture and engineering?

47. Thanks!
Andrew Jones
@whereismytaco
akjones@thoughtworks.com