Recommended
Recommended
More Related Content
Similar to Data Mesh - Anders Boje - Copenhagen Data Engineering Meetup (24 mar 2022)
Similar to Data Mesh - Anders Boje - Copenhagen Data Engineering Meetup (24 mar 2022) (20)
Recently uploaded
Recently uploaded (20)
Data Mesh - Anders Boje - Copenhagen Data Engineering Meetup (24 mar 2022)
- 1. Data Mesh Rethinking Enterprise Data Architecture
- 2. Anders Boje Hertz Head of AI & Data Platforms at INTELLISHORE anders.hertz@Intellishore.dk https://www.linkedin.com/in/andersboje/
- 3. We focus on providing the next generation of consultancy for the digital age: a true passion for data, a combination of strategy advisors and native technologists, and a dedication to delivering actionable insights and cultural change 45+ Employees In 2013 It all started 3,500 employees trained In analytics solutions developed by Intellishore 200+ analytics projects completed 34.8% Top-line growth in 2020 & 29.7% in 2019
- 4. AND A COMPREHENSIVE TOOLKIT TO SUPPORT THE DIGITAL TRANSFORMATION | NON-EXHAUSTIVE | IGNITING DATA – ENGAGING PEOPLE | 2021 ENGAGING PEOPLE IGNITING DATA Sources & Platforms Industry Automation Decision Maturity Learning Platform Visual Analytics Algorithmic Science User Journey Strategizing | Data & AI
- 5. Defining a shared action plan for all elements of transformation: business, technical and organizational 001 Establishing the setup to continuously validate, access, explore and expand the foundation of quality data 002 Turning the insights from the data into accessible reports and tools that can be used in day-to-day operations 003 Building up the organization’s capabilities and ways of working to drive data literacy and usage 004 Ensuring that data scientists, engineers and their analyses are fully engaged and integrated with the day- to-day business 005 TRANSFORMATIONS ARE DRIVEN BY HOW FAST COMPANIES MASTER 5 DISCIPLINES IGNITING DATA – ENGAGING PEOPLE | 2020
- 6. let's get data messy
- 7. CENTRALIZED | MONOLITHIC Ubiquitous Data Innovation Agenda BIG DATA | AI PLATFORM
- 8. Warehouse Data Lake Cloud THE NEXT GENERATION DATA PLATFORM … and yet, the failure modes remain
- 9. Data Mesh A decentralized sociotechnical approach in managing and accessing analytical data at scale.
- 10. IGNITING DATA – ENGAGING PEOPLE | 2021 Domain-oriented decentralised data ownership and architecture Data as a product Self-serve data infrastructure as a platform Federated computational governance THE FOUR PRINCIPLES OF DISTRIBUTED ARCHITECTURE
- 11. IGNITING DATA – ENGAGING PEOPLE | 2021 Domains aligned with the origin of data Domains aligned with shared aggregates Domains aligned with the consumption DECOMPOSE DATA AROUND DOMAINS Distribute the ownership
- 12. IGNITING DATA – ENGAGING PEOPLE | 2021 Domain Data Product Owner L T E Domain Data Product SERVE DATA AS A PRODUCT Delight the consumer with ease of data discovery and use
- 13. IGNITING DATA – ENGAGING PEOPLE | 2021 ENABLE AUTONOMY Abstract technical complexity in self-serve data infrastructure Data | ML Infrastructure as a Platform Data Infra Team
- 14. IGNITING DATA – ENGAGING PEOPLE | 2021 BUILD AN ECOSYSTEM Create a federated and global governance Data Infra as a Platform Global Governance| Open Standards
- 15. IGNITING DATA – ENGAGING PEOPLE | 2021 Federated Computational Governance Apps Multi-plane Data Platforms (Transactional Data, Code) Domain’s App Devs Data Platform Teams Micro Service Legacy App Data Product as Architecture Quantum (Analytical Data, Meta-data, Code / Pipeline, Policy sidecar) Domain’s Data Product Devs and Data Product Owner Domain representative Domain DP owners
- 16. I though this was a tech community?
- 17. IGNITING DATA – ENGAGING PEOPLE | 2021
- 18. IGNITING DATA – ENGAGING PEOPLE | 2021
- 19. IGNITING DATA – ENGAGING PEOPLE | 2021
- 20. But why?
- 21. IGNITING DATA – ENGAGING PEOPLE | 2021
- 22. Data Mesh A decentralized sociotechnical approach in managing and accessing analytical data at scale.
- 23. IGNITING DATA – ENGAGING PEOPLE | 2021 While data (for most) may not be a “product” in a strictly economic sense, it is still the life-blood of organizational decision making, and should not be allowed to become a by-product. Serge Gershkovich
- 24. REFERENCES Dehghani, Z. (2020). Data Mesh Principles and Logical Architecture. Retrieved from Martin Fowler: https://martinfowler.com/ Moses, B. (2022). Data Mesh 101: Everything You Need To Know to Get Started. Retrieved from Monte Carlo Data: https://www.montecarlodata.com/blog-data-mesh-101-everything-you-need-to-know-to-get-started/ Thoughtworks. (2022). Introduction to Data Mesh A principled approach. Retrieved from Thoughtworks: https://www.thoughtworks.com/what-we-do/data-and-ai/data-mesh Data Mesh Learning. (2022). Intro to Data Mesh. Retrieved from Data Mesh Learning: https://datameshlearning.com/ Dehghani, Z. (2019). How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh. Retrieved from Martin Fowler: https://martinfowler.com/articles/data-monolith-to-mesh.html Gershkovich, S. (2022). Data Mesh: overhyped, misunderstood, and useful! Retrieved from Medium: https://medium.com/sqldbm/data-mesh-overhyped-misunderstood-and-useful-e65c60ba6643
- 26. Thanks
- 27. Anders Boje Hertz Head of AI & Data Platforms at INTELLISHORE anders.hertz@Intellishore.dk https://www.linkedin.com/in/andersboje/
Editor's Notes
- Data is always own by a specific domain in the business. Access to that data I decentralized A team is a business oriented technology team. Data Mesh – we are extending the capabilities and also the accountability of those teams to serve and share data for analytical puporses and also embed ML and analytics in the domain. Removing every data middelman EXIT Some other problem may arise, where you could imaging that this could create these data siloing, right? A sales team have all the data they need to optimize processing my sales and I have no incentive to share that data with anybody else. Beacuase I don’t care about their needs really. So the second principle is “data as a product” that tries to address is the data silings.
- Data is consided as a product by each team that publish the data. Product thinking about that data. It is not a asset we collect but it’s a product that we share and we are accountable of the experience for the analysis using the data. Accountability is the key The team is wholly responsible for it Its quality Its representation Its cohesiveness Like it was a thing they where sculpting something out of glass and polishing it and putting in at the storefront. They just want that thing to look nice and usefull.
- Data is available everywhere and self-serve anywhere in the company. Now I hear what you’re saying. You’re thinking about governance, that still is a thing but in principle, these data products are published and there are available everywhere. If you are producing sales report for at sales forecast for a company in Germany, you can find and source all of the data you need to drive that report. Getting that data from all that data where it lives to some database you have control of. Address the cost and feasibility problem, that arise form this decentralized ownership of data prodcuts. Look at infrastructure with a higher level of abstraction of complexity of data infrastructure then what we have to today. So we can lower the cognitive loads of these teams so that genealists, ap developers, programmers, people we have in or organization are able to do data work, are able to create those analytics, insights and create and share data products
- Last principle is about of doing all that with out compromising security, privacy, making sure that data is discoverable, when we blend different domains, we can still make sense of who the customer is. https://www.youtube.com/watch?v=zfFyE3xmJ7I&ab_channel=Confluent For any of these operations to be possible, a data mesh implementation requires a governance model that embraces decentralization and domain self-sovereignty, interoperability through global standardization, a dynamic topology and most importantly automated execution of decisions by the platform. a “common ground” for the whole platform where all data products conform to a shared set of rules, where necessary while leaving enough space for autonomous decision-making. The idea is to localize decisions as close to the source as possible while keeping interoperability and integration standards at a global level, so the mesh components can be easily integrated. In a data mesh, tools can be used to enforce global policies such as GDPR enforcement or access management and also local policies where each domain sets its own policies for their data products such as access control or data retention.
- Last principle is about of doing all that with out compromising security, privacy, making sure that data is discoverable, when we blend different domains, we can still make sense of who the customer is. https://www.youtube.com/watch?v=zfFyE3xmJ7I&ab_channel=Confluent For any of these operations to be possible, a data mesh implementation requires a governance model that embraces decentralization and domain self-sovereignty, interoperability through global standardization, a dynamic topology and most importantly automated execution of decisions by the platform. a “common ground” for the whole platform where all data products conform to a shared set of rules, where necessary while leaving enough space for autonomous decision-making. The idea is to localize decisions as close to the source as possible while keeping interoperability and integration standards at a global level, so the mesh components can be easily integrated. In a data mesh, tools can be used to enforce global policies such as GDPR enforcement or access management and also local policies where each domain sets its own policies for their data products such as access control or data retention.