More Related Content
![[DSC Europe 22] Lakehouse architecture with Delta Lake and Databricks - Draga...](https://cdn.slidesharecdn.com/ss_thumbnails/draganberic-lakehousearchitecturewithdeltalakeanddatabricks-221130080712-6e817e95-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Data weekender4.2 azure purview erwin de kreuk
![[DSC Europe 25] Ivan Peric - Intelligence Swarm Logic and Techno-Functional M...](https://cdn.slidesharecdn.com/ss_thumbnails/7my7c97fsduiccadgavw-2-251212103249-5a03f7c6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Miodrag Pesovic & Vladislav Radonjic - Federated Data Archite...](https://cdn.slidesharecdn.com/ss_thumbnails/gsbe3y5it5uhndi4e08e-1-251212103249-f1008e0c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jovan Bogicevic - Legacy to AI-Driven Defense: Transforming D...](https://cdn.slidesharecdn.com/ss_thumbnails/rsarluadt563hntyfc8q-3-251211083849-3e7bc4c0-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jon Dajci - Bridging TradFi and DeFi: Building the Future of ...](https://cdn.slidesharecdn.com/ss_thumbnails/fqmhfvlbqhkihjvqvhmu-7-251211083849-6af7e325-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Bassam Maharmeh - Artificial Intelligence: Opportunities and ...](https://cdn.slidesharecdn.com/ss_thumbnails/thhfmr2fqpawzj7hsjpg-5-251211083048-2c23204f-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dusan Nesic - Securing Tomorrow’s Infrastructure: Why Cyber-P...](https://cdn.slidesharecdn.com/ss_thumbnails/qikbszfftyowjm2q6duw-1-251211083848-8f2ead6b-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Branko Urosevic -Rethinking Financial Talent: Integrating Cod...](https://cdn.slidesharecdn.com/ss_thumbnails/8jjrus8ttko6qj64f58f-3-251212103250-642c6374-thumbnail.jpg?width=640&height=640&fit=bounds)
Data weekender4.2 azure purview erwin de kreuk
- 1. InSpark Azure Purview Microsoft's answerto Data Governance and Data Lineage Erwin de Kreuk @erwindekreuk https://erwindekreuk.com
- 2. InSpark We help organizations acceleratingtheir digital transformation with impactful Microsoft solutions & expertise We Are InSpark
- 3. InSpark Unified data governanceto maximize the business value of data Azure Purview
- 4.
- 5. InSpark Data governance isbecoming increasingly interdisciplinary DISCOVERY COMPLIANCE ChiefDataOfficer
- 6. InSpark Data governance isbecoming increasingly interdisciplinary What data do I have? Where did the data originate? Can I trust it? What’s my exposure to risk? Is my usage compliant? How do I control access & use? What is required by regulation X? Data Compliancy Data Access Data Security Data Privacy Data Management Data Stewardship Data Engineering Data Science Data Analysist ChiefDataOfficer
- 7. InSpark Overcome operational silos Manage growing datalandscape Elements of successful data governance Increase data agility Comply with industry regulations
- 8. InSpark Data Map Multicloud On-prem Data Insights AzurePurview Data Catalog SaaS Data Map Automate and manage metadata at scale Elastic data map Resource sets Scans and ingestion Data lineage Data Catalog Enable effortless discovery for data consumers Catalog search Business glossary Asset normalization Data Insights Assess data usage across your organization
- 9. InSpark Unified data governanceto maximize the business value of data Azure Purview Reimagine data governance in the cloud Set the foundation for effective data governance Maximize business value of data for data consumers Gain insight into data use across the estate
- 10. InSpark Manage andgovern operational, transactional and analytical data Cloud-native, purpose-built service to address discovery and compliance needs Fully managed, serverless, PaaS service Eliminate manual, ad-hoc and homegrown solutions Reimagine data governance in the cloud
- 11. InSpark Automate discoveryof data in on- premises, multicloud and SaaS sources Classify data at scale to specify sensitivity, compliance, industry, business and company-specific value Know where data came from and what was derived from it with data lineage Set the foundation for effective data governance
- 12. InSpark Connect businessand technical data analysts, data scientists, and data engineers to a trusted data catalog Enable users to quickly find data and view its lineage and sensitivity Deliver a curated and consistent glossary of business terms and definitions Maximize business value of data for data consumers
- 13. InSpark Understand ata glance how data is being created and used across your data estate Visually assess the state of data assets, scans, business glossary and sensitive data Gain insight into data use across the estate
- 14. InSpark Azure Purview Features AzurePurview Azure Purview Platform Azure Purview Studio Automated Scanning & Classification • Dedicated per customer on shared infra • Provisioned default capacity with option to add-on capacity Data Map • Serverless, pay per use • Includes connectors, scanning of sources, processing into data assets, lineage capture, classification • Search, browse, asset details • Automated meta-data and lineage extraction • Automated classification based on content inspection • Private Endpoint • Management center On-prem & Multi-cloud Operational, Analytical, SaaS Azure Purview Catalog included with Platform (C0) Power BI SQL Server on-prem Azure Synapse Azure Data Services M365 Compliance Cen Open APIs (Apache Atlas 2.0)
- 15. InSpark Azure Purview Features AzurePurview Azure Purview Platform Azure Purview Studio Azure Purview Catalog (C1) Automated Scanning & Classification • Dedicated per customer on shared infra • Provisioned default capacity with option to add-on capacity Data Map • Serverless, pay per use • Includes connectors, scanning of sources, processing into data assets, lineage capture, classification • Search, browse, asset details • Automated meta-data and lineage extraction • Automated classification based on content inspection • Private Endpoint • Management center On-prem & Multi-cloud Operational, Analytical, SaaS • Business Glossary templates • Lineage visualization & workflows Azure Purview Catalog included with Platform (C0) Data Producers & Consumers Open APIs (Apache Atlas 2.0) Power BI SQL Server on-prem Azure Synapse Azure Data Services M365 Compliance Cen
- 16. InSpark Azure Purview Features AzurePurview Azure Purview Platform Azure Purview Studio Azure Purview Catalog (C1) Automated Scanning & Classification • Dedicated per customer on shared infra • Provisioned default capacity with option to add-on capacity Data Map • Serverless, pay per use • Includes connectors, scanning of sources, processing into data assets, lineage capture, classification • Search, browse, asset details • Automated meta-data and lineage extraction • Automated classification based on content inspection • Private Endpoint • Management center On-prem & Multi-cloud Operational, Analytical, SaaS Azure Purview Data Insights (D1) • Business Glossary templates • Lineage visualization & workflows Azure Purview Catalog included with Platform (C0) • Catalog Insights (Asset, Scan, Glossary) • Sensitive Information Types & Labeling insights Data Producers & Consumers Data Officers & Security Officers Open APIs (Apache Atlas 2.0) Power BI SQL Server on-prem Azure Synapse Azure Data Services M365 Compliance Cen
- 17. InSpark Azure Purview –Access Control Collection Admins • Can edit the collection, its details, and add subcollections. • Can also add data curators, data readers, and other Purview roles to a collection scope. • Collection admins that are automatically inherited from a parent collection can't be removed.
- 18. InSpark Azure Purview –Access Control Data Source Admins Collection Admins • can manage data sources and data scans
- 19. InSpark Azure Purview –Access Control Data Source Admins Data Curator Collection Admins • can perform create, read, modify, and delete actions on catalog data objects and establish relationships between objects.
- 20. InSpark Azure Purview –Access Control Data Source Admins Data Reader Data Curator Collection Admins • can access but not modify catalog data objects.
- 21. InSpark Azure Purview –Access Control Data Source Administrator Data Reader Data Curator
- 22. InSpark Azure Purview -Pricing • Capacity Unit • €0.353 per 1 Capacity Unit Hour • 1 Capacity Unit supports requests of up to 25 data map operations per second and includes storage of up to 2 GB of metadata about data assets. Azure Purview Data Map
- 23. InSpark Azure Purview -Pricing • Power BI Online • Free in Preview • SQL Server On Prem • Free in Preview • Other Data Sources • Free in Preview • €0.540 per 1 vCore Hour Azure Purview Data Map Scanning and Classification
- 24. InSpark Azure Purview -Pricing • Power BI Online • Free in Preview • SQL Server On Prem • Free in Preview • Other Data Sources • Free in Preview • €0.540 per 1 vCore Hour Azure Purview Data Map Scanning and Classification
- 25. InSpark Azure Purview –Pricing-Example • Resource Set • €0.18 per 1 vCore Hour Azure Purview Data Map Scanning and Classification Other Features
- 26. InSpark Azure Purview -Pricing Azure Purview Data Map 1 CPU x €0.353 X 730 hours € 257,69 Scanning and Classification 4 scans x 2 hours x 32 VCore x €0.540 per vCore per hour € 138,24 Other Features 4 scans x 1 hour x €0.18 per vCore per hour € 0,72 €396,65
- 27. InSpark Azure Purview StudioUpdates Accounts Notifications Feedback Metrics Search Bar Usefull Links Recently Accessed Entities Search Bar Key Actvities
- 28. InSpark • Quick Actions,recently accessed items, owned Items, search bar and Documentation Azure Purview Studio - Activity hubs • Create collections, register data sources, setup Scans, Integration runtime • Manage Glossary Items, search, manage terms templates and custom attributes, import and export Terms using csv • Insights on your data • Meta Data Management Security, ADF and data share Connections
- 29.
- 30.
- 31.
- 32. InSpark Purview Data Map Unifyand make data meaningful Automated metadata scanning and lineage identification of hybrid data stores 100+ built-in and custom classifiers Microsoft Information Protection sensitivity labels
- 33. InSpark Purview Data Map Automated metadata scanning and lineage identification of hybrid data stores 200+ built-in and custom classifiers Microsoft Information Protection sensitivity labels Unify and make data meaningful
- 34. InSpark Azure Purview DataCatalog Enable effortless discovery Semantic search and browse Business glossary and workflows Data lineage with sources, owners, transformations, and lifecycle
- 35. InSpark Azure Purview DataCatalog Unify and make data meaningful
- 36. InSpark Unify and makedata meaningful Azure Purview Data Catalog •Azure Data Factory lineage •Azure Data Share lineage •Azure Synapse Analytics lineage •Cassandra lineage •Erwin lineage •Looker lineage •Google BigQuery lineage •Oracle lineage •Power BI lineage •SAP S/4HANA lineage •SAP ECC lineage •Spark lineage •SQL Server Integration Services lineage •Teradata lineage
- 37. InSpark Unify and makedata meaningful Azure Purview Data Catalog
- 38. InSpark Insights Reports onAssets, Scans, Glossary, Classification, and Labeling Get a bird’s-eye view of sensitive data
- 39.
- 40. InSpark Register and Scana Power BI Tenant Discover data registered and scanned by Azure Purview
- 41. InSpark Register and Scana Power BI Tenant Discover data registered and scanned by Azure Purview Allow service principals to use read-only Power BI admin APIs Enhance admin APIs responses with detailed metadata Enhance admin APIs responses with DAX and mashup expressions (Preview)Enabled for a subset of the organization
- 42. InSpark Register and Scana Power BI Tenant Discover data registered and scanned by Azure Purview Allow service principals to use read-only Power BI admin APIs Enhance admin APIs responses with etailed metadata Enhance admin APIs responses with DAX and mashup expressions (Preview)
- 43.
- 44. InSpark Integrate Azure Purviewin Azure Synapse Analytics Discover data registered and scanned by Azure Purview
- 45. InSpark Integrate Azure Purviewin Azure Synapse Analytics Discover data registered and scanned by Azure Purview
- 46. InSpark Integrate Azure Purviewin Azure Synapse Analytics Discover data registered and scanned by Azure Purview
- 47. InSpark Integrate Azure Purviewin Azure Synapse Analytics Discover data registered and scanned by Azure Purview
- 48.
- 49.
- 50.
- 51. InSpark Roadmap /Preview features Data Quality Data Privacy Master Data Management Data Policy Microsoft Defender for Cloud
- 52. InSpark Policy in AzurePurview What's New in Azure Purview at Microsoft Ignite 2021 - Microsoft Tech Community
- 53.
- 54.
- 55.
Editor's Notes
- #2 Hallo and Welcome to my session about Azure Purview My name is Erwin de Kreuk and I’m working as a Lead Data and AI for InSpark a Microsoft Partner in the Netherlands
- #3 Hallo and Welcome to my session about Azure Purview My name is Erwin de Kreuk and I’m working as a Lead Data and AI for InSpark a Microsoft Partner in the Netherlands
- #5 Azure Purview is a unified data governance service. During this session I will explain what Azure Purview is. The position of Azure Purview within your Data Estate And how it works with some practical examples If you have questions, please feel free to ask them
- #6 History Blue Talon June 2019 With Azure Purview Microsoft has now his own Cloud Native Service for Data Governance and Data Lineage. I'm curious what the future will bring, but also which position it will take compared to Colibra / Informatica / AWS Glue Data Catalog or other Data Governance products
- #7 As we all know Data Governance is becoming more and more becoming increasingly interdisciplinary. A chief data officer (CDO) is a corporate officer who is responsible for enterprise-wide governance and utilization of information as an asset, via data processing, analysis, data mining, information trading and other means. He will be one of the users who will use Azure Purview to get answers On what kind of do I have within my Data Estate Where is the data coming from but also I can trust the data. But also compliance is getting more and more important with all the required regulations from the local government or industries. F.E ISO and NEN certifications. Besided these questions the CDO wants to have also answers based On what are the risk to exposure mu data How can we control the access and use of data and compliant is our data.
- #8 As we all know Data Governance is becoming more and more becoming increasingly interdisciplinary. A chief data officer (CDO) is a corporate officer who is responsible for enterprise-wide governance and utilization of information as an asset, via data processing, analysis, data mining, information trading and other means. He will be one of the users who will use Azure Purview to get answers On what kind of do I have within my Data Estate Where is the data coming from but also I can trust the data. But also compliance is getting more and more important with all the required regulations from the local government or industries. F.E ISO and NEN certifications. Besided these questions the CDO wants to have also answers based On what are the risk to exposure mu data How can we control the access and use of data and compliant is our data.
- #9 The following elements can lead to a successful data governance which is one of the key components in a modern Data Estate: You need to have control on your growing data landscape You want to Overcome operational silos A data silo is a collection of data held by one group that is not easily or fully accessible by other groups. ... Finance, administration, HR, and other departments need different information to do their work, and those individual collections of often overlapping-but-inconsistent data are in separate silos You want Increase the flexibility/agility of your data And You want make sure you comply with all different industry regulations and local government regulations. Azure Purview can help you with these elements
- #10 Azure Purview organizes metadata that enables your organization to break down silos and derive meaning from data. Once data can be understood and annotated, it then lends itself to several applications – Data map where automate and manage metadata at scale Data catalog to Discover and search for data Data insights. To get an overview of the data in our Data Estate This’s what Azure Purview currently has to offer In the future, policies, privacy, quality and master data management will follow.
- #11 There are 4 pilars which helps you to maximize the business value of data in your organization Data Governance Set the Foundation Create Business Value for the consumers And of course, insights should not be missing
- #12 Key features of Reimagine data governance in the cloud Cloud Native Managed Serverless PaaS
- #13 Key features for the foundation are Automate and Discover data of different sources Classify data to specify sensitivity Know where your data is coming from
- #14 Key features to maximize the business values Connect the different roles within your organization to a trusted data catalog Enable them to quickly find this data
- #15 Key features to gain insights Understand at a glance how data is being created and used across your data estate Visually the state of data assets, scans, business glossary and sensitive data
- #16 Datasource Power BI, SQL Sever on-prem, Azure Data Services including Synapse, Cosmos DB & Storage, Non-Microsoft systems including SAP ECC, SAP S4 HANA & Teradata, Multi-cloud systems including AWS S3 With Purview Platform: Automate scanning and classification of multicloud, SaaS, on-prem data. 25 plus out of box connectors and file formats supported Modernize homegrown catalogs built on opensource technology with Purview using Apache Atlas APIs supported out-of-the-box Get catalog features (C0 Tier) for FREE included with Purview platform: Search and browse Empower business and technical data analysts via a catalog to find and interpret data. Power data scientists and engineers with business context to drive BI, Analytics, AI and ML initiatives Automated metadata and lineage extraction Enrich the business value of data with technical, business and semantic metadata Scale understanding of data with automated, fully managed, serverless metadata management capability Leverage support of Apache Atlas’s open-source Lineage APIs to push lineage information into the Purview Data Map. Analyze impact of changes to data and understand dependencies visually.
- #17 Azure Purview Catalog (C1 Tier) includes the following in addition to the free features included with the platform: Business Glossary Deliver a curated and consistent understanding of business terms and definitions. Import existing glossary terms from existing data dictionaries easily. Also add ability to define custom attributes for the glossary terms and create templates for different domains like ‘Finance’, ‘Sales’ etc. Lineage views Ensure data provenance with a visual representation of owners, sources, transformation, and lifecycle Built-in integrations with solutions to automatically extract lineage such as Synapse Analytics, Azure Data Factory, Azure Data Share etc.
- #18 Data Insights (D1 Tier) provides a bird’s eye view of your data landscape intended to help users such as Chief Data Officers quickly understand their data estate at large and gain key insights such as where sensitive data resides. It includes: Catalog insights: Asset Insights: Quickly see where all your data resides across a range of data sources Scan Insights: Success/failures/cancellations over a period Glossary Insights: Quickly understand changes made to the glossary over time and assess how much coverage glossary has over your data map. Sensitive data insights Simplify compliance risk assessment across all your operational and transactional data sources. Assess risk and derive audit trails of data qualified by sensitivity and business relevance.
- #19 Collection admins - can edit the collection, its details, and add subcollections. They can also add data curators, data readers, and other Purview roles to a collection scope. Collection admins that are automatically inherited from a parent collection can't be removed. Data source admins - can manage data sources and data scans. Data curators - can perform create, read, modify, and delete actions on catalog data objects and establish relationships between objects. Data readers - can access but not modify catalog data objects.
- #20 Collection admins - can edit the collection, its details, and add subcollections. They can also add data curators, data readers, and other Purview roles to a collection scope. Collection admins that are automatically inherited from a parent collection can't be removed. Data source admins - can manage data sources and data scans. Data curators - can perform create, read, modify, and delete actions on catalog data objects and establish relationships between objects. Data readers - can access but not modify catalog data objects.
- #21 Collection admins - can edit the collection, its details, and add subcollections. They can also add data curators, data readers, and other Purview roles to a collection scope. Collection admins that are automatically inherited from a parent collection can't be removed. Data source admins - can manage data sources and data scans. Data curators - can perform create, read, modify, and delete actions on catalog data objects and establish relationships between objects. Data readers - can access but not modify catalog data objects.
- #22 Collection admins - can edit the collection, its details, and add subcollections. They can also add data curators, data readers, and other Purview roles to a collection scope. Collection admins that are automatically inherited from a parent collection can't be removed. Data source admins - can manage data sources and data scans. Data curators - can perform create, read, modify, and delete actions on catalog data objects and establish relationships between objects. Data readers - can access but not modify catalog data objects.
- #23 When deploying an Azure Purview Account on or after August 18th, 2021 you now can also assign roles bases on Collection So as you can see in the Example you can restricted people to see data in the Collection Assets Revenue. How this all works, I will show that I a later demo
- #24 4 capacity units are only for some subscriptions types
- #27 Resource sets are a built-in feature of the Data Map used to optimize the storage and search of data assets associated with partitioned files in data lakes. A customer can turn on Advanced Resource Sets to gain additional enrichment of resource set dataset properties and for customization of how Azure Purview groups these assets. Note: Resource set processing is run every 12 hours for all the systems configured for scanning with Advanced Resource Sets toggle enabled. Regardless of if "Advanced Resource Sets" is on or off, the data catalog will have resource set assets representing partitioned data.
- #28 Scanning is done once a week
- #29 A single, centralized place that provides unified experience for data producers, data consumers, data & security officers
- #30 Home Quick Actions, recently accessed items, owned Items, search bar and Documentation Sources Create collections, register data sources, setup Scans, Integration runtimeGlossary Manage Glossary Items, search, manage terms templates and custom attributes, import and export Terms using csv Insights Insights on your data Management Center Meta Data Management Security, ADF and data share Connections
- #33 Demo Activity Hubs Home Page Tabs
- #34 Table view-Map View Amazon RDS PostgressSQL and SQL in Preview but also a default PostgressSQL and Salesfore are now available for scanning
- #35 Scan ADLS Define Scope
- #37 All Source are categorized Pay Attention when you have enabled Private endpoint that you can access selected networks/sources
- #40 Intended to help users such as Chief Data Officers quickly understand their data estate at large and gain key insights such as where sensitive data resides Asset Insight Understand distribution of data assets across a range of data sources & environments Scan Insight Number of successful, failed and cancelled scans over time Glossary Insights Understand changes made to business terms and assess how much coverage glossary has over the data map Classifications Insights Understand what sensitive data exists across the data estate from various lens Sensitivity Labels Insights Understand what sensitivity labels have been applied across the data estate File Extensions Insights Recently scanned files based on their extensions Reports on Assets, Scans, Glossary, Classification, and Labeling
- #43 You need make sure that your Azure Purview Account as permission to read the PowerBI Tenant. You need to be a Power BI Admin to see the tenant settings page. First of all create a Security Group and add your Purview Account as a Member Then you need to add this Security Group to the tenant setting Allow service principals to use read-only Power BI admin APIs to allow Purview to scan your PowerBI Metadata you need to enable Enhance admin APIs responses with detailed metadata Enhance admin APIs responses with DAX and mashup expressions (Preview): See this documentation for details. Purview support for the functionality enabled by this switch will be available in the future.
- #44 Make sure that before you start scanning your Power BI Dataset and to get the metadata, you must schedule a refresh in the powerbi service.
- #46 I immediately thought back to a keynote from Pass Summit 2015, in which , Microsoft's new vision immediately became clear Walk with your head in the Cloud and your feet on the ground. I don’t why but it just came up. But it makes it clear that Microsoft is now busy to create a Unified experience for his customers. Where Azure Synapse is the heart and with the link to Azure Purview and Azure Cosmos DB and DataVerse is getting even more simple.
- #47 Once you created this connection you directly search with the Azure Purview catalog Data Lineage will be enabled also when connecting your Purview Account Azure Purview drops lineage if the source or destination uses an unsupported data storage system.
- #48 You may see below warning if you have the privilege to read Purview role assignment information and the needed role is not granted. To make sure the connection is properly set for the pipeline lineage push, go to your Purview account and check if Purview Data Curator role is granted to the Synapse workspace's managed identity. If not, manually add the role assignment.
- #49 Once you created this connection you directly search with the Azure Purview catalog And for 2 weeks your Data Lineage will be enabled also when connecting your Purview Account Azure Purview drops lineage if the source or destination uses an unsupported data storage system.
- #52 Source Collection Scan + Scan Rule set + Custom File Type Schedule Search catalog cities Lineage Browse Assets Edit/Overview/Lineage/Contacts Show Insights Show Synapse Integration
- #53 Master Data Management Integration with Profisee Prioritize security actions by data sensitivity (powered by Azure Purview) (in preview)
- #54 What's New in Azure Purview at Microsoft Ignite 2021 - Microsoft Tech Community