Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open Metadata and Governance with Apache Atlas

3,314 views

Published on

Learn how Apache Atlas is being enhanced to provide a universal open metadata and governance platform for all data processing across the enterprise. With open metadata, multiple metadata repositories, potentially from different vendors, can operate collaboratively to create an enterprise catalog of data that can be located, understood, used and governed. In this talk we will provide a detailed description of the extensions to the type system, new APIs, the connector framework, metadata discovery framework, governance action framework and the inter-operability that we are adding to Apache Atlas. We will show examples of these features in operation. For example, (1) how metadata is discovered and gathered into Apache Atlas, (2) how applications and tools access metadata, (3) how enforcement engines such as Apache Ranger keep synchronized with the latest governance requirements and (4) how to build an adapter to allow other vendor's metadata repositories can exchange metadata with Apache Atlas repositories. We will also explain how these features can be deployed together to support the Hadoop platform, and the enterprise beyond. This session will be presented by Nigel Jones - IBM & Ferd Schapers - ING Chief Information Architect

Speaker:
Nigel Jones, Software Architect, IBM Analytics Group, IBM

Published in: Technology
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yxufevpm } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Open Metadata and Governance with Apache Atlas

  1. 1. Apache Atlas & Open Metadata Dataworks Sydney 2017 Nigel Jones, Software Architect IBM Ferd Scheepers Chief Information Architect ING
  2. 2. 2 Open Metadata and Governance will allow… … metadata to be captured when the data is created, moved with the data and be augmented and processed by any of the vendor tools.
  3. 3. Open Metadata and Governance consists of: 1. Standardized, extensible set of metadata types 2. Metadata exchange APIs and notifications 3. Frameworks for automated governance Open Metadata and Governance will allow you to have: 1. An enterprise data catalogue that lists all of your data, where it is located, its origin (lineage), owner, structure, meaning, classification and quality 2. New data tools (from any vendor) connect to your data catalogue out of the box 3. Metadata being added automatically to the catalogue as new data is created and analysed 4. Subject matter experts collaborating around the data 5. Automated governance processes protect and manage your data 3 What is Open Metadata and Governance?
  4. 4. 4 Positioning of Apache Atlas for Open Metadata Open and Unified Metadata Metadata repository Apache Atlas Metadata repository IBM Metadata repository SAS Open Metadata Repository Service OMRS Open Metadata Access Service OMAS Components defined and being developed by Open Metadata & Governance project Metadata highway
  5. 5. • Apache Atlas provides an open community for developing the reference implementation for open metadata and governance. In essence Apache Atlas delivers 2 main capabilities: • it plays a role of a metadata repository (Graph Database) for a metadata end-user tool • and, it plays the important role of delivering the federated/unified metadata layer across the entire landscape of an enterprise • The software development governance from the Apache Software Foundation (ASF) creates confidence that the technology will be maintained and enhanced as appropriate in an equitable manner. Role of Apache Atlas 5
  6. 6. … because Apache is mostly focused on development and we are missing a governance body for managing the adoption of and compliance to the Open Metadata and Governance standards. We envision the following roles for ODPI: 1. Be an advocate of the Open Metadata and Governance standards, make them visible and their value understood. 2. Facilitate discussions around the Open Metadata and Governance standards evolution, maintenance and development. 3. Test and sign-off compliance of vendor offerings to the Open Metadata and Governance standards. 6 Doing all of this under Apache Atlas flag is not enough…
  7. 7. 1. Hands-on Community members: • ING • IBM • HortonWorks 2. Companies we have had conversations with: • CIBC • SAS • Microsoft • Oracle • Informatica • Waterline • RBC • DBS 7 Who is in ?
  8. 8. 1. Ambition level: • End of September 2017: Open Metadata working demo. • Mid-December November 2017: first version of user access. • Google for Data 2. Next steps: • End of Q2 2018: production ready version of Virtual Data Connector. 8 Timeline and next steps
  9. 9. About Me https://www.linkedin.com/in/nigelljones https://www.twitter.com/planetf1 jonesn@uk.ibm.com@
  10. 10. Objective Why How Excite & Engage Apache Atlas Open Metadata
  11. 11. Atlas has graduated! DOB: 2015-05-05R: 0.8.1
  12. 12. Atlas Architecture Storage Repository Graph  Type System  REST API  Models  UI & Apps Hooks & Bridges https://cwiki.apache.org/confluence/display/ATLAS/Open+Metadata+and+Governance
  13. 13. A reminder of our problem.. And solution Open and Unified Metadata
  14. 14. Extend beyond Hadoop ++
  15. 15. Common Core Data model Data Assets Governance Lineage Glossary Collaboration Models & Reference Data Base Types, Systems & Infrastructure Metadata Discovery https://cwiki.apache.org/confluence/display/ATLAS/Building+out+the+Open+Metadata+Typesystem
  16. 16. Open APIs - OMRS Metadata Highway Adapter Plugin Open Connector Framework https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258803
  17. 17. Open APIs - OMAS OMRS Governance Engine OMAS Glossary OMAS Asset OMAS Information View OMAS ++...... https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258799
  18. 18. OMAS – detail Project List Metadata Service Data/Asset Community Metadata Service Landscape Definition Metadata Service Asset Catalog Metadata Service Classification and Mapping Metadata Service Information View Metadata Service Connector Directory Metadata Service Governance Definitions Metadata Service Information Process Metadata Service Glossary and Taxonomy Metadata Service Asset Metadata Service Discovery Metadata Service Governance Action Metadata Service Roles and Access Metadata Service Models and Schema Metadata Service Connector https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258799
  19. 19. Business metadata Structural metadata for a data store New glossary function for semantic processing EMPNAME EMPNO JOBCODE SALARY EMPLOYEE RECORD Employee Work Location Annual Salary Job Title Employee Id Employee Name Hourly Pay Rate Manager Compensation Plan HAS-A HAS-A HAS-A HAS-A HAS-A HAS-A IS-A IS-A Sensitive IS-A Data 00 3809890 6 7 Lemmie Stage 818928 3082 4 New York 4 27 DataStage Expert 1 45324 300 27 Code St Harlem NY 1 3 https://cwiki.apache.org/confluence/display/ATLAS/Area+3+-+Glossary
  20. 20. Replacing v1 Taxonomy (tech preview) Categories Terms hierarchies Rich Relationships Classifications Glossary https://cwiki.apache.org/confluence/display/ATLAS/Area+3+-+Glossary
  21. 21. Open Discovery Framework Open Framework Plugins characterize data & relationships Updates metadata with results Initial implementation in master https://cwiki.apache.org/confluence/display/ATLAS/Automated+metadata+discovery
  22. 22. Governance Action Framework metadata drives enforcement Classification (tag) based – scalable, glossary driven Access, Masking, Filtering Supports Apache Ranger but open APIs for others Audit,Rights - Exception management, Rights, Privacy (to look at in future) https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=70258801
  23. 23. Open ecosystem https://cwiki.apache.org/confluence/display/ATLAS/Open+Metadata+and+Governance
  24. 24. Summary Open Metadata Enterprise Catalog Discovery Multi Vendor Open, Layered APIs Metadata store integration Open Source & Governance ubiquitous Standard Models
  25. 25. How can I get involved? Discuss: Mailing List Document, Explain: Wiki Report, Design: Jira Face to face Code Vendors! https://cwiki.apache.org/confluence/display/ATLAS/Getting+Involved
  26. 26. Governance & Security BOF Thursday 18:00 C4.7
  27. 27. Backup
  28. 28. VDC End to End https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69407333

×