Downloaded 302 times
Uploaded byDataWorks Summit/Hadoop Summit
Apache Atlas: Tracking dataset lineage across Hadoop components
Apache Atlas provides centralized metadata services and cross-component dataset lineage tracking for Hadoop components. It aims to enable transparent, reproducible, auditable and consistent data governance across structured, unstructured, and traditional database systems. The near term roadmap includes dynamic access policy driven by metadata and enhanced Hive integration. Apache Atlas also pursues metadata exchange with non-Hadoop systems and third party vendors through REST APIs and custom reporters.
In this document
Powered by AI
Introduction to Apache Atlas, enhancing data governance, and tracking dataset lineage across Hadoop components.
Outline of enterprise data governance vision across various systems with focus on transparency, reproducibility, auditability, and consistency.
Timeline of Apache Atlas development from proto-type to GA release, showcasing progress and major milestones.Future enhancements in Apache Atlas with emphasis on dynamic access policies and maintaining a business catalog.
Tracking and integrating metadata and lineage within Hadoop components like Sqoop, Kafka, and Hive for streamlined governance.
Dataflow governance requirements focusing on data provenance, security, and integration with enterprise-level solutions.
Information about demo sessions, tech previews, and upcoming availability of the product.
Open floor for questions and references/resources for further exploration and information on Apache Atlas.
More Related Content
Similar to Apache Atlas: Tracking dataset lineage across Hadoop components
More from DataWorks Summit/Hadoop Summit
Apache Atlas: Tracking dataset lineage across Hadoop components
- 1. Apache Atlas: Tracking datasetlineage across Hadoop components
- 2. 2 © HortonworksInc. 2011 – 2016. All Rights Reserved Disclaimer This document may contain product features and technology directions that are under development, may be under development in the future or may ultimately not be developed. Project capabilities are based on information that is publicly available within the Apache Software Foundation project websites ("Apache"). Progress of the project capabilities can be tracked from inception to release through Apache, however, technical feasibility, market demand, user feedback and the overarching Apache Software Foundation community development process can all effect timing and final delivery. This document’s description of these features and technology directions does not represent a contractual commitment, promise or obligation from Hortonworks to deliver these features in any generally available product. Product features and technology directions are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Since this document contains an outline of general product development plans, customers should not rely upon it when making purchasing decisions.
- 3. 3 © HortonworksInc. 2011 – 2016. All Rights Reserved Speakers Andrew Ahn Governance Director Product Management
- 4. 4 © HortonworksInc. 2011 – 2016. All Rights Reserved Agenda • Atlas Overview • Near term roadmap • Cross Component Lineage • Questions
- 5. 5 © HortonworksInc. 2011 – 2016. All Rights Reserved Apache Atlas Overview
- 6. 6 © HortonworksInc. 2011 – 2016. All Rights Reserved STRUCTURED UNSTRUCTURED Vision - Enterprise Data Governance Across Platfroms TRADITIONAL RDBMS METADATA MPP APPLIANCES Project 1 Project 5 Project 4 Project 3 Metadata Project 6 DATA LAKE GOAL: Provide a common approach to data governance across all systems and data within the enterprise Transparent Governance standards and protocols must be clearly defined and available to all Reproducible Recreate the relevant data landscape at a point in time Auditable All relevant events and assets but be traceable with appropriate historical lineage Consistent Compliance practices must be consistent
- 7. 7 © HortonworksInc. 2011 – 2016. All Rights Reserved Ready for Trusted Governance OPERATIONS SECURITY GOVERNANCE STORAGE STORAGE Machine Learning Batch StreamingInteractive Search GOVERNANCE YA R N D A T A O P E R A T I N G S Y S T E M Data Management along the entire data lifecycle with integrated provenance and lineage capability Modeling with Metadata enables comprehensive data lineage through a hybrid approach with enhanced tagging and attribute capabilities Interoperable Solutions across the Hadoop ecosystem, through a common metadata store
- 8. 8 © HortonworksInc. 2011 – 2016. All Rights Reserved DGI* Community becomes Apache Atlas May 2015 Proto-type Built Apache Atlas Incubation DGI group Kickoff Feb 2015 Dec 2014 July 2015 HDP 2.3 Foundation GA Release First kickoff to GA in 7 months Global Financial Company * DGI: Data Governance Initiative Faster & Safer Co-Development driven by customer use cases
- 9. 9 © HortonworksInc. 2011 – 2016. All Rights Reserved Apache Atlas: Metadata Services • Cross- component dataset lineage. Centralized location for all metadata inside HDP • Single Interface point for Metadata Exchange with platforms outside of HDP • Business Taxonomy based classification. Conceptual, Logical And Technical Apache Atlas Hive Ranger Falcon Sqoop Storm Kafka Spark NiFi
- 10. 10 © HortonworksInc. 2011 – 2016. All Rights Reserved Big Data Management Through Metadata Management Scalability Many traditional tools and patterns do not scale when applied to multi-tenant data lakes. Many enterprise have silo’d data and metadata stores that collide in the data lake. This is compounded by the ability to have very large windows (years). Can traditional EDW tools manage 100 million entities effectively with room to grow ? Metadata Tools Scalable, decoupled, de-centralized manage driven through metadata is the only via solution. This allows quick integration with automation and other metamodels Tags for Management, Discovery and Security Proper metadata is the foundation for business taxonomy, stewardship, attribute based security and self-service.
- 11. 11 © HortonworksInc. 2011 – 2016. All Rights Reserved Apache Atlas High Level Architecture Type System Repository Search DSL Bridge Hive Storm Falcon Others REST API Graph DB Search Kafka Sqoop Connectors MessagingFramework
- 12. 12 © HortonworksInc. 2011 – 2016. All Rights Reserved Near Term Roadmap: Summer 2016
- 13. 13 © HortonworksInc. 2011 – 2016. All Rights Reserved Dynamic Access Policy Driven by metadata
- 14. 14 © HortonworksInc. 2011 – 2016. All Rights Reserved Business Catalog Breadcrumbs for taxonomy context path Contents at taxonomy context
- 15. 15 © HortonworksInc. 2011 – 2016. All Rights Reserved Hadoop Cross Component Data Lineage
- 16. 16 © HortonworksInc. 2011 – 2016. All Rights Reserved Sqoop Teradata Connector Apache Kafka Atlas: Tracks Metadata + Lineage in one place Custom Activity Reporter Metadata Repository RDBMS
- 17. 17 © HortonworksInc. 2011 – 2016. All Rights Reserved Technical and Logical Metadata Exchange Knowledge Store Atlas REST API Structured Unstructured Files: XML / JSON 3rd Party Vendors Custom Reporter Non-Hadoop
- 18. 18 © HortonworksInc. 2011 – 2016. All Rights Reserved Hive Integration: Model for integration Apache Atlas Hive Bridge (Client) Hive Hook (Post-execution) REST API
- 19. 19 © HortonworksInc. 2011 – 2016. All Rights Reserved HDF: Dataflow Governance Solution
- 20. 20 © HortonworksInc. 2011 – 2016. All Rights Reserved HORTONWORKS CONFIDENTIAL & PROPRIETARY INFORMATION Dataflow Security Use case Requirements Accelerated Data Collection: An integrated, data source agnostic collection platform Increased Security and Unprecedented Chain of Custody: Secure from source to storage with high fidelity data provenance The Internet of Any Thing (IoAT): A Proven Platform for the Internet of Things http://hortonworks.com/hdf/
- 21. 21 © HortonworksInc. 2011 – 2016. All Rights Reserved Enterprise Grade Governance Dataflow Solution Filtered Metadata • HDP Taxonomy • Centrallized Metadata Repository • Downstream HDP Impacts • Cross component lineage • 3rd Party integration • Guaranteed Delivery • Data Buffering • Prioritized Queueing • Flow specific QoS • Visual Command & Control Months Lineage Years Lineage Reference Taxonomy (Tags) Event level versus Dataset level HDF - NiFI Operation Control Maximum Fidelity Event Level HDP – Atlas Governance Management Medium / Low Fidelity Dataset Level
- 22. 23 © HortonworksInc. 2011 – 2016. All Rights Reserved Demo • Tutorial • Atlas Tour • Sqoop Lineage • Kafka / Storm Linage
- 23. 24 © HortonworksInc. 2011 – 2016. All Rights Reserved Availability: - Tech Preview VMs: May 2016 - GA Release: Summer 2016
- 24. 25 © HortonworksInc. 2011 – 2016. All Rights Reserved Questions ?
- 25. 26 © HortonworksInc. 2011 – 2016. All Rights Reserved Reference
- 26. 27 © HortonworksInc. 2011 – 2016. All Rights Reserved Online Resources VM: https://s3.amazonaws.com/demo-drops.hortonworks.com/HDP- Atlas-Ranger-TP.ova —> Download Public Preview VM Tutorial: https://github.com/hortonworks/tutorials/tree/atlas-ranger- tp/tutorials/hortonworks/atlas-ranger-preview Blog: http://hwxjojo.wpengine.com/blog/the-next-generation-of- hadoop-based-security-data-governance/ (this is giving an error, right now) Learn More: http://hortonworks.com/solutions/atlas-ranger- integration/
- 27. 32 © HortonworksInc. 2011 – 2016. All Rights Reserved Thank You
Editor's Notes
- #2 TALK TRACK Data is powering successful clinical care and successful operations. [NEXT SLIDE]
- #7 6
- #8 TALK TRACK Open Enterprise Hadoop enables trusted governance, with: Data lifecycle management along the entire lifecycle Modeling with metadata, and Interoperable solutions that can access a common metadata store. [NEXT SLIDE] SUPPORTING DETAIL Trusted Governance Why this matters to our customers: As data accumulates in an HDP cluster, the enterprise needs governance policies to control how that data is ingested, transformed and eventually retired. This keeps those Big Data assets from turning into big liabilities that you can’t control. Proof point: HDP includes 100% open source Apache Atlas and Apache Falcon for centralized data governance coordinated by YARN. These data governance engines provide those mature data management and metadata modeling capabilities, and they are constantly strengthened by members of the Data Governance Initiative. The Data Governance Initiative (DGI) is working to develop an extensible foundation that addresses enterprise requirements for comprehensive data governance. The DGI coalition includes Hortonworks partner SAS and customers Merck, Target, Aetna and Schlumberger. Together, we assure that Hadoop: Snaps into existing frameworks to openly exchange metadata Addresses enterprise data governance requirements within its own stack of technologies Citation: “As customers are moving Hadoop into corporate data and processing environments, metadata and data governance are much needed capabilities. SAS participation in this initiative strengthens the integration of SAS data management, analytics and visualization into the HDP environment and more broadly it helps advance the Apache Hadoop project. This additional integration will give customers better ability to manage big data governance within the Hadoop framework,” said SAS Vice President of Product Management Randy Guard.” | http://hortonworks.com/press-releases/hortonworks-establishes-data-governance-initiative/
- #9 How fast ? 7 months !
- #12 Apache Atlas is the only open source project created to solve the governance challenge in the open. The founding members of the project include all the members of the data governance initiative and others from the Hadoop community. The core functionality defined by the project includes the following: Data Classification – create an understanding of the data within Hadoop and provide a classification of this data to external and internal sources Centralized Auditing – provide a framework to capture and report on access to and modifications of data within Hadoop Search & Lineage – allow pre-defined and ad hoc exploration of data and metadata while maintaining a history of how a data source or explicit data was constructed Security and Policy Engine – implement engines to protect and rationalize data access and according to compliance policy
- #17 Show – clearly identify customer metadata. Change Add customer classification example – Aetna – make the use case story have continuity. Use DX procedures to diagnosis ** bring meta from external systems into hadoop – keep it together
- #18 Show – clearly identify customer metadata. Change Add customer classification example – Aetna – make the use case story have continuity. Use DX procedures to diagnosis ** bring meta from external systems into hadoop – keep it together
- #19 Specify Metrics – Time / Success /user /etc… Contrast with Ranger plug-in – pre execute
- #23 Show – clearly identify customer metadata. Change Add customer classification example – Aetna – make the use case story have continuity. Use DX procedures to diagonsis ** bring meta from external systems into hadoop – keep it together
- #32 The Data Governance Framework will enable Freddie Mac to design Data Index tool from the ground up for scalability, security and reliability