Uploaded byDataWorks Summit
Driving Enterprise Data Governance for Big Data Systems through Apache Falcon
The document provides an overview of Apache Falcon, focusing on its capabilities for managing data pipelines in the Hadoop ecosystem. Key features include centralized lifecycle management, data governance, monitoring, and audit compliance, as well as support for complex data processing and replication. It also discusses Falcon's architecture, basic concepts, and new features for disaster recovery and cloud replication.
More Related Content
Similar to Driving Enterprise Data Governance for Big Data Systems through Apache Falcon
More from DataWorks Summit
Driving Enterprise Data Governance for Big Data Systems through Apache Falcon
- 1. Page1 © HortonworksInc. 2011 – 2014. All Rights Reserved Apache Falcon Hadoop Data Governance Hortonworks. We do Hadoop.
- 2. Page2 © HortonworksInc. 2011 – 2014. All Rights Reserved Venkatesh Seetharam Architect, Data Management Hortonworks Inc. PMC, Apache Falcon PMC, Apache Knox Proposed Apache Atlas
- 3. Page3 © HortonworksInc. 2011 – 2014. All Rights Reserved Agenda Overview Components Features Governance
- 4. Page4 © HortonworksInc. 2011 – 2014. All Rights Reserved Motivation for Apache Falcon
- 5. Page5 © HortonworksInc. 2011 – 2014. All Rights Reserved Simple Data Pipeline… Page 5 HDFS YARN Landing Materialized Views Oozie Workflow source_db.raw_input_table Partition 2014-01-01-10 Partition 2014-01-01-12 Partition 2014-01-01-12 Partition N Pig JobHive Job source_db.input_table Partition 2014-01-01-10 Partition 2014-01-01-12 Partition N
- 6. Page6 © HortonworksInc. 2011 – 2014. All Rights Reserved Add Data Management Capability to the Pipeline Page 6 HDFS YARN Landing Materialized Views Oozie Workflow source_db.raw_input_table Partition 2014-01-01-10 Partition 2014-01-01-12 Partition 2014-01-01-12 Partition N Pig JobHive Job source_db.input_table Partition 2014-01-01-10 Partition 2014-01-01-12 Partition N Frequent Feeds Late Data Arrival Replication Rentention Archival Exception Handling Lineage Audit Monitoring
- 7. Page7 © HortonworksInc. 2011 – 2014. All Rights Reserved Pipeline Becomes Considerably More Complex Oozie Workflow Pig JobHive Job Results in Many Complex Oozie Workflows Frequent Feeds Late Data Arrival Replication RententionArchival Exception Handling Lineage AuditMonitoring Data Management Requirements
- 8. Page8 © HortonworksInc. 2011 – 2014. All Rights Reserved Introduction to Apache Falcon
- 9. Page9 © HortonworksInc. 2011 – 2014. All Rights Reserved Falcon Overview Centrally Manage Data Lifecycle – Centralized definition & management of pipelines for data ingest, process & export Business Continuity & Disaster Recovery – Out of the box policies for data replication & retention – End to end monitoring of data pipelines Address audit & compliance requirements – Visualize data pipeline lineage – Track data pipeline audit logs – Tag data with business metadata The data traffic cop
- 10. Page10 © HortonworksInc. 2011 – 2014. All Rights Reserved Complicated Pipeline Simplified with Apache Falcon Falcon Generates and Instruments Oozie Workflows Falcon Engine Lineage AuditMonitoring Frequent Feeds Late Data Arrival Replication RententionArchival Exception Handling Frequent Feeds Submit & Schedule Falcon Entities Cluster Cluster Feed Feed Feed Process
- 11. Page11 © HortonworksInc. 2011 – 2014. All Rights Reserved Falcon Architecture Centralized Falcon Orchestration Framework Hadoop ecosystem tools Falcon Server JMS API & UI AMBARI HDFS / Hive Oozie Entity Specs Scheduled Jobs Process Status MapRed / Pig / Hive / Sqoop / Flume / DistCP Data stewards + Hadoop admins
- 12. Page12 © HortonworksInc. 2011 – 2014. All Rights Reserved Falcon Basic Concepts • Cluster: Represents the “interfaces” to a Hadoop cluster • Feed: Defines a “dataset” File, Hive Table or Stream • Process: Consumes feeds, invokes processing logic & produces feeds Page 12 All these put together represent ‘Data Pipelines’ in Hadoop CLUSTER FEED aka DATASET PROCESS INPUT TO CREATES
- 13. Page13 © HortonworksInc. 2011 – 2014. All Rights Reserved Data Pipeline: Definition • Flexible based pipeline specification –JAXB / JSON / JAVA / XML –Modular - Clusters, feeds & processes defined separately and then linked together –Easy to re-use across multiple pipelines • Out of the box policies –Predefined policies for replication, late data handling & eviction –Easily customization of policies • Extensible –Plug in external solutions at any step of the pipeline –Eg. Invoke third party data obfuscation components
- 14. Page14 © HortonworksInc. 2011 – 2014. All Rights Reserved Flexibility in Processing Common types of processing engines can be tied to Falcon processes Oozie workflows Pig scripts HQL scripts
- 15. Page15 © HortonworksInc. 2011 – 2014. All Rights Reserved Data Pipeline: Monitoring DATA Primary site DR site Centralized monitoring of data pipeline With Falcon + Ambari Pipeline run alerts Hadoop Cluster-1 Hadoop Cluster-2 Pipeline run history Pipeline Scheduling raw clean prep raw clean prep
- 16. Page16 © HortonworksInc. 2011 – 2014. All Rights Reserved Replication with Falcon Staged Data Presented Data Cleansed Data Conformed Data Staged Data Presented Data Replication Failover Hadoop Cluster Primary Hadoop Cluster Replication BI / Analytics BusinessObjects BI • Falcon manages workflow and replication • Enables business continuity without requiring full data reprocessing • Failover clusters can be smaller than primary clusters
- 17. Page17 © HortonworksInc. 2011 – 2014. All Rights Reserved Data Retention with Falcon Staged Data Presented Data Cleansed Data Conformed Data Retain 5 Years Retain Last Copy Only Retain 3 Years Retain 3 Years • Sophisticated retention policies expressed in one place • Simplify data retention for audit, compliance, or for data re-processing Retention Policy
- 18. Page18 © HortonworksInc. 2011 – 2014. All Rights Reserved Late Data Handling with Falcon Staged Data Combined Data Online Transaction Data (via Sqoop) Web Log Data (via FTP) Wait up to 4 hours for FTP data to arrive • Processing waits until all required input data is available • Checks for late data arrivals, issues retrigger processing as necessary • Eliminates writing complex data handling rules within applications
- 19. Page19 © HortonworksInc. 2011 – 2014. All Rights Reserved HCatalog Table access Aligned metadata REST API • Raw Hadoop data • Inconsistent, unknown • Tool specific access Apache Falcon provides metadata services via HCatalog Metadata Services with HCatalog • Consistency of metadata and data models across tools (MapReduce, Pig, Hbase, and Hive) • Accessibility: share data as tables in and out of HDFS • Availability: enables flexible, thin-client access via REST API Shared table and schema management opens the platform Page 19
- 20. Page20 © HortonworksInc. 2011 – 2014. All Rights Reserved Data Governance in Apache Falcon
- 21. Page21 © HortonworksInc. 2011 – 2014. All Rights Reserved Data Pipeline: Tracing . Purchase feed Customer feed Product feed Store feed View dependencies between clusters, datasets and processes Data pipeline dependencies Add arbitrary tags to feeds & processes Data pipeline tagging Coming Soon Know who modified a dataset when and into what Data pipeline audits Analyze how a dataset reached a particular state Data pipeline lineage
- 22. Page22 © HortonworksInc. 2011 – 2014. All Rights Reserved Custom Metadata in Falcon • Metadata on Ingest (Content) – What is the format I expect my data to be in? – What source systems did the data come from, owners? – Answer: ingest descriptors + Hcat schema versioning • Metadata for Security (Access Controls) – How is each column blinded or encrypted? – Can I trust that I can join data across tables? What if email is encrypted differently? – Answer: security descriptors • Metadata for lineage (Source, History) – How do I chase down sources of data leading to reports and data? – Answer: lineage carried forward per workflow • Metadata for marts (Usage Constraints, Enrichment) – How do I materialize views and drop views as needed?
- 23. Page23 © HortonworksInc. 2011 – 2014. All Rights Reserved Entity Dependency in Falcon • Dependencies between Falcon entity definitions: cluster, feed & process – Lineage attributes: workflows, input/output feed windows, user, input and output paths, workflow engine, input/output size
- 24. Page24 © HortonworksInc. 2011 – 2014. All Rights Reserved Lineage in Falcon
- 25. Page25 © HortonworksInc. 2011 – 2014. All Rights Reserved Audit, Tagging and Access Control • Tagging – Allows custom tags in entities – Can decorate process entities pipeline names • Access Control – Support for ACL in entities – Authorization driven based on ACLs in entities • Audit – Each execution is controlled by Falcon and runs are audited – Correlate the execution with Lineage (Design) • Search – Search based on Tags, Pipelines, etc. – Full-text search
- 26. Page26 © HortonworksInc. 2011 – 2014. All Rights Reserved Technology • Metadata Repository – Titan Graph Database – Pluggable backing store, berkelydbje, Hbase • Entity Metadata – Tags, Entities are stored in the repository • Execution Metadata – Execution metadata are stored in the repository as well – this is unique to Falcon – Optional inputs • Search – Pluggable backend – Solr or Elastic Search
- 27. Page27 © HortonworksInc. 2011 – 2014. All Rights Reserved New in Apache Falcon 0.6.0 What is coming soon?
- 28. Page28 © HortonworksInc. 2011 – 2014. All Rights Reserved DR Mirroring of HDFS with Recipes •Mirroring for Disaster Recovery and Business continuity use cases. •Customizable for multiple targets and frequency of synchronization •Recipes: Template model re-use of complex workflows Recipe Reduce Cleanse Replicate Propertie s Workflow Template RecipePropertie s RecipePropertie s Workflow Template
- 29. Page29 © HortonworksInc. 2011 – 2014. All Rights Reserved Replication to Cloud •Seemlessly replicate to Cloud targets •Replicate from Cloud as a source. •Support for Amazon S3 and Microsoft Azure Azure Amazon S3 On Prem Cluster
- 30. Page30 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 31. Page31 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 32. Page32 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 33. Page33 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 34. Page34 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 35. Page35 © HortonworksInc. 2011 – 2014. All Rights Reserved
- 36. Page36 © HortonworksInc. 2011 – 2014. All Rights Reserved Q & A
- 37. Page37 © HortonworksInc. 2011 – 2014. All Rights Reserved Thank you! Learn more at: hortonworks.com/hadoop/falcon/
Editor's Notes
- #4 ET: Tactical POS Added benefits – consortium
- #5 Transition to Andrew
- #9 Transition to Andrew
- #10 Thanks Justin, Here are Falcon’s primary features. 1 The first is to manage the data lifecycle in one common place. 2 The second is to facilitate quick deployment of replication for business continuity and disaster recovery use cases. This includes monitoring and a base set of policies for replication and retention 3 Lastly, Falcon provide foundation audit and compliance features – visuallization and tracking of entity lineage and collection of audit logs
- #12 This is the high level Falcon Architecture Falcon runs as a standalone server as part of your Hadoop cluster A user creates entity specifications and submits to Falcon using the API Falcon validates and saves entity specifications to HDFS Falcon uses Oozie as its default scheduler Dashboard for entity viewing in Falcon UI Ambari integration for management
- #13 Feeds have location, replication schedule and retention policies Meta info including frequency, where data is coming from (source), where to replicate (target), how to long to retain
- #14 Let take a look at the Data Pipeline or workflow. ** read high level **
- #15 Hive – HQL scripts Pig scipts Oozie workflows
- #16 Once a pipeline is create you’ll want to run it. This means you probably want to monitoring as well. Falcon in conjunction with Ambari has centralized monitor ** bullets **
- #17 Ok let chat about Replication with Falcon – which is very efficient. In this example with a primary cluster with a typical workflow There is business requirement to replicate this to a Failover cluster ** builett **
- #18 Falcon has flexible data retention policies, it’s able to model the business compliance requirements. Sophisticated retention policies expressed in one place Simplify data retention for audit, compliance, or for data re-processing In this example, different dataset in a workflow can have different retention policies.
- #19 We realize at many type of workflow have inputs from different system with may be in different regions. Falcon has logic built-in to handle this potentially tricky situation.
- #20 HCatalog – metadata shared across whole platform File locations become abstract (not hard-coded) Data types become shared (not redefined per tool) Partitioning and HDFS-optimized
- #21 Transition to Andrew
- #22 Last but not least you’ll want to Trace or track the Data Pipeline We trace:
- #29 The first is DR mirroring with Recipes. Actually recipes can be used in number different use cases, but we’ll just focus on mirroring.
- #30 Place holder pic
- #31 Dashboard view Summary counts Inplace filters – by user defined tags
- #32 Entity creation interface is contextual and has field level sematic check to help the user along.
- #33 As you can see on the right – we have the actual XML being generated as the UI field are being filled out.
- #34 This can be help if you want copy portions to skip repeating entity from scratch.
- #36 Lastly the new UI allow to drilll down to the detail level for each entity types.