Download as PDF, PPTX
Uploaded byZaloni
Creating a Modern Data Architecture
The document discusses creating a modern data architecture using a data lake approach. It describes the key components of a data lake including different zones for landing raw data, refining it into trusted datasets, and using sandboxes. It also summarizes challenges of data lakes and how an integrated data lake management platform can help with ingestion, governance, security, and enabling self-service analytics. Finally, it briefly discusses considerations for implementing cloud-based and hybrid data lakes.
More Related Content
Similar to Creating a Modern Data Architecture
Creating a Modern Data Architecture
- 1. Creating a moderndata architecture September 28, 2016 Ben Sharma | CEO ben@zaloni.com
- 2. • Award-winning providerof enterprise data lake management solutions: Integrated data lake management platform Self-service data preparation • Data Lake Design and Implementation Services: POC, Pilot, Production, Operations, Training • Data Science Professional Services
- 3. 3 Zaloni Proprietary Increased Agility New Insights Improved Scalability Datalakes are central to the modern data architecture
- 4. 4 Zaloni Proprietary • Store all types of data in its raw format • Create Refined, Standardized, Trusted datasets for various use cases • Store data for longer periods of time to enable historical analysis • Query and Access the data using a variety of methods • Manage streaming and batch data in a converged platform • Provide shorter time-to-insight with proper data management and governance The data lake promise
- 5. 5 Zaloni Proprietary Dataarchitecture modernizationTraditionalModern Data Lake Sources ETL EDW Derived (Transformed) Discovery Sandbox EDW Streaming Unstructured Data Various Sources Data Discovery Analytics BI Data Science Data Discovery Analytics BI
- 6. 6 Zaloni Proprietary Datalake – the challenges and the solution • Ingestion • Lack of Visibility • Privacy and Compliance • Quality Issues • Reliance on IT • Reusability • Rate of Change • Skills Gap • Complexity Managing: Delivering:Building:
- 7. 7 Zaloni Proprietary DataLake Reference Architecture • Data required for LOB specific views - transformed from existing certified data • Consumers are anyone with appropriate role-based access • Standardized on corporate governance/ quality policies • Consumers are anyone with appropriate role-based access • Single version of truth Transient Landing Zone Raw Zone Refined Zone Trusted Zone Sandbox Data Lake • Temporary store of source data • Consumers are IT, Data Stewards • Implemented in highly regulation industries • Original source data ready for consumption • Consumers are ETL developers, data stewards, some data scientists • Single source of truth with history • Data required for LOB specific views - transformed from existing certified data • Consumers are anyone with appropriate role-based access Sensors (or other time series data) Relational Data Stores (OLTP/ODS/ DW) Logs (or other unstructured data) Social and shared data
- 8. 8 Zaloni Proprietary Inputs: • Sources: RDBMS, File, Streaming, Structure/ Unstructured, External Data Processes: • Data transfer and intake: Managed and scheduled • Discover metadata • Register in the catalog • Apply Zone specific policies • Capture operational metrics and monitoring • Post-ingestion validations and clean up Outputs: • Data transfer to Raw Zone Policies: • Data privacy – tokenization, masking • Data security – user access • Data quality – profiling, entity level checks • Data lifecycle management – short lived, temporary Transient Landing Zone Transient Landing Zone • Temporary stores source data • Limited access • Consumers are IT, Data Stewards • Implemented in highly regulation industries • This zone collapses with Raw Zone if security is not needed Transient Landing Zone Raw Zone Refined Zone Trusted Zone Sandbox Data Lake
- 9. 9 Zaloni Proprietary Inputs: • Output from TLZ (Hcatalog entries) • Source inputs if TLZ is skipped Policies: • Data privacy – tokenization, masking • Data security – user access • Data Quality – profiling, entity level, field level • Data transformations required for Single View of Truth • Combine attributes to one entity • Change data formats (e.g. VSAM binary of JSON) • Derived columns • Field mappings • Drop columns • Data lifecycle management • Could be a candidate for S3 or Object store Outputs: • Data transfer to Trusted Zone • Data transfer to Sandbox Processes: • Register and update catalog • Apply zone specific policies • Operational metrics and monitoring Raw Zone Transient Landing Zone Raw Zone Refined Zone Trusted Zone Sandbox Data Lake Raw Zone • Original source data • Ready for consumption • Treated for basic validation and privacy • Metadata available to everyone but data access limited based on role • Consumers are ETL developers, data stewards, some data scientists • Single source of truth with history
- 10. 10 Zaloni Proprietary Inputs: • Output from Raw Zone Processes: • Register and update catalog • Apply zone specific policies • Data transformation required for refined use cases for LOB such as • Customer360 view • Periodic snapshots of revenue Outputs: • Data transfer to Refined Policies: • Data security – user access • Data lifecycle management • Lifetime of use case • Use case specific Trusted Zone Raw Zone Refined Zone Trusted Zone Sandbox Data Lake • Standardized on corporate governance/ quality policies • Consumers are anyone with appropriate role-based access • Metadata catalog available to all • Single version of truth Trusted Zone Transient Landing Zone
- 11. 11 Zaloni Proprietary Inputs: • Output from Trusted Zone • Output from Raw Zone for LOB-specific use cases Processes: • LOB specific transformations • Aggregates • De-normalized • Apply zone specific policies • Model building for reports • Optionally a cube generation Outputs: • Transformed data can be saved back to Refined Zone • Applications such as BI tools • Transferred to sandbox if required Policies: • Data security – user access • Data lifecycle management • Lifetime of use case • Use case specific • De tokenization if required (based on access) Refined Zone Raw Zone Refined Zone Trusted Zone Sandbox Data Lake Transient Landing Zone Refined Zone • Data required for LOB specific views - transformed from existing certified data • Consumers are anyone with appropriate role- based access • Metadata catalog available to all
- 12. 12 Zaloni Proprietary Inputs: • Output from Raw, Trusted and Refined Zones • Self-service ingestion Sandbox Raw Zone Refined Zone Trusted Zone Sandbox Data Lake Transient Landing Zone Processes: • Data scientists drive analysis • Self-service for ad-hoc Outputs: • Models that can later be operationalized • Optionally, results/data can be sent back to the Raw Zone Policies: • Data security – user access • Data lifecycle management • Lifetime of use case • Use case specific • Data required for LOB specific views - transformed from existing certified data • Consumers are anyone with appropriate role- based access • Metadata catalog available to all Sandbox
- 13. 13 Zaloni Proprietary Datalake Reference Architecture with Zaloni Consumption ZoneSource System File Data DB Data ETL Extracts Streaming Transient Landing Zone Raw Zone Refined Zone Trusted Zone Sandbox APIs Metadata Management Data Quality Data Catalog Security Data Lake Business Analysts Researchers Data Scientists DATA LAKE MANAGEMENT & GOVERNANCE PLATFORM Sensors (or other time series data) Relational Data Stores (OLTP/ODS/ DW) Logs (or other unstructured data) Social and shared data
- 14. 14 Zaloni Proprietary DataLake 360°: A holistic approach to actionable big data 1. Enable the lake 2. Govern the data 3. Engage the business • Foster a data-driven business through self-service data discovery and preparation • Safeguard sensitive data and enable regulatory compliance • Improve data visibility, reliability and quality to reduce time-to- insight • Leverage the full power of a scale-out architecture with an actionable, scalable data lake
- 15. 15 Zaloni Proprietary • Managed Ingestion § Ability to ingest vast amounts of data § Ability to handle a wide variety of formats (streaming, files, custom) and sources § Build in repeatability through automation to pick up incoming data and apply pre-defined processing • Metadata Management § Capture and manage operational, technical and business metadata § Provides visibility and reliability – key to finding data in the lake § Reduced time to insight for analytics § File and record level watermarking provides data lineage, enables audit and traceability Enable the lake
- 16. 16 Zaloni Proprietary • Data Lineage § See how data moves and how it is consumed in the data lake. § Safeguard data and reduce risk, always knowing where data has come from, where it is, and how it is being used. • Data Quality § Rules based Data validation § Integration with the Managed Data Pipeline § Stats and metrics for reporting and actions Govern the data
- 17. 17 Zaloni Proprietary • Data Security and Privacy § Differing permissions require enhanced data security § Mask or tokenize data before published in the lake for consumption § Policy-based security • Data lifecycle management across tiered storage environments § Hot -> Warm -> Cold on an entity level based on policies/SLAs § Across on-premise and cloud environments § Provide data management features to automate scheduling and orchestration of data movement between heterogeneous storage environments Govern the data
- 18. 18 Zaloni Proprietary Engagethe business • Data Catalog § See what data is available across your enterprise § Contribute valuable business information to improve search and usage § Use a shopping cart experience to create sandbox for ad- hoc and exploratory analytics • Self-service Data Preparation § Blend data in the lake without a costly IT project § Perform interactive data-driven transformations § Collaborate and share data assets and transformations with peers
- 19. 19 Zaloni Proprietary • Rapid increase of Data Lake platforms in the Cloud • Hybrid cloud and multi-cloud considerations • Support sensitive data on premise and external data in the cloud (e.g. client data, machine-generated) • Key challenges: § Leverage Cloud native features § Consistence Data Management and Governance Emergence of cloud-based and hybrid data lakes GOVERNANCE VISIBILITY
- 20. 20 Zaloni Proprietary • How do you create a cloud agnostic data lake platform? • How deploy a cost-effective compute layer? § Elastic compute layer § Batch and near real-time • How do you optimize storage? § Support polyglot persistence § DLM • How do you optimize network connectivity between Ground to Cloud? • How do you meet enterprise security requirements? Considerations for data lake in the cloud CLOUD and HYBRID ENVIRONMENTS
- 21. 21 Zaloni Proprietary CloudData Lake Maturity model Lift and Shift Cloud Native features Multi and Hybrid Cloud Replicate on- premise Data Lake in the cloud Leverage Object stores, Transient compute platforms, Messaging systems Abstraction over multiple clouds, consistent Data Management and Governance
- 22. 22 Zaloni Proprietary Buildingyour blueprint 1. Questions 2. Inputs 3. Outcomes Business Drivers AND Business Questions: e.g. Where is fraud occurring? How do I optimize inventory? Data Use Cases Platform Subject Areas Source System Capabilities, Process Ingest, Organize, Enrich, Explore Roadmap Managed Data Lake Analytics Strategy = ++
- 23. 23 Zaloni Proprietary Typicaldata lake implementation timeline POC Weeks Weeks Production Data Lake Platform Proof of Concept: ü Demonstrate technical capabilities of the platform in the context of selected use cases Data Lake Implementation: ü Planning, Installation, Training ü Sample data sets ingested ü Pilot uses cases created Business Use Case Delivered: ü Engage business stakeholders to identify production use cases at scale ü Review learnings and optimize the data lake Data Lake Use Case Implement Business Use Case Varies by Use Case
- 24. DATA LAKE MANAGEMENT ANDGOVERNANCE PLATFORM SELF-SERVICE DATA PREPARATION
- 25. FREE T-SHIRT! Building aModern Data Architecture Ben Sharma, CEO and Founder, Zaloni Wednesday, 2:05 p.m. – 1 E 09 Demo and FREE copy of book “Architecting Data Lakes” Speaking Sessions: Cloud Computing and Big Data Ben Sharma, CEO and Founder, Zaloni Tuesday, 9:30 a.m. – 1B 01/02 Visit Booth #644 for these giveaways!