Designing an Agile Fast Data Architecture for Big Data Ecosystem using Logical Data Warehouse and Data Virtualization

© 2016 Autodesk | Enterprise Information Services
Designing an Agile Fast Data Architecture for Big Data Ecosystem
using Logical Data Warehouse and Data Virtualization
Kurt Jackson
Autodesk Enterprise Information Services

Introduction

© 2016 Autodesk | Enterprise Information Services 3
Some Definitions
 Agile
 “The division of tasks into short
phases of work and frequent
reassessment and adaptation of
plans.”
 Data Architecture
 “The models, policies, rules or
standards that govern which data is
collected, and how it is stored,
arranged, integrated.”
 Logical Data Warehouse
 “A logical abstraction layer which sits
on top of a variety of enterprise data
sources. The logical layer provides
durable data views without needing to
move or transform data from the
sources.”
 Data Virtualization
 “Data management that allows an
application to retrieve and
manipulate data without knowing
specific details about the data, such as
how it is formatted or where it is
physically located.”

Agile
Data
Architecture
Logical Data
Warehouse
Data
Virtualization
Agile Data Architecture Lifecycle

Business Problem

Multi-year Transition
Autodesk’s Business Challenge
Subscription
and
Rental
Perpetual

Most of us are in the same boat

The Autodesk Agile Data Architecture

Philosophy
 Access and refine data
near the source
 Published logical data
interfaces
 Truly agile data
environment

Autodesk Data Architecture

Why Build the Logical Data Warehouse Data virtualization can be used
throughout your data pipeline!

Big Data Ecosystem

One More Definition
 Data Governance
 “The management of the
availability, usability, integrity,
and security of
the data employed in an
enterprise.”

Logical Data Warehouses are an essential part of your Data
Governance Strategy for your Big Data Ecosystem
 Availability
 Channeling end user access
through a single governance
point simplifies administration
 Usability
 The LDW provides a single
repository for schema
definitions
 Simplifies end-user access for
visualization and interpretation
 Integrity
 Only published views in the LDW
are publically available
 Coupled with ownership,
guarantees the quality of the
data set
 Security
 The LDW can provide a single
point for authentication,
authorization and audit trail for
end user access

The Logical Data Warehouse implements the philosophy
 Access and refine data near the source
 No painful ETL pipelines for data
derivation
 Leverage power of Spark for fast access
 Published logical data interfaces
 Single access point for all of external data
sets
 Enterprise-class governance across the
big data ecosystem
 Truly agile data environment
 Facilitates rapid change/evolution in your
big data ecosystem
 Rip and replace becomes almost
transparent – replace the system that
delivers those views and you’re done

Building the Agile Data Architecture at Autodesk

Implementation Approach
 Identify enterprise data sources
 Harder than you think
 All new custom streaming, highly-available
ingestion mechanism
 Self-service or nearly so
 Kafka/Flume
 Leverage best-of breed for individual
components
 Spark for ETL and fast access
 Hcatalog/Oozie for metadata and job
orchestration
 Denodo for LDW
 Leverage highly-redundant cloud storage for
the data lake
 S3
 Develop canonical representations for your
data sets
 Freakin’ hard!
 Virtualize Spark fast access, data
warehouses and marts with a next
generation Logical DW
 New implementations leverage the LDW
 Legacy migrates opportunistically to Spark
fast access

Data Consumers
Architecting the Data Virtualization Layer
Corporate
LDAP
Data Virt
Instance
1
Data Virt
Instance
n
…
Logging Infrastructure
CI/CD
Source
Repository
Data
Data
Code
Audit
Audit
Legacy
Data Sources

Build an Information Architecture
 Base views to abstract data sources
 Layered derived views to reflect successively refined
derivations
 Create the notion of publication for curated, externally
visible views
 Expose services on top of views to make views more
accessible
 Separate namespaces (schemas) by project or
subject area
 Build the notion of commonality for views shared
across schemas
 Naming conventions for all objects
 Data portal for one-stop shopping for data consumers

Building an LDW makes your Big
Data Ecosystem Enterprise-Ready

Autodesk is a registered trademark of Autodesk, Inc., and/or its subsidiaries and/or affiliates in the USA and/or other countries. All other brand names, product names, or trademarks belong to their respective holders. Autodesk
reserves the right to alter product and services offerings, and specifications and pricing at any time without notice, and is not responsible for typographical or graphical errors that may appear in this document.
© 2016 Autodesk | Enterprise Information Services. All rights reserved

Designing an Agile Fast Data Architecture for Big Data Ecosystem using Logical Data Warehouse and Data Virtualization

More Related Content

What's hot

Viewers also liked

Similar to Designing an Agile Fast Data Architecture for Big Data Ecosystem using Logical Data Warehouse and Data Virtualization

More from Denodo

Recently uploaded

Designing an Agile Fast Data Architecture for Big Data Ecosystem using Logical Data Warehouse and Data Virtualization