SlideShare a Scribd company logo
Modernize & Automate Analytics
Data Pipelines
Attunity and Microsoft Azure
2© 2018 Attunity 2© 2017 Attunity
AGENDA
09:00 AM - 09:15 AM Introductions
09:15 AM - 09:45 AM The Business Value of Real-Time Analytics
09:45 AM - 10:30 AM Why you should use Microsoft Azure as your Analytics Platform
10:30 AM - 10:45 AM Break
10:45 AM - 11:30 AM Delivering Real-Time Data to the Azure Cloud
11:30 AM - 12:15 PM Data Warehouse Automation for Azure (without ETL coding!)
12:15 PM - 01:00 PM Automate Analytics Ready Data Sets in Azure Data Lake
01:00 PM - 02:00 PM Lunch and Q&A with the Data Platform experts
3© 2018 Attunity 3© 2017 Attunity
DATA AS THE NEW OIL
https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data
4© 2018 Attunity
INSIGHT-DRIVEN BUSINESSES
5© 2018 Attunity
FRAUD-DETECTION BEFORE
POS Backend
System(s)
Analytics
Repository
X X
Fraud Analytics Model
$$$
Cost
6© 2018 Attunity
FRAUD-DETECTION REAL-TIME FLOW
HDFS
POS Backend
System(s)
Analytics
Repository
X
Fraud Prediction Model
Fraud Prediction Model
Fraudulent TransactionX
7© 2019 Attunity 7© 2019 Attunity
Predict energy
prices
Reduce energy
consumption and
outages
Predictive
maintenance
Forward capacity
planning
Personalization
Contextual
Recommendations
Dynamic Pricing
REAL-TIME ANALYTICS IMPACTS ALL
INDUSTRIES
Retail &
Consumer Services
Manufacturing &
Supply Chain
Utilities
Fraud Detection
Portfolio Analysis
Risk Management
Financial Services
Social media
sentiment analysis
Customer churn
Customer 360
8© 2019 Attunity 8© 2019 Attunity
REAL-TIME DASHBOARDS – LATE
SHIPMENTS
9© 2019 Attunity
CURRENT ANALYTICS REPOSITORY
CHALLENGES
Source
Systems
Existing Analytics Repository Current Challenges
• Real-time data needs are not being met
• Can not support the pace of new strategic
business initiatives
• Costly to maintain
• Inflexible infrastructure / must support largest
analytics workloads
• Supports multiple problem domains –
operational reporting & analytics
10© 2019 Attunity 10© 2019 Attunity
Analyze a broader set of data structures
as well as structured data
Faster and improved decision making
Leverage AI/ML, IoT and decision
automation for a competitive advantage
Requires managed Data Lake creation
and Big Data processing at scale
Requires real-time data from on-
premise systems and cloud platforms
Next Generation Analytics
Reduce the costs associated with legacy
EDW’s and provide elasticity
Meet new business requirements
Support more advanced analytics
Replace traditional ETL with modern
self-service capabilities
Requires real-time data from on-
premise systems and cloud platforms
Data Warehouse Modernization
SaaS
IaaS
PaaS
DB
MF
EDW
FILES
DWaaS
TRENDS DRIVING INTEGRATION MODERNIZATION &
AUTOMATION
DATA
CONSUMPTION
& ANALYTICS
DB
MF
EDW
FILES
11© 2017 Attunity 11© 2017 Attunity
Data Delivery
SQL DW
DATA WAREHOUSE
Data Lake
ADLS
Operational Management
MONITORINGALERTS ANALYTICS METADATACENTRALIZED
RAW  ASSEMBLED  PROVISIONED
The Attunity Platform
Accelerate your Azure Analytics Journey
Enterprise
Data Sources
APPS / OTHER
RDBMS
FILE
SAP
MAINFRAME
Data Streaming
KAFKAEVENT HUB
Operational Management
MONITORINGALERTS ANALYTICS METADATACENTRALIZEDMONITORINGALERTS ANALYTICS METADATACENTRALIZED
E-
LT
Data
Ingestion
REAL-TIME
DATA MOVEMENT AUTOMATION
Data Science Processing
PREDICTIVEAL / ML
DATA LAKE / DATA WAREHOUSE AUTOMATION
Consumers
POWER BI
Analysis Services
12© 2017 Attunity 12© 2017 Attunity
Agenda
09:00 AM - 09:15 AM Introductions
09:15 AM - 09:45 AM The Business Value of Real-Time Analytics
09:45 AM - 10:30 AM Why you should use Microsoft Azure as your Analytics Platform
10:30 AM - 10:45 AM Break
10:45 AM - 11:30 AM Delivering Real-Time Data to the Azure Cloud
11:30 AM - 12:15 PM Data Warehouse Automation for Azure (without ETL coding!)
12:15 PM - 01:00 PM Automate Analytics Ready Data Sets in Azure Data Lake
01:00 PM - 02:00 PM Lunch and Q&A with the Data Platform experts
Real-Time Data Ingestion
to the Azure Cloud
14© 2017 Attunity
Accelerate your Azure Analytics Journey
Data Ingestion – Attunity Replicate
15© 2017 Attunity
Universal Solution
for the Microsoft Data Platform
EASY NO DOWNTIME
HETEROGENEOU
S
MIGRATION
LOW IMPACT OPTIMIZED PERFORMANCE
ANALYTICS/BI
REAL-TIME REPLICATION
ON PREM
CLOUD
MAINFRAMES
SQL Database
SQL Data Warehouse
ADLS & BLOB
Event Hubs
2012
Parallel Data Warehouse
Analytics Platform System
Azure DB for MySQL
Azure DB for PostgreSQL
16© 2017 Attunity 16© 2017 Attunity
Pre-packaged
automation of complex
tasks
Modern user
experience
Zero source footprint
Change data
capture (CDC)
Stream live updates
Optimized for high-
performance
movement
All major platforms
DB | DW | Hadoop |
Legacy
On Premises | Cloud
SAP | Mainframe
Simplified Real-Time Universal
Data Delivery with Attunity Replicate
17© 2017 Attunity 17© 2017 Attunity
Attunity Replicate Architecture
TRANSFER
IN-MEMORY
FILTER
DATA LAKE
RDBMS
DATA
WAREHOUSE
FILES
MAINFRAME
TRANSFORM
PERSISTENT
STORE
LOG BASED
CDC
BATCH
INCREMENTAL
BATCH
RDBMS
DATA
WAREHOUSE
STREAMING
FILES
ATTUNITY POWERPOINT ASSET LIBRARY
DATA
LAKE
18© 2017 Attunity
TARGET SCHEMA
CREATION
SAP
RDBMS
EDW
FILE
MAINFRAME
HETEROGENEOUS
DATA TYPE MAPPING
BATCH TO CDC
TRANSITION
DDL CHANGE
PROPAGATION
FILTERING
TRANSFORMATIONS
RDBMS
EDW
FILES
STREAMING
REPLICATE
Attunity Replicate - End to End Automation
DATA LAKE
19© 2017 Attunity 19© 2017 Attunity
SOURCES
CLOUD
Amazon RDS
(SQL Server, Oracle,
MySQL, Postgres)
Amazon Aurora
(MySQL)
Amazon Redshift
Azure SQL Server
M1 (Q1)
ATTUNITY - PLATFORM SUPPORTABILITY MATRIX
SAP
ECC
ERP
CRM
SRM
GTS
MDG
S/4HANA
(on Oracle, SQL,
DB2, HANA)
DATABASE
Oracle
SQL Server
DB2 iSeries
DB2 z/OS
DB2 LUW
MySQL
PostgeSQL
Sybase ASE
Informix
ODBC
EDW
Exadata
Teradata
Netezza
Vertica
Pivotal
MAINFRAME
DB2 z/OS
IMS/DB
VSAM
FLAT FILES
Delimited
(e.g., CSV, TSV)
TARGETS
FLAT FILES
Delimited
(e.g., CSV, TSV)
STREAMING
Kafka
Amazon Kinesis
Azure Event Hubs
MapR Streams
SAP
HANA
EDW
Exadata
Teradata
Netezza
Vertica
Sybase IQ
SAP HANA
Microsoft PDW
GOOGLE
Cloud SQL (MySQL,
Postgres)
Cloud Storage
Dataproc
PubSub (‘19)
Big Query (Q2)
DATA LAKE
Hortonworks
Cloudera
MapR
Amazon EMR
Azure HDInsight
Google Dataproc
DATABASE
Oracle
SQL Server
DB2 LUW
MySQL
PostgreSQL
Sybase ASE
Informix
MemSQL
Compose support
AZURE
DBaaS (SQL DB)
DBaaS (MySQL,
Postgres)
ADLS
BLOB
HDInsight
Event Hub
SQL DW
Snowflake (Q1)
Databricks (Q2)
AWS
RDS (MySQL,
Postgres, MariaDB,
Oracle, SQL Server)
Aurora (MySQL,
Postgres)
S3
EMR
Kinesis
Redshift
Snowflake (Q1)
Databricks (Q2)
SaaS
Salesforce (Q2)
20© 2017 Attunity 20© 2017 Attunity
Attunity Replicate Demo
Attunity Data Warehouse Automation
Solution Overview
22© 2017 Attunity
Accelerate your Azure Analytics Journey
Data Warehouse Automation– Attunity Compose
23© 2017 Attunity 23© 2019 Attunity
STAGING
AREA
--------
Trunc &
Load
EDW
-------
3NF
DATA MART
-------
Star Schema
CRM
ERP
FINANCE
LEGACY
SOURCES
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
ETL
That’s not
exactly what I
wanted
Why is my
data always
X day(s) old
BUSINESS / CONSUMER / REQUIREMENTS CHANGES
IMPACT IMPACT IMPACT
We don’t
need that
anymore
Complex transformations
Requirement / Source chg.
Data quality & validation
Manual Modelling
Complex ETL design
DevOps design
Why Data Warehouse Automation ?
Traditional Data Warehousing Methods are failing the business
Complex Design
Impact to source
Bulk – not change data
Long running extracts
Batch/EOD Based
Complex Build
Long, manual coding effort
Long testing cycles
Slow to react to changes
Time to Market
24© 2017 Attunity 24© 2017 Attunity
AUTOMATED WORKFLOW
Real-Time
Extract
Auto Extraction,
Loading,
Mapping
Auto Generated
Transformations
Change
Propagation
Auto Design with
Best Practices
“DWA will accomplish an initial BI implementation up to five times faster than traditional methods”*
*TDWI Data Warehouse Automation Course
REAL-TIME
ODS
STAGING EDW MARTS
Data Pipeline for data warehouses
commit to model architecture
Azure SQL DW
Oracle
SQL Server
RedShift
Snowflake **
DATA MOVEMENT AUTOMATION DATA LAKE / DATA WAREHOUSE AUTOMATION
25© 2017 Attunity 25© 2017 Attunity
Compose for Data Warehouse Demo
EDW MARTS
MDM
Sales
Sales
Service
Ticket
What can we do
to better manage
late shipments?
26© 2017 Attunity 26© 2017 Attunity
CUSTOM MAPPINGS
DATA MART
DESIGN
SOURCE
MODEL
Data Warehouse Model Generation
Automated Mapping Generation
Data Warehouse ETL Generation
Error Mart Generation
Data Mart (Star Schema) ETL Generation
Workflow Generation & Orchestration
Automates
 Native CDC integration
 E-LT Set based, best practice data loads
 Transparent, editable E-LT
 Surrogate Key
 Type 1 / Type 2
 Referential integrity (late arriving dimensions)
 Error mart automation
 Data validation
 Automated Workflow & Dependencies
DW MODEL
Automates
 Flexible physical model
 3NF / Data Vault methodology
 Transparent, editable DDL
ROBUST E-LT
DATA VALIDATION /
QUALITY RULES
STAR
SCHEMAS
Automates
 Type 1 / Type 2 conformed dimensions
 Automatic Incremental processing from DW
 Automated flattening of dimensions
 Granular, Aggregate & Time-Oriented Fact support
Documentation & Deployment
Compose for data warehouses
Automation of complex data processing requirements
Attunity Data Lake Automation
Solution Overview
28© 2017 Attunity
Accelerate your Azure Analytics Journey
Data Lake Automation– Attunity Compose
29© 2017 Attunity 29© 2017 Attunity
The Attunity Difference
Automating Data Lake Ingestion
DATA LAKE AUTOMATION
“Current view example”
30© 2017 Attunity 30© 2017 Attunity
1. Land 2. Store 3. Provision Consume
CAPTURE
PARTITION
ENRICH
SUBSET
STANDARDIZE
MERGE
FORMAT
ANALYZE
PREPARE
CLEANSE
JOIN
Raw
Deltas
Full
Change
History
ODS
HDS
Snapshot
Data sets
SAP
RDBMS
DATA
WAREHOUSE
FILES
MAINFRAME
Source
Deliver Analytics Optimized Data Sets
FOR DATA LAKES
 Real-time high volume delivery
 Consistent data
 Write optimized format
 Standardized historical view
 Read optimized format
 Automated at scale 1,000’s of source
entities
 Current / Type 2 / Snapshot
 Read optimized format
 Automated loads w/ Spark & Hive
31© 2017 Attunity 31© 2017 Attunity
Data Lake Storage <bucket/container/folder>
Source -> Landing -> Storage Data Flow
Source Landing
customer
customer__ct
.seq / .csv
customer
update customers
set name = ‘Maria Anders’
where id = 1;
Delete from
customers where id = 2;
Insert into customers
values (3, ‘New Customer’);
Storage
customer
customer__delta
.snappy.parquet
32© 2017 Attunity 32© 2017 Attunity
Data Lake Storage
Provisioning *Storage
Storage -> Provisioning Data Flow
customer
customer__delta
HDS (“Type 2”)
Snapshot (Point-in-time)
ODS (“current”)
.orc
.snappy.parquet
.snappy.parquet / .orc /.avro
Compactor
Task <Spark>
*Each provisioning task has its own
bucket / container / storage location
33© 2017 Attunity 33© 2017 Attunity
Azure Data Lake Storage
Azure Data Lake Architecture with Attunity
Automated Data Ingestion and Provisioning
Enterprise
Data Sources
APPS / OTHER
RDBMS
FILE
SAP
MAINFRAME
Raw
Transactiona
l Data
<SEQ/CSV>
Standardized
Historical
Raw Data
<Parquet>
Provisioned
Data Sets
<Parquet/
ORC/Avro>
HDInsight
Attunity
Replicate
Batch Load
CDC
Attunity Compose
for Data Lakes
(Cloud VM or on prem)
Compose
Agent
 Industry leading CDC and data ingestion with Attunity Replicate
 Automate standardization and provisioning of consumer ready data sets with Attunity Compose
 Automated handling of schema evolution across 1,000’s of entities
Metadata / Instruction
channel
Data flow
BI / Data
Science etc.
 Azure ADLS
 Google Storage
 HDFS
 AWS S3
 Azure HDInsight
 Azure Databricks**
 AWS EMR
 Google DataProc
 Hortonworks
 Cloudera
34© 2017 Attunity 34© 2017 Attunity
Compose for Data Lakes Demo
35© 2017 Attunity 35© 2017 Attunity
AGILE DATA DELIVERY
WHAT YOU CAN ACHIEVE WITH ATTUNITY
COMPOSE
High levels of satisfaction for business
Significantly improved utilization of resources
Maximized productivity
Rapid adaption to business changes
Vastly improved data quality, delivered real-time
36© 2017 Attunity 36© 2017 Attunity
Trusted by 2000 Customers Worldwide
And Half the Fortune 100
FIN. SERVICES MANUF. / INDUS. GOVERNMENTHEALTH CARE
TECHNOLOGY / TELECOM OTHER INDUSTRIESRETAIL
Thank you
attunity.com

More Related Content

What's hot

Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
Databricks
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
Denodo
 
Snowflake Datawarehouse Architecturing
Snowflake Datawarehouse ArchitecturingSnowflake Datawarehouse Architecturing
Snowflake Datawarehouse Architecturing
Ishan Bhawantha Hewanayake
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
Databricks
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
LibbySchulze
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
James Serra
 
Azure Data Lake Intro (SQLBits 2016)
Azure Data Lake Intro (SQLBits 2016)Azure Data Lake Intro (SQLBits 2016)
Azure Data Lake Intro (SQLBits 2016)
Michael Rys
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta Lake
Databricks
 
Data Governance Takes a Village (So Why is Everyone Hiding?)
Data Governance Takes a Village (So Why is Everyone Hiding?)Data Governance Takes a Village (So Why is Everyone Hiding?)
Data Governance Takes a Village (So Why is Everyone Hiding?)
DATAVERSITY
 
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
Jochem van Grondelle
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
Databricks
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
DATAVERSITY
 
Data Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future OutlookData Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future Outlook
James Serra
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
James Serra
 
BDA311 Introduction to AWS Glue
BDA311 Introduction to AWS GlueBDA311 Introduction to AWS Glue
BDA311 Introduction to AWS Glue
Amazon Web Services
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
James Serra
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Tristan Baker
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Eric Kavanagh
 
Snowflake Overview
Snowflake OverviewSnowflake Overview
Snowflake Overview
Snowflake Computing
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
Databricks
 

What's hot (20)

Introduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse ArchitectureIntroduction SQL Analytics on Lakehouse Architecture
Introduction SQL Analytics on Lakehouse Architecture
 
Enabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data VirtualizationEnabling a Data Mesh Architecture with Data Virtualization
Enabling a Data Mesh Architecture with Data Virtualization
 
Snowflake Datawarehouse Architecturing
Snowflake Datawarehouse ArchitecturingSnowflake Datawarehouse Architecturing
Snowflake Datawarehouse Architecturing
 
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2
 
Time to Talk about Data Mesh
Time to Talk about Data MeshTime to Talk about Data Mesh
Time to Talk about Data Mesh
 
Building a modern data warehouse
Building a modern data warehouseBuilding a modern data warehouse
Building a modern data warehouse
 
Azure Data Lake Intro (SQLBits 2016)
Azure Data Lake Intro (SQLBits 2016)Azure Data Lake Intro (SQLBits 2016)
Azure Data Lake Intro (SQLBits 2016)
 
Intro to Delta Lake
Intro to Delta LakeIntro to Delta Lake
Intro to Delta Lake
 
Data Governance Takes a Village (So Why is Everyone Hiding?)
Data Governance Takes a Village (So Why is Everyone Hiding?)Data Governance Takes a Village (So Why is Everyone Hiding?)
Data Governance Takes a Village (So Why is Everyone Hiding?)
 
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
To mesh or mess up your data organisation - Jochem van Grondelle (Prosus/OLX ...
 
Modernizing to a Cloud Data Architecture
Modernizing to a Cloud Data ArchitectureModernizing to a Cloud Data Architecture
Modernizing to a Cloud Data Architecture
 
How a Semantic Layer Makes Data Mesh Work at Scale
How a Semantic Layer Makes  Data Mesh Work at ScaleHow a Semantic Layer Makes  Data Mesh Work at Scale
How a Semantic Layer Makes Data Mesh Work at Scale
 
Data Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future OutlookData Warehousing Trends, Best Practices, and Future Outlook
Data Warehousing Trends, Best Practices, and Future Outlook
 
Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)Data Lakehouse, Data Mesh, and Data Fabric (r2)
Data Lakehouse, Data Mesh, and Data Fabric (r2)
 
BDA311 Introduction to AWS Glue
BDA311 Introduction to AWS GlueBDA311 Introduction to AWS Glue
BDA311 Introduction to AWS Glue
 
Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)Data Lakehouse, Data Mesh, and Data Fabric (r1)
Data Lakehouse, Data Mesh, and Data Fabric (r1)
 
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021
 
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data PipelinesBest Practices in DataOps: How to Create Agile, Automated Data Pipelines
Best Practices in DataOps: How to Create Agile, Automated Data Pipelines
 
Snowflake Overview
Snowflake OverviewSnowflake Overview
Snowflake Overview
 
Architect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh ArchitectureArchitect’s Open-Source Guide for a Data Mesh Architecture
Architect’s Open-Source Guide for a Data Mesh Architecture
 

Similar to Modernize & Automate Analytics Data Pipelines

Streaming Real-time Data to Azure Data Lake Storage Gen 2
Streaming Real-time Data to Azure Data Lake Storage Gen 2Streaming Real-time Data to Azure Data Lake Storage Gen 2
Streaming Real-time Data to Azure Data Lake Storage Gen 2
Carole Gunst
 
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the CloudBring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
DataWorks Summit/Hadoop Summit
 
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Matt Stubbs
 
Accelerate and modernize your data pipelines
Accelerate and modernize your data pipelinesAccelerate and modernize your data pipelines
Accelerate and modernize your data pipelines
Paul Van Siclen
 
The Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data IntegrationThe Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data Integration
Eric Kavanagh
 
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
DATAVERSITY
 
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEEDTHE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
webwinkelvakdag
 
Azure Data.pptx
Azure Data.pptxAzure Data.pptx
Azure Data.pptx
FedoRam1
 
Data Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptxData Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptx
ArunPandiyan890855
 
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsHow to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
Informatica
 
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
StampedeCon
 
Trivadis Azure Data Lake
Trivadis Azure Data LakeTrivadis Azure Data Lake
Trivadis Azure Data Lake
Trivadis
 
Demystifying Data Warehouse as a Service
Demystifying Data Warehouse as a ServiceDemystifying Data Warehouse as a Service
Demystifying Data Warehouse as a Service
Snowflake Computing
 
Analytics on the Cloud with Tableau on AWS
Analytics on the Cloud with Tableau on AWSAnalytics on the Cloud with Tableau on AWS
Analytics on the Cloud with Tableau on AWS
Amazon Web Services
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use Cases
James Serra
 
Architecting Solutions Leveraging The Cloud
Architecting Solutions Leveraging The CloudArchitecting Solutions Leveraging The Cloud
Architecting Solutions Leveraging The Cloud
David Chou
 
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on AzureGlobal Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
Karim Vaes
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Amazon Web Services
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Amazon Web Services
 
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
Amazon Web Services
 

Similar to Modernize & Automate Analytics Data Pipelines (20)

Streaming Real-time Data to Azure Data Lake Storage Gen 2
Streaming Real-time Data to Azure Data Lake Storage Gen 2Streaming Real-time Data to Azure Data Lake Storage Gen 2
Streaming Real-time Data to Azure Data Lake Storage Gen 2
 
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the CloudBring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
Bring your SAP and Enterprise Data to Hadoop, Apache Kafka and the Cloud
 
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TI...
 
Accelerate and modernize your data pipelines
Accelerate and modernize your data pipelinesAccelerate and modernize your data pipelines
Accelerate and modernize your data pipelines
 
The Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data IntegrationThe Future of Data Warehousing and Data Integration
The Future of Data Warehousing and Data Integration
 
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
Slides: Proven Strategies for Hybrid Cloud Computing with Mainframes — From A...
 
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEEDTHE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
THE FUTURE OF DATA: PROVISIONING ANALYTICS-READY DATA AT SPEED
 
Azure Data.pptx
Azure Data.pptxAzure Data.pptx
Azure Data.pptx
 
Data Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptxData Modernization_Harinath Susairaj.pptx
Data Modernization_Harinath Susairaj.pptx
 
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data AnalyticsHow to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
How to Architect a Serverless Cloud Data Lake for Enhanced Data Analytics
 
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
Best Practices For Building and Operating A Managed Data Lake - StampedeCon 2016
 
Trivadis Azure Data Lake
Trivadis Azure Data LakeTrivadis Azure Data Lake
Trivadis Azure Data Lake
 
Demystifying Data Warehouse as a Service
Demystifying Data Warehouse as a ServiceDemystifying Data Warehouse as a Service
Demystifying Data Warehouse as a Service
 
Analytics on the Cloud with Tableau on AWS
Analytics on the Cloud with Tableau on AWSAnalytics on the Cloud with Tableau on AWS
Analytics on the Cloud with Tableau on AWS
 
Big Data: It’s all about the Use Cases
Big Data: It’s all about the Use CasesBig Data: It’s all about the Use Cases
Big Data: It’s all about the Use Cases
 
Architecting Solutions Leveraging The Cloud
Architecting Solutions Leveraging The CloudArchitecting Solutions Leveraging The Cloud
Architecting Solutions Leveraging The Cloud
 
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on AzureGlobal Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
Global Azure Bootcamp 2017 - Why I love S2D for MSSQL on Azure
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
 
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
Understanding AWS Managed Database and Analytics Services | AWS Public Sector...
 
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
AWS re:Invent 2016: How to Build a Big Data Analytics Data Lake (LFS303)
 

Recently uploaded

Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
shyamraj55
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
Matthew Sinclair
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
Claudio Di Ciccio
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
panagenda
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Paige Cruz
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 

Recently uploaded (20)

Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with SlackLet's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
20240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 202420240605 QFM017 Machine Intelligence Reading List May 2024
20240605 QFM017 Machine Intelligence Reading List May 2024
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”“I’m still / I’m still / Chaining from the Block”
“I’m still / I’m still / Chaining from the Block”
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAUHCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfObservability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 

Modernize & Automate Analytics Data Pipelines

  • 1. Modernize & Automate Analytics Data Pipelines Attunity and Microsoft Azure
  • 2. 2© 2018 Attunity 2© 2017 Attunity AGENDA 09:00 AM - 09:15 AM Introductions 09:15 AM - 09:45 AM The Business Value of Real-Time Analytics 09:45 AM - 10:30 AM Why you should use Microsoft Azure as your Analytics Platform 10:30 AM - 10:45 AM Break 10:45 AM - 11:30 AM Delivering Real-Time Data to the Azure Cloud 11:30 AM - 12:15 PM Data Warehouse Automation for Azure (without ETL coding!) 12:15 PM - 01:00 PM Automate Analytics Ready Data Sets in Azure Data Lake 01:00 PM - 02:00 PM Lunch and Q&A with the Data Platform experts
  • 3. 3© 2018 Attunity 3© 2017 Attunity DATA AS THE NEW OIL https://www.economist.com/leaders/2017/05/06/the-worlds-most-valuable-resource-is-no-longer-oil-but-data
  • 5. 5© 2018 Attunity FRAUD-DETECTION BEFORE POS Backend System(s) Analytics Repository X X Fraud Analytics Model $$$ Cost
  • 6. 6© 2018 Attunity FRAUD-DETECTION REAL-TIME FLOW HDFS POS Backend System(s) Analytics Repository X Fraud Prediction Model Fraud Prediction Model Fraudulent TransactionX
  • 7. 7© 2019 Attunity 7© 2019 Attunity Predict energy prices Reduce energy consumption and outages Predictive maintenance Forward capacity planning Personalization Contextual Recommendations Dynamic Pricing REAL-TIME ANALYTICS IMPACTS ALL INDUSTRIES Retail & Consumer Services Manufacturing & Supply Chain Utilities Fraud Detection Portfolio Analysis Risk Management Financial Services Social media sentiment analysis Customer churn Customer 360
  • 8. 8© 2019 Attunity 8© 2019 Attunity REAL-TIME DASHBOARDS – LATE SHIPMENTS
  • 9. 9© 2019 Attunity CURRENT ANALYTICS REPOSITORY CHALLENGES Source Systems Existing Analytics Repository Current Challenges • Real-time data needs are not being met • Can not support the pace of new strategic business initiatives • Costly to maintain • Inflexible infrastructure / must support largest analytics workloads • Supports multiple problem domains – operational reporting & analytics
  • 10. 10© 2019 Attunity 10© 2019 Attunity Analyze a broader set of data structures as well as structured data Faster and improved decision making Leverage AI/ML, IoT and decision automation for a competitive advantage Requires managed Data Lake creation and Big Data processing at scale Requires real-time data from on- premise systems and cloud platforms Next Generation Analytics Reduce the costs associated with legacy EDW’s and provide elasticity Meet new business requirements Support more advanced analytics Replace traditional ETL with modern self-service capabilities Requires real-time data from on- premise systems and cloud platforms Data Warehouse Modernization SaaS IaaS PaaS DB MF EDW FILES DWaaS TRENDS DRIVING INTEGRATION MODERNIZATION & AUTOMATION DATA CONSUMPTION & ANALYTICS DB MF EDW FILES
  • 11. 11© 2017 Attunity 11© 2017 Attunity Data Delivery SQL DW DATA WAREHOUSE Data Lake ADLS Operational Management MONITORINGALERTS ANALYTICS METADATACENTRALIZED RAW  ASSEMBLED  PROVISIONED The Attunity Platform Accelerate your Azure Analytics Journey Enterprise Data Sources APPS / OTHER RDBMS FILE SAP MAINFRAME Data Streaming KAFKAEVENT HUB Operational Management MONITORINGALERTS ANALYTICS METADATACENTRALIZEDMONITORINGALERTS ANALYTICS METADATACENTRALIZED E- LT Data Ingestion REAL-TIME DATA MOVEMENT AUTOMATION Data Science Processing PREDICTIVEAL / ML DATA LAKE / DATA WAREHOUSE AUTOMATION Consumers POWER BI Analysis Services
  • 12. 12© 2017 Attunity 12© 2017 Attunity Agenda 09:00 AM - 09:15 AM Introductions 09:15 AM - 09:45 AM The Business Value of Real-Time Analytics 09:45 AM - 10:30 AM Why you should use Microsoft Azure as your Analytics Platform 10:30 AM - 10:45 AM Break 10:45 AM - 11:30 AM Delivering Real-Time Data to the Azure Cloud 11:30 AM - 12:15 PM Data Warehouse Automation for Azure (without ETL coding!) 12:15 PM - 01:00 PM Automate Analytics Ready Data Sets in Azure Data Lake 01:00 PM - 02:00 PM Lunch and Q&A with the Data Platform experts
  • 13. Real-Time Data Ingestion to the Azure Cloud
  • 14. 14© 2017 Attunity Accelerate your Azure Analytics Journey Data Ingestion – Attunity Replicate
  • 15. 15© 2017 Attunity Universal Solution for the Microsoft Data Platform EASY NO DOWNTIME HETEROGENEOU S MIGRATION LOW IMPACT OPTIMIZED PERFORMANCE ANALYTICS/BI REAL-TIME REPLICATION ON PREM CLOUD MAINFRAMES SQL Database SQL Data Warehouse ADLS & BLOB Event Hubs 2012 Parallel Data Warehouse Analytics Platform System Azure DB for MySQL Azure DB for PostgreSQL
  • 16. 16© 2017 Attunity 16© 2017 Attunity Pre-packaged automation of complex tasks Modern user experience Zero source footprint Change data capture (CDC) Stream live updates Optimized for high- performance movement All major platforms DB | DW | Hadoop | Legacy On Premises | Cloud SAP | Mainframe Simplified Real-Time Universal Data Delivery with Attunity Replicate
  • 17. 17© 2017 Attunity 17© 2017 Attunity Attunity Replicate Architecture TRANSFER IN-MEMORY FILTER DATA LAKE RDBMS DATA WAREHOUSE FILES MAINFRAME TRANSFORM PERSISTENT STORE LOG BASED CDC BATCH INCREMENTAL BATCH RDBMS DATA WAREHOUSE STREAMING FILES ATTUNITY POWERPOINT ASSET LIBRARY DATA LAKE
  • 18. 18© 2017 Attunity TARGET SCHEMA CREATION SAP RDBMS EDW FILE MAINFRAME HETEROGENEOUS DATA TYPE MAPPING BATCH TO CDC TRANSITION DDL CHANGE PROPAGATION FILTERING TRANSFORMATIONS RDBMS EDW FILES STREAMING REPLICATE Attunity Replicate - End to End Automation DATA LAKE
  • 19. 19© 2017 Attunity 19© 2017 Attunity SOURCES CLOUD Amazon RDS (SQL Server, Oracle, MySQL, Postgres) Amazon Aurora (MySQL) Amazon Redshift Azure SQL Server M1 (Q1) ATTUNITY - PLATFORM SUPPORTABILITY MATRIX SAP ECC ERP CRM SRM GTS MDG S/4HANA (on Oracle, SQL, DB2, HANA) DATABASE Oracle SQL Server DB2 iSeries DB2 z/OS DB2 LUW MySQL PostgeSQL Sybase ASE Informix ODBC EDW Exadata Teradata Netezza Vertica Pivotal MAINFRAME DB2 z/OS IMS/DB VSAM FLAT FILES Delimited (e.g., CSV, TSV) TARGETS FLAT FILES Delimited (e.g., CSV, TSV) STREAMING Kafka Amazon Kinesis Azure Event Hubs MapR Streams SAP HANA EDW Exadata Teradata Netezza Vertica Sybase IQ SAP HANA Microsoft PDW GOOGLE Cloud SQL (MySQL, Postgres) Cloud Storage Dataproc PubSub (‘19) Big Query (Q2) DATA LAKE Hortonworks Cloudera MapR Amazon EMR Azure HDInsight Google Dataproc DATABASE Oracle SQL Server DB2 LUW MySQL PostgreSQL Sybase ASE Informix MemSQL Compose support AZURE DBaaS (SQL DB) DBaaS (MySQL, Postgres) ADLS BLOB HDInsight Event Hub SQL DW Snowflake (Q1) Databricks (Q2) AWS RDS (MySQL, Postgres, MariaDB, Oracle, SQL Server) Aurora (MySQL, Postgres) S3 EMR Kinesis Redshift Snowflake (Q1) Databricks (Q2) SaaS Salesforce (Q2)
  • 20. 20© 2017 Attunity 20© 2017 Attunity Attunity Replicate Demo
  • 21. Attunity Data Warehouse Automation Solution Overview
  • 22. 22© 2017 Attunity Accelerate your Azure Analytics Journey Data Warehouse Automation– Attunity Compose
  • 23. 23© 2017 Attunity 23© 2019 Attunity STAGING AREA -------- Trunc & Load EDW ------- 3NF DATA MART ------- Star Schema CRM ERP FINANCE LEGACY SOURCES ETL ETL ETL ETL ETL ETL ETL ETL ETL ETL That’s not exactly what I wanted Why is my data always X day(s) old BUSINESS / CONSUMER / REQUIREMENTS CHANGES IMPACT IMPACT IMPACT We don’t need that anymore Complex transformations Requirement / Source chg. Data quality & validation Manual Modelling Complex ETL design DevOps design Why Data Warehouse Automation ? Traditional Data Warehousing Methods are failing the business Complex Design Impact to source Bulk – not change data Long running extracts Batch/EOD Based Complex Build Long, manual coding effort Long testing cycles Slow to react to changes Time to Market
  • 24. 24© 2017 Attunity 24© 2017 Attunity AUTOMATED WORKFLOW Real-Time Extract Auto Extraction, Loading, Mapping Auto Generated Transformations Change Propagation Auto Design with Best Practices “DWA will accomplish an initial BI implementation up to five times faster than traditional methods”* *TDWI Data Warehouse Automation Course REAL-TIME ODS STAGING EDW MARTS Data Pipeline for data warehouses commit to model architecture Azure SQL DW Oracle SQL Server RedShift Snowflake ** DATA MOVEMENT AUTOMATION DATA LAKE / DATA WAREHOUSE AUTOMATION
  • 25. 25© 2017 Attunity 25© 2017 Attunity Compose for Data Warehouse Demo EDW MARTS MDM Sales Sales Service Ticket What can we do to better manage late shipments?
  • 26. 26© 2017 Attunity 26© 2017 Attunity CUSTOM MAPPINGS DATA MART DESIGN SOURCE MODEL Data Warehouse Model Generation Automated Mapping Generation Data Warehouse ETL Generation Error Mart Generation Data Mart (Star Schema) ETL Generation Workflow Generation & Orchestration Automates  Native CDC integration  E-LT Set based, best practice data loads  Transparent, editable E-LT  Surrogate Key  Type 1 / Type 2  Referential integrity (late arriving dimensions)  Error mart automation  Data validation  Automated Workflow & Dependencies DW MODEL Automates  Flexible physical model  3NF / Data Vault methodology  Transparent, editable DDL ROBUST E-LT DATA VALIDATION / QUALITY RULES STAR SCHEMAS Automates  Type 1 / Type 2 conformed dimensions  Automatic Incremental processing from DW  Automated flattening of dimensions  Granular, Aggregate & Time-Oriented Fact support Documentation & Deployment Compose for data warehouses Automation of complex data processing requirements
  • 27. Attunity Data Lake Automation Solution Overview
  • 28. 28© 2017 Attunity Accelerate your Azure Analytics Journey Data Lake Automation– Attunity Compose
  • 29. 29© 2017 Attunity 29© 2017 Attunity The Attunity Difference Automating Data Lake Ingestion DATA LAKE AUTOMATION “Current view example”
  • 30. 30© 2017 Attunity 30© 2017 Attunity 1. Land 2. Store 3. Provision Consume CAPTURE PARTITION ENRICH SUBSET STANDARDIZE MERGE FORMAT ANALYZE PREPARE CLEANSE JOIN Raw Deltas Full Change History ODS HDS Snapshot Data sets SAP RDBMS DATA WAREHOUSE FILES MAINFRAME Source Deliver Analytics Optimized Data Sets FOR DATA LAKES  Real-time high volume delivery  Consistent data  Write optimized format  Standardized historical view  Read optimized format  Automated at scale 1,000’s of source entities  Current / Type 2 / Snapshot  Read optimized format  Automated loads w/ Spark & Hive
  • 31. 31© 2017 Attunity 31© 2017 Attunity Data Lake Storage <bucket/container/folder> Source -> Landing -> Storage Data Flow Source Landing customer customer__ct .seq / .csv customer update customers set name = ‘Maria Anders’ where id = 1; Delete from customers where id = 2; Insert into customers values (3, ‘New Customer’); Storage customer customer__delta .snappy.parquet
  • 32. 32© 2017 Attunity 32© 2017 Attunity Data Lake Storage Provisioning *Storage Storage -> Provisioning Data Flow customer customer__delta HDS (“Type 2”) Snapshot (Point-in-time) ODS (“current”) .orc .snappy.parquet .snappy.parquet / .orc /.avro Compactor Task <Spark> *Each provisioning task has its own bucket / container / storage location
  • 33. 33© 2017 Attunity 33© 2017 Attunity Azure Data Lake Storage Azure Data Lake Architecture with Attunity Automated Data Ingestion and Provisioning Enterprise Data Sources APPS / OTHER RDBMS FILE SAP MAINFRAME Raw Transactiona l Data <SEQ/CSV> Standardized Historical Raw Data <Parquet> Provisioned Data Sets <Parquet/ ORC/Avro> HDInsight Attunity Replicate Batch Load CDC Attunity Compose for Data Lakes (Cloud VM or on prem) Compose Agent  Industry leading CDC and data ingestion with Attunity Replicate  Automate standardization and provisioning of consumer ready data sets with Attunity Compose  Automated handling of schema evolution across 1,000’s of entities Metadata / Instruction channel Data flow BI / Data Science etc.  Azure ADLS  Google Storage  HDFS  AWS S3  Azure HDInsight  Azure Databricks**  AWS EMR  Google DataProc  Hortonworks  Cloudera
  • 34. 34© 2017 Attunity 34© 2017 Attunity Compose for Data Lakes Demo
  • 35. 35© 2017 Attunity 35© 2017 Attunity AGILE DATA DELIVERY WHAT YOU CAN ACHIEVE WITH ATTUNITY COMPOSE High levels of satisfaction for business Significantly improved utilization of resources Maximized productivity Rapid adaption to business changes Vastly improved data quality, delivered real-time
  • 36. 36© 2017 Attunity 36© 2017 Attunity Trusted by 2000 Customers Worldwide And Half the Fortune 100 FIN. SERVICES MANUF. / INDUS. GOVERNMENTHEALTH CARE TECHNOLOGY / TELECOM OTHER INDUSTRIESRETAIL

Editor's Notes

  1. These are some of the industries and types of use cases where we’ve enabled digital transformation While Customer 360 isn’t an industry, it’s a use case that goes across all industries
  2. Industry standard architecture Kappa / Lambda architecture for on-prem or cloud based analytics Many customers don’t implement this entire architecture – only components that fit their use cases. Eg. Only a data warehouse Only a data lake A combination – a data lake that feeds a dw Typical challenge for implementing components of this architecture How do we get data ingested quickly? How do we conform data so its analytics and data scientist ready? How do we become agile in our data warehouse and data integration architecture? How can we automate these end to end processes? Attunity’s solutions provide easy to use, standardized methods for creating automated data pipelines for any aspect of this architecture. Ensuring you can meet your business needs while also providing flexibility to evolve your architecture over time. While our solutions don’t typically integrate or interact with the data consumers or data scientist community – those we do impact those data consumers and their ability to leverage right-time information that we automate and curate for them. Discuss the Attunity components and where they fit.
  3. Let’s look briefly at the architecture. Attunity Replicate is hosted on an intermediate Windows or Linux server that sits between one or more sources and one or more targets. We support one to one (one way or two way), one to many/many to one (hub and spoke) and logically independent bi-directional replication topologies. Data transfer is executed in memory. Attunity Replicate is primary focused on extracting and loading data, but does perform light filtering and transformations. Complex transformations are handled by Attunity Compose. We support a range of end points both on premises and in the cloud. In almost all cases we require no software to be installed on either source or target, which simplifies administration and minimizes impact on production applications. More on that to come.
  4. Attunity Replicate automatically generates target databases based on metadata definitions in the source schema. You can use a graphical task map to configure database schema mappings between heterogeneous sources and targets. CDC can run concurrently with a batch load, then continue upon batch completion to ensure targets remain up to date. Any DDL changes made to source schema, such as table/column additions or changes to data types, can be replicated dynamically to the target. You can define which data to replicate, filtering by column, value range or data type Users also can perform transformations such as the addition, deletion or renaming of target columns or the changing of data types.
  5. To understand why we have invested so much in our Data Warehouse Automation technology you have to understand the issues with the traditional method of deploying a dw. Traditional data warehouse processing doesn’t meet today’s business needs. Data is often consumed in batch with a large impact to source systems and only providing eod analytics. Modelling is manual process which often leads to a complex etl design and build. DW architects have to build custom frameworks to support DevOps and data quality and data validation All this results in a delayed time to market with long often manual coding efforts and long testing cycles By the time the business sees the output its often not what they truly wanted, not what they need or the data is not timely enough for them operationally. This leads to changes to requirements and a feedback loop that in turn impacts the end to end dw process.
  6. When we look at what delivering analytics and consumer ready data sets mean, we started by looking at our customers need. Ingest the data with low impact capture mechanisms and deliver in real-time to the lake. This requires a write-optimized format to keep up with data changes Customers also insist that as data is delivered even to data lakes – there is consistency there. We handle this via our built in partitioning mechanism. This is all handled by our best of breed cdc solution – Replicate. Customers want a standardized set of historical data that they can leverage to provision other data sets. This is our storage or assembly zone. It provides a standardized historical view of data delivered by Replicate But in a READ-OPTIMIZED Parquet format. We need to deliver this at scale and we leverage Spark to do so which is an increasing customer requirement. Customers also want to provision data sets and provide subset or enriched data to the consumers. This means being able to treat the data lake like a database and provide a current view, or a type 2 historical view with effective and end dates as well as point in time snapshots. For analytics consumers this means read-optimized again. Columnar formats like parquet or ORC. Automated at scale. This is handled by Compose. It understands the data delivered consistently by Replicate and automates the generation of spark flows to assemble and provision data – fulfilling customers analytics and read optimized requirements