In this session you will learn how Qlik’s Data Integration platform (formerly Attunity) reduces time to market and time to insights for modern data architectures through real-time automated pipelines for data warehouse and data lake initiatives. Hear how pipeline automation has impacted large financial services organizations ability to rapidly deliver value and see how to build an automated near real-time pipeline to efficiently load and transform data into a Snowflake data warehouse on AWS in under 10 minutes.

1. Accelerate and Modernize Your
Data Pipelines
with Qlik Data Integration
February 26, 2020
Tim Garrod
Analytics Data Architect

2. © 2019 QlikTech International AB. All rights reserved.
3
IaaS
PaaS
Micro
Services
DB
MF
EDW
FILES
DWaaS
DB
MF
EDW
FILES
IaaS
PaaS
SaaS
DB
MF
EDW
FILES
Data Lake
DATA
CONSUMPTION
& ANALYTICS
Trends Driving Integration Modernization & Automation
CLOUD APPLICATION DEVELOPMENT
Legacy application modernization
Faster, easier new application
development
Higher scalability/elasticity
Infrastructure and maintenance cost
savings
Requires real-time data from
on-premise systems
DATA WAREHOUSE MODERNIZATION
Reduce the costs associated with
legacy EDW’s and provide elasticity
Meet new business requirements
Support more advanced analytics
Data Warehouse Automation
replaces traditional ETL with modern
self-service capabilities
Requires real-time data from on-
premise systems and cloud platforms
NEXT GENERATION ANALYTICS & DATA
MONETIZATION
Analyze a broader set of data structures
along with structured data
Faster and improved decision making
Leverage AI/ML, IoT and decision
automation for a competitive advantage
Requires Managed Data Lake Creation
and Big Data processing at scale
Requires real-time data from on-
premise systems and cloud platforms

3. 4
Data Architecture evolution
Bulk data movement
Brittle hand-coded ETL
Monolithic / appliance driven architectures (one size fits all)
Slow time to market / react to change
Legacy Data Warehouse Architecture
✓ Real-time data movement
✓ Automated design and code generation
✓ Use-case driven scalable technologies
✓ Faster Time to Market / Value
Modern Analytics Data Management
Architecture
Platform
Development lifecycle?

4. 5
Qlik Data Integration
Automated Data Pipelines
• Data Movement Automation
- Low impact, real-time data replication
- Heterogeneous sources & targets
• Data Lake Automation
- Automated schema evolution
- Analytics ready data provisioning
• Data Warehouse Automation
- Agile end-to-end dw lifecycle
- ETL code generation
• Operational Data Catalog
- Smart data catalog w/ intelligent tagging
- Shop & Publish
Heterogeneous Low Code Platform
Enterprise Manager
CENTRALIZED OPERATIONAL MANAGEMENT, CONTROL & ALERTS
MAINFRAME
SAAS
FILES
RDBMS
NoSQL
SAP
Replicate
REAL-TIME / BATCH
CHANGE DATA CAPTURE
CATALOG > PROFILE > ENRICH > SHOP > PUBLISH
Data Catalyst
Compose for
Data Warehouses
STAGING > EDW > MARTS
DATA WAREHOUSE
AUTOMATION
Compose for
Data Lakes
RAW > ASSEMBLE > PROVISION
DATA LAKE
AUTOMATION
STREAM GENERATION
STREAMING &
RDMBS Platforms
HETEROGENEOUS
DATA REPLICATION
DATA
INTELLIGENCE
ENTERPRISE DATA
SOURCES
formerly

5. 7
QDI - Platform Supportability Matrix
SOURCES TARGETS
▪ Oracle
▪ SQL Server
▪ DB2 iSeries
▪ DB2 z/OS
▪ DB2 LUW
▪ MySQL
▪ PostgeSQL
▪ Sybase ASE
▪ Informix
▪ ODBC
▪ MongoDB
▪ DB2 z/OS
▪ IMS/DB
▪ VSAM
▪ COBOL Copybooks
▪ ECC
▪ ERP
▪ CRM
▪ SRM
▪ GTS
▪ MDG
▪ S/4HANA
▪ (on Oracle, SQL, DB2, HANA)
Database Mainframe SAP
▪ Exadata
▪ Teradata
▪ Netezza
▪ Vertica
▪ Pivotal
▪ Amazon RDS
(SQL Server, Oracle,
MySQL, Postgres)
▪ Amazon Aurora (MySQL)
▪ Amazon Redshift
▪ Azure SQL Server MI
▪ Google Cloud SQL
▪ (MySQL, Postgres)
▪ SalesforceEDW
Cloud
SaaS
▪ XML
▪ JSON
▪ Delimited
(e.g., CSV, TSV)
Flat Files
▪ RDS (MySQL, Postgres, MariaDB,
Oracle, SQL Server)
▪ Aurora (MySQL, Postgres)
▪ S3
▪ EMR
▪ Kinesis
▪ Redshift
▪ Snowflake
▪ Databricks (Q2)
▪ DBaaS (SQL DB)
▪ DBaaS (MySQL, Postgres)
▪ ADLS
▪ BLOB
▪ HDInsight
▪ Event Hub
▪ Synapse aka SQL DW
▪ Snowflake
▪ Databricks
▪ Cloud SQL (MySQL, Postgres)
▪ Cloud Storage
▪ Dataproc
▪ PubSub (‘20)
▪ Big Query (Q2)
AWS Azure Google
▪ Hortonworks
▪ Cloudera
▪ MapR
▪ Amazon EMR
▪ Azure HDInsight
▪ Google Dataproc
▪ Kafka
▪ Amazon Kinesis
▪ Azure Event Hubs
▪ MapR Streams
▪ Exadata
▪ Teradata
▪ Netezza
▪ Vertica
▪ Sybase IQ
▪ SAP HANA
▪ Microsoft PDW
Data Lake
Streaming
EDW
▪ Oracle
▪ SQL Server
▪ DB2 LUW
▪ MySQL
▪ PostgreSQL
▪ Sybase ASE
▪ Informix
▪ MemSQL
Database
▪ HANA
SAP
▪ Delimited
(e.g., CSV, TSV)
Flat Files
1. DL/DW Automation

6. 8
Solution Demo
Build an Automated Pipeline to an
AWS Data Lake

7. 9
Unlock your
Operational Data
Streaming CDC for analytics

8. 10
Free your data
Streaming CDC data delivery

9. 12
Vanguard
Stream Processing & Data Lake Hydration in AWS
• Largest provider of
Mutual Funds in the
world
• Over $5.3 trillion in AUM
Challenges
• Reduce mainframe MIPS
• Modernize data architecture in AWS to
• enable stream processing
• reduce friction for data lake hydration
Solution
• Qlik Data Integration
• Replicate for delivery
to stream based
processing in AWS
Kinesis
• Fan-out delivery to S3
for lake hydration
**Qlik Replicate –
formerly Attunity
Replicate
Results
• Fan-out architecture for streaming
• Modernize “data-lake hydration” and ingestion
AWS re:Invent 2019: Data platform engineering: How Vanguard is migrating data to AWS

10. 13
Managed Data
Lake Creation
Remove the friction associating with
loading and managing a data lake

11. 14
Deliver Analytics Ready Data Sets
Automated Spark data pipelines
Automated Schema Evolution
Analytics-ready data sets

12. 17
Large Multinational Bank
Expand data access via Data Lake
• Large multinational bank
• Over 25 million customers
and over $975bn in
assets
Challenges
• Enable data management and access to more data in the data lake
• Deliver data that is ready for use
• Extraction of data from heterogeneous data sources to support batch and
event-driven workloads
Solution
• QDI Managed Data Lake
Solution
• Replicate for mainframe,
Oracle and other
heterogeneous data
sources
• Compose for Data Lakes
for data lake
management
Results
• Low impact, low latency capture from heterogeneous source systems
• Enable lambda architecture (streaming via Kafka and batch with hive)
• Transactional data stitching automated by Compose for Data Lakes
• Reduction in time spent building scripts for data lake stitching allowed
resources to focus on curating data through the Silver/Gold/Mastered layers
Large
Bank

13. 18
Accelerate Cloud
Data Warehousing

14. 19
Qlik Data Warehouse Automation
Redshift
Azure Synapse

15. 20
Compose for Data Warehouse Automation
Reduce Time to Market with Automation
• End-to-end dw lifecycle
management
• Automated E-LT code generation
• Architected for near real-time
processing with Replicate
ingestion

16. 21
Top Investment Management platform provider
Modernize Analytics Pipelines
• Leading provider of front
and middle office
investment management
platform
• The providers platform is
relied on by investment
firms, wealth managers,
hedge funds and insurers
in more than 30 countries
to manage more than
US$30 Trillion in AUM.
Challenges
• Deliver right-time data and analytics for investment firms
• Build differentiated client value via DW & Analytics as a Service
• Complex data processing requirements for Accounts, Securities, Positions and
Transactions
• Platform agnostic requirement as they evaluate cloud data platforms
Solution
• QDI Data Warehouse
Automation solution
• Replicate
• Compose for Data
Warehouses
Results
• Delivery of first data warehouse solution within weeks instead of months /
years
• Agile solution that provides self-service capabilities to their customers with
improved data quality
• Ability to migrate to new cloud environments with no recoding effort

17. 22
Data Catalyst – Increasing Data Intelligence
The Smart Data Catalog
Semi-Structured
Data
Automated, 1 step process
Complete profiling and validation
Rule-Driven Tagging
Semi-structured Data
Data Relationships
Integrated security
Analytical
Database
Analytical
Datawarehouse
Analytical
Data Lake
Enterprise
Data Sources
Compose for
Data Warehouses
STAGING > EDW > MARTS
Compose for
Data Lakes
RAW > ASSEMBLE > PROVISION

18. 23
Data Catalyst – Increasing Data Intelligence
The Actionable Data Catalog
Find what you’re looking for
Take action
Understand the data assets

19. 24
Accelerating the data supply chain

20. 25
Trusted by over 2500 customers worldwide
And Half the Fortune 100
FIN. SERVICES MANUF. / INDUS. GOVERNMENTHEALTH CARE
TECHNOLOGY / TELECOM OTHER INDUSTRIESRETAIL

21. Learn more about Qlik Data Integration
qlik.com/dataintegration