This session focuses on the needs of the data integrator and data engineer whether that be for data warehousing & BI, advanced analytics of data for SaaS applications. We walk through a comprehensive set of new additions to Azure Data Factory and SSIS for moving and integrating data across on-premises and cloud. Topics and examples will include simple, scalable and reliable data pipelines in ADF using a serverless, parallel data movement service to/from the cloud, provisioning of Azure-SSIS Integration Runtime (IR) – dedicated servers for lifting & shifting SSIS packages to cloud– and its customization with your own/3rd party extensions, the execution of SSIS packages as first-class activities in ADF pipelines and their combination with other ADF activities to create modern ETL/ELT workflows all through the new code-free experience.

4. Modern DW for BI
Modern DW for SaaS
Lift & Shift Existing SSIS Packages to Cloud

5. Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure storage
Polybase
Azure SQL Data Warehouse
Data factory
Data factory
Azure Databricks
(Spark)
Analytical dashboards
(PowerBI)
Model & ServePrep & TrainStoreIngest Intelligence
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING
Azure Analysis ServicesOn Prem, Cloud
Apps & Data

6. Business / custom apps
(Structured)
Logs, files and media
(unstructured)
Azure storage
Polybase
Data factory
Data factory
Azure Databricks
(Spark)
Browser/Device
Model & ServePrep & TrainStoreIngest Intelligence
AZURE DATA FACTORY ORCHESTRATES DATA PIPELINE ACTIVITY WORKFLOW & SCHEDULING
App Storage
SaaS App

7. On-Premise data sources
SQL DB Managed Instance
SQL Server
VNET
Azure Data Factory
Cloud data sources
Cloud
On-premises

9. Flexible Pipeline Model
Rich pipeline orchestration
Triggers: on-demand, schedule, event
Data Movement as a Service
Cloud, Hybrid
70+ connectors provided
SSIS Package Execution
In a managed cloud environment
Use familiar tools, SSMS & SSDT
Author & Monitor
Programmability (Python, .NET, Powershell, etc.)
Visual Tools

10. SSIS
Package
Pipeline

11. Modern DW for BI

12. 140 MillionPassengers are stranded
annually due to IROPS
IRREGUL AR OPERATIONS (IROPS) COST US AIRLINES
8.3 BillionIn revenue
USD
250Rebooking per passenger
from a cancelled flight
USD
4,000For rebooking
crew per airline
USD
GLOBALLY Passengers most frustrated by a lack of communication
51%
USD 16.7 Billion
Estimates cost of passengers’ time
lost due to IROPS
Loss in demand due to IROPS
USD 3.9 Billion
In annual costs to Airlines and customers
USD60Billion
Airline Pain Points
Poor Communication | Lost Revenue | High Re-accommodational Costs |
Upset Customers | Brand Damage
Airline IROPS (IRREGULAR OPERATIONS)
The Billion Dollar Problem Airlines Must Solve

13. Activity 1 Activity 2
Activity 3
“On Error”
Activity 1
params
params
My Pipeline 1
…
My Pipeline 2
For Each…
Activity 4
params
Trigger
Event
Wall Clock
On Demand
params

14. Scalable
Per job elasticity
Up to 1 GB/s
Simple
Visually author or via code (Python, .Net, etc)
Serverless, no infrastructure to manage
Access all your data
70+ connectors provided and growing (cloud, on premises, SaaS)
Data Movement as a Service: 20+ points of presence world wide
Self-hostable Integration Runtime for hybrid movement
Data Movement

15. Azure Database File Storage NoSQL Services and Apps Generic
Azure Blob Storage Amazon Redshift SQL Server Amazon S3 Couchbase Dynamics 365 Salesforce HTTP
Azure Data Lake
Store
Oracle MySQL File System Cassandra Dynamics CRM Salesforce Service
Cloud
OData
Azure SQL DB Netezza PostgreSQL FTP MongoDB SAP C4C ServiceNow ODBC
Azure SQL DW SAP BW SAP HANA SFTP Oracle CRM Hubspot
Azure Cosmos DB Google BigQuery Informix HDFS Oracle Service
Cloud
Marketo
Azure DB for
MySQL
Sybase DB2 SAP ECC Oracle Responsys
Azure DB for
PostgreSQL
Greenplum MariaDB Zendesk Oracle Eloqua
Azure Search Microsoft Access Drill Zoho CRM
Salesforce
ExactTarget
Azure Table
Storage
Hive Phoenix Amazon
Marketplace
Atlassian Jira
Azure File Storage Hbase Presto Megento Concur
Impala Spark PayPal QuickBooks Online
Vertica Shopify Xero
GE Historian Square
Web table
* Supported file formats: CSV, AVRO, ORC, Parquet, JSON

17. Data Movement Performance Optimization
Copy Scenario Behind the Scenes
Loading data into Azure Cosmos DB
• 10+ times throughput boost comparing to previous solution
• Scalable to utilize 100% Request Units (Rus) during ingestion
• Improved reliability
Loading from Azure Blob or ADLS into
SQL DW
Polybase used whenever possible. BULKINSERT otherwise
Applicable when format is text, ORC, Parquet and meets these criteria.
Loading from data sources other than
Azure Blob/ADLS into SQL DW
Use staged copy via Azure Blob
Loading data from Amazon Redshift
Use UNLOAD to copy data from Amazon Redshift to Amazon S3

18. Data Movement Security
Command
Channel
Data
Channel
Data
Channel
Local

20. On-premises data sources SQL Server
 OS: Windows/Linux
 SCALABILITY: Scale-Out feature
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS to design/deploy/
manage/execute/monitor packages
 EXTENSIBILITY: ISVs can build
components/extensions on SSIS
 PRICING: Bundled w/ on-prem SQL Server

21. On-premises data sources
Azure SQL DB/Managed Instance
VNet
Azure Data FactoryCloud data sources
Cloud
On-premises
SQL Server
 LIFT & SHIFT: Use Azure SQL DB/Managed
Instance (MI) to host SSISDB
 SCALABILITY: Use ADF to provision a
managed cluster of Azure VMs dedicated to
run your packages – Azure-SSIS Integration
Runtime (IR)
 EDITION: Standard/Enterprise
 TOOLS: SSDT/SSMS + ADF app to
design/deploy/manage/execute/monitor
packages (activities)
 EXTENSIBILITY: ISVs can build
components/extensions + SaaS on SSIS in
ADF via custom setup + 3rd party licensing
 PRICING: Pay per hour + Azure Hybrid Benefit
(AHB) to Bring Your Own License (BYOL)

22. On-premises data sources
Azure SQL DB/Managed Instance
VNet
Azure Data FactoryCloud data sources
Cloud
On-premises
SQL Server
 HYBRID: Join Azure-SSIS IR to a VNet that is
connected to your on-prem network to
enable on-prem data access
 MODERNIZATION: Schedule first-class SSIS
activities in ADF pipelines via SSMS and
chain/group them w/ other activities via ADF
app
 COMPLEMENTARY: Splice/inject built-in/3rd
party SSIS tasks and transformations in ADF
pipelines
 READINESS: Public Preview w/ 24/7 live-site
support

25. Azure-SSIS IR node
Container
ISV Setup1. Specify Product Key in setup script ISV Activation Server
2. Get Activation Key by submitting Cluster ID + Product Key
Local Store
(e.g. Registry)
3. Write Activation Key
SSIS Executor
ISV Extension
4. Read Activation Key and
validate it with Cluster ID
Setup
Runtime
4. Get Cluster ID
4. Report on Node Count (Optional)
SSIS Runtime
2. Activation Key

29. Area Details
Control Flow • Control Flow-Released!!
• Azure Data Bricks Integration-Released!!
• File arrival trigger-Released!!
• Schedule roll up
• 20 plus activities and growing
Data Transformation Visual author data transformation executed with Spark
Data Movement • More connectors (Hybrid data movement)
• More geographic regions
• Continue performance Improvements (10x increase for CosmosDB connector)-Released!!
SSIS • More Region and VM Choices
• Enterprise Edition-Released!!
• Azure Hybrid Benefit (BYOL)-Released!!
• Custom Setup + 3rd Party Licensing/Extensibility/Ecosystem-Released!!
• First-Class SSIS Activities in ADF Pipelines-Released!!
Tools Visual Control flow, data movement and monitoring-Released!!
Programmatic Interfaces (.NET, Python, Powershell, Rest APIs, ARM templates)-Released!!
Source Control Integration- VSTS GIT & GitHub
Monitoring Single pane of glass to monitor & manage your pipelines