The document outlines an architecture for data exposure in Azure, addressing a client's requirement for seamless access to over 6TB of data across 1000+ staging data warehouses while maintaining data security and isolation. It introduces three solution approaches: OData API, Data Abstraction, and Tabular Model, each with distinct benefits and limitations, such as support for various BI tools and challenges related to data extraction and integration. Ultimately, it assesses a hybrid solution that combines these approaches to optimize data access and analytics capabilities.

1. Data exposure in Azure:
Production use-case
April, 2018

2. 2
FEW WORDS ABOUT MYSELF…
I’m Alexander Laysha
• Solution Architect at EPAM Systems
• Co-Head of TR Cloud Center of Excellence at EPAM Systems
• Microsoft Azure MVP
• Focused on backend cloud solutions
• Leader of Belarus Azure Community
My contacts
• Email: layshaalex@gmail.com
• Twitter: @layshaalexander
• Facebook: alexander.laysha

3. 3
• Overview & Requirements
• OData API Solution
• Data Abstraction Solution
• Tabular Model Solution
• Summary
AGENDA

4. 4
OVERVIEW & REQUIREMENTS

5. 5
• Сustomer has 1000+ staging data warehouses
for it’s clients with 6TB of data in overall
• Customer clients has access to data from
staging data warehouses through OLAPs that
are consumed from web applications
CONTEXT
SSIS Server
- Orchestrator
- Packages
- Jobs
On-Premise
– OLTP Source
– 7 DB Servers
Master DB
Tenant Data
Warehouses
SSAS Servers
- Dashboard CUBES
- Benchmark CUBES
Web APP
– DASHBOARDS
- Reporting
Company
Databases
Benchmark
Database
SSIS Server
- Orchestrator
- Packages
- Jobs

6. 6
• Integration Client - would like to perform full & incremental extract of it's own raw data from
staging data warehouses
• BI Client - would like to connect to it’s data in staging data warehouses using own BI Tools
NEW USE-CASES

7. 7
• Data warehouses are exposed as read-only sources
• Client can’t connect to data warehouse directly because of customer security policies
• Source database schema change propagation
• Multi-tenant support & strong tenant data isolation
• Azure AD authentication
• Role-based data access
• Row-level security in the future
• Support of major BI Tools: PowerBI, Tableau, Qlik
• Clients should not wait hours to load their data
• Data warehouses should not be affected by load spikes
ARCHITECTURE REQUIREMENTS

8. 8
PROPOSED ARCHITECTURE APPROACHES
• OData API Solution – abstracts clients
from staging data warehouses and
provides integration points for clients
• Data Abstraction Solution – contains ETL
process to extract, transform and load
data into separate storage that acts as
an integration point for clients
• Tabular Model Solution – memory
optimized databased for analytical
workloads hosted in Analysis Service. It
extracts data from staging data
warehouses and provides integration
point for clients
Storage Area
Data
Warehouse
Tenant1
Data
Warehouse
Tenant2
Data
Warehouse
Tenant3
...
Data
Warehouse
TenantN
Consumers Area
Power BI Tableau Excel Any Compatible
Client
Exposure Area
OData API
Solution
Data
Abstraction
Solution
Tabular Model
Solution
Authentication/
Authorization

9. 9
ODATA API SOLUTION

10. 10
SOLUTION OVERVIEW
Storage Area
- Data Warehouses- Solution Components
Master DB
Data
Warehouse
Tenant1
Data
Warehouse
Tenant2
...
...
Data
Warehouse
TenantN
Exposure Area
Configuration Logging &
Monitoring
Consumers Area
Power BI Tableau Excel Any Compatible
Client
Authentication
BasicOpenID/OAuth 2.0
Authorization
Table-levelTenant-level
API
ODATA Engine (Maskx.Odata)
Data Access

11. 11
AZURE ARCHITECTURE
Client Autoscaling
Azure Key Vault
Master DB
Client Azure AD
API App
Instance #1
API App
Instance #N
Custom OData API
Storage Area
TCP
Application
Insights
HTTPSInternet
Authentication
Configuration Logging & monitoring
Data Warehouse
Tenant1
Data
Warehouse
TenantN
Azure
AD
(B2B or B2C)
Authentication
Data
Warehouse
Tenant2
Read-only replica

12. 12
• Most BI tools support an “extract” integration model when using ODATA API as a data source.
• No possibility to perform incremental extract, only reloading the entire dataset on schedule.
“Incremental load” feature is in development now.
• Power BI has strict 2-hour timeout that cannot be exceeded during the data import, which raises
additional concerns on exposing large data sets using the API.
• Power BI requires API endpoint and Azure AD tenant to be hosted under custom domain name to
work with Azure AD authentication.
• For Azure AD B2C prototyping we had to develop a custom policy using the Identity Experience
Framework, which may introduce additional risk since the feature is still in the public preview.
• Lack of developer documentation for the Identity Experience Framework: we had to submit issue
for assistance to Microsoft team.
SOLUTION LIMITATIONS

13. 13
Solution Pain Points
• BI tools use “extract” integration model when working with API.
• Most of BI tools are not able to extract data from API incrementally. Can be potentially mitigated by some pre-aggregation
on the database side.
Solution Benefits
• Simple on-boarding mechanism for new clients since HTTP-based integration does not introduce any dependency on the
implementation stack.
• Standardized mechanism of exposing datasets with their metadata. Good number of client libraries available for wide set
of programming languages.
• Ability for clients to select only subset of data.
• Solution allows adopting authentication and authorization logic already applied in customer company. Level of flexibility is
higher comparing to direct usage of Azure services like Azure Storage.
Optimal strategy: exposing Custom API along with alternative integration mechanism that could provide “direct query”
integration model.
SUMMARY

14. 14
DATA ABSTRACTION
SOLUTION

15. 15
SOLUTION OVERVIEW
Storage Area
- Data Warehouses- Solution Components
Data
Warehouse
Tenant1
Data
Warehouse
Tenant2
Data
Warehouse
Tenant3
...
Data
Warehouse
TenantN
ETL Area
Cross-cutting Area
Consumers Area
Power BI Tableau Excel Any Compatible
Client
Multi-tenant
ETL Engine
Tenant1
ETL Process
Tenant2
ETL Process
Tenant3
ETL Process
TenantN
ETL Process
...
Exposure Area
Exposed
Tenant1 DB
Exposed
Tenant2 DB
Exposed
Tenant3 DB ... Exposed
TenantN DB
Monitoring
Logging
Security

16. 16
AZURE ARCHITECTURE
DW
TenantN
DW
TenantN
TCP/IP
TCP/IP
...
Storage Area ETL Area
SQLAzureCluster
Pipeline per Tenant
Version of successfully
extracted data from every
sql table is stored in Version
Table of Tenant Storage
Account
Multi-tenant
Data Factory
Tenant 1
Storage Tables
Storage Account
per Tenant
Tenant N
Storage Tables
Storage Account
per Tenant
HTTPS
Exposure Area
...
Consumers Area
HTTPS
HTTPS
Power BI
Power BI
TenantNToolsTenant1Tools
HTTPS
Tenant1
AAD
Any Client
Any Client
Cross-cutting Area
Log Analytics
HTTPS HTTPS HTTPS
Tenant2
AAD
HTTPS
Access Control Monitor

17. 17
• Table Storage is supported only by PowerBI as data source.
• PowerBI supports only “extract” integration model using Table Storage as data source.
• Incremental data load to PowerBI is not supported thus PowerBI needs to reload the whole dataset during
refresh.
• Power BI has strict 2-hour timeout that cannot be exceeded during the data import.
• Number of storage accounts per Azure subscription is limited to 250 maximum.
• Storage Account does not support integration with AAD in area of authenticating users and authorizing access
to stored data.
• Row-level security is not supported by Table Storage.
• PowerBI supports authentication with Storage Account only using Account Key.
• Table storage (and storage account) might be throttled at high-scale (max 20.000 req/sec per storage account
for 1KB entity, 2000 req/sec per table partition).
SOLUTION LIMITATIONS

18. 18
Solution Pain Points (because of Storage Account)
• Filtering by columns not included into Partition and Row keys might lead to poor performance depending on data volume
• Absent of integration with AAD in area of authentication and authorization
• Supported only by PowerBI as a data source at the moment
• PowerBI use “extract” integration model when working with Table Storage
Solution Benefits (in case of use of SQL Azure as an Exposure Area)
• Integration with AAD for authentication and data access authorization
• Supported table and row-level security
• Supported by popular BI Tools using “direct mode” – BI Tool translates chart queries into sql queries and sends to SQL Azure
for execution
• Any client can connect to SQL Azure
• Allows to limit set of exposed tables by modifying ETL process or perform transformation of data during ETL into
materialized view with pre-aggregated data for better performance and lower DTU usage of SQL Azure
Optimal strategy: implementing Data Abstraction solution using SQL Azure or Azure Data Lake Store as an Exposure Area
SUMMARY

19. 19
TABULAR MODEL SOLUTION

20. 20
SOLUTION OVERVIEW
Storage Area
- Data Warehouses- Solution Components
Data
Warehouse
Tenant1
Data
Warehouse
Tenant2
Data
Warehouse
Tenant3
...
Data
Warehouse
TenantN
Exposure Area Cross-cutting Area
Consumer Area
Power BI Tableau Excel Any Compatible
Client
Multi-tenant Analytical Data Engine
Tenant1
Tabular
Model
Monitoring
Logging
Security
Tenant2
Tabular
Model
Tenant3
Tabular
Model
Analytical Data Engine
TenantN
Tabular
Model

21. 21
AZURE ARCHITECTURE
Multi-Tenant
Analysis Service
DW
Tenant 2
DW
Tenant 3
Single Tenant
Analysis Service
TCP/IP
TCP/IP
Storage Area Exposure Area
SQLAzureCluster
HTTPS
In-Memory Model
Per Tenant
One In-Memory Model
AAD
DW
Tenant 1
asazure://
asazure://
TenantNToolsTenant1Tools
Consumers Area
Power BI
Power BI
HTTPS
Any Client
Any Client
Monitor
Cross-cutting Area
Log Analytics
HTTPS HTTPS
Access Control

22. 22
• Expensive for huge data volumes (5.920$ for 100GB).
• Supports authentication only for organizational accounts that are members of default AAD
of subscription where Analysis Service resides.
• Isn’t supported by AWS QuickSight.
• PowerBI doesn’t allow to create/modify relations for model imported from Analysis
Service. All relations, measures, calculations and other entities should be defined in Tabular
model.
SOLUTION LIMITATIONS

23. 23
Solution Pain Points:
• Expensive… ~3000$/month for 50GB and 200 QPUS
Solution Benefits
• Can be queried directly using BI Tool or custom application using DAX queries
• Supported by popular BI Tools like PowerBI, Tableau, Qlik
• Supports integration with AAD
• Pure PaaS offering that can scale-out if needed
• Can ingest data from multiple sources
• Powerful role-based security implementation that is supported on all levels: model, table, row
• Even big source databases (>100GB) can nicely fit into Analysis Service for analytics scenarios in a cost effective way by
extracting only needed information and aggregated data (materialized views)
Optimal strategy: use AAS for analytical purposes along with integration mechanism that could provide better approach for raw
data extraction in cost effective way
SUMMARY

24. 24
SUMMARY

25. 25
SOLUTIONS COMPARISON
OData API Solution Data Abstraction Solution Tabular Model Solution
Scenarios
Integration scenario suitability High Medium Medium
Analytics scenario suitability Low Low High
Quality attributes
Authentication High Low High
Authorization High Low High
Sync with source High Medium Medium
Maintainability Medium Medium Medium
Schema Medium High High
Infrastructure cost Low Low High
Scalability High High High

26. 26
HYBRID SOLUTION
Multi-Tenant
Analysis Service
Monitor
DW
Tenant1
DW
TenantN
Tenant N
Analysis Service
asazure://
asazure://
... ...
Storage Area Exposure Area
Consumers
SQLAzureCluster
Power BI
Power BI
Cross-Cutting Area
Log Analytics
HTTPS
TenantNToolsTenant1Tools
In-Memory Model per tenant
In-Memory Model
HTTPS
...
Application
Insights
Master DB
Multi-tenant OData API
API App
Instance #1
API App
Instance #N
Key Vault
HTTPS
TCP/IP
Access control
OData client
Pre-aggregated
model
QuickSight
TCP/IP,
HTTPS
Data Factory
ETL Area
Autoscaling
Azure AD

27. 27
THANK YOU!