Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Analytics as a Service
with MariaDB AX and Microsoft Azure
Alexander Aldev, CTO
Agenda
● A little quiz
● Objectives
● Business Case and Requirements
● On-premises Reporting Architecture
● Integrating Ma...
Business Case
The Problem
● UK-based loyalty programme operator
● ~ 70 mln cards and vouchers
● ~ 120 mln transactions
● Limited reporti...
System Requirements
● BUDGET is very tight!
● Analytical system must be available online
● Customers may not see each othe...
Reporting Requirements
● Printable reports distributed automatically
● Online report templates
● Ad-hoc reporting tool
● E...
On-premises Solution
On-Premises System Architecture
This Photo by Unknown Author is licensed under CC BY-NC-SA
On-Premises System Architecture
Icons made by https://thoseicons.com
file server
extract delivery
MariaDB AX
DWH
E-LT proc...
Why MariaDB AX?
The Good
● Affordable DWH infrastructure
● Mature analytics product
● Amazing performance for OLAP queries...
MariaDB AX
The Data Flow
core CM system
extract changed
records
FS store
incremental
extracts
Staging Area
1:1 load with
C...
Data Modeling Primer
Why Bother Transform Data?
Transactional Workloads
● Normalized
● Referential integrity
● Consistency
● Read/write access ...
The Star-Schema Query
card
issuer
insurance_claim
claim_date transaction
transaction_type
transaction_date
(SELECT is.name...
Moving to The Cloud
Cloud-based System Architecture (Migration)
Icons made by https://thoseicons.com
file server
extract delivery
MariaDB AX
D...
Opportunities for Improvement (aka Issues)
● Many moving parts, difficult and expensive to maintain
● High licensing cost ...
Microsoft Azure
● Azure Analysis Services: metadata store, Excel & ad-hoc reporting, row-
level security
● Power BI: dashb...
Cloud-based System Architecture (Target)
Icons made by https://thoseicons.com
file server
extract delivery
MariaDB AX
DWH
...
Data Integration
Integration MariaDB AX - Azure AS
● Tabular mode: Azure Analysis Services only provides tabular mode
● MySQL connection: u...
Steps to Connect to MariaDB AX
● Download and Install On-Premises Data Gateway (Windows box)
● Register On-Premises Data G...
Incremental Data Load
● Table Partitions: separate queries, each loaded independently
● ProcessAdd: insert new rows to exi...
Front-End
User Security
● Azure AD B2B collaboration: external email for user identification
● Federated SSO: external identity prov...
Report Delivery
● Live Reports: Excel templates based on a pivot table with a live connection
to Azure AS instance, refres...
Future Work
DirectQuery Mode
Approach 1: HDInsight Spark Integration
● Create custom DataSource to interface
MariaDB AX PM storage dir...
Q & A
Thank you!
Upcoming SlideShare
Loading in …5
×

M|18 Analytics as a Service

222 views

Published on

M|18 Analytics as a Service

Published in: Data & Analytics
  • Be the first to comment

M|18 Analytics as a Service

  1. 1. Analytics as a Service with MariaDB AX and Microsoft Azure Alexander Aldev, CTO
  2. 2. Agenda ● A little quiz ● Objectives ● Business Case and Requirements ● On-premises Reporting Architecture ● Integrating MariaDB AX with MS Azure AS ● Incremental Data Loading ● Authentication and Data Security ● Report Delivery ● Future Work ● Q & A
  3. 3. Business Case
  4. 4. The Problem ● UK-based loyalty programme operator ● ~ 70 mln cards and vouchers ● ~ 120 mln transactions ● Limited reporting functionality in core CM system ● External customer reports take a lot of time and effort ● On-line analytics could be a paid service ● Soooo … could we get a self-service analytics platform?
  5. 5. System Requirements ● BUDGET is very tight! ● Analytical system must be available online ● Customers may not see each other’s data ● Internal analysts need to see and compare all data ● Some attributes should not be visible to analysts ● Analytical queries should not impact the core system ● Same user credentials should be used for core and analytical system
  6. 6. Reporting Requirements ● Printable reports distributed automatically ● Online report templates ● Ad-hoc reporting tool ● Each level of service priced separately ● … and, btw, can we also have our transaction records in Excel?
  7. 7. On-premises Solution
  8. 8. On-Premises System Architecture This Photo by Unknown Author is licensed under CC BY-NC-SA
  9. 9. On-Premises System Architecture Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery MySQL Proxy query rewriting decryption MS SSAS metadata (R)OLAP XML/A interface Active Directory intl authentication Web Portal SSO ext authentication authorization content delivery XML/A Proxy authorization b/w intl and ext ID Excel! reports pivot tables MariaDB TX users data security contents
  10. 10. Why MariaDB AX? The Good ● Affordable DWH infrastructure ● Mature analytics product ● Amazing performance for OLAP queries ● Good performance for data integration ● Easy to scale storage ● Integration with MS SSAS works well The Ugly ● External driver needed for full VS and SSAS integration ● SSAS sends SELECT DISTINCT * when populating (large) dimensions ● Using ROLAP storage in SSAS against AX requires a bit of tinkering around ● Dialect differences with MariaDB TX
  11. 11. MariaDB AX The Data Flow core CM system extract changed records FS store incremental extracts Staging Area 1:1 load with CPImport Target Area normalized encrypted full history DM Area star-schema snapshot of current state MD Cube metadata hierarchies data security FStrigger CPImport SQL SQL SQL MDX FTP Icons made by https://thoseicons.com
  12. 12. Data Modeling Primer
  13. 13. Why Bother Transform Data? Transactional Workloads ● Normalized ● Referential integrity ● Consistency ● Read/write access to a few rows per query ● 1000s of TPS (ms response time) Analytical Workloads ● De-normalized ● Attributes from different domains ● Filtering and aggregation of millions of rows per query ● 10s of TPS (s response time)
  14. 14. The Star-Schema Query card issuer insurance_claim claim_date transaction transaction_type transaction_date (SELECT is.name, SUM(cl.amount) claims FROM insurance_claims cl JOIN issuer is ON is.issuer_id = cl.issuer_id JOIN claim_date dt ON dt.date_id = cl.date_id WHERE dt.month_id = 201801 GROUP BY 1) t1 (SELECT is.name, SUM(tr.amount) topups FROM trans tr JOIN issuer is ON is.issuer_id = tr.issuer_id JOIN transaction_date dt ON dt.date_id = cl.date_id JOIN transaction_type tp ON tp.type_id = tr.type_id WHERE dt.month_id = 201801 AND tp.type_cd = ‘Top-up’ GROUP BY 1) t2 SELECT t1.name, COALESCE(t1.claims,0.00), COALESCE(t2.topups,0.00) FROM FULL JOIN ON t1.name = t2.name
  15. 15. Moving to The Cloud
  16. 16. Cloud-based System Architecture (Migration) Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery MySQL Proxy query rewriting decryption MS SSAS metadata (R)OLAP XML/A interface Active Directory intl authentication Web Portal SSO ext authentication authorization content delivery XML/A Proxy authorization b/w intl and ext ID Excel! reports pivot tables MariaDB TX users data security contents
  17. 17. Opportunities for Improvement (aka Issues) ● Many moving parts, difficult and expensive to maintain ● High licensing cost of Windows components (SSAS, Server, AD) ● Requires a dedicated Windows OPS person ● Security concerns by client ● Multidimensional model in SSAS suboptimal fit for “lists of records”
  18. 18. Microsoft Azure ● Azure Analysis Services: metadata store, Excel & ad-hoc reporting, row- level security ● Power BI: dashboards, pretty and pixel-perfect reports (via export to PPT) ● SharePoint: content sharing ● Azure AD B2B: single sign-on and integrated authentication ● VM instances: optional, MariaDB AX can also be on-premise ● Pay-as-you-go model: no capex, low integration cost
  19. 19. Cloud-based System Architecture (Target) Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery Azure AS metadata tabular model XML/A interface Azure AD B2B SSO authentication Excel! reports pivot tables PowerBI dashboards Sharepoint content delivery
  20. 20. Data Integration
  21. 21. Integration MariaDB AX - Azure AS ● Tabular mode: Azure Analysis Services only provides tabular mode ● MySQL connection: uses MySQL Connector/Net ● On-premises Data Gateway: required for a MySQL connection ● No DirectQuery support: only supported for MS, Oracle, SAP, Teradata, etc MariaDB AX On-premises Data Gateway (Windows x64) Azure AS Icons made by https://thoseicons.com
  22. 22. Steps to Connect to MariaDB AX ● Download and Install On-Premises Data Gateway (Windows box) ● Register On-Premises Data Gateway ● Add GW in Azure AS instance ● Download and Install Connector/Net 6.x (on the same Windows box) ● In Visual Studio: New AS Tabular Project -> Import Data -> MySQL ● Specify TCP/IP connection setting Icons made by https://thoseicons.com
  23. 23. Incremental Data Load ● Table Partitions: separate queries, each loaded independently ● ProcessAdd: insert new rows to existing partition ● Our Design: multiple partitions (card_id) for large tables: cards, transactions ● Native Query DAX functions: using AES_DECRYPT ● Daily process: Automated ProcessAdd TMSL script for all partitions ● Monthly process: Re-process partitions with added records above a threshold
  24. 24. Front-End
  25. 25. User Security ● Azure AD B2B collaboration: external email for user identification ● Federated SSO: external identity provider based on SAML ● User provisioning: automatically generated PowerShell script using New- AzureADMSInvitation from list of users provided from CM system ● Data security: using Analysis Services roles ● Roles: one role per client organization, DAX formula [Client ID] = <literal> ● Role management: automatically generated TMSL script from user data ● Role membership: automatically generated TMSL script from user data or manually through SSMS
  26. 26. Report Delivery ● Live Reports: Excel templates based on a pivot table with a live connection to Azure AS instance, refreshed with client data on access ● Delivery platform: templates published on Sharepoint Online ● Dashboards: PowerBI connected to Azure AS instance ● Printed Reports: experimenting with PowerBI export as PowerPoint, immature functionality, need to find a solution
  27. 27. Future Work
  28. 28. DirectQuery Mode Approach 1: HDInsight Spark Integration ● Create custom DataSource to interface MariaDB AX PM storage directly ● use Spark as a SQL engine Approach 2: TDS protocol handler ● Create MaxScale protocol plugin to handle TDS connections ● Cheat Azure AS that is sees a SQL Server PM PM UM MariaDB AX E-LT Spark Azure AS E-LT MariaDB AX MaxScale Azure AS TDSprotocol Qryrewrite DataSource TDS MySQL
  29. 29. Q & A
  30. 30. Thank you!

×