Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

M|18 Analytics as a Service

100 views

Published on

M|18 Analytics as a Service

Published in: Data & Analytics
  • Be the first to comment

M|18 Analytics as a Service

  1. 1. Analytics as a Service with MariaDB AX and Microsoft Azure Alexander Aldev, CTO
  2. 2. Agenda ● A little quiz ● Objectives ● Business Case and Requirements ● On-premises Reporting Architecture ● Integrating MariaDB AX with MS Azure AS ● Incremental Data Loading ● Authentication and Data Security ● Report Delivery ● Future Work ● Q & A
  3. 3. Business Case
  4. 4. The Problem ● UK-based loyalty programme operator ● ~ 70 mln cards and vouchers ● ~ 120 mln transactions ● Limited reporting functionality in core CM system ● External customer reports take a lot of time and effort ● On-line analytics could be a paid service ● Soooo … could we get a self-service analytics platform?
  5. 5. System Requirements ● BUDGET is very tight! ● Analytical system must be available online ● Customers may not see each other’s data ● Internal analysts need to see and compare all data ● Some attributes should not be visible to analysts ● Analytical queries should not impact the core system ● Same user credentials should be used for core and analytical system
  6. 6. Reporting Requirements ● Printable reports distributed automatically ● Online report templates ● Ad-hoc reporting tool ● Each level of service priced separately ● … and, btw, can we also have our transaction records in Excel?
  7. 7. On-premises Solution
  8. 8. On-Premises System Architecture This Photo by Unknown Author is licensed under CC BY-NC-SA
  9. 9. On-Premises System Architecture Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery MySQL Proxy query rewriting decryption MS SSAS metadata (R)OLAP XML/A interface Active Directory intl authentication Web Portal SSO ext authentication authorization content delivery XML/A Proxy authorization b/w intl and ext ID Excel! reports pivot tables MariaDB TX users data security contents
  10. 10. Why MariaDB AX? The Good ● Affordable DWH infrastructure ● Mature analytics product ● Amazing performance for OLAP queries ● Good performance for data integration ● Easy to scale storage ● Integration with MS SSAS works well The Ugly ● External driver needed for full VS and SSAS integration ● SSAS sends SELECT DISTINCT * when populating (large) dimensions ● Using ROLAP storage in SSAS against AX requires a bit of tinkering around ● Dialect differences with MariaDB TX
  11. 11. MariaDB AX The Data Flow core CM system extract changed records FS store incremental extracts Staging Area 1:1 load with CPImport Target Area normalized encrypted full history DM Area star-schema snapshot of current state MD Cube metadata hierarchies data security FStrigger CPImport SQL SQL SQL MDX FTP Icons made by https://thoseicons.com
  12. 12. Data Modeling Primer
  13. 13. Why Bother Transform Data? Transactional Workloads ● Normalized ● Referential integrity ● Consistency ● Read/write access to a few rows per query ● 1000s of TPS (ms response time) Analytical Workloads ● De-normalized ● Attributes from different domains ● Filtering and aggregation of millions of rows per query ● 10s of TPS (s response time)
  14. 14. The Star-Schema Query card issuer insurance_claim claim_date transaction transaction_type transaction_date (SELECT is.name, SUM(cl.amount) claims FROM insurance_claims cl JOIN issuer is ON is.issuer_id = cl.issuer_id JOIN claim_date dt ON dt.date_id = cl.date_id WHERE dt.month_id = 201801 GROUP BY 1) t1 (SELECT is.name, SUM(tr.amount) topups FROM trans tr JOIN issuer is ON is.issuer_id = tr.issuer_id JOIN transaction_date dt ON dt.date_id = cl.date_id JOIN transaction_type tp ON tp.type_id = tr.type_id WHERE dt.month_id = 201801 AND tp.type_cd = ‘Top-up’ GROUP BY 1) t2 SELECT t1.name, COALESCE(t1.claims,0.00), COALESCE(t2.topups,0.00) FROM FULL JOIN ON t1.name = t2.name
  15. 15. Moving to The Cloud
  16. 16. Cloud-based System Architecture (Migration) Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery MySQL Proxy query rewriting decryption MS SSAS metadata (R)OLAP XML/A interface Active Directory intl authentication Web Portal SSO ext authentication authorization content delivery XML/A Proxy authorization b/w intl and ext ID Excel! reports pivot tables MariaDB TX users data security contents
  17. 17. Opportunities for Improvement (aka Issues) ● Many moving parts, difficult and expensive to maintain ● High licensing cost of Windows components (SSAS, Server, AD) ● Requires a dedicated Windows OPS person ● Security concerns by client ● Multidimensional model in SSAS suboptimal fit for “lists of records”
  18. 18. Microsoft Azure ● Azure Analysis Services: metadata store, Excel & ad-hoc reporting, row- level security ● Power BI: dashboards, pretty and pixel-perfect reports (via export to PPT) ● SharePoint: content sharing ● Azure AD B2B: single sign-on and integrated authentication ● VM instances: optional, MariaDB AX can also be on-premise ● Pay-as-you-go model: no capex, low integration cost
  19. 19. Cloud-based System Architecture (Target) Icons made by https://thoseicons.com file server extract delivery MariaDB AX DWH E-LT processing core CM system extract delivery Azure AS metadata tabular model XML/A interface Azure AD B2B SSO authentication Excel! reports pivot tables PowerBI dashboards Sharepoint content delivery
  20. 20. Data Integration
  21. 21. Integration MariaDB AX - Azure AS ● Tabular mode: Azure Analysis Services only provides tabular mode ● MySQL connection: uses MySQL Connector/Net ● On-premises Data Gateway: required for a MySQL connection ● No DirectQuery support: only supported for MS, Oracle, SAP, Teradata, etc MariaDB AX On-premises Data Gateway (Windows x64) Azure AS Icons made by https://thoseicons.com
  22. 22. Steps to Connect to MariaDB AX ● Download and Install On-Premises Data Gateway (Windows box) ● Register On-Premises Data Gateway ● Add GW in Azure AS instance ● Download and Install Connector/Net 6.x (on the same Windows box) ● In Visual Studio: New AS Tabular Project -> Import Data -> MySQL ● Specify TCP/IP connection setting Icons made by https://thoseicons.com
  23. 23. Incremental Data Load ● Table Partitions: separate queries, each loaded independently ● ProcessAdd: insert new rows to existing partition ● Our Design: multiple partitions (card_id) for large tables: cards, transactions ● Native Query DAX functions: using AES_DECRYPT ● Daily process: Automated ProcessAdd TMSL script for all partitions ● Monthly process: Re-process partitions with added records above a threshold
  24. 24. Front-End
  25. 25. User Security ● Azure AD B2B collaboration: external email for user identification ● Federated SSO: external identity provider based on SAML ● User provisioning: automatically generated PowerShell script using New- AzureADMSInvitation from list of users provided from CM system ● Data security: using Analysis Services roles ● Roles: one role per client organization, DAX formula [Client ID] = <literal> ● Role management: automatically generated TMSL script from user data ● Role membership: automatically generated TMSL script from user data or manually through SSMS
  26. 26. Report Delivery ● Live Reports: Excel templates based on a pivot table with a live connection to Azure AS instance, refreshed with client data on access ● Delivery platform: templates published on Sharepoint Online ● Dashboards: PowerBI connected to Azure AS instance ● Printed Reports: experimenting with PowerBI export as PowerPoint, immature functionality, need to find a solution
  27. 27. Future Work
  28. 28. DirectQuery Mode Approach 1: HDInsight Spark Integration ● Create custom DataSource to interface MariaDB AX PM storage directly ● use Spark as a SQL engine Approach 2: TDS protocol handler ● Create MaxScale protocol plugin to handle TDS connections ● Cheat Azure AS that is sees a SQL Server PM PM UM MariaDB AX E-LT Spark Azure AS E-LT MariaDB AX MaxScale Azure AS TDSprotocol Qryrewrite DataSource TDS MySQL
  29. 29. Q & A
  30. 30. Thank you!

×