A Journey to a Serverless Business Intelligence, Machine Learning and Big Data platform
•Download as PPTX, PDF•
4 likes•520 views
In this talk we will describe the journey we made with one of our customers, Volotea, to deploy a serverless Business Intelligence (BI), Machine Learning (ML) and Big Data (BD) platform on the Cloud. The new platform leverages Platform-as-a-Service (PaaS) Cloud services, and it is the result of the reengineering and extension of an existing platform based on Cloud Infrastructure-as-a-Service (IaaS) services and bare-metal systems. Managing and maintaining BI, ML and BD platforms based on bare-metal or IaaS deployments is not a straightforward task, and as size and complexity grow, we often find ourselves spending more and more time in tasks that are rather administrative, more than of a development or analytics nature. That is exactly what Volotea realized, and together we envisioned and executed a plan to lift and reengineer their platform into a new solution that leverages Microsoft Azure PaaS services. We have delivered a solution that manages to greatly reduce the administrative burden as well as the technical complexity when implementing new use cases. The new platform is based on the Microsoft Azure stack and it includes Azure Data Lake, Azure Data Lake Analytics, Azure Data Factory, Azure Machine Learning and Azure SQL Database. Join us in this talk where we will share our lessons learned and we will discuss how to plan and execute such an endeavor.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to A Journey to a Serverless Business Intelligence, Machine Learning and Big Data platform
Similar to A Journey to a Serverless Business Intelligence, Machine Learning and Big Data platform (20)
More from DataWorks Summit
More from DataWorks Summit (20)
Recently uploaded
Recently uploaded (20)
A Journey to a Serverless Business Intelligence, Machine Learning and Big Data platform
- 1. A JOURNEY TO A SERVERLESS BUSINESS INTELLIGENCE, MACHINE LEARNING & BIG DATA PLATFORM OSCAR MARTINEZ RUBI March 18–21, 2019 Barcelona, Spain
- 2. AGENDA 1. ABOUT CLEARPEAKS 2. EVOLUTION OF ENTERPRISE DATA ANALYTICS ARCHITECTURES 3. ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD 4. ABOUT VOLOTEA 5. VOLOTEA OBJECTIVES & CHALLENGES 6. VOLOTEA CLOUD REFACTORING JOURNEY 7. LESSONS LEARNED 8. CONCLUSIONS 9. CONTACT US
- 5. more than 34INDUSTRIES Founded in 2000 382 CLIENTS +524 PROJECTS MISSION Through the deployment of quality, business focused and technically robust analytical solutions, we strive to ensure our customers meet their business objectives and can demonstrate undisputed ROI on their investment. ABOUT CLEARPEAKS
- 6. ORACLE OPENWORLD FINALIST AWARD ITC ACHIEVEMENT AWARD Energy sector Deployment of the year Big Data & Business Analytics Offices in BARCELONA DUBAI ABU DHABI 80+CONSULTANTS +100 CERTIFICATIONS TECHNOLOGIES ORACLE MICROSOFT TABLEAU QLIK AWS CLOUDERA TALEND INFORMATICA JAVASCRIPT R / PYTHON ABOUT CLEARPEAKS
- 7. ClearPeaks © 2019 - Confidential Data Monetization Improved Business Performance Accelerate path to a data-driven organisation OUR VALUE PROPOSITION ABOUT CLEARPEAKS Management Dashboards, Data Visualization & Discovery The Executive Experience Advanced Analytics Corporate Scorecards & Performance Management Corporate Insights
- 9. ClearPeaks © 2019 - Confidential TRADITIONAL BI | ENTERPRISE DATA WAREHOUSE EVOLUTION OF ENTERPRISE DATA ANALYTICS ARCHITECTURES STRUCTURED DATA Batch Data Ingestion DWH ANALYTICS & DASHBOARDS ETL/ELT Aggregated data storage
- 10. ClearPeaks © 2019 - Confidential TYPICAL BI/EDW + BIG DATA + ML ARCHITECTURE EVOLUTION OF ENTERPRISE DATA ANALYTICS ARCHITECTURES STRUCTURED DATA UNSTRUCTURED DATA Real-time data ingestion ADVANCED ANALYTICS & DASHBOARDS Batch Data Ingestion BIG DATA PLATFORM Aggregated data storage DWH DATA VIRTUALIZATION
- 12. ClearPeaks © 2019 - Confidential CLOUD MODELS ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD Managed by customer Managed by Cloud Provider “Managed” by Cloud provider (setup by customer) On-premise IaaS PaaS SaaS Identity & Access Management Identity & Access Management Identity & Access Management Identity & Access Management Customer Data Customer Data Customer Data Customer Data Application Application Application Application Cloud Network Cloud Network Cloud Network Cloud Network Cloud Storage Cloud Storage Cloud Storage Cloud Storage Platform Platform Platform Platform Guest OS Guest OS Guest OS Guest OS Virtualization Virtualization Virtualization Virtualization Compute Compute Compute Compute Infrastructure Network Infrastructure Network Infrastructure Network Infrastructure Network Infrastructure Storage Infrastructure Storage Infrastructure Storage Infrastructure Storage
- 13. ClearPeaks © 2019 - Confidential EVOLUTION OF ENTERPRISE DATA ANALYTICS PLATFORMS ON CLOUD ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD FIRST WAVE OF ENTERPRISE DATA ANALYTICS PLATFORMS ON CLOUD SECOND WAVE OF ENTERPRISE DATA ANALYTICS PLATFORMS ON CLOUD PaaS-centricIaaS-centric
- 14. ClearPeaks © 2019 - Confidential NATIVE PAAS SERVICES IN AZURE AND AWS ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD COLLECT/ACQUIRE STORE/ORGANIZE ANALYSE/PROCESS VISUALISE AWS Direct Connect Amazon Kinesis AWS Import/Export Amazon Kinesis Firehose AWS Database Migration Amazon S3 Amazon RDS Aurora Amazon Glacier Amazon DynamoDB Amazon CloudSearch Amazon Elasticsearch AWS Data Pipeline AWS Lambda Amazon EMR Amazon SageMaker Amazon Redshift Amazon Kinesis Analytics Amazon Athena AI Services Amazon QuickSight COLLECT/ACQUIRE STORE/ORGANIZE ANALYSE/PROCESS VISUALISE Event Hubs Service Bus External Data Sources SQL Database Data Lake Store Table Blob Storage External Data Sources Cosmos DB Data Factory Machine Learning SQL Data Warehouse Stream Analysis Data Lake Analytics HDInsight AI Platform App Service Mobile Services Power BI BizTalk Services Notification Hubs
- 15. ClearPeaks © 2019 - Confidential OTHER PAAS SERVICES BUILT ON TOP OF AZURE AND AWS ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD
- 16. ClearPeaks © 2019 - Confidential 6 STRATEGIES FOR MIGRATING APPLICATIONS TO THE CLOUD ENTERPRISE DATA ANALYTICS PLATFORM MEETS CLOUD Source: https://aws.amazon.com/blogs/enterprise-strategy/6-strategies-for-migrating-applications-to-the-cloud/
- 17. ABOUT VOLOTEA
- 18. ClearPeaks © 2019 - Confidential CUSTOMER PROFILE ABOUT VOLOTEA 30 AIRCRAFTS 86% AVERAGE SEAT LOAD 81 DESTINATIONS 13 COUNTRIES 4.82 m PASSENGERS IN 2017 44,500 FLIGHTS IN 2017
- 20. ClearPeaks © 2019 - Confidential OBJECTIVES VOLOTEA OBJECTIVES & CHALLENGES BI+BIG DATA+ML PLATFORM • • • HADOOP DEPLOYMENT ON IAAS SERVICES IN AMAZON REFACTOR BI+BIG DATA+ML PLATFORM • • • SERVER-LESS ARCHITECTURE LEVERAGING AZURE PAAS SERVICES LACK OF RESOURCES in-house to architect and implement data management including BI, Big Data and ML solutions NEED OF CLOUD EXPERTISE especially in how to leverage PaaS Services CHALLENGES&
- 21. ClearPeaks © 2019 - Confidential OBJECTIVES VOLOTEA OBJECTIVES & CHALLENGES Decrease administration efforts and costs that are related to the maintenance of IaaS-based solutions, specially Big Data platforms, and increase agility in resizing platforms. Implementation of BI Big Data and ML pipelines leveraging the server-less PaaS services in Azure. Delivery of expert data management services such as the creation of an efficient replication strategy for various databases with hundreds of tables.
- 23. ClearPeaks © 2019 - Confidential STEPS VOLOTEA CLOUD REFACTORING JOURNEY Analysis & Design Analysis of the existing Enterprise Data Analytics Platform based on IaaS services containing several BI & Big Data & ML pipelines + design of the PaaS-based architecture on Microsoft Azure. 1 Setup & PoC Creation & setup of the new Microsoft Azure architecture (create and link services) + migration of the representative pipeline to have an understanding of costs and other implications. 2 3 Plan and execute full migration Design and execution of a 1-year plan for the full migration of all existing BI & Big Data pipelines while extending and adding new pipelines as new business requirements occur.
- 24. ClearPeaks © 2019 - Confidential TECHNOLOGY STACK VOLOTEA CLOUD REFACTORING JOURNEY Industry: Airlines/Aviation Departments: Revenue, Finance, Operations, HR, Marketing, Sales, IT, Maintenance Technology: Microsoft Duration: Ongoing Effort: 3.5 resources Timing: Start: January 2018 Finish (Refactor): February 2019 FACTS Microsoft & Azure stack: • Azure Data Lake & Azure Data Lake Analytics • Azure HDInsight • Azure Data Factory • Azure SQL Database • Azure Machine Learning Studio • Azure Events Hub • Microsoft SQL Server • Microsoft SQL Server Integration Services
- 25. ClearPeaks © 2019 - Confidential ARCHITECTURE VOLOTEA CLOUD REFACTORING JOURNEY OPERATIONAL DATABASES FTP GOOGLE ANALYTICS BIGQUERY ERP (NAVISION) HADOOP DATA GATEWAY VISUALIZATION VIRTUAL MACHINES STORAGE SQL DATABASEDATA LAKE STORE ORCHESTRATION INGESTION PROCESSING DATA FACTORY DATA FACTORY COPY DATA LAKE ANALYTICS AZURESOURCES NAVITAIRE IATA TTBS HDINSIGHT MACHINE LEARNING MACHINE LEARNING STUDIO EVENT HUBS
- 26. ClearPeaks © 2019 - Confidential BENEFITS VOLOTEA CLOUD REFACTORING JOURNEY Decrease in the number of Big Data, Database and ETL/ELT/EHL (Extract – Hadoop – Load) platforms to administer causing a decrease in administration costs. 1 2 3 Drop in reaction time - from defining a new need to the final solution - in BI, Big Data, ML and other data management projects. Reduction in platform costs due to the migration from a 24/7 Hadoop cluster to pay-per-use cloud server-less PaaS pipelines.
- 27. ClearPeaks © 2019 - Confidential BENEFITS VOLOTEA CLOUD REFACTORING JOURNEY Strong partner and advisor to tackle Cloud, BI, Big Data and data management projects. Easy-to-use platform that makes it possible to tackle existing and future BI, Big Data and ML use cases with more agility and efficiency 4 5
- 28. LESSONS LEARNED
- 29. ClearPeaks © 2019 - Confidential LESSONS LEARNED There are Clouds of all sorts Be careful with the one-armed bandits Clouds are in the Sky Don’t get lost in translationTHINK BIG, START SMART & SMALL, ITERATE OFTEN
- 30. CONCLUSIONS
- 31. ClearPeaks © 2019 - Confidential CONCLUSIONS 1 2 3 4 Enterprise Data Analytics Platforms have evolved, the underlying technologies, its features and what we expect from them More and more business are moving their applications to Cloud, in the first wave to IaaS and now more and more leveraging PaaS There are different strategies to move to Cloud. Our advise: analyse, PoC, plan and move We have driven Volotea in its journey from an IaaS to a PaaS Enterprise Data Analytics Platform yielding many benefits
- 32. CONTACT US
- 33. ClearPeaks © 2019 - Confidential CONTACT US CLEARPEAKS PATHFINDER PROGRAM CLEARPEAKS DATA ANALYTICS LAB CLOUD MIGRATION SERVICES BIG DATA
- 34. Proprietary and Confidential | © 2019 ClearPeaks All of the images included in this presenta tion are real screensho ts , but because of confidentia lity data has been modifie d. All rights reserved. All informa tion contained in this document is confidentia l and proprie ta ry to ClearPeaks. No part of this document may be photocopie d, electronica lly transferred, modified, or reproduc ed i n any manner without the prior written consent of ClearPea ks. clearpeaks.com info@clearpeaks.com Barcelona +34 93 272 15 46 Abu Dhabi +971 2 417 05 60
Editor's Notes
- In this talk we will describe the journey we made with one of our customers, Volotea, to deploy a serverless Business Intelligence (BI), Machine Learning (ML) and Big Data (BD) platform on the Cloud. The new platform leverages Platform-as-a-Service (PaaS) Cloud services, and it is the result of the reengineering and extension of an existing platform based on Cloud Infrastructure-as-a-Service (IaaS) services and bare-metal systems. Managing and maintaining BI, ML and BD platforms based on bare-metal or IaaS deployments is not a straightforward task, and as size and complexity grow, we often find ourselves spending more and more time in tasks that are rather administrative, more than of a development or analytics nature. That is exactly what Volotea realized, and together we envisioned and executed a plan to lift and reengineer their platform into a new solution that leverages Microsoft Azure PaaS services. We have delivered a solution that manages to greatly reduce the administrative burden as well as the technical complexity when implementing new use cases. The new platform is based on the Microsoft Azure stack and it includes Azure Data Lake, Azure Data Lake Analytics, Azure Data Factory, Azure Machine Learning and Azure SQL Database. Join us in this talk where we will share our lessons learned and we will discuss how to plan and execute such an endeavor.
- Only structured sources ETL/ELT tool in single server, sometimes embedded with DWH DB engine (ELT) DWH using RDBMS technology, initially in single-core solutions, later with multi-core systems, and later distributed DBs (multi-node) Reports and Dashboards with reporting tool; use cases usually descriptive analytics All deployments are on premise
- The Big Data and ML era! Not only structured sources, massive volumes, velocity – “big data” data ETL/ELT tool replaced by distributed processing engine (Big Data processing engine) in Big Data platform where also real time streaming applications find its space Structured (and optionally aggregated) data stored in DWH – sometimes DWH is integrated in Big Data platform; if DWH is used it is most possible a MPP system (Distributed DB) Data Virtualization tools can help to federate and ease access to the various analytical sources Analytical use cases go beyond descriptive analytics ->advanced analytics: predictive and prescriptive analytics (ML & AI) For these AA use cases BD platforms are used to deal with high data volumes Deployments can be made on-premise but also on Cloud; and once in Cloud they can be done in different models (IaaS vs PaaS)
- Of course there are other Cloud providers: IBM, Google, Oracle An within AWS and Azure there are many other services, we put the ones most relevant for analytics
- Volotea is a Spanish low-cost airline registered in Spain with bases in Spain, Italy, France and Greece. Their fleet consists of 17 Boeing 717 aircraft and 13 Airbus A319s. Volotea opened 90 routes in its first year and operates almost 300 routes in summer 2018 including 220 openings. This could double to at least 500-600 across Europe. Volotea has been profitable since 2014, a turnover of €360 million ($431 million) is expected in 2018 after $347M in 2017, carrying 5.7 to 6 million passengers: 50% are travelling for leisure, 35% to visit friends and relatives, and 15% for business.
- Volotea was looking for a new Enterprise Data Analytics architecture: offer additional functionalities and an improved efficiency in comparison with the platform they were working with. Together we envisioned a Refactoring (Lift & Reengineering & Shift) of their Enterprise Data Analytics Platform from an IaaS model to server-less PaaS model to make platform management more efficient but they were confronted with: a lack of resources in-house to architect and implement data management including BI, Big Data and ML solutions an indispensable need for experience in Cloud services – especially in how to leverage PaaS Services
- Simpler BI and Big Data architectures Easy-to-use development tools Less administration concerns, almost no maintenance efforts and services easier to resize Higher technical expertise on Cloud, BI, Big Data and data management technologies
- Think Big, start smart & small, iterate often: First analyse current environment and consider various journey options/techs and their pros/cons. Then do a PoC to understand costs, effort, etc. ; and that the platform delivers what is expected Yes, that’s a brain and a Cloud! There are Clouds of all sorts: Consider different Clouds (even consider IaaS): some have easier-to-use PaaS services, some are great in intra-interoperability, some have more connectors to external sources, etc. Consider scope of work and limitations of PaaS services and possible tools to operate them Be careful with the one-armed bandits: Be aware of PaaS pipeline runs during development – they also cost money! Estimating PaaS costs is very difficult; it is hard to know in advance how many “processing units” you will need A PaaS that is 24/7 running will be more expensive than a equivalent IaaS – for equal time and size, the more Cloud provider do for you the more it will charge you Clouds are in the Sky: Don’t forget latency and connection lines, if you are operational systems are on-premise be carefull with the connection you have with your Cloud provider – consider options like Azure ExpressRoute Don’t get lost in translation: Do not underestimate the difference between SQL dialects and migrating one application to a different interface, language or even dialect