Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Finavia: Agile, cloud-based DW development

295 views

Published on

Datastrategian toteutus pilvessä -tapahtuma järjestettiin yhteistyössä Snowflaken kanssa 24.1.2018. Tilaisuudessa kuultiin Tommi Vihervaaran puheenvuoro Finavian kokemuksista liittyen ketterään ja pilvipohjaiseen dw-kehitykseen. Lue lisää tapahtumasta täältä: https://www.solita.fi/tapahtumat/datastrategian-toteutus-pilvessa/

Published in: Data & Analytics
  • Be the first to comment

Finavia: Agile, cloud-based DW development

  1. 1. 25/01/2018 1 Agile, cloud-based DW-development – experiences from the field and lessons learned. 24.1.2018 - Finavia’s journey: the beginning, setting goals and how to get there? - Enablers: Snowflake and Solita ADE - Case examples - Key takeaways Tommi Vihervaara 25.1.20182 “Building the Data-Driven culture in aviation business.”
  2. 2. 25/01/2018 2 25 January 2018 Finavia yritysesitys 20173 Our task is to develop the competitiveness, mobility and internationalisation of Finnish society by providing our customers with safe, high-quality and cost-effective air travel services. We enable international accessibility for Finland through the most widely-linked air traffic network in Northern Europe. 25 January 2018 Finavia yritysesitys 20174 Thanks to the wide network of routes from Helsinki Airport, we offer superb connections to all corners of the world. Transfer connections ensure an extensive market area for Finland. Passengers and airlines are our customers. We produce air traffic services in cooperation with a large group of partners.
  3. 3. 25/01/2018 3
  4. 4. 25/01/2018 4 25 January 2018 Finavia yritysesitys 20178
  5. 5. 25/01/2018 5 Our data journey. In the beginning… Q1 / 2017 – from where did we start from ? 25 January 2018 Finavia yritysesitys 20179
  6. 6. 25/01/2018 6 Target H2/2017 and beyond…
  7. 7. 25/01/2018 7 ”Everyone in the organization should have access to as much data as legally possible” – DJ Patil and Hilary Mason (http://bit.ly/data-driven-report) How the journey began and whats up next? There (and back again? )
  8. 8. 25/01/2018 8 25 January 2018 Finavia yritysesitys 201715 ”Low-hanging fruit”
  9. 9. 25/01/2018 9 Cases 25.1.201817 ”If the organization wants to be data driven there has to be an evangelist that makes noise about what it needs.” ”Son, your ego is writing checks your body can’t cash” http://whitepapers.dataversity.net/content42609/
  10. 10. 25/01/2018 10 25.1.201819 We went agile. Why agile? Data Warehousing projects have their reputation …
  11. 11. 25/01/2018 11 2008 vs 2018 ? 2008
  12. 12. 25/01/2018 12 https://i.giphy.com/media/xUPGczofzl3QEh4glW/giphy.mp4 Just do it. 2018
  13. 13. 25/01/2018 13 25.1.201826
  14. 14. 25/01/2018 14 25 January 2018 Finavia yritysesitys 201728 "keep it short and simple"
  15. 15. 25/01/2018 15 Agile Data Warehousing Old-school DW + ETL + BI doesn’t work anymore! You need a Data Platform  all data: relational data, big data, structured/unstructured, batch and realtime. Manual ETL is a no-no  Automatically generated/maintained ”ETL code” and db schema. - Time to Market: Let’s get going already!  Learning by doing! - Do it once. Do it right. Focus on proper data modeling. - Dev + Ops  ”Data Devops” Old-school BI/DW-development vs. software development?  Getting finally together: ”Data Devops”  Question for you: Which vendors can (really) master it? Enablers: Tools & methods and architecture #architecture&standards #snowflake #solitaADE #datavault2.0 #dwautomation #analyticalMDM
  16. 16. 25/01/2018 16 #architecture Architecture Flights ERP A Cloud API A Legacy sources (relational db) Customer feedback data, security throuputs and other sensor data (non-relational) csv Flights ERP B finavia-stage-data S3 bucket AWS Lambda S3 bucket COPY DDL statements Stage loads Data vault loads FINAVIA_DATA_HUB No separate ”Cube or inmemory” hassle” INTO (Finavia integration platform) DW automation/accelerator solution (”Data Devops”) Import (cache) or DirectQuery #analyticalMDM (MDS) csv Batch Real-time json
  17. 17. 25/01/2018 17 Legendary 3 tier meets 3½: Bimodal 25.1.201833 ”Data Vault. This is where the magik happens”landing / stage consume experiment Storage vs publish !! Since 1974 … #snowflake
  18. 18. 25/01/2018 18 Snowflake • Data Warehouse as a Service • Decisions now, must support requirements N years ahead • Per second pricing (!) • No-Admin !? ~Zero Admin • No Backups!? • Storage vs compute • Separate compute workloads • On-line up/downscaling • No-hadoop etc. • ”It Just Works”™ • Love it. 25.1.201835 #solitaADE #dwautomation
  19. 19. 25/01/2018 19 Solita Agile Data Engine – Data DevOps • ”No ETL”: ETL  L(T) • Separate ETL Tool = Silo ? Does not scale! • ”Model – Map – and Load”  Automation ! • ~ No transformation  Mapping. • Put proper master data in work!  Analytical MDM • Cases – Dimension  Analytical MDM – Fact  ”you got problems elsewhere son” • ”Time to market” • Proper data modeling / standardization: business entities  hubs / satellites / links •  No need for lookups etc. • Automate-everything? • BI & dashboards automation? 25.1.201837 #others Task management and monitoring
  20. 20. 25/01/2018 20 #trello Task management #grafana Monitoring
  21. 21. 25/01/2018 21 #slack Communicating Profit? Some case examples…
  22. 22. 25/01/2018 22 …just to point out only 3 of them. Flight & PAX data • The most crucial data asset of all – drives the whole business • Once we made the data modeling properly it’s been easy to connect more data assets and domains to it •  Pay serious attention to data modeling. Security access control data • Three sources: Boarding pass stamps, line queue length sensors, security lane throughput sensors • Analytic MDM: harmonized master data • Thanks to business- driven and holistic approach to data modeling: Joined the data to flight data  detailed insights: dwell times and throughtput times per destination etc. Strategic data acquisitions • Connect data from various API sources (internal and external) • Once new operational systems are introduced, make sure that the data is available via APIs • Start collecting and introduce early prototypes and test versions in advance. 25.1.201843 #AI
  23. 23. 25/01/2018 23 ? PAX Nationality User experience 25.1.201846 ”Tämä on just niin kuin pitikin.” Datahubin mahdollistama tulevien lentojen työkalu on käytössä eikä vain paranna työtäni, mutta myös vähentää tästä eteenpäin manuaalisen työni määrää useita tunteja jokainen kuukausi. Power BI -työkalun vaatimat muutokset Datahubin puolelle sain läpi monesti jo saman päivän aikana ilmoituksesta ja useat mieleeni tulleet kehitysehdotukset olivat itse asiassa työnalla jo ennen kuin ehdin niistä sanomaan. Huippuduunia! Delay seurantaan ja analyysiin kehiteltävänä oleva Delay Dashboard näyttää jo kehityksen tässä vaiheessa olevan matkalla kohden toimivaa, monipuolista, joustavaa sekä helppokäyttöistä työkalua jota on jo kauan kaivattu niiden käyttäjien toimesta jotka tarvitsevat ja analysoivat näitä tietoja kehittääkseen toimintaa eri vastuualueilla.
  24. 24. 25/01/2018 24 Lessons learned and key takeaways Lessons learned and key takeaways 25.1.201848 Be brave: ”roppaa likoon” • Have the courage! Move forward step by step. • ”Learning by doing”: ask and collect feedback! Technology • Technology does matter • Cloud, [nnn] as a Service, Agile, focus on bringing value! Change • Old habits die hard. Question the old ways to do things. Doubt the old ways to think. • But not all of them! Praise the ”good old” stuff!
  25. 25. 25/01/2018 25 #bigdata #saddata* #IoT BUT beware of… ”Data you forced to collect even though no-one wants it as a customer & no-one needs it in your business & no-one can utilize.” -Jarno Kartela, Artificial Intelligence Partner @ Fourkind #machinelearning #analytics #API
  26. 26. 25/01/2018 26 “Knowledge that does not change behavior is useless.” Yuval Noah Harari, Homo Deus: A Brief History of Tomorrow Thank you. tommi.vihervaara@finavia.fi @GreenyDangerous +358 40 518 1090 https://www.linkedin.com/in/tommi-vihervaara-1593733/

×