Successfully reported this slideshow.
Your SlideShare is downloading. ×

MATS stack (MLFlow, Airflow, Tensorflow, Spark) for Cross-system Orchestration of Machine Learning Pipelines

Ad

MATS stack (MLFlow, Airflow, Tensorflow,
Spark) for cross-system orchestration of
machine learning pipelines
João Da Silva...

Ad

Intro
Yury Kasimov
Data engineer at Avast with background in
Machine Learning and Network security, tennis
player on even ...

Ad

Intro
Yury Kasimov
Data engineer at Avast with background in
Machine Learning and Network security, tennis
player on even ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Check these out next

1 of 64 Ad
1 of 64 Ad

MATS stack (MLFlow, Airflow, Tensorflow, Spark) for Cross-system Orchestration of Machine Learning Pipelines

Download to read offline

At Avast we complete over 17 million phishing detections a day, providing crucial online protection for this type of attacks.

In this talk Joao Da Silva and Yury Kasimov will present the MATS stack for productionisation of Machine Learning and their journey into integrating model tracking, storage, cross-system orchestration and model deployments for a complete and modern machine learning pipeline.

At Avast we complete over 17 million phishing detections a day, providing crucial online protection for this type of attacks.

In this talk Joao Da Silva and Yury Kasimov will present the MATS stack for productionisation of Machine Learning and their journey into integrating model tracking, storage, cross-system orchestration and model deployments for a complete and modern machine learning pipeline.

More Related Content

Slideshows for you (19)

Similar to MATS stack (MLFlow, Airflow, Tensorflow, Spark) for Cross-system Orchestration of Machine Learning Pipelines (20)

More from Databricks (20)

MATS stack (MLFlow, Airflow, Tensorflow, Spark) for Cross-system Orchestration of Machine Learning Pipelines

  1. 1. MATS stack (MLFlow, Airflow, Tensorflow, Spark) for cross-system orchestration of machine learning pipelines João Da Silva & Yury Kasimov
  2. 2. Intro Yury Kasimov Data engineer at Avast with background in Machine Learning and Network security, tennis player on even days, chess on odd days
  3. 3. Intro Yury Kasimov Data engineer at Avast with background in Machine Learning and Network security, tennis player on even days, chess on odd days João Da Silva Scala & FP enthusiast, Lead Data Engineer @avast, DJ @sonuz, capoeirista and co-organizer of Prague @functional_jvm meetup
  4. 4. Agenda ● Intro: The saga begins ● Problems: Clone wars ● Goals: Insidious plan ● Solutions: Spark of a rebellion ● Challenges: Technologies strike back ● Successes: A new hope
  5. 5. Avast Avast is dedicated to creating a world that provides safety and privacy for all, no matter who you are, where you are, or how you connect.
  6. 6. Intro: The saga begins
  7. 7. Intro: The saga begins
  8. 8. Intro: The saga begins
  9. 9. Intro: The saga begins
  10. 10. Intro: The saga begins
  11. 11. Problems: Clone wars
  12. 12. Problems: Clone wars ● A lot of duplicated effort between different teams
  13. 13. Problems: Clone wars ● A lot of duplicated effort between different teams ● No overview of different experiments in one place
  14. 14. Problems: Clone wars ● A lot of duplicated effort between different teams ● No overview of different experiments in one place ● No automated process for moving from experiments to production
  15. 15. Problems: Clone wars ● A lot of duplicated effort between different teams ● No overview of different experiments in one place ● No automated process for moving from experiments to production ● Scaling and monitoring of deployed models
  16. 16. Goals: Insidious plan
  17. 17. Goals:Insidiousplan
  18. 18. Goals:Insidiousplan
  19. 19. Goals:Insidiousplan
  20. 20. Goals:Insidiousplan
  21. 21. Goals: Insidious plan ● Define a common ground for data science team and data engineering team ● Structured, fast and reproducible experiments ● Cross-system orchestration/scheduling ● Automated model serving
  22. 22. Solutions: Spark of a rebellion
  23. 23. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure
  24. 24. ML Project Lifecycle, Design and Structure Solutions: Spark of a rebellion
  25. 25. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure
  26. 26. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Data: Data Engineering Stages
  27. 27. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Data: Data Engineering Stages ○ Model: Machine Learning Stages
  28. 28. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Data: Data Engineering Stages ○ Model: Machine Learning Stages ○ Code: CI/CD
  29. 29. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Standard repository structure
  30. 30. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Standard repository structure ○ Standard ML Development at Avast
  31. 31. Solutions: Spark of a rebellion ● ML Project Lifecycle, Design and Structure ○ Standard repository structure ○ Standard ML Development at Avast ○ Standard Tooling
  32. 32. Solutions: MATS Stack ● MLFlow ● Airflow ● Tensorflow ● Spark MATS
  33. 33. Solutions: Spark of a rebellion ● MLFlow for experiment tracking and Model management
  34. 34. Solutions: Spark of a rebellion ● MLFlow for experiment tracking and Model management ○ Open Source ML Platform ○ Easy experiment tracking ○ Model packaging, storage, version management and deployment ○ Rich API and CLI which can be used by any language or ML Library
  35. 35. Solutions: Spark of a rebellion
  36. 36. Solutions: Spark of a rebellion
  37. 37. Solutions: Spark of a rebellion
  38. 38. Solutions: Spark of a rebellion
  39. 39. Solutions: Spark of a rebellion ● Airflow for cross-system scheduling
  40. 40. Solutions: Spark of a rebellion ● Airflow for cross-system scheduling ○ Message driven architecture ○ It’s extensible, it’s Python ;-) ○ Templating, default_args and connections removes boilerplate
  41. 41. Solutions: Spark of a rebellion Kubernetes GPU/Tensorflow Yarn/Spark data_dump Spark / HDFS
  42. 42. Solutions: Spark of a rebellion Kubernetes GPU/Tensorflow Yarn/Spark data_dump Spark / HDFS
  43. 43. Solutions: Spark of a rebellion Kubernetes GPU/Tensorflow Yarn/Spark data_dump Spark / HDFS
  44. 44. Solutions: Spark of a rebellion Kubernetes GPU/Tensorflow Yarn/Spark data_dump Spark / HDFS
  45. 45. Solutions: Spark of a rebellion Spark / HDFS Kubernetes GPU/Tensorflow Yarn/Spark data_dump
  46. 46. Solutions: Spark of a rebellion ● Tensorflow for high performance training
  47. 47. Solutions: Spark of a rebellion ● Tensorflow for high performance training ○ TFRecords ○ TensorFlow Serving ○ Rich ecosystem
  48. 48. Solutions: Spark of a rebellion ● Spark for distributed big data processing
  49. 49. Solutions: Spark of a rebellion ● Spark for distributed big data processing ○ Extensive usage and knowledge at Avast ○ Really, Spark it’s king for big data processing ;-)
  50. 50. Solutions: MATS Stack MATS
  51. 51. Challenges: Technologies strike back
  52. 52. Challenges: Technologies strike back ▪ Lack of event based notifications for model registry changes ▪ https://github.com/mlflow/mlflow/issues/2740
  53. 53. Challenges: Technologies strike back ▪ Lack of event based notifications for model registry changes ▪ https://github.com/mlflow/mlflow/issues/3015 ▪ Lack of support Tensorflow ModelServer for serving ▪ MLFlow does not support tensorflow model logging in saved_model format ▪ https://github.com/mlflow/mlflow/issues/2740
  54. 54. Challenges: Technologies strike back ▪ Lack of event based notifications for model registry changes ▪ https://github.com/mlflow/mlflow/issues/2740 ▪ Lack of support Tensorflow ModelServer for serving ▪ MLFlow does not support tensorflow model logging in saved_model format ▪ https://github.com/mlflow/mlflow/issues/2740 ▪ Airflow deployment, security and quirks
  55. 55. Successes: A new hope
  56. 56. Successes: A new hope ● Delivered Angler ML pipeline for url phishing classifier
  57. 57. Successes: A new hope ● Delivered Angler ML pipeline for url phishing classifier ● Established processes for faster productization of ML Models
  58. 58. Successes: A new hope ● Delivered Angler ML pipeline for url phishing classifier ● Established processes for faster productization of ML Models ● Interest from other teams to adopt our solution
  59. 59. Successes: A new hope ● Delivered Angler ML pipeline for url phishing classifier ● Established processes for faster productization of ML Models ● Interest from other teams to adopt our solutions ● MATS Stack
  60. 60. We would like to thank ● Tomas Trnka – our first “customer” and the creator of Angler projects ● Vojtech Tuma – our manager for guiding and supporting us ● Our colleagues for their help and suggestions ● All of you that attended this presentation
  61. 61. Reach out Yury Kasimov @LunaticInHall João Da Silva @imjsilva
  62. 62. Q&A
  63. 63. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.

×