SlideShare a Scribd company logo
1 of 34
Download to read offline
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Challenges for Machine Learning
Systems toward Continuous
Improvement
IBIS 2019 機械学習工学セッション @ Nagoya
2019.11.22
Arm Treasure Data
Aki Ariga
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[Sculley, 2015] より翻訳をして引用
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
●
●
●
https://sites.google.com/view/sig-mlse/wg
試行錯誤から本番適用にもっていくにあたり、継続的に改善を続
けていく上で適切な機械学習システムを作るため
- 論文になりづらいLesson Learntを収集し
- アーキテクチャパターンを体系化したい
(鷲崎先生の取り組み [Washizaki 2019]以外、まだ十分に整備されていない)
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● Project process, organization structure/management
● Machine Learning systems for distributed training
● Hardware acceleration e.g. GPU, TPU, FPGA, etc...
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Typical steps for a Machine Learning project
1. ビジネス課題を
2. 類似の課題を、論文を中心にサーベイ
3. 方法を考える
4. システム設計
5. 学習データ(特徴量+ラベル)の設計
6. 実データの収集と前処理をする
7. 探索的データ分析とアルゴリズムの選定
8. 学習・パラメータチューニング
9. システムに組み込む
10. 予測精度・ビジネス指標をモニタリング
実験ループ:
5〜8を繰り返し
本番ループ:
8〜10を繰り返し
(4に戻ることも)
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
1. ビジネス課題を
2. 類似の課題を、論文を中心にサーベイ
3. 機械学習をしない方法を考える
4. システム設計
5. 学習
6. 実データの収集と前処理
7. 探索的データ分析とアルゴリズムの選定
8. 学習・パラメータチューニング
9. システムに組み込む
10. をモニタリング
1. を書く
2. コードを書く
3. Pull Request/CIでの
4. コードレビュー、マージ
5. コード/バイナリのビルド、デプロイ
6. を と
共にモニタリング
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
1. ビジネス課題を
2. 類似の課題を、論文を中心にサーベイ
3. 機械学習をしない方法を考える
4. システム設計
5. 学習
6. 実データの収集と前処理
7. 探索的データ分析とアルゴリズムの選定
8. 学習・パラメータチューニング
9. システムに組み込む
10. をモニタリング
1. を書く
2. コードを書く
3. Pull Request/CIでの
4. コードレビュー、マージ
5. コード/バイナリのビルド、デプロイ
6. を と
共にモニタリング
● に振る舞いが決まる
● 仕様を網羅して ができる
● エラーは一意に定義でき、コードの
ロジックを することができ
る
● 入力 に対して にに振る
舞いが決まる
● 仕様を網羅できず、
しかできない
● エラーかどうかの検出が難しく、モ
デル更新など しかでき
ない
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● “Data defines everything”
● “Change Anything Changes Everything” [Sculley, 2015]
● データが振る舞いを決めるため、確率的にエラーを抑えることし
かできない
● 確率的な挙動をするため、変更の影響範囲を事前に予見できない
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● した音声認識システム
○ リリース後、 に → 予測性能が劣化
● 工場の画像検査システムで、一ヶ月以上現場の画像で学習
○ が変化 → 予測性能が劣化
● ウェブサイトのランキングシステム
○ により検索語のトレンドが変遷 → 性能が劣化
[Batch] より翻訳をして引用
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[Ziobaite, 2010]より引用
Production models should be
retrained continuously to avoid
concept drift
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[CD4ML]より引用
モデルを本番に学習・デプロイし続けるには、
チームをまたいだ協力が必要
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
🤔
● : ”Infrastructure as Code” made Ops of Dev
○ which improves
● : Isolate DS/Researcher’s code from production system
○ which aims to improve
○ researcher’s code by Dev is
■ Those Dev can be DS/researcher and get high salary!
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[Ariga] より
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Preprocessing
EDA
Training
Deploy/Serving
Audit
[Ariga] より
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
[Marsden, 2019]より翻訳して引用
●
○ 9ヶ月前に学習したモデルが全く同じ環境で、同じデータで再学習でき、
ほぼ同じ(数%以内の差)の精度を得られるべきである
●
○ 本番で稼働しているどのモデルも、作成時のパラメータと学習データ、
更に生データまでトレースできるべきである
●
○ 他の同僚の作ったモデルを本人に聞くことなく改善でき、
非同期で改善とコードやデータのマージができるべきである
●
○ 手動での作業0でモデルはデプロイできるべき。
統計的にモニタリングできるべき
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
1. Data/Model management
2. Experiment Tracking
3. Reproducible experiment
4. Pipeline management
5. ML framework abstraction
6. Model Serving/Deployment
7. Testing and quality check
8. Explaining model/data
9. Monitoring/Observability
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Reproducible / Accountable
Collaborative / Continuous1. Data/Model management
How do you manage 10000+ models
in production?
● Phase
○ EDA, Training, Audit
● Related concepts
○ Data/Model Versioning,
Data/Model Lineage,
Metadata management,
Feature store
● OSS
○ ModelDB, pachyderm, DVC,
ML Metadata, Feast
Example of Data Lineage. [Uber] より引用
EDA Training
Audit
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Reproducible / Accountable
Collaborative / Continuous2. Experiment tracking
What parameter/condition did you choose
with the production model?
● Phase
○ Training, Audit
● Related concepts
○ Parameter management, Artifact
management
Model tracking example
[MLFlow tracking]
より引用
● OSS
○ kubeflow pipelines, MLflow
tracking, polyaxon, comet.ml
Training Audit
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
3. Reproducible experiments
Reproducible / Accountable
Collaborative / Continuous
Is you notebook can reproducible in
your peer’s environment?
● Phase
○ EDA, Training, Audit
● Related concepts
○ Reproducible notebook,
○ Dependency management
● OSS
○ Jupyter notebook, Polynote,
Docker
https://twitter.com/keigohtr/status/1197321232800071680
もちろん、実装によるが...
EDA Training
Audit
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Reproducible / Accountable
Collaborative / Continuous4. Pipeline management
How do you manage consistent
end-to-end ML pipeline?
● Phase
○ Preprocessing, Training,
Deploy/Serving, Audit
● Related concepts
○ Pipeline versioning, Workflow,
CI
● OSS
○ kubeflow, Argo, ArgoCD,
MLFlow Projects, Tensorflow
Transform, [General workflow
engines]
Training
Audit
Preprocessing
Deploy/Serving
[Kubeflow pipeline] より引用
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Reproducible / Accountable
Collaborative / Continuous5. ML framework abstraction
How can Dev make DS’s code deployable
without understanding the code?
● Phase
○ Preprocessing, Training, Deploy
● Related concepts
○ Deploy with function/decorator,
Configuration based training,
AutoML
● OSS & Known frameworks
○ TensorFlow Transform,
Metaflow
○ (FBLeaner Flow, Overton,
Bighead, Boson, Metaflow)
TrainingPreprocessing
Deploy/Serving
Metaflow enables to deploy first models within a week for most
of projects [Metaflow]
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Overton: A Data System for Monitoring and Improving
Machine-Learned Products [Overton]
“domain engineers should not be
forced to write traditional deep learning
modeling code”
“Engineers are Comfortable with
Automatic Hyperparameter Tuning”
Apple implemented ML system with weak
supervision and slicing.
Engineers are required to
1) create/select schema and input payload.
2) Add slices or labeling functions, or synthetic
examples
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Reproducible / Accountable
Collaborative / Continuous6. Testing and quality check
What metrics should we track for
quality?
● Phase
○ Training, Deploy
● Related concepts
○ Data validation, Component
integration validation, Model
quality check, Adversarial
example detection
● OSS
○ TensorFlow Data Validation,
Deequ
Training Deploy/Serving
[TFDV] Understand data from stats
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Data validation: Testing toward train-serve skew
教師あり学習の予測モデルは、学
習データの分布に近いという仮定
学習データの仮定をvalidationす
れば良い
● Check with schema
○ Categorical variable
○ Numerical value range
○ Similarity of
distribution
[Baylor, 2017]より引用
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
7. Explaining model/data
Can you convince your boss why we
should choose this model?
● Phase
○ Training, Audit
● Related concepts
○ Model explainability,
Bias/fairness and ethics
check
● OSS
○ TensorFlow Model Analysis,
Facets, LIME, ELI5, SHAP,
Manifold
Reproducible / Accountable
Collaborative / Continuous
Training Audit
Manifold can compare two sliced population with multiple
models/features [Manifold]
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
“Explain what’s happening and why. There is still
significant fear, uncertainty and doubt (FUD) about AI. I have
seen that providing a basic education — along the lines of
the AI for Everyone curriculum — eases these conversations.
Other tactics including explainability, visualization,
rigorous testing, and auditing also help build trust in an AI
system and convince our customers (and ourselves!) that it
really works.”
[Batch 2] より引用。下線部は筆者強調
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
8. Model Serving/Deployment
How rapidly can you deploy a model to production?
● Phase
○ Deploy/Serving
● Related concepts
○ Rollout (Canary rollouts, A/B testing, Shadowing)
○ Model serialization/export
● OSS
○ TF Serving, Seldon, KFServing, MLFlow Model registry, Clipper
○ PMML, PFA, ONNX, Menoh
Reproducible / Accountable
Collaborative / Continuous
Deploy/Serving
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
“Time is money” - impact of serving latency [Bernardi 2019]
+30% latency reduces 0.5% CVR at
Booking.com
Techniques to reduce latency:
● Model Redundancy
● In-house developed Linear Prediction
engine
● Sparse models
● Precomputation and caching
● Bulking
● Minimum Feature Transformation
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
9. Monitoring/Observability
How do you notice production
model/data is corrupt?
● Phase
○ Deploy/Serving
● Related concepts
○ Data Validation, Outlier
detection, Concept drift
detection, Delayed feedback
● OSS
○ [General Monitoring tools],
[General Workflow engine],
TFDV
Reproducible / Accountable
Collaborative / Continuous
Deploy/Serving
“Smooth bimodal distributions with one clear
stable point are signs of a model that successfully
distinguishes two classes” [Bernardi 2019]
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
Challenges Phase
Reprod
ucible
Accou
ntable
Collab
orative
Contin
uous
Data/Model management EDA/Training/Audit ✅ ✅ ✅
Experiment Tracking Training/Audit ✅ ✅ ✅
Reproducible experiment EDA/Training/Audit ✅ ✅ ✅
Pipeline management
Preprocessing/Training/S
erving/Deploy/Audit ✅ ✅ ✅ ✅
ML framework abstraction
Preprocessing/Training/D
eploy ✅ ✅ ✅
Testing and quality check Training/Deploy ✅ ✅
Explaining model/data Training/Audit ✅
Model Serving/Deployment Serving/Deploy ✅
Monitoring/Observability Serving/Deploy ✅ ✅ ✅
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● Introduced the importance of Continuous Improvement for machine learning
system
● Introduced 9 Challenges for ML systems in production
● Let’s join MLSE slack and collect/share knowledge on GitHub Wiki!
https://github.com/chezou/ml_in_production/wiki
○ Planning for the Workshop for Machine Learning system in production
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● https://github.com/EthicalML/awesome-production-machine-learning
● https://hackernoon.com/why-is-devops-for-machine-learning-so-different-384z32f1
● https://martinfowler.com/articles/cd4ml.html
Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved.
● [Sculley, 2015] Sculley, David, et al. "Hidden technical debt in machine learning systems." Advances in neural information processing systems.
2015.
● [Washizaki 2019] Washizaki, Hironori, et al. "Studying Software Engineering Patterns for Designing Machine Learning Systems." arXiv
preprint arXiv:1910.04736 (2019).
● [Batch] The Batch Nov. 6, 2019
https://info.deeplearning.ai/the-batch-deepmind-masters-starcraft-2-ai-attacks-on-amazon-a-career-in-robot-management-banks-embrace-b
ots-1
● [Batch 2] The Batch Nov. 20, 2019
https://info.deeplearning.ai/the-batch-artificial-noses-surveillance-on-wheels-unwelcome-researchers-privacy-problems-beyond-
bounding-boxes
● [Kubeflow pipeline] https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/
● [TFDV] https://www.tensorflow.org/tfx/data_validation/get_started
● [Zliobaite, 2010] Žliobaitė, Indrė. "Learning under concept drift: an overview." arXiv preprint arXiv:1010.4784 (2010).
● [Marsden, 2019] Marsden, Luke. "The Future of MLOps"
https://docs.google.com/presentation/d/17RWqPH8nIpwG-jID_UeZBCaQKoz4LVk1MLULrZdyNCs/edit#slide=id.p
● [Baylor, 2017] D. Baylor et al., “TFX: A tensorflow-based production-scale machine learning platform,” in Proceedings of the 23rd acm
sigkdd international conference on knowledge discovery and data mining, 2017, pp. 1387–1395
● [CD4ML] D. Sato, et al. “Continuous Delivery for Machine Learning” , https://martinfowler.com/articles/cd4ml.html
● [Uber] “Databook: Turning big data into knowledge with metadata at uber.” https://eng.uber.com/databook/
● [Manifold] “Manifold: A Model-Agnostic Visual Debugging Tool for Machine Learning at Uber” https://eng.uber.com/manifold/
● [Overton] Ré, Christopher & Niu, Feng & Gudipati, Pallavi & Srisuwananukorn, Charles. (2019). Overton: A Data System for Monitoring and
Improving Machine-Learned Products.
● [Bernardi 2019] Bernardi, Lucas, et al. "150 Successful Machine Learning Models: 6 Lessons Learned at Booking. com." Proceedings of
the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.
● [Metaflow] Julie Pitt, et al. “A Human-Friendly Approach to MLOps”, MLOps NYC19, https://youtu.be/fOSZuONmLbA
● [Ariga] “MLOpsの歩き方”, n月刊ラムダノートvol.1 no.1, 2019
Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2019
Thank You!
Danke!
Merci!
谢谢!
ありがとう!
Gracias!
Kiitos!

More Related Content

What's hot

MLflowによる機械学習モデルのライフサイクルの管理
MLflowによる機械学習モデルのライフサイクルの管理MLflowによる機械学習モデルのライフサイクルの管理
MLflowによる機械学習モデルのライフサイクルの管理Takeshi Yamamuro
 
「機械学習:技術的負債の高利子クレジットカード」のまとめ
「機械学習:技術的負債の高利子クレジットカード」のまとめ「機械学習:技術的負債の高利子クレジットカード」のまとめ
「機械学習:技術的負債の高利子クレジットカード」のまとめRecruit Technologies
 
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...NTT DATA Technology & Innovation
 
OSS強化学習フレームワークの比較
OSS強化学習フレームワークの比較OSS強化学習フレームワークの比較
OSS強化学習フレームワークの比較gree_tech
 
機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design Patterns機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design PatternsHironori Washizaki
 
機械学習モデルのサービングとは?
機械学習モデルのサービングとは?機械学習モデルのサービングとは?
機械学習モデルのサービングとは?Sho Tanaka
 
機械学習応用のためのソフトウェアエンジニアリングパターン
機械学習応用のためのソフトウェアエンジニアリングパターン機械学習応用のためのソフトウェアエンジニアリングパターン
機械学習応用のためのソフトウェアエンジニアリングパターンHironoriTAKEUCHI1
 
SlideShareをやめて Speaker Deckに移行します
SlideShareをやめて Speaker Deckに移行しますSlideShareをやめて Speaker Deckに移行します
SlideShareをやめて Speaker Deckに移行しますMoriwaka Kazuo
 
機械学習で泣かないためのコード設計 2018
機械学習で泣かないためのコード設計 2018機械学習で泣かないためのコード設計 2018
機械学習で泣かないためのコード設計 2018Takahiro Kubo
 
Machine learning CI/CD with OSS
Machine learning CI/CD with OSSMachine learning CI/CD with OSS
Machine learning CI/CD with OSSyusuke shibui
 
プロジェクトマネージャのための機械学習工学入門
プロジェクトマネージャのための機械学習工学入門プロジェクトマネージャのための機械学習工学入門
プロジェクトマネージャのための機械学習工学入門Nobukazu Yoshioka
 
Apache Arrow - データ処理ツールの次世代プラットフォーム
Apache Arrow - データ処理ツールの次世代プラットフォームApache Arrow - データ処理ツールの次世代プラットフォーム
Apache Arrow - データ処理ツールの次世代プラットフォームKouhei Sutou
 
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)NTT DATA Technology & Innovation
 
Teslaにおけるコンピュータビジョン技術の調査
Teslaにおけるコンピュータビジョン技術の調査Teslaにおけるコンピュータビジョン技術の調査
Teslaにおけるコンピュータビジョン技術の調査Kazuyuki Miyazawa
 
Amazon SageMakerでカスタムコンテナを使った学習
Amazon SageMakerでカスタムコンテナを使った学習Amazon SageMakerでカスタムコンテナを使った学習
Amazon SageMakerでカスタムコンテナを使った学習西岡 賢一郎
 
Deep Learningを用いた経路予測の研究動向
Deep Learningを用いた経路予測の研究動向Deep Learningを用いた経路予測の研究動向
Deep Learningを用いた経路予測の研究動向HiroakiMinoura
 
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜Preferred Networks
 
機械学習で嘘をつく話
機械学習で嘘をつく話機械学習で嘘をつく話
機械学習で嘘をつく話Satoshi Hara
 

What's hot (20)

MLflowによる機械学習モデルのライフサイクルの管理
MLflowによる機械学習モデルのライフサイクルの管理MLflowによる機械学習モデルのライフサイクルの管理
MLflowによる機械学習モデルのライフサイクルの管理
 
「機械学習:技術的負債の高利子クレジットカード」のまとめ
「機械学習:技術的負債の高利子クレジットカード」のまとめ「機械学習:技術的負債の高利子クレジットカード」のまとめ
「機械学習:技術的負債の高利子クレジットカード」のまとめ
 
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...
MLOps に基づく AI/ML 実運用最前線 ~画像、動画データにおける MLOps 事例のご紹介~(映像情報メディア学会2021年冬季大会企画セッショ...
 
OSS強化学習フレームワークの比較
OSS強化学習フレームワークの比較OSS強化学習フレームワークの比較
OSS強化学習フレームワークの比較
 
機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design Patterns機械学習デザインパターン Machine Learning Design Patterns
機械学習デザインパターン Machine Learning Design Patterns
 
機械学習モデルのサービングとは?
機械学習モデルのサービングとは?機械学習モデルのサービングとは?
機械学習モデルのサービングとは?
 
機械学習応用のためのソフトウェアエンジニアリングパターン
機械学習応用のためのソフトウェアエンジニアリングパターン機械学習応用のためのソフトウェアエンジニアリングパターン
機械学習応用のためのソフトウェアエンジニアリングパターン
 
Data-Centric AIの紹介
Data-Centric AIの紹介Data-Centric AIの紹介
Data-Centric AIの紹介
 
SlideShareをやめて Speaker Deckに移行します
SlideShareをやめて Speaker Deckに移行しますSlideShareをやめて Speaker Deckに移行します
SlideShareをやめて Speaker Deckに移行します
 
機械学習で泣かないためのコード設計 2018
機械学習で泣かないためのコード設計 2018機械学習で泣かないためのコード設計 2018
機械学習で泣かないためのコード設計 2018
 
Machine learning CI/CD with OSS
Machine learning CI/CD with OSSMachine learning CI/CD with OSS
Machine learning CI/CD with OSS
 
プロジェクトマネージャのための機械学習工学入門
プロジェクトマネージャのための機械学習工学入門プロジェクトマネージャのための機械学習工学入門
プロジェクトマネージャのための機械学習工学入門
 
Apache Arrow - データ処理ツールの次世代プラットフォーム
Apache Arrow - データ処理ツールの次世代プラットフォームApache Arrow - データ処理ツールの次世代プラットフォーム
Apache Arrow - データ処理ツールの次世代プラットフォーム
 
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)
入門 Kubeflow ~Kubernetesで機械学習をはじめるために~ (NTT Tech Conference #4 講演資料)
 
Teslaにおけるコンピュータビジョン技術の調査
Teslaにおけるコンピュータビジョン技術の調査Teslaにおけるコンピュータビジョン技術の調査
Teslaにおけるコンピュータビジョン技術の調査
 
Amazon SageMakerでカスタムコンテナを使った学習
Amazon SageMakerでカスタムコンテナを使った学習Amazon SageMakerでカスタムコンテナを使った学習
Amazon SageMakerでカスタムコンテナを使った学習
 
Deep Learningを用いた経路予測の研究動向
Deep Learningを用いた経路予測の研究動向Deep Learningを用いた経路予測の研究動向
Deep Learningを用いた経路予測の研究動向
 
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜
PFN のオンプレML基盤の取り組み / オンプレML基盤 on Kubernetes 〜PFN、ヤフー〜
 
BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装BlackBox モデルの説明性・解釈性技術の実装
BlackBox モデルの説明性・解釈性技術の実装
 
機械学習で嘘をつく話
機械学習で嘘をつく話機械学習で嘘をつく話
機械学習で嘘をつく話
 

Similar to Challenges for machine learning systems toward continuous improvement

20210428 - Sustainable Engineering practices & API Communities: Adoption Best...
20210428 - Sustainable Engineering practices & API Communities: Adoption Best...20210428 - Sustainable Engineering practices & API Communities: Adoption Best...
20210428 - Sustainable Engineering practices & API Communities: Adoption Best...Angel Alberici
 
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...Amazon Web Services Korea
 
Data meets AI - ATP Roadshow India
Data meets AI - ATP Roadshow IndiaData meets AI - ATP Roadshow India
Data meets AI - ATP Roadshow IndiaSandesh Rao
 
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerceWhat We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerceVMware Tanzu
 
SRE (service reliability engineer) on big DevOps platform running on the clou...
SRE (service reliability engineer) on big DevOps platform running on the clou...SRE (service reliability engineer) on big DevOps platform running on the clou...
SRE (service reliability engineer) on big DevOps platform running on the clou...DevClub_lv
 
Pre-Con Education: Introduction to Mainframe Academy With CA Technologies
Pre-Con Education: Introduction to Mainframe Academy With CA TechnologiesPre-Con Education: Introduction to Mainframe Academy With CA Technologies
Pre-Con Education: Introduction to Mainframe Academy With CA TechnologiesCA Technologies
 
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_have
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_haveHow_to_build_your_cloud_enablement_engine_with_the_people_you_already_have
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_haveAmazon Web Services
 
The Strategic Role of the Enterprise Application Framework
The Strategic Role of the Enterprise Application FrameworkThe Strategic Role of the Enterprise Application Framework
The Strategic Role of the Enterprise Application FrameworkJean-Marc Desvaux
 
Empowering your Process Automation with Machine Learning
Empowering your Process Automation with Machine LearningEmpowering your Process Automation with Machine Learning
Empowering your Process Automation with Machine LearningLykle Thijssen
 
Methods Over Madness 2003 Ver.
Methods Over Madness 2003 Ver.Methods Over Madness 2003 Ver.
Methods Over Madness 2003 Ver.Tom Weinberger
 
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...Amazon Web Services
 
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...Anderson Bassani
 
Mobile to Mainframe - the Challenges of Enterprise DevOps Adoption
Mobile to Mainframe - the Challenges of Enterprise DevOps AdoptionMobile to Mainframe - the Challenges of Enterprise DevOps Adoption
Mobile to Mainframe - the Challenges of Enterprise DevOps AdoptionSanjeev Sharma
 
Good Design is Good Business: Business Design with RSA and SA
Good Design is Good Business: Business Design with RSA and SAGood Design is Good Business: Business Design with RSA and SA
Good Design is Good Business: Business Design with RSA and SARoger Snook
 
IBM Rational Developer for System z Quick Start Sales Presentation
IBM Rational Developer for System z Quick Start Sales PresentationIBM Rational Developer for System z Quick Start Sales Presentation
IBM Rational Developer for System z Quick Start Sales PresentationIBM Rational software
 
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...sumitahuja94
 
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...Building a Mass Migration Program Moving Through Vision to Experiment and Sca...
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...Amazon Web Services
 
Drive Digital Transformation using Machine Learning
Drive Digital Transformation using Machine LearningDrive Digital Transformation using Machine Learning
Drive Digital Transformation using Machine LearningAmazon Web Services
 

Similar to Challenges for machine learning systems toward continuous improvement (20)

20210428 - Sustainable Engineering practices & API Communities: Adoption Best...
20210428 - Sustainable Engineering practices & API Communities: Adoption Best...20210428 - Sustainable Engineering practices & API Communities: Adoption Best...
20210428 - Sustainable Engineering practices & API Communities: Adoption Best...
 
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...
Datarobot, 자동화된 분석 적용 시 분석 절차의 변화 및 효용 - 홍운표 데이터 사이언티스트, DataRobot :: AWS Sum...
 
Data meets AI - ATP Roadshow India
Data meets AI - ATP Roadshow IndiaData meets AI - ATP Roadshow India
Data meets AI - ATP Roadshow India
 
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerceWhat We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
What We're Learning Adopting Spring Boot and PCF for Dell.com's eCommerce
 
SRE (service reliability engineer) on big DevOps platform running on the clou...
SRE (service reliability engineer) on big DevOps platform running on the clou...SRE (service reliability engineer) on big DevOps platform running on the clou...
SRE (service reliability engineer) on big DevOps platform running on the clou...
 
Pre-Con Education: Introduction to Mainframe Academy With CA Technologies
Pre-Con Education: Introduction to Mainframe Academy With CA TechnologiesPre-Con Education: Introduction to Mainframe Academy With CA Technologies
Pre-Con Education: Introduction to Mainframe Academy With CA Technologies
 
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_have
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_haveHow_to_build_your_cloud_enablement_engine_with_the_people_you_already_have
How_to_build_your_cloud_enablement_engine_with_the_people_you_already_have
 
The Strategic Role of the Enterprise Application Framework
The Strategic Role of the Enterprise Application FrameworkThe Strategic Role of the Enterprise Application Framework
The Strategic Role of the Enterprise Application Framework
 
Empowering your Process Automation with Machine Learning
Empowering your Process Automation with Machine LearningEmpowering your Process Automation with Machine Learning
Empowering your Process Automation with Machine Learning
 
Methods Over Madness 2003 Ver.
Methods Over Madness 2003 Ver.Methods Over Madness 2003 Ver.
Methods Over Madness 2003 Ver.
 
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...
Train once, deploy anywhere on the cloud and at the edge with Neo - AIM301 - ...
 
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...
IBM RACEv - Ferramenta para análise técnico e financeira para Plataforma Main...
 
Mobile to Mainframe - the Challenges of Enterprise DevOps Adoption
Mobile to Mainframe - the Challenges of Enterprise DevOps AdoptionMobile to Mainframe - the Challenges of Enterprise DevOps Adoption
Mobile to Mainframe - the Challenges of Enterprise DevOps Adoption
 
Good Design is Good Business: Business Design with RSA and SA
Good Design is Good Business: Business Design with RSA and SAGood Design is Good Business: Business Design with RSA and SA
Good Design is Good Business: Business Design with RSA and SA
 
IBM Rational Developer for System z Quick Start Sales Presentation
IBM Rational Developer for System z Quick Start Sales PresentationIBM Rational Developer for System z Quick Start Sales Presentation
IBM Rational Developer for System z Quick Start Sales Presentation
 
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...
Kochi Mulesoft Meetup #10 - MuleSoft Composer: Connect apps and data easily w...
 
mulecomposer.pdf
mulecomposer.pdfmulecomposer.pdf
mulecomposer.pdf
 
Global Squad Leader - Resume
Global Squad Leader - ResumeGlobal Squad Leader - Resume
Global Squad Leader - Resume
 
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...Building a Mass Migration Program Moving Through Vision to Experiment and Sca...
Building a Mass Migration Program Moving Through Vision to Experiment and Sca...
 
Drive Digital Transformation using Machine Learning
Drive Digital Transformation using Machine LearningDrive Digital Transformation using Machine Learning
Drive Digital Transformation using Machine Learning
 

More from Aki Ariga

Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataAki Ariga
 
仕事ではじめる機械学習
仕事ではじめる機械学習仕事ではじめる機械学習
仕事ではじめる機械学習Aki Ariga
 
主人が外資系IT企業に転職して4ヶ月が過ぎました
主人が外資系IT企業に転職して4ヶ月が過ぎました主人が外資系IT企業に転職して4ヶ月が過ぎました
主人が外資系IT企業に転職して4ヶ月が過ぎましたAki Ariga
 
R&D at Foodtech company - #CookpadTechConf 2016
R&D at Foodtech company - #CookpadTechConf 2016R&D at Foodtech company - #CookpadTechConf 2016
R&D at Foodtech company - #CookpadTechConf 2016Aki Ariga
 
Why I started Machine Learning Casual Talks? #MLCT
Why I started Machine Learning Casual Talks? #MLCTWhy I started Machine Learning Casual Talks? #MLCT
Why I started Machine Learning Casual Talks? #MLCTAki Ariga
 
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題Aki Ariga
 
Rubyistがgemの前にPypiデビューするのは間違っているだろうか
Rubyistがgemの前にPypiデビューするのは間違っているだろうかRubyistがgemの前にPypiデビューするのは間違っているだろうか
Rubyistがgemの前にPypiデビューするのは間違っているだろうかAki Ariga
 
Machine Learning Casual Talks Intro #MLCT
Machine Learning Casual Talks Intro #MLCTMachine Learning Casual Talks Intro #MLCT
Machine Learning Casual Talks Intro #MLCTAki Ariga
 
Make Julia more popular in Japan!!1 #JuliaTokyo
Make Julia more popular in Japan!!1 #JuliaTokyoMake Julia more popular in Japan!!1 #JuliaTokyo
Make Julia more popular in Japan!!1 #JuliaTokyoAki Ariga
 
Refrection of kawasaki.rb
Refrection of kawasaki.rbRefrection of kawasaki.rb
Refrection of kawasaki.rbAki Ariga
 
Introduction and benchmarking of MeCab.jl #JapanR
Introduction and benchmarking of MeCab.jl  #JapanRIntroduction and benchmarking of MeCab.jl  #JapanR
Introduction and benchmarking of MeCab.jl #JapanRAki Ariga
 
Recommendation for iruby #tqrk08
Recommendation for iruby #tqrk08Recommendation for iruby #tqrk08
Recommendation for iruby #tqrk08Aki Ariga
 
The book that changed me
The book that changed meThe book that changed me
The book that changed meAki Ariga
 
Introduction of Mecab.jl #JuliaTokyo
Introduction of Mecab.jl #JuliaTokyoIntroduction of Mecab.jl #JuliaTokyo
Introduction of Mecab.jl #JuliaTokyoAki Ariga
 
Introduction to Kanagawa Ruby Kaigi01 #kana01
Introduction to Kanagawa Ruby Kaigi01 #kana01Introduction to Kanagawa Ruby Kaigi01 #kana01
Introduction to Kanagawa Ruby Kaigi01 #kana01Aki Ariga
 
Julia 100 exercises #JuliaTokyo
Julia 100 exercises #JuliaTokyoJulia 100 exercises #JuliaTokyo
Julia 100 exercises #JuliaTokyoAki Ariga
 
Machine Learning Casual Talks opening talk
Machine Learning Casual Talks opening talkMachine Learning Casual Talks opening talk
Machine Learning Casual Talks opening talkAki Ariga
 
Gong anyware
Gong anywareGong anyware
Gong anywareAki Ariga
 
gsub with ActiveSupport::SafeBuffer
gsub with ActiveSupport::SafeBuffergsub with ActiveSupport::SafeBuffer
gsub with ActiveSupport::SafeBufferAki Ariga
 
はじめて翻訳記事を書いたら300ブクマ超えた話
はじめて翻訳記事を書いたら300ブクマ超えた話はじめて翻訳記事を書いたら300ブクマ超えた話
はじめて翻訳記事を書いたら300ブクマ超えた話Aki Ariga
 

More from Aki Ariga (20)

Managing Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure DataManaging Machine Learning workflows on Treasure Data
Managing Machine Learning workflows on Treasure Data
 
仕事ではじめる機械学習
仕事ではじめる機械学習仕事ではじめる機械学習
仕事ではじめる機械学習
 
主人が外資系IT企業に転職して4ヶ月が過ぎました
主人が外資系IT企業に転職して4ヶ月が過ぎました主人が外資系IT企業に転職して4ヶ月が過ぎました
主人が外資系IT企業に転職して4ヶ月が過ぎました
 
R&D at Foodtech company - #CookpadTechConf 2016
R&D at Foodtech company - #CookpadTechConf 2016R&D at Foodtech company - #CookpadTechConf 2016
R&D at Foodtech company - #CookpadTechConf 2016
 
Why I started Machine Learning Casual Talks? #MLCT
Why I started Machine Learning Casual Talks? #MLCTWhy I started Machine Learning Casual Talks? #MLCT
Why I started Machine Learning Casual Talks? #MLCT
 
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題
クックパッドサマーインターン2015 機械学習・自然言語処理 実習課題
 
Rubyistがgemの前にPypiデビューするのは間違っているだろうか
Rubyistがgemの前にPypiデビューするのは間違っているだろうかRubyistがgemの前にPypiデビューするのは間違っているだろうか
Rubyistがgemの前にPypiデビューするのは間違っているだろうか
 
Machine Learning Casual Talks Intro #MLCT
Machine Learning Casual Talks Intro #MLCTMachine Learning Casual Talks Intro #MLCT
Machine Learning Casual Talks Intro #MLCT
 
Make Julia more popular in Japan!!1 #JuliaTokyo
Make Julia more popular in Japan!!1 #JuliaTokyoMake Julia more popular in Japan!!1 #JuliaTokyo
Make Julia more popular in Japan!!1 #JuliaTokyo
 
Refrection of kawasaki.rb
Refrection of kawasaki.rbRefrection of kawasaki.rb
Refrection of kawasaki.rb
 
Introduction and benchmarking of MeCab.jl #JapanR
Introduction and benchmarking of MeCab.jl  #JapanRIntroduction and benchmarking of MeCab.jl  #JapanR
Introduction and benchmarking of MeCab.jl #JapanR
 
Recommendation for iruby #tqrk08
Recommendation for iruby #tqrk08Recommendation for iruby #tqrk08
Recommendation for iruby #tqrk08
 
The book that changed me
The book that changed meThe book that changed me
The book that changed me
 
Introduction of Mecab.jl #JuliaTokyo
Introduction of Mecab.jl #JuliaTokyoIntroduction of Mecab.jl #JuliaTokyo
Introduction of Mecab.jl #JuliaTokyo
 
Introduction to Kanagawa Ruby Kaigi01 #kana01
Introduction to Kanagawa Ruby Kaigi01 #kana01Introduction to Kanagawa Ruby Kaigi01 #kana01
Introduction to Kanagawa Ruby Kaigi01 #kana01
 
Julia 100 exercises #JuliaTokyo
Julia 100 exercises #JuliaTokyoJulia 100 exercises #JuliaTokyo
Julia 100 exercises #JuliaTokyo
 
Machine Learning Casual Talks opening talk
Machine Learning Casual Talks opening talkMachine Learning Casual Talks opening talk
Machine Learning Casual Talks opening talk
 
Gong anyware
Gong anywareGong anyware
Gong anyware
 
gsub with ActiveSupport::SafeBuffer
gsub with ActiveSupport::SafeBuffergsub with ActiveSupport::SafeBuffer
gsub with ActiveSupport::SafeBuffer
 
はじめて翻訳記事を書いたら300ブクマ超えた話
はじめて翻訳記事を書いたら300ブクマ超えた話はじめて翻訳記事を書いたら300ブクマ超えた話
はじめて翻訳記事を書いたら300ブクマ超えた話
 

Recently uploaded

2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.elesangwon
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunicationnovrain7111
 
Python Programming for basic beginners.pptx
Python Programming for basic beginners.pptxPython Programming for basic beginners.pptx
Python Programming for basic beginners.pptxmohitesoham12
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...gerogepatton
 
Detection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackingDetection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackinghadarpinhas1
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Sumanth A
 
STATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectSTATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectGayathriM270621
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliNimot Muili
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communicationpanditadesh123
 
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...IJAEMSJORNAL
 
Immutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfImmutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfDrew Moseley
 
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...shreenathji26
 
ADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studyADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studydhruvamdhruvil123
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labsamber724300
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSneha Padhiar
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 
Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Communityprachaibot
 

Recently uploaded (20)

ASME-B31.4-2019-estandar para diseño de ductos
ASME-B31.4-2019-estandar para diseño de ductosASME-B31.4-2019-estandar para diseño de ductos
ASME-B31.4-2019-estandar para diseño de ductos
 
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
2022 AWS DNA Hackathon 장애 대응 솔루션 jarvis.
 
The Satellite applications in telecommunication
The Satellite applications in telecommunicationThe Satellite applications in telecommunication
The Satellite applications in telecommunication
 
Python Programming for basic beginners.pptx
Python Programming for basic beginners.pptxPython Programming for basic beginners.pptx
Python Programming for basic beginners.pptx
 
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
March 2024 - Top 10 Read Articles in Artificial Intelligence and Applications...
 
Detection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and trackingDetection&Tracking - Thermal imaging object detection and tracking
Detection&Tracking - Thermal imaging object detection and tracking
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
 
Versatile Engineering Construction Firms
Versatile Engineering Construction FirmsVersatile Engineering Construction Firms
Versatile Engineering Construction Firms
 
STATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subjectSTATE TRANSITION DIAGRAM in psoc subject
STATE TRANSITION DIAGRAM in psoc subject
 
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot MuiliStructural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
Structural Integrity Assessment Standards in Nigeria by Engr Nimot Muili
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
multiple access in wireless communication
multiple access in wireless communicationmultiple access in wireless communication
multiple access in wireless communication
 
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
Guardians of E-Commerce: Harnessing NLP and Machine Learning Approaches for A...
 
Immutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfImmutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdf
 
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...
Introduction to Artificial Intelligence: Intelligent Agents, State Space Sear...
 
ADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain studyADM100 Running Book for sap basis domain study
ADM100 Running Book for sap basis domain study
 
Secure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech LabsSecure Key Crypto - Tech Paper JET Tech Labs
Secure Key Crypto - Tech Paper JET Tech Labs
 
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATIONSOFTWARE ESTIMATION COCOMO AND FP CALCULATION
SOFTWARE ESTIMATION COCOMO AND FP CALCULATION
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 
Prach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism CommunityPrach: A Feature-Rich Platform Empowering the Autism Community
Prach: A Feature-Rich Platform Empowering the Autism Community
 

Challenges for machine learning systems toward continuous improvement

  • 1. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Challenges for Machine Learning Systems toward Continuous Improvement IBIS 2019 機械学習工学セッション @ Nagoya 2019.11.22 Arm Treasure Data Aki Ariga
  • 2. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [Sculley, 2015] より翻訳をして引用
  • 3. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● ● ● https://sites.google.com/view/sig-mlse/wg 試行錯誤から本番適用にもっていくにあたり、継続的に改善を続 けていく上で適切な機械学習システムを作るため - 論文になりづらいLesson Learntを収集し - アーキテクチャパターンを体系化したい (鷲崎先生の取り組み [Washizaki 2019]以外、まだ十分に整備されていない)
  • 4. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● Project process, organization structure/management ● Machine Learning systems for distributed training ● Hardware acceleration e.g. GPU, TPU, FPGA, etc...
  • 5. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Typical steps for a Machine Learning project 1. ビジネス課題を 2. 類似の課題を、論文を中心にサーベイ 3. 方法を考える 4. システム設計 5. 学習データ(特徴量+ラベル)の設計 6. 実データの収集と前処理をする 7. 探索的データ分析とアルゴリズムの選定 8. 学習・パラメータチューニング 9. システムに組み込む 10. 予測精度・ビジネス指標をモニタリング 実験ループ: 5〜8を繰り返し 本番ループ: 8〜10を繰り返し (4に戻ることも)
  • 6. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 1. ビジネス課題を 2. 類似の課題を、論文を中心にサーベイ 3. 機械学習をしない方法を考える 4. システム設計 5. 学習 6. 実データの収集と前処理 7. 探索的データ分析とアルゴリズムの選定 8. 学習・パラメータチューニング 9. システムに組み込む 10. をモニタリング 1. を書く 2. コードを書く 3. Pull Request/CIでの 4. コードレビュー、マージ 5. コード/バイナリのビルド、デプロイ 6. を と 共にモニタリング
  • 7. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 1. ビジネス課題を 2. 類似の課題を、論文を中心にサーベイ 3. 機械学習をしない方法を考える 4. システム設計 5. 学習 6. 実データの収集と前処理 7. 探索的データ分析とアルゴリズムの選定 8. 学習・パラメータチューニング 9. システムに組み込む 10. をモニタリング 1. を書く 2. コードを書く 3. Pull Request/CIでの 4. コードレビュー、マージ 5. コード/バイナリのビルド、デプロイ 6. を と 共にモニタリング ● に振る舞いが決まる ● 仕様を網羅して ができる ● エラーは一意に定義でき、コードの ロジックを することができ る ● 入力 に対して にに振る 舞いが決まる ● 仕様を網羅できず、 しかできない ● エラーかどうかの検出が難しく、モ デル更新など しかでき ない
  • 8. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● “Data defines everything” ● “Change Anything Changes Everything” [Sculley, 2015] ● データが振る舞いを決めるため、確率的にエラーを抑えることし かできない ● 確率的な挙動をするため、変更の影響範囲を事前に予見できない
  • 9. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● した音声認識システム ○ リリース後、 に → 予測性能が劣化 ● 工場の画像検査システムで、一ヶ月以上現場の画像で学習 ○ が変化 → 予測性能が劣化 ● ウェブサイトのランキングシステム ○ により検索語のトレンドが変遷 → 性能が劣化 [Batch] より翻訳をして引用
  • 10. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [Ziobaite, 2010]より引用 Production models should be retrained continuously to avoid concept drift
  • 11. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [CD4ML]より引用 モデルを本番に学習・デプロイし続けるには、 チームをまたいだ協力が必要
  • 12. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 🤔 ● : ”Infrastructure as Code” made Ops of Dev ○ which improves ● : Isolate DS/Researcher’s code from production system ○ which aims to improve ○ researcher’s code by Dev is ■ Those Dev can be DS/researcher and get high salary!
  • 13. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [Ariga] より
  • 14. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Preprocessing EDA Training Deploy/Serving Audit [Ariga] より
  • 15. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. [Marsden, 2019]より翻訳して引用 ● ○ 9ヶ月前に学習したモデルが全く同じ環境で、同じデータで再学習でき、 ほぼ同じ(数%以内の差)の精度を得られるべきである ● ○ 本番で稼働しているどのモデルも、作成時のパラメータと学習データ、 更に生データまでトレースできるべきである ● ○ 他の同僚の作ったモデルを本人に聞くことなく改善でき、 非同期で改善とコードやデータのマージができるべきである ● ○ 手動での作業0でモデルはデプロイできるべき。 統計的にモニタリングできるべき
  • 16. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 1. Data/Model management 2. Experiment Tracking 3. Reproducible experiment 4. Pipeline management 5. ML framework abstraction 6. Model Serving/Deployment 7. Testing and quality check 8. Explaining model/data 9. Monitoring/Observability
  • 17. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Reproducible / Accountable Collaborative / Continuous1. Data/Model management How do you manage 10000+ models in production? ● Phase ○ EDA, Training, Audit ● Related concepts ○ Data/Model Versioning, Data/Model Lineage, Metadata management, Feature store ● OSS ○ ModelDB, pachyderm, DVC, ML Metadata, Feast Example of Data Lineage. [Uber] より引用 EDA Training Audit
  • 18. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Reproducible / Accountable Collaborative / Continuous2. Experiment tracking What parameter/condition did you choose with the production model? ● Phase ○ Training, Audit ● Related concepts ○ Parameter management, Artifact management Model tracking example [MLFlow tracking] より引用 ● OSS ○ kubeflow pipelines, MLflow tracking, polyaxon, comet.ml Training Audit
  • 19. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 3. Reproducible experiments Reproducible / Accountable Collaborative / Continuous Is you notebook can reproducible in your peer’s environment? ● Phase ○ EDA, Training, Audit ● Related concepts ○ Reproducible notebook, ○ Dependency management ● OSS ○ Jupyter notebook, Polynote, Docker https://twitter.com/keigohtr/status/1197321232800071680 もちろん、実装によるが... EDA Training Audit
  • 20. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Reproducible / Accountable Collaborative / Continuous4. Pipeline management How do you manage consistent end-to-end ML pipeline? ● Phase ○ Preprocessing, Training, Deploy/Serving, Audit ● Related concepts ○ Pipeline versioning, Workflow, CI ● OSS ○ kubeflow, Argo, ArgoCD, MLFlow Projects, Tensorflow Transform, [General workflow engines] Training Audit Preprocessing Deploy/Serving [Kubeflow pipeline] より引用
  • 21. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Reproducible / Accountable Collaborative / Continuous5. ML framework abstraction How can Dev make DS’s code deployable without understanding the code? ● Phase ○ Preprocessing, Training, Deploy ● Related concepts ○ Deploy with function/decorator, Configuration based training, AutoML ● OSS & Known frameworks ○ TensorFlow Transform, Metaflow ○ (FBLeaner Flow, Overton, Bighead, Boson, Metaflow) TrainingPreprocessing Deploy/Serving Metaflow enables to deploy first models within a week for most of projects [Metaflow]
  • 22. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Overton: A Data System for Monitoring and Improving Machine-Learned Products [Overton] “domain engineers should not be forced to write traditional deep learning modeling code” “Engineers are Comfortable with Automatic Hyperparameter Tuning” Apple implemented ML system with weak supervision and slicing. Engineers are required to 1) create/select schema and input payload. 2) Add slices or labeling functions, or synthetic examples
  • 23. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Reproducible / Accountable Collaborative / Continuous6. Testing and quality check What metrics should we track for quality? ● Phase ○ Training, Deploy ● Related concepts ○ Data validation, Component integration validation, Model quality check, Adversarial example detection ● OSS ○ TensorFlow Data Validation, Deequ Training Deploy/Serving [TFDV] Understand data from stats
  • 24. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Data validation: Testing toward train-serve skew 教師あり学習の予測モデルは、学 習データの分布に近いという仮定 学習データの仮定をvalidationす れば良い ● Check with schema ○ Categorical variable ○ Numerical value range ○ Similarity of distribution [Baylor, 2017]より引用
  • 25. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 7. Explaining model/data Can you convince your boss why we should choose this model? ● Phase ○ Training, Audit ● Related concepts ○ Model explainability, Bias/fairness and ethics check ● OSS ○ TensorFlow Model Analysis, Facets, LIME, ELI5, SHAP, Manifold Reproducible / Accountable Collaborative / Continuous Training Audit Manifold can compare two sliced population with multiple models/features [Manifold]
  • 26. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. “Explain what’s happening and why. There is still significant fear, uncertainty and doubt (FUD) about AI. I have seen that providing a basic education — along the lines of the AI for Everyone curriculum — eases these conversations. Other tactics including explainability, visualization, rigorous testing, and auditing also help build trust in an AI system and convince our customers (and ourselves!) that it really works.” [Batch 2] より引用。下線部は筆者強調
  • 27. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 8. Model Serving/Deployment How rapidly can you deploy a model to production? ● Phase ○ Deploy/Serving ● Related concepts ○ Rollout (Canary rollouts, A/B testing, Shadowing) ○ Model serialization/export ● OSS ○ TF Serving, Seldon, KFServing, MLFlow Model registry, Clipper ○ PMML, PFA, ONNX, Menoh Reproducible / Accountable Collaborative / Continuous Deploy/Serving
  • 28. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. “Time is money” - impact of serving latency [Bernardi 2019] +30% latency reduces 0.5% CVR at Booking.com Techniques to reduce latency: ● Model Redundancy ● In-house developed Linear Prediction engine ● Sparse models ● Precomputation and caching ● Bulking ● Minimum Feature Transformation
  • 29. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. 9. Monitoring/Observability How do you notice production model/data is corrupt? ● Phase ○ Deploy/Serving ● Related concepts ○ Data Validation, Outlier detection, Concept drift detection, Delayed feedback ● OSS ○ [General Monitoring tools], [General Workflow engine], TFDV Reproducible / Accountable Collaborative / Continuous Deploy/Serving “Smooth bimodal distributions with one clear stable point are signs of a model that successfully distinguishes two classes” [Bernardi 2019]
  • 30. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. Challenges Phase Reprod ucible Accou ntable Collab orative Contin uous Data/Model management EDA/Training/Audit ✅ ✅ ✅ Experiment Tracking Training/Audit ✅ ✅ ✅ Reproducible experiment EDA/Training/Audit ✅ ✅ ✅ Pipeline management Preprocessing/Training/S erving/Deploy/Audit ✅ ✅ ✅ ✅ ML framework abstraction Preprocessing/Training/D eploy ✅ ✅ ✅ Testing and quality check Training/Deploy ✅ ✅ Explaining model/data Training/Audit ✅ Model Serving/Deployment Serving/Deploy ✅ Monitoring/Observability Serving/Deploy ✅ ✅ ✅
  • 31. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● Introduced the importance of Continuous Improvement for machine learning system ● Introduced 9 Challenges for ML systems in production ● Let’s join MLSE slack and collect/share knowledge on GitHub Wiki! https://github.com/chezou/ml_in_production/wiki ○ Planning for the Workshop for Machine Learning system in production
  • 32. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● https://github.com/EthicalML/awesome-production-machine-learning ● https://hackernoon.com/why-is-devops-for-machine-learning-so-different-384z32f1 ● https://martinfowler.com/articles/cd4ml.html
  • 33. Copyright 1995-2019 Arm Limited (or its affiliates). All rights reserved. ● [Sculley, 2015] Sculley, David, et al. "Hidden technical debt in machine learning systems." Advances in neural information processing systems. 2015. ● [Washizaki 2019] Washizaki, Hironori, et al. "Studying Software Engineering Patterns for Designing Machine Learning Systems." arXiv preprint arXiv:1910.04736 (2019). ● [Batch] The Batch Nov. 6, 2019 https://info.deeplearning.ai/the-batch-deepmind-masters-starcraft-2-ai-attacks-on-amazon-a-career-in-robot-management-banks-embrace-b ots-1 ● [Batch 2] The Batch Nov. 20, 2019 https://info.deeplearning.ai/the-batch-artificial-noses-surveillance-on-wheels-unwelcome-researchers-privacy-problems-beyond- bounding-boxes ● [Kubeflow pipeline] https://www.kubeflow.org/docs/pipelines/overview/pipelines-overview/ ● [TFDV] https://www.tensorflow.org/tfx/data_validation/get_started ● [Zliobaite, 2010] Žliobaitė, Indrė. "Learning under concept drift: an overview." arXiv preprint arXiv:1010.4784 (2010). ● [Marsden, 2019] Marsden, Luke. "The Future of MLOps" https://docs.google.com/presentation/d/17RWqPH8nIpwG-jID_UeZBCaQKoz4LVk1MLULrZdyNCs/edit#slide=id.p ● [Baylor, 2017] D. Baylor et al., “TFX: A tensorflow-based production-scale machine learning platform,” in Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, 2017, pp. 1387–1395 ● [CD4ML] D. Sato, et al. “Continuous Delivery for Machine Learning” , https://martinfowler.com/articles/cd4ml.html ● [Uber] “Databook: Turning big data into knowledge with metadata at uber.” https://eng.uber.com/databook/ ● [Manifold] “Manifold: A Model-Agnostic Visual Debugging Tool for Machine Learning at Uber” https://eng.uber.com/manifold/ ● [Overton] Ré, Christopher & Niu, Feng & Gudipati, Pallavi & Srisuwananukorn, Charles. (2019). Overton: A Data System for Monitoring and Improving Machine-Learned Products. ● [Bernardi 2019] Bernardi, Lucas, et al. "150 Successful Machine Learning Models: 6 Lessons Learned at Booking. com." Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019. ● [Metaflow] Julie Pitt, et al. “A Human-Friendly Approach to MLOps”, MLOps NYC19, https://youtu.be/fOSZuONmLbA ● [Ariga] “MLOpsの歩き方”, n月刊ラムダノートvol.1 no.1, 2019
  • 34. Confidential © Arm 2017Confidential © Arm 2017Confidential © Arm 2019 Thank You! Danke! Merci! 谢谢! ありがとう! Gracias! Kiitos!