Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ML 모델 생성 및 운영 효율화를 높이는
Amazon SageMaker 의 신규 기능들...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• AI/ML on AWS
• ML 모델 생성 및 운영 효율화를 위한 Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATI...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jupyter
Notebook 기반
서비스
SageMaker
Ground Truth
D...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Data Visualization
& Analysis
Business Problem –...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ML vs. human reviews . . .
• ML 과학자, ML 엔지니어, ML...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I : ML 예측 결과에 대한 효율적인 휴먼 리뷰 지원 서비스
휴먼 리...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Human review workforce options
Amazon Mechanical...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I : how it works
클라이언트 애플리케이션이
입력 데이터를 ...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How to use Amazon A2I
Step 1:
휴먼 리뷰 워크플로우 정의
Ste...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Textract: Defining condit...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Textract: (실행 예) 문서 내 오탈자...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Textract: (실행 예) 폼 데이터 추출...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Textract: (실행 예) 데이터 서브셋에...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Rekognition: Defining con...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I with Amazon Rekognition: 이미지 조정을 위한 사...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
다른 ML 모델에도 Amazon A2I 를 사용할 수 있습니다
사용자 ML 모델을
이용...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon A2I: 비용
aws.amazon.com/augmented-ai
Prici...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : 배포된 ML 모델의 관리/업...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : 배포된 ML 모델의 지속적인...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
(S...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
(S...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
(S...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
(S...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
Am...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
(S...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker Model Monitor : How it works
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker multi-model endpoints
 사용자별 모델...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Introducing Amazon SageMaker Multi-model Endpoin...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker multi-model endpoints
Dynamical...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker multi-model endpoints
Can achie...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker managed spot training
ML 모델의 학습...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker managed spot training
각 학습 작업에 ...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker managed spot training
Dataset V...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker managed spot training
Experimen...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SageMaker managed spot training
Experimen...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS 머신러닝(ML) 교육 및 자격증
Amazon의 개발자와 데이터 과학자를 교육하는...
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2019, Amazon Web Services, Inc. or its Affilia...
Thank you!
© 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
[AWS Innovate 온라인 컨퍼런스]  ML 모델 생성 및 운영 효율화를 높이는 Amazon SageMaker의 신규 기능들 - 남궁영환, AWS Sr. AI/ML 컨설턴트
Upcoming SlideShare
Loading in …5
×

[AWS Innovate 온라인 컨퍼런스] ML 모델 생성 및 운영 효율화를 높이는 Amazon SageMaker의 신규 기능들 - 남궁영환, AWS Sr. AI/ML 컨설턴트

136 views

Published on

발표자료 다시보기: https://youtu.be/XVbhLKHC06U

기계 학습 모델링에는 여전히 많은 수작업이 수반됩니다. 여기에는 모델 평가, 성능 모니터링, 실효성 검증 등 다양한 요소들이 포함되어 있습니다. 본 세션에서는 기계 학습 모델에서 데이터 라벨링 작업의 어려움을 해소하는 SageMaker Ground Truth, 모델 예측 결과에 대한 사람에 의한 리뷰 작업을 도와 주는 Augmented AI (A2I), 모델에 대한 성능 모니터링을 도와주는 SageMaker Model Monitor 등에 대해 알아봅니다.

Published in: Technology
  • Be the first to comment

[AWS Innovate 온라인 컨퍼런스] ML 모델 생성 및 운영 효율화를 높이는 Amazon SageMaker의 신규 기능들 - 남궁영환, AWS Sr. AI/ML 컨설턴트

  1. 1. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. ML 모델 생성 및 운영 효율화를 높이는 Amazon SageMaker 의 신규 기능들 남궁영환 Senior AI/ML Consultant AWS, Professional Services
  2. 2. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Agenda • AI/ML on AWS • ML 모델 생성 및 운영 효율화를 위한 Amazon SageMaker의 신규 기능들 • Amazon SageMaker A2I (Augmented AI) • Amazon SageMaker Model Monitor • Amazon SageMaker Multi-model endpoints • Amazon SageMaker Managed spot training • 정리
  3. 3. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  4. 4. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATION FORECASTING FRAUD DEVELOPMENT CONTACT CENTERS Amazon SageMaker Ground Truth Augmented AI SageMaker Neo Built-in algorithms SageMaker Notebooks SageMaker Experiments Model tuning SageMaker Debugger SageMaker Autopilot Model hosting SageMaker Model Monitor Deep Learning AMIs & Containers GPUs & CPUs Elastic Inference Inferentia (Inf2) FPGA Amazon Rekognition Amazon Polly Amazon Transcribe +Medical Amazon Comprehend +Medical Amazon Translate Amazon Lex Amazon Personalize Amazon Forecast Amazon Fraud Detector Amazon CodeGuru AI SERVICES ML SERVICES ML FRAMEWORKS & INFRASTRUCTURE Amazon Textract Amazon Kendra Contact Lens For Amazon Connect SageMaker Studio IDE NEW NEW NEW NEW NEW NEW NEW NEW NEWNEW NEWNEW NEW
  5. 5. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Jupyter Notebook 기반 서비스 SageMaker Ground Truth D A T A P R E P A R A T I O N 높은 정확도의 트레이닝 데이터셋 생성 및 70 % 레이블링 비용 절감 고성능 빌트인 알고리즘 제공 AWS ML Marketplace B U I L T - I N A L G O R I T H M S 파트너사의 수백 가지 새로운 알고리즘에 액세스 Hyperparameter 최적화 SageMaker Neo T R A I N A N D T U N E 정확도 손실 없이 1/10미만의 메모리로 2배 빠른 추론 가능 원클릭 데이터 트레이닝 SageMaker RL O N E - C L I C K T R A I N I N G 강화학습 알고리즘 및 시뮬레이터 추가 원클릭 배포 Training with Spot 온-디멘드 가격 대비 약 90 % 저렴 O N E - C L I C K T R A I N I N G 완전 관리 및 자동 스케일링 최근 1년간 기능 업데이트수 Amazon SageMaker를 이용한 빠른 혁신 손쉬운 기계 학습 모델 생성, 훈련 및 서비스 배포 완전 관리 서비스
  6. 6. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Visualization & Analysis Business Problem – Data Collection Data Integration Data Preparation & Cleaning Feature Engineering Model Training & Parameter Tuning Model Evaluation Are Business Goals met? Model Deployment Monitoring & Debugging – Predictions YesNo DataAugmentation Feature Augmentation Re-training ML problem formulation and/or definition Machine Learning Process : Revisited
  7. 7. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  8. 8. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. ML vs. human reviews . . . • ML 과학자, ML 엔지니어, ML 운영팀이 있어야 합니다 • 대규모 리뷰를 관리해야 합니다 • 리뷰 작업을 관리하기 위한 맞춤형 소프트웨어가 필요합니다 • 휴먼 리뷰 수준의 정확도는 매우 도전적인 목표일 수 있습니다 • 다양한 애플리케이션에 ML 기술을 주입시킵니다. • ML을 통해 빠른 속도와 저렴한 비용으로 주요 유스케이스를 처리합니다. • ML 은 확률론적 결과를 제공합니다. • 낮은 신뢰도의 ML 모델에는 휴먼 리뷰가 필요합니다. Current status ML and humans working together
  9. 9. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I : ML 예측 결과에 대한 효율적인 휴먼 리뷰 지원 서비스 휴먼 리뷰 워크플로우를 손쉽게 구현합니다 Multiple workforce options 미리 구축되어 있는 워크플로우와 UI를 이용하여 시장 출시 소요 시간을 단축시킵니다 사용자 ML 모델과 통합할 수 있습니다 사전에 만들어 놓은 알고리즘을 통해 정확도를 높일 수 있습니다
  10. 10. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Human review workforce options Amazon Mechanical Turk 전세계 500,000 이상의 독립 계약 업체가 온디맨드 형태로 24/7 서비스 지원 Private Vendors 자체 지원 가능한 작업팀 (사내 직원 또는 계약 업체로 구성) 휴먼 리뷰 작업을 전문으로 하는 AWS Marketplace 벤더(vendor) 업체
  11. 11. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I : how it works 클라이언트 애플리케이션이 입력 데이터를 전달 AWS AI 서비스 또는 사용자 ML 모델을 이용하여 예측 수행 Amazon S3에 결과를 저장 1 2 64 낮은 신뢰도의 예측 결과는 휴먼 리뷰를 위해 따로 전송 3 높은 신뢰도의 예측 결과는 클라이언트 애플리케이션으로 즉시 리턴됨 5 Amazon A2I answer 통합 알고리즘을 사용하여 통합 리뷰를 진행 클라이언트 애플리케이션
  12. 12. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. How to use Amazon A2I Step 1: 휴먼 리뷰 워크플로우 정의 Step 2: API call 에서 Amazon A2I 워크플로우를 사용
  13. 13. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Textract: Defining conditions Confidence score Important keys Random sample
  14. 14. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Textract: (실행 예) 문서 내 오탈자 탐지 Amazon TextractInput ML 신뢰도 < 80% 잘못된 스펠링 파악
  15. 15. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Textract: (실행 예) 폼 데이터 추출을 위한 사전 구축 UI
  16. 16. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Textract: (실행 예) 데이터 서브셋에만 휴먼 리뷰 진행
  17. 17. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Rekognition: Defining conditions Single Confidence score Random sampleConfidence score per label
  18. 18. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I with Amazon Rekognition: 이미지 조정을 위한 사전-구축된 UI
  19. 19. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. 다른 ML 모델에도 Amazon A2I 를 사용할 수 있습니다 사용자 ML 모델을 이용한 예측 수행 Amazon S3에 결과를 저장 1 64 네: 휴먼 리뷰를 위해 예측 결과를 전달 3 아니오: 클라이언트 애플리케이션으로 예측 결과를 바로 리턴 5 Amazon A2I 가 작업을 수행하고 결과를 수집 클라이언트 애플리케이션 휴먼 리뷰 여부 결정 (네 / 아니오) 2
  20. 20. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon A2I: 비용 aws.amazon.com/augmented-ai Pricing: 휴먼 리뷰가 필요한 객체에 대해서면 비용이 부과됨
  21. 21. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  22. 22. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : 배포된 ML 모델의 관리/업데이트 • 프로덕션(운영) 환경의 ML 모델은 시간 경과에 따라 에러 및 추론 성능 저하(Model drift)가 발생할 수 있습니다.  트레이닝에 사용된 데이터 와 프로덕션 환경에서 유입되는 데이터의 차이 (통계 분포의 변화를 초래할 수 있음)  예측 결과의 성능에 심각한 영향을 줄 수 있음 • ML 모델 성능의 지속적인 모니터링 및 관리 필요  ML 모델 성능 저하를 빠르게 탐지하고 대규모 ML 모델 관리도 가능한 적절한 방안이 필요  외부 솔루션 활용도 좋지만 비용 효율성 및 확장성 면에서 적합하지 않을 수 있음 ML 모델은 프로덕션 환경에 배포된 후에도 지속적으로 모니터링하고 적절한 주기로 업데이트가 이뤄져야 합니다.
  23. 23. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : 배포된 ML 모델의 지속적인 모니터링 수집된 데이터와 베이스라인 데이터를 대상으로 룰(Rules) 평가 및 편차를 탐지 CloudWatch 로그 기반 측정 지표의 변화 계산 및 CloudWatch 경고(alerts) 기반의 수정 작업 자동화 Amazon SageMaker Studio를 통한 모니터링 이력 및 엔드포인트 확인 데이터 통계 시각화 및 통합 가능 운영 환경의 ML모델을 위한 예측 데이터 수집 추론 요청, 응답 및 메타데이터 model name, sampling rate, timestamp, etc. 수집된 신규 데이터의 S3 저장 Notebook에서 분석 작업 시 손쉬운 데이터 액세스 가능 ( ) 베이스라인 데이터 기반 Schema 생성 및 수집된 신규 데이터 분석 (기초 통계량 계산) 수행 ( )
  24. 24. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining [Overview]
  25. 25. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #1) Amazon SageMaker Endpoint 생성/업데이트
  26. 26. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #2) 생성된 Amazon SageMaker Endpoint 에서 데이터 수집 활성화
  27. 27. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #2) 생성된 Amazon SageMaker Endpoint 에서 데이터 수집 활성화 from import = 'UC-DEMO-xgb-churn-pred-model-monitor-' "%Y-%m-%d-%H-%M-%S" = = True = 100 = = = 1 = 'ml.m5.xlarge' = =
  28. 28. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #3) training/validation 데이터셋에서 베이스라인 생성
  29. 29. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #3) training/validation 데이터셋에서 베이스라인 생성 from import from import = = = 1 = 'ml.m5.xlarge' = 20 = 3600 = '/training-dataset-with-header.csv' = =True = = True
  30. 30. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works (Step #3) training/validation 데이터셋에서 베이스라인 생성 예) Amazon S3에 저장된 baseline 데이터셋의 기초 통계량 계산 결과 (sagemaker/UC-DEMO-ModelMonitor/baselining/results/statistics.json 예) Amazon S3에 저장된 suggested constraints의 결과 (sagemaker/UC-DEMO-ModelMonitor/baselining/results/constraints.json
  31. 31. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #4) 모니터링 스케줄 생성
  32. 32. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #4) 모니터링 스케줄 생성 from import from import = 'DEMO-xgb-churn-pred-model-monitor-schedule' "%Y-%m-%d-%H-%M-%S" = = = = = = = = True
  33. 33. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #5) 모니터링 결과 확인
  34. 34. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works (Step #5) 모니터링 결과 확인 1. Monitoring Job status 종류 • Completed • Completed with violations • Failed • Stopped 2. Monitoring Job 각각에 대해 생성되는 결과 • Violations report for each job in Amazon S3 • Statistics report for data collected during the run • Suggested constraints (which can be used to create new baseline)
  35. 35. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works (Step #5) 모니터링 결과 확인
  36. 36. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works (Step #5) 모니터링 결과 확인 Violation Check Type Description Data type check If the data types in the current execution are not the same as in the baseline dataset, this violation is flagged. Completeness check If the completeness (% of non-null items) observed in the current execution exceeds the threshold specified in completeness threshold specified per feature, this violation is flagged. Drift check If the calculated distribution distance between the current and the baseline datasets is more than the threshold specified Missing column check If the number of columns in the current dataset is less than the number in the baseline dataset, this violation is flagged. Extra column check If the number of columns in the current dataset is more than the number in the baseline, this violation is flagged. Categorical values check If there are more unknown values in the current dataset than in the baseline dataset, this violation is flagged. Model Monitor에서 탐지하는 Violation 타입
  37. 37. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works Amazon SageMaker Training job Model Amazon SageMaker Endpoint Applications Results: statistics and violations Baseline statistics and constraints Amazon CloudWatch metrics Requests, predictions Analysis of results Notifications • Model updates • Training data updates • Retraining (Step #6) 모니터링 알람 확인 및 수정 보완 작업 수행
  38. 38. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works (Step #6) 모니터링 알람 확인 및 수정 보완 작업 수행
  39. 39. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  40. 40. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  41. 41. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  42. 42. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  43. 43. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  44. 44. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker Model Monitor : How it works
  45. 45. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  46. 46. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker multi-model endpoints  사용자별 모델이 많은 경우  유사한 모델이 많은 경우  모든 ML 모델이 각각 다른 액세스 패턴을 지니고 있을 경우 (액세스 빈도가 높은/낮은 경우)  프로덕션 (운영) 환경에 모든 ML 모델이 로딩되어 있어야 함  프로덕션에 로딩되어 있는 ML 모델은 낮은 레이턴시로 추론 결과를 제공해야 함 대규모 모델 배포 작업은 관리 용이성, 비용 효율성 측면에서 쉽지 않습니다. 대규모 모델의 관리에는 많은 배포 비용이 수반됨
  47. 47. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Introducing Amazon SageMaker Multi-model Endpoints 학습된 모델을 Amazon S3에 저장 단일 엔드포인트에서 모든 모델 제공 동일한 엔드 포인트에서 여러 모델을 동시에 호출 트래픽을 기준으로 관리되는 메모리 엔드포인트 및 인스턴스 활용도 향상 수천 개의 모델 배포 및 관리 손쉬운 ML 모델의 배포 및 관리 엔드포인트에 여러개의 ML 모델 배포 타깃 ML 모델 호출 자동화된 메모리 관리 큰 비용 절감 효과 Seoul GA
  48. 48. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker multi-model endpoints Dynamically load models from S3 when invoked
  49. 49. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker multi-model endpoints Can achieve the Significant cost savings ! [Example] When XGBoost-based ML models to predict housing prices for individual market segments are deployed:
  50. 50. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker managed spot training ML 모델의 학습 과정에서 많은 비용이 발생할 수 있습니다 ! $$$  ML 모델의 학습은 수 분(min)에서 수 주(week)까지 소요될 수 있음  비용 효율적인 EC2 스팟 인스턴스 사용 시 ML 모델의 학습이 중단될 수 있음 (ML 모델 학습이 중단되지 않도록 해야 함) ML 모델 학습에 스팟 인스턴스를 사용하기 위해선 복잡한 툴 구축 작업이 필요합니다
  51. 51. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker managed spot training 각 학습 작업에 대한 비용 절감을 시각화 Amazon EC2 On-Demand 인스턴스와 비교하여 학습 비용 절감 스팟 용량 관리 및 중단 자동 처리 빌트인 알고리즘 및 사용자 정의 알고리즘 및 프레임워크 지원 모든 SageMaker 학습 가능 중단 없음 알고리즘 및 프레임워크 지원 완벽한 가시성 자동 모델 튜닝 및 강화 활용 90%까지 비용 절감 Seoul GA 서울 리전 기준 Image Classification & Object Detection 약 65% 절감, BlazingText 약 75% 절감 ML 모델 학습 비용을 최대 90%까지 절감할 수 있습니다!
  52. 52. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker managed spot training Dataset Volume: 1.2GB Number of Samples: Training set about 15,420 , Validation set 15,187 Image Shape: 224x224x3 (Resized) Model: SageMaker Built-in ResNet-50 (50 layers) Epochs: 20, Mini-batch size: 128 Instance Type Total Elapsed Secs Training Secs Billable Secs Training Savings Time increased p3.2xlarge On-demand 1579 1445 1445 - - Managed Spot 1676 1478 522 64.7% 6.1% p3.8xlarge On-demand 1518 1384 1384 - - Managed Spot 1602 1396 493 64.7% 5.5% Experiment: Caltech-256 Image Classification
  53. 53. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker managed spot training Experiment: Pascal VOC Object Detection Dataset Volume: 2GB Number of Samples: Training set about 16,000 , Validation set 4,000 Image Shape: 300x300x3 (Resized) Model: SageMaker Built-in ResNet-50 (50 layers) Epochs: 10, Mini-batch size: 32 Instance Type Total Elapsed Secs Training Secs Billable Secs Training Savings Time increased p3.2xlarge On-demand 2173 2022 2022 - - Managed Spot 2334 2094 739 64.7% 7.4% p3.8xlarge On-demand 1828 1688 1688 - - Managed Spot 1913 1698 598 64.7% 4.6%
  54. 54. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon SageMaker managed spot training Experiment: ImageNet Image Classification Dataset Volume: 138GB for training, 5GB for validation Number of Samples: Training set about 1.2million , Validation set 50,000 Image Shape: 224x224x3 (Resized) Model: SageMaker Built-in ResNet-50 (50 layers) image classification Number of GPUS: 16 (two p3.16xlarge instances) Epochs: 2, Mini-batch size: 256 Type Training Secs Billable Secs Training Savings Time increased Type On-demand 1894 1894 - - On-demand Managed Spot 1963 693 64.69% 3.64% Managed Spot
  55. 55. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS 머신러닝(ML) 교육 및 자격증 Amazon의 개발자와 데이터 과학자를 교육하는 데 직접 활용 되었던 커리큘럼을 기반으로 학습하세요! 전체 팀을 위한 머신러닝 교육 원하는 방법으로! 교육 유연성 제공 전문성에 대한 검증 비즈니스 의사 결정자, 데이터 과학자, 개발자, 데이터 플랫폼 엔지니어 등 역할에 따라 제공되는 맞춤형 학습 경로를 확인하세요. 약 65개 이상의 온라인 과정 및 AWS 전문 강사를 통해 실습과 실적용의 기회가 제공되는 강의실 교육이 준비되어 있습니다. 업계에서 인정받는 ‘AWS 공인 머신러닝 – 전문분야’ 자격증을 통해 머신러닝 모델을 구축, 학습, 튜닝 및 배포하는 데 필요한 전문 지식이 있음을 입증할 수 있습니다. https://aws.amazon.com/ko/training/ learning-paths/machine-learning/
  56. 56. © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Innovate 온라인 컨퍼런스에 참석해주셔서 대단히 감사합니다. aws-korea-marketing@amazon.com twitter.com/AWSKorea facebook.com/amazonwebservices.ko youtube.com/user/AWSKorea slideshare.net/awskorea twitch.tv/aws 저희가 준비한 내용, 어떻게 보셨나요? 더 나은 세미나를 위하여 설문을 꼭 작성해 주시기 바랍니다.
  57. 57. Thank you! © 2020, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×