Apache MXNet/Gluon을 이용한 입술 읽기(Lipreading) 모델 만들기 김형준, SK텔레콤 입술 읽기(Lipreading)는 화자의 입의 움직임으로부터 텍스트를 디코딩하는 과제입니다. 대표적인 LipNet 모델을 Apache MXNet 기반 Gluon을 이용하여 구현한 과정과 이를 MXNet Model Example에 소스 공헌을 하면서 겪었던 과정들을 공유합니다.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Apache MXNet/Gluon을 이용한
입술 읽기(Lip Reading) 모델 만들기
김형준
Research Engineer
SK Telecom

2. 발표자료 바로 공개
발표자료는 발표 종료 후 해당
사이트에서 바로 보실 수 있습니다
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Who am I
김 형 준
- AI Research Engineer
- AI/Machine Learning Application Engineer
- Data Analysis Platform Engineer
- Data Scientist

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SK Telecom AI Center

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Open Source & Community
(2018.07 ~ ) Apache MXNet Medium(Tech Blog) Writer & Code Contributor
(2018.06) A way to Benchmark Your Deep Learning Framework, Psygrams
(2018.01) Sentiment Analysis with grad-CAM and Target2Vec, Microsoft X 바벨피쉬
(2017.10) Machine Learning Platform Development, 데이터야놀자
(2015.12) Anomaly Detection using MRFs, Naver D2 개발자 커뮤니티
(2015.06) Sentiment Analysis, R User Conference in Korea 2015

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Contribution

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MXNet/Gluon Model

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
1. 도움이 된 것들
2. 처음 Pull Request 하던 날
3. Lip Model 예제 작성
4. Review & Merged
5. gluonCV, gluonNLP
6. Discussion

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MODEL: Deep Learning Frameworks
https://www.slideshare.net/awskorea/2-deep-learning-with-gluon
* Low level-control
- Hybridized Gluon(Symbolic)
* High level
- Gluon (Imperative)
* GluonCV, Gluon-NLP
* Semantic Versioning
- MXNet Community

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MODEL: MXNet/Gluon Study (17.12 ~ 18.06)

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MODEL: MXNet/Gluon Study (17.12 ~ 18.06)
- (NLP) Word2vec (skip-gram)
- (Vision) GAN
- (Vision) Pix2Pix
- (Vision) Grad-cam & Integrated-gradients
- (NLP) End to end Memory Network
- (NLP) LSTM with Custom Embedding

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MODEL: MXNet/Gluon
- 때로는 다른 누군가가 만들어주더라도
- 때로는 실패하고

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MODEL: MXNet/Gluon
- 성공했을 때의 경험은 잊을 수 없습니다

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SOSCON 2018(https://www.slideshare.net/awskorea/apache-mxnet-soscon-2018/)
Contribution: Apache MXNet 공헌하는 방법

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Contribution: Apache MXNet 공헌하는 방법
- Source Code Contribution
: [MXNet Github](https://github.com/apache/incubator-mxnet)
- Blog
: [Medium Blog](https://medium.com/apache-mxnet)
- Community
: [MXNet Forum](https://discuss.mxnet.io/)
[Slack – mxnet, AWSKRUG ]
[Facebook – MXNet Korea User Group](https://www.facebook.com/groups/mxnetkr/)

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Medium Blog
(18.07) A way to Benchmark Your Deep Learning Framework On-premise

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Medium Blog
MXNet Makes Us Faster And Stronger!

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution
[MXNet Github](https://github.com/apache/incubator-mxnet)

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution
오타 수정도
번역도
가이드 문서 작성도
디자인 작업도
의견 제시도
모두 컨트리뷰션입니다
“More Common Than You Think: An In-Depth Study of Casual
Contributors”, 2016”
https://naver.github.io/OpenSourceGuide/book/BetterContribution/why-contribute-to-open-source.html

22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution

23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution
https://mxnet.apache.org/versions/master/community/contribute.html#setup-mxnet-for-development

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution
[MXNet Github](https://github.com/apache/incubator-mxnet)

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution

26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Source Code Contribution
해당 이슈에 다음 행동들을 통해 기여해 주세요
- 의견 및 아이디어 제시
- 관심 어필
- Pull Request

27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bug Fix
- Issues

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bug Fix
- PR(Pull Request)

30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- https://arxiv.org/abs/1611.01599
- LipNet: End-to-End Sentence-level Lipreading (2016)
- please
입술 읽기(Lip reading)는 화자의 입의 움직임으로부터 텍스트를 디코딩하는 과제입니다.

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- https://arxiv.org/abs/1611.01599
- LipNet: End-to-End Sentence-level Lipreading (2016)

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MXNet 공헌하기
- Example

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- https://github.com/apache/incubator-mxnet/tree/master/example/gluon/lipnet
Data
Collect
Data
Prepare
- Example
Model
Build
Model
Train
Model
Inference
utils./download_data.py utils/preprocess_data.py data_loader..py
models/network.py
trainer.py
main.py
Infer.py
- 정승환, 김형준(SK Telecom)
- 18.12.01 ~ 12.14 (약 2주간 개발)
- 18.12.14 ~ 02.15 (약 2달간 리뷰)

35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Collect
- Example
bbaf3s.mpg

36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Collect
- Example
- 16GB
- Multiple processes

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Prepare
- Example
- Using Face
Landmark
Detection(http://dlib
.net/), It’s able to
extract the mouth
from a video.
- Using ffmpeg, It
changes 1 video to
75 frames

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Prepare
- Example
- Preprocessing
: Video →
cropped images
- DataLoader
MXNet/Gluon
: Generator
(nd.array)

39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Prepare
전처리 시간이 너무 오래 걸림
• 54 hours  1 process
• 1 hour  54 processes

40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Prepare
Public S3 이용

41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
Data
Prepare
“직접 다운로드 & 전처리” 또는 “전처리된 데이터 이용”
- https://github.com/apache/incubator-mxnet/tree/master/example/gluon/lipnet

42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- Model: (ST)CNN + GRU
Model
Build

43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- Model: (ST)CNN + GRU
Model
Build
https://arxiv.org/pdf/1611.01599.pdf

44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- Model: (ST)CNN + GRU
Model
Build
https://arxiv.org/pdf/1611.01599.pdf

45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- Model: (ST)CNN + GRU
Model
Build
https://arxiv.org/pdf/1611.01599.pdf

46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- 학습
Model
Train
trainer = Train(config)
trainer.build_model()
trainer.load_dataloader()
trainer.run()

47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- 추론
Model
Inference
trainer = Train(config)
trainer.build_model()
trainer.load_dataloader()
trainer.infer_batch(trainer.valid_dataloader)

48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet
- 추론 예제
Model
Inference

49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- (1) Fork
- (2) New branch
- (3) Code/Doc
- (4) Test

51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- (5) Fill in PR template

52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- (6) Review
https://github.com/apache/incubator-mxnet/pull/13647

53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- (6) Review

54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- (7) Merged

55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- 고려사항들 요약
- pylint
- huge data → s3
- Multi gpus
- Hybridize
- Test code
- Pretrained model

56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- pylint

57. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- huge data → s3

58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- Multi gpus

59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- Hybridize
https://github.com/apache/incubator-mxnet/pull/13647/commits/f487255be48f89231ba917c570266af759cc1116

60. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- Test code

61. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- Pretrained model

62. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
- 실수?!
- https://github.com/apache/incubator-
mxnet/tree/master/example/gluon/lipnet

63. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
LipNet Pull Requests
가장 힘들었던 점
- 미국과 한국의 시간 차
- 의사소통: 영어
- 체력 & 끈기: 과도한 요구? 이쯤이면 ‘approve’ 해 줍시다

64. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

65. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GluonCV
- Model zoo

66. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GluonCV
- Model zoo

67. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GluonCV
- Model zoo

68. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
GluonNLP
- Model zoo

69. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

70. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lip Reading
- Large-scale Visual Speech Recognition
: https://arxiv.org/pdf/1807.05162.pdf

71. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lip Reading
- Large-scale Visual Speech Recognition
: https://arxiv.org/pdf/1807.05162.pdf

72. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Contribution?
- 시간 & 노력 vs 보람, 성취감, 자부심
- 실력 향상
- 최고의 실력자들로 부터 피드백
- 협업
- 테스트 코드
- 컨벤션
- “스프린트 / 핸즈업 / 밋업” 통한 사용자 확대 및 인적 네트워크
- 타인에게 Inspire
- 스터디
- 발표
- 스프린트 / 핸즈업 / 밋업

73. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Life is too short, Dive into Deep Learning!
http://ko.d2l.ai/

74. 여러분의 피드백을 기다립니다!
#AWSSummit 해시태그로
소셜미디어에 여러분의
행사소감을 올려주세요.
AWS Summit Seoul 2019
모바일 앱과 QR코드를 통해
강연평가 및 설문조사에
참여하시고 재미있는 기념품을
받아가세요.
내년 Summit을 만들 여러분의
소중한 의견 부탁 드립니다.
© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.