Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

from ai.backend import python @ pycontw2018

Title: from ai.backend import python
Author: Joe Tseng from Umbo CV

  • Login to see the comments

from ai.backend import python @ pycontw2018

  1. 1. Joe | AI Engineer
  2. 2. from ai.backend import python A (backend → AI) story and several thoughts about the future
  3. 3. Joe Tseng • Backend (AI) engineer • Tainan.py - organizer • PyConTW 2015/2016 - speaker
  4. 4. Outline About the story Umbo Computer Vision intro Develop and maintain CV services Build our machine learning pipelines What I have learned About the future My imagination
  5. 5. Outline About the story ➡ Umbo Computer Vision intro Develop and maintain CV services Build our machine learning pipelines What I have learned About the future My imagination
  6. 6. Autonomous video security systems
  7. 7. 400 million cameras ref: UK Police Scientific and Development Branch (PSDB) 12 or less cameras per operator - UK Police Scientific Department
  8. 8. Our Solution Existing cameras Umbo Learning Cameras Real-time alerts Umbo Light
  9. 9. Infra We use a lot of cloud services and have our own GPU clusters
  10. 10. Backend Languages All our AI team members speak Python ❤ …
  11. 11. Flow ./enter_dev_env.sh + write code with unit tests + code review + CI/CD
  12. 12. Monitoring We log, monitor and measure our services
  13. 13. Teams Hardware, Dev, CV, Infra, Research, PoC, QA, FAE …
  14. 14. Develop and maintain all computer vision and machine learning features The definition of an AI Engineer
  15. 15. 1. Develop and maintain CV services 2. Build our machine learning pipelines Our jobs
  16. 16. Backend → AI How ?
  17. 17. Outline About the story Umbo Computer Vision intro ➡ Develop and maintain CV services Build our machine learning pipelines What I have learned About the future My imagination
  18. 18. • On production in 2016 • From PoC projects to online services Previously on SmartCloud and Umbo Light
  19. 19. The Simplified CV Architecture Media Server CV Forwarder Load Balancer ZMQ Stream Router Stream Manager Stream Manager Stream Manager Load Balancer CV Worker CV Worker CV Worker HTTP Alert Endpoint
  20. 20. Where is AI?
  21. 21. Media Server CV Forwarder Load Balancer ZMQ Stream Router Stream Manager Stream Manager Stream Manager Load Balancer CV Worker CV Worker CV Worker HTTP Alert Endpoint
  22. 22. Media Server CV Forwarder Load Balancer ZMQ Stream Router Stream Manager Stream Manager Stream Manager Load Balancer CV Worker CV Worker CV Worker HTTP Alert Endpoint +
  23. 23. How much can the backend do?
  24. 24. 87% How much the backend can do
  25. 25. • Infra • LB / ASG / VPC … • Monitoring • PaperTrail, Cloud Watch … • Development • Add new features • CI/CD • Maintenance • Upgrade packages • Refactor • Debug 87% MS CF LB SR SM SM SM LB CW CW CW AE
  26. 26. • Infra • LB / ASG / VPC … • Monitoring • PaperTrail, Cloud Watch … • Development • Add new features • CI/CD • Maintenance • Upgrade packages • Refactor • Debug 87% MS CF LB SR SM SM SM LB CW CW CW AE
  27. 27. • Stream Router/Manager was written by Python2 ❤ ❤ • Python2 => Python3 • byte string / decode • isinstance • format • API changes • Results • Python3 ❤ ❤ ❤ • Enable a lot of cool features • F-strings, typing, asyncio, tracemalloc, etc. 87%
  28. 28. • Refactor Stream Manager • Happy to use async/await • Use pipeline pattern • Still use run_in_executor/ ThreadPoolExecutor if needed • Results • Cleaner architecture • Performance boost 87%
  29. 29. • Refactor CV Worker • Lua => Python • Torch => PyTorch • Use aiohttp server • Results • Easier to maintain/upgrade • High performance • GPU resources are the bottleneck • Python package ecosystem 👍 87%
  30. 30. • Debug • Use pyflame to profile the program • Use tracemaclloc to find memory usage • Use Valgrind to check your C++ code • Results • Understand more and gotcha! 87%
  31. 31. Ping me if you want to listen this story (and also give us a talk) in Tainan.py 😼
  32. 32. 13% How much the backend cannot handle
  33. 33. • About Machine Learning • Not so easy to learn 13%
  34. 34. More than 13% In fact
  35. 35. • Computer Vision and Machine Learning • They are everywhere • The learning curve is steep More than 13% MS CF LB SR SM SM SM LB CW CW CW AE
  36. 36. You need to delve really deep into Computer Vision and Machine Learning to build a self-improving system
  37. 37. Outline About the story Umbo Computer Vision intro Develop and maintain CV services ➡ Build our machine learning pipelines What I have learned About the future My imagination
  38. 38. Why machine learning pipelines? Improve services => customers 😆
  39. 39. How to improve your services Measure first New models / heuristics approaches Need to do training or redesigning Need more (or specific) labeled data Training! Need to compare with the existing ones Get the best on production … Measure again 👷
  40. 40. Machine Learning Pipelines
  41. 41. • Meet Michelangelo: Uber’s Machine Learning Platform • Machine Learning Pipeline for Real- Time Forecasting @Uber Marketplace Michelangelo (Uber)
  42. 42. How much can backend do?
  43. 43. 0%~87%
  44. 44. Why 0%? 🐵 without domain knowledge
  45. 45. Why 87%? With domain knowledge + good communication with others + speaking the same language (Python!)
  46. 46. Catch a glimpse of our pipelines
  47. 47. • Can dispatch tasks to • Our own labelers • Third-party partners • MTurkers • Backend: • Flask + plugins • NumPy • Celery/Redis/Pymongo • uWSGI Labeling Platform source: https://github.com/nightrome/cocostuff
  48. 48. • Can train you a lot of new models with different parameters • Backend: • Airflow • Celery Training Pipeline
  49. 49. • Can visualize • The ground truth data • The evaluation results • Backend: • Flask + plugins • NumPy • SQLAlchemy Visualization Platform
  50. 50. A faster pipeline means a shorter turnaround time! Speed matters for a startup
  51. 51. Outline About the story Umbo Computer Vision intro Develop and maintain CV services Build our machine learning pipelines ➡ What I have learned About the future My imagination
  52. 52. Recap Services & Pipelines
  53. 53. Collaboration Researcher vs. Engineer
  54. 54. Domain Knowledge Have a basic understanding of ML/CV with practical experience
  55. 55. Python From different aspects
  56. 56. Outline About the story Umbo Computer Vision intro Develop and maintain CV services Build our machine learning pipelines What I have learned About the future ➡ My imagination
  57. 57. from ai.backend import * Python, Golang, Java, C#, Javascript, PHP, Ruby, etc.
  58. 58. To build your pipelines, Python is the best default language Pipeline → Service
  59. 59. Every young engineer knows machine learning But some of them may have not heard of GIL 😥
  60. 60. High-quality (ai) backends are needed
  61. 61. Thank you
  62. 62. • Backend, CV, Front-end, App, QA, FAE, Firmware • joe.tseng@umbocv.com We are hiring

×