Title: from ai.backend import python Author: Joe Tseng from Umbo CV

1. Joe | AI Engineer

2. from ai.backend import python
A (backend → AI) story and several thoughts about the future

3. Joe Tseng
• Backend (AI) engineer
• Tainan.py - organizer
• PyConTW 2015/2016 - speaker

4. Outline
About the story
Umbo Computer Vision intro
Develop and maintain CV services
Build our machine learning pipelines
What I have learned
About the future
My imagination

5. Outline
About the story
➡ Umbo Computer Vision intro
Develop and maintain CV services
Build our machine learning pipelines
What I have learned
About the future
My imagination

6. Autonomous video security systems

7. 400 million cameras
ref: UK Police Scientific and Development Branch (PSDB)
12 or less cameras per
operator
- UK Police Scientific Department

8. Our Solution
Existing cameras
Umbo Learning
Cameras
Real-time alerts
Umbo Light

9. Infra
We use a lot of cloud services and have our own GPU clusters

10. Backend Languages
All our AI team members speak Python ❤
…

11. Flow
./enter_dev_env.sh + write code with unit tests + code review + CI/CD

12. Monitoring
We log, monitor and measure our services

13. Teams
Hardware, Dev, CV, Infra, Research, PoC, QA, FAE …

14. Develop and maintain all computer vision and
machine learning features
The definition of an AI Engineer

15. 1. Develop and maintain CV services
2. Build our machine learning pipelines
Our jobs

16. Backend → AI
How ?

17. Outline
About the story
Umbo Computer Vision intro
➡ Develop and maintain CV services
Build our machine learning pipelines
What I have learned
About the future
My imagination

18. • On production in 2016
• From PoC projects to online services
Previously on SmartCloud and Umbo Light

19. The Simplified CV Architecture
Media
Server
CV
Forwarder
Load
Balancer
ZMQ
Stream
Router
Stream
Manager
Stream
Manager
Stream
Manager
Load
Balancer
CV
Worker
CV
Worker
CV
Worker
HTTP
Alert Endpoint

20. Where is AI?

21. Media
Server
CV
Forwarder
Load
Balancer
ZMQ
Stream
Router
Stream
Manager
Stream
Manager
Stream
Manager
Load
Balancer
CV
Worker
CV
Worker
CV
Worker
HTTP
Alert Endpoint

22. Media
Server
CV
Forwarder
Load
Balancer
ZMQ
Stream
Router
Stream
Manager
Stream
Manager
Stream
Manager
Load
Balancer
CV
Worker
CV
Worker
CV
Worker
HTTP
Alert Endpoint
+

23. How much can the backend do?

24. 87%
How much the backend can do

25. • Infra
• LB / ASG / VPC …
• Monitoring
• PaperTrail, Cloud Watch …
• Development
• Add new features
• CI/CD
• Maintenance
• Upgrade packages
• Refactor
• Debug
87%
MS
CF
LB SR
SM
SM
SM
LB
CW
CW
CW
AE

26. • Infra
• LB / ASG / VPC …
• Monitoring
• PaperTrail, Cloud Watch …
• Development
• Add new features
• CI/CD
• Maintenance
• Upgrade packages
• Refactor
• Debug
87%
MS
CF
LB SR
SM
SM
SM
LB
CW
CW
CW
AE

27. • Stream Router/Manager was written by Python2 ❤ ❤
• Python2 => Python3
• byte string / decode
• isinstance
• format
• API changes
• Results
• Python3 ❤ ❤ ❤
• Enable a lot of cool features
• F-strings, typing, asyncio, tracemalloc, etc.
87%

28. • Refactor Stream Manager
• Happy to use async/await
• Use pipeline pattern
• Still use run_in_executor/
ThreadPoolExecutor if needed
• Results
• Cleaner architecture
• Performance boost
87%

29. • Refactor CV Worker
• Lua => Python
• Torch => PyTorch
• Use aiohttp server
• Results
• Easier to maintain/upgrade
• High performance
• GPU resources are the bottleneck
• Python package ecosystem 👍
87%

30. • Debug
• Use pyflame to profile the program
• Use tracemaclloc to find memory
usage
• Use Valgrind to check your C++
code
• Results
• Understand more and gotcha!
87%

31. Ping me if you want to listen this
story (and also give us a talk) in
Tainan.py 😼

32. 13%
How much the backend cannot handle

33. • About Machine Learning
• Not so easy to learn
13%

34. More than 13%
In fact

35. • Computer Vision and Machine
Learning
• They are everywhere
• The learning curve is steep
More than 13%
MS
CF
LB SR
SM
SM
SM
LB
CW
CW
CW
AE

36. You need to delve really deep into
Computer Vision and Machine Learning to
build a self-improving system

37. Outline
About the story
Umbo Computer Vision intro
Develop and maintain CV services
➡ Build our machine learning pipelines
What I have learned
About the future
My imagination

38. Why machine learning pipelines?
Improve services => customers 😆

39. How to improve your services
Measure first
New models / heuristics approaches
Need to do training or redesigning
Need more (or specific) labeled data
Training!
Need to compare with the existing ones
Get the best on production
…
Measure again
👷

40. Machine Learning Pipelines

41. • Meet Michelangelo: Uber’s Machine
Learning Platform
• Machine Learning Pipeline for Real-
Time Forecasting @Uber Marketplace
Michelangelo (Uber)

42. How much can backend do?

43. 0%~87%

44. Why 0%?
🐵 without domain knowledge

45. Why 87%?
With domain knowledge + good communication with others + speaking the same language (Python!)

46. Catch a glimpse of our pipelines

47. • Can dispatch tasks to
• Our own labelers
• Third-party partners
• MTurkers
• Backend:
• Flask + plugins
• NumPy
• Celery/Redis/Pymongo
• uWSGI
Labeling Platform
source: https://github.com/nightrome/cocostuff

48. • Can train you a lot of new
models with different parameters
• Backend:
• Airflow
• Celery
Training Pipeline

49. • Can visualize
• The ground truth data
• The evaluation results
• Backend:
• Flask + plugins
• NumPy
• SQLAlchemy
Visualization Platform

50. A faster pipeline means a shorter turnaround time!
Speed matters for a startup

51. Outline
About the story
Umbo Computer Vision intro
Develop and maintain CV services
Build our machine learning pipelines
➡ What I have learned
About the future
My imagination

52. Recap
Services & Pipelines

53. Collaboration
Researcher vs. Engineer

54. Domain Knowledge
Have a basic understanding of ML/CV with practical experience

55. Python
From different aspects

56. Outline
About the story
Umbo Computer Vision intro
Develop and maintain CV services
Build our machine learning pipelines
What I have learned
About the future
➡ My imagination

57. from ai.backend import *
Python, Golang, Java, C#, Javascript, PHP, Ruby, etc.

58. To build your pipelines, Python is the best
default language
Pipeline → Service

59. Every young engineer knows machine learning
But some of them may have not heard of GIL 😥

60. High-quality (ai) backends are needed

61. Thank you

62. • Backend, CV, Front-end, App,
QA, FAE, Firmware
• joe.tseng@umbocv.com
We are hiring