Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Build and Host Real-world Machine Learning Services from Scratch @ pycontw2019

Title: Build and Host Real-world Machine Learning Services from Scratch
Author: Joe Tseng from Umbo CV

  • Be the first to comment

Build and Host Real-world Machine Learning Services from Scratch @ pycontw2019

  1. 1. Joe | AI Engineer
  2. 2. Building and Hosting Real-World Machine Learning Services from Scratch How to make A.I. work?
  3. 3. Joe Tseng • AI engineer • Tainan.py - organizer • PyConTW 15/16/18/19 - speaker • Our booth
  4. 4. Why this talk? 😎 Cool stuff to share 🛠 An extended version of the previous talk
  5. 5. November 2014 @AppWorks
  6. 6. Conventional 2D Trip Zone Umbo 3D Virtual Tripwire Home Surveillance: Intruder Detection pedestriancamera intruder cat Object/Event Recognition ! high precision / low false alarm 
 performance • 3D localized detection • precise dimension measurement • machine recognition DEMO1 DEMO2 DEMO3 Advantage November 2014 @AppWorks
  7. 7. @Songde Rd. November 2015
  8. 8. @Songde Rd. November 2015
  9. 9. June 2016 @Songde Rd.
  10. 10. Image- LQ (t+1) DQTN DQTN Image- LQ (t+1) Image- HQ (t) Image- HQ (t+3) Image- LQ (t+2) DQTN DQTN Image- LQ (t+2) CNN-3x3(3) CNN-1x1(16) CNN-3x3(16) Reconstructed HQ(t+1) CNN-3x3(3) CNN-1x1(16) CNN-3x3(16) Reconstructed HQ(t+2) Fusion Fusion ACCV 2016 @Songde Rd.
  11. 11. Demo Workshop in CVPR 2016 @Songde Rd.
  12. 12. • Human segmentation + others • Cloud-based solution Convergence I usually recommend to brainstorm at least six different projects … source
  13. 13. Idea: send camera streams to the cloud, and then do cool stuff 🐣 Autonomous video security platform
  14. 14. Our Solution Umbo Learning Cameras Real-time alerts Umbo Light
  15. 15. Issues Need to Be Solved Cameras - hardware/firmware/design Cloud - provider/streaming/application A.I. - PoC => service WebSocket
  16. 16. • On production in 2016 • From PoC projects to online services TruePlatform and Umbo Light
  17. 17. • Bugs • Feature improvement • Infrastructure improvement • Deployment • Healthy check • Monitoring 🚒 🔥 Service 🔥 🚒
  18. 18. When Real-World Services Meet Real-World Challenges Factory 95% accuracy 70% 45% 80% Parking Lot 55% 40%
  19. 19. How to improve this?
  20. 20. Our A.I. solution consists of heuristics algorithms, machine learning models, and, data processing pipelines. Which one should we improve first? 🧐
  21. 21. Collect More Data Data Data and tweak the machine learning models in the meantime.
  22. 22. Your data processing pipeline should support data collection tasks: customer feedback sampling algorithms false-negative miner … How and where to collect data?
  23. 23. • Can dispatch tasks to • Our own labelers • Third-party labelling services • AWS Mechanical Turk • 😃 integration welcome! • Audit mechanism • Backend: • Flask + plugins • NumPy • Celery/Redis/Pymongo • uWSGI Labeling Platform source: https://github.com/nightrome/cocostuff
  24. 24. The Simplified CV Architecture Media Server CV Forwarder Load Balancer ZMQ Stream Router Stream Manager Stream Manager Stream Manager Load Balancer CV Worker CV Worker CV Worker HTTP Alert Endpoint +
  25. 25. The Simplified CV Architecture Media Server CV Forwarder Load Balancer ZMQ Stream Router Stream Manager Stream Manager Stream Manager Load Balancer CV Worker CV Worker CV Worker HTTP Alert Endpoint +
  26. 26. • Stream Router/Manager was written by Python2 ❤ ❤ • Python2 => Python3 • byte string / decode • isinstance • format • API changes • Results • Python3 ❤ ❤ ❤ • Enable a lot of cool features • F-strings, typing, asyncio, tracemalloc, etc. Python 2 to 3
  27. 27. • Refactor Stream Manager • Happy to use async/await • Use pipeline pattern • Still use run_in_executor/ ThreadPoolExecutor if needed • Results • Cleaner architecture • Performance boost Adopt Python asyncio
  28. 28. • Refactor CV Worker • Lua => Python • Torch => PyTorch • Use aiohttp server • Results • Easier to maintain/upgrade • High performance • GPU resources are the bottleneck • Python package ecosystem 👍 Torch to PyTorch
  29. 29. • Debug • Use pyflame to profile the program • Use tracemaclloc to find memory usage • Use Valgrind to check your C++ code • Results • Understand more and gotcha! Fighting Bugs
  30. 30. 🏗 Boost maintainability 🕵 Better infrastructure monitoring 🐞 Have more features Outcomes
  31. 31. Machine Learning Pipelines
  32. 32. Good examples Uber: Michelangelo Airbnb: Bighead
  33. 33. Please make sure you have clear goals, practical user stories, and enough resources Before you start building the pipeline
  34. 34. • Goals • Timeline • Outcome • User stories • Blueprint • Concerns • Resource • Infra eng • Data eng • Researcher Behind the Machine Learning Pipelines
  35. 35. • ✅ Data collection • ✅ Model training • ✅ Model evaluation • 🧐 Model deployment Our Progress
  36. 36. • Cloud • Computing/Network => architecture • GPU => throughput + algorithm • ML pipeline • Computing • GPU => provider + integration • Labeling => automation Cost
  37. 37. • SLO • Monitor first • Set up alerts • Oncall process • Maintainability • DevOps • Adopt engineering best practices Service Level Objectives and Maintainability
  38. 38. Why Edge? Cost / Performance / Latency / Network Traffic / Security
  39. 39. • Launched in 2019.06 • From Chaos to Stability 🚒 • Edge + Cloud solution • ML pipeline is necessary • Data • Model Our AiCamera
  40. 40. This is our story, how about yours? 🤔
  41. 41. Product mindset matters After all, you have to serve your ML service to the real-world 😏 The takeaway
  42. 42. What’s next?
  43. 43. Thank you
  44. 44. • AI Engineer • Backend Engineer • Front-end Engineer • https://umbocv.ai/join_us • Our booth: Join Our Story

×