This document summarizes the development of an image processing API and ML system to meet requirements of processing images in under 2 seconds, with visibility, persistence, scalability, and extensibility. It describes splitting the API and ML into separate services, optimizing processing times from 4 to 1 second through changes like using Kafka, scaling the system, and troubleshooting issues with performance spikes, message brokers, and databases. Lessons learned include distinguishing CPU from IO-bound tasks and the challenges of Go versus Python. The overall goal of under 1 second per request processing time was eventually achieved.

1. Go && Python
Event driven ML
Lalafo case

3. How it all started
Part 1. The Phantom Menace

5. StackOverflow: Ready to rescue

6. 200 RPM

8. $
Feedback

9. $
Feedback
Processes Monitoring
RAM - expiration?
Relational DB?

10. Any issues there?
1. Request monitoring (?)
2. Hard to reason GPU usage (varies 3 to 7GB)
3. ~5Gb RAM out of the box (scale?)
4. Redis is in-memory storage (temporary)
5. Threading != Parallel

11. http://www.nooooooooooooooo.com/

12. What if we split
API and ML
Part 2. A new hope

13. Requirements
Lalafo case
1. Single image/request processing time: < 2 seconds
2. Visibility
3. Persistence
4. Scalability
5. Make it extendable for new features
a. Price prediction
b. Similarity search
c. Segmentation
6. SDK friendly (well documented, tested etc)

14. $
Feedback
Processes Monitoring
RAM - expiration?
Relational DB?

15. After 2 months
API Listeners
Feedback
$

16. Requirements
Lalafo case
1. Single image/request processing time: < 2 seconds
2. Visibility - Decouple request and prediction
3. Persistence
4. Scalability
5. Make it extendable for new features
a. Price prediction
b. Similarity search
c. Segmentation
6. SDK friendly (well documented, tested etc)

17. 2 seconds per request
Part 3. The Empire Strikes Back

18. 4 seconds per request
API Listeners
1 second
0.01 sec

19. segmentio/kafka-go -> shopify/sarama
API Listeners
0.02 sec
0.01 sec

20. 3 seconds per request
API Listeners
0.02 sec
0.3 second
0.3 second
Synchronous request

21. How large is every image

22. 2.4 seconds per request
API Listeners
0.02 sec
Synchronous request

23. 2.4 seconds per request
API Listeners
0.02 sec
Synchronous request

24. 2 seconds per request
API Listeners
0.02 sec
Synchronous request
Add ThreadPool

25. 2 seconds per request
Success?
Not yet

26. 3 seconds per request
API Listeners
0.02 sec

27. 3 seconds per request
API Listeners
0.02 sec
1 second and growing

28. 3 seconds per request

29. 2 seconds per request

30. ¯_(ツ)_/¯

31. 2 seconds per request
API Listeners
0.02 sec

32. 200 RPM -> 1200 RPM

33. 2 seconds per request
API Listeners
0.02 sec
Scale
Scale
Master -> Slave -> balancer
Topics -> Partitions

34. < 1 second per request
Success?
Not yet

36. Out of 8Gb

37. How to PyTorch in production

38. 7.6 Gb -> 600 Mb

39. 1 second per request
Success?
Not yet

40. Back to 2, 5, 10 seconds per request

41. Troubleshooting
API Listeners
0.02 sec
? ?

42. Troubleshooting
API Listeners
0.02 sec
?

43. Troubleshooting
API Listeners
0.02 sec
?
175k records per request

44. Troubleshooting
API Listeners
0.02 sec
?

45. Troubleshooting
API

46. SELECT * FROM ads LIMIT 1

47. 25 seconds -> 0.02 seconds

48. 0.02 seconds -> 0.002 second

49. Success?
Not yet

50. Faust 1.4.6: No latest offset
https://github.com/robinhood/faust

51. 3K images per minute 0_o

52. Success?
Not yet

53. Issues to solve
1. Occasional spikes in performance (GC, network latency)
2. Message broker (Kafka rebalancing, offset etc)
3. How to handle DB migrations
4. Something we are not aware of yet

54. Lessons learnt
1. CPU bound tasks != IO bound (¯_(ツ)_/¯)
2. High coupling - low cohesion
3. You need to know how to cook MongoDB
4. Go is not that obvious and library reach as Python
5. Simple != Easier
6. Concurrency != Parallelism (obviously)

55. < 1 second per request

56. + Live statistics from PostgreSQL
API Listeners

57. Thank you everyone
Success?