In this talk I will cover our experience moving CNET, from an in-house monolith image resizing application, to a containerized open source micro-service. I will detail the planning and development phases of the micro-service, which utilizes Thumbor. I will also cover the testing, deployment and monitoring strategies used in the process. Running in Docker Swarm Mode and delivering millions of image requests a day, the service has greatly improved stability and reduced infrastructure needs by 80%.

1. ON-DEMAND IMAGE
RESIZING
FROM PART OF THE MONOLITH
TO CONTAINERIZED
MICROSERVICE

2. Jonathan Lee
Lead Software Engineer, CNET
@ CBS Interactive

3. About CNET
● Dedicated to tech
product reviews, news,
price comparisons, free
software downloads,
videos, how-tos, and
podcasts.
● Established in 1994,
created by Halsey Minor
and Shelby Bonnie
● CNET Networks was
acquired by CBS
Interactive in 2008.
Monthly Users ~ 68 M
Monthly Unique Users ~ 45 M
Monthly Image Requests ~ 2.1 B

4. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

5. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

6. Legacy Architecture
Load Balancer
Nginx
Image
Resizer
Image
Resizer
Nginx
Server 2
NFS File system
Server 1
Image
Resizer
Nginx
Server 3
CDN

7. Legacy Architecture
Load Balancer
Nginx
Image
Resizer
Image
Resizer
Nginx
Server 2
NFS File system
Server 1
Image
Resizer
Nginx
Server 3

8. Legacy Architecture
Nginx
Image
Resizer
Image
Resizer
Nginx
Server 2Server 1
Image
Resizer
Nginx
Server 3

9. Legacy Architecture
Nginx
Image
Resizer
Server 1
Nginx
Image
Resizer
Server 2
Nginx
Image
Resizer
Server 3

10. Legacy Image Resizer
● Required a large
number of servers to run
● Large spikes in new
resize requests caused
micro-outages
● New effects for
redesigns (i.e. blur)
● Deploying new effects
required orchestration

11. ● Crop and Resizing
● Various built-in filters
● Image optimization
● Detectors
● Image source loaders
● Community dockerized
version
Thumbor https://store.docker.com/community/images/apsl/thumbor

12. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

13. ● Current Image
Resizer
○ Upgrade to
Imagemagick
Testing/Investigation
● Thumbor
single-process
○ Optimizers
on/off
○ Smart-crop
● Thumbor
multi-process
○ Optimizers
on/off
○ Smart-crop
Testing Scenarios:

14. Load Testing Choices:
• Siege
• AB
• GoBench
• Gatling.io
Decision:
Siege,
based on previous experience and simplicity
Testing/Investigation

15. Testing/Investigation

16. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 3.9G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

17. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 3.9G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

18. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 2.0G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

19. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 3.9G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

20. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 3.9G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

21. Testing/Investigation
Concurrent
Users
Memory CPU Load Requests Hits/s
10 4.8G 92 178 121973 7
20 N/A N/A N/A N/A N/A
30 N/A N/A N/A N/A N/A
... ... ... ... ... ...
10 3.9G 10.3 0.9 122670 21
20 2.4G 12.5 0.9 245340 23
30 2.7G 12.5 0.9 368010 23
LegacyThumbor

22. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

23. ● Real-time error tracking
● Support for python
tornado
● Out of the box support
with apsl/thumbor
Sentry

24. ● Time-series DB
● Based on Google’s
Borgmon
● Pull model
● Service Federation
Prometheus

25. ● Graphite is a time
series DB
● Push model
● Statsd is a data
statistics aggregator
● Statsd pushes
aggregate data to
Graphite for long term
storage
Graphite/Statsd

26. ● Metric analytics &
visualization suite
● Multiple data sources
● Alerting
● Custom dashboards
Grafana

27. The ELK Stack is a collection of
three open-source products from
Elastic.
● Elasticsearch is a NoSQL
database that is based on the
Lucene search engine.
● Logstash accepts and parses
messages and sends them to
Elasticsearch for storage.
● Kibana is a visualization layer
that works on top of
Elasticsearch.
ELK

28. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

29. Docker 1.10.x and 1.11.x
Server 1 Server 2 Server 3
Env fileEnv file Env file

30. Docker 1.10.x and 1.11.x
Thumbor
Server 1 Server 2 Server 3
Env file
Thumbor
Env file
Thumbor
Env file

31. Docker 1.10.x and 1.11.x
Thumbor
Server 1 Server 2 Server 3
Env file
Thumbor
Env file
Thumbor
Env file

32. Docker 1.10.x and 1.11.x
Thumbor
Nginx
Server 1 Server 2 Server 3
Env file
Thumbor
Nginx
Env file
Thumbor
Nginx
Env file

33. Docker 1.10.x and 1.11.x
Thumbor
Nginx
Server 1 Server 2 Server 3
Env file
Thumbor
Nginx
Env file
Thumbor
Nginx
Env file

34. Docker 1.10.x and 1.11.x
Thumbor
Nginx
Env
file
Thumbor
Nginx
Env
file
Thumbor
Nginx
Env
file
Load Balancer
NFS File system
CDN

35. ● Rolling deployment strategy
● Path based targeting
○ /news/*, /products/*,
/pictures/*, etc.
● Monitor system performance
and deploy additional servers if
needed
Push It Live!

36. Previous IMG Tier - 22 servers in rotation
• 96% avg memory usage
• 5.5 avg nginx active connections
• 1.68 nginx requests a second
• 1.88 avg active processes
• 2.27 requests a second
• 280 ms avg resize response time
Thumbor Tier - 3 servers in rotation
• 12.5% avg memory usage
• 13 avg nginx requests a second
• 0.384 avg requests per second
• 255 ms avg resize response time
*Statistics are per box
Results

37. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

38. Docker 1.12.x, 1.13.x, and 17.03
Server 1 Server 2 Server 3
Manager

39. ~ docker swarm init
Swarm initialized: current node (o2lqnophten3ybufd996zxc6m) is now a
manager
To add a worker to this swarm, run the following command:
docker swarm join
--token
SWMTKN-1-1kw6ht8rie6aeuokaeym2wc2yazb2vqxi6g92rog55o7sok2t8-0lxneopk6
7mto3o6ltuavckb8
192.168.65.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and
follow the instructions
Docker Swarm Mode

40. ~ docker swarm init
Swarm initialized: current node (o2lqnophten3ybufd996zxc6m) is now a
manager.
To add a worker to this swarm, run the following command:
docker swarm join
--token
SWMTKN-1-1kw6ht8rie6aeuokaeym2wc2yazb2vqxi6g92rog55o7sok2t8-0lxneopk6
7mto3o6ltuavckb8
192.168.65.2:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and
follow the instructions.
Docker Swarm Mode

41. Docker 1.12.x, 1.13.x, and 17.03
Server 1 Server 2 Server 3
docker swarm join --token
SWMTKN-1-1kw6ht8rie6aeuokaeym2wc2yazb2vqxi6g92rog55o7sok2t8-0lxneopk67mto3
o6ltuavckb8 192.168.65.2:2377

42. Docker 1.12.x, 1.13.x, and 17.03
Server 1 Server 2 Server 3
docker service create --name thumbor --replicas 3 ... thumbor

43. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Server 1 Server 2 Server 3
ThumborThumbor
Thumbor service
docker service create --name nginx -p 8080:80 --replicas 2 ... nginx

44. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
ThumborThumbor
Nginx service
Thumbor service
docker service scale thumbor=5

45. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor
docker service update --image <new-image> thumbor

46. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

47. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

48. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

49. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

50. Docker 1.12.x, 1.13.x, and 17.03
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

51. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

52. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor

53. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

54. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor

55. Docker 1.12.x, 1.13.x, and 17.03
Thumbor
Nginx Nginx
Server 1 Server 2 Server 3
Thumbor
Thumbor
Nginx service
Thumbor service
Thumbor
Thumbor

56. Agenda
1. Background
2. Testing
3. Monitoring
4. Deployment
5. Swarm mode
6. Roadmap

57. Finished:
● Webp
Upcoming:
● no_upscaling by default
● Gifv
● Smart Cropping
● Prometheus support for runtime metrics
Thumbor

58. Finished:
● Docker stack deploy
Upcoming:
● Auto Scaling containers
● Prometheus monitoring
● Secrets
● Volume plugins
Docker

59. Open Source, FTW!
statsd
Prometheus.io

60. Thank You!
@beenanner #thumborswarm