SQS lesson learnt

1. Gogolook SQS
lesson learnt

2. Hello!
I am Kakashi
You can find me at:
Twitter: @kakashiliu

3. The story is ...

5. Previous Status
◎ Worker: c3.large * 30~50, spot instance
◎ CPU utilization: less than 20%
◎ Messages: over than 1 billion / per day

6. SQS 基本介紹

7. SQS 基本設定 2
◎ First 1 million SQS requests per month are free
◎ $0.476 / per 1 million SQS requests
◎ A single request can have from 1 to 10 messages,
up to maximum total payload of 256 kb
◎ requests including:
CreateQueue, ListQueues, DeleteQueue, SendMessage,
SendMessageBatch, ReceiveMessage, ChangeMessageVisibility,
ChangeMessageVisibilityBatch, DeleteMessage, DeleteMessageBatch,
SetQueueAttributes, GetQueueAttributes, GetQueueUrl, AddPermission,
and RemovePermission.

8. SQS新政1 : 流程改善
◎ 每個worker每隔1min, 用crontab叫他們復活, 沒事幹就死掉
=> cpu utilization low, cpu loading high
◎ Solution:
利用 while True 加上 SQS long polling (20s) , 沒事做的時候
sleep 60s
◎ 結果: Loading降低, CPU utilization 沒提升很多

9. SQS新政2 : 斷開鎖鏈
◎ 發現有network bound: 3rd library loggly 沒有使用keep-alive
的方式連接
◎ Solution:
1) python lib 替換 urllib2 為 requests.session
2) 並且使用batch sent log
◎ 結果: 清queue速度快一倍
worker cpu utilization 上升數倍

10. SQS新政3 : 減少request量
◎ 利用sent message batch的方式, Batch sent 100 messages or
塞滿256kb在發送
◎ Solution:
patch掉原本送queue的function, 並且利用 atexit 去避免
worker被關閉時, 有些message還在 local queue裡面
◎ 結果: sent, receive , delete 量減少99%

11. SQS新政4 : 調整單機工人數目
◎ Message 多的開多一點 thread去處理
◎ 其中還可以利用 gevent 去避開一些 network bound的問題

12. Result
◎ Spot instance: 30~50 -> 2
◎ CPU utilization: 20% -> 80%
◎ Message Counts: 1 billion -> 10million
◎ 處理時間

13. Result
Cost Reduction around 70%

14. Bonus: SQS alternative

15. Fluentd (Before)

16. Fluentd (After)

17. Traditional Flow
AP server SQS Worker Mongodb
Big Query

18. Fluentd Flow
AP server
Fluentd
Forwarder
Fluentd
Station
Mongodb
Big Query

19. Fluentd Flow
AP server
Fluentd
Forwarder
Fluentd
Station
ELB
Fluentd
Station
Fluentd
Station
Fluentd
Station

20. Thanks!
Any questions?
You can find me at:
@kakashiliu
kakashi@gogolook.com