1
2
3
Mongodb, UGC project
- billions objects;
- terabytes of data MongoDB;
- high Disk I/O;
- big size Indexes
- balancing & ...
4
Yandex.Disk is a service for storing and sharing files that
also provides access to files from any device connected to
t...
5
~6,4 billion objects:
6
~6 Tb filesize:
7
~1.5 Tb IndexSize;
8
> 6 billion operations every day;
9
- over 50000 chunks
10
Notes:
- write global lock to DB;
- no transactions;
- single threaded chunk moving job;
- 20000 connections limit to m...
11
12
13
14
15
DBA at work:
16
Usual RS:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UPUP
UP
UP
UP delay
17
Clean hidden and Startup:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
Startup
UP
UP
UP
18
Recover Secondary:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
DOWN
DOWN
UP
UP
rsync
19
Chunk-defrag RS:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UPUP
UP
Defrag UP
Defrag UP delay
20
Recover Secondary:
Secondary Secondary
Secondary
hidden
Primary
ReplicaSet:
DOWN
DOWN
UP
Defrag UP
rsync
21
Usual RS:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UPUP
UP defrag
UP defrag
UP defrag
22
Disk I/O:
- ext4
- raid0 и raid10
- mount options
- dirty pages
- SSD
23
Disk I/O:
24
Monitoring:
- db.serverStatus()
- rs-indexes-consistency
- indexes-miss-ratio
- backgroundFlushing
- heap_usage_bytes
-...
25
mongostat --discover:
26
Collections qps:
27
Balancing:
- tags
- sh.getBalancerState()
- moveChunk
28
Mongos:
29
Mongos drop tcp cfg:
drop tcp
30
Mongos drop tcp shard:
drop tcp
31
Mongos fast fix:
drop tcp
-j REJECT --reject-with tcp-reset
32
Keepalived config:
33
Mongos RS down — ALL DOWN:
34
Summary:
1) SERVER-7008
2) SERVER-9041
3) SERVER-1240
35
Thank you! Questions?
Andrey Godin
http://yandex.ru
@airesp_ya
agodin@yandex-team.ru
Upcoming SlideShare
Loading in …5
×

Использование MongoDB как основной метабазы в UGC-сервисах

1,345 views

Published on

Сервис, на котором хранятся данные миллиардов пользователей, обязан быть масштабируем, отказоустойчив и производителен. Одним из важных его компонентов является метабаза — сердце сервиса. За время использования MongoDB в сервисах с нагрузкой более 50 тысяч запросов в секунду мы научились многому.

В докладе пойдёт речь о том, как:

хранить десятки миллиардов объектов и терабайты данных в MongoDB;
настраивать дисковую подсистему с БД и следить за ней;
управлять индексами, соединениями и балансировкой данных между несколькими ReplicaSet’ами;
жить со специфическими проблемами в эксплуатации MongoDB, которые мы заметили.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,345
On SlideShare
0
From Embeds
0
Number of Embeds
768
Actions
Shares
0
Downloads
15
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Использование MongoDB как основной метабазы в UGC-сервисах

  1. 1. 1
  2. 2. 2
  3. 3. 3 Mongodb, UGC project - billions objects; - terabytes of data MongoDB; - high Disk I/O; - big size Indexes - balancing & connections to DB; - troubleshooting mongodb;
  4. 4. 4 Yandex.Disk is a service for storing and sharing files that also provides access to files from any device connected to the internet. > 19M Users > 10M file uploads per day SDK – http://api.yandex.ru/disk/ The Yandex.Disk SDK is intended for applications that work with the files of Yandex.Disk users or store their own files and settings on Yandex.Disk.
  5. 5. 5 ~6,4 billion objects:
  6. 6. 6 ~6 Tb filesize:
  7. 7. 7 ~1.5 Tb IndexSize;
  8. 8. 8 > 6 billion operations every day;
  9. 9. 9 - over 50000 chunks
  10. 10. 10 Notes: - write global lock to DB; - no transactions; - single threaded chunk moving job; - 20000 connections limit to mongod/mongos; - Mongos unable to work in RO mode; - no connection-pool
  11. 11. 11
  12. 12. 12
  13. 13. 13
  14. 14. 14
  15. 15. 15 DBA at work:
  16. 16. 16 Usual RS: Secondary Secondary Secondary hidden Primary ReplicaSet: UPUP UP UP UP delay
  17. 17. 17 Clean hidden and Startup: Secondary Secondary Secondary hidden Primary ReplicaSet: Startup UP UP UP
  18. 18. 18 Recover Secondary: Secondary Secondary Secondary hidden Primary ReplicaSet: DOWN DOWN UP UP rsync
  19. 19. 19 Chunk-defrag RS: Secondary Secondary Secondary hidden Primary ReplicaSet: UPUP UP Defrag UP Defrag UP delay
  20. 20. 20 Recover Secondary: Secondary Secondary Secondary hidden Primary ReplicaSet: DOWN DOWN UP Defrag UP rsync
  21. 21. 21 Usual RS: Secondary Secondary Secondary hidden Primary ReplicaSet: UPUP UP defrag UP defrag UP defrag
  22. 22. 22 Disk I/O: - ext4 - raid0 и raid10 - mount options - dirty pages - SSD
  23. 23. 23 Disk I/O:
  24. 24. 24 Monitoring: - db.serverStatus() - rs-indexes-consistency - indexes-miss-ratio - backgroundFlushing - heap_usage_bytes - connections - replication
  25. 25. 25 mongostat --discover:
  26. 26. 26 Collections qps:
  27. 27. 27 Balancing: - tags - sh.getBalancerState() - moveChunk
  28. 28. 28 Mongos:
  29. 29. 29 Mongos drop tcp cfg: drop tcp
  30. 30. 30 Mongos drop tcp shard: drop tcp
  31. 31. 31 Mongos fast fix: drop tcp -j REJECT --reject-with tcp-reset
  32. 32. 32 Keepalived config:
  33. 33. 33 Mongos RS down — ALL DOWN:
  34. 34. 34 Summary: 1) SERVER-7008 2) SERVER-9041 3) SERVER-1240
  35. 35. 35 Thank you! Questions? Andrey Godin http://yandex.ru @airesp_ya agodin@yandex-team.ru

×