This document discusses the operations and troubleshooting of a large MongoDB deployment containing billions of objects and terabytes of data. It covers monitoring disk usage and performance, recovering replicas, rebalancing shards, mongos configuration, and other administrative tasks such as checking replica set status and starting up hidden nodes. The deployment requires ongoing maintenance to optimize disk usage, monitor system resources, and address any issues that arise with replication, sharding or client connections.

1. 1

2. 2

3. 3
Mongodb,
UGC
project
-­‐
billions
objects;
-­‐
terabytes
of
data
MongoDB;
-­‐
high
Disk
I/O;
-­‐
big
size
Indexes
-­‐
balancing
&
connec?ons
to
DB;
-­‐
troubleshoo?ng
mongodb;

4. 4
диск
~6,4
billion
objects:

5. 5
диск
~6
Tb
filesize:

6. 6
диск
~1.5
Tb
IndexSize;

7. 7
>
6
billion
opera?ons
every
day;
диск

8. 8
-­‐
over
50000
chunks
диск

9. 9
диск
Notes:
-­‐
write
global
lock
to
DB;
-­‐
no
transac?ons;
-­‐
single
threaded
chunk
moving
job;
-­‐
20000
connec?ons
limit
to
mongod/mongos;
-­‐
Mongos
unable
to
work
in
RO
mode;
-­‐
no
connec?on-­‐pool

10. 10

11. 11

12. 12

13. 13

14. 14
DBA
at
work:

15. 15
Usual
RS:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UP
UP
UP
UP
delay

16. 16
Clean
hidden
and
Startup:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
Startup
UP
UP
UP

17. 17
Recover
Secondary:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
DOWN
DOWN
UP
UP
rsync

18. 18
Chunk-­‐defrag
RS:
Defrag
UP
delay
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UP
UP
Defrag
UP

19. 19
Recover
Secondary:
Secondary
Secondary
hidden
Secondary
Primary
ReplicaSet:
DOWN
DOWN
UP
Defrag
UP
rsync

20. 20
Usual
RS:
Secondary
Secondary
Secondary
hidden
Primary
ReplicaSet:
UP
UP
defrag
UP
defrag
UP
defrag

21. 21
Disk
I/O:
-­‐
ext4
-­‐
raid0
и
raid10
-­‐
mount
op?ons
-­‐
dirty
pages
-­‐
SSD

22. 22
Monitoring:
-­‐
db.serverStatus()
-­‐
rs-­‐indexes-­‐consistency
-­‐
indexes-­‐miss-­‐ra?o
-­‐
backgroundFlushing
-­‐
heap_usage_bytes
-­‐
connec?ons
-­‐
replica?on
-­‐
qr|qw

23. 23
mongostat
-­‐-­‐discover:

24. 24
Balancing:
-­‐
sh.getBalancerState()
-­‐
moveChunk

25. 25
Mongos:

26. 26
Mongos
drop
tcp
cfg:
drop
tcp

27. 27
Mongos
drop
tcp
shard:
drop
tcp

28. 28
Mongos
fast
fix:
drop
tcp
-­‐j
REJECT
-­‐-­‐reject-­‐with
tcp-­‐reset

29. 29
Keepalived
config:

30. 30
Mongos
RS
down
—
ALL
DOWN:

31. 31
Thank
you!
Ques?ons?
Andrey
Godin
h=p://yandex.ru
@airesp_ya
agodin@yandex-­‐team.ru