Structure of presentation
●Short overview of project and infra
5 mins
●Issue describe (spoiler – production not works)
●5 mins
●Investigate & triangle
15 mins
●Fix

E-goverment platform

RabbitMQ infrastructure
prod-rabbit-dr-1
prod-rabbit-dr-3 prod-rabbit-dr-4
prod-rabbit-dr-1 prod-rabbit-dr-2
prod-rabbit-new-lb

Problem definition phase

Problem description
●Alarm on average page processing time fired
●Alarm on number of gateway timeouts fired

Problem description
●Alarm on average page processing time fired
●Alarm on number of gateway timeouts fired
●We check manually – web site works very slow
●And situation becomes worser, after ~15 mins of
incedent detection site almost stop works

Problem triangulation

Anamnes checklist
●Do we change anything in system recently ?
●Do we have deployments recently?
●Is RMQ works fine? Do we have overloads ?
●Do we have anything abnormal in logs or monitoring
system ?
●Check system elements one-by-one if required

●Huge increase on prod-rabbit-lb, up to 300MBit
●Number of messages in queues normal
●But in/out rate in RMQ really slow
●Nothing incorrect in log
●So first result – something keeps RMQ busy

EUREKA

Problem solve

●Solve is trivial – just increase memory limit for
problematic server in short perspective
●And review service to generate smaller RMQ
messages in longer perspective

Lesson learned

●Improve monitoring:
– Check for often restarting services
– Check for docker OOM killer events
– Check for system-wide OOM killer