This document summarizes the author's experience optimizing Kamailio and Redis for horizontal scaling. Some of the key points covered include: - Using Redis as a persistent data store for Kamailio provides better scaling than MySQL - Tuning disk I/O settings in Debian was needed to optimize Redis performance - Tuning network I/O such as rx queues was also important for fully utilizing CPU resources - Defining custom keys for Redis is necessary to map data in a way that supports lookups by AoR - Redis persistence settings like AOF and RDB snapshots can be tuned to balance data integrity with infrastructure costs - Future work includes shipping default Redis key definitions and leveraging additional Redis features.

1. Super-Charging Kamailio
and learning some new tricks
Andreas Granig • 15.05.2018
<agranig@sipwise.com>

2. Goal: Full Horizontal Scale
kamailio
mysql
kamailio
?
kamailio kamailio
??
load-balancer
a little bit of coding

3. Reality
kamailio
mysql
kamailiokamailio kamailio
redis
done!
huh? oh!
wtf?
… to be continued

4. Telling the story
backwards
● When you’re facing systemd
● When Kamailio + Redis
outperforms your system
defaults
● When all finally comes together

5. Everything looks great, except when it doesn’t

6. Know your Disk I/O!
● Introducing systemd, which
comes with journald…
● which uses memory-mapped file
access…
● and Debian has certain default
virtual memory settings!

7. Tuning Disk I/O
# sysctl -a | grep dirty | grep centi
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500
I see a pattern forming...
vm.dirty_expire_centisecs = 200
vm.dirty_writeback_centisecs = 100
… works for us
… and we disabled journald persistence

8. Kamailio + Redis +
rtpengine =
fast
● Especially when running on a
brand new Xeon Gold 16 Core
CPU
● But what if you only ever can
utilize 60% of the overall CPU?

9. Tuning Network I/O
Check number of rx queues
# ethtool -l eth2 | grep Combined
Combined: 8
Check your rx queue hashing algorithm
# ethtool -n eth2 rx-flow-hash udp4
UDP over IPV4 flows use these fields for computing Hash flow key:
IP SA
IP DA
L4 bytes 0 & 1 [TCP/UDP src port]
L4 bytes 2 & 3 [TCP/UDP dst port]
The default for our driver is 4...
Know your network topology, or
else you’ll starve queues
https://blog.cloudflare.com/how-to-receive-a-million-packets/

10. Now to the meat of the talk

11. Selecting a
backend
for a horizontally scalable, persistent
data store for mostly volatile data, e.g.
● Raw acc records (r/o)
● Registrations (r/w)
● Dialog info (r/w)
● Presence info (r/w)

12. Requirements
● Well known and understood within Sipwise
● Shardable on protocol level
● Simple highly available architecture
● Survivable without quorum (geographical site
survivability)
… which for us is Redis

13. A Redis Primer
● KV-Store on Steroids
● Simple Wire Protocol
● Replication Support
● Persistence

14. The problem with the “Key” in “Key-Value Store”
● Key must be unique
○ AoR (user@domain) won’t work (parallel registrations)
○ Let’s use RUID then
● You can only select by Key (unless you table-scan)
○ But… usrloc selects by AoR?
Let’s define entry keys for actual rows
… and map keys to simulate a schema

15. Defining your Keys
● It’s not plug & play yet!
● modparam(“db_redis”, “keys”, …)

16. Anatomy of a Key Definition
location=entry:ruid&usrdom:user,domain&timer:partition,keepalive
location:entry::my-ruid-1
username: testuser1
domain: example.org
contact: sip:me@1.2.3.4
location:usrdom::testuser1:example.org
location:entry::my-ruid-1
location:entry::my-ruid-2
location:entry::my-ruid-2
username: testuser1
domain: example.org
contact: sip:also-me@2.3.4.5
location:timer::23:1
location:entry::my-ruid-1
location:entry::my-ruid-2

17. But… how would I
know the keys?
● Problem is, I don’t know either
● If you get them wrong, it will still
work (resorting to full scans),
but sloooow
● Except if you get them really
wrong, it won’t work
● Goal is to ship kamailio with
pre-defined keys in the future

18. How to use Redis persistence?
● AOF with per-second vs immediate fsync
○ Similar to innodb_flush_log_at_trx_commit
● RDB Snapshots
What does a second of data loss cost, vs. operating 3x the infrastucture?
● If in doubt, run multiple Redis instances for different purposes
https://redis.io/topics/persistence

19. TODOs
● Ship default key definitions in a
file, as we do with schema
● Leverage Redis Key Expire
Mechanism
● Add Sentinel Support
● Use PubSub for updating cache,
anyone?

20. Questions?
<agranig@sipwise.com>