The document details the architecture and performance of WhatsApp's server infrastructure, which supports over 600 million FreeBSD users. It discusses the hardware and software configurations used to manage high concurrency and messaging rates, as well as the tuning and optimizations implemented for performance. Key topics include system metrics, software versions, and troubleshooting strategies to ensure stability and efficiency.

1. Half a Billion 600M+
Unsuspecting FreeBSD Users
Rick Reed
WhatsApp

2. topics
history
statistics
architecture
wins
challenges

3. rr
Joined WhatsApp server team in 2011
Performance, multimedia, security, ops
Previously: Yahoo!, SGI, Metaphor, Aion
First FreeBSD: 2.2.8 at Yahoo!

4. Why FreeBSD?
Previous Yahoo! experience
High-quality, high-performance TCP stack
Scale
Stability

6. Numbers
Over 600M monthly users
140M concurrent connections
440k connections per second
1.1M messages sent per second
1B pics sent per day

7. Numbers
~800 servers (incl. 200 chat + 350 mms)

10. Performance
SMP scaling is great
kernel relatively free of contention
Erlang (actor model)
BEAM (Erlang VM) very SMP-aware

15. System metrics

16. Hardware (typical)
SoftLayer bare-metal
1U/2U/4U Supermicro
dual E5-2690v2 (40 threads)
(public, private) x 2 lagg x 1g igb (some 10g ix)
64-768GB RAM
1-12 800GB SSD (12x4TB HDD for video)
JBOD w/ mobo, Adaptec, and LSI controllers

17. Software Platform
FreeBSD 9.1 - 9.3
ufs
gmirror for HDD video servers
starting to evaluate 10.1

18. Software Platform
Erlang R16B01 (w/ our patchset)
rtprio
patches address specific contention issues
schedulers bound to cpus
allocators tuned for superpage promotion
yaws for web-serving (but disable sendfile)

19. Software Platform
stud for SSL termination (proxy via loopback)

20. Data storage
Erlang mnesia
account data, group membership, etc.
in-memory + dump/log persistence
Flat-file on hashed directory tree
offline mailboxes, media
fs layout tuned per workload

21. System metrics
6 x 800GB SSD per offline storage server

22. Versions
FreeBSD 7: 2009-2010
good
bugs: some

23. Versions
FreeBSD 8: 2011-2012
better
bugs: zero-copy sockets, igb, tsc tc
tuning got us to ~2.8M conns/server

24. Versions
FreeBSD 9: 2013-
yes!
bugs: very, very rare panic
enhancements:
>2G dirhash size
max tcp rexmit interval

25. Tuning
kernconf
remove unused modules
enable debugging/tracing

26. Tuning
loader vars
igb (numq=8, rx_process_limit=-1, r/txd=4k)
kern.nbuf=128k
kern.max{files,filesperproc,proc}
net.inet.tcp.tcbhashsize=512k
net.inet.tcp.hostcache.bucketlimit=1

27. Tuning
Network
capacity (mbufs, sockets, etc.)

28. Tuning
vfs
ufs.dirhash_maxmem (patched for >2G)
dirty buffer limits & sync delays

29. Issues
ufs panics (fs inconsistencies after crash/pfail)
bursty syncer behavior
filedesc table contention
pcb table contention (1.5m igb + 3m loopback)

30. Now vs. then
machdep.panic_on_nmi
VM pressure

31. Wishlist
Multipath TCP
Faster large RAM boot

32. Questions?