This document discusses Kea, an open source DHCP server from ISC. It provides high-level summaries of Kea's design, features, and deployments. Kea was designed for high performance, modularity, and scalability. Notable deployments include a large ISP serving over 2 million users and Facebook using Kea in their datacenters for bare metal provisioning. Kea provides advantages like failure resilience, rapid configuration updates, and metrics collection.

1. © 2015 ISC
Kea Deployments
High Performance DHCP
PLNOG15, Kraków, 2015-09-29
Tomek Mrugalski
<tomasz(at)isc(dot)org>

2. © 2015 ISC
Who is Tomek?
M.Sc., Ph.D from Gdansk University of Technology
Primary author of Dibbler
Portable DHCPv6 implementation (srv, cli, relay)
Supports Win 2k-Win8, Linux, BSD, ~Solaris
7 years at Intel (Network Quality Labs, chipsets)
4 years at ISC
Lead Developer of Kea (former BIND10 DHCP)
Occasional contributor to ISC-DHCP
Active IETF participant since 2009
DHC Working Group co-chair
Author of 5 RFCs, 15+ drafts
2

3. © 2015 ISC
Why re-write DHCP?
• Existing code is old enough to buy alcohol
• Hardware changed (many cores)
• Networks changed
• DHCP landscape changed
• New software development techniques
• Lacking performance
• Monolithic
• Documentation is lacking
• Complex code, difficult to extend
3

4. © 2015 ISC
Kea
 Open Source
 BIND10 component,
now standalone
 C++
 IPv6 first
 High Performance
 On-line configuration
 Switchable backends
Multi-core support*
• Flexible Logging
• Statistics
• Hooks
• Modular
• Portable (Linux,
Free/Net/OpenBSD,
Mac OS)
• Well documented
User's Guide,
Developer's guide,
Contributors Guide
4

5. © 2015 ISC
Design Assumptions
 Long-lived project, design for 2035
 Modular
 Scalable
 Networks with millions of devices
– Many thousands of leases/sec
– IPv4 and IPv6 on equal footing
 Maintainable
 Unit-tests
 Portable
– Linux, BSD, Mac OS (Windows, Solaris possible)
5

6. © 2015 ISC
Kea :: Overview
DHCPv4
Server
DHCPv4
Server
DHCPv6
Server
DHCPv6
Server
DNS
Updates
DNS
Updates
perfdhcp
•general purpose DHCP library
•IPv4/IPv6 packet parsing/assembly
•IPv4/IPv6 options parsing/assembly
•interface detection (Linux, BSD/Mac OS)
•socket management
•general purpose DHCP library
•IPv4/IPv6 packet parsing/assembly
•IPv4/IPv6 options parsing/assembly
•interface detection (Linux, BSD/Mac OS)
•socket management
libdhcp++
JSON
config
JSON
config
6

7. © 2015 ISC
Kea :: current Status
Basic
- v4, v6 parity
- Address assignment, renewal, release, expiry
- On-line configuration
- Switchable backends: MySQL, PostgreSQL, memfile
- Options (std)
Advanced
- Hooks (external libraries, a'la apache modules)
- Options (custom, vendor, nested, namespaces)
- Host reservation (addresses, prefixes)
- MAC in DHCPv6
- Statistics (JSON)
- Control Channel (JSON over Unix socket)
- Flexible (per component) logging
7

8. © 2015 ISC
Kea :: Roadmap
0.92
Aug 2015
Statistics
Flexible
Logging
Control
Channel
1.1
Spring 2016
Client
classification
Remote
mgmt
1.0
Dec 2015
Lease
expiration
Decline
PXE boot
Public
benchmarks
1.2
Late 2016
?Radius auth
?Hooks config
?Lease query
?Server-
initiated
reconfig
Contributors!
DHCPv4 over v6, Secure DHCP
fixed flexiblefixed flexible
8

9. © 2015 ISC
Switchable Lease Database
 MySQL, PostgreSQL
– Standard tools to read/update
 Memfile
– Custom developed in C++
– Flat file storage (CSV)
– Memory-only or memory+disk-write
– Very high performance in mem-only mode
– High performance in memory+disk mode
 Abstract LeaseMgr
– C++ class, add your favourite storage
DHCPv4/6
server
MySQL
PgSQL
Memfile
C++
Abstract
LeaseMgr
Scalable2Mdevices
9

10. © 2015 ISC
DHCPpacketprocessingflow
Hooks :: Extending the Server
Kea Process Custom Library 1 Custom Library 2
F1()
F2()
G1()
G2()
Receive packet
Select subnet
Select lease
Send packet
10

11. © 2015 ISC
Hooks API
• Many hook points defined (more being added)
• buffer4_receive, buffer6_receive
• pkt4_receive, pkt6_receive
• subnet4_select, subnet6_select
• lease4_select, lease6_select
• lease{4,6}_{renew, release, decline}
• pkt4_send, pkt6_send
• buffer4_send, buffer6_send
• Replaces conditional configuration processing in ISC-DHCP
• Very well documented (developer’s guide, examples)
http://git.kea.isc.org/~tester/kea/doxygen/
=> Hooks Framework
11

12. © 2015 ISC
Performance Results
Preliminary data. Your mileage may vary.
This is single threaded, single process for now.
host: iMac 27" Mid 2011, 3.4 GHz i7, 16 GB 1333 MHz DDR3, OS X 10.10.1, VMWare Fusion Pro 7.1.0
Guest: Fedora 20 64 bits, 2 GB, 2 cores, gcc/g++ 4.8.3, glibc 2.16, MariaDB 5.5.41 (MySQL fork), PostgreSQL 9.3.6.
12

13. © 2015 ISC
Deployments
Large ISP
13

14. © 2015 ISC
Large ISP Deployment
Large, country wide ISP in western Europe needs
DHCP solution to support his hotspots with 2M
users
Timeline:
• Approached ISC in Dec. 2014
• Requested several features
• Roadmap updated, beta drop in Feb. 2015
• On-site developer training
• Signed support contract
• Kea 0.9.1 with all features in Mar 2015
• Field trials in May 2015
• Country wide deployment – Aug 2015 14

15. © 2015 ISC
Wireless Access
Gateway 1
memfile
open secure
Wireless Access
Gateway 2
open secure
Kea1
Pool Open WAG-1
Pool Open WAG-2
Kea2
Pool Secure WAG-1
Pool Secure WAG-2
memfile
DRBD
RedHat Cluster Suite
Kea1
Pool Open WAG-1
Pool Open WAG-2
Kea2
Pool Secure WAG-1
Pool Secure WAG-2
IP1 IP2 IP1 IP2
DRBD
RedHat Cluster Suite
memfile memfile
Node A
Active
Node B
Stand-by
Large ISP Architecture
15

16. © 2015 ISC
Large ISP :: Conclusions
• Fast response (new feature in ~2 months)
• Proven to work for 2M users, will work for more
• High performance (7500 new leases/sec)
• Support contract
• 1 request so far, solved promptly
Status:
Serving country-wide WiFi network
in western Europe, ~2 million users
16

17. © 2015 ISC
Kea @ Facebook Datacenter

18. © 2015 ISC
Datacenter Deployment
Overview:
Large DC with inventory database, multiple DCs
Redundancy problems, wants to use inventory DB
for bare metal provisioning, PXE boot.
Had complex git/rsync config generation with
periodic restarts, took hours to propagate changes
18

19. © 2015 ISC
Datacenter :: Overview
server
server
server
server
…
Rack switch
19

20. © 2015 ISC
server
server
server
…rack
switch
server
server
server
…
rack
switch
DHCP relay
DHCP relay
DHCP
Server 1
active
stand by
DHCP
Server 2
HW load balancers cost $$$
Both servers down => 
LB down => 
Problem 1 :: Failure domain
20

21. © 2015 ISC
Problem 2 :: Bootstrapping
DHCP
Server
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
serve
r
…
switc
h
serve
r
serve
r
remote cluster
21

22. © 2015 ISC
Problem 3 :: Configuration
Pipeline
DHCP server
Inventory
system
Periodic
job
Git repo
delivery
system
Chef
Infrastructu
re
Nanny
script
/etc/dhcpd/
…
/etc/init.d/
restart
• People changing hw => HW addr changing
• Inventory update took up to 4h to propagate
• Lack of instrumentation
• Latencies
• Errors? Exceptions? 22

23. © 2015 ISC
Kea Deployment
Kea (v4, v6)
• buffer4_receive
• pkt4_receive
• subnet4_select
• lease4_select
• pkt4_send
• buffer4_send
Hooklibrary
• Facebook specific hook lib extracts addresses from DB
• Effectively “stateless” DHCP
• DB changes are applicable immediately (no config sync
delay, no restart, no reconfiguration downtimes) 23

24. © 2015 ISC
DHCP Anycasting
Announcing single host route over BGP, the same on all servers
exaBGP
DHCP
server
exaBGP
DHCP
server
Region 1 Region 2
exaBGP
DHCP
server
exaBGP
DHCP
server
infrastructure routers
BGP
BGP
BGP
BGP
24

25. © 2015 ISC
Facebook :: Timeline
Timeline
•Month 1: proof of concept developed, test deployment in
1 cluster
•Month 2-3: productize, integrate with provisioning
infrastructure
•Month 4: mass deployment, serving all clients with Kea!
25

26. © 2015 ISC
Facebook :: Benefits
Benefits
•Anycast: packets sent to the nearest server
• => No more HW load balancing
• => Any DHCP server instance can handle any
request (in the same or other cluster, it doesn’t
matter)
•HW replacement is easier, changes picked up instantly
•JSON configuration, trivial to manage/regenerate
•Metrics!
26
https://www.isc.org/blogs/how-facebook-is-using-kea-
in-the-datacenter/
https://www.usenix.org/conference/srecon15europe/p
rogram/presentation/failla

27. © 2015 ISC
Get Involved
Open source
Community asset
Core repository is
public
Bug datatase public
Mailing lists, jabber
public
Test, report bugs
Contribute
Support contract
Submit patches (read
Contributor’s Guide
first)
Sponsorship
Development
contracts
Review design
documents
Looking for contributions:
• Additional back ends
• Apps via Hooks API
• Basic GUI
•…
github.com/isc-projects/kea
http://kea.isc.org
27

28. © 2015 ISC
Questions?
http://kea.isc.org, https://www.isc.org/kea/
github.com/isc-projects/kea
28

29. © 2015 ISC
Thank you