This document proposes an alternative to the XenStore called NoXS. The goals of NoXS are to provide a simple and specialized mechanism for storing domain and device information that is efficient, fast and has few lines of code. Rather than using a centralized daemon, NoXS would store this information in memory pages directly accessible to domains. The design of NoXS includes Device Information Pages and Device Control Pages to store data about devices and allow control of guest VMs. Initial implementations have been done for Linux guests and the Chaos toolstack. Further work remains to complete features like crash recovery and device hotplugging. Performance tests show NoXS can significantly reduce domain boot times compared to using the standard XenStore.

1. NoXS: Death to the XenStore
Filipe Manco NEC Europe Ltd.
Costin Lupu University Politehnica of Bucharest
Florian Schmidt NEC Europe Ltd.

2. Oh… but, why?
2

3. Why do we care?
3

4. Why do we care?
4
● Specialized applications on Unikernels
○ ClickOS: Middleboxes (ClickOS)
○ MiniCache: Edge content caches
● Micro-services with VMs
● Per-user virtualized services
● Edge devices
○ Constrained ARM boards

5. Why do we care?
5
● Specialized applications on Unikernels
○ ClickOS: Middleboxes (ClickOS)
○ MiniCache: Edge content caches
● Micro-services with VMs
● Per-user virtualized services
● Edge devices
○ Constrained ARM boards
● Performance
○ Low boot times
○ Good scalability
○ Low overhead

6. Why do we care?
6
● Specialized applications on Unikernels
○ ClickOS: Middleboxes (ClickOS)
○ MiniCache: Edge content caches
● Micro-services with VMs
● Per-user virtualized services
● Edge devices
○ Constrained ARM boards
● Performance
○ Low boot times
○ Good scalability
○ Low overhead
● Specific functionality
○ PV or PVH (no QEMU)
○ Usually only vif and block devices
○ No “legacy”

7. What’s wrong with the XenStore?
7

8. What’s wrong with the XenStore?
● Performance
○ Biggest contributor for domain creation time after few hundreds
8

9. Domain creation time breakdown for booting 1000 MiniOS guests (8MB RAM, 1 vCPU and 1 VIF) on Xen 4.8 9

10. What’s wrong with the XenStore?
● Performance
○ Biggest contributor for domain creation time after few hundreds
10

11. What’s wrong with the XenStore?
● Performance
○ Biggest contributor for domain creation time after few hundreds
● Protocol
○ Too generic: requires dozens of messages to create a domain
○ Limits directory size, i.e. number of VMs
○ Searches are expensive
11

12. What’s wrong with the XenStore?
● Performance
○ Biggest contributor for domain creation time after few hundreds
● Protocol
○ Too generic: requires dozens of messages to create a domain
○ Limits directory size, i.e. number of VMs
○ Searches are expensive
● Centralized service
○ Central point-of-failure
○ Harder to make Dom0 rebootable
12

13. What’s wrong with the XenStore?
● Performance
○ Biggest contributor for domain creation time after few hundreds
● Protocol
○ Too generic: requires dozens of messages to create a domain
○ Limits directory size, i.e. number of VMs
○ Searches are expensive
● Centralized service
○ Central point-of-failure
○ Harder to make Dom0 rebootable
● Certification
○ One more piece of software to certificate 13

14. What can we do?
14

15. Create a new
protocol…
15

16. Improve the
implementation…
16

17. LiXS
LightWeight XenStore
https://github.com/cnplab/lixs
17

18. 18
Add new
functionality...

19. Why not kill it?
J.J. at the English language
Wikipedia / CC BY-SA 3.0
19

20. NoXS: No XenStore
20

21. NoXS Goals
21

22. XenStore functionality
22

23. XenStore functionality
● Keep basic domain information
23

24. XenStore functionality
● Keep basic domain information
24

25. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
25

26. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
26

27. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
27

28. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
28

29. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
29

30. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
30

31. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
31

32. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
32

33. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
● Power control, i.e., power off and suspend
33

34. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
● Power control, i.e., power off and suspend
34

35. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
● Power control, i.e., power off and suspend
● Guest specific functionality
○ ClickOS uses XenStore to install configurations
○ Mirage’s Jitsu uses XenStore to advertise service endpoints
35

36. XenStore functionality
● Keep basic domain information
● CPU hot-plugging
● Dynamic setup of balloon target
● Keep toolstack related information
● Virtual device setup a.k.a. XenBus
● Power control, i.e., power off and suspend
● Guest specific functionality
○ ClickOS uses XenStore to install configurations
○ Mirage’s Jitsu uses XenStore to advertise service endpoints
36

37. NoXS Goals
37

38. ● Features
○ XenBus
○ Guest control
NoXS Goals
38

39. ● Features
○ XenBus
○ Guest control
● Simple and specialized mechanism
○ Efficient and fast
○ Few lines-of-code: easy to certify
NoXS Goals
39

40. ● Features
○ XenBus
○ Guest control
● Simple and specialized mechanism
○ Efficient and fast
○ Few lines-of-code: easy to certify
● Not based on centralized daemon
NoXS Goals
40

41. ● Features
○ XenBus
○ Guest control
● Simple and specialized mechanism
○ Efficient and fast
○ Few lines-of-code: easy to certify
● Not based on centralized daemon
● Run side-by-side with the xenstore
NoXS Goals
41

42. NoXS Design
42

43. 43NoXS basic architecture

44. 44NoXS basic architecture

45. 45NoXS basic architecture

46. Device Info Page (DevIP)
typedef struct noxs_devip {
uint32_t version;
uint32_t dev_count;
noxs_devip_entry_t devs[NOXS_DEV_COUNT_MAX];
} noxs_devip_t;
46
typedef struct noxs_devip_entry {
noxs_dev_type_t type;
noxs_dev_id_t id;
domid_t be_id;
grant_ref_t grant;
evtchn_port_t evtchn;
} noxs_devip_entry_t;

47. 47NoXS basic architecture

48. Device Control Page (DevCP) for Network
typedef struct noxs_devcp_vif {
noxs_devcp_t hdr;
int vifid;
struct vif_be_features be_feat;
int multi_queue_max_queues;
int multi_queue_num_queues;
grant_ref_t tx_ring_ref;
grant_ref_t rx_ring_ref;
evtchn_port_t event_channel_tx;
evtchn_port_t event_channel_rx;
unsigned int request_rx_copy;
struct vif_fe_features fe_feat;
grant_ref_t ctrl_ring_ref;
evtchn_port_t event_channel_ctrl;
uint8_t mac[ETH_LEN];
uint32_t ip;
char bridge[IF_LEN];
} noxs_devcp_vif_t;
48
typedef struct noxs_devcp_hdr {
int devid;
int be_state;
int fe_state;
noxs_watch_state_t fe_watch_state;
noxs_watch_state_t be_watch_state;
} noxs_devcp_hdr_t;

49. 49Domain creation with NoXS

50. Guest initialization with NoXS 50

51. 51NoXS basic architecture

52. 52NoXS basic architecture

53. Guest Control
53

54. ● Virtual device
○ Based on the split driver model
● Features
○ Power control
○ CPU hot-plugging
○ Ballooning
● Can be extended to add other features
gCtl: Guest Control Device
54

55. Suspend with NoXS’ gCtl 55

56. What’s done already
56

57. Implemented features
57
● Basic mechanism
● Linux support
○ Network
○ Block
○ gCtl
● Toolstack support
○ Chaos toolstack
● gCtl
○ Power control

58. Chaos
Virtualization Toolstack
https://github.com/cnplab/chaos
58

59. Xen
[N-0978] xen $ git diff RELEASE-4.8.1 --stat
xen/arch/x86/hypercall.c | 2 +
xen/common/Makefile | 2 +
xen/common/devctl.c | 100 ++++++++++++++++++++++++++++
xen/common/domain.c | 12 +++-
xen/common/noxs.c | 94 ++++++++++++++++++++++++++
xen/include/public/devctl.h | 63 ++++++++++++++++++
xen/include/public/io/noxs.h | 250 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
xen/include/public/xen.h | 1 +
xen/include/xen/hypercall.h | 4 ++
xen/include/xen/noxs.h | 18 +++++
xen/include/xen/sched.h | 3 +
11 files changed, 548 insertions(+), 1 deletion(-)
59

60. Linux
[N-0978] linux $ git diff v4.8.17 --shortstat
70 files changed, 9272 insertions(+), 1576 deletions(-)
[N-0978] linux $ git diff v4.8.17 --dirstat
12.6% drivers/block/xen-blkback/
9.3% drivers/block/
11.9% drivers/net/xen-netback/
7.1% drivers/net/
47.4% drivers/xen/xenbus/
9.4% include/xen/
60

61. What’s missing
61

62. ● Basic mechanism
○ Crash-recovery
○ Device hot-plugging
● Linux support
○ Stub-domain support
○ Serial console drivers
● Toolstack support
○ xl/libxl
● gCtl
○ CPU hot-plugging
○ Ballooning
Missing features
62

63. Performance
63

64. 64Domain boot times comparison: Standard vs NoXS with Chaos

65. 65
Questions