In this Joyent Technical Discussion, Mike Gerdts and Pedro Palazón Candel describe and demonstrate the Flexible Disk Space feature that was recently added to Triton and SmartOS. With Flexible Disk Space, bhyve instances can have a variable number of disks, can add/remove/resize disks, and use snapshots.

1. 1© 2019 Joyent, Inc.
Joyent Technical
Discussion
Flexible Disk Space
Mike Gerdts
Pedro Palazón
Candel
August 1, 2019

2. © 2019 Joyent, Inc. 2
Agenda
● Problem statement
● Solution overview
● Pray to Demo Gods
● SmartOS changes
● Triton changes
CC 2.0 https://www.flickr.com/photos/wwarby/19447531358/

3. Problem Statement
With bhyve HVM instances...
● Some people want one large disk
● Some people want several disks
● No one can agree on the size of any of
them
● Operators hate snowflakes
● Oh, and we want snapshots too!
3© 2019 Joyent, Inc.

4. Solution Overview
With bhyve HVM instances...
● Allow space to be reserved for future use
● Allow variable number of disks
● Allow snapshots
4© 2019 Joyent, Inc.

5. Considerations
● Predictable often more important than most efficient
● Virtual disks are a different from file systems
○ Grow generally safe, shrink is not
○ Free space management complicates snapshots
● Snapshots can be abused to consume all space
● RAIDZ and small blocks can be surprisingly inefficient
○ https://www.youtube.com/watch?v=sTvVIF5v2dw
5© 2019 Joyent, Inc.

6. SmartOS Changes
For bhyve instances...
● Persistent pci_slot
● New property: flexible_disk_size
● Add, remove, and resize disks with vmadm update
● Snapshots and Rollback
● ZFS fixes
6© 2019 Joyent, Inc.

7. Important ZFS Properties
● Reserve space
○ refreservation
■ for a specific file system or volume
■ auto is a special value
○ reservation - for a dataset and its children
● Limit disk space utilization
○ volsize - logical size of a ZFS volume
○ refquota - for a specific file system (or volume)
○ quota - for a file system and its children
7© 2019 Joyent, Inc.

8. © 2019 Joyent, Inc.
flexible_disk_size, aka...
Interface Attribute Units Notes
PAPI quota MiB When pkg.flexible_disk set to true
CloudAPI pkg.disk MiB triton pkg get $pkg | json disk
vmadm flexible_disk_size MiB Set when pkg.flexible_disk is set.
vmadm quota GiB Not related. This is refquota on the
zonepath dataset (zpds), which limits size of
logs, metadata, etc.
ZFS quota on zpds Any Related, but larger: flexible_disk_size +
metadata size + refquota on zonepath
dataset.
ZFS refquota on zpds Any Not related. See vmadm quota.
8

9. Triton Changes
PAPI
● Add flexible_disk (Bool) and disks attribute support to
Packages (TRITON-964)
● disks is a collection of JSON objects, either empty or
containing the property size which can take as value a numeric
size in MiB or the string "remaining"
9© 2019 Joyent, Inc.

10. Triton Changes
VMAPI
● Allow snapshots of Bhyve branded VMs (TRITON-716)
● Add flexible_disk_size support (TRITON-961). Note this
means that, when a VM brand is bhyve and package's
flexible_disk is set to true, the VM will have a
flexible_disk_size attribute with the value
Number(pkg.quota)
● Expose free_space for bhyve VMs (TRITON-1099)
10© 2019 Joyent, Inc.

11. Triton Changes
VMAPI
● Added CreateDisk, ResizeDisk and DeleteDisk
end-points to VMAPI (TRITON-962)
● Added support for remaining value for disk size, both for disk
(TRITON-1641) and VM creation (TRITON-1764)
● Stop passing quota value to vmadm for KVM and Bhyve VMs.
We should rely into vmadm default values. Pass only disks and
flexible_disks_size (TRITON-1783)
11© 2019 Joyent, Inc.

12. Triton Changes
CloudAPI
● Support flexible disk information for Machines/Packages (TRITON-965)
This adds attributes disks, flexible and free_space to CloudAPI's
representation of Bhyve VMs
● The disks attribute is an array where each one of the members has the
attributes id and size. The OS disk also has the attributes image and
boot
● Added CreateMachineDisk, ResizeMachineDisk and
DeleteMachineDisk end-points to CloudAPI (TRITON-966)
12© 2019 Joyent, Inc.

13. Triton Changes
CloudAPI
● Added support for remaining disk size value for
CreateMachineDisk and CreateMachine (TRITON-1631)
● Bhyve instances disk attribute has the value of
flexible_disk_space instead of quota (TRITON-1780)
● Instance creation does not permit first disk assignment to be
lesser than image size (TRITON-1785)
13© 2019 Joyent, Inc.

14. Triton Changes
node-triton
● Add support for flexible disk, disks information and disks
modifications to triton (TRITON-967)
14© 2019 Joyent, Inc.

15. Triton Changes
AdminUI
● Added initial support for multiple disks, packages flexible disk
and instances disk. (More to come for 2019 second half).
● TRITON-1775 Flexible disk and disks values should be
displayed on the vm#view page for Bhyve VMs
● TRITON-1649 Add flexible_disk to AdminUI’s packages UI
15© 2019 Joyent, Inc.

16. Triton Changes
Creating snapshots of Bhyve machines through
CloudAPI
● This feature isn't enabled by default from CloudAPI.
● Need to add the following SAPI config:
16© 2019 Joyent, Inc.
# cloudapi_svc=$(sdc-sapi /services?name=cloudapi | json -H 0.uuid)
# sapiadm update "$cloudapi_svc"
metadata.experimental_cloudapi_bhyve_snapshots=true

17. 17© 2019 Joyent, Inc.