Running without a ZFS system pool
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Running without a ZFS system pool

on

  • 935 views

One of the main design princples of ZFS is merging the management of physical volumes with individual filesystems. Instead of relying on an underlying volume manager, ZFS manages disks directly and ...

One of the main design princples of ZFS is merging the management of physical volumes with individual filesystems. Instead of relying on an underlying volume manager, ZFS manages disks directly and aggregates them into pools from which individual filesystems are allocated. Storage servers using ZFS typically configure two pools: one pool onto which the system’s root filesystem is installed, and a second for the data to be managed by that system.

At Joyent we’ve taken a different approach and discarded the root pool in favor of a single system-wide pool. Not only does this approach free up an additional two drives to be used for main storage, it also provides us flexibility in upgrading system software, higher customer multitenancy, and ease of deploying new machines. In this talk, I’ll describe our overall architecture, talk about challenges we faced in constructing such an architecture, and characterize our experiences having deployed this model in production over the last 18 months.

Statistics

Views

Total Views
935
Views on SlideShare
934
Embed Views
1

Actions

Likes
0
Downloads
15
Comments
0

1 Embed 1

http://www.slashdocs.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Running without a ZFS system pool Presentation Transcript

  • 1. Running ZFS without a system pool Bill Pijewski Software Engineer, Joyent @pijewskiTuesday, October 2, 2012
  • 2. Agenda • Why ZFS is important to Joyent • Evolution of USB and PXE boot architectures • Running with no system poolTuesday, October 2, 2012
  • 3. ZFS at Joyent • We run a production cloud with many servers in datacenters worldwide • Two kinds of zones (covered in detail in other talks): • Zones: sparse zones share libraries with the platform • VMs: fully virtualized GNU/Linux, Windows, FreeBSD, etc. machines • Use small number of NFS machines to provide additional storage capacity in each datacenterTuesday, October 2, 2012
  • 4. ZFS for Zones and VMs • Zones are allocated two ZFS datasets • One dataset for data in that zone • Another for core files -- to prevent cores from exceeding quota • VMs have a ZFS volume into which the VM image is installed, plus one or more additional volumes presented to guest as disks • Guest filesystems are installed into volumesTuesday, October 2, 2012
  • 5. ZFS in different contexts • For Joyent, two main contexts: SmartOS and SDC • SmartOS: community distribution, illumos + lightweight virtualization tools • SmartDataCenter (SDC): SmartOS + full cloud management and orchestration stackTuesday, October 2, 2012
  • 6. Important ZFS features • As with all ZFS users, we take for granted rely on end-to-end data integrity • Copy-on-write architecture: snapshots, clones • Compression • Space management tools: quotas and reservations • Replication to move customers around between different machinesTuesday, October 2, 2012
  • 7. Delegated administration! • In our next SDC release, we enable delegated administration • Allows customers to: • Take snapshots outside of Joyentʼs API • Create child datasets • Snapshot and clone datasets • Replicate or migrate data between instances • Open work: basic limits on delegated activity to avoid DOSTuesday, October 2, 2012
  • 8. ZFS Performance • SSDs for ZIL • ARC • We hold back some portion of a serverʼs total memory, knowing that a good portion of this memory will be consumed by the ARC • Committing memory achieves greater I/O performance • ZFS I/O throttle for QoS controls • For more information, check out Brendan Greggʼs excellent talk next doorTuesday, October 2, 2012
  • 9. Read-only system pool • At Fishworks, we decided to have a read-only system pool • Necessary for OS install as well as analytics data • Simplified some things: • No unnecessary customizations from customers • Discouraged hot patching • Other disadvantages: • Upgrade, rollback, and factory reset were trickyTuesday, October 2, 2012
  • 10. SmartOS USB Boot • Instead of installing OS to root disks, SmartOS boots from a USB key • Entire kernel and userland fit in about 200 MB (compressed) • Other software can be installed from pkgsrc • Single ZFS pool for all zonesTuesday, October 2, 2012
  • 11. USB Boot Advantages • All disks are available for zone/VM storage, thereby increasing both performance and capacity • Encourages users to provision a zone for each application rather than using the global zone • Discourages customization and one-off patching • Fast to get up and running • Easy to “bring your OS with you”Tuesday, October 2, 2012
  • 12. SmartDataCenter (SDC) Architecture • Two kinds of servers: head nodes and compute nodes • Head nodes run management, provisioning, monitoring, and boot services • Compute nodes contain customer zones • Head nodes are similar to SmartOS installs • Each compute node PXE boots its platform from the head node • Both head nodes and compute nodes have a single ZFS poolTuesday, October 2, 2012
  • 13. SDC Diagram DC 0 DC 1 DC 2 Headnode Headnode Headnode PXE PXE PXE CN 0 CN 10 CN 20 CN 1 CN 11 CN 21 CN 2 CN 12 CN 22 ...... ...... ......Tuesday, October 2, 2012
  • 14. PXE Boot Advantages • Ben Rockwood, 10/1/2012: “Apparently other people spend time installing software. I think thats stupid.” • As with SmartOS, operators encouraged to put applications in zones instead of global • Upgrade = rollback = reboot, nothing more • Newer platforms can be staged and machines rebooted later • Any machine which hits a known fixed problem will automatically boot onto fresh platformTuesday, October 2, 2012
  • 15. Storage pools! • Most OSes assume the existence of a “system” pool -- a pool onto which the OS, applications, and configuration information is installed • Joyent moving away from single-vdev pools backed by hardware RAID • Embracing hybrid storage pool (HSP) using an SSD for the ZFS intent log (ZIL) • Everything else worked on RAID-Z pools except for saving a crash dumpTuesday, October 2, 2012
  • 16. RAID-Z Crash Dump • Problem: have only one RAID-Z or mirrored pool but cannot save crash dump on said pool • Implement crash dumps on RAID-Z (majority of work) and pools with multiple vdevs • Not necessarily to save parity bits for crash dump data: • Crash dump is immediately saved upon reboot • Needs to be reliable, simple, and (hopefully) fastTuesday, October 2, 2012
  • 17. Why no parity bits? • Since DVAs on the dump device are preallocated, use those 128K blocks for each write • Most calls into dump entry point are not block aligned • Rather than write variable size, use original 128K • I first calculated parity bits, only my test machine took three hours to save a crash dump • No parity calculated -- on a pool with n vdevs, each write could require n-1 (synchronous) readsTuesday, October 2, 2012
  • 18. Other system components • Swap device (thankfully) supports RAID-Z pools • /var, /opt have their own datasets • /etc not persistent • /root also not persistent, again incentivizing people to configure applications in zones rather than using the GZTuesday, October 2, 2012
  • 19. Summary • The single ZFS pool has simplified Joyentʼs deployment • Delegated administration has given customers more power • ZFS has been and will continue to be a crucial component of our architecture for many yearsTuesday, October 2, 2012
  • 20. Questions?Tuesday, October 2, 2012
  • 21. Running ZFS without a system pool Bill Pijewski Software Engineer, Joyent @pijewskiTuesday, October 2, 2012
  • 22. Backup slidesTuesday, October 2, 2012
  • 23. ZFS 101 • ZFS is a copy-on-write filesystem from Sun originally shipped with Solaris 10 • Many innovative features: data compression, snapshot/rollback, ZFS send/receive, SSD integration • Enterprise-grade reliability and data integrity • Two main components relevant here: • ZFS pools • ZFS datasetsTuesday, October 2, 2012
  • 24. ZFS Pools • Aggregate disks into a single storage pool from which “datasets” are allocated • No parted/LVM needed • Mix both spinning disks and SSDs: • L2ARC: extends filesystem buffer cache • ZIL: absorbs synchronous write activityTuesday, October 2, 2012
  • 25. ZFS Datasets • Datasets are a tree of blocks within the storage pool, presented as either: • A filesystem (file interface) • A volume (block interface) • Datasets can be flexibly resized, and volumes can even be thinly provisioned • Administrative controls on datasetsTuesday, October 2, 2012
  • 26. Zones and VMs • A zone is a lightweight software-virtualized container • Uses the systemʼs OS platform • Allocated its own ZFS filesystem (more in a sec) • A VM is a hardware-virtualized container for GNU/ Linux, Windows, BSD, etc. • Uses its own ZFS volume • VMʼs filesystem installed into ZFS volume • Both machines have resource controls for CPU, memory, and disk I/OTuesday, October 2, 2012
  • 27. Advantages of ZFS • Snapshots: zone/VM backup and recovery • Space management: reservations and quota flexibly allocate space between zones • Delegated administration: each tenant can administer their own dataset: • Set compression level and other properties • Take snapshots of application data • Generate send streams for replication/backupTuesday, October 2, 2012
  • 28. Advantages of ZFS (2) • Data integrity: verifies data of VM guest filesystems (ext4, XFS, NTFS, etc.) • Multiple storage configurations available: mirrored, RAID-Z2, and others • System fully supported on any storage configurations, can even take a crash dump to a RAID-Z poolTuesday, October 2, 2012