vmfs intro

VMFS IntroductionBergwolf@linuxfb.org

AgendaESX IntroductionVMFS Design GoalsVMFS ArchitectureSAN ImpactConclusion

ESX System Setup

Guest Memory Layers Shadow page tables (VA- MA). Page sharing (BA-MA).

ESX IO Stack Average IO requests just involves offset remapping.

AgendaESX IntroductionVMFS Design GoalsVMFS ArchitectureSAN Influence and ImpactConclusion

Use CaseSmall number of files (30~100 per VM)Files either very small (~a few KBs), or verylarge (many GBs)SAN storage is the underlying substrate.All storage exported by these storage systemsis shared among all ESX servers

Design GoalsMetadata overhead should be very lowVM IO throughput and latency should be asgood as directly attached raw deviceA clustered lock manager for moderatingaccess to files among ESX serversHelp VM deterministically react to transientand non-transient SAN events and errorconditions.

VMFS ArchitectureA volume is an aggregation of resources and on-disklocks.A resource is either an inode, a file block, a sub-block or an indirect block.Each lock moderates access to a subset of resources.Hosts negotiate access to resource by acquiringrelevant locks.VMFS = a clustered lock manager + a resourcemanager + a journaling module + a data mover + aVM IO manager + POSIX system call frantend

VMKernel Logical VolumeVMFS are by default created inside VMKernel logical volumes. VMKernel logical volumes can be spanned across multiple devices.

VMFS on disk Layout

Four Resources file blocks sub-blocks pointer blocks file descriptorsResources are grouped together into collections called CLUSTERs and clusters are further grouped together into CLUSTER GROUPS.

Block Mapping Packed inside inode Sub block addressing File block addressing Pointer block addressingCan upgrade automatically.

System FilesSystem files are created at file system format time, and each manages one type of resources.

System FilesUse file blocks.Same read/write method as regular files.Checking file data consistency essentiallyprovides metadata consistency.

Cluster GroupsCluster groups are repeated to create a file system.An existing VMFS volume grows over unused spaceon the disk or spans new disks by laying out newcluster groups that refer to the newly added space.VMFS resource manager makes hosts operate ondifferent and distant cluster groups within a systemfile. This reduces the possibility of mutiple hostscontending on the same lock(s) and increases theefficiency of the clustered lock manager.

On-disk LockA single sector datastructure.Locking is based on lease.Atomic disk operations (SCSIreserve-read-modify-write-SCSI release)

On-disk Lock Data StructureHostID: This is a 128-bit unique identifier that identifies the ESX host thatowns the lock at a given point in time. All zeros means no owner.Mode: A set of non-zero values to indicate whether a lock is free, heldexclusively, held by multiple hosts for shared read access, or held bymultiple hosts for shared read and write access.Generation: A monotonically increasing counter, updates every time a lockis acquired, released or broken. While the hostID field sufficientlydisambiguates operations on a lock from different hosts, this fielddisambiguates multiple operations on a lock by the same host.HBregion: For each valid hostID (if any) currently using the lock, a pointerto the on disk heartbeat region of the host.HBgen: A generation number to validate the HBregion reference as beingcurrent or stale. It disambiguates locks held by a given host before andafter a host crash and before and after a storage outage.

On-disk HeartbeatA single sector data structureEvery host accessing a VMSF volume acquiresa heartbeat on disk to declare liveness toother hosts.Allocated from a 1MB reserved region of thevolume. 2048 concurrent hosts access.

HB Failure HandlingHosts are free to break locks if heartbeat’stimestamp does not change for 20 second. Shouldreplay journal when taking stale lock.If failing to update heartbeat timestamp in five HBperiod (about 15 sec and 40 HB IO tries), host willfence itself and abort all inflight IOs.Lock manager tries to rejoin the cluster if IO error isnot permanent, and reclaims HB slot.

On-disk Lock & HBEach host can join a cluster by acquiring a on-disk HB.It can also hold thousands of on-disk locks

JournalingEach host maintains its own journal on thevolume.HB region on disk stores journal location.

Transaction State Machine

Optimistic LockingAll hosts in a VMFS cluster generally operate onmutually exclusive subsets of locks on the volume.A host that is interested in acquiring a given lock willtypically find it to be free on disk.In stead of acquiring all locks, host first reads alllocks, if they are free, modify in memory metadataand then upgrade locks and commit.

Transaction State Machine w/ op lock

Transaction State Machine w/ op lock Upgrade Lock1: reserve disk;2: issue asynchronous (async) reads of allrequired locks;3: if any lock is acquired by remote host,abort and fall back to normal TSM;4: issue async writes of all required locks;5: wait for all async writes to complete;6: release disk;

Adaptive SAN-aware retriesFor some SAN errors, instead of letting guestOS retry IO, VMkernel retries the IO after anoptimal time.

Adaptive SAN-aware retries

Data Moverclone(srcFileHandle, srcFileOffset,dstFileHandle, dstFileOffset, length, policies)

Data Mover

Directive SCSI CMDoperator(VMID, source_blocklist,destination_blocklist)Zero, clone, delete

Directive SCSI CMDatomic_test_and_set(block_number, old_image,new_image)For VMFS lock manager, new lock algorithm: reads alock image from disk, and if the lock is free, issuesan atomic_test_and_set with a new_imagecontaining host specific hostID, generation andheartbeat information.4 IOs -> 2 IOs

Performance

http://bardofschool.blogspot.com	137
http://bardofschool.blogspot.in	7
http://bardofschool.blogspot.de	2
http://bardofschool.blogspot.co.uk	2
http://bardofschool.blogspot.ca	1
http://bardofschool.blogspot.pt	1
http://bardofschool.blogspot.com.es	1

vmfs intro

by bergwolf on Aug 27, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 151

Statistics

vmfs intro — Presentation Transcript