Tailoring NAS Proxies for Virtual Machines

3,326 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,326
On SlideShare
0
From Embeds
0
Number of Embeds
2,615
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Tailoring NAS Proxies for Virtual Machines

  1. 1. Caitlin.Bestler@nexenta.com Alex.Aizman@nexenta.comSponsored by: &
  2. 2. • Traditional tradeoff: Local vs. Central storage: – Central storage migrates very simply. • Just complete last work from old location – Then start at the new location. • Extracts non-local latency tax on every access. – Purely local storage makes migration a challenge. • Must migrate VM image and local storage. • And synchronize them. Xen Summit Aug 2, 2011 –Caitlin.Bestler/Alex.Aizman@nexenta.com 2
  3. 3. • SSD vs HDD: –SSDs provide most IOPS per $. • SSDs need to be on host, or at least near the VM, or network latencies minimize SSD speed. –HDDs provide most TBs per $. • HDDs provide capacity, not processing speeds. Xen Summit Aug 2, 2011 –Caitlin.Bestler/Alex.Aizman@nexenta.com
  4. 4. • A very common solution currently features Host DomU a Block Backend that co-ordinates a local App block device (typically SSD) with remote VFS File System block device (still NAS or SAN). Block Driver• Problem: Device Domain – Any cache solution only stores a portion Block Backend SSD Proxy of a VM‟s storage locally. Remote Block Local Block Driver Access – With NAS Proxy the “portion stored locally” can be understood as files and File System Block NAS Client directories. Network Driver Network • A block-oriented solution never sees Driver Driver more than a collection of blocks. Local Block Device • Many solutions are enhanced or External NAS External SAN made possible with File-orientation. Server(s) Server(s) Xen Summit Aug 2, 2011 –Caitlin.Bestler/Alex.Aizman@nexenta.com 4
  5. 5. Host• DomU can support both NAS DomU App access and Virtual Disk Access VFS – Most VMs will prefer directly-attached boot File NAS Client System drive. Block Network Driver Driver• Backend can map virtual drive to Device Domain Block Network NAS. Backend Backend• Meanwhile NAS Client can access NAS Proxy Network Block NAS Proxy via local internal Driver Driver network. Local Block Device• NAS Proxy can utilize local SSD External NAS Server(s) drive to optimize either.5 Xen Summit Aug 2, 2011 – Caitlin.Bestler/Alex.Aizman@nexenta.com
  6. 6. • NAS Benefits – Files enable understanding of context: – Deduplication of files is far easier than deduplication of block. – Hierarchical Storage Management works on files, not blocks.• Traditional NAS Proxy Caveats – Traditional NAS is unaware of VM-specific semantics. – Excessive syncing – Historical assumptions are that files are shared by default. – Proxy overhead for non-cached files – Historically, the only easy configuration was to proxy all NAS access.• VM Storage Proxy provides full benefits of NAS without limitations of traditional NAS Proxies.6 Xen Summit Aug 2, 2011 – Caitlin.Bestler/Alex.Aizman@nexenta.com
  7. 7. • Distributed NAS protocols such as NFSv4 and DFS/CIFS have features that make NAS proxies far more efficient.• They allow efficient caching via exclusive delegation and oplocking (“opportunistic locking”). – Proxy can temporarily claim a file.• They have advanced session features to protect against data loss from connection loss. – File reverts to central NAS server if Proxy dies. – System recovers from network partitions and temporary outages of distributed servers and/or proxies as long as they all have persistent redundant storage. Xen Summit Aug 2, 2011 –8 Caitlin.Bestler/Alex.Aizman@nexenta.com
  8. 8. • Client can simply mount from NAS server. VM Client• NAS server can refer select directories to a NAS “Server” that is actually the NAS Proxy. NAS Proxy / Server – NAS Proxy/Server must act as Referral a NAS Server for NAS Replication replication procedures. Cached Files• Client does not need to identify Directory access itself. – NAS server can infer identity from IP address. All NAS Server Directories• Advantage that Proxy/Server is enabled/disabled for each directory from the central NAS Server. Xen Summit Aug 2, 2011 –9 Caitlin.Bestler/Alex.Aizman@nexenta.com
  9. 9. • NAS Proxy can act as a NAS VM Client client and claim an exclusive delegation on a file.• It can then treat its cached copy Local copy can be treated as THE copy. as the copy. Cached NAS Proxy /• It must replicate it back to the Files Server central NAS server before Delegation ending the delegation. Granted Claim Exclusive Delegation• Keep-alive messages track the NAS client with the delegation. All NAS Server Directories• Protocol rules already cover loss of contact/failure scenarios. Xen Summit Aug 2, 2011 –10 Caitlin.Bestler/Alex.Aizman@nexenta.com
  10. 10. • Generic NAS Proxy does not understand that it is the sole writer for the VM Private Directory.• NAS Proxy management is not integrated with VM Management. – The NAS system must detect that the VM using its VM Private Directory has migrated.• Proxies only optimize storage for selective files – but all traffic goes through them. – If there are directories/files that are not worth caching the clients would ideally just go directly to the real server. – NAS Referrals can be applied selectively, but Management Plane controls are awkward for dynamically controlling referrals.• Optimized cache sharing across multiple clients is challenging without understanding of VM life-cycle. Xen Summit Aug 2, 2011 – 12 Caitlin.Bestler/Alex.Aizman@nexenta.com
  11. 11. • Allow VM‟s files to be stored and archived as files.• Allow storage associated with commodity servers to be deployed effectively for VMs in combination with NAS storage.• Don‟t replicate to local storage those files that probably won‟t be accessed anyway.• Allow very thin provisioning of local storage. – SSDs are designed to provide IOPs cost effectively, not storage. – If IOPS are not needed, avoid cost of redundant storage. Xen Summit Aug 2, 2011 –13 Caitlin.Bestler/Alex.Aizman@nexenta.com
  12. 12. • The VM NAS Proxy is a specialized NAS Proxy that is optimized to support VMs and the Hypervisor Platform supporting them:• A VM Private Directory is maintained for each VM: – A per VM directory in the NAS global name space that holds files that are “local” for this VM. – A “platform” sub-directory hold virtual disk images and other files used by the Hypervisor and/or Dom0. – A “mount” sub-directory holds NAS mount points that are available only to this VM. • These are implicitly named. The client does not know the path to its VM Private Directory because it does not know which VM it is. • A “proxy” sub-directory holds files used by the VM NAS Proxy for this VM, such as the NAS Response Cache. Xen Summit Aug 2, 2011 –14 Caitlin.Bestler/Alex.Aizman@nexenta.com
  13. 13. VM NAS Proxy as a VM. VM NAS Proxy for a Rack Virtualization Host Rack (or other enclosure) Virtualization Host Hypervisor Platform VM NAS Proxy VM13478 VM Hypervisor Platform VM13478 VM /foo VM NAS Proxy /foo /bar /bar NAS VMS /vmexports /vmexports /vm13478 NAS VMS /vm13478 /platform /platform /mounts /mounts NAS NAS /foo /foo ` /bar ` /bar /vm13483 /vm13483 /platform Virtualization Management Virtualization Management /platform /mounts /mounts /foo /foo /bar /bar15 Xen Summit Aug 2, 2011 – Caitlin.Bestler/Alex.Aizman@nexenta.com
  14. 14. • Virtualization Management – Unchanged, but new information is available to it.• NAS VMS (Virtualization Management System) – Extends Virtualization Management to assign VM NAS Proxies.• Hypervisor Platform – The Hypervisor and supporting domains.• NAS system with NFSv4/v4.1 and/or DFS/CIFS. – Unchanged, but optional enhancements are defined.• VMs – Unchanged, but optional enhancements for NAS Client are defined.• VM NAS Proxy – A specialized NAS Proxy optimized for support of VMs Xen Summit Aug 2, 2011 –16 Caitlin.Bestler/Alex.Aizman@nexenta.com
  15. 15. • A VM NAS Proxy can control what is stored locally through several techniques: – Which sub-directories of the VM‟s private directory are referred to it. – Whether it provides local storage for specific files within those directories. – Which blocks of a file that is stored locally are actually stored locally, vs. being accessed over the network. – pNFS-aware clients can be enabled to access network block servers directly. Xen Summit Aug 2, 2011 –17 Caitlin.Bestler/Alex.Aizman@nexenta.com
  16. 16. • VM will be migrated from Current Host Virtualization Host Current Home to a new Host. 10.1.3.115Hypervisor Platform VM• VM has public IP address, which 192.168.1.115 10.1.1.1 192.168.1.1 Virtualization Management will migrate. VM NAS Proxy 10.1.1.2• VM has private IP Address which it will 192.168.1.2 use in new network without realizing that it is on a new network. Planned Home• New VM NAS Proxy will have same Virtualization Host private IP address on the new network. 10.1.3.115 Hypervisor Platform 10.1.2.1 192.168.1.1 NAS 192.168.1.115• New VM NAS Proxy does not respond 10.1.2.2 192.168.1.2 to the VM‟s private IP Address until it is ready. VM Storage Proxy Xen Summit Aug 2, 2011 – 19 Caitlin.Bestler/Alex.Aizman@nexenta.com
  17. 17. • By assigning VM NAS Proxies to an Internal Network migration of a VM can be made transparent to the VM. – All Internal Networks can look alike. – Just as almost all home networks are 192.168.1.*• NAS client does not need to know about the migration. – NAS clients will see a dropped connection to a NAS server. – NAS clients will assume the NAS server is simply rebooting. – Recovery will be complete and transparent to client layers. – No change to the IP address of the VM NAS Proxy the Client uses.• As far as the client is concerned it just reconnected to the same server – It even has the same response cache. – The response cache is stored within the VM Private Directory.20 Xen Summit Aug 2, 2011 – Caitlin.Bestler/Alex.Aizman@nexenta.com
  18. 18. • VM will be migrated from Current Current Home Virtualization Host Host to a new Host. – 10.1.3.115 is migrated. VM Hypervisor Platform 10.1.3.115 10.1.1.1• VM is still instructed to use the old VM NAS Proxy (10.1.1.2). 10.1.1.2 VM NAS Proxy – It will continue to use it until explicitly redirected to the new VM NAS Proxy ac ce ss d ue (10.1.2.2). Co nt in Planned Home – This is very network inefficient. Virtualization Host – Current RTT measurements and Hypervisor Platform 10.1.3.115 congestion window will be wildly 10.1.2.1 NAS incorrect. 10.1.2.2 – VM NAS Proxy must accept clients from anywhere. VMNAS Proxy Xen Summit Aug 2, 2011 – 21 Caitlin.Bestler/Alex.Aizman@nexenta.com
  19. 19. • Distributed NAS features for Referral and Delegation have nothing to do with each other. – Referral/redirection is a Management Plane feature. • It is used to refer/redirect clients to other NAS servers. – Delegation/oplocking is a Control Plane feature. • It is used by NAS clients.• A VM NAS Proxy integrates them in collaboration with NAS servers. Xen Summit Aug 2, 2011 –23 Caitlin.Bestler/Alex.Aizman@nexenta.com
  20. 20. • VM NAS Proxies use Delegation/Oplocking to exclusively claim specific sub-directories within a VM Private Directory. – This is an enhancement to a NAS server.• Supporting NAS servers will then automatically direct referrals for the claimed sub-directories to that VM NAS Proxy for the duration of the delegation/oplock. – Reverting is automatic, preferable to having to reverse a management plane edit. Xen Summit Aug 2, 2011 –24 Caitlin.Bestler/Alex.Aizman@nexenta.com
  21. 21. • Only the correct clients are referred,• Other clients are refused access.• Allowed access types: – Hypervisor Platform access a „platform‟ sub- directory for virtual disk images and snapshots. – VM itself accesses NAS mounts in a „mount‟ sub- directory. – VM NAS Proxy keeps its own data in a „proxy‟ sub-directory. • This is where the NAS Response Cache is kept. Xen Summit Aug 2, 2011 –25 Caitlin.Bestler/Alex.Aizman@nexenta.com
  22. 22. 2 3 1 VM NAS Proxy NAS NAS Server Server Locally Cached Files• 1 – VM NAS Proxy takes exclusive delegation on a directory.• 2 – Client referencing delegated directory is referred to VM NAS Proxy• 3 – Client accesses directory on VM NAS Proxy Xen Summit Aug 2, 2011 –26 Caitlin.Bestler/Alex.Aizman@nexenta.com
  23. 23. • Progression of Solutions for Controlling File Locations: – Without NFSv4 Referrals/DFS Redirections: • Load balancing requires editing each client‟s configuration. • Or directing all traffic through a proxy. – Using just standard Referrals/Redirections: • Central control of which directories are referred • But only via Management Plane edits of exports. – Integrating Delegation to allow Automatic Referrals: • Dynamic control over which directories are referred is exercised by the VM NAS Proxy and/or VM NAS Management. • Dynamic referral automatic ends when delegation ends. Xen Summit Aug 2, 2011 –27 Caitlin.Bestler/Alex.Aizman@nexenta.com
  24. 24. • One of the features of the VM NAS Proxy is that it can dynamically control which of a VM‟s private directories are referred to it. – But standard granularity for referral is an entire directory.• Partial Referrals is a new technique to publish alternate referrals for subsets of a directory: – Each partial referral identified by a regular expression. – The alternate server is then specified for that regular expression, which can be “revert to default” to ignore the main referral. – This information is published as named attributes for the directory. • If a NAS client does not understand them then it will simply go to the default server for the directory • The default server then acts as a proxy for the actual server. Xen Summit Aug 2, 2011 –29 Caitlin.Bestler/Alex.Aizman@nexenta.com
  25. 25. 2B 1 2A VM NAS Proxy NAS NAS Server Server Locally Cached Files• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.• 2a – Client continues to access non-excluded portions of directory on the VM NAS Proxy, which caches them locally.• 2b – But now accesses files covered by partial referral on the home NAS Server Xen Summit Aug 2, 2011 – 30 Caitlin.Bestler/Alex.Aizman@nexenta.com
  26. 26. 1 2 3 VM NAS Proxy NAS NAS Server Server Locally Cached Files• 1 – VM NAS Proxy advertises portions of directory that Client should access directly on the home NAS server.• 2 – Client does not understand or ignores the partial referral, so all access for the directory goes through the VM NAS Proxy• 3 – When files are not stored locally access is proxied via the NAS Server. Xen Summit Aug 2, 2011 –31 Caitlin.Bestler/Alex.Aizman@nexenta.com
  27. 27. • When the VM‟s NAS Client indicates that it understands pNFS, the VM NAS Proxy Locally Cached Blocks can supply pNFS layouts Logical Physical Block # Block # that mix references to its Logical Physical Proxy pNFS Layout own storage and NAS Block # Block # Logical Block # Local Volume ID Physical Block # storage. Logical Physical• Volume ID Referencing pNFS blocks Block # Block # Logical Physical directly is safe, because the Block # Volume ID Block # VM NAS Proxy is the only pNFS Layout Logical Local Physical client allowed to update the Logical Physical Block # Volume ID Block # Volume ID Logical Physical file while it is claimed. Block # Block # Block # Volume ID Block #• pNFS-aware clients can Logical Block # Volume ID Physical Block # Logical Block # Volume ID Physical Block # access unmodified and Logical Volume ID Physical Logical Physical Block # Block # Volume ID uncached blocks without Logical Physical Block # Block # overhead of going through Block # Volume ID Block # the proxy. Xen Summit Aug 2, 2011 –33 Caitlin.Bestler/Alex.Aizman@nexenta.com
  28. 28. • Named attributes can identify files that are to be stored only in the local cache. – The file is never updated on the NAS server.• Regular expression can easily identify common temporary files.• These files would be stored only on local SDD – never consuming any network bandwidth. Xen Summit Aug 2, 2011 –34 Caitlin.Bestler/Alex.Aizman@nexenta.com
  29. 29. • VM NAS Proxy is only on the path for a VM pNFS Block Server accessing storage Direct access to un-cached blocks when it wants to be. VM – Control is fully dynamic. Default Access, may be referred to VM Storage Proxy Access to non-referred directories Access to partially un-referred files• Access to Non- NAS Server optimized Access to referred directories, files directories/files can VM NAS PRoxy bypass the Proxy.• VM NAS Proxy is only identified as referral target for the correct clients. Xen Summit Aug 2, 2011 –36 Caitlin.Bestler/Alex.Aizman@nexenta.com

×