Leases and-caching final

1. Soumya Koduri
Poornima G
Lease Locks and
Caching
Gluster design Summit
May. 2015

2. 05/17/16 2GlusterFS Meetup
Agenda
● Quick intro to Lease locks
● Why implement in Gluster
● Design
● Network Partitions
● Challenges / limitations
● Future enhancements
● Client side caching with leases

3. 05/17/16 3GlusterFS Meetup
● Locking mechanism, which guarentees the
client, certain semantics with respect to the
sharing of the file/directory with other clients.
● Granted on regular files (file/directory).
● Protocol for cache consistency and aggressive
caching.
● Enhances performance.
Quick intro to Lease locks

4. 05/17/16 4GlusterFS Meetup
Quick intro to Lease locks
Client 1
Server
Request for
lease on file1
file1
file1
Granted,
cache file1
Client 2
Request for
access to file1
Lease Break file1
Flush all the
cached data
Flush file1,
Unlock lease
6.Granted access
to file1
1
2
6
4
3
5

5. 05/17/16 5GlusterFS Meetup
Why support it in Gluster
● NFS-Ganesha/Samba applications
● Multi-protocol access on Gluster
● A new client side caching xlator based on leases
– File data
– Directory entries (+ readdirplus)
– Metadata

6. 05/17/16 6GlusterFS Meetup
Lease locks and its
manifestations
● Linux lease locks:
- Read lease
- Write lease
- Process based
● NFS V4 delegations:
- Read delegations
- Write delegations
- Directory delegations
- Client based

7. 05/17/16 7GlusterFS Meetup
Lease locks and its
manifestations
● SMB
- Oplocks (opurtunistic locking)
Level 1 (exclusive lock)
Level 2 (Read lock)
Batch (handle caching +
exclusive lock)
Fd based
Leases introduced in SMB 2.1
Read
Read/Handle
Read/Write
Read/Write/handles
Client based

8. 05/17/16 8GlusterFS Meetup
On Gluster
● Currently supported
- Read Lease
- Write Lease
- fd based
● Plan is to support a super set of all standards:
- Handle Lease, Read handle, Read Write Handle Lease
- Lease Key support i.e. client based.
- Lease upgrade/downgrade.
- Directory Lease

9. 05/17/16 9GlusterFS Meetup
Design
Gluster Server
Locks xlator
Lease xlator
libgfapi
Protocol clnt
Protocol server
glfs_adv_lock()
Lk Request
Lk Response
Recall Lease
Uses Upcall
Infra
- Stores the lease info
- Check conflict,grant/
Recall lease
- Timer based recall

10. 05/17/16 10GlusterFS Meetup
Design
● New API introduced in libgfapi to request lease locks.
int glfs_*advanced*_lock ();
● Lease xlator on server side above Posix locks:
- Maintains the lease state of a file
- Checks for conflicts, Grants, Recall, lease locks
● Recall lease is implemented using Gluster file system
notifications.
● Timer based recall

11. 05/17/16 11GlusterFS Meetup
Network Parition
● Gluster Server disconnects:
➢ Replica brick will serve the lease state.
➢ Non-replica - the lease state is lost forever.
● Gluster Client disconnects:
➢ All the locks held by this client will be unlocked.
● Gluster Server/Client Reconnects:
➢
The client will replay all the lease locks it has held.

12. 05/17/16 12GlusterFS Meetup
Challenges / Limitations
● Rebalance/Tiering
● Self-Heal
● Client lock replay not enabled.
● Inconsistency when used along with
performance xlators
● Delegation throttling
● Tracking non fd based fops and the actual
client associated

13. 05/17/16 13GlusterFS Meetup
Future Enhancements
● Super set of all leases
● Upgrade and downgrade leases
● Heuristics based lease grant
– Conflict frequency
– Limit on leases granted
– Quota limits
● Migrate/heal lease state
● Locks recovery during network disconnects

14. 05/17/16 14GlusterFS Meetup
Client side caching

15. 05/17/16 15GlusterFS Meetup
Common challenges of
caching
● Challenges with caching
➢ Mostly in memory hence less aggressive
➢ Not distributed cache coherent
➢ Performance hit with cache-validation
requests
➢ Not tolerant with file-system notification
failures

16. 05/17/16 16GlusterFS Meetup
Benifits of using leases
● Cache Coherency
● Aggressive persistent data caching
● Reduces application-level response time
● Suitable for distributed systems of larger
scale with their larger ratio of processor
speed to network delay
● Performance improvement in the workloads
with little write-sharing

17. 05/17/16 17GlusterFS Meetup
Benifits of using leases
● Directory leases will help in caching readdir and
readdirplus coherently
● Well suited for home directory use cases
● Decreases file creation latency

18. 05/17/16 18GlusterFS Meetup
Client cache based on leases
Gluster Server
Locks xlator
Lease xlator
Libgfapi/FUSE
Protocol clnt
Protocol server
Open file1
Open + Lease req
Lease Granted rsp
Lease Cache
Open
Open+
lease
Cache file1
if lease granted
Gluster Client

19. 05/17/16 19GlusterFS Meetup
Thank You!