Gluster Summit 2017 - Michael Goulet

1. Gluster QoS at Facebook
Michael Goulet, Caltech

2. Overview
• Goals/non-goals
• Namespace tagging
• Io-threads changes
• Changes in-action (data!)
• Future work

3. Problem Statement
• Increasing storage utilization requires higher-density deployments
• Many Gluster users at FB only use a fraction of their deployed capacity!
• Sharing volumes between tenants is one way to increase density
• However, it can be unreliable to share with “noisy neighbors”
• Goal of better space utilization does not justify worse performance

4. Goals and Non Goals
• Need to spread I/O fairly between tenants
• “fairly” = policy-based share of total IOPS, not naïve first-come first-
served
• “tenant” = real person/org (or internal activity), may have multiple
clients/connections
• Not trying to achieve fine-grained perfect ratios
• Just “keeping a lid” on greedy or malfunctioning users
• Try to make sure everyone has some traffic going through
• Avoid absolute numbers (e.g. IOPS), tricky tuning
• Not scalable and difficult to manage

5. Goals and Non Goals
• Need to spread I/O fairly between tenants
• “fairly” = policy-based share of total IOPS, not naïve first-come first-
served
• “tenant” = real person/org (or internal activity), may have multiple
clients/connections
• Not trying to achieve fine-grained perfect ratios
• Just “keeping a lid” on greedy or malfunctioning users
• Try to make sure everyone has some traffic going through
• Avoid absolute numbers (e.g. IOPS), tricky tuning
• Not scalable and difficult to manage

6. ”Who should I say is calling?”
• Before we can do anything, we have to know which request belongs to
which tenant.
• “namespace” = top level directory under volume root
• Namespace translator runs on bricks, right below server translator.
• Tags FOPs with namespace information for use in later translators.
• Brick-side tagging has pros/cons:
• Impartial to any specific connection method (NFS, FUSE, etc)
• However, we have no client-side information to guide FOP-tagging process

7. ”Who should I say is calling?”
• Before we can do anything, we have to know which request belongs to
which tenant.
• “namespace” = top level directory under volume root
• Namespace translator runs on bricks, right below server translator.
• Tags FOPs with namespace information for use in later translators.
• Brick-side tagging has pros/cons:
• Impartial to any specific connection method (NFS, FUSE, etc)
• However, we have no client-side information to guide FOP-tagging process

8. Namespace Tagging at a High-level
• For each request, find top-level directory
(much like quota does)
• Hash directory name to a namespace ID
• Tag FOP for subsequent translators to use
• Other translators just need to convert
namespace ID back to a namespace string
• Bonus: Enables per-namespace statistics
and other namespace tracking for free
• Invaluable for identifying “bad actors” in
production

9. Namespace Tagging at a High-level
• For each request, find top-level directory
(much like quota does)
• Hash directory name to a namespace ID
• Tag FOP for subsequent translators to use
• Other translators just need to convert
namespace ID back to a namespace string
• Bonus: Enables per-namespace statistics
and other namespace tracking for free
• Invaluable for identifying “bad actors” in
production

10. Sharing Time: Ensuring Fairness
• Modifications to io-threads translator, group FOPs
by namespace
• Separate queue per-tenant instead of per-connection
• General approach to scheduling is Weighted Fair
Queuing
• Each tenant gets a “share” (weight)
• FOPS[tenant] = FOPS[total] * share[tenant] / share[total]
• Tenant queues are strung together in a ring-of-queues called “the
clock.”
• The same queue can show up multiple times in the clock based on
weight
• Future: experiment with other priority algorithms
• Abstract io-threads so different algorithms can be plugged in and
chosen
• e.g. Deficit Round Robin (to deal with very expensive operations)

11. Sharing Time: Ensuring Fairness
• Modifications to io-threads translator, group FOPs
by namespace
• Separate queue per-tenant instead of per-connection
• General approach to scheduling is Weighted Fair
Queuing
• Each tenant gets a “share” (weight)
⌾ FOPS[tenant] = FOPS[total] * share[tenant] / share[total]
• Tenant queues are strung together in a ring-of-queues called “the
clock.”
• The same queue can show up multiple times in the clock based on
weight
• Future: experiment with other priority algorithms
• Abstract io-threads so different algorithms can be plugged in and
chosen
• e.g. Deficit Round Robin (to deal with very expensive operations)

12. Sharing Time: Ensuring Fairness
• Modifications to io-threads translator, group FOPs
by namespace
• Separate queue per-tenant instead of per-connection
• General approach to scheduling is Weighted Fair
Queuing
• Each tenant gets a “share” (weight)
⌾ FOPS[tenant] = FOPS[total] * share[tenant] / share[total]
• Tenant queues are strung together in a ring-of-queues called “the
clock.”
• The same queue can show up multiple times in the clock based on
weight
• Future: experiment with other priority algorithms
• Abstract io-threads so different algorithms can be plugged in and
chosen
• e.g. Deficit Round Robin (to deal with very expensive operations)

13. Relative FOP numbers
FOPs / second
High-priority Share Low-priority Share
1:1 1:2
1:3
1:10 Off

14. Relative FOP numbers
FOPs / second
High-priority Share Low-priority Share
1:1 1:2
1:3
1:10 Off

15. Ratios!
0
2
4
6
8
10
12
Ratio between FOP rates
1:3
1:10 Off
1:1
1:2

16. Ratios!
0
2
4
6
8
10
12
Ratio between FOP rates
1:3
1:10 Off
1:1
1:2

17. Future Plans
• Full-scale, production rollout of Namespace changes
• Upstreaming to 3.8, then 4.x
• Gather high-IO and high-capacity customers onto shared volumes
• Explore new queueing algorithms in io-threads (e.g. DRR)
• Tighter integration with other Gluster features
• Self-heal, rebalance
• Snapshots
• Multiplexing

18. Future Plans
• Full-scale, production rollout of Namespace changes
• Upstreaming to 3.8, then 4.x
• Gather high-IO and high-capacity customers onto shared volumes
• Explore new queueing algorithms in io-threads (e.g. DRR)
• Tighter integration with other Gluster features
• Self-heal, rebalance
• Snapshots
• Multiplexing

19. Questions