Accidental PVC delete or namespace delete can cause the Persistent Volume to get deleted. Such volumes lose their data and the stateful applications lose their state. By the use of Persistent Volume TrashCan, users can get a grace period to undo such unintended delete operation. The deleted Persistent Volumes are staged for delayed deletes. They continue to live even after being deleted from k8 perspective, for a configurable time(retention period) and based on the system’s usage. The storage class of the PVC can dictate if they need to be staged for a delayed delete. StorageClass can also allow for configurable retention period. To recover a deleted PersistentVolume, users can create a new namespace with the same name and reapply the original PVC spec. The PVC will reference a special StorageClass to indicate that the new PersistentVolume needs to be restored from the TrashCan. This will allow the application to restart with the right state and data. This talk will showcase how to overcome one of the admin’s pain point seen in field involving accidental deletions of PVCs by using advanced storage management solutions in Kubernetes. This talk was given by Veda Talakad, Aditya Kulkarni, and Aditya Dani for DoK Day Europe @ KubeCon 2022.

1. DoK Day Europe 2022 @ KubeCon
Protect data from accidental deletes
Aditya Dani
adani@purestorage.com
Aditya Kulkarni
akulkarni@purestorage.com
Veda Talakad
vtalakad@purestorage.com
PV TrashCan

2. DoK Day Europe 2022 @ KubeCon
Does this sound familiar?

3. DoK Day North America 2021 @ KubeCon
Accidental deletion is
a real problem!
● Accidental deletions of Kubernetes namespaces and/or PersistentVolumes is a common problem in our
community.
● Losing Data is the most expensive and painful of the known problems.
● Human error is the number one cause of data loss in Organizations.
Source - https://webtribunal.net/blog/data-loss-statistics/ - gref

4. Existing ways to recover
data
DoK Day North America 2021 @ KubeCon
● Regular Backups: backup tools that would backup your namespaces.
○ You may not have the latest data.
○ Time to recover can potentially be a lot.
● DR
○ Great if you have it!
○ However, cost of sub-second RPO may not be warranted in all scenarios.

5. Volume TrashCan
DoK Day North America 2021 @ KubeCon
Prevent data loss due to
unintended volume deletes
Recover deleted
volumes
Protect the
deleted volumes
from overwrites
Control the lifespan of volumes in
trash can
Better user experience by
handling manual deletion errors
Simplified operations for
data recovery
Benefits:

6. PVC TrashCan to the rescue!
DoK Day North America 2021 @ KubeCon
● When the volume is deleted the volume state and the data is captured and retained in a staging
area.
● One way to achieve this is by using a volume snapshot and copying over the labels and other
storage class provided options.
● Another way will be to tier out the data to cold storage to free up the capacity on the cluster for hot
data.

7. Steps to Resurrect the PVC
DoK Day North America 2021 @ KubeCon
1. Annotate the CR with restore-from-trashcan to provide a hint to the storage provider
2. Scale up the applications
3. Storage provider will use the annotations to search for the volume in TrashCan. This volume will be
restored instead of creating a new one.
Search will find a TrashCan entity with matching PVC name and Namespace annotation.

8. DoK Day North America 2021 @ KubeCon
Controlling
duration of
TTL for a
volume in
TrashCan

9. DoK Day North America 2021 @ KubeCon
Mark PVC to
restore from
TrashCan

10. Resurrecting a Namespace
DoK Day North America 2021 @ KubeCon
Namespace States
• Namespace Ready:
Scale up creates new volumes
• Namespace In-TrashCan:
Namespace does not exist. No
new volumes can be created in
this namespace.
• Namespace Restore:
Any new volume create
operation searches for a volume
to restore from TrashCan

11. In-Restore State
DoK Day North America 2021 @ KubeCon
• No more TrashCan cleanup until the namespace is out of In-restore state, i.e. Ready or In-
Trashcan states.
• New volume creates do search in TrashCan and if found will do a restore.
• New volume creates will end up creating a new volume, when TrashCan search does not yield a
matching volume.

12. DoK Day North America 2021 @ KubeCon
Restoring
Namespace
from
TrashCan

13. Side effects of staging the
PVCs
DoK Day North America 2021 @ KubeCon
● Storage capacity is not immediately visible after intended namespace/PVC deletion(choose a
meaningful TTL duration for TrashCan!)

14. Demo
DoK Day North America 2021 @ KubeCon

15. What’s Next?
DoK Day North America 2021 @ KubeCon
● Make TrashCan generic for all storage providers
○ Hook onto the PVC controller
■ Invoke CSI snapshots as a part of PV deletions.
■ Invoke CSI snapshot restore as a part of PV creations.
● Introduce TrashCan for CRs
○ Keep the PVC and PV CRs in a trashcan, so the K8 objects can also be restored.
○ Use the trashcan-minutes, to determine how long to keep the PVC and PV objects in Kubernetes
ETCD.

16. DoK Day North America 2021 @ KubeCon