Use EasyHSM and Nirvana to migrate a storage system containing a couple PBytes of data into IBM Spectrum Scale (GPFS) over the weekend.

1. 1
How to migrate a storage system
holding a couple PBytes of data
into IBM Spectrum Scale (GPFS)
over the weekend
HSM migration
with EasyHSM and Nirvana
Feb 2017 Igor Sfiligoi

2. 2
• You have a storage system containing
a couple PBytes of data
• And you want to migrate off of it
– Due to cost, performance, age, etc.
• How long will it take?
– At 10 Gbps – at least 3 weeks
– Longer if you have smaller files
• What will your users do during that period?
– How do you handle consistency?
So, you want to migrate a PByte storage system
2PBytes/10 Gbits/s ~= 1.7M secs
1.7M secs / 3600 / 24 ~= 20 days

3. 3
• Moving 2 PBytes of data in one shot
takes a long time
– Not much you can do about it
(short of spending a fortune in networking, and even then…)
• But why do you want to do it in the first place?
– User convenience
(single namespace)
– Data consistency
(which copy is the authoritative)
– Let’s be done with it
But why move all the data at once?

4. 4
• Moving 2 PBytes of data in one shot
takes a long time
– Not much you can do about it
(short of spending a fortune in networking, and even then…)
• But why do you want to do it in the first place?
– User convenience
(single namespace)
– Data consistency
(which copy is the authoritative)
– Let’s be done with it
But why move all the data at once?
Don’t need data to be moved,
just the file structure
(the namespace, metadata)

5. 5
• A PByte-class storage system will likely have
about 100M files in it
• Using a dedicated tool, you can
catalogue them in less than half a day
– General Atomics’ Nirvana has proven
it can crawl 100M files
in 3h when on IBM Spectrum Scale™(GPFS), and
in 6h when on Isilon™
How long will it take to migrate just the metadata?
2PBytes/ 100M files = 20MBytes/file

6. 6
• A PByte-class storage system will likely have
about 100M files in it
• Using a dedicated tool, you can
catalogue them in less than half a day
– General Atomics’ Nirvana has proven
it can crawl 100M files
in 3h when on IBM Spectrum Scale™(GPFS), and
in 6h when on Isilon™
How long will it take to migrate just the metadata?
2PBytes/ 100M files = 20MBytes/file
OK, now I know what is
in my old system.
Now what???

7. 7
• If you are migrating to an
IBM Spectrum Scale (GPFS) storage resource,
you can use General Atomics’ EasyHSM to create
HSM stubs to the old storage system
• Creating the HSM stubs
can be done fast
– A few hours for 100M files
• Users see all the files
in the new storage immediately
– Data is transparently (reverse) migrated
into GPFS when needed
Using EasyHSM on IBM Spectrum Scale (GPFS)
Storage System to Decommission
EasyHSM +
Nirvana
Also known as recall
in HSM terminology

8. 8
• The old storage system is scanned once
– Direct user access should be prohibited at this point
• HSM stubs are created in target GPFS system
– Users can start using the GPFS system
– They should not notice any difference
• When a user reads the content of a file,
data is recalled from old storage system into GPFS
– Transparently, through EasyHSM, using
GPFS-native HSM capabilities
• Sysadmins can schedule also explicit recalls
– Possibly during low-activity hours
• Once data in GPFS, copy in old storage not needed
– EasyHSM provides tool for automatic deletion
In more detail

9. 9
• The old storage system is scanned once
– Direct user access should be prohibited at this point
• HSM stubs are created in target GPFS system
– Users can start using the GPFS system
– They should not notice any difference
• When a user reads the content of a file,
data is recalled from old storage system into GPFS
– Transparently, through EasyHSM, using
GPFS-native HSM capabilities
• Sysadmins can schedule also explicit recalls
– Possibly during low-activity hours
• Once data in GPFS, copy in old storage not needed
– EasyHSM provides tool for automatic deletion
In more detail
Can easily be completed
over a week-end

10. 10
• The old storage system is scanned once
– Direct user access should be prohibited at this point
• HSM stubs are created in target GPFS system
– Users can start using the GPFS system
– They should not notice any difference
• When a user reads the content of a file,
data is recalled from old storage system into GPFS
– Transparently, through EasyHSM, using
GPFS-native HSM capabilities
• Sysadmins can schedule also explicit recalls
– Possibly during low-activity hours
• Once data in GPFS, copy in old storage not needed
– EasyHSM provides tool for automatic deletion
In more detail
Data migration can
span months, without
any major disruption
to business processes

11. 11
• Nirvana and EasyHSM are both developed and
maintained by General Atomics
– Nirvana has been around for a long time
• See Wikipedia page for more details
https://en.wikipedia.org/wiki/Nirvana_(software)
– EasyHSM has been released in Feb 2017
• Employ a subscription licensing model
– Pay for the period you use them
– Not capacity based
(fixed price, no matter how big is your storage system)
About the products

12. 12
• If you want to try it out, fill the request form
on the Nirvana home page:
http://www.ga.com/nirvana
• If you just want to know more, feel free to
– Contact me at
Igor.Sfiligoi@ga.com
– Contact the program manager at
Robert.Murphy@ga.com
For more information