Spectra Developer Summit 2016
Event Agenda:
• Corporate updates from Spectra CEO Nathan Thompson
• BlackPearl product overview and enhancements
• Learn what tools are available to help develop a BlackPearl client
• Partner presentation detailing client development
• Learn how easy it is to integrate an existing BlackPearl client into your workflow
• Demonstration of Avid PAM and BlackPearl integration
• Question and answer with our BlackPearl Developer Program team
2. Developer Summit Agenda:
Nathan Thompson – CEO Spectra Logic
Steven Easter – BlackPearl Product Manager
Jeff Braunstein – Senior Product Manager, Client Development
Sharon Shabtai – Senior Software Engineer, Client Development
Sven Howarth-Moore – Professional Services Lead for CatDV
5. 2U BlackPearl + T50e Tape Library BlackPearl + T120 Tape Library BlackPearl + T200 Tape Library BlackPearl + T380 Tape Library BlackPearl + T680 Tape Libraries
80TB Raw Disk Storage 160TB Raw Disk Storage 928TB Raw Disk Storage 1,696TB Raw Disk Storage 6.1PB+ Raw Disk Storage
300TB Uncompressed Tape Storage 720TB Uncompressed Tape Storage 1,200TB Uncompressed Tape Storage 2,280TB Uncompressed Tape Storage 4,020TB Uncompressed Tape Storage
Multi–RackFull RackHalf RackQuarter Rack
Spectra’s BlackPearl S3 Gateway To Tape
Eighth Rack
6. System Configuration:
BlackPearl 4U Deep Storage
Gateway
• 6.1PB Raw Capacity Cache
5 Frame TFinity Tape Library
5,590 LTO Slots
• 13.9PB Uncompressed Capacity
using LTO-6 tape technology
• 33.5PB Uncompressed Capacity
using LTO-7 tape technology
4,239 TS Enterprise Slots
• 42.4PB Uncompressed Capacity
using TS1150 tape technology
BlackPearl™ ExaScale Edition TFinity™
7. Exploring The New World Of Storage
• At the last conference event I spoke about two upcoming publications
– Spectra’s Storage Forecast which you can download from the Spectra Logic
web site for free (http://spectra.cc/Igh)
– The Book “Society’s Genome”
• Both are now published
– I will tell you a few of our findings
– Jeff can send you an electronic copy of the book
8. Major Change In Tape Roadmap
• Tape is in a position to more rapidly deliver declining cost per bit over
the next 10 years
– Tape has plenty of potential for capacity improvement to reach at least four or
five more doublings of current capacity, entirely due to surface area and
Tunneling Magnetoresistive (TMR) heads
– Fujifilm and IBM have demonstrated BaFe media has been proven to reach
220TB or more / cartridge
– IBM and HP have partnered on the same head design at LTO-7 and beyond which
gives them 98% market share (at LTO-7 and beyond) with plans to double every
two years
• This gives users confidence in a roadmap that can span for at least four more generations of
drives
– Nearly all of the large cloud companies see a significant role for tape
9. Here Is The Economic Opportunity For Tape
• With or without HAMR, disk prices will level out at
$300 for capacity 3.5” disk drives, for 16TB, or
about 2 cents per Gigabyte (raw) in 2019 and will
increase with inflation from there
• Assuming constant media price and the industry
stays true to a two year improvement cycle
– LTO-7 is about 1.6c per Gigabyte now and will reach
~1c in 2017
– LTO-8 will .8c per Gigabyte in 2018
– LTO-9 tape will be .4c per Gigabyte in 2020
– LTO-10 in ~2022 will be .2c per Gigabyte
10. What Does This Mean For Large BlackPearl/Tape Opportunities?
• Ultra cost effective solutions can be built
• Declining cost point into the future
• Let’s look at some examples:
11. 10 PB Archive System (no compression)
5 PB Nearline Copy + 5 PB Ejected Copy of Data
Systems Configured with 12 LTO Drives
7x24 Support adds 6%-9% to total US Price, annually
12. 100 PB Archive System (no compression)
50 PB Nearline Copy + 50 PB Ejected Copy of Data
Systems Configured with 48 LTO Tape Drives
7x24 Support adds 6%-9% to total US Price, annually
13. 1000 PB / 1 Exabyte System (no compression)
500 PB Nearline Copy + 500 PB Ejected Copy of Data
Systems Configured with 144 LTO Tape Drives
7x24 Support adds 6%-9% to total US Price, annually
14. Applications Written To BlackPearl Can Take Advantage Of All
Storage Types
• Tape in libraries
– LTO
– IBM TS
• Ejected tape
• SAS disk
• Power-Down SMR SATA
disk
• Public Cloud (late
‘16/early ‘17)
• Future storage media
types
17. What Makes BlackPearl Revolutionary
BlackPearl – Is a living archive, preserving digital data
The most data durability available
Highly efficient tape writing
Replication Targets now available
18. Two Types of Replication Target
BlackPearl to BlackPearl replication
BlackPearl to 3rd Party (public) Cloud
19. BlackPearl Replication
• Provide two copies of data – two locations
• Data is more ‘available’,
• if one site is down your app can point to the other BlackPearl to GET data
• Solution then adds some HA capability
• More flexible platform to fit more customer needs
20. BlackPearl Replication Example
Process to setup bi-directional replication
1. At each site: Setup other BlackPearl Replication target
• Use Data Port for replication
• Set default read preference
• Optional – Enable User Replication (access control replication)
2. At each site: Add BlackPearl Target to Data Policy
• Different policy for each BP:
• HQ system has a Data Policy for: One Copy on a tape Storage Domain and One Replication
Target
• Colo system has a Data Policy for: One Copy on a ArcticBlue Storage Domain and One
Replication Target (pointing back at HQ system)
3. Create a Bucket using these Data Policies
• Either a new PUT Job will use default Policies for the user and replication
target, or
• Best to create the same bucket on both systems first (specify the Data Policy
created in step 2), before a PUT job is done
Tape LibrarySite – Primary – “HQ”
Site – Secondary – “Colo”
21. BlackPearl Replication Example (cont.)
Data flow in bi-directional replication
A. User does A Bulk PUT into Bucket “EditingArchive” at HQ BlackPearl
i. Creates a S3 Job – e.g. Job 1 on HQ BlackPearl, and Replication Job 1 on Colo BlackPearl
ii. Both databases are synced with Replicated Job 1 at time of Job 1 creation, that way the same data cannot be sent to other BlackPearl
B. The Application sends data to HQ BlackPearl
i. Data lands in HQ BlackPearl cache, stays here until the Job 1 data has been persisted to all Storage Domains
ii. For Example we’ll say the data is replicated to the Colo BlackPearl first,
iii. Colo BlackPearl writes it to ArcticBlue quickly
• Replication Job 1 is now complete. Data in Colo BlackPearl cache can be reclaimed when needed
iv. Tape Drive resources become available at HQ BlackPearl and the Data is now written to Tape
• Job 1 is now complete. Data in HQ BlackPearl can now be reclaimed. Host application can remove any file locks for deleting data from primary
storage.
Site – Primary – “HQ"
Cache:
Tape Storage Domain:
Site – Secondary – “Colo”
Cache:
ArcticBlue Storage Domain:
EditingArchive
Data Policy:
Tape+Replication
EditingArchive
Data Policy:
ArcticBlue+Replication
Job 1
Replication Job 1
Host
User
data datadata
data
data
22. BlackPearl P Series
spectralogic.com/products/BlackPearl/specs/
• Very high performance system – up to 3GBps
• Customers will need scale-out application & client
• Or customer may have multiple different applications
• Also useful if BlackPearl makes multiple copies
23. BlackPearl Developer Updates - Agenda
• General Program Updates
• Certification Program
• SDK Updates
• End to End Data Integrity
• When Files are Safely Archived to BlackPearl
• New Deep Storage Browser Demonstration
• Java Command Line Interface Demo and Updates
24. Developer Program Updates
• Mature Program
• Certification
• Simulator and Remote BlackPearl
• Spectra BlackPearl Lab
• Forums
25. BlackPearl Certification Program
• Goal: Ensure Clients are Robust and Reliable
• Includes the Following Components:
• Program Overview and Timeline
• Legal Agreements
• Questionnaire
• Test Plan
• Testing Performed in Spectra Lab in Boulder
• Certified Clients Able to Participate in Joint Marketing
• Retesting Available for New Releases
• Currently have 3 Spectra staff doing certification testing
• Goal: Ensure Clients are Robust and Reliable
• Includes the Following Components:
• Program Overview and Timeline
• Questionnaire
• Test Plan
• Testing Performed in Spectra Lab in Boulder
• Certified Clients Able to Participate in Joint Marketing
• Retesting Available for New Releases
• Currently have 2 dedicated staff doing certification testing
26. Software Development Kit (SDK) Updates
• Four Complete SDKs
• Java
• C#/.NET
• Python
• C
• Java and C#/.NET Include “Helper” Functions
• All SDKs Can Now Access All 250+ BlackPearl APIs
27. End-To-End Data Integrity
• BlackPearl includes comprehensive data integrity once data
received
• Developers to be concerned with “in flight” data integrity
between client and BlackPearl
• Client can use checksums to guarantee integrity
• Checksums done on each file or file parts sent to BlackPearl
• HTTPS/SSL is a good optional checksum data integrity method
28. When Are Files Safely Archived to BlackPearl?
• Clients archive groups of files to BlackPearl in “jobs”
• Job is “Complete” when files are on all storage targets
• BlackPearl’s database tracks the location of files on storage
targets
• This database is typically backed up daily
• Files are not truly, safely archived until database backup with their
location has been made
• Clients should not purge/delete files off of their system until
safely archived to BlackPearl
31. The .Net SDK and Demonstration of Avid PAM plugin
Sharon Shabtai - Senior Software Engineer
32. Table Of Contents
• Spectra Logic .Net SDK
• Helper functions
• The Spectra S3 Client
• Instantiating the API example
• NuGet Package
• AVID Plugin
33. Spectra Logic .Net SDK
• Open source (GitHub) can be imported to VS and modify to you needs
• Using .Net 4.5.1
• Latest SDK release version v3.2.7
• The SDK consists of two levels of abstraction:
1. A high level interface (Ds3.Helpers.IDs3ClientHelpers)
2. The core client interface (Ds3.IDs3Client)
34. Helper Functions
• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
35. Helper Functions
• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
36. Helper Functions
• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
o EnsureBucketExists
37. Helper Functions
• A high level interface that abstracts several very common application requirements.
• Constantly working on adding and improving the API:
o StartWriteJob
o StartReadJob
o StartReadAllJob
o StartPartialReadJob
o ListObjects - with and without prefix
o EnsureBucketExists
o RecoverWriteJob
o RecoverReadJob
38. Helper Functions
• The Helper functions support:
o Parallel Requests
o Using the right Chunking strategy:
Random access
Streaming
o Metadata
o Checksum
o Cancellation token
o Job Events (“Data transferred”, “Item completed”, “On failure”)
39. The Spectra S3 Client
• The Spectra S3 client implements all of the BlackPearl REST API and can be use when the desire
behavior is not implemented in the helper function.
• You can find the full API in our IDs3Client interface
40. Instantiating the API example
The example below shows how to configure and instantiate IDs3Client and IDs3ClientHelpers.
using Ds3;
using Ds3.Helpers;
using System.Configuration;
namespace YourApplication
{
class YourClass
{
public void YourMethod()
{
// Configure and build the core client.
IDs3Client client = new Ds3Builder(
"http://ds3-endpoint",
new Credentials("access key", "secret key")
).Build();
// Set up the high-level abstractions.
IDs3ClientHelpers helpers = new Ds3ClientHelpers(client);
// Use functionality from 'helpers' and 'client', preferring 'helpers'.
}
}
}
• More detailed examples can be found in our Examples Project
41. NuGet Package
• The SDK is distributed as a NuGet package for .Net 4.5.1 and above.
• How to get the NuGet package:
1. Download the .nupkg file from the Releases page
2. Use Spectra Logic Bintray NuGet repository
42. • Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o We Created a plugin for BlackPearl
43. • Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o Created a plugin for BlackPearl
44. • Avid Technology, Inc.
• Avid 3rd party plugin API
o Archive
o Restore (including partial restore)
o Delete
o Pause and Resume
• Avid and Spectra Logic
o Created a plugin for BlackPearl
I’d like to highlight an Overview of Why BlackPearl
And enhancements to the storage platform over the last year.
3rd time here. I have some controversial views which I’m going to share with you today. I can assure my partners in the audience that I am only using public information that did not come to me under NDA
Spectra’s Headquarters has always been in Boulder, CO. I have a friend to does Drone photography and he shot this over labor day.. Spectra builds a variety of disk, tape and interface products. We use IBM tape drives, Seagate SMR disk drives and HGST Helium drives to build the world’s largest storage system.
Since I was with many of you last year, Spectra moved much more into solutions, has done a lot with our BlackPearl and TFinity tape libraries. We have a range of BlackPearl S3 Gateways with 10GigE, 40GigE and back ends ranging from SAS disk, PowerDown SATA Disk, LTO tape, TS Tape and Public Cloud. BlackPearl takes a lot of the complexity out of tape—it makes multiple copies, it helps with off-siting, it migrates to future media. With tape it is S3 on the front-end, and LTFS format on the backend.
We have also released a new tape library—the ExaScale Edition TFinity which can contain up to 1Exabyte of storage
With both disk and tape the overall system cost is 3-5 times higher once you power supplies, interconnect, sheet metal, robotics, etc.
BlackPearl enables users to create massive storage applications using Spectra’s SDK’s. With our SDK’s or partner’s storage applications can be rapidly created for everything from police video, to seismic, to media and entertainment, to scientific, to genomics
In the end, it doesn’t matter. Well written BlackPearl clients can leverage any media type
I’d like to highlight an Overview of Why BlackPearl
And enhancements to the storage platform over the last year.
Why BP? Why does customer want or need a system like BP?
Living Archive to preserve digital data. The data must carry on, migrate, never change (checksum), be highly resilient (safe storage – tape or triple parity).
Next developer conference, we’ll be talking about BlackPearl’s ability to migrate data.
Durability is ability to not lose data. One copy is not as durable as two, etc.
Using Disk and Tape provides most protection, data is written with different device drivers.
Even TS1150 and TLO provide extremely high durability.
From a high level data is managed as one object, but on the backend, BlackPearl manages each copy individually (via Storage Domain).
BlackPearl P Series really takes advantage of Spectra S3 - BULK PUT. With the ability to have over 7GBps throughput in our cache (Read+Write) (sustaining 3GBps to tape, on a single server)
BlackPearl Replication is two distinct BPs, each has it’s own data policy. i.e. one copy on ArcitcBlue for 2 years, one copy tape, and one replication target.
Public Cloud support (e.g. Data on Glacier) is a great option for DR.
Implemented at a Data Policy level.
Have different read preference, i.e. last resort or never.
One way replication, with a permanent local copy required.
If customer has Cloud mandate.
Replication provides more flexibility.
It is pretty common that DR or data spread across geography is a way to prevail after a local disaster.
Pretty common use case is Enterprise Tape in one location, and LTO Tape in another.
Many Different Read Preferences for different use cases.
In this example the AB is in the co-location (“colo”) because the customer does not have personnel onsite to fix robot. However they do have great Ethernet connection with very low latency – at Primary Location when setting up Replication Target for colo they set Default Read Preference to “After Nearline Pool” meaning a GET Job at HQ BlackPearl will get data from the replication BlackPearl if the data is not in cache and before it gets it from tape.
Details on steps:
A. Bucket is a logic location – The Bucket Name and BlackPearl IP Address are the only things specified to create the Job, and location to send the data to.
A. i. The Replication Job 1 is needed, this prevents the user or application form trying to send duplicate data to the BlackPearl Colo.
A. ii. BlackPearl needs constant replication link, that way both BlackPearls stay in sync for Jobs being processes
B. Data in this example is a specific set of data. Like a project, or video asset, or a directory of files. Each green Data represents the same exact data in different physical locations (5 physical locations: Host System, HQ BP Cache, HQ BP Tape Cartridge(s), Colo BP Cache, and Colo BP AB).
B. i. Data stays in cache until all persisted copies are made Tape and AB in this example
B. ii. iii. and iv. These steps can happen in a different order, or in parallel. This is just one example of how data is persisted.
Sustain 3GBps (6 total). Almost 11TB per hour to tape.
However may not be able to get 3GBps into BlackPearl.
P Series is still useful. If customer needs to make multiple copies (two local copies to tape, one to ArcticBlue) they will need P Series to average 1GBps.
BlackPearl S Series system may only average 300MBps with 3 copies.
Mention what we are testing – network outage, large files, lots of files, cache full, etc. If you’re client is certified, you are going to have a good experience. For building their own clients, they can still use this program.
