1.
Cost, Risk, Loss and other fun things Screening the Future 15 March 2011 Matthew Addis, IT Innovation Centre [email_address]

2.
THEMES

3.
Themes: cost of compromise Costs Opportunities Risks

4.
Themes: it never ends

5.
Themes: storage and online access <ul><li>Access is the ‘steam engine’ of preservation </li></ul><ul><li>Storage is needed whatever you do </li></ul>

6.
COSTS

7.
Costs <ul><li>All preservation activities have a cost </li></ul><ul><li>No ‘one size fits all’ </li></ul><ul><li>No single answer to ‘what will it cost’? </li></ul>&quot; Preservation is the totality of the steps necessary to ensure the permanent access ibility – forever - of an audiovisual document with the maximum integrity &quot;. (CCAAA)

8.
British Library: LIFE cost model L­ T Total cost Aq Acquisition cost I Ingest cost M metadata cost Ac Access cost S Storage cost P Preservation cost. The subscript T means that costs have to be calculated over the lifetime of the items being preserved.

9.
<ul><li>Detailed breakdown into functional areas </li></ul><ul><li>Examples </li></ul><ul><li>Spreadsheets </li></ul><ul><li>Guideline </li></ul>

10.
NASA Cost Estimation Toolkit <ul><li>Spreadsheet </li></ul><ul><li>year-by-year staff effort </li></ul><ul><ul><li>ingest </li></ul></ul><ul><ul><li>processing </li></ul></ul><ul><ul><li>documentation </li></ul></ul><ul><ul><li>archive </li></ul></ul><ul><ul><li>access and distribution </li></ul></ul><ul><ul><li>user support </li></ul></ul><ul><ul><li>facility /infrastructure </li></ul></ul>

11.
Some ‘numbers’ for AV <ul><li>Preserving Digital Public Television </li></ul><ul><ul><li>Strategies for Sustainable Preservation </li></ul></ul><ul><li>Blue Ribbon Task Force </li></ul><ul><ul><li>Long term access to digital content </li></ul></ul><ul><li>Sun </li></ul><ul><ul><li>Archiving Movies in a Digital World </li></ul></ul><ul><li>Academy of Motion Pictures Arts and Sciences </li></ul><ul><ul><li>The Digital Dilemma </li></ul></ul><ul><li>JISC </li></ul><ul><ul><li>Understanding the Costs of Digitisation </li></ul></ul>

12.
COSTS: STORAGE AND ONLINE ACCESS

13.
Trend: increasing storage capacity <ul><li>Doubles every 18 months </li></ul><ul><li>100 times every decade </li></ul><ul><li>1 million times every 30 years </li></ul>

14.
Trend: increasing recording density http://www.americanscientist.org/issues/pub/2010/3/avoiding-a-digital-dark-age

15.
Trend: storage cost improvement <ul><li>http://www.mattscomputertrends.com/harddrives.html </li></ul>

16.
TCO of storage (SDSC)

17.
TCO of storage + processing (Google)

19.
Amazon S3 <ul><li>£1000 per TB per year </li></ul>

20.
Access costs <ul><li>Distribution costs are high for frequently used content </li></ul><ul><li>Amazon S3 </li></ul><ul><ul><li>$0.10 per GB per month storage </li></ul></ul><ul><ul><li>$0.10 per GB transfer </li></ul></ul><ul><li>INA public access </li></ul><ul><ul><li>1.5 million accesses from 100,000 items in 1 month </li></ul></ul><ul><ul><li>Network costs are 15x storage (Amazon rates) </li></ul></ul><ul><li>BBC iPlayer </li></ul><ul><ul><li>Web distribution estimated at 1500x storage cost </li></ul></ul><ul><li>On the other hand.. </li></ul><ul><ul><li>20% of BBC archive accessed each year </li></ul></ul>

21.
TCO over time: migration <ul><li>Encoding formats </li></ul><ul><li>Media formats </li></ul><ul><li>Storage hardware </li></ul><ul><li>Operating systems </li></ul><ul><li>Management software </li></ul><ul><li>Networking </li></ul>

22.
TCO over time: automation

23.
TCO over time: falls more slowly <ul><li>SDSC </li></ul><ul><ul><li>$1500 per TB per year on disk in early 2007 </li></ul></ul><ul><ul><li>$1000 in 2008 </li></ul></ul><ul><ul><li>$650 as of the end of 2009 </li></ul></ul><ul><li>Amazon </li></ul><ul><ul><li>$1800 per TB per year in early 2007 </li></ul></ul><ul><ul><li>$1260 per TB per year end 2009 (over 500TB) </li></ul></ul><ul><ul><li>$950 per TB per year end 2010 (over 500TB) </li></ul></ul><ul><li>Annual storage costs halve every 2-3 years </li></ul>

24.
Implications: ‘Forever’ costs of storage <ul><li>‘ Endowment’ model of sustaining content </li></ul><ul><li>Needs never ending growth – don’t turn off the tap! </li></ul>Half life of annual cost Multiplier for ‘forever’ cost 1 2 2 3.3 3 4.9 5 7.7 10 14.9

25.
Implications: outsourced archive hosting <ul><li>Data Centre and Access costs become dominant </li></ul><ul><li>Economies of scale drives cost per down </li></ul><ul><li>Location will mater (operations and access) </li></ul><ul><li>Data and system management at scale is a skilled job </li></ul><ul><li>For many, outsourced hosting will become the only economically viable option in the long-term </li></ul><ul><li>But this needs a new breed of ‘trusted cloud’ Service Providers…. </li></ul>

26.
RISKS

27.
Risks <ul><li>Top down risk assessment process </li></ul><ul><li>Context, objectives, policies </li></ul><ul><li>Activities, assets, owners </li></ul><ul><li>Risks, treatment, management </li></ul>

30.
Lots of risks <ul><li>Technical obsolescence, e.g. formats and players </li></ul><ul><li>Hardware failures, e.g. digital storage systems </li></ul><ul><li>Loss of staff, e.g. skilled transfer operators </li></ul><ul><li>Insufficient budget, e.g. digitisation too expensive </li></ul><ul><li>Accidental loss, e.g. human error during QC </li></ul><ul><li>Stakeholders, e.g. preservation no longer a priority </li></ul><ul><li>Underestimation of resources or effort </li></ul><ul><li>Fire, flood, meteors, aliens… </li></ul>

31.
Files: 37 risks from ‘IT’ <ul><li>Risks of loss of data authenticity and integrity </li></ul><ul><ul><li>Loss of ability to track and record what’s been done </li></ul></ul><ul><ul><li>Changes to integrity or authenticity go unnoticed. </li></ul></ul><ul><li>Risks of data destruction or degradation </li></ul><ul><ul><li>Loss or corruption of data </li></ul></ul><ul><ul><li>People: deliberate or accidental damage </li></ul></ul><ul><ul><li>Technology: bit rot, obsolescence </li></ul></ul><ul><li>Risks to data through loss of services </li></ul><ul><ul><li>E.g. loss of routine integrity checks </li></ul></ul><ul><ul><li>Loss or pressure on resources used to do preservation </li></ul></ul><ul><li>Risks to through mismatch of expectations </li></ul><ul><ul><li>Service providers don’t meet archive needs </li></ul></ul>

32.
Example risks Risk ID Title Example R30 Hardware Failure A storage system corrupts files (bit rot) or loses data due to component failures (e.g. hard drives). R31 Software Failure A software upgrade to the system looses or corrupts the index used to locate files. R32 Systems fail to meet archive needs The system can ’ t cope with the data volumes and the backups fail. R33 Obsolescence of hardware or software A manufacturer stops support for a tape drive and there is insufficient head life left in existing drives owned by the archive to allow migration R34 Media degradation or obsolescence The BluRay optical discs used to store XDCAM files develop data loss. R35-R38 Security Insufficient security measures allow unauthorised access that results undetected modification of files.

33.
Loss of data authenticity and integrity (origins) <ul><li>Lack of, or failure to follow, proper process </li></ul><ul><li>Failure to record all actions performed within the archive </li></ul><ul><li>Failure of archive storage systems or processing of content </li></ul><ul><li>Failure to record attempts (deliberate or otherwise) to breach systems </li></ul><ul><li>Failures at remote storage service providers </li></ul><ul><li>Deliberate attack by disgruntled employees </li></ul><ul><li>Deliberate attack by hackers or other third-parties </li></ul><ul><li>Failure of preservation systems to correctly apply preservation actions </li></ul>

34.
Loss of data authenticity and integrity (things at risk) <ul><li>Audiovisual content </li></ul><ul><li>Descriptive Metadata </li></ul><ul><li>Contracts, agreements, audit trail </li></ul>

35.
Loss of data authenticity and integrity (consequences) <ul><li>Loss of reputation </li></ul><ul><li>Financial penalties (service provider) </li></ul><ul><li>Extra time and resources needed to fix it again </li></ul><ul><li>Loss of ability to use content (customer) </li></ul><ul><li>Failure to record details of transactions with consequent denial by customer or service provider that they have agreed obligations </li></ul>

36.
Loss of data authenticity and integrity (counter measures) <ul><li>Enforce authentication and access control so only trusted individuals have ability to manipulate assets (both within and external to the organisation) </li></ul><ul><li>Record all actions to content that take place (who did what and when) to create a complete audit trail </li></ul><ul><li>Digital signatures (e.g. hashing) and integrity monitoring to detect changes in digital content, both within storage systems and in transit over networks </li></ul><ul><li>Log any attempted breaches , deliberate or accidental, and whether they were successful or not to allow security effectiveness to be measured. </li></ul><ul><li>Regular security audits of technology, processes, staff skills etc. </li></ul><ul><li>Evaluate and take into account any increased risk from using data encryption in storage systems as a potential degradation amplifier. </li></ul><ul><li>Use appropriate integrity assurance processes that match the frequency, timescales and severity of the ways in which integrity could be lost </li></ul><ul><li>Ensure integrity records (e.g. checksums or signatures) are kept safe and are themselves subject to integrity control </li></ul><ul><li>Ensure integrity control is comprehensive and consistent , i.e. applied to all forms of data (metadata, identifiers, checksums, logs, credentials, audiovisual content) </li></ul>

38.
RISKS: STORAGE AND ONLINE ACCESS

39.
Trend: obsolescence <ul><li>Each change in ‘technology’ is 1000 times denser </li></ul><ul><li>But the media lasts 0.1 times as long </li></ul>Medium Storage Density bits/cm² Life, years Stone 10 10000 Paper 10 4 1000 Film 10 7 100 Disc 10 10 10

40.
Data tape (LTO) 6 years 2 years Ultrium LTO roadmap

41.
Data tape <ul><li>Relatively ‘safe’ technology (compared to HDD) </li></ul><ul><li>Typical ‘problem rates’ are 0.1 – 1% of tapes </li></ul><ul><li>Most problems from data tape come from drives </li></ul><ul><ul><li>Malfunctioning or worn drives that damage tapes </li></ul></ul><ul><ul><li>New drives that don’t handle older generations properly </li></ul></ul><ul><li>Field studies show data rarely lost where multiple copies have been made and integrity checked </li></ul>

42.
Eggs in one basket <ul><li>Today: 1000hrs of video </li></ul><ul><ul><ul><li>1000 tapes LTO2 (200GB), 2 copies </li></ul></ul></ul><ul><ul><ul><li>Need to migrate and 1% of tapes problematic </li></ul></ul></ul><ul><ul><ul><li>90% no data lost, 10% chance 1 hr lost, 1% chance 2hrs lost </li></ul></ul></ul><ul><ul><ul><li>Practically zero chance of all data being lost </li></ul></ul></ul><ul><li>10 years later </li></ul><ul><ul><ul><li>Tapes hold 30hrs each, 2 copies </li></ul></ul></ul><ul><ul><ul><li>Need to migrate 1% of tapes problematic </li></ul></ul></ul><ul><ul><ul><li>0.3% chance of loosing 30 hrs in one go </li></ul></ul></ul><ul><li>Another 10 years later </li></ul><ul><ul><ul><li>One tape holds 1000hrs, 2 copies </li></ul></ul></ul><ul><ul><ul><li>0.01% chance of loosing everything – all or nothing </li></ul></ul></ul>

43.
HDD error rates <ul><li>1000 times more HDD capacity over last 15 years </li></ul><ul><li>Only 10 times lower Bit Error Rates (BER) </li></ul><ul><li>HDD BER = 10 -14 </li></ul><ul><li>1 TB = 10 13 bits </li></ul><ul><li>10% chance of an error when reading all of a HDD </li></ul><ul><li>Within a few years, more likely than not to get a read error when copying a HDD </li></ul>

44.
HDD lifetime <ul><li>Manufacturers say: </li></ul><ul><li>‘ Mean Time To Failure’ = 1 million hours </li></ul><ul><li>What does a MTTF of 1,000,000 hrs mean? </li></ul><ul><li>What is does not mean: </li></ul><ul><ul><li>A HDD will typically last 100 years </li></ul></ul><ul><ul><li>Or, the failure rate is 1% each year </li></ul></ul><ul><li>Lifetime of a HDD is 3-5 years </li></ul>

45.
HDD failure rates <ul><li>Google study of Annual Failure Rates in HDD servers </li></ul>

46.
The IT Industry knows this already

47.
But systems bring their own problems <ul><li>“ Disk failures are not always a dominant factor of storage subsystem failures, and a reliability study for storage subsystems cannot only focus on disk failures. Resilient mechanisms should target all failure types” </li></ul><ul><li>2008 NetApp study of 1.8M HDD in 155,000 systems </li></ul>

48.
‘ bit rot’ <ul><li>Errors can be silent (latent) </li></ul><ul><ul><li>Permanent and undetected corruption of data </li></ul></ul><ul><ul><li>Deeply worrying for archives </li></ul></ul><ul><ul><li>Seen in field studies (if you know how to look) </li></ul></ul>2007 study into data corruption by CERN David Rosenthal’s blog http://blog.dshr.org/

49.
Cost of reducing chance of loss <ul><li>Storage capacity increasing very quickly </li></ul><ul><li>Storage speed and error rates not keeping pace </li></ul><ul><li>Increasingly complex measures needed </li></ul><ul><li>Disproportionate time and cost needed to manage integrity </li></ul>

50.
Cost of not reducing loss (1) JPEG2000 with one error per 100KB Volker Heydegger study on file format sensitivity to corrupton <ul><li>Compression = Corruption amplifier </li></ul><ul><ul><li>Corrupting 0.001% of encoded image results in 30% of pixels affected in decoded image </li></ul></ul>

51.
COST OF RISK: STORAGE AND ONLINE ACCESS

52.
Storage: cost of risk of loss <ul><li>Which storage technologies should I use? </li></ul><ul><li>How many copies to make, where to put them? </li></ul><ul><li>How often to check them, how to repair them? </li></ul><ul><li>Cost </li></ul><ul><li>Safety (risk of loss) </li></ul><ul><li>Accessibility </li></ul><ul><li>Retention time </li></ul>

53.
Comparing ‘cost of risk of loss’ <ul><li>Multiple independent copies </li></ul><ul><li>Detection and correction of failures </li></ul><ul><li>Migration to address obsolescence </li></ul><ul><li>All activities have a cost, including access </li></ul>

54.
Approaches <ul><li>Use longer lived storage technology </li></ul><ul><ul><li>E.g. Printing bits to film </li></ul></ul><ul><li>Use more reliable storage technology </li></ul><ul><ul><li>E.g. data tape instead of HDD on shelves </li></ul></ul><ul><li>Make more copies </li></ul><ul><ul><li>E.g. off site deep archiving </li></ul></ul><ul><li>Encode so content is more resilient </li></ul><ul><ul><li>E.g. Graceful degradation </li></ul></ul><ul><li>Use concealment </li></ul><ul><ul><li>E.g. Interpolation to replace corrupted frames or blocks </li></ul></ul><ul><li>Check often and fix quickly </li></ul><ul><ul><li>E.g. scrubbing of HDD servers </li></ul></ul>

55.
Detailed Comparison <ul><li>From data tape to </li></ul>

56.
Detailed Comparison <ul><li>From data tape to </li></ul>

57.
Tipping points <ul><li>All on tape (2 copies), hard disk only for staging </li></ul><ul><li>Frequently used on hard disk, two copies tape </li></ul><ul><li>All on hard disk (1 copy), safety copy on tape </li></ul><ul><li>All on hard disk (2 or more copies) </li></ul><ul><li>All on flash (2 copies, e.g. USB sticks) </li></ul>Increasing archive size

58.
TOOLS

59.
Two tools <ul><li>Long term planning </li></ul><ul><ul><li>25 years </li></ul></ul><ul><ul><li>High level choices </li></ul></ul><ul><ul><li>Estimates of total cost and loss </li></ul></ul><ul><ul><li>Narrow down the options </li></ul></ul><ul><li>Short to medium term simulation </li></ul><ul><ul><li>Simulates actual events </li></ul></ul><ul><ul><li>Corruption, loss, catastrophes </li></ul></ul><ul><ul><li>Ingest, access, ‘active preservation’ </li></ul></ul><ul><ul><li>Impact of limited resources </li></ul></ul>

60.
Challenges <ul><li>Hard to get input data </li></ul><ul><ul><li>Costs for storage and access </li></ul></ul><ul><ul><li>Failure modes and frequencies </li></ul></ul><ul><li>Diverse range of storage models </li></ul><ul><ul><li>Data tapes on shelves </li></ul></ul><ul><ul><li>HDD in servers </li></ul></ul><ul><ul><li>Storage as a Service </li></ul></ul><ul><ul><li>Manual operation, machine automation </li></ul></ul><ul><li>Simplicity v.s. accuracy v.s. longevity </li></ul><ul><ul><li>Model the whole world and be instantly out of date! </li></ul></ul>

61.
Approach <ul><li>Storage and access costs </li></ul><ul><ul><li>Annual costs </li></ul></ul><ul><ul><li>Trends </li></ul></ul><ul><li>Failures and loss </li></ul><ul><ul><li>Latent, Access </li></ul></ul><ul><ul><li>Human, Machine </li></ul></ul><ul><ul><li>Focus on files </li></ul></ul><ul><li>Best practice </li></ul><ul><ul><li>Fixity checks (read, write, scrubbing, migration) </li></ul></ul><ul><ul><li>Careful selection of media and systems </li></ul></ul><ul><ul><li>Monitoring and reaction to extant errors </li></ul></ul>

62.
Examples Data tape on shelves HDD in servers Storage as a Service Storage Cost Low (media, shelves, climate control) High (servers, power, cooling, maintenance) High (fully managed service) Access Cost High (people retrieve and load media) Low (internal network, automated) High (bandwidth, charges for i/o) Latent Failures Low (data tape is reliable) Med (‘bit rot’) Low (replication and monitoring) Access Failures Medium (drives eat tapes) Low/Medium (depends on system) Low (automated checks)

67.
Long term loss

68.
Long term cost

69.
Main sources of risk

71.
Use of resources <ul><li>Ingest, access, migration, scrubbing </li></ul><ul><ul><li>Use resources </li></ul></ul><ul><ul><li>Take time </li></ul></ul><ul><ul><li>Cost money </li></ul></ul><ul><li>Resources are often limited </li></ul><ul><ul><li>People, servers, bandwidth </li></ul></ul><ul><ul><li>Contention and priorities </li></ul></ul><ul><li>Capacity planning, Disaster simulation, Training </li></ul>

79.
Can I use the tools? <ul><li>Yes </li></ul><ul><li>http://prestoprime.it-innovation.soton.ac.uk </li></ul><ul><li>‘ Beta quality’ </li></ul><ul><li>Long-term planning tool already available </li></ul><ul><li>Simulation tool within the next few weeks </li></ul><ul><li>We welcome suggestions for making them better </li></ul><ul><li>Next steps: </li></ul><ul><ul><li>Cost, quality, throughput in transfer chains </li></ul></ul>

80.
SERVICE MANAGEMENT

81.
Making the plan happen <ul><li>Why plan if you can’t control what happens in practice? </li></ul><ul><li>Treat everything as services </li></ul><ul><li>Define SLAs </li></ul><ul><li>Measure and monitor </li></ul><ul><li>Use policies to define what actions are taken </li></ul>

82.
Trust Questionnaire <ul><li>Before you worry about SLAs etc, why would you even trust a service provider with your data? </li></ul><ul><li>Online survey asking service providers what they think is important in determining whether a user trust them with their data. </li></ul><ul><li>36 responses were received. </li></ul><ul><li>Based on TRAC: </li></ul><ul><ul><li>Asks how important aspects of governance, rights management and security are. </li></ul></ul><ul><ul><li>Presents TRAC: do they know about it? </li></ul></ul><ul><ul><li>Would an audit certificate be useful to them? </li></ul></ul><ul><ul><li>Asks about other criteria not in TRAC </li></ul></ul>

83.
Questionnaire results Governance AV Material Management Utility

84.
SLA Terms <ul><li>Ten quality of service terms proposed, e.g. </li></ul>ID Name Description Metrics Bounds Monitoring frequency QS-01 Availability The guarantee that the service will be available (up and exploitable) ME-01 The availability should never go below a specific threshold, there could be more than a threshold (e.g. for business hours and night) Once per fixed period like month or year OR on a sliding window with an appropriate width and moving with a convenient step QS-03 SIP ingestion time Total elapsed time from the SIP submission to the confirmation from the system that everything has been correctly acquired. ME-05 The SIP ingestion time should never go above a specific threshold, there could be more than a threshold (e.g. for business hours and night). It can be given as percentage, e.g. 90% of deliveries are done under a threshold 1 and the rest under threshold 2. Every time there is a SIP ingestion or periodically if percentage check is assumed on the base of precalculated statistics

85.
SLA Terms <ul><li>Four constraints proposed, e.g. </li></ul>ID Name Description Units Metrics Bounds C-02 Maximum amount of storage The maximum amount allowed to a specific customer by contract GBytes ME-04 The occupied storage should never cross a specific threshold, partial exceeding for a limited period of time could be tolerated C-03 Maximum number of simultaneous users Maximum number of users logged in at the same time Positive integer ME-14 The actual number of logged users should never cross a specific threshold, partial exceeding for a limited period of time could be tolerated

86.
SLA Terms <ul><li>In total: </li></ul><ul><ul><li>21 capabilities </li></ul></ul><ul><ul><li>12 features of interest </li></ul></ul><ul><ul><li>15 metrics </li></ul></ul><ul><ul><li>12 quality of service terms </li></ul></ul><ul><ul><li>4 constraints </li></ul></ul><ul><ul><li>6 pricing terms </li></ul></ul><ul><ul><li>7 penalty terms </li></ul></ul>

88.
Automation <ul><li>Factory approach reduces cost by 50% </li></ul><ul><li>Needs to run smoothly: 10% exceptions  25% more cost </li></ul>

89.
Management Services and Resources Ingest, Access, Quality Control, Metadata validation… Management Modelling Set policies Service Manager Resources, SLAs: Monitoring, Managing, Automation

90.
Content Producers and Consumers Suppliers e.g. 3 rd party storage Modelling Set policies QoE and QoS Workflow and Access control

96.
CONCLUSIONS

97.
Cost, Risk, Loss, Opportunity <ul><li>Long term planning </li></ul><ul><li>Day to day management </li></ul><ul><li>Making compromises </li></ul><ul><li>Managing uncertainty </li></ul><ul><li>Optimising use of resources </li></ul><ul><li>Making it work </li></ul>Costs Opportunities Risks

98.
Some reference documents from PrestoPRIME <ul><li>D2.1.1 Preservation Strategies </li></ul><ul><li>D2.1.2 Preservation Modelling Tools </li></ul><ul><li>D2.3.1 SOA for AV storage </li></ul><ul><li>D3.2.1 Threats from mass storage </li></ul><ul><li>D3.4.1 Service Level Agreements </li></ul><ul><li>D7.1.4 Annual AV preservation report </li></ul><ul><li>www.prestoprime.org </li></ul>

99.
More <ul><li>Beta versions of planning/simulation tools </li></ul><ul><ul><li>http://prestoprime.it-innovation.soton.ac.uk </li></ul></ul><ul><li>Everything else </li></ul><ul><ul><li>www.prestocentre.eu </li></ul></ul>