Who Decides? Reinterpreting archival processes for the management of digital research


Published on

Management of digital records can benefit from the contribution of digital curators and archivists. The presentation outlines the efforts of the PEKin project at King's College London to develop a management strategy that combines these disparate skillsets

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I present a case study of work being performed at King’s College London to revise its records management strategy. The presentation will cover 3 key areas: The importance of re-evaluating definitions and criteria for what constitutes a Record to an institution on a regular basis Examine challenges that may be posed when attempting to manage the lifecycle of digital record Tthe technical architecture and processes that KCL has adopted to manage digital records.
  • This slide introduces a record, describing its purpose and role within an institution. Definitions provided by the ISO Records Management specification and ICA establish a distinction between information and records While an institution will produce or maintain many types of information, not all of this information will constitute a record
  • In this slide, I highlight key terms in the definition that are useful for making a distinction between the two. Key is the relationship between the data, key stakeholders and the set of activities that are performed within the institution. The Records Management standard places an emphasis upon activities related to business transactions & legal requirements, whereas The ICA adopts a broader perspective, by considering the evidence base necessary to understand any type of activity. It is left to the institution to decide which business transactions or activities should be recorded.
  • Criteria for determining types of resource to maintain must be continuously reviewed and updated to meet with contemporary aims and objectives of the institution The criteria for determining what is a Record may evolve over time as a result of many factors, including alterations to the legal framework and changing interpretations of the business of the institution. For an academic institution, records traditionally include its operational conduct and teaching activities. In recent years, institutions have begun to recognise that other types of activity play a key role in their business and have developed broad criteria to encapsulate research and other types of information that represent the memory of institution
  • We are currently addressing many of these issues at King’s College. College data is managed a central archives service. The service currently collects business records, college heritage resources & private papers. For the most part, these resources are paper-based. Evidence of this can be seen in the preservation policy that emphasises maintaining records in their original form & storing them in appropriate environmental conditions.
  • In recent years there has been a recognition there is an increasing amount of information being created & published of archival value that are not covered by the policy and that new practices are required to manage this content. Specifically, the growth of electronic creation & publication methods presents new challenges in the archival approach that is taken. Resources are increasingly complex, in terms of their composition, e.g. hybrid archives that are composed of paper & digital, as well as the type of content that is created (e.g. CAD drawings, interactive resources). Many of these resources do not have a paper equivalent. The frequency of content creation, revision and publication and technological dependencies presents new challenges for an archivist.
  • To examine these issues in detail, we obtained JISC funding and are currently developing a management solution that can be applied to the broad range of business and research data that is being produced by the college. Our strategy combines the skills and expertise of archivists and digital curators to ensure that digital resources of archival value remain accessible and usable for the period of retention. This is achieved through a two-layered strategy: Work with academic units to understand their production workflow and identify the type of risks that may limit the long-term access. Work with central services in the college to identify gaps and areas for improvement in their management infrastructure. The outcome of the project will be the development of a digital repository for the preservation of research and business data of archival value and a over-archiving strategy for managing data within the college.
  • To understand the type of information being produced and their management requirements, we developed a bespoke framework for auditing each academic unit. The framework draws upon and combines sections of several existing methodologies, including Data Audit Framework, DRAMBORA and others We have extended these, through modification of the lifecycle model and extension of the vocabulary that was used. Although certain aspects of these frameworks were designed for a digital repository environment, they have proved useful for understanding data management issues in the wider context
  • A data audit was applied to selected departments within the college that were identified as having data of particular value or at particular risk. Wished to understand the functional activities performed, the type of information created, the reason that the data was used, who was responsible for these resources and the time period that they were stored for. The audit process was time-consuming to perform and required perserverance. However, by end of audit, we were able to produce a detailed case study for each department or service.
  • The audit identified many types of digital information being stored in each dept. These were frequently produced in-house, but also came from external sources. Each department considered their data assets to be valuable, but used different criteria for understanding value & adopted different approaches for maintaining them.
  • A second aspect of the audit work was to develop an understanding of the digital object lifecycle. Digital information may be subject to several types of lifecycle We have the record lifecycle (on left) which emphasises the creation & use of a resource, as well as an appraisal process that may result in its destruction or re-use for different purpose We also have the cycle of access, represented by the digital lifecycle, that focuses upon the access period of the storage format. An encoding format may be created by a business, who will publish it, revise it over time to perform new functionality, and at some stage, make a decision to withdraw the format from usage or make the specification available for use. The ability to access information stored in the format will often depend upon where the format is in its lifecycle. Content stored in a older format that has been subsequently revised, replaced or withdrawn may be more difficult to access and recreate.
  • As a result of the recognition that digital information has a lifecycle, we must also recognise that events may occur that reduce the likelihood that long-term access can be retained. To understand these events, we applied a ‘light touch’ version of the DRAMBORA risk assessment methodology to each case study. DRAMBORA was originally developed to analyse risks that may occur in a digital repository. For the ‘light touch’ version, we focused upon risks during the lifecycle, from creation to dissemination, as well as aspects of the staff and technical infrastructure components that affected the object. Each risk was documented, describing when it may occur, the potential consequences and one or more mitigation strategies that could be implemented.
  • As a result of the risk analysis, we identified a number of management issues. These may be classified into four categories: Storage issues were the most common. Many staff were reliant upon USB disks & various free & for-cost 3rd party systems to store their data. This presented the risk that data could be lost or corrupted at any stage. Authenticity was a concern for all staff. Several less technical staff were reliant upon printed copies because they did not trust the digital copy that had been produced. In addition, the archival value & retention requirements were unknown. Assessment of the value of digital information was intrinsic in many department’s operation, but implementation varied between staff members.
  • To mitigate the risk of loss or corruption of data during its lifecycle, we identified four key issues that we would have to address The First and perhaps most obvious issue is that a technical infrastructure must be provided to each dept. to store their data. These system should provide services to maintain authenticity and integrity Second, education is required to help staff understand data management and archival principles through provision of training events and supporting documents Third, consistent policies are required at the institutional level that indicates the type of digital information that have archival value. Finally, procedures are required to acquire, curate and preserve data in an archival environment.
  • To reduce the likelihood of many risks, we’re developing a preservation repository that will be used to manage college data of short & long-term value. For the last section of this presentation, I will examine various aspects of the repository. In abstract, it was identified that the repository should: comply with necessary standards, such as OAIS and Trusted Repositories comply with records management strategies, by providing bitstream and content preservation. It should also be able to interoperate with other institution systems and provide controlled access to resources.
  • A widely known repository specification is the OAIS Reference model, which provides a consistent method of expressing the components of a digital repository – Ingest, Data Management, Archival storage, Administration, Preservation Planning and Access. These components are responsible for processing the digital object at each stage of the workflow, changing it from a Submission Information Package into an Archival Package suitable for preservation and a Dissemination Package suitable for access & use
  • In practice, the preservation repository is composed of several technical components that enable the archive to fulfil the functions of an OAIS. The Alfresco CMS (on left of screen) is being used to provide submission interface and data processing functions. Alfresco was chosen based upon its extensive plugin support & ability to create complex content models. Data is stored temporarily in Alfresco, until a time when the collection is closed. At this stage the data is transferred into Fedora Commons for archiving. This stores all data and metadata in its various versions. To provide controlled access, we’re currently testing Muradora as a front-end and using add-ons such as Tika and Solr to provide metadata and full text searching
  • The processing workflow is expressed as a Business Processing Model (BPM). During the workflow, key activities are performed on each digital object. Alfresco actions are used to configure the type of events that occur, when they are initiated and the type of resource they are applied to. These are configured by defining various parameters. These activities can be organised into sequential or concurrent stages as required
  • The primary stage where actions are performed is on receipt of the data. Actions will include: Checking that the data structure conforms to the specified content model (mentioned in later slide) Creation of fixity values for each file Format identification, which will lead into use of format specific tools to perform technical metadata extraction. And conversion into appropriate preservation and dissemination formats Each activity will be logged & stored as PREMIS Event metadata
  • At a later stage, the data and metadata will be archived. A collection is held in Alfreso until one of two parameters are met: The collection is designated as closed by an archivist, or A specified time period has elapsed, e.g. 2 months after deposit. The parameters that are set will vary, dependent upon the broad category of data. Actions will be performed at later periods during the lifecycle. E.g. fixity checks will be performed to validate that the on-disk data is unchanged. Additional actions may be set by the archivist or depositor. For example, re-appraise content every 5 years to determine if it should continue to be stored, or to change the access status of the resource after an embargo.
  • Content models define the organisational structure of different types of collection, type of content that can be stored & behaviours of individual components. For the repository, we adopted a three layer structure for each collection type. These are organised hierarchically. A comparison may be made to a filing cabinet that contains a set of shelves, each of which may contain one or more folders
  • The diagrams on screen show examples of the content model that have been developed for committee records and research projects. A committee records is sub-divided into meetings that occurred on specific dates. Each meeting will posses an agenda, minutes of previous meeting, various papers. At a later stage, the minutes of the current meeting will also be added. Research project also have a 3 layer structure broken into year of project, which has been separated into different types of content.
  • To conclude, digital resource represents a challenge for archives to maintain, but also present opportunities for re-assessing implementation of archival practices. We have found that data audit & risk analysis provide useful frameworks for understanding the information lifecycle and justifying archiving of these resources Although digital resources created for business and research purposes perform different functions in institution, process of managing them remains broadly similar and can be automated to some degree in a shared system. Finally, I think the process has demonstrated the value of applying archival techniques to the management of digital resources. It has been invaluable in understanding the storage and management requirements of the college and identifying further areas for development.
  • Who Decides? Reinterpreting archival processes for the management of digital research

    1. 1. Who Decides? Reinterpreting archival processes for the management of digital research Gareth Knight Centre for e-Research, King’s College London
    2. 2. Presentation Themes <ul><li>Need to re-appraise definition & criteria for a Record </li></ul><ul><li>Challenges posed when attempting to archive digital records </li></ul><ul><li>Technical architecture and processes required to manage digital records </li></ul>
    3. 3. What is a record? <ul><li>“ information created, received, and maintained as evidence and information by an organization or person, in pursuance of legal obligations or in the transaction of business ” </li></ul><ul><li>ISO 15489-1:2001. Records management </li></ul><ul><li>“ a specific piece of recorded information generated, collected or received in the initiation, conduct or completion of an activity and that comprises sufficient content, context and structure to provide proof or evidence of that activity “ </li></ul><ul><li>The International Committee on Archives (ICA) Committee </li></ul>
    4. 4. What is a record? <ul><li>“ information created, received, and maintained as evidence and information by an organization or person, in pursuance of legal obligations or in the transaction of business ” </li></ul><ul><li>ISO 15489-1:2001. Records management </li></ul><ul><li>“ a specific piece of recorded information generated, collected or received in the initiation, conduct or completion of an activity and that comprises sufficient content, context and structure to provide proof or evidence of that activity “ </li></ul><ul><li>The International Committee on Archives (ICA) Committee </li></ul>
    5. 5. What is a Record today? What is a Record tomorrow? <ul><li>Criteria for defining records that have archival value is subject to range of factors and may change over time: </li></ul><ul><li>Legal obligation: </li></ul><ul><ul><li>Varies between institution of different type </li></ul></ul><ul><ul><li>Geographic location </li></ul></ul><ul><ul><li>E.g. Public Records Act 1958 (government), Data Protection Act, Freedom of Information (FoI) Act 2000, Environmental Information Regulations (2004), etc. </li></ul></ul><ul><li>Proof of business: </li></ul><ul><ul><li>What type of business? Commercial, non-commercial </li></ul></ul><ul><ul><li>Research as a business activity </li></ul></ul><ul><li>Proof of activity: </li></ul><ul><ul><li>May be broad. Influenced by function, evidential value, uniqueness </li></ul></ul><ul><li>Memory of institution </li></ul><ul><ul><li>Same as above – requires consideration of function, value, uniqueness and other criteria </li></ul></ul>
    6. 6. Re-evaluating Records management at King’s College London <ul><li>College Archives acquire, preserve and makes available all material of long-term, evidential and research interest that forms part of the College’s heritage </li></ul><ul><li>Acquisition policy: </li></ul><ul><ul><li>Paper and electronic records of day-to-day business operation, private papers of academics and researchers and material related to College heritage </li></ul></ul><ul><li>Preservation policy: </li></ul><ul><ul><li>Preserve archives in their original physical format (storage, packaging). </li></ul></ul><ul><ul><li>Maintain in appropriate environmental conditions – in c ompliance with BS5454: 2000 </li></ul></ul><ul><li>http://www.kcl.ac.uk/iss/archivespec/spcm.html </li></ul>
    7. 7. New archiving challenges <ul><li>New types of information: </li></ul><ul><ul><li>Published research papers, research data – funder requirements for data management </li></ul></ul><ul><li>Creation of increasingly diverse content types: </li></ul><ul><ul><li>Hybrid (paper+digital), digital content </li></ul></ul><ul><ul><li>CAD designs, interactive resources, datasets. Publication of dynamic content </li></ul></ul><ul><li>Update frequency: </li></ul><ul><ul><li>Web site, blogs, twitter, object revisions. Versioning </li></ul></ul><ul><li>Access lifecycle: </li></ul><ul><ul><li>Technology dependencies – software & hardware </li></ul></ul><ul><li>Uncertain value: </li></ul><ul><ul><li>What is the business value? What period of time? </li></ul></ul>
    8. 8. Preservation Exemplars at King’s (PEKin) <ul><li>Project objectives: </li></ul><ul><ul><li>Develop a management system capable of handling digital business records AND research material </li></ul></ul><ul><ul><li>Adopt a management strategy that brings together archival AND data curation approaches </li></ul></ul><ul><ul><li>Embed preservation practices within the institution through a multi-layered strategy: </li></ul></ul><ul><ul><ul><li>work with central services to develop a preservation strategy and service for digital records </li></ul></ul></ul><ul><ul><ul><li>work with academic units and professional services to ensure local data producers and systems managers are provided with targeted advice, guidance and tools to support decision-making </li></ul></ul></ul><ul><li>Project Partners: </li></ul><ul><li>Centre for e-Research (CeRch) & Archives & Information Management (AIM) at King’s College London </li></ul><ul><li>Funder: </li></ul><ul><li>JISC Information Environment 09-11 preservation exemplars strand </li></ul>
    9. 9. Audit framework PEKin audit framework combines sections of DAF, DRAMBORA, DIRKS & other audit work
    10. 10. Audit management practices <ul><li>Purpose of audit was to identify: </li></ul><ul><ul><li>Functions within the organisation that create records </li></ul></ul><ul><ul><li>Who used records and for what purpose </li></ul></ul><ul><ul><li>Location and responsibility for storage </li></ul></ul><ul><ul><li>Time period they are currently / should be retained for </li></ul></ul><ul><ul><li>Future stakeholders that need/may wish to use records </li></ul></ul><ul><li>Survey academic and business units </li></ul><ul><li>Core business units: </li></ul><ul><ul><li>College Estates, Student Records, College Committees </li></ul></ul><ul><li>Research groups: </li></ul><ul><ul><li>Twins Early Development Study, Regional Information Collection Centre, Environment Research Group, Randall Division of Cell and Molecular Biophysics </li></ul></ul>
    11. 11. Types of digital information <ul><li>Business records: </li></ul><ul><ul><li>Estates records – Property records, contract for building maintenance, Computer Aided Design (CAD) </li></ul></ul><ul><ul><li>Student Records – Students, courses, grades </li></ul></ul><ul><ul><li>Committee Records – Structure, operation </li></ul></ul><ul><li>Research records: </li></ul><ul><ul><li>Commercial – research data created for commercial purposes, e.g. pollution monitoring, patents </li></ul></ul><ul><ul><li>Funded research – Contracts awarded by funding bodies </li></ul></ul><ul><ul><li>Unfunded research – Academic researcher who has interest in topic </li></ul></ul><ul><li>Each record has different value and retention period. </li></ul>
    12. 12. Many types of lifecycle Record lifecycle (variants: Information, data lifecycle) Access lifecycle (e.g. digital lifecycle)
    13. 13. Analysis of lifecycle risks <ul><li>Identify & evaluate risks that occur in the lifecycle </li></ul><ul><li>Applied a ‘light touch’ DRAMBORA ( http://www.repositoryaudit.eu/) methodology to case studies. Influenced by DIRKS </li></ul><ul><li>Risk categories </li></ul><ul><ul><li>Organisation Management, Staff, Tech Infrastructure, Acquisition & Ingest, Preservation & Storage, Access & Dissemination </li></ul></ul><ul><li>Risk Description </li></ul><ul><ul><li>Definition, manifestations, consequences, severity (risk impact x probability), mitigation strategies </li></ul></ul>
    14. 14. Recognised risks <ul><li>Storage: </li></ul><ul><ul><li>Insufficient capacity: local drives, network drives, 3rd party server </li></ul></ul><ul><li>Authenticity & integrity: </li></ul><ul><ul><li>Unidentified/unknown change. Some staff rely upon print-outs of digital original </li></ul></ul><ul><li>Archival value and retention period: </li></ul><ul><ul><li>Different criteria & quality thresholds </li></ul></ul><ul><ul><li>Business records – Recognised legal value & retention period </li></ul></ul><ul><ul><li>Research data – Archival value of research papers understood. Retention period of data has, until recently, not been recognised </li></ul></ul><ul><li>Access and usage: </li></ul><ul><ul><li>Business records have well-defined period of primary use, but unrecognised secondary use </li></ul></ul><ul><ul><li>Research papers understood, but do not always consider datasets & other outputs </li></ul></ul>
    15. 15. Risk management Strategy <ul><li>Storage and management infrastructure: </li></ul><ul><ul><li>Technical infrastructure to store and manage their data </li></ul></ul><ul><li>Education: </li></ul><ul><ul><li>Develop staff understanding of data management and archival principles </li></ul></ul><ul><ul><ul><li>Topics include: Authenticity and integrity, assessment of archival value </li></ul></ul></ul><ul><ul><ul><li>Methods: Practical documentation on data creation/management, training events </li></ul></ul></ul><ul><li>Policies: </li></ul><ul><ul><li>type of record collected, time period for collection, appraisal criteria for long-term retention </li></ul></ul><ul><li>Procedures: </li></ul><ul><ul><li>Data capture, curation, preservation </li></ul></ul><ul><li>Developed with consideration of cost implications </li></ul>
    16. 16. KCL Archives Preservation Repository <ul><li>An preservation repository for college data of short/long-term value: </li></ul><ul><li>Standards compliance </li></ul><ul><ul><li>OAIS Reference Model, TRAC, ISO 15489 </li></ul></ul><ul><li>Bitstream preservation: </li></ul><ul><ul><li>fixity creation/verification, online + offline storage </li></ul></ul><ul><li>Information Content Preservation: </li></ul><ul><ul><li>Format conversion, event logging – audit trail </li></ul></ul><ul><li>Access: </li></ul><ul><ul><li>Limited to archive reading room, catalogue descriptive MD to common standard </li></ul></ul><ul><li>Interoperable: </li></ul><ul><ul><li>Interact with other college & public systems eg student records </li></ul></ul>
    17. 17. OAIS Reference Model
    18. 18. Technical Infrastructure
    19. 19. Alfresco Actions <ul><li>Actions - a parameterized unit of work that can be applied to a node </li></ul><ul><li>Parameters – rules for action execution and type to be performed </li></ul><ul><li>jBPM synchronous or asynchronous workflows </li></ul><ul><li>Actions be performed at different stages of workflow </li></ul>
    20. 20. Ingest Actions <ul><li>Content model compliance </li></ul><ul><ul><li>Conforms to defined structure & object types </li></ul></ul><ul><li>Fixity generation </li></ul><ul><ul><li>All: MD5, SHA-1, CRC </li></ul></ul><ul><li>Format identification </li></ul><ul><ul><li>All: File(1), DROID </li></ul></ul><ul><li>Technical metadata extraction </li></ul><ul><ul><li>Format specific: JHOVE, MP3Info, others </li></ul></ul><ul><li>Specification conformance </li></ul><ul><ul><li>branch workflow according to threshold </li></ul></ul><ul><li>Conversion to preservation & dissemination derivative </li></ul><ul><ul><li>parameters for each format & licence </li></ul></ul><ul><ul><li>OpenOffice, ImageMagick, SoX </li></ul></ul><ul><li>Data Packaging </li></ul><ul><ul><li>Generate METS package, record action results as PREMIS Event </li></ul></ul>
    21. 21. Archiving actions <ul><li>Transfer into Fedora archive </li></ul><ul><ul><li>when collection closed (e.g. all papers submitted for meeting collection </li></ul></ul><ul><ul><li>After specified time period, e.g. 3 months </li></ul></ul><ul><li>Fixity verification: </li></ul><ul><ul><li>Conform that fixity unchanged </li></ul></ul><ul><li>Manual activity for future date: </li></ul><ul><ul><li>Anonymisation in 2 years, </li></ul></ul><ul><ul><li>Re-appraisal in xx years – Retain or remove </li></ul></ul><ul><li>Obsolescence monitoring? </li></ul><ul><ul><li>Possible future implementation </li></ul></ul>
    22. 22. Content Models <ul><li>Content models define rules that govern collection structure, data type & behaviour </li></ul><ul><li>Preservation Archive Content Models designed with consideration of resource type and Alfresco & Fedora capabilities </li></ul><ul><li>Content model for each resource type composed of 3 layers </li></ul><ul><li>Each layer is a Fedora Object that contains different metadata </li></ul>Filing Cabinet Drawer Folder
    23. 23. Content Model examples Each item represents a Fedora Object Each FO holds user provided metadata Different MD required at each layer Filing Cabinet Drawer Folder
    24. 24. Findings (so far) <ul><li>Data audit & risk analysis provide useful frameworks for analysing data management practices & justifying data archiving </li></ul><ul><li>No single definition of archival value – different criteria & quality thresholds </li></ul><ul><li>Application of archival principles to data assets provide demonstrable benefits </li></ul><ul><li>Duration requirements of business & research data are broadly similar </li></ul><ul><li>Digital repository architecture provide sufficient flexibility to manage data assets created for different purposes </li></ul>
    25. 25. Contact <ul><li>Gareth Knight </li></ul><ul><li>Centre for e-Research, King’s College London </li></ul><ul><li>[email_address] </li></ul><ul><li>020 7848 1979 </li></ul>http://www.kcl.ac.uk/iss/cerch/projects/portfolio/pekin.html Centre for e-Research : www.kcl.ac.uk/iss/cerch Archives and Information Management (AIM) : http://www.kcl.ac.uk/iss/explore/team/aim