Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Microfilm or Digitize: Which is Right for You?


Published on

Presentation on reformatting options for active and inactive records. Originally presented at the 2009 Annual Conference of the International Institute of Municipal Clerks, May 20, 2009

Published in: Business, Technology
  • Be the first to comment

Microfilm or Digitize: Which is Right for You?

  1. 1. Microform or Digitize: Which is Right For You? Brad Houston Records Manager, University of Wisconsin-Milwaukee
  2. 2. Reformatting <ul><li>v. 1. To create a copy with a format or structure different from the original, especially for preservation or access. 2. To migrate information from one carrier to another. </li></ul><ul><li>Source: Richard Pearce-Moses, A Glossary of Archival and Records Terminology, Society of American Archivists, 2005 </li></ul><ul><ul><li>Encompasses microfilming, digitization, photocopying, redevelopment, you name it! </li></ul></ul>
  3. 3. Why reformat your records? <ul><li>Save space in the office or records center </li></ul><ul><li>Improve access to frequently-requested records </li></ul><ul><li>Move records to format more suitable for long-term preservation </li></ul><ul><ul><li>Alternately: Provide duplication for redundancy/off-site storage </li></ul></ul>
  4. 4. Goals of this presentation <ul><li>Discuss general issues to consider before embarking on reformatting project </li></ul><ul><li>Examine the two main forms of reformatting practical for most municipalities </li></ul><ul><ul><li>Digital and Microfilming, of course! </li></ul></ul><ul><li>Describe scenarios in which either, both, or neither type of reformatting is appropriate </li></ul>
  5. 5. PART I: GENERAL CONSIDERATIONS <ul><li>Legal </li></ul><ul><li>Operational </li></ul><ul><li>Administrative </li></ul>
  6. 6. Before you start reformatting… <ul><li>Why is my office thinking about this in the first place? </li></ul><ul><ul><li>Space concerns? </li></ul></ul><ul><ul><li>Vital Records Protection? </li></ul></ul><ul><ul><li>Increased Public Access? </li></ul></ul><ul><ul><li>Better Discoverability? </li></ul></ul><ul><ul><li>Improved Security? </li></ul></ul><ul><ul><li>Do these records NEED to be reformatted? </li></ul></ul>
  7. 7. How are these records used? <ul><li>Preservation Concerns </li></ul><ul><li>Vital records </li></ul><ul><ul><li>Deeds, Birth Certificates, etc. </li></ul></ul><ul><li>Private-use records </li></ul><ul><li>Archival Documents </li></ul><ul><ul><li>Policies, reports, minutes </li></ul></ul><ul><li>Infrequently accessed </li></ul><ul><li>Unlikely to be subpoenaed/discovered </li></ul><ul><li>Access Concerns </li></ul><ul><li>Administrative records </li></ul><ul><ul><li>Notarization, budgets, etc. </li></ul></ul><ul><li>Public-use records </li></ul><ul><li>Short-retention documents </li></ul><ul><ul><li>Procedures, expense reports </li></ul></ul><ul><li>Frequently accessed </li></ul><ul><li>Increased likelihood of subpoena/discovery </li></ul>
  8. 8. What are the startup costs? <ul><li>Time/Labor needed to reformat </li></ul><ul><ul><li>Specialized labor/skills needed for project </li></ul></ul><ul><li>Equipment outlays </li></ul><ul><li>Public notices (as needed) </li></ul><ul><li>Other buy-in costs (appeals to officials?) </li></ul>
  9. 9. What are the maintenance costs? <ul><li>Cost of physical equipment maintenance </li></ul><ul><li>Time/labor needed to maintain </li></ul><ul><li>Time/labor needed to provide access </li></ul><ul><li>Media stability offsets </li></ul><ul><li>Is the maintenance cost offset by return on investment? </li></ul>
  10. 10. What are the security implications? <ul><li>Does the information in the reformatted records require protection? </li></ul><ul><li>If so: </li></ul><ul><ul><li>Reformatting a net increase or decrease in security? </li></ul></ul><ul><ul><li>Initial security measure outlays </li></ul></ul><ul><ul><li>Staff information security training </li></ul></ul><ul><ul><li>Secure access to public (as applicable) </li></ul></ul>
  11. 11. What are the legal implications? <ul><li>Federal Rules of Civil Procedure Sec. 34 (December 2006) </li></ul><ul><ul><li>Discovery applies equally to reformatted records! </li></ul></ul><ul><li>Public Records Laws in various states </li></ul><ul><ul><li>Example: Wis. Stat. 19.32: Open Records Act </li></ul></ul><ul><li>Other federal/state disclosure laws </li></ul><ul><ul><li>Sarbanes-Oxley, Gramm-Leach-Bliley, etc. </li></ul></ul><ul><ul><li>May apply if working with private contractors </li></ul></ul>
  12. 12. Should I reformat at all? <ul><li>Do reformatting candidates have high retention periods (>10 years)? </li></ul><ul><li>Is my estimated ROI (in either time or cost) more than the time or cost to reformat? </li></ul><ul><li>Do I have the infrastructure (or can I build it) to adequately support and maintain system? </li></ul><ul><li>If no to any of the above, consider postponing reformatting program! </li></ul>
  13. 13. PART II: MICROFILM AND MICROFICHE <ul><li>An Oldie But a Goodie </li></ul>
  14. 14. The least you need to know: Microfilm <ul><li>Preservation : Well-suited </li></ul><ul><li>Access : Requires time investment </li></ul><ul><li>Initial outlay : Medium equipment/low training costs </li></ul><ul><li>Continued Maintenance : Low Cost </li></ul><ul><li>Security Impact : Neutral </li></ul><ul><li>Legal impact: Potentially high </li></ul>
  15. 15. First steps: Needs planning <ul><li>Active vs. Inactive records for reformatting? </li></ul><ul><li>Storage and use copies? One or both? </li></ul><ul><li>Need for long-term preservation? </li></ul><ul><li>Microfilming from source documents or data from a computer? </li></ul><ul><li>Large or small amount of documents to be filmed? </li></ul>
  16. 16. Film Selection: Physical Medium <ul><li>Diazo Film </li></ul><ul><ul><li>Best for use copies; less stable for permanent preservation </li></ul></ul><ul><li>Vesicular Film </li></ul><ul><ul><li>Flips master image polarity; good for non-printing films. Non-ideal for preservation </li></ul></ul><ul><li>Silver Halide Film </li></ul><ul><ul><li>Best for long-term preservation; easily damaged with repeated use </li></ul></ul>
  17. 17. Film Selection: Format <ul><li>Microfilm roll (16mm or 35mm) </li></ul><ul><ul><li>Inexpensive and compact, but cumbersome to access </li></ul></ul><ul><ul><li>Best format for storage of vital or archival documents </li></ul></ul><ul><li>Microfiche (105mm, rows/columns) </li></ul><ul><ul><li>Easier to organize/access, but more expensive to produce and store </li></ul></ul><ul><ul><li>Best format for case files, semi-active records </li></ul></ul>
  18. 18. Initial Outlay: Equipment <ul><li>Microfilm Camera </li></ul><ul><ul><li>Rotary—best for initial reformatting project </li></ul></ul><ul><ul><li>Planetary—better image quality, diverse sizes </li></ul></ul><ul><li>Microform reader </li></ul><ul><ul><li>Different designs for film and fiche– some do both, for a price </li></ul></ul><ul><ul><li>Built in printer? </li></ul></ul>Planetary microfilm camera
  19. 19. Initial Outlay: Document Prep <ul><li>Remove documents from folders, remove fasteners, etc. </li></ul><ul><li>William Saffady: 1000 pages/hour (2.5 drawers/hour) as a benchmark </li></ul><ul><ul><li>This figure increases for non-standard file sizes </li></ul></ul><ul><li>Do you need a target ? Find out before you film! </li></ul>
  20. 20. Reduction Parameters <ul><li>Reduction: Number of times a page is reduced through microphotography </li></ul><ul><ul><li>24x reduction for standard-sized documents </li></ul></ul><ul><ul><li>30x reduction for large documents </li></ul></ul><ul><ul><li>48x for COM reduction </li></ul></ul><ul><li>Remember to make sure your microform reader can read the specified reduction! </li></ul><ul><ul><li>Variable-zoom lens, as appropriate </li></ul></ul>
  21. 21. Quality Control <ul><li>Should be done for each slide to check for: </li></ul><ul><ul><li>Legibility (naked eye overview) </li></ul></ul><ul><ul><li>Resolution: ability of reader to distinguish fine detail </li></ul></ul><ul><ul><ul><li>Specialized resolution chart </li></ul></ul></ul><ul><ul><li>Density: Contrast between info/non-info areas </li></ul></ul><ul><ul><ul><li>Densitometer </li></ul></ul></ul><ul><li>For standards, see: ANSI/AIIM MS23-1998 </li></ul>
  22. 22. Systems Maintenance <ul><li>Low cost of providing access, training staff/public on use </li></ul><ul><li>Cameras may need to be replaced periodically </li></ul><ul><li>Readers may last decades, depending on complexity </li></ul><ul><ul><li>If all else fails, records readable with a light table and magnifying glass! </li></ul></ul>
  23. 23. Environmental/Security <ul><li>On-site records=your responsibility to maintain good preservation environment! </li></ul><ul><ul><li>Windowless room if possible </li></ul></ul><ul><ul><li>No more than 50% relative humidity </li></ul></ul><ul><ul><li>No more than 70 degrees Fahrenheit </li></ul></ul><ul><ul><li>Acid-free/chemically stable sleeves (microfiche) </li></ul></ul><ul><li>Same security implications as with paper records </li></ul><ul><ul><li>Maintain secure room, locked cabinets, etc. as needed </li></ul></ul>
  24. 24. Access concerns <ul><li>Microfiche may be indexed/organized same way as “regular” case files </li></ul><ul><li>Microfilm: roll format means special index should be created </li></ul><ul><ul><li>Consider primary access point of records </li></ul></ul><ul><ul><li>Entries should include: reel, frame, secondary access </li></ul></ul><ul><ul><li>Provide cross-references as needed </li></ul></ul><ul><ul><li>Indicate in index when filming took place, and by whom the records were filmed </li></ul></ul>
  25. 25. Legal Issues and Records Retention <ul><li>Will you be able to produce the documents in a timely manner? </li></ul><ul><li>Uniform Photographic Copies Act (28 USC 1732) </li></ul><ul><li>If records have 10 year retention period or less, off-site paper storage may be more cost-effective </li></ul>
  26. 26. Final Verdict: <ul><li>Microfilm remains useful for its relative lack of expense, easy reproduction, and extreme longevity (>500 years!). However, the issue of accessibility, particularly with microfilm rolls, must be taken into account when deciding to film records for preservation or access. </li></ul>
  27. 27. PART III: DIGITIZATION <ul><li>New Paradigms, New Problems </li></ul>
  28. 28. The Least You Need to Know: Digitization <ul><li>Preservation: Potentially problematic </li></ul><ul><li>Access: Very good, with the right tools </li></ul><ul><li>Initial Outlay: Low equipment/medium software/high staff training </li></ul><ul><li>Continued Maintenance : high level of staff time/effort </li></ul><ul><li>Security Impact: Potentially high </li></ul><ul><li>Legal Impact: High, but manageable </li></ul>
  29. 29. Digitization is NOT a panacea! <ul><li>Potentially very powerful productivity tool and space saver </li></ul><ul><li>At minimum, the following are required for an effective program: </li></ul><ul><ul><li>Content Management System </li></ul></ul><ul><ul><li>Digital Preservation Plan </li></ul></ul><ul><ul><li>Disaster Recovery Plan </li></ul></ul>
  30. 30. Needs Planning: Digitization <ul><li>Active vs. Inactive records </li></ul><ul><ul><li>For digital records, this is a function of retrieval speed </li></ul></ul><ul><li>Public vs. Private access to the records </li></ul><ul><li>What are your preservation needs? </li></ul><ul><li>File-sharing needs? </li></ul>
  31. 31. Initial Outlay: Hardware <ul><li>Scanner, preferably with a document feed </li></ul><ul><ul><li>Resolution required depends on type of scan </li></ul></ul><ul><li>Digital Camera/Overhead Scanner </li></ul><ul><li>Importing/editing computer </li></ul><ul><li>Public Access computer (as needed) </li></ul><ul><li>Storage Media </li></ul>
  32. 32. Digital Storage Media <ul><li>Good: Removable storage media (CD-ROMs, DVD-ROMs, External Hard Drive) </li></ul><ul><ul><li>Good for quick access; low longevity </li></ul></ul><ul><li>Better: Disc Array or Server Array </li></ul><ul><ul><li>Medium longevity and access speed; better for redundancy of data </li></ul></ul><ul><li>Best: Distributed Storage Space </li></ul><ul><ul><li>Maximum redundancy; slowest retrieval time </li></ul></ul>
  33. 33. Initial Outlay: Software <ul><li>Scanning/Import software </li></ul><ul><ul><li>Usually comes with scanner </li></ul></ul><ul><li>Image Manipulation software </li></ul><ul><ul><li>For document reformatting, include Optical Character Recognition (OCR) capability </li></ul></ul><ul><li>Content Management System (CMS) </li></ul><ul><ul><li>The most important (and most expensive) component of digitization management </li></ul></ul>
  34. 34. Content Management Systems <ul><li>Should include robust metadata at point of entry: </li></ul><ul><ul><li>Subject </li></ul></ul><ul><ul><li>Author </li></ul></ul><ul><ul><li>Recipient </li></ul></ul><ul><ul><li>Date </li></ul></ul><ul><ul><li>Other keywords/tags </li></ul></ul><ul><li>Some CMSs extract this automatically– ask your vendor! </li></ul>
  35. 35. Content Management, Records Management <ul><li>Consider a CMS with integrated Records Management: </li></ul><ul><ul><li>Facilitated indexing </li></ul></ul><ul><ul><li>Automated or semi-automated disposition </li></ul></ul><ul><ul><li>Improved information security </li></ul></ul><ul><ul><li>Legal protection under “Safe Harbor” rule (FRCP 37) </li></ul></ul>
  36. 36. DoD 5015.2 RM standard <ul><li>Standards for ERMS– often advertised! </li></ul><ul><li>A compliant system will include: </li></ul><ul><ul><li>Metadata entry and file plans (including retention scheduling) </li></ul></ul><ul><ul><li>Records declaration and classification </li></ul></ul><ul><ul><li>Records protection and/or change auditing </li></ul></ul><ul><ul><li>Designation of official ERMS administrator(s) </li></ul></ul>
  37. 37. Imaging Parameters <ul><li>Pixels per inch/dots per inch (ppi/dpi) </li></ul><ul><li>Capture mode </li></ul><ul><li>Bit Depth </li></ul><ul><li>File Format </li></ul><ul><li>Compression </li></ul><ul><li>n.b. Higher values increase image quality AND file size/storage requirements </li></ul>
  38. 38. PPI/Resolution <ul><li>Should be as high as possible, but consider display and space needs! </li></ul><ul><li>To adjust resolution of already-imported image, adjust image size (smaller=sharper) </li></ul>Document Type PPI (master) PPI (access) Text 400 150 Photographs 400-800 200-300 Maps 600 (400 if oversized) 150-300 Graphics (including arch. Drawings) 600-800 200-400
  39. 39. Capture Mode/Bit Depth <ul><li>A measure of the amount of colors able to be displayed in the image </li></ul><ul><li>USUALLY an easy call to make– with some exceptions </li></ul>Document Type Capture Mode Bit Depth Text Bitonal 1 bit B/W photos/drawings Grayscale 8 bit Color photos/drawings RGB 24-48 bit (access v. master) Maps Grayscale/RGB 24-48 bit (access v. master)
  40. 40. File Format/Compression <ul><li>Higher compression=greater degradation of data over time </li></ul><ul><li>Lossy vs. Lossless compression </li></ul><ul><li>Proprietary vs. Non-Proprietary formats </li></ul>Document Type File Format (Preservation) File Format (Access) Text TIFF or PDF/A JPEG or PDF Photographs TIFF or JPEG 2000 JPEG Graphics TIFF or JPEG 2000 JPEG Maps TIFF or JPEG 2000 JPEG
  41. 41. What’s in a name? <ul><li>Don’t rely on the CMS to find files for you! </li></ul><ul><li>File names should be easily-browsable: </li></ul><ul><ul><li>Date of imaging or use </li></ul></ul><ul><ul><li>Type of file (e.g. dept. acronym?) </li></ul></ul><ul><ul><li>Unique identifier (e.g. project name, serial #) </li></ul></ul><ul><li>File naming not a substitute for proper metadata in the CMS </li></ul>
  42. 42. Quality Control: Physical Aspects <ul><li>Looking for similar defects as with microfilm QC </li></ul><ul><ul><li>Legible images </li></ul></ul><ul><ul><li>Correct orientation/resolution </li></ul></ul><ul><ul><li>Density, color balance, etc. </li></ul></ul><ul><ul><ul><li>Targets and color bars assist with this check </li></ul></ul></ul><ul><li>Some QC defects are digitization-specific: </li></ul><ul><ul><li>Correct size, name, file format, etc. </li></ul></ul><ul><ul><li>Digital artifacts (lines on screen, etc.) </li></ul></ul>
  43. 43. Quality Control: Content Aspects <ul><li>Has the OCR read the content properly? </li></ul><ul><ul><li>Usually a cursor drag determines this </li></ul></ul><ul><ul><li>Most OCR programs allow for manual corrections </li></ul></ul><ul><li>Is the metadata complete and appropriate? </li></ul><ul><ul><li>Dublin Core: search and use metadata </li></ul></ul><ul><ul><li>PREMIS: Preservation metadata </li></ul></ul><ul><ul><li>Administrative metadata (rights, statutes, etc.) </li></ul></ul><ul><ul><li>Structural metadata (how records tie together) </li></ul></ul>
  44. 44. Systems Maintenance <ul><li>Maintenance of a digitization system often comprises up to two-thirds of total budget! </li></ul><ul><ul><li>Cataloging, description, indexing </li></ul></ul><ul><ul><li>Ongoing staff training </li></ul></ul><ul><ul><li>Physical data storage </li></ul></ul><ul><ul><li>Data migration </li></ul></ul><ul><li>Consider cost-sharing with neighboring municipalities </li></ul>
  45. 45. Access to Digitized Records <ul><li>The key upside to digitization! </li></ul><ul><li>Public Access Terminal/Web Access/Admin use only? </li></ul><ul><li>Rights management? </li></ul><ul><ul><li>Will you allow printing? </li></ul></ul><ul><ul><li>Watermarks </li></ul></ul><ul><li>Electronic file structure </li></ul><ul><li>Searchability? </li></ul>
  46. 46. Preservation Concerns <ul><li>IT department usually keeps data storage secure, environmentally stable </li></ul><ul><li>Data itself may have stability problems! </li></ul><ul><ul><li>Software obsolescence </li></ul></ul><ul><ul><li>Hardware obsolescence </li></ul></ul><ul><ul><li>Data integrity loss </li></ul></ul><ul><ul><li>File format obsolescence </li></ul></ul>
  47. 47. Preservation Solutions? <ul><li>Non-proprietary formats, lossless compression </li></ul><ul><ul><li>Usually implies access copy creation </li></ul></ul><ul><li>REGULAR (5 years/3 version) reformatting: </li></ul><ul><ul><li>Migration to new systems/file formats </li></ul></ul><ul><ul><li>Emulation of programs to read files on old systems </li></ul></ul><ul><ul><li>Encapsulation in XML “wrappers” </li></ul></ul><ul><li>Work with IT to see which is right for your organization </li></ul>
  48. 48. Information Security <ul><li>Risk of security breach is MUCH higher because of ease of reproducibility </li></ul><ul><li>Potential Solutions: </li></ul><ul><ul><li>Versioning/Logging of imaged documents </li></ul></ul><ul><ul><li>Declaration of imaged documents as “records” in DoD 5015.2 system </li></ul></ul><ul><ul><li>Visible or metadata-based watermarks </li></ul></ul><ul><ul><li>Embargo on portable storage for imaged documents </li></ul></ul>
  49. 49. Digitization Legal Issues <ul><li>Zubulake Case (2004): If you have the records, you must produce them or pay penalties </li></ul><ul><li>Federal Rules of Civil Procedure </li></ul><ul><ul><li>Electronic records are discoverable </li></ul></ul><ul><ul><li>Safe Harbor rule DOES NOT APPLY if there is reasonable expectation of litigation (Broccoli) </li></ul></ul><ul><ul><li>Plaintiff may request format for production </li></ul></ul>
  50. 50. Legal Issues: Responses <ul><li>Make sure your CMS has a robust search feature </li></ul><ul><ul><li>Semantic web /fuzzy logic? </li></ul></ul><ul><li>Be prepared to convert formats as needed </li></ul><ul><li>Talk to legal team about Litigation Hold procedures </li></ul><ul><li>Formalize records retention/destruction schedules and procedures </li></ul>
  51. 51. Final Verdict <ul><li>Digitization offers organizations excellent options for saving space and making reformatted records accessible to staff and to the public. However, great care must be taken to ensure that preservation, security, and legal issues are addressed before a digitization solution is implemented. </li></ul>
  52. 52. PART IV: CONCLUSION <ul><li>Summation, scenarios, resources, questions </li></ul>
  53. 53. In Summary: <ul><li>Microfilm and Microfiche </li></ul><ul><li>Excellent preservation medium </li></ul><ul><li>Low-to-medium startup costs </li></ul><ul><li>Low maintenance and staff training requirements </li></ul><ul><li>Potential accessibility issues </li></ul><ul><li>High security, reduced legal response ability </li></ul><ul><li>Digitization </li></ul><ul><li>Superb accessibility if managed within CMS </li></ul><ul><li>Increased ability to respond to legal requirements </li></ul><ul><li>Medium-to-high startup costs </li></ul><ul><li>High maintenance and staff training requirements </li></ul><ul><li>Security and preservation issues </li></ul>
  54. 54. Scenarios: Office A <ul><li>Reformatting target: Birth Records </li></ul><ul><ul><li>Considered a “vital record” by statute </li></ul></ul><ul><ul><li>Medium-to-high usage </li></ul></ul><ul><ul><li>High security needs </li></ul></ul><ul><ul><li>Medium-to-low subpoena likelihood </li></ul></ul><ul><li>What would you suggest? </li></ul><ul><ul><li>My answer: Microfilm </li></ul></ul>
  55. 55. Scenarios: Office B <ul><li>Reformatting Target: Financial Records </li></ul><ul><ul><li>Records Schedule states 7-year retention time </li></ul></ul><ul><ul><li>Subject to public access under state records law </li></ul></ul><ul><ul><li>Very high use– referred to on daily basis </li></ul></ul><ul><ul><li>High likelihood of audit/subpoena </li></ul></ul><ul><ul><li>Medium security requirements </li></ul></ul><ul><li>What would you suggest? </li></ul><ul><ul><li>My answer: Do not reformat OR digitize as requested </li></ul></ul>
  56. 56. Scenarios: Office C <ul><li>Reformatting Target: Child Welfare Case Files </li></ul><ul><ul><li>Records Schedule states 20-year retention time </li></ul></ul><ul><ul><li>High use while clients active; low use thereafter </li></ul></ul><ul><ul><li>Very high security requirements </li></ul></ul><ul><ul><li>Medium-to-high potential for subpoena </li></ul></ul><ul><li>What would you suggest? </li></ul><ul><ul><li>My answer: Microfiche case files and/or digitize </li></ul></ul>
  57. 57. Scenarios: Office D <ul><li>Reformatting Target: Committee Minutes </li></ul><ul><ul><li>Permanent Retention/Archival </li></ul></ul><ul><ul><li>Subject to public access under state records law </li></ul></ul><ul><ul><li>High use by both public and staff users </li></ul></ul><ul><ul><li>Low security requirements </li></ul></ul><ul><ul><li>Medium likelihood of audit/subpoena </li></ul></ul><ul><li>What would you suggest? </li></ul><ul><ul><li>My answer: Digitize with web access </li></ul></ul>