You ask, we scan MARAC Conference October 30 2009 The Amsterdam City Archives  and the Archiefbank
This morning  <ul><li>Ellen: Amsterdam City Archives  </li></ul><ul><ul><li>a new Service Concept </li></ul></ul><ul><li>M...
Growing FAST <ul><li>5.000 different archives </li></ul><ul><li>15 repositories with 20 miles of shelf-length </li></ul><u...
Growing FAST <ul><li>5.000 different archives </li></ul><ul><li>15 repositories with 20 miles of shelf-length </li></ul><u...
BUT… less visitors each year MARAC Conference October 30 2009 17.958 2006 25.014 2002 26.598 1998 27.738 1992 29.788 1988 ...
Archives are dusty MARAC Conference October 30 2009
MORE webvisitors  And… MARAC Conference October 30 2009 512.592 17.958 2006 224.050 25.014 2002 40.048 26.598 1998   27.73...
New <ul><li>1. We want visitors to come to the archives  </li></ul><ul><ul><li>to experience the look and feel of authenti...
How <ul><li>To experience look and feel of an archive  </li></ul><ul><ul><li>be where the visitors are: in the city centre...
How <ul><li>To experience look and feel of an archive  </li></ul><ul><ul><li>be where the visitors are: in the city centre...
How <ul><li>To experience look and feel of an archive  </li></ul><ul><ul><li>be where the visitors are: in the city centre...
How <ul><li>To experience look and feel of an archive  </li></ul><ul><ul><li>be where the visitors are: in the city centre...
How <ul><li>To experience look and feel of an archive  </li></ul><ul><ul><li>be where the visitors are: in the city centre...
How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use the internet, use refere...
How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use t...
How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use t...
How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use t...
How <ul><li>All documents online? </li></ul><ul><ul><li>do not think about the 20 miles in your repository, think about th...
MARAC Conference October 30 2009
You ask Scanning on customer’s request, economic principles, technical issues  and work process We Scan MARAC Conference O...
You ask We Scan MARAC Conference October 30 2009 We Store We Do Scanning on customer’s request, economic principles Image ...
Q. How long does it take to scan it all? MARAC Conference October 30 2009 1 feet = 2.000 scans Production = 10.000 scans a...
The user doesn’t commit to anything by placing a request, but neither does the archive You ask We let our users set priori...
Costs for purchasing scans are equal for all users (the more you buy, the cheaper it gets) Scans available are integrated ...
Customers think a low price is important This means that costs for producing and storing scans have to be as low as possib...
This asks for a streamlined, efficiently organized work process You ask Digitization takes time, but research should not h...
An efficiently organized work process Low incidental and structural costs You ask MARAC Conference October 30 2009 Conclus...
Besides scanning on request projects are based on: In this presentation the focus is on large scale digitization at custom...
Goals of digitization projects vary from access to substitution of the originals In every project quality standard and met...
Experience shows that a constant production of 10.000 scans (at cutomer’s request) each  week is achievable This way tasks...
Documents that are being digitized in this reproduction process can have the following forms We scan MARAC Conference Octo...
Costs for producing and storing scans are determined to a high extent by the quality standard  set for the scans Purpose o...
But has no added value for the customer at all A quality higher than that inevitably will push up both incidental and stru...
We scan MARAC Conference October 30 2009 Scan quality and legibility High quality scan Modified scan (contrast) Optimal to...
Skimming on the quality of scans (it can be better) is purely an economic decision, not one taken  on principle We scan MA...
This way damage or loss of the originals is ruled out After digitization the originals can not be requested in the reading...
A file can contain one – hundreds of documents We scan By definition the entire file is scanned Never just a selection of ...
Contracting out of scanning was a logical choice We scan The in-house scan facilities are not designed for large-scale dig...
This calls for intensive collaboration Also, the workflows of archive and digitizer have to dovetail We scan There are man...
We use a combination of 1 and 3 We store Storage costs still are considerably high when producing large quantities of scan...
We store Resolution, compression and legibility: an example MARAC Conference October 30 2009 300 dpi, high quility JPEG 20...
We store Storage of compressed files as master images was “not done” The main arguments where Research after these argumen...
MARAC Conference October 30 2009 Comparison between file format, compression,  resolution and file size Scans with a file ...
TIFF uncompressed MARAC Conference October 30 2009 Comparison between file format, compression,  resolution and file size ...
JPEG (psd) 10 MARAC Conference October 30 2009 Comparison between file format, compression,  resolution and file size Scan...
JPEG (psd) 4 MARAC Conference October 30 2009 Comparison between file format, compression,  resolution and file size Scans...
JPEG2000 lossless MARAC Conference October 30 2009 Comparison between file format, compression,  resolution and file size ...
We store Comparison storage costs MARAC Conference October 30 2009 Storage of 500.000 images Avg size per scan uncompresse...
Projects with different goals, document types and partners take place at the same time A streamlined, standardized process...
We developed a simple, but effective workflow application in-house This asks for workflow management with a user-friendly ...
In the following slides we focus on the weekly production of 10.000 scans in the digitizing on request service We develope...
All public files can be requested for digitization via the findings aids in the Archiefbank Just by clicking on the “digit...
A unit to be digitized must be able to be identified at each step of the handling process The units therefore get a unique...
A unit to be digitized must be able to be identified at each step of the handling process The units therefore get a unique...
The workflow system generates a list of all originals to asses from the repositories The list is sorted on repository / sh...
MARAC Conference October 30 2009 All assessed originals are stored in a special room In this room all checks are executed ...
MARAC Conference October 30 2009 Information about the originals in our management  systems is not always complete If an i...
MARAC Conference October 30 2009 Information about the originals in our management  systems is not always complete If an i...
Material preparation is limited to the most minimal We Do 4. Checking the originals MARAC Conference October 30 2009 Stapl...
But this is only true when the numbering tallies exact, because: Numbering the originals has one advantage: We Do Not numb...
Securing completeness can be realized by other means: We Do MARAC Conference October 30 2009 Comparing scans to originals ...
For secure transport, special flight cases are used We Do 5. Transport MARAC Conference October 30 2009
It has to be perfectly clear which filenames this should be After scanning the scan operator or data manager has to assign...
Assigning filenames at City Archives Amsterdam MARAC Conference October 30 2009 Customer request Management systems First ...
An application from which all checks can be executed is in development Scans and metadata are checked efficiently Where po...
After import the “order for digitization” of each unit is completed After approving of all checks, scans and metadata are ...
After import the metadata are optimized for the search system For exchange of finding aids we use EAD From any workstation...
Until then scans are transported by use of portable USB harddisks Bandwith of the internet connections at the archive is s...
Derivates for use of thumbnails and zoom / contrast functionality are made After connecting the harddisk to the server the...
MARAC Conference October 30 2009 The requester can decide whether to buy scans or not When both scans and metadata have be...
MARAC Conference October 30 2009 The requester can decide whether to buy scans or not When both scans and metadata have be...
MARAC Conference October 30 2009
MARAC Conference October 30 2009
Mission accomplished <ul><li>Government satisfied: number of visitors increased fivefold </li></ul><ul><li>Management sati...
Government satisfied  Government MARAC Conference October 30 2009 531.143 77.298  2009 (3/4) 538.483 118.312 2008 520.483 ...
Management Management satisfied MARAC Conference October 30 2009 €  200,000 Digitization projects €  52,000 Webservices € ...
Customers <ul><li>Registered users:  ca. 15.000 </li></ul><ul><li>Requests:  10.605 </li></ul><ul><li>Scans online: more t...
Thanks Free scancredits? [email_address] MARAC Conference October 30 2009
 
Upcoming SlideShare
Loading in …5
×

You Ask We Scan

2,703 views
2,617 views

Published on

This is a copy of the presentation given by Ellen Fleurbaay and Marc Holtman of the Amsterdam City Archives at the the MARAC Plenary Session in Jersey City on Friday October 30, 2009.

Published in: Education, Travel
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,703
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
23
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • I will take you a step deeper into the workprocess of creating large amounts of scans. I’ll tell you about starting points and choises we have made and I’ll show you the result of some research we have done, particularyu towards image quality and filesize. Also, I’ll sohw you some back- and frontoffice tools from our webstie.
  • I will take you a step deeper into the workprocess of creating large amounts of scans. I’ll tell you about starting points and choises we have made and I’ll show you the result of some research we have done, particularyu towards image quality and filesize. Also, I’ll sohw you some back- and frontoffice tools from our webstie.
  • You Ask We Scan

    1. 1. You ask, we scan MARAC Conference October 30 2009 The Amsterdam City Archives and the Archiefbank
    2. 2. This morning <ul><li>Ellen: Amsterdam City Archives </li></ul><ul><ul><li>a new Service Concept </li></ul></ul><ul><li>Marc: large scale scanning </li></ul><ul><ul><li>on request, by order and subsidised projects </li></ul></ul><ul><ul><li>economic principles and workflow </li></ul></ul><ul><li>Ellen: Mission accomplished </li></ul><ul><ul><li>government and customers satisfied </li></ul></ul>MARAC Conference October 30 2009 MARAC Conference October 30 2009
    3. 3. Growing FAST <ul><li>5.000 different archives </li></ul><ul><li>15 repositories with 20 miles of shelf-length </li></ul><ul><li>91.000 prints, maps and drawings </li></ul><ul><li>824.000 photo’s </li></ul><ul><li>372.000 reference books </li></ul><ul><li>16.000 video- and audio tapes </li></ul>MARAC Conference October 30 2009 City Archives 1848 - 2009
    4. 4. Growing FAST <ul><li>5.000 different archives </li></ul><ul><li>15 repositories with 20 miles of shelf-length </li></ul><ul><li>91.000 prints, maps and drawings </li></ul><ul><li>824.000 foto’s </li></ul><ul><li>372.000 referencebooks </li></ul><ul><li>16.000 video- and audiotapes </li></ul>MARAC Conference October 30 2009 City Archives 1848 - 2009
    5. 5. BUT… less visitors each year MARAC Conference October 30 2009 17.958 2006 25.014 2002 26.598 1998 27.738 1992 29.788 1988 24.027 1982 Reading rooms Year Visitors
    6. 6. Archives are dusty MARAC Conference October 30 2009
    7. 7. MORE webvisitors And… MARAC Conference October 30 2009 512.592 17.958 2006 224.050 25.014 2002 40.048 26.598 1998   27.738 1992   29.788 1988   24.027 1982 Website Reading rooms Year Visitors
    8. 8. New <ul><li>1. We want visitors to come to the archives </li></ul><ul><ul><li>to experience the look and feel of authentic archival documents </li></ul></ul><ul><ul><li>to teach them the pleasure of doing their your own historical research </li></ul></ul><ul><li>2. Everybody should be able to use all archival collections at home 24/7 </li></ul>Service Concept MARAC Conference October 30 2009
    9. 9. How <ul><li>To experience look and feel of an archive </li></ul><ul><ul><li>be where the visitors are: in the city centre </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    10. 10. How <ul><li>To experience look and feel of an archive </li></ul><ul><ul><li>be where the visitors are: in the city centre </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    11. 11. How <ul><li>To experience look and feel of an archive </li></ul><ul><ul><li>be where the visitors are: in the city centre </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    12. 12. How <ul><li>To experience look and feel of an archive </li></ul><ul><ul><li>be where the visitors are: in the city centre </li></ul></ul><ul><ul><li>new corporate identity, new name, new logo </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    13. 13. How <ul><li>To experience look and feel of an archive </li></ul><ul><ul><li>be where the visitors are: in the city centre </li></ul></ul><ul><ul><li>new corporate identity, new name, new logo: City Archives </li></ul></ul><ul><ul><li>new products: museumnight, historical building, weekend open on Saturday and Sunday </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    14. 14. How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use the internet, use reference library, no silence please, do discuss with your fellow researchers </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    15. 15. How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use the internet and no silence please </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    16. 16. How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use the internet and no silence please </li></ul></ul><ul><ul><li>staff is walking around and offers free assistance </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    17. 17. How <ul><li>To experience pleasure of research </li></ul><ul><ul><li>New readingroom formula: use reference library, use the internet and no silence please </li></ul></ul><ul><ul><li>staff is walking around and offers free assistance </li></ul></ul><ul><ul><li>Staff is trained in educational and social skills </li></ul></ul>to attract visitors? MARAC Conference October 30 2009
    18. 18. How <ul><li>All documents online? </li></ul><ul><ul><li>do not think about the 20 miles in your repository, think about the few thousand documents your customers use per week </li></ul></ul><ul><li>Realistic and economic principles </li></ul><ul><ul><li>estimate costs of complete proces, not just costs of scanproduction </li></ul></ul><ul><ul><li>Dutch legislation: consult original is free, reproduction is paid for, so scans are to be paid for </li></ul></ul>to create an internet reading room? MARAC Conference October 30 2009
    19. 19. MARAC Conference October 30 2009
    20. 20. You ask Scanning on customer’s request, economic principles, technical issues and work process We Scan MARAC Conference October 30 2009
    21. 21. You ask We Scan MARAC Conference October 30 2009 We Store We Do Scanning on customer’s request, economic principles Image quality and workflow principles Compression and filesize Workflow, tools and practical issues
    22. 22. Q. How long does it take to scan it all? MARAC Conference October 30 2009 1 feet = 2.000 scans Production = 10.000 scans a week A. 406 years Will this be our ultimate solution? Q. How many scans can be made from 20 miles of archives? A. 739.200.001 scans
    23. 23. The user doesn’t commit to anything by placing a request, but neither does the archive You ask We let our users set priorities in digitization In principle all requests are honored, unless We speak of a request for digitization and not of an order MARAC Conference October 30 2009 1. Scanning at customer’s request It can not be digitized for material reasons Copyright material Disclosure restrictions apply All archive files can be requested for digitization via the online the finding aids
    24. 24. Costs for purchasing scans are equal for all users (the more you buy, the cheaper it gets) Scans available are integrated in the online finding aids All scans made are available for all users The requester is not obliged to buy all scans MARAC Conference October 30 2009 You ask 1. Scanning at customer’s request
    25. 25. Customers think a low price is important This means that costs for producing and storing scans have to be as low as possible Archival research easily runs into the use of dozens to hundreds of documents You ask The price of an ordinary copy in our reading room should be the benchmark MARAC Conference October 30 2009 2. Low costs 100 scans should not cost $ 100 The costs when purchasing scans online should be competitive with travel costs when visiting our reading room
    26. 26. This asks for a streamlined, efficiently organized work process You ask Digitization takes time, but research should not have to be planned weeks ahead Delivery time in a scanning on request service should be as short as possible MARAC Conference October 30 2009 3. Fast delivery Aim is a delivery time of 2 – 3 weeks
    27. 27. An efficiently organized work process Low incidental and structural costs You ask MARAC Conference October 30 2009 Conclusion If we can make sure that All finding aids can be selected for digitization by users The scans are delivered in short time For low costs it can be stated that we have no backlog in digitizing and the objective that the customer is able to consult digitized item has been achieved We need:
    28. 28. Besides scanning on request projects are based on: In this presentation the focus is on large scale digitization at customer’s request We scan However, scanning on request is only a part of all digitization that takes place in the archives MARAC Conference October 30 2009 Digitization at the Amsterdam City Archives in general Grant money (often on specific topics, like WWII) Selections of photographs, drawings etc for the Imagebank (Beeldbank) Cooperation with Amsterdam district councils and services
    29. 29. Goals of digitization projects vary from access to substitution of the originals In every project quality standard and method are set, depending on purpose and type of material For all projects we have one workflow We always work on project basis We scan MARAC Conference October 30 2009 Digitization at the Amsterdam City Archives in general
    30. 30. Experience shows that a constant production of 10.000 scans (at cutomer’s request) each week is achievable This way tasks can be planned best and deployment of staff is most efficient We scan 1. At large scale the more scans being made, the lower the price per scan Large scale production is a prerequisite in order to keep production costs as low as possible MARAC Conference October 30 2009 2. With a constant production Large scale production can only be organized effectively when constant production is assumed
    31. 31. Documents that are being digitized in this reproduction process can have the following forms We scan MARAC Conference October 30 2009 Small and large size Bound and loose-leafed entities Card indexes Old and modern material Low and high contrast documents Text alone, text and image together Hybrid forms 3. A broad spectrum of document types
    32. 32. Costs for producing and storing scans are determined to a high extent by the quality standard set for the scans Purpose of the scans: archival research using the web, straight from screen or print We scan 4. For archival research from screen or print The higher the standard of quality, the higher the costs will be In order to keep costs low it is prudent to allow the standard of quality follow from the requirement the end user places on the scan Textual information legible in de originals must be legible in the scans MARAC Conference October 30 2009
    33. 33. But has no added value for the customer at all A quality higher than that inevitably will push up both incidental and structural costs We scan 4. For archival research from screen or print Specified (basic) quality standard: MARAC Conference October 30 2009 Reproduction of all significant information Reproduction of details which are not part of the textual information is not required
    34. 34. We scan MARAC Conference October 30 2009 Scan quality and legibility High quality scan Modified scan (contrast) Optimal tonal range Example: very “light” original Excellent flexibility Poor tonal range Little flexibility Experience in practice learns that what is experienced as being “good legibility” is very personal. We decided to solve this problem with a smart filter in the document viewer. Poor legibility Excellent legibility Which one would you buy?
    35. 35. Skimming on the quality of scans (it can be better) is purely an economic decision, not one taken on principle We scan MARAC Conference October 30 2009 4. For archival research from screen or print It does make sense to let the standard of quality follow from the purpose the end-uses places on of the scans Price rates scanning, external partner 0,05 $ Legibility, auto-feed 0,30 – 0,75 $ Legibility 3 – 10 $ High-end Price comparison scanning costs
    36. 36. This way damage or loss of the originals is ruled out After digitization the originals can not be requested in the reading room anymore We scan 5. For conservation and security The scans in the scanning on request service are made for the purpose of access / archival research Not as a substitute for the originals Nevertheless, digitization does have a real conservation function MARAC Conference October 30 2009 Conservation of the originals remains the major concern
    37. 37. A file can contain one – hundreds of documents We scan By definition the entire file is scanned Never just a selection of pages There are a few reasons for this: MARAC Conference October 30 2009 6. Always complete files The costs for scanning are not so much a factor of quantity, but rather of the manual processing involving in it In the originals or the metadata it has to be indicated which documents are being digitized When shown in the Archiefbank, the user expects completeness When non-scanned pages have to be digitized later, the entire preparation process has to be gone through once again
    38. 38. Contracting out of scanning was a logical choice We scan The in-house scan facilities are not designed for large-scale digitizing The complexity of the workflow and material to be scanned calls for Investing only makes sense by very high production, organized on a large scale MARAC Conference October 30 2009 7. Contracting out the scanning to external partners Specialized hard- and software Specialized set-ups Knowledge Very complex technical infrastructure
    39. 39. This calls for intensive collaboration Also, the workflows of archive and digitizer have to dovetail We scan There are many scanning companies Most do have experience in bulk processing But not in this degree of complexity and diversity MARAC Conference October 30 2009 7. Contracting out scanning is more than awarding a contract to a supplier Contracting out the scanning to external partners
    40. 40. We use a combination of 1 and 3 We store Storage costs still are considerably high when producing large quantities of scans In order to bring structural costs down file size of the scans has to be as low as possible This can be achieved in three ways MARAC Conference October 30 2009 Scans with a file size as small as possible 1. Skimming on resolution 3. Using (lossless or lossy) compression on the files 2. Skimming on bit depth / amount of colors (only possible in formats like TIFF and PNG)
    41. 41. We store Resolution, compression and legibility: an example MARAC Conference October 30 2009 300 dpi, high quility JPEG 200 dpi, low quility JPEG Scans with a file size as small as possible
    42. 42. We store Storage of compressed files as master images was “not done” The main arguments where Research after these arguments learned: MARAC Conference October 30 2009 When using lossy compression you’ll loose information Compressed files are more vulnerable (preservation) Even when using strong lossy compression legibility is still guaranteed Compressed files are not more vulnerable to loss then uncompressed files But no compression means: large files  high storage costs Storage of uncompressed files is not necessary Scans with a file size as small as possible
    43. 43. MARAC Conference October 30 2009 Comparison between file format, compression, resolution and file size Scans with a file size as small as possible We store 55% 6 Tb 12 MB 24 bits 300 dpi Lossless Part 1 JPEG2000 0,5% 59 Gb 120 Kb 24 bits 300 dpi Lossy Part 6 34% 3,7 Tb 7,5 Mb 24 bits 300 dpi Lossy Qua (ps) 12 JPEG 10% 1,1 Tb 2,1 Mb 24 bits 300 dpi Lossy Qua (ps) 10 1,1% 124 Gb 255 Kb 24 bits 200 dpi Lossy Qua (ps) 4 Filesize 3,3 Mb 22,1 Mb Avg Lossy --- Type 15% 100% % 1,6 Tb 11 Tb 500.000 400 dpi 300 dpi Resolution Qua (ps) 10 No Compression 24 bits 24 bits TIFF Color Format
    44. 44. TIFF uncompressed MARAC Conference October 30 2009 Comparison between file format, compression, resolution and file size Scans with a file size as small as possible We store 55% 6 Tb 12 MB 24 bits 300 dpi Lossless Part 1 JPEG2000 0,5% 59 Gb 120 Kb 24 bits 300 dpi Lossy Part 6 34% 3,7 Tb 7,5 Mb 24 bits 300 dpi Lossy Qua (ps) 12 JPEG 10% 1,1 Tb 2,1 Mb 24 bits 300 dpi Lossy Qua (ps) 10 1,1% 124 Gb 255 Kb 24 bits 200 dpi Lossy Qua (ps) 4 Filesize 3,3 Mb 22,1 Mb Avg Lossy --- Type 15% 100% % 1,6 Tb 11 Tb 500.000 400 dpi 300 dpi Resolution Qua (ps) 10 No Compression 24 bits 24 bits TIFF Color Format
    45. 45. JPEG (psd) 10 MARAC Conference October 30 2009 Comparison between file format, compression, resolution and file size Scans with a file size as small as possible We store 55% 6 Tb 12 MB 24 bits 300 dpi Lossless Part 1 JPEG2000 0,5% 59 Gb 120 Kb 24 bits 300 dpi Lossy Part 6 34% 3,7 Tb 7,5 Mb 24 bits 300 dpi Lossy Qua (ps) 12 JPEG 10% 1,1 Tb 2,1 Mb 24 bits 300 dpi Lossy Qua (ps) 10 1,1% 124 Gb 255 Kb 24 bits 200 dpi Lossy Qua (ps) 4 Filesize 3,3 Mb 22,1 Mb Avg Lossy --- Type 15% 100% % 1,6 Tb 11 Tb 500.000 400 dpi 300 dpi Resolution Qua (ps) 10 No Compression 24 bits 24 bits TIFF Color Format
    46. 46. JPEG (psd) 4 MARAC Conference October 30 2009 Comparison between file format, compression, resolution and file size Scans with a file size as small as possible We store 55% 6 Tb 12 MB 24 bits 300 dpi Lossless Part 1 JPEG2000 0,5% 59 Gb 120 Kb 24 bits 300 dpi Lossy Part 6 34% 3,7 Tb 7,5 Mb 24 bits 300 dpi Lossy Qua (ps) 12 JPEG 10% 1,1 Tb 2,1 Mb 24 bits 300 dpi Lossy Qua (ps) 10 1,1% 124 Gb 255 Kb 24 bits 200 dpi Lossy Qua (ps) 4 Filesize 3,3 Mb 22,1 Mb Avg Lossy --- Type 15% 100% % 1,6 Tb 11 Tb 500.000 400 dpi 300 dpi Resolution Qua (ps) 10 No Compression 24 bits 24 bits TIFF Color Format
    47. 47. JPEG2000 lossless MARAC Conference October 30 2009 Comparison between file format, compression, resolution and file size Scans with a file size as small as possible We store 55% 6 Tb 12 MB 24 bits 300 dpi Lossless Part 1 JPEG2000 0,5% 59 Gb 120 Kb 24 bits 300 dpi Lossy Part 6 34% 3,7 Tb 7,5 Mb 24 bits 300 dpi Lossy Qua (ps) 12 JPEG 10% 1,1 Tb 2,1 Mb 24 bits 300 dpi Lossy Qua (ps) 10 1,1% 124 Gb 255 Kb 24 bits 200 dpi Lossy Qua (ps) 4 Filesize 3,3 Mb 22,1 Mb Avg Lossy --- Type 15% 100% % 1,6 Tb 11 Tb 500.000 400 dpi 300 dpi Resolution Qua (ps) 10 No Compression 24 bits 24 bits TIFF Color Format
    48. 48. We store Comparison storage costs MARAC Conference October 30 2009 Storage of 500.000 images Avg size per scan uncompressed = 22,1 MB Price rate : 1 TB, storage in a controlled e-repository environment on two separate locations, including IT costs $ 7.000 (NLD, nov 2009) Scans with a file size as small as possible (File)size still does matter! $ 420.000 $ 8.680 $ 77.000 $ 770.000 Costs 10 years $ 42.000 $ 868 $ 7.700 $ 77.000 Costs 1 year 6 TB 124 GB 1,1 TB 11 TB Storage JPEG 2000 (part 1, ll) JPEG 4 (200 dpi) JPEG 10 Tiff uncompressed Fileformat
    49. 49. Projects with different goals, document types and partners take place at the same time A streamlined, standardized process is indispensable when digitizing on a large scale Guidelines and best practices often take no account of these complex factors and the amount of scans to be produced We developed a process in which large scale and flexibility are starting points All digitization projects follow this process Developing the reproduction process MARAC Conference October 30 2009 We Do
    50. 50. We developed a simple, but effective workflow application in-house This asks for workflow management with a user-friendly application For all projects, at any moment, it has to be clear: We Do MARAC Conference October 30 2009 What the current status is of each to digitize unit Where each unit can be located What current and succeeding tasks are to be performed on each unit Developing the reproduction process
    51. 51. In the following slides we focus on the weekly production of 10.000 scans in the digitizing on request service We developed a simple, but effective workflow application in-house This asks for workflow management with a user-friendly application For all projects, at any moment, it has to be clear: We Do MARAC Conference October 30 2009 What the current status is of each to be digitized unit Where each unit can be located What current and succeeding tasks are to be performed on each unit Developing the reproduction process
    52. 52. All public files can be requested for digitization via the findings aids in the Archiefbank Just by clicking on the “digitize” button Production of 10.000 scans on weekly basis 1. Requesting for digitization MARAC Conference October 30 2009 We Do
    53. 53. A unit to be digitized must be able to be identified at each step of the handling process The units therefore get a unique meaningless order number An order number is provided by the metadata management system and is the basis for In practice: all units to be digitized get an order ticket 2. Providing ordernumbers MARAC Conference October 30 2009 Communication with the digitizer Scanning Assigning filenames Registration of filenames Billing by digitizer We Do
    54. 54. A unit to be digitized must be able to be identified at each step of the handling process The units therefore get a unique meaningless order number An order number is provided by the metadata management system and is the basis for In practice: all units to be digitized get an order ticket 2. Providing ordernumbers MARAC Conference October 30 2009 Communication with the digitizer Scanning Assigning filenames Registration of filenames Billing by digitizer We Do
    55. 55. The workflow system generates a list of all originals to asses from the repositories The list is sorted on repository / shelf to make retrieval efficient We Do 3. Assessing the originals MARAC Conference October 30 2009
    56. 56. MARAC Conference October 30 2009 All assessed originals are stored in a special room In this room all checks are executed We Do 4. Checking the originals
    57. 57. MARAC Conference October 30 2009 Information about the originals in our management systems is not always complete If an item falls into one of these categories the request is rejected B. Condition of the material A rough check of the originals takes place A. Content We Do 4. Checking the originals Copyrights Publicity Privacy Items that are in such a condition that digitizing or transport could cause damage, or are packaged in a way that scanning in conventional set-ups is not possible do not qualify for standard way of digitization
    58. 58. MARAC Conference October 30 2009 Information about the originals in our management systems is not always complete If an item falls into one of these categories the request is rejected B. Condition of the material A rough check of the originals takes place A. Content We Do 4. Checking the originals Copyrights Publicity Privacy Items that are in such a condition that digitizing or transport could cause damage, or are packaged in a way that scanning in conventional set-ups is not possible do not qualify for standard way of digitization
    59. 59. Material preparation is limited to the most minimal We Do 4. Checking the originals MARAC Conference October 30 2009 Staples are being removed as a rule Small reparations are executed by our restoration employees The sequence of the originals as found in the repository is not checked or altered We Do We don’t The originals are not numbered
    60. 60. But this is only true when the numbering tallies exact, because: Numbering the originals has one advantage: We Do Not number the originals MARAC Conference October 30 2009 The completeness of the scans (compared to the originals) can be guaranteed Numbers that are assigned double lead to illogical end numbers (100 scans: scan 100 has been numbered as 99) Experiments with numbering in practice learned that faultless numbering can not be realized A missing number in a sequence of scans leads to the conclusion that there is one original that has not been scanned
    61. 61. Securing completeness can be realized by other means: We Do MARAC Conference October 30 2009 Comparing scans to originals 1:1 after digitization Scanning the originals twice # scans = 365 # scans = 365 Low quality High quality master files Not number the originals
    62. 62. For secure transport, special flight cases are used We Do 5. Transport MARAC Conference October 30 2009
    63. 63. It has to be perfectly clear which filenames this should be After scanning the scan operator or data manager has to assign filenames to the scans Because, when the meaning changes, filenames should change too As a rule filenames contain no meaningful information We Do 6. / 7. Scanning and assigning filenames MARAC Conference October 30 2009 Filenames are the key between scans metadata
    64. 64. Assigning filenames at City Archives Amsterdam MARAC Conference October 30 2009 Customer request Management systems First 6#: ordernr Last 6#: serial nr Order ticket Filename Scanning the order A20758000001 A20758000002 A20758000003 Range A20758000001 – A20758999999 Archive 195 File 836 Order: A20758 A20758000004 A20758000005 Scan report A20758000001 A20758000002 A20758000003 A20758000004 A20758000005 12 digits Registration filenames Import
    65. 65. An application from which all checks can be executed is in development Scans and metadata are checked efficiently Where possible checks are automated 10. 11. Checking scans and metadata Basic checks We Do MARAC Conference October 30 2009 Depends on project Completeness Script Filenames Visual check production scans Visual check reference scans Quality scans Jhove File format validity MD-5 checksum comparison Data integrity Virus checker Viruses Method Check
    66. 66. After import the “order for digitization” of each unit is completed After approving of all checks, scans and metadata are imported into the management systems The imports are executed automatically, on basis of scripts and standard protocols for file transfer 13. 14. Import metadata and scans into management systems We Do MARAC Conference October 30 2009
    67. 67. After import the metadata are optimized for the search system For exchange of finding aids we use EAD From any workstation at the archive, directly via the CMS of the website The website is hosted from an external location Metadata are uploaded to the webserver by simple HTTP transfer 18. Import metadata into the website We Do MARAC Conference October 30 2009
    68. 68. Until then scans are transported by use of portable USB harddisks Bandwith of the internet connections at the archive is still too small for direct sFTP (or suchlike) upload of large quantities of scans to the webserver It seems likely that in the near future this will change 17. Import scans into the website Transport medium We Do MARAC Conference October 30 2009
    69. 69. Derivates for use of thumbnails and zoom / contrast functionality are made After connecting the harddisk to the server the import process starts Some basic checks are executed on the scans Import 17. Import scans into the website We Do MARAC Conference October 30 2009
    70. 70. MARAC Conference October 30 2009 The requester can decide whether to buy scans or not When both scans and metadata have been imported, automatically an e-mail is sent to the requester for digitization This email contains a link to the finding aid and thumbnails on the website Request complete! The happy customer: We Do
    71. 71. MARAC Conference October 30 2009 The requester can decide whether to buy scans or not When both scans and metadata have been imported, automatically an email is send to the requester for digitization This email contains a link to the finding aid and thumbnails on the website Request completed We Do The happy customer:
    72. 72. MARAC Conference October 30 2009
    73. 73. MARAC Conference October 30 2009
    74. 74. Mission accomplished <ul><li>Government satisfied: number of visitors increased fivefold </li></ul><ul><li>Management satisfied: costs and funding balance each other </li></ul><ul><li>Staff satisfied: enjoy their new role </li></ul><ul><li>Customers satisfied: lots of compliments </li></ul>Accomplished MARAC Conference October 30 2009
    75. 75. Government satisfied Government MARAC Conference October 30 2009 531.143 77.298  2009 (3/4) 538.483 118.312 2008 520.483 92.678 2007 512.592 17.958 2006 224.050 25.014 2002 40.048 26.598 1998   27.738 1992   29.788 1988   24.027 1982 Website Reading rooms Year Visitors
    76. 76. Management Management satisfied MARAC Conference October 30 2009 € 200,000 Digitization projects € 52,000 Webservices € 140,000 Digitsation on request Costs Archiefbank (2008) € 40,000 Government € 330,350 Project funding € 100,000 Digitsation on request Income Archiefbank (2008)
    77. 77. Customers <ul><li>Registered users: ca. 15.000 </li></ul><ul><li>Requests: 10.605 </li></ul><ul><li>Scans online: more than 7 million </li></ul><ul><li>Archives Next and Computable awards </li></ul>Customers satisfied MARAC Conference October 30 2009
    78. 78. Thanks Free scancredits? [email_address] MARAC Conference October 30 2009

    ×