Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Digitalización de Literatura de Biodiversidad: an overview of the BHL for CONABIO

779 views

Published on

Presentation for CONABIO staff during a Workshop on Dec 17, 2014 at Mexico City.

  • Be the first to comment

  • Be the first to like this

Digitalización de Literatura de Biodiversidad: an overview of the BHL for CONABIO

  1. 1. Digitizando Literatura sobre Biodiversidad (Contenido Técnico) CONABIO, México William Ulate, Director Técnico de BHL 17 Diciembre 2014
  2. 2. Digitization Workflow Insert Smithsonian Macaw software here
  3. 3. Hardware & Software Hardware Usando una Estación Scribe
  4. 4. Escaneo por Internet Archive Northeast Regional Scanning Facility (Boston) New Jersey Facility Natural History Museum, London Fedscan (Library of Congress) Internet Archive (San Francisco) Smithsonian Libraries Missouri Botanical Garden (Non- Scribe operation)
  5. 5. Hardware & Software Hardware Usando una Estación Scribe “Off-the-shelf” escaners o cámaras digitales de buena calidad Software Wonderfetch -> Partner Meta App (si usan máquinas Scribe) identifier search_id title volume creator date call_number language subject publisher description page-progression possible-copyright- status licenseurl
  6. 6. Hardware & Software Hardware Usando una Estación Scribe “Off-the-shelf” escaners o cámaras digitales de buena calidad Software Wonderfetch -> Partner Meta App (when using Scribe machines) Macaw
  7. 7. Software de Escaneo: Macaw
  8. 8. Hardware & Software Hardware Usando una Estación Scribe “Off-the-shelf” escaners o cámaras digitales de buena calidad Software Wonderfetch -> Partner Meta App (when using Scribe machines) Macaw Uploading directly to Internet Archive (for example: MBG‟s Botanicus http://www.botanicus.org/)
  9. 9. Standards and formats to consider The simplest way to contribute a text item to IA is currently as a single pdf file. IA creates a second pdf with a text layer, if none exist. Items can be submitted as a stack of image files, one image per page. The files can be in JPEG2000, JPG, or TIFF format, but with strict requirements for how the files in an image stack are to be named, and the stack needs to be packed into a single .zip or .tar file before submission. When IA (Archive.org) scans a book for a Contributing Library, they use the custom-engineered "Scribe" workstation, but for many materials, adequate images can be made with off-the-shelf scanners or good-quality digital cameras. For best results, it is recommended to use the highest resolution your device is capable of. Most images IA processes were produced at a resolution of 300- 600 ppi.
  10. 10. Standards and formats to consider BHL recommends following, in part, the DLF's "Benchmark for Faithful Digital Reproductions of Monographs and Serials" (available online at http://www.diglib.org/standards/bmarkfin.htm). Bitonal: 600 dpi, 1-bit or bitonal TIFF images Grayscale: 300 dpi, 8-bit grayscale uncompressed TIFF, or lossless compressed image (e.g. LZW, JPEG2000 [*.jp2]). Color: 300 dpi, 24-bit color uncompressed TIFF, or lossless compressed images (e.g. LZW, JPEG2000 [*.jp2]). NOTE: the above specifications are the preferred ones. BHL will, however, accept lossy files. In the case of JPEG2000, files with a compression level of 85% are acceptable.
  11. 11. Standards and formats to consider Currently, BHL data can be downloaded as MODS, EndNote and BibTex. See our wiki page with more information: http://biodivlib.wikispaces.com/Data+Exports#x--MODS Title metadata as well as pagination, descriptive and page order (structural) metadata is being copied into METS files in the <biodiveristy> collection at IA. The purpose of these METS files is to accommodate the need of our pagination data. These METS files are pagination specific and they do not have the item/volume information included. If bibliographic metadata for BHL content was required, it should be found in the MODS files on the Data Exports page.
  12. 12. Standards and formats to consider For the future, we are looking at serving OLEF as an envelope format to share information with other BHL Nodes. See http://www.bhle.eu/bhl-schema/v0.3/ and http://www.slideshare.net/HeimoRainer/bhleuropemet adataharmonisationtdwg20111018kollerwhrainer/6 )
  13. 13. Metadata generation and indexing strategy Each item to be uploaded needs a unique identifier within our central repository, currently Internet Archive (archive.org) and a folder with such name is created to hold the uploaded and generated (derivative) files. Within BHL we record metadata at 3 levels of bibliographic granularity – Title, Item & Page – as well as metadata for the Creator(s) of the title.
  14. 14. Metadata generation and indexing strategy Scanned material (jp2.zip) and basic title-level metadata content (marc.xml), item-level metadata (meta.xml) and page-level metadata (scandata.xml) are uploaded to Internet Archive (IA), in the „biodiversity‟ collection. JP2.zip: The compressed JP2 images (Compression Quality 15) that IA will use for delivering pages to the Read Online feature following a very specific naming convention for the filenames: Master images files named with local library identifier + 4-digit sequence number (with no gaps). MARC.xml: The MARC record for the title from the library catalog in MARCXML format Title, *Abbreviation, *Creator, Description, Publisher, Start Date Published, End Date Published, Local Library Identifier, *OCLC Number, *ISSN, *ISBN, *Call Number, *Subject, *Language, Date Created, Date Last Modified, *Foreign Keys
  15. 15. Metadata generation and indexing strategy META.xml: The item level information (even redundant with the title-level information) including the title, author, publisher, copyright information, digitizing sponsor, date published, type of item, and who originally uploaded it. IA may also update this XML file with information as it processes the pages of the item. Barcode, Sequence, Local Library Identifier, +Start Volume, End Volume, +Start Date, End Date, *Language, Scanning Institution, *Scanning Contributor, *Scanning Sponsor, Date Created, Date Last Modified SCANDATA.xml: An XML file (scandata.xml) recording information about each page image (handSide, cropBox, original width & height, etc. ) FileName, Sequence, *Page number, *Page Type, Year, Volume, IssuePrefix, Issue, Date Created, Date Last Modified
  16. 16. Metadata generation and indexing strategy CREATOR: A “Creator” is defined as a person or company responsible for the creation of the Title. Name, *Role, Date of Birth, Date of Death, Biography A detailed description of the contents of each one of these files and the whole process of Uploading content to IA is available at: http://biodivlib.wikispaces.com/Upload
  17. 17. Metadata generation and indexing strategy Internet Archive runs the OCR process and generates “derivative files” that include: The resulting files of the OCR process with ABBYY FineReader (djvu, djvu.txt, djvu.xml, abby.gz) A 100x152 pixel GIF with a looping, animated thumbnail of the first 20 pages of a book. The presentation version on BHL in PDF format. The MARC record in binary and XML formats. And others ( for a more detailed description you can see http://biodivlib.wikispaces.com/Download+All+File+Type s+and+Descriptions )
  18. 18. Metadata generation and indexing strategy The metadata from new items included in the BHL collection is included in the database and indexed to be used in searches through the Portal and API services. Periodically, the OCR pages are ran through taxonomic names services to mine for new taxa names like TaxonFinder (ubio.org) or GNRDS (Global Names resolution tools and services: resolver.globalnames.org) soon. Taxa names are added to the database and written back into Internet Archive (names.xml)
  19. 19. Online Platform Capture System Scribe machines Macaw Publication BHL Portal BookViewer PDF Generator
  20. 20. Online Platform Publication BHL API (biodivlib.wikispaces.com/Developer+Tools+and+API) The BHL Application Programming Interface (API) is a set of REST-like web services that can be invoked via HTTP queries (GET/POST requests) or SOAP. Responses can be received in one of three formats: JSON, XML, or XML wrapped in a SOAP envelope. We are currently developing a new API v3, closer to a RESTful design than previous versions, using resource-centric URLs (where possible) and GET/PUT/POST/DELETE verbs.
  21. 21. Online Platform Publication Data Exports (biodivlib.wikispaces.com/Data+Exports)
  22. 22. Online Platform Management BHL Admin Dashboard Admin Functions (Alert Message, Image Server, Collections, Institutions, Languages, Page Types, PDF Requests, Segment Types) Library Functions (Titles/Items/Segments /Pagination/Authors) Science Functions (Names (Taxa) on a Page) Library Statistics (Titles/Items/Pages/Names/Segments/Items with Segments, Names, Pages with Names) Growth Statistics (Titles/Items/Pages/Names/Segments new this Month/Year)
  23. 23. Online Platform Management BHL Admin Dashboard PDF Generation Statistics (Generated: 174,162) Internet Archive Harvesting Statistics (Complete: 119,125 items) BioStor Harvest Statistics (Published: 11,126 as of Aug. 29, 2013) DOI Assignment Statistics (DOI Approved: 57,338 as of Aug 29, 2013) Web Traffic Statistics (API v2, OpenURL) Reports (Item Pagination, Title Import History, Character Encoding Problems, DOIs by Institution, Monographic Contributions, Items by Contributor)
  24. 24. Deduplication • We try to avoid duplication where possible • Tools • Serials = Scanlist • Monographs = Monographic deduper • Check the BHL before you send for scanning • We do our best but duplication happens • Post-digitization, we merge titles as necessary
  25. 25. Online Platform Management Monographic Deduping Tool The MBLWHOI Library has been working on a tool that assists with de-duplicating the monographs that BHL members are sending to IA for scanning. The application is ready for use and it‟s entirely web-based, requiring no client or user configuration. The monographic deduper acts as a master database that contains records for all of the monographs that any BHL partner institution has scanned.
  26. 26. Online Platform Management Monographic Deduping Tool In addition, there is a process also in place that allows for material ingested from the Internet Archive, but not contributed by a BHLpartner institution, to be added to the deduper database. Ultimately, the Monographic deduper database should be seen as living record of accountability that communicates to staff collaborating in the BHL network, a partner‟s promise to digitize a particular monographic title.
  27. 27. Online Platform Management Serials Bid List It is a catalogue that allows users to browse and search Serials titles held by BHL member institutions using advanced filtering.
  28. 28. Technical Group at MBG Mike Lichtenberg Developer Trish Rose-Sandler Data Analyst William Ulate Technical Director
  29. 29. Technical Support MBG IT Division Manage servers, systems and telecommunications. Installs software needed And others: MBL Smithsonian Internet Archive BHL-Australia BHL-Europe
  30. 30. Technical Advisory Group
  31. 31. Firewall Images (JP2) PDF Coordinate-based OCR XML metadata BHL Architecture: Window Seat Ed. BHL DB Internet Archive Storage Logic APIs UI Data Exports Access Data Transform Utilities Geocoding Name Finding
  32. 32. Projects Global Names Art of Life Purposeful Gaming Digging into Data
  33. 33. Scientific Name Extraction TaxonFinder algorithm in production since 2008 More than 100 million candidate name strings More than 1.5 million unique, verified names Available through UI, APIs, Data Exports & Internet Archive New collaboration with Global Names project Improved algorithm, better precision & recall More data with TaxonFinder and Neti Neti! http://gnrd.globalnames.org/
  34. 34. Taxon Names BEFORE Name Instances 101,591,803 101,288,804 Unique Names 7,498,554 7,464,924 Verified Names 1,905,507 1,902,803 EOL Names 63,130,350 62,963,582 EOL Pages 13,579,868 13,532,684 AFTER Name Instances 151,222,182 150,066,425 Unique Names 29,246,382 29,091,767 Verified Names 10,153,165 10,109,540 EOL Names 87,791,695 87,135,089 EOL Pages 15,466,713 15,342,867
  35. 35. Article-level metadata Chapter-level metadata Treatment-level metadata Part-level metadata
  36. 36. Articles in the BHL UI
  37. 37. See also:
  38. 38. Related Titles
  39. 39. Digitization workflow 1. Titles vs. Items vs. Segments 2. Metadata we need: • MARC for book and journal titles • Volume information • Page data BHL Term Titles Items Segments Library Term Book or Journal Titles Volume, Piece Articles, Book chapters, Meaning Conceptual unit Object Section of consecutive pages
  40. 40. Art of Life
  41. 41. Art of Life
  42. 42. Art of Life
  43. 43. Art of Life
  44. 44. Art of Life
  45. 45. Art of Life
  46. 46. Macaw https://github.com/cajunjoel/macaw-book-metadata-tool
  47. 47. Reviewing Metadata
  48. 48. Reviewing Metadata
  49. 49. Manually built: 1,714 sets 89,457 images
  50. 50. Purposeful Gaming
  51. 51. *E.xvi�c�piteI von c. cXx.WptdvonfnrWmn bu�fbe;bcn.5 am cix bIa � S &3rn~ 41X a�m cv(f b1air�'o�et ert oiensr �; �', :�hlrfc�c wa ff�4am.diug bist a 6aiw~s ff oJrJtwt nof bL4ecImt& blfafra mem b t wag `wr 4 cn wiu 4 e8t5m.ed bvUratflb ck wuo, ma144'*4I bttE5rmbebt =rt3'kn am4ra tif vrmr Waff C * t6rmnli an `tn�ciblatGteaM w ?ffoaifrn w4wmeu nu weib e , wpiteI voE5teiri ct c ober gtUcr cit cm` 91 cLi biar J ' >bSciatl�Oiff ;Bruet wacfttc n qmcx b1a bl: bt5c lttmtt bb9 lkr w.llr#e iti ncn xoa ff cu :r trtuft *e t � B Rn "� trv W1Rt' ?Cm c blas waIwutr Ober �ci ti 1V Ces ' wt gbtiemwwajfu tpctt, afferain 9 c: b�titbfof �r f eran m rs bra wlg auig4;f aer�m *mc vrt blatcabtfm wfru an'deg~m rt blas Iaum bwWt� run f ncmai b14ianf tJobrrfan ebrut4net vnber Brwt Ober awawi*m.crriii btafwfm uww c on$ 'it ttu wttkc 5,10 $ m~C fca trc* cx u W�e�&mcyfbq4 Mabtt mmw rc a iiu bc Jcn ncI.end.*, blat s. a u:�rprd3 rw4ftf wm c ii,+ ttCC tn wa frr9fr orfab fcfbt enb c optiti bt -r9 ceDa ttDcn i34M sn Sem i
  52. 52. OCR Improvements Gaming Transcription
  53. 53. OCR Improvements Transcription Purposeful Gaming Looking at… Crowdsourcing Markup & Annotation
  54. 54. Purposeful Gaming DIGITALKOOT Joint project run by the National Library of Finland and Microtask to index the library's enormous archives so that they are searchable on the Internet for easier access to the Finnish cultural heritage. .
  55. 55. Purposeful Gaming DIGITALKOOT Launched on Feb 8 2011, nearly 110 000 participants completed over 8 million word fixing tasks by Nov 29 2012 DigiTalkoot enabled volunteers to participate in this fixing work by playing games. .
  56. 56. Purposeful gaming and BHL: engaging the public in improving and enhancing access to digital texts IMLS Grant Program: National Leadership Grants for Libraries Partners: Missouri Botanical Garden Harvard University Cornell University New York Botanical Garden P.I.: Trish Rose-Sandler, Missouri Botanical Garden Dates: Dec 2013 – Nov. 2015
  57. 57. Project objectives and benefits Test new means of crowdsourcing to support the enhancement of content in BHL Demonstrate if digital games are an effective tool for analyzing and improving digital outputs from OCR and transcription Benefits of gaming include: improved access to content by providing richer and more accurate data; an extension of limited staff resources; and exposure of library content to communities who may not know about the collections otherwise.
  58. 58. OCR Improvements German text interpreted by the OCR process as: “unb auf ben ©elnrgen be6 fublic{)en”
  59. 59. AOCR Improvements Different resulting texts from parsing the phrase: “und auf den Gebirgen des südlichen Deutschlands” (“and on the mountains of southern Germany”) IA OCR OCR 2 Transcriptio n 1 Transcriptio n 2 1 unb und und und Ok 2 den ben den den Ok 3 ©elnrgen ©ebirgen Bebirgen Gebirgen X 4 be6 des de5 des Chk 5 fublic{)en fublichen Füdlichen Südlichen X 6 £)eittfc{)(anb6 Deutfchlanbs Deutfchlands Deutschland s X
  60. 60. Purposeful Gaming
  61. 61. iDigBio‟s aOCR Hackathon Improve OCR parsing of labels with clear metrics (datasets, output formats, scoring algorithm) Libraries of regular expr. to clean up each field (different error correction for latitude/longitude coordinates than personal names or herbarium catalog numbers) Tool for classifying segments of the image before submitting to OCR Do a first pass of OCR to clean images before sending them to a second, 'real' pass of OCR
  62. 62. iDigBio‟s CITScribe Hackathon 1. Interoperability betweenpublic participation tools and biodiversity data systems, 2. Transcription quality assessment/quality control (QA/QC) and the reconciliation of replicatetranscriptions, 3. Integration of optical character recognition (OCR) into thetranscription workflow 4. User engagement
  63. 63. NfN & iDigBio‟s CITScribe Hackathon Jason Best‟s DarwinScore Ben Brumfield‟s Handwriting Gibberish Detector Dictionaries to improve crowdsourcing consensus (e.g., names of collectors, scientific names) Word Clouds created using n-gram scoring, faceting, and Solr for indexing + Carrot2 for specimen selection (visualize and explore of the use with a word of interest from the word cloud) and a data cleaning step (highlight infrequent words by the system).
  64. 64. NESCent EOL-BHL Research Sprint There is no place like home: Defining “habitat” for biodiversity science Robert D. Stevenson UMass Boston, Dept. of Biology, 100 Morrissey Blvd., Boston, MA 02125-3393 Carl Nordman (Natureserve) and Evangelos Pafilis Hellenic Centre for Marine Research, P.O. Box 2214, Heraklion, 71003, Crete, Greece
  65. 65. NESCent EOL-BHL Research Sprint Assessing Risk Status of Mexican Amphibians Through Data Mining. Esther Quintero and Bárbara Ayala National Commission for Knowledge and Use of Biodiversity (CONABIO) and Anne Thessen Marine Biological Laboratory and Arizona State University
  66. 66. Planning for global change: using species interactions in conservation Nicole F. Angeli, Emma P. Gomez, Margot A. Wood, Applied Biodiversity Sciences Program, Texas A&M University, College Station, Texas nangeli1@jhu.edu Tweet me @auratus_nicole and Javier Otegui University of Colorado-Boulder
  67. 67. There is no place like home: Defining “habitat” for biodiversity science Robert D. Stevenson UMass Boston, Dept. of Biology, 100 Morrissey Blvd., Boston, MA 02125-3393 Carl Nordman (Natureserve) Evangelos Pafilis Hellenic Centre for Marine Research, P.O. Box 2214, Heraklion, 71003, Crete, Greece http://epafilis.info/ , vagpafilis@gmail.com
  68. 68. Evolution in the usage of anatomical concepts in the biodiversity literature Todd Vision (tjv@bio.unc.edu), Prashanti Manda (manda.prashanti@gmail.com), and Dongye Meng (dmeng@cs.unc.edu) University of North Carolina at Chapel Hill
  69. 69. NESCent EOL-BHL Research Sprint Evolution in the usage of anatomical concepts in the biodiversity literature Todd Vision (tjv@bio.unc.edu), Prashanti Manda (manda.prashanti@gmail.com), and Dongye Meng University of North Carolina at Chapel Hill
  70. 70. Some preliminary observations… Our API seemed to work fine Access via a taxon (or a group), for example: “I want to harvest all pages with names from this taxon (Chordata) or this common name (Vertebrate)”. Groups started getting results after 2.5 days. The structure of BHL was explained so researchers could understand the title, item, page and part levels and define what they wanted. Ex: one group was looking for terms in the titles and the parts‟ titles. Some others said they would Harvest the OCR from IA although they will not be able to harvest the text on a page by page granularity (only item level).
  71. 71. NESCent EOL-BHL Research Sprint There is no place like home: Defining “habitat” for biodiversity science Robert D. Stevenson UMass Boston, Dept. of Biology, 100 Morrissey Blvd., Boston, MA 02125-3393 Carl Nordman (Natureserve) and Evangelos Pafilis Hellenic Centre for Marine Research, P.O. Box 2214, Heraklion, 71003, Crete, Greece
  72. 72. Mining Biodiversity
  73. 73. Mining Biodiversity Mining Biodiversity: Enriching Biodiversity Heritage with Text Mining and Social Media One of the international projects that won in the third round of the 2013 Digging Into Data Challenge Promote the development of innovative computational techniques to apply into big data in the humanities and social sciences The National Centre for Text Mining (UK) Missouri Botanical Garden (US) Dalhousie University's Big Data Analytics Institute (Canada) Social Media Lab (Canada)
  74. 74. MiBIO: Mining Biodiversity 1. Automatic error correction of OCR text errors. 2. Crowdsource annotation of legacy texts with semantic metadata. 3. Adapt text mining techniques to extract terminology, entities and significant events automatically and to track terminology evolution over time. 4. Use Interactive visualization techniques to help users manage search results through next generation browsing capabilities, assisted by a semantic similarity network of important terms and entities. 5. Design of a social media layer, serving as an environment for diverse users to interact and collaborate on science, public education, awareness and outreach.
  75. 75. MiBIO: Mining Biodiversity
  76. 76. Crowdsource Markup Display text Species Profile Model category General/summary TaxonBiology Geographic range Distribution Habitat Habitat Food sources and feeding behavior TrophicStrategy Physical description (general) Description Physical description (detailed morphology) DiagnosticDescription
  77. 77. Visit to NaCTeM, Feb. 17, 2014
  78. 78. NaCTeM‟s Biodiversity- relevant tools
  79. 79. ANNNOTATION PLATFORM
  80. 80. Remote Processing Workflows processed on remote machines. No attendance needed Workflows GUI for creating single-flow and multi-branch workflows Workflow Designer User Interaction Annotation Editor allows for making changes while processingAnnotator/Curator WebService Third-party applications Processing Components Data (de)serialisation, search engines, NLP, NER, etc. Developers
  81. 81. Workflows view
  82. 82. Processes View
  83. 83. Documents view
  84. 84. Workflow editor
  85. 85. Workflow as a Web service
  86. 86. Workflow as a Web service http://argo.nactem.ac.uk/test/services/webservice/314 INPUT OUTPUT
  87. 87. NAMED ENTITY RECOGNISERS AND NORMALISERS
  88. 88. ✔ ✔ ✔ ✔ ✔
  89. 89. Automatically recognised named entities
  90. 90. Linking to external dictionaries
  91. 91. Species and habitat recognition
  92. 92. EVENT EXTRACTORS
  93. 93. Events: associations between entities
  94. 94. SEMANTIC SEARCH
  95. 95. TERM EXTRACTION
  96. 96. Ryerson University SocialLab‟s Netlytic.org
  97. 97. http://miningbiodiversity.comhttp://miningbiodiversity.org/
  98. 98. Thank you William Ulate BHL Technical Director Missouri Botanical Garden william.ulate@mobot.org Skype: william_ulate_r
  99. 99. Thank you! And thanks to Bianca Crowley for the workflow slides

×