Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Geometric and Semantic Matching for Cultural Heritage Artefacts

706 views

Published on

By Stephen C. Philips
IT Innovation

GRAVITATE participating @ Understanding Europe – Promoting the European Public and Cultural Space

  • Be the first to comment

  • Be the first to like this

Geometric and Semantic Matching for Cultural Heritage Artefacts

  1. 1. Geometric and Semantic Matching for Cultural Heritage Artefacts Stephen C. Phillips IT Innovation
  2. 2. https://www.flickr.com/photos/lunamodule/ © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 2
  3. 3. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 3
  4. 4. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 4
  5. 5. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 5
  6. 6. Front Back Faded Cracked Eroded © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
  7. 7. British Museum, London Cyprus Museum, Cyprus Ashmolean Museum, Oxford Fitzwilliam Museum, Cambridge © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
  8. 8. Jigsaws in 3D most pieces missing all the edges worn off many puzzles jumbled together pieces spread across many countries © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
  9. 9. Re-Assembly Re-Unification Re-Association _______________ Similarity Search © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 9
  10. 10. Re-Assembly You have some (eroded) pieces. The computer fits them together. This is hard in 2 dimensions, never mind in 3D! Localised geometry 3D descriptors Localised detail Colour descriptors © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 10
  11. 11. Re-Unification Head in the British Museum Statue in the Cyprus Museum © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 11  Re-unify 3D models of the statue  3D print copies for re-unification  Re-unify statue in a virtual museum
  12. 12. Re-Association Artefact with semantic description Material, decoration, size, shape, period, style, glaze, … Data in another museum catalogue © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 12
  13. 13. Re-Association Find similarities leading to new insights about past cultures © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 13
  14. 14. How? © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 14
  15. 15. What’s the data? © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 15
  16. 16. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 16 Fragments of Terracotta Statues from Salamis
  17. 17. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 17 We need to Gather all the Knowledge Catalogue data with text descriptions Archaeological papers Excavation notes Chemical analysis X-ray fluorescence 3D scanning  models
  18. 18. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 18 We need to Gather all the Knowledge Catalogue data with text descriptions Archaeological papers Excavation notes Chemical analysis X-ray fluorescence 3D scanning  models
  19. 19. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 19 We need to Gather all the Knowledge Catalogue data with text descriptions Archaeological papers Excavation notes The British Museum has 2.5M objects described in CIDOC CRM Catalogue data from Ashmolean, Fitzwilliam and Cyprus museums is now mapped to the same data model Object  hasNote  “…lots of free-form text added by the curator” Using Natural Language Processing to extract and encode meaning from this text: • References to papers, to catalogue entries • Parts and features • Conservation condition • Measurements • …  CIDOC CRM / CRMarchaeo / etc
  20. 20. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 20 We need to Gather all the Knowledge Chemical analysis X-ray fluorescence 3D scanning  models Volume, area thickness, curvature, … Colour, texture, distance from convex hull, … Feature detection Faceting: front / back / fracture Part annotation CRMdig CH Artefact Partonomy All fragments (211) are scanned, in a variety of resolutions and with a variety of scanners
  21. 21. Faceting  Assume local fold model  Measure local fold angle  Select locally salient angles  Connect to form facet curves  Determine the facet types  Robust for our fragments  Very few parameters © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 21 90ᵒ
  22. 22. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 22 Feature Detection  Morphological and stylistic feature detection and characterization  Reasoning on similarity among fragments −1 −0.5 0 0.5 1 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 x y
  23. 23. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 23 Using CH Artefact Partonomy within CIDOC-CRM
  24. 24. How do I use it? © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 24
  25. 25. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 25 Six views in the dashboard  Inspection  Users can inspect 3D fragments and metadata and run a search  Fragment  3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated  ReAssembly  Exploration  Datasets can be explored according to specific selected properties  History  To preserve the list of operations performed in the session  Clipboard  To save all the data the user is interested in (e.g. notes, models, annotation)
  26. 26. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 26 Six views in the dashboard  Inspection  Users can inspect 3D fragments and metadata and run a search  Fragment  3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated  ReAssembly  Exploration  Datasets can be explored according to specific selected properties  History  To preserve the list of operations performed in the session  Clipboard  To save all the data the user is interested in (e.g. notes, models, annotation)
  27. 27. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 27 Six views in the dashboard  Inspection  Users can inspect 3D fragments and metadata and run a search  Fragment  3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated  ReAssembly  Exploration  Datasets can be explored according to specific selected properties  History  To preserve the list of operations performed in the session  Clipboard  To save all the data the user is interested in (e.g. notes, models, annotation)
  28. 28. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 28 Six views in the dashboard  Inspection  Users can inspect 3D fragments and metadata and run a search  Fragment  3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated  ReAssembly  Exploration  Datasets can be explored according to specific selected properties  History  To preserve the list of operations performed in the session  Clipboard  To save all the data the user is interested in (e.g. notes, models, annotation)
  29. 29. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 29 Six views in the dashboard  Inspection  Users can inspect 3D fragments and metadata and run a search  Fragment  3D assets can be processed (faceting, geometric characterisation, feature identification) and annotated  ReAssembly  Exploration  Datasets can be explored according to specific selected properties  History  To preserve the list of operations performed in the session  Provenance and argumentation  Clipboard  To save all the data the user is interested in (e.g. notes, models, annotation)
  30. 30. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016
  31. 31. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 31 Prototype
  32. 32. And Finally: ReAssembly Use all the data to guide the process © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 32 Selection by Similarity Positioning clues Matching Mating Jigsaws in 3D most pieces missing all the edges worn off many puzzles jumbled together pieces spread across many countries
  33. 33. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 33 Mannequin
  34. 34. Matching: geometry-based © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 34
  35. 35. © Copyright University of Southampton IT Innovation Centre and other GRAVITATE partners, 2015-2016 35 Mating  Mating digitally mimics gluing of the fragments  We place selected fragments in their optimal relative position based on:  Geometric complementarity  Skin pattern continuity  Semantic constraints  Global alignment  Mathematical morphology used  Final approval of proposed assemblage is requested of the user (SotA mating from predecessor PRESIOUS)
  36. 36. GRAVITATE-PROJECT.EU Coordinator: Stephen C Phillips scp@it-innovation.soton.ac.uk

×