When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018
•
1 like•336 views
EuropeanaTech Conference 15 & 16 May 2018, Rotterdam, the Netherlands
Recommended
Recommended
More Related Content
More from Europeana
More from Europeana (20)
Recently uploaded
Recently uploaded (20)
When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018
- 1. When automated analysis goes wrong Tristan Roddis Cogapp EuropeanaTech 2018
- 2. Tristan Roddis. Cogapp • Digital publishing systems • Website and mobile app development • Digital strategy • Innovation and research projects
- 3. Tristan Roddis. Cogapp In this session • Term extraction • Machine learning • Computer vision • Successes • FAILURES
- 4. Tristan Roddis. Cogapp Failure: Term extraction
- 5. Tristan Roddis. Cogapp Success: Constrain results
- 6. Tristan Roddis. Cogapp Failure: term extraction
- 7. Tristan Roddis. Cogapp Failure: term extraction
- 8. Tristan Roddis. Cogapp Failure: term extraction
- 9. Tristan Roddis. Cogapp Failure 1: term extraction
- 10. Tristan Roddis. Cogapp Failure 1: term extraction
- 11. Tristan Roddis. Cogapp Failure: term extraction
- 12. Tristan Roddis. Cogapp Success: constrain results (using a human)
- 13. Tristan Roddis. Cogapp Failure: Interesting images
- 14. Tristan Roddis. Cogapp The problem
- 15. Tristan Roddis. Cogapp The problem • No useful information
- 16. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.56
- 17. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.54
- 18. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.20
- 19. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.19
- 32. Tristan Roddis. Cogapp Success: Train your own neural network
- 33. Tristan Roddis. Cogapp • Automatically tag, organize, and search visual content with machine learning. • Create concepts
- 34. Tristan Roddis. Cogapp • Existing concepts: • Illustration • Drawing
- 35. Tristan Roddis. Cogapp • Negative concept: • Text • Manuscripts
- 36. Tristan Roddis. Cogapp • Combination of two custom concepts • arabic_manuscript • arabic_ manuscript_with_image
- 37. Tristan Roddis. Cogapp 45 images 45 images
- 38. Tristan Roddis. Cogapp 26 images 26 images
- 39. Tristan Roddis. Cogapp • Python script to create sets and train • Test and train
- 45. Tristan Roddis. Cogapp Next steps • Custom manifest • Add to search results • Boost • Filter • Display screens etc.
- 46. Tristan Roddis. Cogapp Other uses • Blank sheets vs. writing • Handwritten vs. typewritten • [your collection-specific question here]
- 47. Tristan Roddis. Cogapp Failure: Finding similar images
- 52. Tristan Roddis. Cogapp Scikit-learn • Hand-rolled machine learning is not easy • Especially for this problem
- 53. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction
- 54. Tristan Roddis. Cogapp Term extraction • Clarifai • Google Vision API • Microsoft Computer Vision
- 56. Tristan Roddis. Cogapp Google Vision API
- 57. Tristan Roddis. Cogapp MS Computer vision
- 58. Tristan Roddis. Cogapp Images • Sourced from Nationalmuseum Sweden • Using Europeana API for discovery • 2000 images • http://labs.cogapp.com/iiif-ml/
- 61. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction • Tune thresholds • Optionally correlate with “known good” terms • Optionally allow people to flag incorrect terms
- 62. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 63. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 64. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 65. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 66. Tristan Roddis. Cogapp Success: Embrace failure • Original goal: Finding similar images • It doesn’t matter what the computer sees!
- 67. Tristan Roddis. Cogapp Success: Hide the details
- 68. Tristan Roddis. Cogapp Unexpected success: Optical music recognition
- 72. Tristan Roddis. Cogapp Failure 4: Automated captions
- 73. A vase of flowers on a display
- 77. Bertel Thorvaldsen sitting on a suitcase
- 80. Emanuel Swedenborgsitting in front of a laptop
- 81. Person using a phone
- 82. Person sitting in a chair talking on a cell phone
- 83. A man and a woman taking a selfie
- 84. A cat with its mouth open
- 85. A pizza sitting on top of a window
- 86. A man riding a bear in the water
- 87. A teddy bear
- 88. A group of sheep standing on top of a horse
- 89. A group of sheep standing on top of a book
- 90. Tristan Roddis. Cogapp Conclusions • Embrace failure • Find workarounds • Constrain results • Tune thresholds • Hide the details • Add humans to the mix
- 91. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience Neil Hawkins, Cogapp • Playing ancient music without an instrument Tristan Roddis, Cogapp • Iconclass and AI Gro Benedikte Pedersen, NMAAD Norway • Computer Vision so good Shelley Bernstein, Barnes Foundation
- 92. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF, Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience, Neil Hawkins, Cogapp • Playing ancient music without an instrument, Tristan Roddis, Cogapp • Iconclass and AI, Gro Benedikte Pedersen, National Museum of Art, Architecture and Design in Norway • Computer Vision so good, Shelley Bernstein, Barnes Foundation • Quantifying Kissenger, Micki Kaufman • Comparison of different computer-vision APIs, Cogapp Labs.