When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018
•
1 like•336 views
The document discusses successes and failures of automated analysis of documents and images. It describes attempts at term extraction, image recognition, and generating captions that either succeeded with human oversight and constraints or failed outright. The key lessons are to embrace failures, find workarounds like pre-trained models or limiting results, and incorporate human input to improve automated analysis.
Recommended
More Related Content
More from Europeana
More from Europeana (20)
Recently uploaded
Recently uploaded (19)
When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018
- 1. When automated analysis goes wrong Tristan Roddis Cogapp EuropeanaTech 2018
- 2. Tristan Roddis. Cogapp • Digital publishing systems • Website and mobile app development • Digital strategy • Innovation and research projects
- 3. Tristan Roddis. Cogapp In this session • Term extraction • Machine learning • Computer vision • Successes • FAILURES
- 4. Tristan Roddis. Cogapp Failure: Term extraction
- 5. Tristan Roddis. Cogapp Success: Constrain results
- 6. Tristan Roddis. Cogapp Failure: term extraction
- 7. Tristan Roddis. Cogapp Failure: term extraction
- 8. Tristan Roddis. Cogapp Failure: term extraction
- 9. Tristan Roddis. Cogapp Failure 1: term extraction
- 10. Tristan Roddis. Cogapp Failure 1: term extraction
- 11. Tristan Roddis. Cogapp Failure: term extraction
- 12. Tristan Roddis. Cogapp Success: constrain results (using a human)
- 13. Tristan Roddis. Cogapp Failure: Interesting images
- 14. Tristan Roddis. Cogapp The problem
- 15. Tristan Roddis. Cogapp The problem • No useful information
- 16. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.56
- 17. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.54
- 18. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.20
- 19. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.19
- 32. Tristan Roddis. Cogapp Success: Train your own neural network
- 33. Tristan Roddis. Cogapp • Automatically tag, organize, and search visual content with machine learning. • Create concepts
- 34. Tristan Roddis. Cogapp • Existing concepts: • Illustration • Drawing
- 35. Tristan Roddis. Cogapp • Negative concept: • Text • Manuscripts
- 36. Tristan Roddis. Cogapp • Combination of two custom concepts • arabic_manuscript • arabic_ manuscript_with_image
- 37. Tristan Roddis. Cogapp 45 images 45 images
- 38. Tristan Roddis. Cogapp 26 images 26 images
- 39. Tristan Roddis. Cogapp • Python script to create sets and train • Test and train
- 45. Tristan Roddis. Cogapp Next steps • Custom manifest • Add to search results • Boost • Filter • Display screens etc.
- 46. Tristan Roddis. Cogapp Other uses • Blank sheets vs. writing • Handwritten vs. typewritten • [your collection-specific question here]
- 47. Tristan Roddis. Cogapp Failure: Finding similar images
- 52. Tristan Roddis. Cogapp Scikit-learn • Hand-rolled machine learning is not easy • Especially for this problem
- 53. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction
- 54. Tristan Roddis. Cogapp Term extraction • Clarifai • Google Vision API • Microsoft Computer Vision
- 56. Tristan Roddis. Cogapp Google Vision API
- 57. Tristan Roddis. Cogapp MS Computer vision
- 58. Tristan Roddis. Cogapp Images • Sourced from Nationalmuseum Sweden • Using Europeana API for discovery • 2000 images • http://labs.cogapp.com/iiif-ml/
- 61. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction • Tune thresholds • Optionally correlate with “known good” terms • Optionally allow people to flag incorrect terms
- 62. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 63. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 64. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 65. Tristan Roddis. Cogapp Failure: when computers get it wrong
- 66. Tristan Roddis. Cogapp Success: Embrace failure • Original goal: Finding similar images • It doesn’t matter what the computer sees!
- 67. Tristan Roddis. Cogapp Success: Hide the details
- 68. Tristan Roddis. Cogapp Unexpected success: Optical music recognition
- 72. Tristan Roddis. Cogapp Failure 4: Automated captions
- 73. A vase of flowers on a display
- 77. Bertel Thorvaldsen sitting on a suitcase
- 80. Emanuel Swedenborgsitting in front of a laptop
- 81. Person using a phone
- 82. Person sitting in a chair talking on a cell phone
- 83. A man and a woman taking a selfie
- 84. A cat with its mouth open
- 85. A pizza sitting on top of a window
- 86. A man riding a bear in the water
- 87. A teddy bear
- 88. A group of sheep standing on top of a horse
- 89. A group of sheep standing on top of a book
- 90. Tristan Roddis. Cogapp Conclusions • Embrace failure • Find workarounds • Constrain results • Tune thresholds • Hide the details • Add humans to the mix
- 91. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience Neil Hawkins, Cogapp • Playing ancient music without an instrument Tristan Roddis, Cogapp • Iconclass and AI Gro Benedikte Pedersen, NMAAD Norway • Computer Vision so good Shelley Bernstein, Barnes Foundation
- 92. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF, Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience, Neil Hawkins, Cogapp • Playing ancient music without an instrument, Tristan Roddis, Cogapp • Iconclass and AI, Gro Benedikte Pedersen, National Museum of Art, Architecture and Design in Norway • Computer Vision so good, Shelley Bernstein, Barnes Foundation • Quantifying Kissenger, Micki Kaufman • Comparison of different computer-vision APIs, Cogapp Labs.