Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018

128 views

Published on

EuropeanaTech Conference 15 & 16 May 2018, Rotterdam, the Netherlands

  • Be the first to comment

  • Be the first to like this

When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018

  1. 1. When automated analysis goes wrong Tristan Roddis Cogapp EuropeanaTech 2018
  2. 2. Tristan Roddis. Cogapp • Digital publishing systems • Website and mobile app development • Digital strategy • Innovation and research projects
  3. 3. Tristan Roddis. Cogapp In this session • Term extraction • Machine learning • Computer vision • Successes • FAILURES
  4. 4. Tristan Roddis. Cogapp Failure: Term extraction
  5. 5. Tristan Roddis. Cogapp Success: Constrain results
  6. 6. Tristan Roddis. Cogapp Failure: term extraction
  7. 7. Tristan Roddis. Cogapp Failure: term extraction
  8. 8. Tristan Roddis. Cogapp Failure: term extraction
  9. 9. Tristan Roddis. Cogapp Failure 1: term extraction
  10. 10. Tristan Roddis. Cogapp Failure 1: term extraction
  11. 11. Tristan Roddis. Cogapp Failure: term extraction
  12. 12. Tristan Roddis. Cogapp Success: constrain results (using a human)
  13. 13. Tristan Roddis. Cogapp Failure: Interesting images
  14. 14. Tristan Roddis. Cogapp The problem
  15. 15. Tristan Roddis. Cogapp The problem • No useful information
  16. 16. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.56
  17. 17. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.54
  18. 18. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.20
  19. 19. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.19
  20. 20. Tristan Roddis. Cogapp
  21. 21. Tristan Roddis. Cogapp
  22. 22. Tristan Roddis. Cogapp
  23. 23. Tristan Roddis. Cogapp
  24. 24. Tristan Roddis. Cogapp
  25. 25. Tristan Roddis. Cogapp
  26. 26. Tristan Roddis. Cogapp
  27. 27. Tristan Roddis. Cogapp
  28. 28. Tristan Roddis. Cogapp
  29. 29. Tristan Roddis. Cogapp
  30. 30. Tristan Roddis. Cogapp
  31. 31. Tristan Roddis. Cogapp
  32. 32. Tristan Roddis. Cogapp Success: Train your own neural network
  33. 33. Tristan Roddis. Cogapp • Automatically tag, organize, and search visual content with machine learning. • Create concepts
  34. 34. Tristan Roddis. Cogapp • Existing concepts: • Illustration • Drawing
  35. 35. Tristan Roddis. Cogapp • Negative concept: • Text • Manuscripts
  36. 36. Tristan Roddis. Cogapp • Combination of two custom concepts • arabic_manuscript • arabic_ manuscript_with_image
  37. 37. Tristan Roddis. Cogapp 45 images 45 images
  38. 38. Tristan Roddis. Cogapp 26 images 26 images
  39. 39. Tristan Roddis. Cogapp • Python script to create sets and train • Test and train
  40. 40. Adrian Hindle Tristan Roddis
  41. 41. Tristan Roddis. Cogapp
  42. 42. Adrian Hindle Tristan Roddis
  43. 43. Adrian Hindle Tristan Roddis
  44. 44. Tristan Roddis. Cogapp
  45. 45. Tristan Roddis. Cogapp Next steps • Custom manifest • Add to search results • Boost • Filter • Display screens etc.
  46. 46. Tristan Roddis. Cogapp Other uses • Blank sheets vs. writing • Handwritten vs. typewritten • [your collection-specific question here]
  47. 47. Tristan Roddis. Cogapp Failure: Finding similar images
  48. 48. Tristan Roddis. Cogapp Scikit-image
  49. 49. Tristan Roddis. Cogapp Scikit-image
  50. 50. Tristan Roddis. Cogapp Scikit-image
  51. 51. Tristan Roddis. Cogapp Scikit-image
  52. 52. Tristan Roddis. Cogapp Scikit-learn • Hand-rolled machine learning is not easy • Especially for this problem
  53. 53. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction
  54. 54. Tristan Roddis. Cogapp Term extraction • Clarifai • Google Vision API • Microsoft Computer Vision
  55. 55. Tristan Roddis. Cogapp Clarifai
  56. 56. Tristan Roddis. Cogapp Google Vision API
  57. 57. Tristan Roddis. Cogapp MS Computer vision
  58. 58. Tristan Roddis. Cogapp Images • Sourced from Nationalmuseum Sweden • Using Europeana API for discovery • 2000 images • http://labs.cogapp.com/iiif-ml/
  59. 59. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction • Tune thresholds • Optionally correlate with “known good” terms • Optionally allow people to flag incorrect terms
  60. 60. Tristan Roddis. Cogapp Failure: when computers get it wrong
  61. 61. Tristan Roddis. Cogapp Failure: when computers get it wrong
  62. 62. Tristan Roddis. Cogapp Failure: when computers get it wrong
  63. 63. Tristan Roddis. Cogapp Failure: when computers get it wrong
  64. 64. Tristan Roddis. Cogapp Success: Embrace failure • Original goal: Finding similar images • It doesn’t matter what the computer sees!
  65. 65. Tristan Roddis. Cogapp Success: Hide the details
  66. 66. Tristan Roddis. Cogapp Unexpected success: Optical music recognition
  67. 67. http://labs.cogapp.com/nls-omr/wavs/91387296.wav
  68. 68. http://labs.cogapp.com/nls-omr/
  69. 69. Tristan Roddis. Cogapp Failure 4: Automated captions
  70. 70. A vase of flowers on a display
  71. 71. Bertel Thorvaldsen
  72. 72. Bertel Thorvaldsen
  73. 73. Bertel Thorvaldsen sitting on a suitcase
  74. 74. Emanuel Swedenborg
  75. 75. Emanuel Swedenborgsitting in front of a laptop
  76. 76. Person using a phone
  77. 77. Person sitting in a chair talking on a cell phone
  78. 78. A man and a woman taking a selfie
  79. 79. A cat with its mouth open
  80. 80. A pizza sitting on top of a window
  81. 81. A man riding a bear in the water
  82. 82. A teddy bear
  83. 83. A group of sheep standing on top of a horse
  84. 84. A group of sheep standing on top of a book
  85. 85. Tristan Roddis. Cogapp Conclusions • Embrace failure • Find workarounds • Constrain results • Tune thresholds • Hide the details • Add humans to the mix
  86. 86. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience Neil Hawkins, Cogapp • Playing ancient music without an instrument Tristan Roddis, Cogapp • Iconclass and AI Gro Benedikte Pedersen, NMAAD Norway • Computer Vision so good Shelley Bernstein, Barnes Foundation
  87. 87. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF, Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience, Neil Hawkins, Cogapp • Playing ancient music without an instrument, Tristan Roddis, Cogapp • Iconclass and AI, Gro Benedikte Pedersen, National Museum of Art, Architecture and Design in Norway • Computer Vision so good, Shelley Bernstein, Barnes Foundation • Quantifying Kissenger, Micki Kaufman • Comparison of different computer-vision APIs, Cogapp Labs.
  88. 88. Thank you. Questions? Tristan Roddis @tristan_roddis tristanr@cogapp.com

×