When automated analysis goes wrong by Tristan Roddis - EuropeanaTech Conference 2018

1. When automated analysis goes wrong Tristan Roddis Cogapp EuropeanaTech 2018

2. Tristan Roddis. Cogapp • Digital publishing systems • Website and mobile app development • Digital strategy • Innovation and research projects

3. Tristan Roddis. Cogapp In this session • Term extraction • Machine learning • Computer vision • Successes • FAILURES

4. Tristan Roddis. Cogapp Failure: Term extraction

5. Tristan Roddis. Cogapp Success: Constrain results

6. Tristan Roddis. Cogapp Failure: term extraction

9. Tristan Roddis. Cogapp Failure 1: term extraction

10. Tristan Roddis. Cogapp Failure 1: term extraction

12. Tristan Roddis. Cogapp Success: constrain results (using a human)

13. Tristan Roddis. Cogapp Failure: Interesting images

14. Tristan Roddis. Cogapp The problem

15. Tristan Roddis. Cogapp The problem • No useful information

16. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.56

17. Adrian Hindle, Tristan Roddis. Cogapp A Syrian Voyage in Central and South America http://www.qdl.qa/en/archive/qnlhc/12957.54

18. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.20

19. Adrian Hindle, Tristan Roddis. Cogapp "Nourishment for the Ailing" and "Nourishment for the Healthy" http://www.qdl.qa/en/archive/qnlhc/9549.19

20. Tristan Roddis. Cogapp

32. Tristan Roddis. Cogapp Success: Train your own neural network

33. Tristan Roddis. Cogapp • Automatically tag, organize, and search visual content with machine learning. • Create concepts

34. Tristan Roddis. Cogapp • Existing concepts: • Illustration • Drawing

35. Tristan Roddis. Cogapp • Negative concept: • Text • Manuscripts

36. Tristan Roddis. Cogapp • Combination of two custom concepts • arabic_manuscript • arabic_ manuscript_with_image

37. Tristan Roddis. Cogapp 45 images 45 images

38. Tristan Roddis. Cogapp 26 images 26 images

39. Tristan Roddis. Cogapp • Python script to create sets and train • Test and train

40. Adrian Hindle Tristan Roddis

45. Tristan Roddis. Cogapp Next steps • Custom manifest • Add to search results • Boost • Filter • Display screens etc.

46. Tristan Roddis. Cogapp Other uses • Blank sheets vs. writing • Handwritten vs. typewritten • [your collection-specific question here]

47. Tristan Roddis. Cogapp Failure: Finding similar images

48. Tristan Roddis. Cogapp Scikit-image

52. Tristan Roddis. Cogapp Scikit-learn • Hand-rolled machine learning is not easy • Especially for this problem

53. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction

54. Tristan Roddis. Cogapp Term extraction • Clarifai • Google Vision API • Microsoft Computer Vision

55. Tristan Roddis. Cogapp Clarifai

56. Tristan Roddis. Cogapp Google Vision API

57. Tristan Roddis. Cogapp MS Computer vision

58. Tristan Roddis. Cogapp Images • Sourced from Nationalmuseum Sweden • Using Europeana API for discovery • 2000 images • http://labs.cogapp.com/iiif-ml/

59. Tristan Roddis. Cogapp Success: Use pre-trained models for term extraction • Tune thresholds • Optionally correlate with “known good” terms • Optionally allow people to flag incorrect terms

60. Tristan Roddis. Cogapp Failure: when computers get it wrong

64. Tristan Roddis. Cogapp Success: Embrace failure • Original goal: Finding similar images • It doesn’t matter what the computer sees!

65. Tristan Roddis. Cogapp Success: Hide the details

66. Tristan Roddis. Cogapp Unexpected success: Optical music recognition

67. http://labs.cogapp.com/nls-omr/wavs/91387296.wav

68. http://labs.cogapp.com/nls-omr/

69. Tristan Roddis. Cogapp Failure 4: Automated captions

70. A vase of flowers on a display

71. Bertel Thorvaldsen

72. Bertel Thorvaldsen

73. Bertel Thorvaldsen sitting on a suitcase

74. Emanuel Swedenborg

75. Emanuel Swedenborgsitting in front of a laptop

76. Person using a phone

77. Person sitting in a chair talking on a cell phone

78. A man and a woman taking a selfie

79. A cat with its mouth open

80. A pizza sitting on top of a window

81. A man riding a bear in the water

82. A teddy bear

83. A group of sheep standing on top of a horse

84. A group of sheep standing on top of a book

85. Tristan Roddis. Cogapp Conclusions • Embrace failure • Find workarounds • Constrain results • Tune thresholds • Hide the details • Add humans to the mix

86. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience Neil Hawkins, Cogapp • Playing ancient music without an instrument Tristan Roddis, Cogapp • Iconclass and AI Gro Benedikte Pedersen, NMAAD Norway • Computer Vision so good Shelley Bernstein, Barnes Foundation

87. Tristan Roddis. Cogapp Further reading • Automated image analysis with IIIF, Adrian Hindle, Cogapp • Using Artificial Intelligence to enhance User Experience, Neil Hawkins, Cogapp • Playing ancient music without an instrument, Tristan Roddis, Cogapp • Iconclass and AI, Gro Benedikte Pedersen, National Museum of Art, Architecture and Design in Norway • Computer Vision so good, Shelley Bernstein, Barnes Foundation • Quantifying Kissenger, Micki Kaufman • Comparison of different computer-vision APIs, Cogapp Labs.

88. Thank you. Questions? Tristan Roddis @tristan_roddis tristanr@cogapp.com