R2-D2 analysed your collection images and here's what he found
•Download as PPTX, PDF•
1 like•428 views
Techniques for the automated review of images continue to improve. Analysis of colours and patterns within your collection images can provide new connections between collection items. Artificial intelligence can work out the subject of an image with increasing accuracy, sometimes rivalling the descriptions created manually by cataloguers. This presentation looks at some of the emerging tools in this field and how they might be applied to your image collections.
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
R2-D2 analysed your collection images and here's what he found
- 1. R2-D2 analysed your collection images and here’s what he found Paul Rowe National Digital Forum, Nov 2017
- 2. Why do it? Provide access to data already there.
- 3. Create text data that we don’t have time to enter manually
- 4. Find patterns: Colours, lines, light, space
- 6. Youtube – Closed Captions
- 7. IBM Watson speech to text
- 8. Image analysis isn’t easy
- 9. Cogapp test site Microsoft / Google / Clarifai Nationalmuseum, Sweden: Painting - "Hedvig Sofia, princess of Sweden"
- 10. Captioning is the weakest link Nationalmuseum, Sweden: Painting Madonna with child
- 13. … and the captions can be surprisingly precise Nationalmuseum, Sweden - Painting, Landskap met kor
- 15. Colour Analysis
- 16. Mapping to smaller palette
- 17. Combine it all and find “house AND white AND landscape”
- 18. These are the droids you’re looking for Websites Barnes Foundation: https://collection.barnesfoundation.org/ Cogapp image tagging test site: http://labs.cogapp.com/iiif-ml/ Invisible Australians: http://invisibleaustralians.org/faces/ Sarjeant Gallery: https://collection.sarjeant.org.nz Tools Artrendex ArtPI: http://www.artrendex.com/ Clarifai: https://www.clarifai.com/demo Color Thief: http://lokeshdhakar.com/projects/color-thief/ Google Cloud Vision API: https://cloud.google.com/vision/docs/ IBM Watson Speech to Text: https://www.ibm.com/watson/services/speech-to-text/ Microsoft Azure Cognitive Services: https://azure.microsoft.com/en- us/try/cognitive-services/ OpenCV Library: https://sourceforge.net/projects/opencvlibrary/
- 19. “That’s no moon” Paul Rowe, Vernon Systems @armchair_caver Generated Subject Tags: Moon Planet Satellite Space
Editor's Notes
- Kia ora. I’m Paul Rowe from Vernon Systems. I’ve been investigating options for automated image analysis, so today’s an opportunity to share what I (or more precisely, R2-D2) has found.
- Why do it in the first place? One reason is to provide access to data already in those images. For example, is the image portrait or landscape orientation? This can then be a way for people to filter their collection searches. Here’s all of the Gretchen Albrecht works from Sarjeant Gallery where the image is in landscape orientation.
- You could use computer analysis to add text data that you haven’t had time to put in manually, like subject keywords or captions. This example is just using the free single image upload screen that Google Cloud provides. We can see that even the cat breed has been correctly detected.
- You can also find patterns that are harder to spot with traditional cataloguing. The Barnes Foundation uses a product called ArtPI to find patterns based on colours, lines, light and space within their art collection. Here are the objects featuring horizontal lines.
- … and lastly, some of the options still feel like magic. Tim Sherratt used the Open Source Computer Vision Library to identify the faces of people in source documents. This provided a dramatic visualisation of those who suffered discrimination under the White Australia Policy and it provided a new way to discover the source documents.
- As a side note, there are already options to analyse audio and video files, not just image files. For example, when you upload a video, Youtube can automatically generate closed captions to help make the content more accessible. The captions aren’t perfect, so usually there’s some manual editing required. This example is from the Bad Lip Reading of Star Wars.
- IBM’s Watson product can do a similar thing with audio, generating text transcripts for you. This is a huge win when you want to provide searching in areas such as oral history.
- I’ll now look at what can be achieved through image analysis. Many images are a challenge even for people to decipher, so don’t be surprised when your painting of a Chihuahua gets tagged as a muffin. The web development company Cogapp have a great site with tests of the image analysis tools provided by Google, Microsoft and Clarifai.
- The site provides a good comparison of the strengths and weaknesses of each tool. This example is a portrait of the princess of Sweden. Clarifai labelled it as Renaissance Cavalry. The Renaissance cavalry weren’t very effective, but they had the *BEST* uniforms.
- Creating a full sentence caption is particular complex and I think it’s the weakest element in the tools available to us. Here we see a typical problem where the caption is describing a person using a popular cell phone model shaped like baby.
- … but the results can be amazing. Here Google Vision has successfully tagged all of the men with facial hair in the Sarjeant Gallery collection, just in time for Movember. To cope with the tags not being perfect we added options for the curators to remove selected tags.
- Automated tags don’t have to be perfect to create interesting results. All of these works have been tagged as ‘circle’. Most of them aren’t strictly circles, but the tags provide new connections to other works with similar shapes and lines which wouldn’t normally be connected in your collection catalogue.
- … and sometimes even the captions created for artworks can be comparable to a caption that any cataloguer might have added.
- It’s also possible to do text analysis to transcribe the text within images. The quality depends on the clarity of the text in the image, but the National Libraries’ Papers Past website is an example of what can be achieved through text analysis of photographs or scanned documents.
- The popular colour analysis tools available are all good. The challenge is that they extract the most dominant colours from the original palette in the image – usually a 16 million colour palette. Unless your collection is enormous, hardly any images will share the same exact colour.
- These original colours need to be matched to a smaller palette – in this example we’re using the 140 named web colours. With a smaller palette, more images will share the same matching colour. However, as the palette gets smaller the colours you are displaying will get further away from the original precise image colours.
- There’s then the potential to search on several of these elements together: keywords, colour names, image orientation. Here’s a search on “house AND white AND landscape” where all of these results were missing at least one of these descriptive elements in the original catalogue record.
- I’ve added a slide with references to all of the websites and tools I’ve mentioned today and I’ve uploaded the slides to Slideshare.
- To sum up, the current state of image analysis isn’t perfect, but the tools make it easier to discover the images and they open up new connections within collections. Thank you!