SuperIDR - ECDL 2009

560 views

Published on

Superimposed Image Description and Retrieval for Fish Species Identification - talk given at ECDL 2009

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
560
On SlideShare
0
From Embeds
0
Number of Embeds
8
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

SuperIDR - ECDL 2009

  1. 1. Superimposed Image Descriptionand Retrieval for Fish SpeciesIdentification29 September, 2009ECDL 2009, Corfu, GreeceUma Murthy, Edward Fox,Yinlin Chen, & Eric Hallerman*Computer Science*Fisheries and Wildlife SciencesRicardo Torres, Evandro Ramos,& Tiago FalcãoInstitute of Computing
  2. 2. Context of this work digital libraries content services users scenariosdocuments annotations search annotation scholars biodiversity text-based fisheries retrieval students facultyimages text sub- content-based researchers species documents image retrieval identification 2
  3. 3. Motivation•  Many scholarly tasks involve working with images with significant number of details –  Species identification, analyzing paintings, studying architecture styles, analyzing medical images, etc.•  Scholars combine paper-based and electronic methods/tools –  Tools are not well-integrated, information is fragmented and tedious to access  ineffective and inefficient task execution 3
  4. 4. Paper-based methods for fishidentificationDichotomous keys Personal Notes1a Paired fins absent; jaws absent, mouth in an oral disk (the disk mostly surrounded by a fleshy hood in larvae); 7 external gill openings present in row behind eye .......... Lampreys - Petromyzontidae p. ooo1b Paired fins present (at least 1 set); jaws present; 1 external gill opening per side ................................... 22a Caudal fin heterocercal or abbreviate heterocercal (Figure 5) ............................................................ 32b Caudal fin protocercal (Figure 13, Part 2, upper left) or homocercal (Figure 5) ............................................ 6 4
  5. 5. Popular electronic methods FISHBASE: Image collection with browsing, field-wise searching and use of forums EFISH: Organization and browsing based on taxonomy EKEY: Text search and shape- based image retrieval on predefined shapes 5
  6. 6. Requirements of a new system•  Incorporate more visual and descriptive information –  Ability to add user content•  Improve information management access to heterogeneous information –  Images, textual descriptions, notes, image markings, etc.•  Provide well-integrated functionalities to support task execution•  Provide capability to share content 6
  7. 7. The Superimposed Image Description and Retrieval Tool (SuperIDR) Content-based image retrieval (CBIR) description and Digital library retrieval of images servicesSuperimposed using image •  searching,information features such as browsing,subdocuments shape, color, etc. indexing, etc.annotations •  Managingnew structure documents, collections, Text-based metadata, etc. image retrieval description and retrieval of images using text 7
  8. 8. SuperIDR: species organization 8
  9. 9. SuperIDR: species description (1/2) 9
  10. 10. SuperIDR: species image annotation 10
  11. 11. SuperIDR: species description 11
  12. 12. SuperIDR: text-based image retrievalquery 12
  13. 13. SuperIDR: text-based image retrievalresults 13
  14. 14. SuperIDR: text- and content-based imageretrieval (1/2) 14
  15. 15. SuperIDR: text- and content-based imageretrieval (2/2) 15
  16. 16. User study in an UG Ichthyology class(Spring 2008) – 1/2•  Goal: to assess the usefulness of SuperIDR in fish species identification•  Participants: 28 undergraduates, working in teams of two with a tablet PC 16
  17. 17. Fill entry questionnaire Get assigned tablet Study procedure PCs with SuperIDR Tutorial Use tool for a week, Compare methods make annotations Session 1: half the teams use SuperIDR and halfTask: identify 20 unknownTask: identify 20 unknown used traditional methods specimens in 2 sessions specimens in 2 sessions Session 2: teams Fill exit Fill exit swapped methods questionnaire questionnaire Keep/return SuperIDR Keep/return SuperIDR 17
  18. 18. Summary of results (1/2)•  Method had a significant impact on task outcome (p-value=0.015), with higher likelihood of success in using SuperIDR than traditional methods•  In general, students were interested in using SuperIDR for species identification –  Positive comments by students –  Six teams chose to keep the tool 18
  19. 19. Comments“it was very helpful”“very helpful for taxonomy but need betterphotos”“very neat but might take a while to master allthe key concepts” 19
  20. 20. Summary of results (2/2)•  No significant evidence to show that annotation or search on parts of images is useful –  Possibly due to timing and duration of the study 20
  21. 21. Conclusion•  Developed SuperIDR that combines text- and content- based image description, retrieval, and browsing, including for parts of images•  Ichthyology students performed better with SuperIDR than with traditional methods in fish species identification•  Current/future work –  Conduct further experiments to test retrieval effectiveness and usefulness of combined search (text + CBIR) on parts of images –  Make SuperIDR available for download –  Connect with Flickr to enable information sharing and to leverage existing Flickr collection 21
  22. 22. Thank you ? ? http://si.dlib.vt.edu/superidr 22
  23. 23. Back-up slides 23
  24. 24. Acknowledgements•  Dr. Orth, Ryan McManamay, Lindsey Pierce, and others at the Fisheries and Wildlife Sciences department•  Microsoft (tablet PC grant)•  Jason Lockhart for loaning tablets•  NSF (DUE-0435059) 24
  25. 25. SuperIDR architecture Annotation Search Browse Annotation SearchCBISC Lucene BrowseFeature Taxonomy Image Species Images descriptions AnnotationsVectors & key data marks 25
  26. 26. Content BasedInformationRetrieval 26
  27. 27. GoalProvide better support to work withimages, parts of images, providingdescription through personal notes,linking related information, providingaccess to existing and new informationeasily, and being able to share all thisinformation 27
  28. 28. Species and image data•  207 species of Virginia freshwater fishes – 25 families and 124 genera•  213 images, mostly from Jenkins and Burkhead’s Freshwater Fishes of Virginia 28
  29. 29. Data collected in experiment•  Entry and exit questionnaire responses•  Species identification responses and time to identify•  Logs of user interaction with the tool•  Data from 9 teams (18 students) was considered in analysis –  Others not present for both sessions –  Incomplete species id. responses 29
  30. 30. Species identification response summaryTraditional methods SuperIdRTeam # Correct Team # Correct ID Session (out of 20) ID Session (out of 20) 2 1 15 2 2 18 4 1 16 4 2 17 6 1 13 6 2 17 11 1 12 11 2 16 3 2 8 3 1 14 5 2 13 5 1 15 9 2 12 9 1 10 10 2 10 10 1 14 13 2 11 13 1 11 Mean 12.2 Mean 14.67 Method had a significant impact on outcome and 30 students did better with SuperIDR
  31. 31. Next steps•  Further evaluation to show that combining text- and content-based image retrieval on parts of images is effective and useful•  Explore use in other fields - art history, architecture, and other biology-based fields –  Involving images with significant number of details 31

×