Outsourcing FrameNet to the Crowd
•
2 likes•533 views
This document discusses outsourcing FrameNet annotation to crowdsourcing. It presents a two-step and simplified one-step methodology for crowdsourcing frame and semantic role annotation. Experiments using these methods on the CrowdFlower platform showed that the simplified one-step approach had higher accuracy and was faster than the two-step approach. Lessons learned include that definitions need to be simplified for non-experts and negation and modality are difficult concepts. Further research directions include larger-scale experiments and linking entities to structured knowledge bases.
![Filling
“A trolley was heaped with beer cans”
ThemeGoal
Frame annotation
2
Lexical
unit
Frame
Frame
elements
to heap Filling [Goal, Theme]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Outsourcing FrameNet to the Crowd
- 1. OUTSOURCING FRAMENET TO THE CROWD Marco Fossati, Claudio Giuliano, and Sara Tonelli fossati@fbk.eu 1
- 2. Filling “A trolley was heaped with beer cans” ThemeGoal Frame annotation 2 Lexical unit Frame Frame elements to heap Filling [Goal, Theme]
- 3. Frame annotation 3 Lexical unit to heap Frame 1 Filling Frame 2 Placing Goal CauseTheme Agent
- 4. CHALLENGE Full frame annotation by the man in the street 4
- 5. 2-step methodology 5 1 Frame discrimination 2 FE recognition Word sense disambiguation Which is the sense of heaped? Filling Semantic role assignment The Theme is... Which is the Theme? with beer cans “A trolley was heaped with beer cans”
- 6. Critical issues in crowdsourcing 6 “The element Theme is generally an NP object” ??? Definition by experts for experts
- 7. Critical issues in crowdsourcing 7 1 Frame discrimination 2 FE recognition !!! Error propagation
- 8. Frame emersion Alternative methodology 8 1-step workflow (bottom-up) 1 FE recognition
- 9. Implementation 9 “A trolley was heaped with beer cans” to heap A trolley with beer cans None None PlacingFilling Theme ? Goal ? Agent ? Cause ?
- 10. Manual simplification a. Replace the FE name with the semantic type b. Simplify complex syntax c. Avoid variability d. Reformulate technical concepts using common words 10
- 11. Simplification impact 11 LU Frame FE Gain to throw Cause motion Theme + 44% to throw Cause motion Goal + 19%to throw Body movement Body part + 31% to guide Influence of event on cognizer Cognizer + 25% Average gainAverage gainAverage gain + 30%
- 12. Simplification examples The element Theme is generally an NP object The Theme is the element that undergoes the motion 12 Theme Cause motion
- 13. Simplification examples 13 With some verbs in this frame, the Body part involved in the action is specified by the meaning of the verb and cannot be expressed separately This element describes the Body part that is involved in the action Body part Body Movement
- 14. EXPERIMENTS with the CrowdFlower platform 14
- 15. Settings Lexical unit Frames to disappear Ceasing to be Departing to guide Cotheme Influence of event on cognizer to heap Filling Placing to throw Body movement Cause motion Judgments Cost per sentence 5 1.83 $ cents 15
- 16. 2-STEP Top-down standard annotation workflow 1. Frame discrimination 2. FE recognition 16 “A trolley was heaped with beer cans” Which is the correct sense? Filling Placing “A trolley was heaped with beer cans” Theme: The Theme is the object which changes location A trolley with beer cans Goal: The Goal is... Choose the right sense of a word Find the participants in the event Simplified FE definitions
- 17. Theme: The Theme is the object which changes location A trolley with beer cans None Goal: The Goal is... 1-STEP Bottom-up workflow 17 “A trolley was heaped with beer cans” Filling correct frame
- 18. Agent: The Agent is the person that cause the theme to move A trolley with beer cans None Cause: The Cause is... 1-STEP Bottom-up workflow “A trolley was heaped with beer cans” Placing wrong frame 18
- 19. Results 2-step 1-step Majority vote accuracy Execution time (h) Cost per sentence ($ cents) .687 .792 171 130 4.57 8.41 19 986 judgments collected so far
- 20. Lessons learnt Difficult gold = low agreement Automatic task takedown = time increase Contested gold is useful Signal for tricky FE definitions Negation and modality are problematic 20
- 21. Negation 21 “On their way to the station she would not throw her coin into the Trevi Fountain” the Goal is the place where the element ends up at the end of the motion Goal A worker said “If she WOULD NOT THROW her coin, it did NOT end up in the fountain. Therefore this answer is wrong. She still has the coin.”
- 22. Conclusion 22 We can crowdsource frame annotation FE definitions simplification Bottom-up approach
- 23. Research directions Larger scale experiments Get rid of FE definitions Entity linking techniques Semantic type information from structured knowledge bases 23