Experiment on Knowledge Acquisition


Published on

This is our experiment in crowd-sourcing knowledge acquisition for universal truth creation out of complicated textbook sentences.
An integration of Project Halo's SMW and AURA efforts.

  • Be the first to comment

Experiment on Knowledge Acquisition

  1. 1. Subject Matter Expert - User Experience Review AuraWiki Jesse Wang, Wil Smith Oct 2012
  2. 2. Project Goals“Crowd Source UT Authoring” Can Subject Matter Experts Author Useful Universal Truths? Can We Speed Up Encoding a Textbook with Input from Subject Matter Experts? Can We Create a UT Authoring Portal for Multiple Textbooks? Can Existing Social Networks Provide Subject Matter Experts Capable of UT Authoring? Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?
  3. 3. Meeting GoalsThis is an end user meeting and we are doing it to hearyour comments on the current system.The team wants to explain the authoring process, anddemonstrate the current application state.Before we continue development on the project we needto review how the application, in the unfinished state,performs with our targeted user base.MID PROJECT FEEDBACK. We do these meetings whenreleasing production products to ensure the finalapplication complies with customer needs and is not “theengineering best way”.
  4. 4. Review SchedulePhase Method Time requiredIntroduction & Training • Goals of Project Halo and Aura 0:45 Wiki • Training for UT Authoring • What is a UT? • What is a concept / context? • How do I formulate UTs • System demonstration & hands-onAssignment: UT Authoring • You work by yourself using our 1:30 system • Support is provided by usQuestionnaire & Discussion • Online questionnaire 0:45 • Group discussion
  5. 5. Subject Matter Expert - User Experience Review How do we author a universal truth?
  6. 6. From Sentence to Universal TruthsWhat is a UT?How to write UTs for a sentenceWhat it a concept?How to choose concepts for a UT
  7. 7. What is a UT?There is no exact definition of “Universal Truth”, but a good UT… expresses a single fact (which can be compound, but must be indivisible). is as simple as possible, but not simpler (to quote Einstein). is context-independent, i.e. meaningful on its own, without surrounding sentences or other context. is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...". is unambiguous. is precise.
  8. 8. How to Write UTs for a SentenceFor each relevant sentence, write one or more UTsThere is no one correct set of UTs for a given sentence.The source sentences are open to interpretation, and it isup to you to determine what the most appropriate UTs areThe interpretation of the source sentences, and theparaphrasing into UTs, should be guided by yourknowledge of the subject matter (and by common sense)The following methods are not exhaustive. When indoubt, refer back to the definition of a “good UT”.
  9. 9. How to Write UTs for a SentenceIn the simplest case, the sentence has only one UT, and itis exactly the same as the UT.Sentence: Matter is made up of elements.UT: Matter is made up of elements.
  10. 10. How to Write UTs for a SentenceSometimes part of a sentence is irrelevant, even thoughthe sentence as a whole has been deemed relevant. Insuch cases the irrelevant part is simply dropped.Sentence: Some symbols are derived from Latin orGerman; for instance, the symbol for sodium is Na.UT: Sodium has the symbol Na.
  11. 11. How to Write UTs for a SentenceOften, a sentence contains several facts. In this case, thesentence needs to be split into several UTs.Sentence: Pure sodium is a metal, and pure chlorine is apoisonous gas.UT: Pure sodium is a metal.UT: Pure chlorine is a poisonous gas.
  12. 12. How to Write UTs for a SentenceWhen a reference like "this", "that", "these", "those", "which", etcoccurs in a source sentence, they should be replaced with thereferent (the word that "this","that", etc refers to).Sentence: Organisms are composed of matter, which is defined asanything that takes up space and has massUT: Organisms are composed of matterUT: Matter has mass and volumeHere, "which" clearly refers to "Organisms", and "is defined as" isunnecessarily verbose, and thus simplified.
  13. 13. How to Write UTs for a SentenceSometimes the referent is in a previous sentence in thesame paragraph.Sentence: These selective channels affect the rate atwhich water moves osmotically across the membraneUT: Aquaporins are selective channelsUT: Aquaporins affect the rate at which water movesosmotically across the membrane
  14. 14. How to Write UTs for a SentenceThere are many other words that are used as linguistic flourishes or to bind sentences in aparagraph together and give the text some "flow". Such words ("another" in the examplebelow) should be eliminated, but can sometimes be used to extract or infer "incidental"information from a sentence.Sentence: Water (H20), another compound, consists of the elements hydrogen (H) andoxygen (O) in a 2:1 ratioUT: Water consists of hydrogen and oxygen in a 2:1 ratioUT: Water is a compoundUT: Hydrogen has symbol HUT: Oxygen has symbol OHere, the first UT contains the "main" information of the sentence, and the last three are"incidental".
  15. 15. How to Write UTs for a SentenceIdeally, a UT should not contain words like "sometimes","usually", "most", "some", "generally", etc. There are acouple of ways to handle them. One way is to generalize theweaker statement into a universal statement, usually just bydropping the word in question.Sentence: Branching generally enables plants to harvestsunlight for photosynthesis more effectively.UT: Branching enables plants to harvest sunlight forphotosynthesis more effectively.
  16. 16. How to Write UTs for a SentenceAnother method of eliminating "some" etc is to instantiate thesentence by providing a name for the case where the sentence istrue.Sentence: Some prokaryotes even contain a row of tiny magneticparticles that allow the cells to orient in Earths magnetic fieldUT: Magnetotactic bacteria contain a row of tiny magnetic particlesthat allow the cells to orient in Earths magnetic fieldHere we replaced "some prokaryotes" with "magnetotacticbacteria". We only do this when the new name occurs somewhere inthe textbook.
  17. 17. How to Write UTs for a Sentence"some", etc are allowed when they do not refer to the main concept ofthe sentence (see "Choosing Concepts for a UT", below). For example,this is a good UT:UT: In vascular plants, some substances may use more than one routeThe UT as a whole states a fact about all vascular plants (the mainconcept), and "some" refers to substances within all vascular plants.One way to remember this distinction is to keep in mind that UniversalTruths are just that -- universally true statements, i.e. statements thatare always true of some type of entity, event, etc. If it is possible, wecan also use Instantiation (see previous slide) in this type of case.
  18. 18. How to Write UTs for a SentenceWhen "for example", "such as", etc occurs in a sentence,encode the specific information in the example.Sentence: Some plants have horizontally oriented leaves;others, such as grasses, have leaves that are verticallyoriented.UT: Grass has vertically oriented leavesAs discussed above, the "some" part of the sentence isproblematic.
  19. 19. How to Write UTs for a SentenceIn other cases, the "for example" does not add any newinformation.Paragraph: We symbolize atoms with the same abbreviationused for the element that is made up of those atoms. Forexample, the symbol C stands for both the element carbonand a single carbon atom.Here, the second sentence does not add any information,since its informational content is subsumed by the general"rule" of the first sentence (assuming we have alreadyencoded that carbon has the symbol C).
  20. 20. How to Write UTs for a SentenceConditional statements should usually be rephrased to a non-conditional form.Sentence: If we place a cell in a solution that is hypotonic to the cell,water will enter the cell faster than it leavesUT: A cell in a hypotonic solution will gain water faster than it losesitHere, the condition has been changed into a context, "in ahypotonic solution". See also the section on "ContextualUniversals" below.
  21. 21. Review: What is a UT?Or rather a good UT… expresses a single fact (which can be compound, but must be indivisible). is as simple as possible is context-independent, i.e. meaningful on its own, without surrounding sentences or other context. is universally true. For example, a UT could start with "All Plants..." but not with "Some plants...". is unambiguous. is precise.
  22. 22. Summary: Methods of UT AuthoringCopy, remove (partial, irrelevant info), splittingReference resolution (from context)Incidental extraction e.g. “Water, another compound, …”Specialization, generalizationInstantiation E.g. “some prokaryotes”  “magnetotactic bacteria”Example-encoding E.g. “some plants…”  “grass”Condition to context
  23. 23. What is a Concept?A concept is a noun or a noun phrase that describes atype of entity or event, in singular formYou may need to create a new concept for a UT if anappropriate one does not already exist.Good concept names: Cell, Eukaryotic Cell, CellularRespirationBad concept names: Cells, The eukaryotic cell, A plant
  24. 24. Choosing Concepts for a UT The main concept of a UT is the entity or event that the UT is about. It is usually the grammatical subject of the UTUT: All plant cells have a nucleusMain concept: Plant CellNote that "All plant cells" is not a good concept, because it is not in singularform.UT: Photosynthesis produces oxygenMain concept: Photosynthesis
  25. 25. Choosing Concepts for a UT Some UTs are true in a specific context. In such cases, also choose a contextual concept –contextual concept is OPTIONAL The contextual concept usually follows a word such as "in", "during", "while", etc.UT: The phospholipids in a membrane move laterallyMain concept: PhospholipidContextual concept: MembraneNote that "the" or "a" is never part of the concept.UT: The oxygen produced during photosynthesis is used during cellular respirationMain concept: oxygenContextual concept: PhotosynthesisUT: Cellular respiration in eukaryotes happens in the mitochondriaMain Concept: Cellular RespirationContextual Concept: Eukaryote
  26. 26. Bring your Best, then Move OnWe hope you got good understanding of what you’resupposed to doGive it your best shot, it does NOT need to be perfectDo not spend too much time on one sentence/UT,move on if it’s too hardFeel free to work on sentences or UT others startedFeel free to ask us if you’re really confused or notsure
  27. 27. Subject Matter Expert - User Experience Review Aura wiki tutorial
  28. 28. Our Solution: Aura WikiContents: A few chapters of Campbell Biology Without figures and using a simplified layoutPurpose Use the power of crowdsourcing and online collaboration Let users „translate“ sentences from the textbook into UTs.
  29. 29. Sample wiki page for the textbook
  30. 30. Sample wiki page for a whole section
  31. 31. Sample wiki page for a sentence
  32. 32. Creating UTsThe translation of sentences to UTs is done on thesentence page.The sentence page can display the paragraphsurrounding the sentence (or even more context) ifneeded for understanding and formulating the UT.Under the heading Universal Truths you can see allUTs that exist for this sentence and you can add yorown.
  33. 33. Sample UTs presented in the sentence page
  34. 34. Creating UTs – Add new UT
  35. 35. Creating UTs – Input Form Input form for new UT. First two inputs are required.
  36. 36. Creating UTs -Autocompletion Autocompletion is automatically triggered and suggests existing concepts
  37. 37. Subject Matter Expert - User Experience ReviewSubject Matter Expert Assignment
  38. 38. The AssignmentNow it is your turnGo to your User pageThere is a list of sentences assigned to youOnce you have added UTs they will be displayed theretogether with the sentences (after refreshing the page)At the top of your user page, you will find links to help-texts, some sample UTs and the questionnaire for afterthe session.
  39. 39. The Assignment This link always brings you back to your user page Help documents and a list of example UTs Sentences (and UTs)
  40. 40. The AssignmentIMPORTANT This evaluation is conducted in order to assess the system, not to assess your work as a SME. You are supporting the evaluation, but you are not a test subject! Please work with the system at your own convenience. You do not need to achieve any quantitative or qualitative goal. (This is of course no excuse for being lazy ) You have pencils and paper available – if you encounter any problems or have ideas you think are worth mentioning, feel free to take a short note so you can bring it up during the discussion later If there are questions please ask them anytime
  41. 41. Subject Matter ExpertSubject Matter Expert - User Experience Review Review
  42. 42. QuestionnaireGo to your user pageThere is a link to the online questionnaire at the topIn the first question, select the username you wereassigned todayFill out the entire questionnaire. Don’t forget tosubmit at the end
  43. 43. Discussion – System FunctionalityDid you have trouble following the UT authoringworkflow? If so, where and why?What changes would you apply to the existingfunctionality and/or workflows to increase productivityand ease of use?What new functionality could you think of that might havea positive influence on your work?
  44. 44. Discussion – UT AuthoringDid you understand the notion of UT and concept?Did the training sufficiently explain the meaning of UT andconcept?Do you feel the training you received in the beginning andthe help materials on the site were sufficient forexecuting your task? If not, what would you improve?Which areas need better coverage?
  45. 45. Subject Matter Expert - User Experience Review Conclusion
  46. 46. Thank you.Contact: Jesse Wang (jessew@vulcan.com)William Smith (wills@vulcan.com )