CoderRank: Creating Gold Standards
•1 like•245 views
Report
Share
Download to read offline
Slides from today's presentation in Brussels at LT-Accelerate. The short talk covered the core ideas underpinning a recent patent for a method known colloquially as CoderRank. I focus on the challenge of human annotation, the role of measurement, and the tools we have built, both free/open source and commercial, to make it easier for teams to create gold standard annotation sets quickly and accurately.
Recommended
More Related Content
What's hot
What's hot(20)
Viewers also liked
Viewers also liked(20)
Similar to CoderRank: Creating Gold Standards
Similar to CoderRank: Creating Gold Standards(20)
More from Stuart Shulman
More from Stuart Shulman(14)
Recently uploaded
Recently uploaded(20)
CoderRank: Creating Gold Standards
- 1. CoderRank: Creating Gold Standards Dr. Stuart W. Shulman Founder & CEO, Texifter @stuartwshulman “…a wealth of information creates a poverty of attention.” - Herbert Simon, 1971
- 2. Text Classification A 2500 year-old problem Plato argued it would be frustrating. It still is.
- 3. Grimmer & Stewart “Text as Data” Political Analysis (2013) Volume is a problem for scholars Coders are expensive Groups struggle to accurately label text at scale Validation of both humans and machines is “essential” Some models are easier to validate than others All models are wrong Automated models enhance/amplify, but don’t replace humans There is no one right way to do this “Validate, validate, validate” “What should be avoided then, is the blind use of any method without a validation step.”
- 4. Free, Open-Source, Web-based Text Analytics Toolkit
- 5. Original Software Kernel: Tools for Measurement
- 6. Avoid Tennis Elbow Items load to the screen and the coder hits the keystroke
- 7. Keystroke Human Coding Human coding can be distributed to individuals, groups & crowds
- 8. Computer Science & National Science Foundation: Measure Everything How fast? How reliable? How accurate?
- 10. Interrater Reliability: A Critical Measurement
- 11. Adjudication
- 12. CoderRank for enhanced machine-learning is our key innovation Patent issued March 1, 2016
- 13. CoderRank for Enhanced Machine-learning CoderRank is to text analytics what PageRank was to search. Just as Google said not all web pages are created equal, Texifter argues that not all humans are created equal. When training machines, it is best to rely most on the humans most likely to create a valid observation. We proposed a unique way to rank humans on trust and knowledge vectors.
- 14. ActiveLearning engines and human coding tools combine… what humans do best… with what computers do best. Humans and machines learning together It is always good to keep humans “in-the-loop”
- 15. Word sense disambiguation (relevance)
- 16. Word sense disambiguation (relevance)
- 17. Word sense disambiguation (relevance)
- 18. Word sense disambiguation (relevance)
- 20. Human coding converts into machine classifiers Accumulated human coding becomes training data via machine-learning
- 21. Crowdsourcing accelerates the insight generation Distribute coding for synchronous & asynchronous collaboration