Downloaded 31 times
![7
From: Google Alerts <googlealerts-noreply@google.com>
Subject: Google Alert - "John R. Frank”
=== Web - 2 new results for ["John R. Frank"] ===
John R. Frank
SPOKANE, Wash. - John R. Frank, 55, died March 4, 2012, in Coeur
d' Alene, Idaho. Survivors include: his wife, Miki; daughter, Patricia
Frank; ...
<http://www.hutchnews.com/obituaries/Frank--John-CP>
In Memory of John R Frank
Biography. John R. Frank, age 55, passed away at Sacred Heart
Medical Center in Spokane, WA, on March 4, 2012. John was born in
Hutchison, KS, ...
<http://www.englishfuneralchapel.com/sitemaker/sites/Englis1/obit.cgi?user=583335Frank>](https://image.slidesharecdn.com/max-kleiman-weiner-open-data-science-conference-odsc-150619132008-lva1-app6891/85/Machine-In-The-Loop-for-Knowledge-Discovery-7-320.jpg)
![8
From: Google Alerts <googlealerts-noreply@google.com>
Subject: Google Alert - "John R. Frank”
=== Web - 2 new results for ["John R. Frank"] ===
John R. Frank
SPOKANE, Wash. - John R. Frank, 55, died March 4, 2012, in Coeur
d' Alene, Idaho. Survivors include: his wife, Miki; daughter, Patricia
Frank; ...
<http://www.hutchnews.com/obituaries/Frank--John-CP>
In Memory of John R Frank
Biography. John R. Frank, age 55, passed away at Sacred Heart
Medical Center in Spokane, WA, on March 4, 2012. John was born in
Hutchison, KS, ...
<http://www.englishfuneralchapel.com/sitemaker/sites/Englis1/obit.cgi?user=583335Frank>](https://image.slidesharecdn.com/max-kleiman-weiner-open-data-science-conference-odsc-150619132008-lva1-app6891/85/Machine-In-The-Loop-for-Knowledge-Discovery-8-320.jpg)
![23
"#both_bol_JJ_0": {
"24 minute": 1,
"american": 5,
"linux": 2,
"pro government": 1,
"#both_co_LOC_3": {
"arab": 2,
"east": 4,
"iran": 5,
"kingdom": 2,
"lebanon": 2,
"london": 2
"#both_co_MAGIC_VALUE_0": {
"pagewanted all r 0accessdate22": 1,
"q cache 9qiv27 ymm8j": 1
},
"#both_co_NATIONALITY_2": {
"american": 6,
"indians": 2,
"iranian": 5,
"saudi": 2,
"syrian": 93
"#entity_type": {
"ORG": 6
},
"#post_bow_RB_2": {
"allegedly": 4,
"falsely": 1,
"openly": 2,
Natural Language
Large Collections
of Unstructured
Documents
StreamCorpus
Pipeline
(FOSS)
3rd Party
NER,
e.g. Serif,
Basis
TreeLab
Feature
Collection
The Syrian Electronic Army (SEA) is a
group of computer hackers who support
the government of Syrian President
Bashar al-Assad. Using spamming,[2]
defacement, malware (including the
Blackworm tool),[3] phishing, and denial
of service attacks, it mainly targets
political opposition groups and western
websites including news organizations
and human rights groups. The Syrian
Electronic Army is the first public, virtual
army in the Arab world to openly launch
cyber attacks on its opponents.[4]
Feature Vectors](https://image.slidesharecdn.com/max-kleiman-weiner-open-data-science-conference-odsc-150619132008-lva1-app6891/85/Machine-In-The-Loop-for-Knowledge-Discovery-23-320.jpg)
More Related Content
Similar to Machine-In-The-Loop for Knowledge Discovery
![[DataCon.TW 2019] Graph Query on Big-data, REST API, and Live Analysis Systems](https://cdn.slidesharecdn.com/ss_thumbnails/graphquery-datacon-190917102738-thumbnail.jpg?width=640&height=640&fit=bounds)
Machine-In-The-Loop for Knowledge Discovery
- 1. MACHINE-IN-THE- LOOP FOR KNOWLEDGE DISCOVERY Max Kleiman- Weiner OP E N D A T A S C I E N C E C O N F E R E N C E_ BOSTON 2015 @opendatasci
- 2. 2 MACHINE-IN-THE-LOOP FOR KNOWLEDGE DISCOVERY ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ BOSTON 2015 @opendatasc iMaxKleiman-Weiner Co-founder of Diffeo Co-organizer of TREC-KBA
- 3. 3 Analysis & Production Collection &Processing Dissemination to Customers People close this gap.
- 4. 4 Search Recommender Engines Text Analytics Analysis & Production Collection &Processing Dissemination to Customers People close this gap.
- 5. 5 Search Recommender Engines Text Analytics Analysis & Production Collection &Processing Dissemination to Customers People close this gap. Reporting Tools
- 6. 6 Search Recommender Engines Text Analytics Analysis & Production Collection &Processing Dissemination to Customers People close this gap. helps. Reporting Tools
- 7. 7 From: Google Alerts<googlealerts-noreply@google.com> Subject: Google Alert - "John R. Frank” === Web - 2 new results for ["John R. Frank"] === John R. Frank SPOKANE, Wash. - John R. Frank, 55, died March 4, 2012, in Coeur d' Alene, Idaho. Survivors include: his wife, Miki; daughter, Patricia Frank; ... <http://www.hutchnews.com/obituaries/Frank--John-CP> In Memory of John R Frank Biography. John R. Frank, age 55, passed away at Sacred Heart Medical Center in Spokane, WA, on March 4, 2012. John was born in Hutchison, KS, ... <http://www.englishfuneralchapel.com/sitemaker/sites/Englis1/obit.cgi?user=583335Frank>
- 8. 8 From: Google Alerts<googlealerts-noreply@google.com> Subject: Google Alert - "John R. Frank” === Web - 2 new results for ["John R. Frank"] === John R. Frank SPOKANE, Wash. - John R. Frank, 55, died March 4, 2012, in Coeur d' Alene, Idaho. Survivors include: his wife, Miki; daughter, Patricia Frank; ... <http://www.hutchnews.com/obituaries/Frank--John-CP> In Memory of John R Frank Biography. John R. Frank, age 55, passed away at Sacred Heart Medical Center in Spokane, WA, on March 4, 2012. John was born in Hutchison, KS, ... <http://www.englishfuneralchapel.com/sitemaker/sites/Englis1/obit.cgi?user=583335Frank>
- 9. KBAa TREC evaluation Howmany days must a news article wait before being cited in Wikipedia?
- 10. KBAa TREC evaluation KnowledgeBase Acceleration? rate of assimilation << stream size # editors << # entities << # mentions (definition of a “large” KB)
- 11. KBAa TREC evaluation TRECKBA 2014 Stream Filtering to Recommend Citations and Fill Slots 1) Initialize with target entities • Start with citations before dateX (20%) 1) Iterate over stream of text items • Predict citations after dateX (80%) • Citation = “vital”, i.e. changes profile 2) Identify infobox slot values Content Stream • 1.2bn texts, 50% English • >500M tagged by Serif • 13,663 hours of data (19 months) • 105 docs-per-hour • News, blogs, forums, preprints, and more Your KBA System Entities in Wikipedia or another Knowledge Base Automatically recommend new edits
- 12. KBAa TREC evaluation 1 10 42504270 4290 4310 4330 4350 Citationworthy doccounts hour number in stream corpus BBC: “What would you say to forming The Beatles - The Next Generation” … James: “… I'd be up for it.” echo Entity profiles evolve over time! Exploring these issues in new track: TREC Dynamic Domain SSF example: “FounderOf” slot on James McCartney
- 13. KBAa TREC evaluation 1 10 100 2011-10-05-00 2011-10-10-17 2011-10-16-09 2011-10-22-01 2011-10-27-17 2011-11-02-09 2011-11-08-01 2011-11-13-17 2011-11-19-09 2011-11-25-01 2011-11-30-17 2011-12-06-09 2011-12-12-01 2011-12-17-17 2011-12-23-09 2011-12-29-01 2012-01-03-17 2012-01-09-09 2012-01-15-01 2012-01-20-17 2012-01-26-09 2012-02-01-01 2012-02-06-17 2012-02-12-09 2012-02-18-01 2012-02-23-17 2012-02-29-09 2012-03-06-01 2012-03-11-17 2012-03-17-09 2012-03-23-01 2012-03-28-17 2012-04-03-09 2012-04-09-01 2012-04-14-17 2012-04-20-09 2012-04-26-01 2012-05-01-17 2012-05-07-09 2012-05-13-01 2012-05-18-17 2012-05-24-09 2012-05-30-01 2012-06-04-17 2012-06-10-09 2012-06-16-01 2012-06-21-17 2012-06-27-09 2012-07-03-02 2012-07-08-20 2012-07-14-12 2012-07-20-04 2012-07-25-20 2012-07-31-12 2012-08-06-04 2012-08-11-20 2012-08-17-12 2012-08-23-04 2012-08-28-20 2012-09-03-12 2012-09-09-04 2012-09-14-20 2012-09-20-12 2012-09-26-04 2012-10-01-20 2012-10-07-12 2012-10-13-04 2012-10-18-20 2012-10-24-12 2012-10-30-04 2012-11-04-20 2012-11-10-12 2012-11-16-04 2012-11-21-20 2012-11-27-12 2012-12-03-04 2012-12-08-20 2012-12-14-12 2012-12-20-04 2012-12-25-20 2012-12-31-12 2013-01-06-04 2013-01-11-20 2013-01-17-12 2013-01-23-04 2013-01-28-20 2013-02-03-12 2013-02-09-04 Corpuscounts per hour Pre-hoc assess all surface mentions “vital” docs change entity profile, usually filling a slot training evaluation http://trec-kba.org/
- 14. KBAa TREC evaluation 1,187,974,321documents total 579,838,246 English docs tagged by BBN Serif • NER, within-doc coref • Sentence Parse Trees • Relations Available at: http://trec-kba.org/
- 15. 15 15 What entity resolversdo easily is boring. Wang Qishan, China's top graft buster and a member of the Politburo Standing Committee, making him one of the seven most powerful people in China Wang Qishan, male, Han nationality, is a native of Tianzhen County, Shanxi province. Director, Rural Development Research Center of the State Council, Communication Office. He served as a member of the CPC Beijing Municipal Committee in 2003, and he was appointed mayor of Beijing in the same year. Wang, 64, is member of the Political Bureau of the 17th CPC Central Committee and vice premier. He was also elected into the new Central Commission for Discipline Inspection.
- 16. 16 16 What entity resolversshould do is exciting. Wang also announced that Trinidad and Tobago would receive a 40M yuan grant (approximately TT$40M) from the Chinese government to fund projects mutually agreed upon by both governments. Do you realize that Harry Reid and his son Rory have a deal for that land with Chinese Vice Premier Wang Qishan for a 5 billion solar plant. Its allover social media… The U.S. Chamber of Commerce co-hosted a dinner last night at the JW Marriot in Washington to honor Chinese Vice Premier Wang Qishan, who is in the United States to co-chair the 23rd China - U.S. Joint Commission on Commerce and Trade
- 17.
- 18.
- 19. 19 Machine-in-the-Loop (active-ranking) 19 Collaborative MediaWiki KB AuthoringTools Novel and Relevant Results Information gathering as training Profile as query
- 21. 21 Large Collections of Unstructured Documents StreamCorpus Pipeline (FOSS) rejester 3rdParty NER, e.g. Serif, Basis dossier.store dossier.label kvlayer TreeLab Feature Collection PostgreSQL Dossier Stack (FOSS)TreeLab Active Learning FOSS development: github.com/streamcorpus github.com/dossier Diffeo ships a stack of docker containers, including a fully integrated MediaWiki for turn-key deployment on a single server or customization on a large cluster. dossier.web Diffeo TreeLab Clustering & Ranking on Bigforest ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ Diffeo: Built for Clouds
- 22. 22 Large Collections of Unstructured Documents StreamCorpus Pipeline (FOSS) rejester 3rdParty NER, e.g. Serif, Basis dossier.store dossier.label kvlayer TreeLab Feature Collection PostgreSQL Dossier Stack (FOSS)TreeLab Active Learning dossier.web Diffeo TreeLab Clustering & Ranking on Bigforest ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ ♒ Scales with Your Database Hierarchical Clustering and Active Ranking Natural Language Processing Pipelines and Feature Vectors Data Center Scaling User Experience for Machine-Assisted Entity Research
- 23. 23 "#both_bol_JJ_0": { "24 minute":1, "american": 5, "linux": 2, "pro government": 1, "#both_co_LOC_3": { "arab": 2, "east": 4, "iran": 5, "kingdom": 2, "lebanon": 2, "london": 2 "#both_co_MAGIC_VALUE_0": { "pagewanted all r 0accessdate22": 1, "q cache 9qiv27 ymm8j": 1 }, "#both_co_NATIONALITY_2": { "american": 6, "indians": 2, "iranian": 5, "saudi": 2, "syrian": 93 "#entity_type": { "ORG": 6 }, "#post_bow_RB_2": { "allegedly": 4, "falsely": 1, "openly": 2, Natural Language Large Collections of Unstructured Documents StreamCorpus Pipeline (FOSS) 3rd Party NER, e.g. Serif, Basis TreeLab Feature Collection The Syrian Electronic Army (SEA) is a group of computer hackers who support the government of Syrian President Bashar al-Assad. Using spamming,[2] defacement, malware (including the Blackworm tool),[3] phishing, and denial of service attacks, it mainly targets political opposition groups and western websites including news organizations and human rights groups. The Syrian Electronic Army is the first public, virtual army in the Arab world to openly launch cyber attacks on its opponents.[4] Feature Vectors
- 24. 24 Large Collections of Unstructured Documents StreamCorpus Pipeline (FOSS) 3rdParty NER, e.g. Serif, Basis TreeLab Feature Collection Feature Vectors "#post_bow_VB_3": { "quote": 6, "infiltrate": 4, "disclose": 2, "#post_co_DATE_2": { "2011": 2, "2011 3 15": 1, "2012": 4, "2013": 26, "2014": 12, "2015": 4, "april": 11, "august": 6, "february": 17, "january": 10, "july": 29, "june": 8, "november": 21, "october": 8, "september": 10 }, "post_co_PER_1": { "37246009": -1, "131492056": 2, "437879322": 1, "664353967": -2, "680387639": -1, "681138746": 1, "740105633": -1, "792534747": 2, "973559166": 3, "1146544765": 6, "1273592341": 2, "#post_co_PER_1": { "al assad": 2, "barack": 2, "bashar": 3, "buscemi": 1, "crook": 1, "foster": 1, "fowler": 1, "jordan": 1, "monde": 1, "obama": 6, "sabari": 2,
- 25. 25 Distantly supervised models not perfect
- 26. 26 Distantly supervised models not perfect Can flatten into a set of sets for Cross Doc Coref Resolution (CDCR) with set-based metrics, e.g., • Bcubed F • CEAF
- 27. 27 Distantly supervised models not perfect Can flatten into a set of sets for Cross Doc Coref Resolution (CDCR) with set-based metrics, e.g., • Bcubed F • CEAF
- 28. 28 Distantly supervised models not perfect Can flatten into a set of sets for Cross Doc Coref Resolution (CDCR) with set-based metrics, e.g., • Bcubed F • CEAF
- 29. 29 Distantly supervised models not perfect Can flatten into a set of sets for Cross Doc Coref Resolution (CDCR) with set-based metrics, e.g., • Bcubed F • CEAF
- 30. 30 Distantly supervised models not perfect Can flatten into a set of sets for Cross Doc Coref Resolution (CDCR) with set-based metrics, e.g., • Bcubed F • CEAF
- 31. 31 Distantly supervised models not perfect Traversing from a leaf starts with higher precision, and progresses to higher recall.
- 32. 32 Ranking models justdifferent Traversing from a leaf starts with higher precision, and progresses to higher recall.
- 33. 33 Ranking models justdifferent Traversing from a leaf starts with higher precision, and progresses to higher recall. Crossing the set boundary can find the most surprising and useful information.
- 34. 34 Ranking models beyondcoref Ranking can recover from coref errors and expose knowledge beyond coref.
- 35. 35 Ranking models beyondcoref Ranking can recover from coref errors and expose knowledge beyond coref.
- 36. 36 Measurement Methodologies Unexpected ExpectedReciprocal Rank u-ERR 1. Submit profile as query to search engine 2. Receive items 3. Start reading through the list: 4. User examines position n 5. If user finds new knowledge: 6. Update profile 7. Go to 1 with updated profile as query 8. else 9. n += 1 10. Go to 4 u-ERR = 1 / (expected list position of surprise) Figure of merit: depth in the list where user discovers new knowledge
- 37. 37 Retrieval results suggest updates tothe profile Entity profiles act as queries for retrieval algorithms User interactions steer the system beyond coref to find surprises Truth data is expensive Observe user actions Learn Actively
- 38.
Editor's Notes
- #3 Diffeo connects documents to the entities mentioned in the text, so you can discover everything written about your entities of interest.
- #8 How many of you have setup a google alert? How many of you have setup a google alert on yourself?
- #10 This shows the lag time between publication and eventual citation of news articles linked from pages in Category:Living_people. The median lag time for articles published after 2001 is more than 1.5 years. You can see the lag time leveling out to a long heavy tail for earlier articles, and then only gradually accelerating for a small handful of news articles that get incorporated rapidly – out of ~50,000 articles only ~300 had a lag time of less than a day.
- #21 Automatically recommends content. Collaborate with the machine and other users on building a profile. As the user builds a profile the system becomes aware of both the information the user knows about, which is contained in the wiki and the information *not* contained in the wiki which can be used for learning. Hard / soft selectors. Hard selectors are phrases that are synonymous with the entity, name, phone number, IP. Soft –selectors are signatures of the entity such as commonly used long phrases or other co-occurring entities.