Knowledge graphs (KGs) have recently emerged as a powerful way to represent knowledge in multiple communities, including data mining, natural language processing and machine learning. Large-scale KGs like Wikidata and DBpedia are openly available, while in industry, the Google Knowledge Graph is a good example of proprietary knowledge that continues to fuel impressive advances in Google's semantic search capabilities. Yet, both crowdsourced and automatically constructed KGs suffer from noise, both during KG construction and during search and inference. In this talk, I will discuss how to build and use such knowledge graphs effectively, despite the noise and sparsity of labeled data, to solve real-world social problems such as providing insights in disaster situations, and helping law enforcement fight human trafficking. I will conclude by providing insight on the lessons learned, and the applicability of research techniques to industrial problems. The talk will be designed to appeal both to business and technical leaders.

1. Applying Noisy Knowledge
Graphs to Real Problems
Mayank Kejriwal
USC Information Sciences Institute
May 2019

2. 2
Acknowledgements

3. Real Problems

5. Web has lowered the barrier to entry!
5

6. 6

7. 7
Pump and dump schemes proliferate online

8. 8

9. Quechua
Fula
Odiya
Maithili Bhojhpuri
Uighyur
Mayan languages
Aboriginal
languages
Tasmanian
languages
Fang
Umbundu
Setswana
Afro-Asiatic
Khoisan Fon
Yoruba
Peulh
Adangame
Erzya Bashkir
Khakas
Udmurt
Ingush
Tagalog
Hilgaynon
Bikol
Waray
Native American
dialects

12. 12

13. 13

14. What do these problems have in common
(besides being really hard)?

15. 1. Very messy, raw data, with both
redundancy and irrelevance
2. Users are also producers i.e. we cannot
just ‘build’ the system and hand it off
3. Domains are largely non-analytic (e.g.,
we don’t have a model/equations for
human trafficking)

16. 1. Very messy, raw data, with both
redundancy and irrelevance
2. Users are also producers i.e. we cannot
just ‘build’ the system and hand it off
3. Domains are largely non-analytic (e.g.,
we don’t have a model/equations for
human trafficking)

17. 1. Very messy, raw data, with both
redundancy and irrelevance
2. Users are also producers i.e. we cannot
just ‘build’ the system and hand it off
3. Domains are largely non-analytic (e.g.,
we don’t have a model/equations for
human trafficking)

19. 19
Space of design decisions
Raw data
Search+GUI
?
??
Representation +
Infrastructure
? ? ?

20. Space of design decisions
Search+GUI
Producer Consumer

21. 21

22. 22

23. 23
Space of design decisions
Raw data
Search+GUI
Knowledge
Graph
??
Representation +
Infrastructure
? ? ?

24. 24
Domain-specific
Insight Graphs
(DIG)

25. 25
Space of design decisions: example from human trafficking
Raw data
Search+GUI
Knowledge
Graph (KG)
Domain
discovery
Define KG
schema
Representation +
Infrastructure
Flexible
inputs
Query
reformulation
KG
Construction

27. The Knowledge Graph is noisy…how do
we cope?

28. Answer: Strategize around each triangle
Search+GUI
ConsumerProducer

29. Example from DIG: consumer triangle
Search+GUI
Consumer

31. 31
Anti-fragile query reformulation to satisfy user intent
SELECT ?ad ?ethnicity
WHERE
{
?ad a :Ad ;
:hair_color 'Auburn' ;
:review_site_id 'cg9469f'
;
:price_per_hour '500' ;
:name ’Claire Gold’ ;
:ethnicity ?ethnicity .
}
query 1
query 2
query 3
query 4
query n
Query
Reformulation
Keyword expansion • Context broadening • Constraint
relaxation
Precision
Recall
Elastic Search
100M entities
Ranked
Candidates

32. 32
Query-centric KG representation

33. 33
Infrastructure: Leverage existing ecosystems (there
are many!)

35. 35
Prosecutions

36. 36
User Testimonials

37. Showcasing THOR

38. 38
Other domains
Narcotics
Illegal weapons
sales
Fraudulent shipments
Securities fraud
Causal exploration
Geopolitical forecasting
Cyberattack
prediction

39. THANK YOU! QUESTIONS...
39

41. BACKUP

42. THOR: Text-enabled Humanitarian Operations in
Real-time

44. Impact and
Measurements

45. 45
Controlled (i.e. academic measurements)
0
10
20
30
40
50
60
70
80
90
100
0 - 0.1 < 0.2 < 0.3 < 0.4 < 0.5 < 0.6 < 0.7 < 0.8 < 0.9 <= 1.0
Average Precision of Retrieved Pages
DARPA MEMEX Eval (90K pages)
Point Fact Cluster ID Aggregate Facet
%Questions
Average Precision

46. 46
In-use impact (sex trafficking)
100 million+ escort ads
3 years data coverage
2 billion triples
100 law enforcement
offices
3 convictions

47. 47
NY County District Attorney (HTRU)
MEMEX tools getting
rolled out

48. 48
Memex tools getting rolled out

49. 49
Academic Output
~15 publications over the course of the program
• 7 more currently under review
• 2 best paper awards
• Upcoming special issue call on knowledge construction and management
• 2 upcoming books, incl. graduate-level textbook on knowledge graphs (MIT Press, 2018)
Multiple tutorials/demonstrations at top-tier academic conferences
• Tutorials on knowledge graph construction and data mining over Web corpora/unusual domains in
KDD17, ISWC17, AAAI18, WWW18
• At ISWC17, only full-day tutorial accepted; had near-capacity attendance
• Demos at ISWC17, AAAI18 (nominated for Best Demo)
• Case study at CHI18
Selected papers
• Knowledge Graphs for Social Good: An Entity-centric Search Engine for the Human Trafficking Domain
(IEEE Transactions on Big Data, 2017)
• Information Extraction in Illicit Domains (WWW, 2017)
• Unsupervised Entity Resolution on Multi-type Graphs (ISWC 2016)

50. 50
Broker
Rich club
effectStar cluster
Web formation
Social Science Studies

51. 51
• Subjective issue
• Architecture-level
evaluation
• Ablation analysis
“Ideal” Evaluation

53. 53
Space of design decisions
Raw data Search+GUI
?
??
? ?

54. 54
Domain-specific
Insight Graphs
(DIG)

55. 55
Structured query execution on noisy data
SELECT ?ad ?ethnicity
WHERE
{
?ad a :Ad ;
:hair_color 'Auburn' ;
:review_site_id 'cg9469f'
;
:price_per_hour '500' ;
:name ’Claire Gold’ ;
:ethnicity ?ethnicity .
}
query 1
query 2
query 3
query 4
query n
Query
Reformulation
Keyword expansion • Context broadening • Constraint
relaxation
Precision
Recall
Elastic Search
100M entities
Ranked
Candidates

56. 56
DIG capabilities
Aggregations
Facets
Dossier Generation
Networks
Provenance
Structured Queries
Interface Customization
• Capabilities that generic search
engines like Google do not
currently support
• Domain-specific
–Allows a user to specify her schema
–No prior constraints
• Insight
–Supports aggregations, network
analysis, faceted search, dossiers...
• Graph
–Uses a knowledge graph
representation + efficient NoSQL
query reformulation

57. Users want Situational Awareness i.e.
equipped with actionable insights

58. • Advanced name matching algorithm based on
machine learning, phonetic similarity and illicit
webpage-specific word embeddings
58
How can we tell when two actors are really one and the same?
Abbie
Candy
Kim
Lea
Nicki
Abby
Kandy
Kimmy
Leah
Nikki

59. • Evaluated on five investigative domains beyond human trafficking,
each with its own domain-specific needs
–Narcotics
–Counterfeit Electronics Manufacturing
–Securities Fraud
–Mail Shipment Fraud
–Illegal Weapons Sales
• User engagement was high
–Investigators were able to customize their domain in just one day, with less
than an hour of training
–Have expressed interest in continuing to refine and use the search engine
internally
59
Other use-cases

60. Relevance score
Matching search
criteria
highlighted
Image
extraction+face
and pose
analytics using
deep learning
Original URL

61. Dossier term
Activity timeline
Co-occurrence
statistics
Related ads