Basis Technology's OSINT is a web-based application that enables analysts to extract and resolve named entities, including people, places, and organizations, from large quantities of multilingual text. OSINT is deployable in the cloud or on premises, and can be customized to address different data sources, knowledge bases, and workflow models.

1. A Collabora)ve, En)ty-­‐Centric Approach to
Mining OSINT Data
Gregor
Stewart
Director
of
Product
Management
Basis
Technology
Corpora=on
Open Source Intelligence !
and Research Association Inaugural Meeting

2. Key Issue: Things not Strings!
2

3. § Brief
Introduc=on
to
Basis
§ Review
two
technologies
that
take
us
from
Strings
to
Things:
– Extrac=on:
finding
names
and
types
in
text
– Resolu=on:
connec=ng
names
to
things
§ A
Search
Example,
illustra=ng:
– How
En=ty-­‐based
querying
can
help
find
things
quickly
– How
human
feedback
can
be
used
to
improve
accuracy
over
=me
Overview
3

4. Introduc)on: Basis Technology
4
English
Pronoun Verb Name
Pronoun+VerbVerb NounConjunction Title Name
Pronoun Verb NameDeterminer Noun
Verb Name Title Name NamePrep.
Noun VerbRel. Pronoun Verb Verb Prep. Verb Noun
Pronoun+Verb Adjective Noun Prep. Adjective Adjective
What is Big Text ?
It's huge volumes of multilingual , unstructured
text that must be processed to deliver insights
and build connections . It’s President Clinton
helping Malawi . Secretary Clinton in .
The 福島 meltdown . This is Big Text .
‫ﻣﻠﺴﺎ‬ ‫داﺑﺂ‬
Concept
Person
ConceptPlace
Place Person Place
Urdu: “Islamabad”
Japanese:
"Fukushima”

5. Introduc)on: Gregor Stewart
5

6. En)ty Extrac)on: What is it?
6
Unhasu Orchestra
Kim Jong-il
Kim
Hyun
Ri Sol-ju
Wangjeasan Light Music Band
Twitter
South Korea
Chosun Ilbo
Chosun
North Korea
China
Hyon Song-wol
Kim Jong-un
Kim Jong-un
Unhasu Orchestra
Chosun Ilbo
China
Hyon
Kim Jong-il
Kim
Ri Sol-ju
Wangjeasan Light
Music Band
Twitter
South Korea
Chosun
North Korea
Kim Jong-un
Kim Jong-un
Unhasu Orchestra
Unhasu Orchestra Chosun Ilbo
Chosun Ilbo
China
China
Hyun
Hyon Song-wol
Hyon

7. En)ty Extrac)on: How is it Done?
7
Probabilis=c
Extractor
Supervised
Model
Unsupervised
Model
Determinis=c
Extractor
Exact
Match
(GazeOeer)
PaOern
Match
(Regex)
En=ty
Redactor
En=ty
Joining
Input
Text
Filtering
Overlap
Adjudica=on
Tagged
Text
Domain
Text
Output
Text
User
Defined
Lists
User
Defined
PaOerns

8. Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
FISHING NEWS
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
FISHING NEWS
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
Un estadounidense que aprendió a saltar en paracaídas
durante la Segunda Guerra Mundial cumplió su sueño de
poner en práctica su habilidad a los 90 años de edad. Lester
Slate saltó este domingo en el estado de Maine acompañado
de un guía paracaidista. A pesar de haber volado en
numerosas ocasiones, según la prensa local, este veterano de
la marina estadounidense nunca se había lanzado en
paracaídas. Slate señaló que se sintió inspirado por el
expresidente de Estados Unidos Goerge H.W. Bush, quien
realizó un salto con motivo de su 85 cumpleaños en 2009.
Tras pisar tierra, el nonagenario dijo que esperaba poder
repetir en su 95 y 100 cumpleaños.
George H. W. Bush
41st U.S. President
ID: USPRES41
DOB: June 12, 1924
George Herbert Walker Bush is an American
politician who served as the 41st President of the
United States.
George W. Bush
43rd U.S. President
ID: USPRES43
DOB: July 6, 1946
George Walker Bush is an American politician and
businessman who was the 43rd President of the
United States of America from 2001 to 2009 and the
46th Governor of Texas from 1995 to 2000.
FISHING NEWS
Noelle Bush
ID: BD239852
Unknown Person
Unknown Organization
Penobscot Boat Builders
ID: TF354723
KENNEBUNKPORT, Maine — Three
generations of an American political
dynasty went fishing off the Maine coast.
The family set out together in the morning
on their new powerboat, built by
Penobscot Boat Builders.
On board were the first President Bush;
the second President Bush
and his niece, Noelle Bush.
1 Identifying Known Entities
2 Inferring New Entities
En)ty Resolu)on: What is it? (1)

9. En)ty Resolu)on: What is it? (2)
Alberto
Alberto
Alberto
Alberto
Alberto
Amos
Fernandez…
Alberto
M.
Fernandez…
Alberto
Fernandez…
Alberto
Fernandiz…
Albert
Fernandez…
Alberto
Alberto
Alberto
Alberto
Alberto
Fernandez…
…
Chief
of
Cabinet
…
Argen=na…
…Prof
of
Criminal
Law…
Alberto
Fernandez…
…
born
Sept
7,
1984
…
cycling
…
Madrid
Alberto
Fernandez…
…
born
in
Cuba
…
US
Ambassador
Alburto
Fernandez…
Alberto
Alberto
Fernandez
de
la
Puebla…
Alberto
Ratio of
Politicians to
Sportsmen?
2:1
Alberto
Fernandez…
Sportsmen?
YES
Nickname
“El Galleta?”
?

10. En)ty Resolu)on: How is it Done? (1)

11. En)ty Resolu)on: How is it Done? (2)
11

12. En)ty Resolu)on: How is it Done? (3)
12

13. En)ty Resolu)on: How is it Done? (4)
13
Resolu=on
Engine
Candidate
Selec=on
En=ty
Index
En=ty
Men=on
+
Context
Link
or
Ghost
Ranking
Knowledge
Base
Learned
Seeded
!

14. § In
a
far
away
place,
not
so
long
ago…
the
CIA
was
asked
by
the
US
President
to
assess
the
likelihood
that
the
Syrian
opposi=on
would
use
chemical
weapons
by
mid-­‐2014.
§ As
part
of
building
that
analysis,
and
because
there
are
Al-­‐Qaeda
elements
in
the
Syrian
opposi=on,
Alice
the
analyst
was
asked
to:
characterize
Al-­‐Qaeda’s
a2tude
to
using
chemical
weapons
against
Middle
Eastern
governments.
A (Convenient) Fic)on…
14

15. 15
A
document
that
Alice
needs
to
read…

16. 16
From:
Ayman
Al-­‐Zawahiri
(?)
To:
“Hafiz
Sultan”

17. 17
From:
Ayman
Al-­‐Zawahiri
(?)
To:
“Hafiz
Sultan”
Dear
Brother,
We
need
guidance
from
you
on
the
issue
of
using
chlorine
gas
technology.
It
was
reported
that
the
brothers
in
Iraq
have
used
it,
but
this
was
implicitly
denied
in
a
statement
issued
by
the
Islamic
State
of
Iraq.
The
brothers
where
Mahmud
is
have
the
poten=al
to
use
chlorine
gas
on
the
forces
of
the
apostates,
Jalal
Talabani
and
Mas'ud
Barzani,
and
have
already
considered
using
it.
However,
I
informed
them
that
maOers
as
serious
as
this
require
centralized
[coordina=on]
and
permission
from
the
senior
[al-­‐Qa'ida]
leadership,
because
the
gas
could
be
difficult
to
control
and
might
harm
some
people,
which
could
tarnish
our
image,
alienate
people
from
us,
and
so
on.”

18. 18
From:
Ayman
Al-­‐Zawahiri
(?)
To:
“Hafiz
Sultan”
Dear
Brother,
We
need
guidance
from
you
on
the
issue
of
using
chlorine
gas
technology.
It
was
reported
that
the
brothers
in
Iraq
have
used
it,
but
this
was
implicitly
denied
in
a
statement
issued
by
the
Islamic
State
of
Iraq.
The
brothers
where
Mahmud
is
have
the
poten=al
to
use
chlorine
gas
on
the
forces
of
the
apostates,
Jalal
Talabani
and
Mas'ud
Barzani,
and
have
already
considered
using
it.
However,
I
informed
them
that
maOers
as
serious
as
this
require
centralized
[coordina=on]
and
permission
from
the
senior
[al-­‐Qa'ida]
leadership,
because
the
gas
could
be
difficult
to
control
and
might
harm
some
people,
which
could
tarnish
our
image,
alienate
people
from
us,
and
so
on.”

19. 19
From:
Ayman
Al-­‐Zawahiri
(?)
To:
“Hafiz
Sultan”
3/28/07

20. 20
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11
…characterize
Al-­‐Qaeda’s
a2tude
to
using
chemical
weapons
against
Middle
Eastern
governments.

21. 21
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

22. 22
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

23. 23
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

24. 24
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

25. 25
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

26. 26
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

27. 27
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

28. 28
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

29. 29
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

30. 30
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

31. 31
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

32. 32
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

33. 33
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

34. 34
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

35. 35
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

36. 36
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

37. 37
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

38. 38
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

39. 39
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

40. 40
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

41. 41
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

42. 42
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

43. 43
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

44. 44
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

45. 45
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

46. 46
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

47. 47
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

48. 48
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

49. 49
3/28/07
Today
5/2/11
5/3/11
5/12/11
5/14/11

50. Summary
§ By
Connec=ng
Strings
to
Things
in
text,
exis=ng
knowledge
can
be
applied:
– And
we
can
know
where
exis=ng
knowledge
is
lacking
(ghosts)
§ Indexing
text
by
Things
allows
each
query
to
be
more
precise
and
produc=ve
– Fewer
queries,
fewer
adjustments,
fewer
things
to
read
§ By
using
abundant
human
feedback,
resolu=on
accuracy
can
get
beOer
over
=me
without
effort.
– More
subtle
dis=nc=ons
between
en==es
can
be
learned
by
example
§ But…

51. …it’s not the shoes
51

52. Thank You!
Gregor
Stewart
(gregor@basistech.com)
Director
of
Product
Management
Basis
Technology
Corpora=on
Open Source Intelligence !
and Research Association Inaugural Meeting