Introduction to AI & NLP (for digital humanities) AND Hands-on tagtog.net, a text annotation tool Lancaster University 2019 workshop, February 7th #dh #textMining #nlproc #nlu #training #trainingData #trainingCorpus #corpus #corpora #deepLearning #machineLearning #session #academia #industry #textAnalytics #juanmirocks #cejuela

1. LANCASTER, FEBRUARY 7TH

2. LANCASTER, FEBRUARY 7TH

4. Dr. Juan Miguel Cejuela Jorge Campos
Focus on product development and
frontend
Focus of his research has been on
Text Mining and Machine Learning
PhD in Computer Science MSc Computer Engineering
Munich, Germany Gdansk, Poland
@tagtog_net

5. LANCASTER, FEBRUARY 7TH

6. Source: Ex Machina; 2015; director&writer: Alex Garland; production: Universal Pictures International (UPI), Film4, DNA Films

7. AI à Machine Learning à Deep Learning

8. AI à Machine Learning à Deep Learning

18. Tokens: a, garage, has, charged,
me, £, 541, …
isEnglishWord(MOT) = no
hasMixedCase(eBay) = yes
Token ! ? :(, :pouting_face:

19. A machine learning model is an automatic algorithm that
maps inputs to outputs, for which it is optimized (learned).

20. A machine learning model is an automatic algorithm that
maps inputs to outputs, for which it is optimized (learned).

23. • Supervised Learning
• Semi-Supervised Learning
• Unsupervised Learning

24. • Online Learning
• Active Learning

25. LANCASTER, FEBRUARY 7TH

26. Unstructured data is text-heavy
Social Media, Voice, PDFs,
scientific articles, reports, etc.
IDC and EMC: Data will grow to 40 ZB by 2020
Unstructured data grows faster than structured data

33. Techniques to turn (large amounts) of unstructured text
that is understandable by humans, i.e. natural language,
into unambiguous, structured knowledge.

36. ?

38. à

39. à
!

40. à
à
!

41. à
à
❤
"

42. à
à
à
❤
"

43. à
à
à
❤
"
#

44. h"ps://www.youtube.com/watch?v=_eewlRCfewQ&
feature=youtu.be&t=560
h"ps://www.bbc.co.uk/programmes/p06jt7j4

45. Ambiguity !
(AI-complete problem)

46. Pipeline of Tasks !
(Errors compound)
… à … à … à …

47. Language Detec+on &
Topic Modeling

48. [Ms. Smith didn’t wait; she bound her dress and left.]
Sentence segmentation

49. [Ms.] [Smith] [didn’t] [wait] [;] …
Word segmenta-on &
Tokeniza-on

50. Ms. à PROPN
Smith à PROPN
did à VERB
n’t à ADV
…
Part-of-Speech (POS) tagging

51. Stemming & Lemmatization
Ms. Smith didn’t wait; she bound her dress and left.
à bind? bound?

52. Spelling Corrector

53. Parsing
(Cons+tuency, Shallow, Dependency)

54. Abbreviation Expansion

55. Coreference Resolution
London is the capital of England. It was founded
by the Romans, who named it Londinium.

56. Coreference Resolu,on
London is the capital of England. It was founded
by the Romans, who named it Londinium.
• London is the capital of England
• London was founded by the Romans
• London was named [by the Romans] Londinium

57. UTF-8, Special Characters !
Córdoba à C□rdoba
Ü õ Å ñ ö Œ Ô è í â
가-힣 русский язык বাংলা !ह#द% ελληνικά ‫ا‬‫ﻟ‬‫ﻌ‬َ‫ر‬َ‫ﺑ‬ِ‫ﯾ‬‫ﱠ‬‫ﺔ‬

58. Many file formats !"

59. LANCASTER, FEBRUARY 7TH

61. • Annota&on Type? NER? RE? Doc. Classifica&on?
• How many annotators?
• How many documents?
• Which documents?
• Training & Tes&ng?
• Time frame?
• Costs?
• …?

62. ! Documents‘ source?
" Representa1ve sample?
# Biases?

64. • Have several rounds to train the annotators
• Redefine the guidelines progressively
• Discard all first annota7ons un7l annota7on quality plateaus

66. true positive (tp)

67. true positive (tp)
false nega2ve (fn)

68. true posi)ve (tp)
false nega)ve (fn)
false posi)ve (fp)

69. true posi)ve (tp)
false negative (fn)
false posi)ve (fp)

70. true posi)ve (tp)
false nega)ve (fn)
false posi)ve (fp)

71. 1 annotator/document vs.
X >= 2 annotators/document ?

72. 1 annotator/document vs.
X >= 2 annotators/document ?
Always some repe,,on to
ensure IAA remains high

73. 1 annotator/document vs.
X >= 2 annotators/document ?
Always some repe,,on to
ensure IAA remains high
New annotators must go
through the same training

74. Test; 20%
Valida/on;
20%
Training; 60%

75. Test; 20%
Validation;
20%
Training; 60%
Don‘t tell the
annotators! !

76. Test; 20%
Validation;
20%
Training; 60%
Don‘t tell the
annotators! !
Report sta:s:cs (IAA, #Docs, etc.)
for each set

77. Ac#ve Learning

78. h"ps://www.tagtog.net/-corpora

79. ✅ Define clear goals
✅ Beware data biases
✅ Ensure high IAA (remains)
❤ Publish your data

80. LANCASTER, FEBRUARY 7TH

82. info@tagtog.net
@tagtog_net
The Text Annotation Tool to Train AI