Specialty gene sets, such as virulence factors and antibiotic resistance genes, are of particular interest to infectious disease researchers. Much of the information about specialty genes’ function is described in literature but unavailable as structured data in bioinformatics databases. The steadily increasing volume of literature makes it difficult to manually find relevant papers and extract assertion sentences about specialty genes. This presentation describes efforts to build and an automatic classifier for such sentences. Experiments were conducted to assess the impact of the imbalance of positive and negative examples in source documents on classification; develop a support vector machine (SVM) classifier using term frequency-inverse document frequency (TF-IDF) representation of text; and assess the marginal benefit of additional training examples on the quality of the classifier. Analysis of learning curves indicates that additional training examples will not likely improve the quality of the classifier. We discuss options for other text representation schemes to investigate in order to improve the quality of the classifier as measured by F-score.

1. GBCB Seminar
April 24, 2014

2.  Biocuration in bacterial infectious diseases
 Existing approaches to biocuration
 Goals of current research
 Sentence classification for virulence factor
(VF) curation
 Future Research

3.  Large scale sequencing,
transcriptomics, proteomics and
metabolomics provide large volumes
of data about structure and function
 Valuable information about genes,
proteins and other biological entities
derived from interpretation of data
 Publications capture information that
researchers extract from data by
aggregating, integrating,
summarizing and analyzing
experiment results and interpreting
those results with respect to other
published results

4.  Gene annotation
◦ Virulence factors
◦ Antibiotic resistance
◦ Genomic metadata
 Experiment Metadata
◦ Transcriptomic metadata
◦ Metabolomic metadata
 Literature
◦ Named entity recognition
◦ Metadata tagging

5.  Automated annotation
◦ Example – RAST
◦ Transfer annotations based on similarity
◦ Metabolic reconstruction
 Community curation
◦ Example – WikiGenes
◦ Collaborative manual curation
 Model building
◦ Example - MetaFlux
◦ Predict missing components of pathways based
on FBA models
 Dedicated manual curation
◦ Example –
◦ PATRIC Curate entries with statements
traceable to literature

6. ◦ In 2009, half of biocurators were using text mining in
support of biocuration1
◦ Common use cases:
 Document prioritization
 Linking entities and relations to biological resources
such as GO or UniProt
 Identification of evidence
◦ Identification of evidence
 Pattern recognition - genomic location information
 Named entity recognition – T4SS components
 Event extraction – positive/negative regulation
1. PMID: 23110974

7.  Manual procedures are time
consuming and costly
 Volume of literature continues
to grow
 Commonly used search
techniques, such as keyword,
similarity searching, metadata
filtering, etc. can still yield
volumes of literature that are
difficult to analyze manually
 Some success with popular
tools but limitations

8. http://www.nature.com/nrmicro/journal/v8/n1/fig_tab/nrmicro2260_F2.html
http://stroke.nih.gov/materials/strokechallenges.htm

9.  Potentially brittle
methods, e.g.
dictionary lookups
 Questions of effort
required to extend
 Named entity
recognition does not
allows disambiguate
correctly
 Prioritizing documents
is still challenging
Textpresso Dictionary Entries
Adhesion to host
Adhesion to hosts
Adhesion to other organism during symbiotic interaction
Adhesion to other organism during symbiotic interactions
Adhesion to symbiont
Adhesion to symbionts
Agglutination during conjugation with cellular fusion
Agglutination during conjugation with cellular fusions
Agglutination during conjugation without cellular fusion
Agglutination during conjugation without cellular fusions

10.  Generalized set of biocuration tools to:
◦ Filter and prioritize documents
◦ Identify relevant assertion sentences within documents
◦ Extract entity and events
◦ Require minimal manual intervention
 Approach
◦ Address each objective separately
◦ Topic modeling and similarity measures for document
classification
◦ Term-frequency Inverse Document Frequency (TF-IDF) for
sentence classification
◦ Shallow semantic parsing for entity and event extraction
 Focus of this presentation is TF-IDF for sentence
classification and its limitations

11.  3 Key Components
◦ Data
◦ Representation scheme
◦ Algorithms
 Data
◦ Positive examples – VF assertion sentences
◦ Negative examples – Randomly selected from same
publications
 Representation
◦ TF-IDF
◦ Vector space representation
◦ Cosine of vectors measure of similarity
 Algorithms
◦ Supervised learning
 SVMs
 Ridge Classifier
 Perceptrons
 kNN
 SGD Classifier
 Naïve Bayes
 Random Forest
 AdaBoost
• Semisupervised Learning
• Label Spreading

12.  “Bacterial virulence factors enable a [pathogen] to replicate and disseminate within a
host in part by subverting or eluding host defenses.”1
 Example assertion sentences about virulence factors
 Mutations in the fimH gene of Salmonella typhimurium result in a non-fimbriate, non-adhesive
phenotype.2
 Unexpectedly, here we find that nonacylated LprG retains TLR2 activity. 3
 The autolysin Ami contributes to the adhesion of Listeria. 4
 Negative examples are randomly selected non-VF assertion sentences from the same set
of publications.
 VF Sentence Set 1 - PATRIC team of biocurators identified 4,696 assertion sentences in
1,127 publications about virulence in 5 genera: Escherichia, Listeria, Mycobacterium,
Salmonella, Shigella
 VF Sentence Set 2 - Second round of curation over initial results yield 3,716 VF assertion
sentences from 787 publications across 6 genera: Bartonella, Escherichia, Listeria,
Mycobacterium, Salmonella, Shigella
1. A. Cross, “What is a Virulence Factor” Crit Care. 2008; 12(6): 196.
2. Hancox, Yeh et al. 1997
3. Drage, Tsai et al. 2010
4. Milohanic, Jonquieres et al. 2001

13.  Term Frequency (TF)
tf(t,d) = # of occurrences of t in d
t is a term
d is a document
 Inverse Document Frequency (IDF)
idf(t,D) = log(N / |{d in D : t in d}|)
D is set of documents
N is number of document
 TF-IDF = tf(t,d) * idf(t,D)
 TF-IDF is
◦ large when high term frequency in document and low
term frequency in all documents
◦ small when term appears in many documents

14.  Bag of word model
 Ignores structure (syntax) and
meaning (semantics) of sentences
 Representation vector length is the
size of set of unique words in
corpus
 Stemming used to remove
morphological differences
 Each word is assigned an index in
the representation vector, V
 The value V[i] is non-zero if word
appears in sentence represented by
vector
 The non-zero value is a function of
the frequency of the word in the
sentence and the frequency of the
term in the corpus

15.  Support Vector Machine (SVM) is large
margin classifier
 Commonly used in text classification
 Initial results based on VF Sentence
Set 1
Image Source:http://en.wikipedia.org/wiki/File:Svm_max_sep_hyperplane_with_margin.png

16.  Non-VF, Predicted VF:
◦ “Collectively, these data suggest that EPEC 30-5-1(3) translocates reduced levels
of EspB into the host cell.”
◦ “Data were log-transformed to correct for heterogeneity of the variances where
necessary.”
◦ “Subsequently, the kanamycin resistance cassette from pVK4 was cloned into the
PstI site of pMP3, and the resulting plasmid pMP4 was used to target a disruption
in the cesF region of EHEC strain 85-170.”
 VF, Predicted Non-VF
◦ “Here, it is reported that the pO157-encoded Type V-secreted serine protease
EspP influences the intestinal colonization of calves. “
◦ “Here, we report that intragastric inoculation of a Shiga toxin 2 (Stx2)-producing
E. coli O157:H7 clinical isolate into infant rabbits led to severe diarrhea and
intestinal inflammation but no signs of HUS. “
◦ “The DsbLI system also comprises a functional redox pair”

17.  Adding additional examples is not likely to
substantially improve results as seen by error curve
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
0 5000 10000
All
Training Error
Validation Error

18.  8 Alternative Algorithms
 VF Sentence Set 2
 Select 10,000 most important features using
chi-square

19.  Machine learning technique that takes
advantage of unlabeled data
 Unlabeled data helps determine shape
of underlying data distribution
 Added randomly selected, unlabeled
sentences from VF publications
 Trained with 842 labeled and 4346
unlabeled
 Label Spreading is a semi-supervised
algorithm somewhat resilient to noise

20.  Algorithms have parameters not learned from
data
 SVMs, for example:
◦ C – balances training error and over-fitting
◦ Kernel – function to map data to high-dimensional
space, e.g. linear, polynomial
◦ Gamma – parameter in non-linear kernels, controls how
far influence of a training instance reaches
 Search combination of parameters
 Optimal results with linear kernel and slightly
smaller C than default

21.  Process of explicitly modeling relations between variables or explicitly
representing information not already in a representation scheme, for
example:
◦ Classify all numbers as NUMBER instead of numerals
◦ Replace gene/protein names with term GENE_Protein
 Used in text classification problems, e.g. phrase-based learning has
improved some rule-based classifiers.1
 Rule based learners may not be generalizable to other domains
 Taxonomic-structure of Unified Medical Language System (UMLS) used
to create semantic similarity measures. 2
 Most informative features can be detected automatically, e.g. chi-
square
 Manual feature engineering is not a viable option if our goal is
topic-independent support for biocuration
1. DOI:10.1.1.36.9770
2. PMID: 22580178

22.  Improve quality of data (quantity not likely helpful)
 Utilize multiple supervised algorithms,
ensemble and non-ensemble
 Use unlabeled data and semi-supervised
techniques
 Feature Selection
 Parameter Tuning
 Feature Engineering
 Given:
◦ High quality data in sufficient quantity
◦ State of the art machine learning algorithms
 How to improve results: Change Representation?

23.  TF-IDF
◦ Loss of syntactic and
semantic information
◦ No relation between
term index and
meaning
◦ No support for
disambiguation
◦ Feature engineering
extends vector
representation or
substitute specific for
more general terms – a
crude way to capture
semantic properties
 Ideal
Representation
◦ Capture semantic
similarity of words
◦ Does not require
feature engineering
◦ Minimal pre-
processing, e.g. no
mapping to
ontologies
◦ Improves precision
and recall

24.  Words represented as set of weights in vector
 Useful properties
◦ Semantically similar words in close proximity
◦ Methods for capturing phrases, e.g. “Secretion system”
◦ Captures some semantic features
 Trained with
◦ Skip-gram or CBOW algorithms
◦ Text, such as PubMed abstracts and open access papers
T. Mikolov, et. al. “Efficient Estimation of Word Representations in Vector Space.” 2013. http://arxiv.org/pdf/1301.3781.pdf

25.  Utilize distributed representations in
classification algorithms
 Compare SVM and multi-layered neural
network for classification
 Build on distributed word representation as
basis for shallow semantic parsing and
information extraction
 Apply to other specialty gene sets

26.  PATRIC Curators: Rebecca
Wattam, Chunhong Mao, David
Abraham, Meredith Wilson,
Yan Zhang
 Resources
◦ PATRIC www.patricbrc.org
◦ Python, NumPy, SciPy, Scikit-
Learn
◦ iPython
◦ Gensim
 Funding
◦ National Institute of Allergy and
Infectious Disease, National
Institutes of Health
CID Photo Here