The document outlines the key components in a text analysis pipeline including: - CorpusReader which reads text from a source into a Corpus without processing. - FeatureExtractor which converts text to features and Annotator which adds annotations to the Corpus. - Dataset contains feature representations of Documents from a Corpus. - Learner uses the Dataset to learn a Model, which is then used by Classifier to produce Classifications, evaluated by ClassificationEvaluator.

1. STAT Requirement Analysis 2/24/09

3. Flowchart and Definitions CorpusReader reads text from a source into Corpus . No processing is done and everything (label, metadata, etc.) stays in text format. FeatureExtrator converts the text in Document s to features* Annotator transforms the corpus into another corpus by adding annotations Corpus is a set of Document s in text format. Dataset is a set of Instance s which are feature representation of Document text. Learner uses the dataset to learn a model Model is a set parameters* learned from the data by Learner * Not modeled Classifier uses the model to predict classes in and produces a Classification Classification contains predictions and information about them ClassificationEvaluator computes the evaluation metrics for the Classification Text Data Corpus Dataset Model Classification ClassificationEvaluation contains evaluation metrics Classification Evaluation

4. CorpusReader FeatureExtractor Dataset Learner Model Classifier Classification ClassificationEvaluator ClassificationEvaluation Document Instance contains contains processed-by produces learns-from produced-by used-by produces evaluated-by classified-by produces produces 1 * 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 * 1 1 STAT Domain Model – v5 1 evaluate-on 1 This object holds the evaluation results calculated by ClassificationEvaluator and also reference to all relevant objects to produce useful reports Annotator Corpus 1 modifies 0..1 * Notably absent from this diagram are classes needed by Corpus (e.g.,WordList, Split, etc.) or by machine learning component s(e.g., DistanceMetric, ProbabilityDistribution, etc.) CorpusReader provides the protected variation for input sources (file, web, etc.) Instances are representation of documents needed for machine learning Classification hold the predictions produced by Classifier Document could contain Labels, Annotations, MetaData, etc. Any class to mutates the corpus, e.g., POSTagger Any class that converts the document components to feature vector. Also includes feature selection, aggregation, etc.