Uploaded byrpatonubi
PDF, PPTX9 views

mallet-tutorial.pdf mallet-tutorial.pdf mallet-tutorial.pdf

The document is a comprehensive tutorial on using MALLET for machine learning applications, focusing on text classification, sequence tagging, and topic modeling. It covers the key components such as data representation, classifiers, training methods, and instance management, detailing the process from data input to model training and evaluation. Additionally, it incorporates historical context and development information relevant to MALLET's implementation and functionality.

Embed presentation

Download as PDF, PPTX
1 / 120
2 / 120
3 / 120
4 / 120
5 / 120
6 / 120
7 / 120
8 / 120
9 / 120
10 / 120
11 / 120
12 / 120
13 / 120
14 / 120
15 / 120
16 / 120
17 / 120
18 / 120
19 / 120
20 / 120
21 / 120
22 / 120
23 / 120
24 / 120
25 / 120
26 / 120
27 / 120
28 / 120
29 / 120
30 / 120
31 / 120
32 / 120
33 / 120
34 / 120
35 / 120
36 / 120
37 / 120
38 / 120
39 / 120
40 / 120
41 / 120
42 / 120
43 / 120
44 / 120
45 / 120
46 / 120
47 / 120
48 / 120
49 / 120
50 / 120
51 / 120
52 / 120
53 / 120
54 / 120
55 / 120
56 / 120
57 / 120
58 / 120
59 / 120
60 / 120
61 / 120
62 / 120
63 / 120
64 / 120
65 / 120
66 / 120
67 / 120
68 / 120
69 / 120
70 / 120
71 / 120
72 / 120
73 / 120
74 / 120
75 / 120
76 / 120
77 / 120
78 / 120
79 / 120
80 / 120
81 / 120
82 / 120
83 / 120
84 / 120
85 / 120
86 / 120
87 / 120
88 / 120
89 / 120
90 / 120
91 / 120
92 / 120
93 / 120
94 / 120
95 / 120
96 / 120
97 / 120
98 / 120
99 / 120
100 / 120
101 / 120
102 / 120
103 / 120
104 / 120
105 / 120
106 / 120
107 / 120
108 / 120
109 / 120
110 / 120
111 / 120
112 / 120
113 / 120
114 / 120
115 / 120
116 / 120
117 / 120
118 / 120
119 / 120
120 / 120
Machine  Learning  with  MALLET http://mallet.cs.umass.edu/mallet-­‐tutorial.pdf David  Mimno   Department  of  Information  Science,   Cornell  University License:  CC  BY  4.0
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Who? • Andrew  McCallum  (most  of  the   work)   • Charles  Sutton,  Aron  Culotta,   Greg  Druck,  Kedar  Bellare,   Gaurav  Chandalia…   • Fernando  Pereira,  others  at   Penn…
Who  am  I? • Chief  maintainer  of  MALLET   • Primary  author  of  MALLET  topic  modeling   package
Why? • Motivation:  text  classification  and   information  extraction   • Commercial  machine  learning  (Just   Research,  WhizBang)   • Analysis  and  indexing  of  academic   publications:  Cora,  Rexa
What? • Text  focus:  data  is  discrete  rather  than   continuous,  even  when  values  could  be   continuous: double value = 3.0
How? • Command  line  scripts:   – bin/mallet  [command]  -­‐-­‐[option]  [value]  …   – Text  User  Interface  (“tui”)  classes   • Direct  Java  API   – http://mallet.cs.umass.edu/api Most of this talk
History • Version  0.4:  c2004   – Classes  in  edu.umass.cs.mallet.base.*   • Version  2.0:  c2008   – Classes  in  cc.mallet.*   – Major  changes  to  finite  state  transducer   package   – bin/mallet  vs.  specialized  scripts   – Java  1.5  generics
Learning  More • http://mallet.cs.umass.edu   – “Quick  Start”  guides,  focused  on  command  line   processing   – Developers’  guides,  with  Java  examples   • mallet-­‐dev@cs.umass.edu  mailing  list   – Low  volume,  but  can  be  bursty
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Models  for  Text  Data • Generative  models  (Multinomials)   – Naïve  Bayes   – Hidden  Markov  Models  (HMMs)   – Latent  Dirichlet  Topic  Models   • Discriminative  Regression  Models   – MaxEnt/Logistic  regression   – Conditional  Random  Fields  (CRFs)  
Representations • Transform  text   documents  to   vectors  x1,  x2,…   • Retain  meaning   of  vector  indices   • Ideally  sparsely Call me Ishmael. … Document
Representations • Transform  text   documents  to   vectors  x1,  x2,…   • Retain  meaning   of  vector  indices   • Ideally  sparsely 1.0 0.0 … 0.0 6.0 0.0 … 3.0 … Call me Ishmael. … xi Document
Representations • Elements  of  vector   are  called  feature   values   • Example:  Feature   at  row  345  is   number  of  times   “dog”  appears  in   document     1.0 0.0 … 0.0 6.0 0.0 … 3.0 … xi
Documents  to  Vectors Call me Ishmael. Document
Documents  to  Vectors Call me Ishmael. Document Call me Ishmael Tokens
Documents  to  Vectors Call me Ishmael Tokens call me ishmael Tokens
Documents  to  Vectors call me ishmael Tokens 473, 3591, 17 Features 17. ishmael … 473. call … 3591 me
Documents  to  Vectors 17 1.0 473 1.0 3591 1.0 Features (bag) 17. ishmael 473. call 3591 me 473, 3591, 17 Features (sequence) 17. ishmael … 473. call … 3591 me 17. ishmael … 473. call … 3591 me
Instances Email  message,  web  page,  sentence,  journal   abstract…     • Name   • Data   • Target/Label   • Source What is it called? What is the input? What is the output? What did it originally look like?
Instances • Name   • Data   • Target   • Source String ! TokenSequence ArrayList<Token> FeatureSequence int[] FeatureVector int -> double map cc.mallet.types
Alphabets TObjectIntHashMap map ArrayList entries ! int lookupIndex(Object o, boolean shouldAdd) ! Object lookupObject(int index) cc.mallet.types, gnu.trove 17. ishmael … 473. call … 3591 me
Alphabets TObjectIntHashMap map ArrayList entries ! int lookupIndex(Object o, boolean shouldAdd) ! Object lookupObject(int index) cc.mallet.types, gnu.trove 17. ishmael … 473. call … 3591 me for
Alphabets TObjectIntHashMap map ArrayList entries ! cc.mallet.types, gnu.trove 17. ishmael … 473. call … 3591 me void stopGrowth() ! void startGrowth() Do not add entries for new Objects -- default is to allow growth.
Creating  Instances • Instance   constructor   method   • Iterators new Instance(data, target, name, source) Iterator<Instance> FileIterator(File[], …) CsvIterator(FileReader, Pattern…) ArrayIterator(Object[]) … cc.mallet.pipe.iterator
Creating  Instances • FileIterator cc.mallet.pipe.iterator /data/bad/ /data/good/ Label from dir name Each instance in its own file
Creating  Instances • CsvIterator cc.mallet.pipe.iterator Name, label, data from regular expression groups. “CSV” is a lousy name. LineRegexIterator? Each instance on its own line 1001. Melville Call me Ishmael. Some years ago… 1002. Dickens It was the best of times, it was… ^([^t]+)t([^t]+)t(.*)
Instance  Pipelines • Sequential   transformations   of  instance  fields   (usually  Data)   • Pass  an   ArrayList<Pipe>   to  SerialPipes cc.mallet.pipe // “data” is a String CharSequence2TokenSequence // tokenize with regexp TokenSequenceLowercase // modify each token’s text TokenSequenceRemoveStopwords // drop some tokens TokenSequence2FeatureSequence // convert token Strings to ints FeatureSequence2FeatureVector // lose order, count duplicates
Instance  Pipelines • A  small  number   of  pipes  modify   the  “target”  field   • There  are  now   two  alphabets:   data  and  label cc.mallet.pipe, cc.mallet.types // “target” is a String Target2Label // convert String to int // “target” is now a Label Alphabet > LabelAlphabet
Label  objects • Weights  on  a   fixed  set  of   classes   • For  training  data,   weight  for   correct  label  is   1.0,  all  others  0.0 cc.mallet.types implements Labeling ! int getBestIndex() Label getBestLabel() You cannot create a Label, they are only produced by LabelAlphabet
InstanceLists • A  List  of  Instance   objects,  along   with  a  Pipe,  data   Alphabet,  and   LabelAlphabet cc.mallet.types InstanceList instances = new InstanceList(pipe); ! instances.addThruPipe(iterator);
Putting  it  all  together ArrayList<Pipe> pipeList = new ArrayList<Pipe>(); ! pipeList.add(new Target2Label()); pipeList.add(new CharSequence2TokenSequence()); pipeList.add(new TokenSequence2FeatureSequence()); pipeList.add(new FeatureSequence2FeatureVector()); ! InstanceList instances = new InstanceList(new SerialPipes(pipeList)); ! instances.addThruPipe(new FileIterator(. . .));
Persistent  Storage • Most  MALLET   classes  use  Java   serialization  to   store  models  and   data java.io ObjectOutputStream oos = new ObjectOutputStream(…); oos.writeObject(instances); oos.close(); Pipes, data objects, labelings, etc all need to implement Serializable. ! Be sure to include custom classes in classpath, or you get a StreamCorruptedException
Review • What  are  the  four  main  fields  in  an   Instance?
Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?
Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?   • How  do  we  modify  the  value  of  Instance   fields?
Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?   • How  do  we  modify  the  value  of  Instance   fields?   • Name  some  classes  that  appear  in  the   “data”  field.
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Classifier  objects • Classifiers  map   from  instances  to   distributions   over  a  fixed  set   of  classes   • MaxEnt,  Naïve   Bayes,  Decision   Trees… cc.mallet.classify Given data Which class is best? ! (this one!) watery NN JJ PRP VB CC
Classifier  objects • Classifiers  map   from  instances  to   distributions   over  a  fixed  set   of  classes   • MaxEnt,  Naïve   Bayes,  Decision   Trees… cc.mallet.classify Labeling labeling = classifier.classify(instance); ! Label l = labeling.getBestLabel(); ! System.out.print(instance + “t”); System.out.println(l);
Training  Classifier  objects cc.mallet.classify ClassifierTrainer trainer = new MaxEntTrainer(); ! Classifier classifier = trainer.train(instances); • Each  type  of   classifier  has  one   or  more   ClassifierTrainer   classes
Training  Classifier  objects cc.mallet.optimize log P(Labels | Data) = log f(label1, data1, w) + log f(label2, data2, w) + log f(label3, data3, w) + … • Some  classifiers   require   numerical   optimization  of   an  objective   function. Maximize w.r.t. w!
Parameters  w • Association   between   feature,  class   label   • How  many   parameters  for  K   classes  and  N   features? action     NN   0.13   action     VB   -­‐0.1   action     JJ   -­‐0.21   SUFF-­‐tion   NN   1.3   SUFF-­‐tion   VB   -­‐2.1   SUFF-­‐tion   JJ   -­‐1.7   SUFF-­‐on   NN   0.01   SUFF-­‐on   VB   -­‐0.02   …  
Training  Classifier  objects cc.mallet.optimize interface Optimizer boolean optimize() ! interface Optimizable interface ByValue interface ByValueGradient Limited-memory BFGS, Conjugate gradient… Specific objective functions
Training  Classifier  objects cc.mallet.classify MaxEntOptimizableByLabelLikelihood double[] getParameters() void setParameters(double[] parameters) … ! double getValue() void getValueGradient(double[] buffer) Log likelihood and its first derivative For Optimizable interface
Evaluation  of  Classifiers • Create   random  test/ train  splits cc.mallet.types InstanceList[] instanceLists = instances.split(new Randoms(), new double[] {0.9, 0.1, 0.0}); 90% training 10% testing 0% validation
Evaluation  of  Classifiers • The  Trial  class   stores  the   results  of   classifications   on  an   InstanceList   (testing  or   training) cc.mallet.classify Trial(Classifier c, InstanceList list) double getAccuracy() double getAverageRank() double getF1(int/Label/Object) double getPrecision(…) double getRecall(…)
Review • I  have  invented  a  new  classifier:  David   regression.   – What  class  should  I  implement  to  classify   instances?
Review • I  have  invented  a  new  classifier:  David   regression.   – What  class  should  I  implement  to  train  a  David   regression  classifier?
Review • I  have  invented  a  new  classifier:  David   regression.   – I  want  to  train  using  ByValueGradient.  What   mathematical  functions  do  I  need  to  code  up,   and  what  class  should  I  put  them  in?
Review • I  have  invented  a  new  classifier:  David   regression.   – How  would  I  check  whether  my  new  classifier   works  better  than  Naïve  Bayes?
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Sequence  Tagging • Data  occurs  in   sequences   • Categorical  labels   for  each  position   • Labels  are   correlated DET          NN          VBS                VBG the          dog        likes          running
Sequence  Tagging • Data  occurs  in   sequences   • Categorical  labels   for  each  position   • Labels  are   correlated ??                ??            ??                        ?? the          dog        likes          running
Sequence  Tagging • Classification:  n-­‐way   • Sequence  Tagging:  nT-­‐way NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC or          red      dogs      on          blue    trees
Avoiding  Exponential  Blowup • Markov  property   • Dynamic  programming Andrei Markov
Avoiding  Exponential  Blowup • Markov  property   • Dynamic  programming This one Given this one Is independent of these Andrei Markov DET  JJ  NN  VB
Avoiding  Exponential  Blowup • Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC or          red      dogs      on          blue    trees Andrei Markov
Avoiding  Exponential  Blowup • Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC red      dogs      on          blue    trees Andrei Markov
Avoiding  Exponential  Blowup • Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC                                    dogs      on          blue    trees Andrei Markov
Hidden  Markov  Models  and  Conditional   Random  Fields • Hidden  Markov   Model:  fully   generative   ! • Conditional  Random   Field:  conditional P(Labels | Data) = P(Data, Labels) / P(Data) P(Labels | Data)
Hidden  Markov  Models  and  Conditional   Random  Fields • Hidden  Markov  Model:   simple  (independent)   output  space   ! • Conditional  Random   Field:  arbitrarily   complicated  outputs “NSF-funded” “NSF-funded” CAPITALIZED HYPHENATED ENDS-WITH-ed ENDS-WITH-d …
Hidden  Markov  Models  and  Conditional   Random  Fields FeatureSequence FeatureVectorSequence ! FeatureVector[] int[] • Hidden  Markov  Model:   simple  (independent)   output  space   ! • Conditional  Random   Field:  arbitrarily   complicated  outputs
Importing  Data • SimpleTagger   format:  one   word  per  line,   with  instances   delimited  by  a   blank  line Call VB me PPN Ishmael NNP . . ! Some JJ years NNS …
Importing  Data • SimpleTagger   format:  one   word  per  line,   with  instances   delimited  by  a   blank  line Call SUFF-ll VB me TWO_LETTERS PPN Ishmael BIBLICAL_NAME NNP . PUNCTUATION . ! Some CAPITALIZED JJ years TIME SUFF-s NNS …
Importing  Data LineGroupIterator ! SimpleTaggerSentence2TokenSequence() //String to Tokens, handles labels ! TokenSequence2FeatureVectorSequence() //Token objects to FeatureVectors cc.mallet.pipe, cc.mallet.pipe.iterator
Importing  Data LineGroupIterator ! SimpleTaggerSentence2TokenSequence() //String to Tokens, handles labels ! [Pipes that modify tokens] ! TokenSequence2FeatureVectorSequence() //Token objects to FeatureVectors cc.mallet.pipe, cc.mallet.pipe.iterator
Importing  Data //Ishmael TokenTextCharSuffix(“C2=”, 2) //Ishmael C2=el RegexMatches(“CAP”, Pattern.compile(“p{Lu}.*”)) //Ishmael C2=el CAP LexiconMembership(“NAME”, new File(‘names’), false) //Ishmael C2=el CAP NAME cc.mallet.pipe.tsf must match entire string one name per line ignore case?
Sliding  window  features a        red        dog        on        a        blue        tree
Sliding  window  features a        red        dog        on        a        blue        tree
Sliding  window  features a        red        dog        on        a        blue        tree red@-1
Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2
Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2 on@1
Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2 on@1 a@-2_&_red@-1
Importing  Data int[][] conjunctions = new int[3][]; conjunctions[0] = new int[] { -1 }; conjunctions[1] = new int[] { 1 }; conjunctions[2] = new int[] { -2, -1 }; ! OffsetConjunctions(conjunctions) ! // a@-2_&_red@-1 on@1 cc.mallet.pipe.tsf previous position next position previous two
Importing  Data int[][] conjunctions = new int[3][]; conjunctions[0] = new int[] { -1 }; conjunctions[1] = new int[] { 1 }; conjunctions[2] = new int[] { -2, -1 }; ! TokenTextCharSuffix("C1=", 1) OffsetConjunctions(conjunctions) ! // a@-2_&_red@-1 a@-2_&_C1=d@-1 cc.mallet.pipe.tsf previous position next position previous two
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden)
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden) DET P(DET)
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden) DET           the P(the | DET)
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN the P(NN | DET)
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN the          dog P(dog | NN)
Finite  State  Transducers • Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN          VBS the          dog P(VBS | NN)
How  many  parameters? • Determines   efficiency  of   training   • Too  many  leads   to  overfitting Trick: Don’t allow certain transitions P(VBS | DET) = 0
How  many  parameters? • Determines   efficiency  of   training   • Too  many  leads   to  overfitting DET          NN          VBS ! the          dog          runs DET          NN          VBS ! the          dog          runs DET          NN          VBS ! the          dog          runs
Finite  State  Transducers abstract class Transducer CRF HMM ! abstract class TransducerTrainer CRFTrainerByLabelLikelihood HMMTrainerByLikelihood cc.mallet.fst
Finite  State  Transducers cc.mallet.fst First order: one weight for every pair of labels and observations. CRF crf = new CRF(pipe, null); crf.addFullyConnectedStates(); // or crf.addStatesForLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
Finite  State  Transducers cc.mallet.fst “three-quarter” order: one weight for every pair of labels and observations. crf.addStatesForThreeQuarterLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
Finite  State  Transducers cc.mallet.fst Second order: one weight for every triplet of labels and observations. crf.addStatesForBiLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
Finite  State  Transducers cc.mallet.fst “Half” order: equivalent to independent classifiers, except some transitions may be illegal. crf.addStatesForHalfLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
Training  a  transducer CRF crf = new CRF(pipe, null); crf.addStatesForLabelsConnectedAsIn(trainingInstances); CRFTrainerByLabelLikelihood trainer = new CRFTrainerByLabelLikelihood(crf); ! trainer.train(); cc.mallet.fst
Evaluating  a  transducer CRFTrainerByLabelLikelihood trainer = new CRFTrainerByLabelLikelihood(transducer); ! TransducerEvaluator evaluator = new TokenAccuracyEvaluator(testing, "testing")); ! trainer.addEvaluator(evaluator); ! trainer.train(); cc.mallet.fst
Applying  a  transducer Sequence output = transducer.transduce (input); ! for (int index=0; index < input.size(); input++) { System.out.print(input.get(index) + “/”); System.out.print(output.get(index) + “ “); } cc.mallet.fst
Review • How  do  you  add  new  features  to   TokenSequences?
Review • How  do  you  add  new  features  to   TokenSequences?   • What  are  three  factors  that  affect  the   number  of  parameters  in  a  model?
Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
Topics:  “Semantic  Groups” News Article
Topics:  “Semantic  Groups” “Sports” “Negotiation” News Article
Topics:  “Semantic  Groups” “Sports” “Negotiation” News Article team player game strike deadline union
Topics:  “Semantic  Groups” News Article team player game strike deadline union
Series  Yankees  Sox  Red  World  League  game  Boston  team  games   baseball  Mets  Game  series  won  Clemens  Braves  Yankee  teams
players  League  owners  league  baseball  union  commissioner   Baseball  Association  labor  Commissioner  Football  major  teams   Selig  agreement  strike  team  bargaining
Training  a  Topic  Model cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate();
Evaluating  a  Topic  Model cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate(); ! MarginalProbEstimator evaluator = lda.getProbEstimator(); ! double logLikelihood = evaluator.evaluateLeftToRight(testing, 10, false, null);
Inferring  topics  for  new  documents cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate(); ! TopicInferencer inferencer = lda.getInferencer(); ! double[] topicProbs = inferencer.getSampledDistribution(instance, 100, 10, 10);
More  than  words… • Text  collections   mix  free  text  and   structured  data David Mimno Andrew McCallum UAI 2008 …
More  than  words… • Text  collections   mix  free  text  and   structured  data David Mimno Andrew McCallum UAI 2008 ! “Topic models conditioned on arbitrary features using Dirichlet-multinomial regression. …”
Dirichlet-­‐multinomial  Regression   (DMR) The  corpus  specifies  a  vector  of  real-­‐valued   features  (x)  for  each  document,  of  length  F.   Each  topic  has  an  F-­‐length  vector  of   parameters.
Topic  parameters  for  feature  “published  in   JMLR” 2.27 kernel,  kernels,  rational  kernels,  string  kernels,  fisher  kernel 1.74 bounds,  vc  dimension,  bound,  upper  bound,  lower  bounds 1.41 reinforcement  learning,  learning,  reinforcement 1.40 blind  source  separation,  source  separation,  separation,  channel 1.37 nearest  neighbor,  boosting,  nearest  neighbors,  adaboost -­‐1.12 agent,  agents,  multi  agent,  autonomous  agents -­‐1.21 strategies,  strategy,  adaptation,  adaptive,  driven -­‐1.23 retrieval,  information  retrieval,  query,  query  expansion -­‐1.36 web,  web  pages,  web  page,  world  wide  web,  web  sites -­‐1.44 user,  users,  user  interface,  interactive,  interface
Feature  parameters  for  RL  topic 2.99 Sridhar  Mahadevan 2.88 ICML 2.56 Kenji  Doya 2.45 ECML 2.19 Machine  Learning  Journal -­‐1.38 ACL -­‐1.47 CVPR -­‐1.54 IEEE  Trans.  PAMI -­‐1.64 COLING -­‐3.76 <default>
Topic  parameters  for  feature  “published  in   UAI” 2.88 bayesian  networks,  bayesian  network,  belief  networks 2.26 qualitative,  reasoning,  qualitative  reasoning,  qualitative  simulation 2.25 probability,  probabilities,  probability  distributions, 2.25 uncertainty,  symbolic,  sketch,  primal  sketch,  uncertain,  connectionist 2.11 reasoning,  logic,  default  reasoning,  nonmonotonic  reasoning -­‐1.29 shape,  deformable,  shapes,  contour,  active  contour -­‐1.36 digital  libraries,  digital  library,  digital,  library -­‐1.37 workshop  report,  invited  talk,  international  conference,  report -­‐1.50 descriptions,  description,  top,  bottom,  top  bottom -­‐1.50 nearest  neighbor,  boosting,  nearest  neighbors,  adaboost
Feature  parameters  for  Bayes  nets  topic 2.88 UAI 2.41 Mary-­‐Anne  Williams 2.23 Ashraf  M.  Abdelbar 2.15 Philippe  Smets 2.04 Loopy  Belief  Propagation  for  Approximate  Inference  (Murphy,  Weiss,   and  Jordan,  UAI,  1999) -­‐1.16 Probabilistic  Semantics  for  Nonmonotonic  Reasoning  (Pearl,  KR,  1989) -­‐1.38 COLING -­‐1.50 Neural  Networks -­‐2.24 ICRA -­‐3.36 <default>
Dirichlet-­‐multinomial  Regression • Arbitrary  observed  features  of  documents   • Target  contains  FeatureVector DMRTopicModel dmr = new DMRTopicModel (numTopics); ! dmr.addInstances(training); dmr.estimate(); ! dmr.writeParameters(new File("dmr.parameters"));
Polylingual  Topic  Modeling • Topics  exist  in  more  languages  than  you   could  possibly  learn   • Topically  comparable  documents  are  much   easier  to  get  than  translation  sets   • Translation  dictionaries   – cover  pairs,  not  sets  of  languages   – miss  technical  vocabulary   – aren’t  available  for  low-­‐resource  languages
Topics  from   European   Parliament   Proceedings
Topics  from   European   Parliament   Proceedings
Topics  from   Wikipedia
Aligned  instance  lists dog… chien… hund… cat… chat… pig… schwein…
Polylingual  Topics InstanceList[] training = new InstanceList[] { english, german, arabic, mahican }; ! PolylingualTopicModel pltm = new PolylingualTopicModel(numTopics); ! pltm.addInstances(training);
MALLET  hands-­‐on  tutorial http://mallet.cs.umass.edu/mallet-­‐handson.tar.gz

More Related Content

PPTX
Gpu programming with java
byGary Sieling
 
PPT
Hands on Mahout!
byOSCON Byrum
 
PDF
WISS 2015 - Machine Learning lecture by Ludovic Samper
byAntidot
 
PDF
Machine learning pipeline with spark ml
bydatamantra
 
PDF
Оформление пайплайна в NLP проекте​, Виталий Радченко. 22 июня, 2019
byMail.ru Group
 
PDF
Mallet
byShatakirti Er
 
PDF
Introduction to and Extending Spark ML
byHolden Karau
 
PDF
Magpie
byJan Stypka
 
Gpu programming with java
byGary Sieling
 
Hands on Mahout!
byOSCON Byrum
 
WISS 2015 - Machine Learning lecture by Ludovic Samper
byAntidot
 
Machine learning pipeline with spark ml
bydatamantra
 
Оформление пайплайна в NLP проекте​, Виталий Радченко. 22 июня, 2019
byMail.ru Group
 
Mallet
byShatakirti Er
 
Introduction to and Extending Spark ML
byHolden Karau
 
Magpie
byJan Stypka
 

Similar to mallet-tutorial.pdf mallet-tutorial.pdf mallet-tutorial.pdf

PPTX
05 -- Feature Engineering (Text).pptxiuy
bySravani477269
 
PPTX
MLforCyber_MLDataSetsandFeatures_Presentation.pptx
byAaryaNaik8
 
PPTX
Apache Spark MLlib
byZahra Eskandari
 
PDF
Machine Learning Pipelines
byjeykottalam
 
PPTX
Introduction to Spark ML
byHolden Karau
 
PDF
Machine learning with Apache Spark MLlib | Big Data Hadoop Spark Tutorial | C...
byCloudxLab
 
PDF
Text categorization with Lucene and Solr
byTommaso Teofili
 
PDF
Automatic Text Classification Of News Blog using Machine Learning
byIRJET Journal
 
PDF
Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...
byChris Fregly
 
PPTX
Learning with classification and clustering, neural networks
byShaun D'Souza
 
PDF
Statistical Machine Learning for Text Classification with scikit-learn and NLTK
byOlivier Grisel
 
PPT
20070702 Text Categorization
bymidi
 
PDF
(Py)testing the Limits of Machine Learning
byRebecca Bilbro
 
PDF
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Real-time Aggregations, Ap...
byData Con LA
 
PDF
Session 2 - Akyildiz, Beinecke, Yee at MLconf NYC
byMLconf
 
PDF
Introduction to Spark ML Pipelines Workshop
byHolden Karau
 
PDF
Extending spark ML for custom models now with python!
byHolden Karau
 
PPTX
From Story-Telling to Production
byKwan-yuet Ho
 
PDF
Text Classification Powered by Apache Mahout and Lucene
bylucenerevolution
 
PDF
Lecture13 xing fei-fei
byTianlu Wang
 
05 -- Feature Engineering (Text).pptxiuy
bySravani477269
 
MLforCyber_MLDataSetsandFeatures_Presentation.pptx
byAaryaNaik8
 
Apache Spark MLlib
byZahra Eskandari
 
Machine Learning Pipelines
byjeykottalam
 
Introduction to Spark ML
byHolden Karau
 
Machine learning with Apache Spark MLlib | Big Data Hadoop Spark Tutorial | C...
byCloudxLab
 
Text categorization with Lucene and Solr
byTommaso Teofili
 
Automatic Text Classification Of News Blog using Machine Learning
byIRJET Journal
 
Practical Data Science Workshop - Recommendation Systems - Collaborative Filt...
byChris Fregly
 
Learning with classification and clustering, neural networks
byShaun D'Souza
 
Statistical Machine Learning for Text Classification with scikit-learn and NLTK
byOlivier Grisel
 
20070702 Text Categorization
bymidi
 
(Py)testing the Limits of Machine Learning
byRebecca Bilbro
 
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Real-time Aggregations, Ap...
byData Con LA
 
Session 2 - Akyildiz, Beinecke, Yee at MLconf NYC
byMLconf
 
Introduction to Spark ML Pipelines Workshop
byHolden Karau
 
Extending spark ML for custom models now with python!
byHolden Karau
 
From Story-Telling to Production
byKwan-yuet Ho
 
Text Classification Powered by Apache Mahout and Lucene
bylucenerevolution
 
Lecture13 xing fei-fei
byTianlu Wang
 

More from rpatonubi

PPT
Reconocimiento de voz RUA - Tema 6.ppt Reconocimiento de voz RUA - Tema 6.ppt
byrpatonubi
 
PPTX
Unit-1.pptxUnit-1.pptxUnit-1.pptxUnit-1.pptx
byrpatonubi
 
PPTX
docker_1.pptxdocker_1.pptxdocker_1.pptxdocker_1.pptx
byrpatonubi
 
PPTX
Unit-1-es.pptx Unit-1-es.pptxUnit-1-es.pptx
byrpatonubi
 
PPT
Regresion T6.ppt Regresion T6.ppt Regresion T6.ppt
byrpatonubi
 
PPTX
ExpAgentesIntro.pptxExpAgentesIntro.pptx
byrpatonubi
 
PPTX
M__tricas_y_KPIs.pptxM__tricas_y_KPIs.pptxM__tricas_y_KPIs.pptxM__tricas_y_KP...
byrpatonubi
 
PPT
Agentes_inteligentes2(2005-II).ppt ExpAgentesIntro.pptx
byrpatonubi
 
PPT
PLAN y GOBIERNO.pptPLAN y GOBIERNO.pptPLAN y GOBIERNO.pptPLAN y GOBIERNO.pptP...
byrpatonubi
 
PPT
SQL_ADO.NET.pptSQL_ADO.NET.pptSQL_ADO.NET.ppt
byrpatonubi
 
PPTX
Chapter1_Introduction_v2_20230112.pptxChapter1_Introduction_v2_20230112.pptx
byrpatonubi
 
PPTX
plantilla-170314003514 plantilla-170314003514.pptx
byrpatonubi
 
PPT
turban_bi2e_pp_ch01.pptturban_bi2e_pp_ch01.pptturban_bi2e_pp_ch01.pptturban_b...
byrpatonubi
 
PPT
T6.pptT6.pptT6.pptT6.pptT6.pptT6.pptT6.ppt
byrpatonubi
 
PPT
PLAN y GOBIERNO - Oscar Vega PLAN y GOBIERNO - Oscar Vega
byrpatonubi
 
Reconocimiento de voz RUA - Tema 6.ppt Reconocimiento de voz RUA - Tema 6.ppt
byrpatonubi
 
Unit-1.pptxUnit-1.pptxUnit-1.pptxUnit-1.pptx
byrpatonubi
 
docker_1.pptxdocker_1.pptxdocker_1.pptxdocker_1.pptx
byrpatonubi
 
Unit-1-es.pptx Unit-1-es.pptxUnit-1-es.pptx
byrpatonubi
 
Regresion T6.ppt Regresion T6.ppt Regresion T6.ppt
byrpatonubi
 
ExpAgentesIntro.pptxExpAgentesIntro.pptx
byrpatonubi
 
M__tricas_y_KPIs.pptxM__tricas_y_KPIs.pptxM__tricas_y_KPIs.pptxM__tricas_y_KP...
byrpatonubi
 
Agentes_inteligentes2(2005-II).ppt ExpAgentesIntro.pptx
byrpatonubi
 
PLAN y GOBIERNO.pptPLAN y GOBIERNO.pptPLAN y GOBIERNO.pptPLAN y GOBIERNO.pptP...
byrpatonubi
 
SQL_ADO.NET.pptSQL_ADO.NET.pptSQL_ADO.NET.ppt
byrpatonubi
 
Chapter1_Introduction_v2_20230112.pptxChapter1_Introduction_v2_20230112.pptx
byrpatonubi
 
plantilla-170314003514 plantilla-170314003514.pptx
byrpatonubi
 
turban_bi2e_pp_ch01.pptturban_bi2e_pp_ch01.pptturban_bi2e_pp_ch01.pptturban_b...
byrpatonubi
 
T6.pptT6.pptT6.pptT6.pptT6.pptT6.pptT6.ppt
byrpatonubi
 
PLAN y GOBIERNO - Oscar Vega PLAN y GOBIERNO - Oscar Vega
byrpatonubi
 

Recently uploaded

PPT
Memory Organization In Assembly Language
byumairmuhammad0409
 
PDF
Design For Test - Getting Test Automation value from Design Expressions.
byDavid Harrison
 
PDF
Beacon Kit slides tech framework for world game (s) pdf
bySteven McGee
 
PPTX
Computer Science Degree for College .pptx
byHanenek1
 
PPTX
mc l2 Overview of Computer and Web-Technology.pptx
byavanikharde
 
PPTX
CaseStudy-CloudMigration -For USE CASE -Customer Success Info -6-Aug-25.pptx
bywinlentech
 
PPTX
Overview-of-Anti-DDoS-Solutions-On-Premise-vs-On-Cloud.pptx
bybunhongkruy
 
PDF
tokiohotellistening.pdf for tokiohotelfans
byrafaelgombos1234
 
PPT
Lecture_3 Network Securityyyeyeyyeeyyeywy.ppt
bysawsan202
 
PPTX
Introduction Digital Signal Processing.pptx
byumairmuhammad0409
 
PPTX
Ideathon template.pptx it is a ideathon template
bykonav71133
 
PPTX
Anti-DDoS Service Introduction（Customer Version） - 2025(1).pptx
bybunhongkruy
 
PDF
Key Duties and Roles of an Ecommerce Developer Singapore
byHachi Web Solution
 
PDF
Here are the winners of KALIDASA SAMMAN.
byglcppro
 
PDF
classification of elements and periodicity.pdf
byaryanshreya2010
 
DOCX
BestMaker.ai: The Complete Brand Story, Founder's Vision, and AI Creative Pla...
byssuser7381d6
 
PPTX
PRITHVI theater, A true artist. Newzfeaturez.pptx
bysc05digital
 
PPTX
SriLank SLT LTE Network Sharing V2.pptx
bynthuang
 
PPTX
COĞRAFYA ödevi 70 sayfa güzeldir hayırlı olsun
bydtilter
 
PPTX
Dalhousie University Diploma
by文凭办理维多利亚大学购买毕业证学位认证知乎【Q微:1954292140】
 
Memory Organization In Assembly Language
byumairmuhammad0409
 
Design For Test - Getting Test Automation value from Design Expressions.
byDavid Harrison
 
Beacon Kit slides tech framework for world game (s) pdf
bySteven McGee
 
Computer Science Degree for College .pptx
byHanenek1
 
mc l2 Overview of Computer and Web-Technology.pptx
byavanikharde
 
CaseStudy-CloudMigration -For USE CASE -Customer Success Info -6-Aug-25.pptx
bywinlentech
 
Overview-of-Anti-DDoS-Solutions-On-Premise-vs-On-Cloud.pptx
bybunhongkruy
 
tokiohotellistening.pdf for tokiohotelfans
byrafaelgombos1234
 
Lecture_3 Network Securityyyeyeyyeeyyeywy.ppt
bysawsan202
 
Introduction Digital Signal Processing.pptx
byumairmuhammad0409
 
Ideathon template.pptx it is a ideathon template
bykonav71133
 
Anti-DDoS Service Introduction（Customer Version） - 2025(1).pptx
bybunhongkruy
 
Key Duties and Roles of an Ecommerce Developer Singapore
byHachi Web Solution
 
Here are the winners of KALIDASA SAMMAN.
byglcppro
 
classification of elements and periodicity.pdf
byaryanshreya2010
 
BestMaker.ai: The Complete Brand Story, Founder's Vision, and AI Creative Pla...
byssuser7381d6
 
PRITHVI theater, A true artist. Newzfeaturez.pptx
bysc05digital
 
SriLank SLT LTE Network Sharing V2.pptx
bynthuang
 
COĞRAFYA ödevi 70 sayfa güzeldir hayırlı olsun
bydtilter
 
Dalhousie University Diploma
by文凭办理维多利亚大学购买毕业证学位认证知乎【Q微:1954292140】
 

mallet-tutorial.pdf mallet-tutorial.pdf mallet-tutorial.pdf

  • 1.
    Machine  Learning  with  MALLET http://mallet.cs.umass.edu/mallet-­‐tutorial.pdf David  Mimno   Department  of  Information  Science,   Cornell  University License:  CC  BY  4.0
  • 2.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 3.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 4.
    Who? • Andrew  McCallum  (most  of  the   work)   • Charles  Sutton,  Aron  Culotta,   Greg  Druck,  Kedar  Bellare,   Gaurav  Chandalia…   • Fernando  Pereira,  others  at   Penn…
  • 5.
    Who  am  I? •Chief  maintainer  of  MALLET   • Primary  author  of  MALLET  topic  modeling   package
  • 6.
    Why? • Motivation:  text  classification  and   information  extraction   • Commercial  machine  learning  (Just   Research,  WhizBang)   • Analysis  and  indexing  of  academic   publications:  Cora,  Rexa
  • 7.
    What? • Text  focus:  data  is  discrete  rather  than   continuous,  even  when  values  could  be   continuous: double value = 3.0
  • 8.
    How? • Command  line  scripts:   – bin/mallet  [command]  -­‐-­‐[option]  [value]  …   – Text  User  Interface  (“tui”)  classes   • Direct  Java  API   – http://mallet.cs.umass.edu/api Most of this talk
  • 9.
    History • Version  0.4:  c2004   – Classes  in  edu.umass.cs.mallet.base.*   • Version  2.0:  c2008   – Classes  in  cc.mallet.*   – Major  changes  to  finite  state  transducer   package   – bin/mallet  vs.  specialized  scripts   – Java  1.5  generics
  • 10.
    Learning  More • http://mallet.cs.umass.edu   – “Quick  Start”  guides,  focused  on  command  line   processing   – Developers’  guides,  with  Java  examples   • mallet-­‐dev@cs.umass.edu  mailing  list   – Low  volume,  but  can  be  bursty
  • 11.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 12.
    Models  for  Text  Data • Generative  models  (Multinomials)   – Naïve  Bayes   – Hidden  Markov  Models  (HMMs)   – Latent  Dirichlet  Topic  Models   • Discriminative  Regression  Models   – MaxEnt/Logistic  regression   – Conditional  Random  Fields  (CRFs)  
  • 13.
    Representations • Transform  text   documents  to   vectors  x1,  x2,…   • Retain  meaning   of  vector  indices   • Ideally  sparsely Call me Ishmael. … Document
  • 14.
    Representations • Transform  text   documents  to   vectors  x1,  x2,…   • Retain  meaning   of  vector  indices   • Ideally  sparsely 1.0 0.0 … 0.0 6.0 0.0 … 3.0 … Call me Ishmael. … xi Document
  • 15.
    Representations • Elements  of  vector   are  called  feature   values   • Example:  Feature   at  row  345  is   number  of  times   “dog”  appears  in   document     1.0 0.0 … 0.0 6.0 0.0 … 3.0 … xi
  • 16.
    Documents  to  Vectors Callme Ishmael. Document
  • 17.
    Documents  to  Vectors Callme Ishmael. Document Call me Ishmael Tokens
  • 18.
    Documents  to  Vectors Callme Ishmael Tokens call me ishmael Tokens
  • 19.
    Documents  to  Vectors callme ishmael Tokens 473, 3591, 17 Features 17. ishmael … 473. call … 3591 me
  • 20.
    Documents  to  Vectors 171.0 473 1.0 3591 1.0 Features (bag) 17. ishmael 473. call 3591 me 473, 3591, 17 Features (sequence) 17. ishmael … 473. call … 3591 me 17. ishmael … 473. call … 3591 me
  • 21.
    Instances Email  message,  web  page,  sentence,  journal   abstract…     • Name   • Data   • Target/Label   • Source What is it called? What is the input? What is the output? What did it originally look like?
  • 22.
    Instances • Name   •Data   • Target   • Source String ! TokenSequence ArrayList<Token> FeatureSequence int[] FeatureVector int -> double map cc.mallet.types
  • 23.
    Alphabets TObjectIntHashMap map ArrayList entries ! intlookupIndex(Object o, boolean shouldAdd) ! Object lookupObject(int index) cc.mallet.types, gnu.trove 17. ishmael … 473. call … 3591 me
  • 24.
    Alphabets TObjectIntHashMap map ArrayList entries ! intlookupIndex(Object o, boolean shouldAdd) ! Object lookupObject(int index) cc.mallet.types, gnu.trove 17. ishmael … 473. call … 3591 me for
  • 25.
    Alphabets TObjectIntHashMap map ArrayList entries ! cc.mallet.types,gnu.trove 17. ishmael … 473. call … 3591 me void stopGrowth() ! void startGrowth() Do not add entries for new Objects -- default is to allow growth.
  • 26.
    Creating  Instances • Instance   constructor   method   • Iterators new Instance(data, target, name, source) Iterator<Instance> FileIterator(File[], …) CsvIterator(FileReader, Pattern…) ArrayIterator(Object[]) … cc.mallet.pipe.iterator
  • 27.
  • 28.
    Creating  Instances • CsvIterator cc.mallet.pipe.iterator Name,label, data from regular expression groups. “CSV” is a lousy name. LineRegexIterator? Each instance on its own line 1001. Melville Call me Ishmael. Some years ago… 1002. Dickens It was the best of times, it was… ^([^t]+)t([^t]+)t(.*)
  • 29.
    Instance  Pipelines • Sequential   transformations   of  instance  fields   (usually  Data)   • Pass  an   ArrayList<Pipe>   to  SerialPipes cc.mallet.pipe // “data” is a String CharSequence2TokenSequence // tokenize with regexp TokenSequenceLowercase // modify each token’s text TokenSequenceRemoveStopwords // drop some tokens TokenSequence2FeatureSequence // convert token Strings to ints FeatureSequence2FeatureVector // lose order, count duplicates
  • 30.
    Instance  Pipelines • A  small  number   of  pipes  modify   the  “target”  field   • There  are  now   two  alphabets:   data  and  label cc.mallet.pipe, cc.mallet.types // “target” is a String Target2Label // convert String to int // “target” is now a Label Alphabet > LabelAlphabet
  • 31.
    Label  objects • Weights  on  a   fixed  set  of   classes   • For  training  data,   weight  for   correct  label  is   1.0,  all  others  0.0 cc.mallet.types implements Labeling ! int getBestIndex() Label getBestLabel() You cannot create a Label, they are only produced by LabelAlphabet
  • 32.
    InstanceLists • A  List  of  Instance   objects,  along   with  a  Pipe,  data   Alphabet,  and   LabelAlphabet cc.mallet.types InstanceList instances = new InstanceList(pipe); ! instances.addThruPipe(iterator);
  • 33.
    Putting  it  all  together ArrayList<Pipe> pipeList = new ArrayList<Pipe>(); ! pipeList.add(new Target2Label()); pipeList.add(new CharSequence2TokenSequence()); pipeList.add(new TokenSequence2FeatureSequence()); pipeList.add(new FeatureSequence2FeatureVector()); ! InstanceList instances = new InstanceList(new SerialPipes(pipeList)); ! instances.addThruPipe(new FileIterator(. . .));
  • 34.
    Persistent  Storage • Most  MALLET   classes  use  Java   serialization  to   store  models  and   data java.io ObjectOutputStream oos = new ObjectOutputStream(…); oos.writeObject(instances); oos.close(); Pipes, data objects, labelings, etc all need to implement Serializable. ! Be sure to include custom classes in classpath, or you get a StreamCorruptedException
  • 35.
    Review • What  are  the  four  main  fields  in  an   Instance?
  • 36.
    Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?
  • 37.
    Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?   • How  do  we  modify  the  value  of  Instance   fields?
  • 38.
    Review • What  are  the  four  main  fields  in  an   Instance?   • What  are  two  ways  to  generate  Instances?   • How  do  we  modify  the  value  of  Instance   fields?   • Name  some  classes  that  appear  in  the   “data”  field.
  • 39.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 40.
    Classifier  objects • Classifiers  map   from  instances  to   distributions   over  a  fixed  set   of  classes   • MaxEnt,  Naïve   Bayes,  Decision   Trees… cc.mallet.classify Given data Which class is best? ! (this one!) watery NN JJ PRP VB CC
  • 41.
    Classifier  objects • Classifiers  map   from  instances  to   distributions   over  a  fixed  set   of  classes   • MaxEnt,  Naïve   Bayes,  Decision   Trees… cc.mallet.classify Labeling labeling = classifier.classify(instance); ! Label l = labeling.getBestLabel(); ! System.out.print(instance + “t”); System.out.println(l);
  • 42.
    Training  Classifier  objects cc.mallet.classify ClassifierTrainertrainer = new MaxEntTrainer(); ! Classifier classifier = trainer.train(instances); • Each  type  of   classifier  has  one   or  more   ClassifierTrainer   classes
  • 43.
    Training  Classifier  objects cc.mallet.optimize logP(Labels | Data) = log f(label1, data1, w) + log f(label2, data2, w) + log f(label3, data3, w) + … • Some  classifiers   require   numerical   optimization  of   an  objective   function. Maximize w.r.t. w!
  • 44.
    Parameters  w • Association   between   feature,  class   label   • How  many   parameters  for  K   classes  and  N   features? action     NN   0.13   action     VB   -­‐0.1   action     JJ   -­‐0.21   SUFF-­‐tion   NN   1.3   SUFF-­‐tion   VB   -­‐2.1   SUFF-­‐tion   JJ   -­‐1.7   SUFF-­‐on   NN   0.01   SUFF-­‐on   VB   -­‐0.02   …  
  • 45.
    Training  Classifier  objects cc.mallet.optimize interfaceOptimizer boolean optimize() ! interface Optimizable interface ByValue interface ByValueGradient Limited-memory BFGS, Conjugate gradient… Specific objective functions
  • 46.
    Training  Classifier  objects cc.mallet.classify MaxEntOptimizableByLabelLikelihood double[]getParameters() void setParameters(double[] parameters) … ! double getValue() void getValueGradient(double[] buffer) Log likelihood and its first derivative For Optimizable interface
  • 47.
    Evaluation  of  Classifiers •Create   random  test/ train  splits cc.mallet.types InstanceList[] instanceLists = instances.split(new Randoms(), new double[] {0.9, 0.1, 0.0}); 90% training 10% testing 0% validation
  • 48.
    Evaluation  of  Classifiers •The  Trial  class   stores  the   results  of   classifications   on  an   InstanceList   (testing  or   training) cc.mallet.classify Trial(Classifier c, InstanceList list) double getAccuracy() double getAverageRank() double getF1(int/Label/Object) double getPrecision(…) double getRecall(…)
  • 49.
    Review • I  have  invented  a  new  classifier:  David   regression.   – What  class  should  I  implement  to  classify   instances?
  • 50.
    Review • I  have  invented  a  new  classifier:  David   regression.   – What  class  should  I  implement  to  train  a  David   regression  classifier?
  • 51.
    Review • I  have  invented  a  new  classifier:  David   regression.   – I  want  to  train  using  ByValueGradient.  What   mathematical  functions  do  I  need  to  code  up,   and  what  class  should  I  put  them  in?
  • 52.
    Review • I  have  invented  a  new  classifier:  David   regression.   – How  would  I  check  whether  my  new  classifier   works  better  than  Naïve  Bayes?
  • 53.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 54.
    Sequence  Tagging • Data  occurs  in   sequences   • Categorical  labels   for  each  position   • Labels  are   correlated DET          NN          VBS                VBG the          dog        likes          running
  • 55.
    Sequence  Tagging • Data  occurs  in   sequences   • Categorical  labels   for  each  position   • Labels  are   correlated ??                ??            ??                        ?? the          dog        likes          running
  • 56.
    Sequence  Tagging • Classification:  n-­‐way   • Sequence  Tagging:  nT-­‐way NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC or          red      dogs      on          blue    trees
  • 57.
    Avoiding  Exponential  Blowup •Markov  property   • Dynamic  programming Andrei Markov
  • 58.
    Avoiding  Exponential  Blowup •Markov  property   • Dynamic  programming This one Given this one Is independent of these Andrei Markov DET  JJ  NN  VB
  • 59.
    Avoiding  Exponential  Blowup •Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC or          red      dogs      on          blue    trees Andrei Markov
  • 60.
    Avoiding  Exponential  Blowup •Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC red      dogs      on          blue    trees Andrei Markov
  • 61.
    Avoiding  Exponential  Blowup •Markov  property   • Dynamic  programming NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC NN JJ PRP VB CC                                    dogs      on          blue    trees Andrei Markov
  • 62.
    Hidden  Markov  Models  and  Conditional   Random  Fields • Hidden  Markov   Model:  fully   generative   ! • Conditional  Random   Field:  conditional P(Labels | Data) = P(Data, Labels) / P(Data) P(Labels | Data)
  • 63.
    Hidden  Markov  Models  and  Conditional   Random  Fields • Hidden  Markov  Model:   simple  (independent)   output  space   ! • Conditional  Random   Field:  arbitrarily   complicated  outputs “NSF-funded” “NSF-funded” CAPITALIZED HYPHENATED ENDS-WITH-ed ENDS-WITH-d …
  • 64.
    Hidden  Markov  Models  and  Conditional   Random  Fields FeatureSequence FeatureVectorSequence ! FeatureVector[] int[] • Hidden  Markov  Model:   simple  (independent)   output  space   ! • Conditional  Random   Field:  arbitrarily   complicated  outputs
  • 65.
    Importing  Data • SimpleTagger   format:  one   word  per  line,   with  instances   delimited  by  a   blank  line Call VB me PPN Ishmael NNP . . ! Some JJ years NNS …
  • 66.
    Importing  Data • SimpleTagger   format:  one   word  per  line,   with  instances   delimited  by  a   blank  line Call SUFF-ll VB me TWO_LETTERS PPN Ishmael BIBLICAL_NAME NNP . PUNCTUATION . ! Some CAPITALIZED JJ years TIME SUFF-s NNS …
  • 67.
    Importing  Data LineGroupIterator ! SimpleTaggerSentence2TokenSequence() //String toTokens, handles labels ! TokenSequence2FeatureVectorSequence() //Token objects to FeatureVectors cc.mallet.pipe, cc.mallet.pipe.iterator
  • 68.
    Importing  Data LineGroupIterator ! SimpleTaggerSentence2TokenSequence() //String toTokens, handles labels ! [Pipes that modify tokens] ! TokenSequence2FeatureVectorSequence() //Token objects to FeatureVectors cc.mallet.pipe, cc.mallet.pipe.iterator
  • 69.
    Importing  Data //Ishmael TokenTextCharSuffix(“C2=”, 2) //IshmaelC2=el RegexMatches(“CAP”, Pattern.compile(“p{Lu}.*”)) //Ishmael C2=el CAP LexiconMembership(“NAME”, new File(‘names’), false) //Ishmael C2=el CAP NAME cc.mallet.pipe.tsf must match entire string one name per line ignore case?
  • 70.
    Sliding  window  features a        red        dog        on        a        blue        tree
  • 71.
    Sliding  window  features a        red        dog        on        a        blue        tree
  • 72.
    Sliding  window  features a        red        dog        on        a        blue        tree red@-1
  • 73.
    Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2
  • 74.
    Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2 on@1
  • 75.
    Sliding  window  features a        red        dog        on        a        blue        tree red@-1 a@-2 on@1 a@-2_&_red@-1
  • 76.
    Importing  Data int[][] conjunctions= new int[3][]; conjunctions[0] = new int[] { -1 }; conjunctions[1] = new int[] { 1 }; conjunctions[2] = new int[] { -2, -1 }; ! OffsetConjunctions(conjunctions) ! // a@-2_&_red@-1 on@1 cc.mallet.pipe.tsf previous position next position previous two
  • 77.
    Importing  Data int[][] conjunctions= new int[3][]; conjunctions[0] = new int[] { -1 }; conjunctions[1] = new int[] { 1 }; conjunctions[2] = new int[] { -2, -1 }; ! TokenTextCharSuffix("C1=", 1) OffsetConjunctions(conjunctions) ! // a@-2_&_red@-1 a@-2_&_C1=d@-1 cc.mallet.pipe.tsf previous position next position previous two
  • 78.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden)
  • 79.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden) DET P(DET)
  • 80.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden) DET           the P(the | DET)
  • 81.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN the P(NN | DET)
  • 82.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN the          dog P(dog | NN)
  • 83.
    Finite  State  Transducers •Finite  state   machine  over   two  alphabets   (observed,   hidden) DET          NN          VBS the          dog P(VBS | NN)
  • 84.
    How  many  parameters? •Determines   efficiency  of   training   • Too  many  leads   to  overfitting Trick: Don’t allow certain transitions P(VBS | DET) = 0
  • 85.
    How  many  parameters? •Determines   efficiency  of   training   • Too  many  leads   to  overfitting DET          NN          VBS ! the          dog          runs DET          NN          VBS ! the          dog          runs DET          NN          VBS ! the          dog          runs
  • 86.
    Finite  State  Transducers abstractclass Transducer CRF HMM ! abstract class TransducerTrainer CRFTrainerByLabelLikelihood HMMTrainerByLikelihood cc.mallet.fst
  • 87.
    Finite  State  Transducers cc.mallet.fst Firstorder: one weight for every pair of labels and observations. CRF crf = new CRF(pipe, null); crf.addFullyConnectedStates(); // or crf.addStatesForLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
  • 88.
    Finite  State  Transducers cc.mallet.fst “three-quarter”order: one weight for every pair of labels and observations. crf.addStatesForThreeQuarterLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
  • 89.
    Finite  State  Transducers cc.mallet.fst Secondorder: one weight for every triplet of labels and observations. crf.addStatesForBiLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
  • 90.
    Finite  State  Transducers cc.mallet.fst “Half”order: equivalent to independent classifiers, except some transitions may be illegal. crf.addStatesForHalfLabelsConnectedAsIn(instances); DET          NN          VBS ! the          dog          runs
  • 91.
    Training  a  transducer CRFcrf = new CRF(pipe, null); crf.addStatesForLabelsConnectedAsIn(trainingInstances); CRFTrainerByLabelLikelihood trainer = new CRFTrainerByLabelLikelihood(crf); ! trainer.train(); cc.mallet.fst
  • 92.
    Evaluating  a  transducer CRFTrainerByLabelLikelihoodtrainer = new CRFTrainerByLabelLikelihood(transducer); ! TransducerEvaluator evaluator = new TokenAccuracyEvaluator(testing, "testing")); ! trainer.addEvaluator(evaluator); ! trainer.train(); cc.mallet.fst
  • 93.
    Applying  a  transducer Sequenceoutput = transducer.transduce (input); ! for (int index=0; index < input.size(); input++) { System.out.print(input.get(index) + “/”); System.out.print(output.get(index) + “ “); } cc.mallet.fst
  • 94.
    Review • How  do  you  add  new  features  to   TokenSequences?
  • 95.
    Review • How  do  you  add  new  features  to   TokenSequences?   • What  are  three  factors  that  affect  the   number  of  parameters  in  a  model?
  • 96.
    Outline • About  MALLET   • Representing  Data   • Classification   • Sequence  Tagging   • Topic  Modeling
  • 97.
  • 98.
    Topics:  “Semantic  Groups” “Sports”“Negotiation” News Article
  • 99.
    Topics:  “Semantic  Groups” “Sports”“Negotiation” News Article team player game strike deadline union
  • 100.
    Topics:  “Semantic  Groups” NewsArticle team player game strike deadline union
  • 101.
    Series  Yankees  Sox  Red  World  League  game  Boston  team  games   baseball  Mets  Game  series  won  Clemens  Braves  Yankee  teams
  • 102.
    players  League  owners  league  baseball  union  commissioner   Baseball  Association  labor  Commissioner  Football  major  teams   Selig  agreement  strike  team  bargaining
  • 103.
    Training  a  Topic  Model cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate();
  • 104.
    Evaluating  a  Topic  Model cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate(); ! MarginalProbEstimator evaluator = lda.getProbEstimator(); ! double logLikelihood = evaluator.evaluateLeftToRight(testing, 10, false, null);
  • 105.
    Inferring  topics  for  new  documents cc.mallet.topics ParallelTopicModel lda = new ParallelTopicModel(numTopics); lda.addInstances(trainingInstances); lda.estimate(); ! TopicInferencer inferencer = lda.getInferencer(); ! double[] topicProbs = inferencer.getSampledDistribution(instance, 100, 10, 10);
  • 106.
    More  than  words… •Text  collections   mix  free  text  and   structured  data David Mimno Andrew McCallum UAI 2008 …
  • 107.
    More  than  words… •Text  collections   mix  free  text  and   structured  data David Mimno Andrew McCallum UAI 2008 ! “Topic models conditioned on arbitrary features using Dirichlet-multinomial regression. …”
  • 108.
    Dirichlet-­‐multinomial  Regression   (DMR) The  corpus  specifies  a  vector  of  real-­‐valued   features  (x)  for  each  document,  of  length  F.   Each  topic  has  an  F-­‐length  vector  of   parameters.
  • 109.
    Topic  parameters  for  feature  “published  in   JMLR” 2.27 kernel,  kernels,  rational  kernels,  string  kernels,  fisher  kernel 1.74 bounds,  vc  dimension,  bound,  upper  bound,  lower  bounds 1.41 reinforcement  learning,  learning,  reinforcement 1.40 blind  source  separation,  source  separation,  separation,  channel 1.37 nearest  neighbor,  boosting,  nearest  neighbors,  adaboost -­‐1.12 agent,  agents,  multi  agent,  autonomous  agents -­‐1.21 strategies,  strategy,  adaptation,  adaptive,  driven -­‐1.23 retrieval,  information  retrieval,  query,  query  expansion -­‐1.36 web,  web  pages,  web  page,  world  wide  web,  web  sites -­‐1.44 user,  users,  user  interface,  interactive,  interface
  • 110.
    Feature  parameters  for  RL  topic 2.99 Sridhar  Mahadevan 2.88 ICML 2.56 Kenji  Doya 2.45 ECML 2.19 Machine  Learning  Journal -­‐1.38 ACL -­‐1.47 CVPR -­‐1.54 IEEE  Trans.  PAMI -­‐1.64 COLING -­‐3.76 <default>
  • 111.
    Topic  parameters  for  feature  “published  in   UAI” 2.88 bayesian  networks,  bayesian  network,  belief  networks 2.26 qualitative,  reasoning,  qualitative  reasoning,  qualitative  simulation 2.25 probability,  probabilities,  probability  distributions, 2.25 uncertainty,  symbolic,  sketch,  primal  sketch,  uncertain,  connectionist 2.11 reasoning,  logic,  default  reasoning,  nonmonotonic  reasoning -­‐1.29 shape,  deformable,  shapes,  contour,  active  contour -­‐1.36 digital  libraries,  digital  library,  digital,  library -­‐1.37 workshop  report,  invited  talk,  international  conference,  report -­‐1.50 descriptions,  description,  top,  bottom,  top  bottom -­‐1.50 nearest  neighbor,  boosting,  nearest  neighbors,  adaboost
  • 112.
    Feature  parameters  for  Bayes  nets  topic 2.88 UAI 2.41 Mary-­‐Anne  Williams 2.23 Ashraf  M.  Abdelbar 2.15 Philippe  Smets 2.04 Loopy  Belief  Propagation  for  Approximate  Inference  (Murphy,  Weiss,   and  Jordan,  UAI,  1999) -­‐1.16 Probabilistic  Semantics  for  Nonmonotonic  Reasoning  (Pearl,  KR,  1989) -­‐1.38 COLING -­‐1.50 Neural  Networks -­‐2.24 ICRA -­‐3.36 <default>
  • 113.
    Dirichlet-­‐multinomial  Regression • Arbitrary  observed  features  of  documents   • Target  contains  FeatureVector DMRTopicModel dmr = new DMRTopicModel (numTopics); ! dmr.addInstances(training); dmr.estimate(); ! dmr.writeParameters(new File("dmr.parameters"));
  • 114.
    Polylingual  Topic  Modeling •Topics  exist  in  more  languages  than  you   could  possibly  learn   • Topically  comparable  documents  are  much   easier  to  get  than  translation  sets   • Translation  dictionaries   – cover  pairs,  not  sets  of  languages   – miss  technical  vocabulary   – aren’t  available  for  low-­‐resource  languages
  • 115.
    Topics  from   European   Parliament   Proceedings
  • 116.
    Topics  from   European   Parliament   Proceedings
  • 117.
  • 118.
    Aligned  instance  lists dog…chien… hund… cat… chat… pig… schwein…
  • 119.
    Polylingual  Topics InstanceList[] training= new InstanceList[] { english, german, arabic, mahican }; ! PolylingualTopicModel pltm = new PolylingualTopicModel(numTopics); ! pltm.addInstances(training);
  • 120.