A Few Useful Things to Know about Machine Learning                                                             Pedro Domin...
Table 1: The three components of learning algorithms.                    Representation                    Evaluation     ...
4. DATA ALONE IS NOT ENOUGH                                                            Low                   HighGeneraliz...
Cross-validation can help to combat overfitting, for exampleby using it to choose the best size of decision tree to learn. ...
dcan find a reasonable frontier between examples of different        total number of functions is 22 . And even for hypothes...
it is to gather data, integrate it, clean it and pre-process it,         N. Bayesand how much trial and error can go into ...
In the end, the biggest bottleneck is not data or CPU cycles,    those produced by (say) bagging or boosting: the latter a...
12. REPRESENTABLE DOES NOT IMPLY                                  Many researchers believe that causality is only a conven...
[12] D. Kibler and P. Langley. Machine learning as an     experimental science. In Proceedings of the Third     European W...
Upcoming SlideShare
Loading in …5

A Few Useful Things to Know about Machine Learning


Published on

Machine learning algorithms can figure out how to perform
important tasks by generalizing from examples. This is often feasible and cost-effective where manual programming
is not. As more data becomes available, more ambitious
problems can be tackled. As a result, machine learning is
widely used in computer science and other fields. However,
developing successful machine learning applications requires
a substantial amount of “black art” that is hard to find in
textbooks. This article summarizes twelve key lessons that
machine learning researchers and practitioners have learned.
These include pitfalls to avoid, important issues to focus on,
and answers to common questions.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

A Few Useful Things to Know about Machine Learning

  1. 1. A Few Useful Things to Know about Machine Learning Pedro Domingos Department of Computer Science and Engineering University of Washington Seattle, WA 98195-2350, U.S.A. pedrod@cs.washington.eduABSTRACT correct output yt for future examples xt (e.g., whether theMachine learning algorithms can figure out how to perform spam filter correctly classifies previously unseen emails asimportant tasks by generalizing from examples. This is of- spam or not spam).ten feasible and cost-effective where manual programmingis not. As more data becomes available, more ambitious 2. LEARNING = REPRESENTATION +problems can be tackled. As a result, machine learning iswidely used in computer science and other fields. However, EVALUATION + OPTIMIZATION Suppose you have an application that you think machinedeveloping successful machine learning applications requires learning might be good for. The first problem facing youa substantial amount of “black art” that is hard to find in is the bewildering variety of learning algorithms available.textbooks. This article summarizes twelve key lessons that Which one to use? There are literally thousands available,machine learning researchers and practitioners have learned. and hundreds more are published each year. The key to notThese include pitfalls to avoid, important issues to focus on, getting lost in this huge space is to realize that it consistsand answers to common questions. of combinations of just three components. The components are:1. INTRODUCTIONMachine learning systems automatically learn programs from Representation. A classifier must be represented in somedata. This is often a very attractive alternative to manually formal language that the computer can handle. Con-constructing them, and in the last decade the use of machine versely, choosing a representation for a learner is tan-learning has spread rapidly throughout computer science tamount to choosing the set of classifiers that it canand beyond. Machine learning is used in Web search, spam possibly learn. This set is called the hypothesis spacefilters, recommender systems, ad placement, credit scoring, of the learner. If a classifier is not in the hypothesisfraud detection, stock trading, drug design, and many other space, it cannot be learned. A related question, whichapplications. A recent report from the McKinsey Global In- we will address in a later section, is how to representstitute asserts that machine learning (a.k.a. data mining or the input, i.e., what features to use.predictive analytics) will be the driver of the next big wave ofinnovation [15]. Several fine textbooks are available to inter- Evaluation. An evaluation function (also called objectiveested practitioners and researchers (e.g, [16, 24]). However, function or scoring function) is needed to distinguishmuch of the “folk knowledge” that is needed to successfully good classifiers from bad ones. The evaluation functiondevelop machine learning applications is not readily avail- used internally by the algorithm may differ from theable in them. As a result, many machine learning projects external one that we want the classifier to optimize, fortake much longer than necessary or wind up producing less- ease of optimization (see below) and due to the issuesthan-ideal results. Yet much of this folk knowledge is fairly discussed in the next section.easy to communicate. This is the purpose of this article. Optimization. Finally, we need a method to search among the classifiers in the language for the highest-scoringMany different types of machine learning exist, but for il- one. The choice of optimization technique is key to thelustration purposes I will focus on the most mature and efficiency of the learner, and also helps determine thewidely used one: classification. Nevertheless, the issues I classifier produced if the evaluation function has morewill discuss apply across all of machine learning. A classi- than one optimum. It is common for new learners tofier is a system that inputs (typically) a vector of discrete start out using off-the-shelf optimizers, which are laterand/or continuous feature values and outputs a single dis- replaced by custom-designed ones.crete value, the class. For example, a spam filter classifiesemail messages into “spam” or “not spam,” and its input may Table 1 shows common examples of each of these three com-be a Boolean vector x = (x1 , . . . , xj , . . . , xd ), where xj = 1 ponents. For example, k-nearest neighbor classifies a testif the jth word in the dictionary appears in the email and example by finding the k most similar training examplesxj = 0 otherwise. A learner inputs a training set of exam- and predicting the majority class among them. Hyperplane-ples (xi , yi ), where xi = (xi,1 , . . . , xi,d ) is an observed input based methods form a linear combination of the features perand yi is the corresponding output, and outputs a classifier. class and predict the class with the highest-valued combina-The test of the learner is whether this classifier produces the tion. Decision trees test one feature at each internal node,
  2. 2. Table 1: The three components of learning algorithms. Representation Evaluation Optimization Instances Accuracy/Error rate Combinatorial optimization K-nearest neighbor Precision and recall Greedy search Support vector machines Squared error Beam search Hyperplanes Likelihood Branch-and-bound Naive Bayes Posterior probability Continuous optimization Logistic regression Information gain Unconstrained Decision trees K-L divergence Gradient descent Sets of rules Cost/Utility Conjugate gradient Propositional rules Margin Quasi-Newton methods Logic programs Constrained Neural networks Linear programming Graphical models Quadratic programming Bayesian networks Conditional random fieldswith one branch for each feature value, and have class predic- Algorithm 1 LearnDT(TrainSet)tions at the leaves. Algorithm 1 shows a bare-bones decision if all examples in TrainSet have the same class y∗ thentree learner for Boolean domains, using information gain and return MakeLeaf(y∗ )greedy search [20]. InfoGain(xj ,y) is the mutual information if no feature xj has InfoGain(xj ,y) > 0 thenbetween feature xj and the class y. MakeNode(x,c0 ,c1 ) re- y∗ ← Most frequent class in TrainSetturns a node that tests feature x and has c0 as the child for return MakeLeaf(y∗ )x = 0 and c1 as the child for x = 1. x∗ ← argmaxxj InfoGain(xj ,y) TS0 ← Examples in TrainSet with x∗ = 0Of course, not all combinations of one component from each TS1 ← Examples in TrainSet with x∗ = 1column of Table 1 make equal sense. For example, dis- return MakeNode(x∗ , LearnDT(TS0 ), LearnDT(TS1 ))crete representations naturally go with combinatorial op-timization, and continuous ones with continuous optimiza-tion. Nevertheless, many learners have both discrete and Contamination of your classifier by test data can occur incontinuous components, and in fact the day may not be insidious ways, e.g., if you use test data to tune parametersfar when every single possible combination has appeared in and do a lot of tuning. (Machine learning algorithms havesome learner! lots of knobs, and success often comes from twiddling them a lot, so this is a real concern.) Of course, holding outMost textbooks are organized by representation, and it’s data reduces the amount available for training. This caneasy to overlook the fact that the other components are be mitigated by doing cross-validation: randomly dividingequally important. There is no simple recipe for choosing your training data into (say) ten subsets, holding out eacheach component, but the next sections touch on some of the one while training on the rest, testing each learned classifierkey issues. And, as we will see below, some choices in a on the examples it did not see, and averaging the results tomachine learning project may be even more important than see how well the particular parameter setting does.the choice of learner. In the early days of machine learning, the need to keep train-3. IT’S GENERALIZATION THAT COUNTS ing and test data separate was not widely appreciated. ThisThe fundamental goal of machine learning is to generalize was partly because, if the learner has a very limited repre-beyond the examples in the training set. This is because, sentation (e.g., hyperplanes), the difference between train-no matter how much data we have, it is very unlikely that ing and test error may not be large. But with very flexiblewe will see those exact examples again at test time. (No- classifiers (e.g., decision trees), or even with linear classifierstice that, if there are 100,000 words in the dictionary, the with a lot of features, strict separation is mandatory.spam filter described above has 2100,000 possible different in-puts.) Doing well on the training set is easy (just memorize Notice that generalization being the goal has an interestingthe examples). The most common mistake among machine consequence for machine learning. Unlike in most other op-learning beginners is to test on the training data and have timization problems, we don’t have access to the functionthe illusion of success. If the chosen classifier is then tested we want to optimize! We have to use training error as a sur-on new data, it is often no better than random guessing. So, rogate for test error, and this is fraught with danger. Howif you hire someone to build a classifier, be sure to keep some to deal with it is addressed in some of the next sections. Onof the data to yourself and test the classifier they give you the positive side, since the objective function is only a proxyon it. Conversely, if you’ve been hired to build a classifier, for the true goal, we may not need to fully optimize it; inset some of the data aside from the beginning, and only use fact, a local optimum returned by simple greedy search mayit to test your chosen classifier at the very end, followed by be better than the global optimum.learning your final classifier on the whole data.
  3. 3. 4. DATA ALONE IS NOT ENOUGH Low HighGeneralization being the goal has another major consequence: Variance Variancedata alone is not enough, no matter how much of it you have.Consider learning a Boolean function of (say) 100 variablesfrom a million examples. There are 2100 − 106 exampleswhose classes you don’t know. How do you figure out whatthose classes are? In the absence of further information, Highthere is just no way to do this that beats flipping a coin. This Biasobservation was first made (in somewhat different form) bythe philosopher David Hume over 200 years ago, but eventoday many mistakes in machine learning stem from failingto appreciate it. Every learner must embody some knowl-edge or assumptions beyond the data it’s given in order togeneralize beyond it. This was formalized by Wolpert inhis famous “no free lunch” theorems, according to which no Lowlearner can beat random guessing over all possible functions Biasto be learned [25].This seems like rather depressing news. How then can weever hope to learn anything? Luckily, the functions we want Figure 1: Bias and variance in dart-throwing.to learn in the real world are not drawn uniformly fromthe set of all mathematically possible functions! In fact,very general assumptions—like smoothness, similar exam- quirks in the data. This problem is called overfitting, and isples having similar classes, limited dependences, or limited the bugbear of machine learning. When your learner outputscomplexity—are often enough to do very well, and this is a a classifier that is 100% accurate on the training data butlarge part of why machine learning has been so successful. only 50% accurate on test data, when in fact it could haveLike deduction, induction (what learners do) is a knowledge output one that is 75% accurate on both, it has overfit.lever: it turns a small amount of input knowledge into alarge amount of output knowledge. Induction is a vastly Everyone in machine learning knows about overfitting, butmore powerful lever than deduction, requiring much less in- it comes in many forms that are not immediately obvious.put knowledge to produce useful results, but it still needs One way to understand overfitting is by decomposing gener-more than zero input knowledge to work. And, as with any alization error into bias and variance [9]. Bias is a learner’slever, the more we put in, the more we can get out. tendency to consistently learn the same wrong thing. Vari- ance is the tendency to learn random things irrespective ofA corollary of this is that one of the key criteria for choos- the real signal. Figure 1 illustrates this by an analogy withing a representation is which kinds of knowledge are easily throwing darts at a board. A linear learner has high bias,expressed in it. For example, if we have a lot of knowledge because when the frontier between two classes is not a hyper-about what makes examples similar in our domain, instance- plane the learner is unable to induce it. Decision trees don’tbased methods may be a good choice. If we have knowl- have this problem because they can represent any Booleanedge about probabilistic dependencies, graphical models are function, but on the other hand they can suffer from higha good fit. And if we have knowledge about what kinds of variance: decision trees learned on different training setspreconditions are required by each class, “IF . . . THEN . . .” generated by the same phenomenon are often very different,rules may be the the best option. The most useful learners when in fact they should be the same. Similar reasoningin this regard are those that don’t just have assumptions applies to the choice of optimization method: beam searchhard-wired into them, but allow us to state them explicitly, has lower bias than greedy search, but higher variance, be-vary them widely, and incorporate them automatically into cause it tries more hypotheses. Thus, contrary to intuition,the learning (e.g., using first-order logic [21] or grammars a more powerful learner is not necessarily better than a less[6]). powerful one.In retrospect, the need for knowledge in learning should not Figure 2 illustrates this.1 Even though the true classifierbe surprising. Machine learning is not magic; it can’t get is a set of rules, with up to 1000 examples naive Bayes issomething from nothing. What it does is get more from more accurate than a rule learner. This happens despiteless. Programming, like all engineering, is a lot of work: naive Bayes’s false assumption that the frontier is linear!we have to build everything from scratch. Learning is more Situations like this are common in machine learning: stronglike farming, which lets nature do most of the work. Farm- false assumptions can be better than weak true ones, becauseers combine seeds with nutrients to grow crops. Learners a learner with the latter needs more data to avoid overfitting.combine knowledge with data to grow programs. 15. OVERFITTING HAS MANY FACES Training examples consist of 64 Boolean features and aWhat if the knowledge and data we have are not sufficient Boolean class computed from them according to a set of “IF . . . THEN . . .” rules. The curves are the average of 100 runsto completely determine the correct classifier? Then we run with different randomly generated sets of rules. Error barsthe risk of just hallucinating a classifier (or parts of it) that are two standard deviations. See Domingos and Pazzani [10]is not grounded in reality, and is simply encoding random for details.
  4. 4. Cross-validation can help to combat overfitting, for exampleby using it to choose the best size of decision tree to learn. 80 BayesBut it’s no panacea, since if we use it to make too many C4.5 Test-Set Accuracy (%)parameter choices it can itself start to overfit [17]. 75Besides cross-validation, there are many methods to combat 70overfitting. The most popular one is adding a regulariza-tion term to the evaluation function. This can, for exam- 65ple, penalize classifiers with more structure, thereby favoringsmaller ones with less room to overfit. Another option is to 60perform a statistical significance test like chi-square beforeadding new structure, to decide whether the distribution of 55the class really is different with and without this structure.These techniques are particularly useful when data is very 50scarce. Nevertheless, you should be skeptical of claims that 10 100 1000 10000a particular technique “solves” the overfitting problem. It’s Number of Exampleseasy to avoid overfitting (variance) by falling into the op-posite error of underfitting (bias). Simultaneously avoiding Figure 2: Naive Bayes can outperform a state-of-both requires learning a perfect classifier, and short of know- the-art rule learner (C4.5rules) even when the trueing it in advance there is no single technique that will always classifier is a set of rules.do best (no free lunch).A common misconception about overfitting is that it is caused trillion examples, the latter covers only a fraction of aboutby noise, like training examples labeled with the wrong class. 10−18 of the input space. This is what makes machine learn-This can indeed aggravate overfitting, by making the learner ing both necessary and hard.draw a capricious frontier to keep those examples on whatit thinks is the right side. But severe overfitting can occur More seriously, the similarity-based reasoning that machineeven in the absence of noise. For instance, suppose we learn a learning algorithms depend on (explicitly or implicitly) breaksBoolean classifier that is just the disjunction of the examples down in high dimensions. Consider a nearest neighbor clas-labeled “true” in the training set. (In other words, the classi- sifier with Hamming distance as the similarity measure, andfier is a Boolean formula in disjunctive normal form, where suppose the class is just x1 ∧ x2 . If there are no other fea-each term is the conjunction of the feature values of one tures, this is an easy problem. But if there are 98 irrele-specific training example). This classifier gets all the train- vant features x3 , . . . , x100 , the noise from them completelying examples right and every positive test example wrong, swamps the signal in x1 and x2 , and nearest neighbor effec-regardless of whether the training data is noisy or not. tively makes random predictions.The problem of multiple testing [13] is closely related to over- Even more disturbing is that nearest neighbor still has afitting. Standard statistical tests assume that only one hy- problem even if all 100 features are relevant! This is becausepothesis is being tested, but modern learners can easily test in high dimensions all examples look alike. Suppose, formillions before they are done. As a result what looks signif- instance, that examples are laid out on a regular grid, andicant may in fact not be. For example, a mutual fund that consider a test example xt . If the grid is d-dimensional, xt ’sbeats the market ten years in a row looks very impressive, 2d nearest examples are all at the same distance from it.until you realize that, if there are 1000 funds and each has So as the dimensionality increases, more and more examplesa 50% chance of beating the market on any given year, it’s become nearest neighbors of xt , until the choice of nearestquite likely that one will succeed all ten times just by luck. neighbor (and therefore of class) is effectively random.This problem can be combatted by correcting the signifi-cance tests to take the number of hypotheses into account, This is only one instance of a more general problem withbut this can lead to underfitting. A better approach is to high dimensions: our intuitions, which come from a three-control the fraction of falsely accepted non-null hypotheses, dimensional world, often do not apply in high-dimensionalknown as the false discovery rate [3]. ones. In high dimensions, most of the mass of a multivari- ate Gaussian distribution is not near the mean, but in an increasingly distant “shell” around it; and most of the vol-6. INTUITION FAILS IN HIGH ume of a high-dimensional orange is in the skin, not the pulp. DIMENSIONS If a constant number of examples is distributed uniformly inAfter overfitting, the biggest problem in machine learning a high-dimensional hypercube, beyond some dimensionalityis the curse of dimensionality. This expression was coined most examples are closer to a face of the hypercube thanby Bellman in 1961 to refer to the fact that many algo- to their nearest neighbor. And if we approximate a hyper-rithms that work fine in low dimensions become intractable sphere by inscribing it in a hypercube, in high dimensionswhen the input is high-dimensional. But in machine learn- almost all the volume of the hypercube is outside the hyper-ing it refers to much more. Generalizing correctly becomes sphere. This is bad news for machine learning, where shapesexponentially harder as the dimensionality (number of fea- of one type are often approximated by shapes of another.tures) of the examples grows, because a fixed-size trainingset covers a dwindling fraction of the input space. Even with Building a classifier in two or three dimensions is easy; wea moderate dimension of 100 and a huge training set of a
  5. 5. dcan find a reasonable frontier between examples of different total number of functions is 22 . And even for hypothesisclasses just by visual inspection. (It’s even been said that if spaces that are “merely” exponential, the bound is still verypeople could see in high dimensions machine learning would loose, because the union bound is very pessimistic. For ex-not be necessary.) But in high dimensions it’s hard to under- ample, if there are 100 Boolean features and the hypothesisstand what is happening. This in turn makes it difficult to space is decision trees with up to 10 levels, to guaranteedesign a good classifier. Naively, one might think that gath- δ = ǫ = 1% in the bound above we need half a million ex-ering more features never hurts, since at worst they provide amples. But in practice a small fraction of this suffices forno new information about the class. But in fact their bene- accurate learning.fits may be outweighed by the curse of dimensionality. Further, we have to be careful about what a bound like thisFortunately, there is an effect that partly counteracts the means. For instance, it does not say that, if your learnercurse, which might be called the “blessing of non-uniformity.” returned a hypothesis consistent with a particular trainingIn most applications examples are not spread uniformly thr- set, then this hypothesis probably generalizes well. Whatoughout the instance space, but are concentrated on or near it says is that, given a large enough training set, with higha lower-dimensional manifold. For example, k-nearest neigh- probability your learner will either return a hypothesis thatbor works quite well for handwritten digit recognition even generalizes well or be unable to find a consistent hypothesis.though images of digits have one dimension per pixel, be- The bound also says nothing about how to select a good hy-cause the space of digit images is much smaller than the pothesis space. It only tells us that, if the hypothesis spacespace of all possible images. Learners can implicitly take contains the true classifier, then the probability that theadvantage of this lower effective dimension, or algorithms learner outputs a bad classifier decreases with training setfor explicitly reducing the dimensionality can be used (e.g., size. If we shrink the hypothesis space, the bound improves,[22]). but the chances that it contains the true classifier shrink also. (There are bounds for the case where the true classi- fier is not in the hypothesis space, but similar considerations7. THEORETICAL GUARANTEES ARE apply to them.) NOT WHAT THEY SEEM Another common type of theoretical guarantee is asymp-Machine learning papers are full of theoretical guarantees. totic: given infinite data, the learner is guaranteed to outputThe most common type is a bound on the number of ex- the correct classifier. This is reassuring, but it would be rashamples needed to ensure good generalization. What should to choose one learner over another because of its asymptoticyou make of these guarantees? First of all, it’s remarkable guarantees. In practice, we are seldom in the asymptoticthat they are even possible. Induction is traditionally con- regime (also known as “asymptopia”). And, because of thetrasted with deduction: in deduction you can guarantee that bias-variance tradeoff we discussed above, if learner A is bet-the conclusions are correct; in induction all bets are off. Or ter than learner B given infinite data, B is often better thansuch was the conventional wisdom for many centuries. One A given finite data.of the major developments of recent decades has been therealization that in fact we can have guarantees on the re- The main role of theoretical guarantees in machine learningsults of induction, particularly if we’re willing to settle for is not as a criterion for practical decisions, but as a source ofprobabilistic guarantees. understanding and driving force for algorithm design. In this capacity, they are quite useful; indeed, the close interplayThe basic argument is remarkably simple [5]. Let’s say a of theory and practice is one of the main reasons machineclassifier is bad if its true error rate is greater than ǫ. Then learning has made so much progress over the years. Butthe probability that a bad classifier is consistent with n ran- caveat emptor: learning is a complex phenomenon, and justdom, independent training examples is less than (1 − ǫ)n . because a learner has a theoretical justification and works inLet b be the number of bad classifiers in the learner’s hy- practice doesn’t mean the former is the reason for the latter.pothesis space H. The probability that at least one of themis consistent is less than b(1 − ǫ)n , by the union bound.Assuming the learner always returns a consistent classifier, 8. FEATURE ENGINEERING IS THE KEYthe probability that this classifier is bad is then less than At the end of the day, some machine learning projects suc-|H|(1 − ǫ)n , where we have used the fact that b ≤ |H|. So ceed and some fail. What makes the difference? Easilyif we want this probability to be less than δ, it suffices to the most important factor is the features used. If you havemake n > ln(δ/|H|)/ ln(1 − ǫ) ≥ 1 ln |H| + ln 1 . ` ´ ǫ δ many independent features that each correlate well with the class, learning is easy. On the other hand, if the class isUnfortunately, guarantees of this type have to be taken with a very complex function of the features, you may not bea large grain of salt. This is because the bounds obtained in able to learn it. Often, the raw data is not in a form that isthis way are usually extremely loose. The wonderful feature amenable to learning, but you can construct features from itof the bound above is that the required number of examples that are. This is typically where most of the effort in a ma-only grows logarithmically with |H| and 1/δ. Unfortunately, chine learning project goes. It is often also one of the mostmost interesting hypothesis spaces are doubly exponential in interesting parts, where intuition, creativity and “black art”the number of features d, which still leaves us needing a num- are as important as the technical stuff.ber of examples exponential in d. For example, consider thespace of Boolean functions of d Boolean variables. If there First-timers are often surprised by how little time in a ma-are e possible different examples, there are 2e possible dif- chine learning project is spent actually doing machine learn-ferent functions, so since there are 2d possible examples, the ing. But it makes sense if you consider how time-consuming
  6. 6. it is to gather data, integrate it, clean it and pre-process it, N. Bayesand how much trial and error can go into feature design.Also, machine learning is not a one-shot process of build- kNNing a data set and running a learner, but rather an iterative SVMprocess of running the learner, analyzing the results, modi-fying the data and/or the learner, and repeating. Learningis often the quickest part of this, but that’s because we’vealready mastered it pretty well! Feature engineering is moredifficult because it’s domain-specific, while learners can belargely general-purpose. However, there is no sharp frontierbetween the two, and this is another reason the most usefullearners are those that facilitate incorporating knowledge. D. TreeOf course, one of the holy grails of machine learning is toautomate more and more of the feature engineering process.One way this is often done today is by automatically gen- Figure 3: Very different frontiers can yield similarerating large numbers of candidate features and selecting class predictions. (+ and − are training examples ofthe best by (say) their information gain with respect to the two classes.)class. But bear in mind that features that look irrelevantin isolation may be relevant in combination. For example, and neural networks. But in fact propositional rules areif the class is an XOR of k input features, each of them by readily encoded as neural networks, and similar relation-itself carries no information about the class. (If you want ships hold between other representations. All learners es-to annoy machine learners, bring up XOR.) On the other sentially work by grouping nearby examples into the samehand, running a learner with a very large number of fea- class; the key difference is in the meaning of “nearby.” Withtures to find out which ones are useful in combination may non-uniformly distributed data, learners can produce widelybe too time-consuming, or cause overfitting. So there is ul- different frontiers while still making the same predictions intimately no replacement for the smarts you put into feature the regions that matter (those with a substantial number ofengineering. training examples, and therefore also where most test ex- amples are likely to appear). This also helps explain why9. MORE DATA BEATS A CLEVERER powerful learners can be unstable but still accurate. Fig- ALGORITHM ure 3 illustrates this in 2-D; the effect is much stronger inSuppose you’ve constructed the best set of features you high dimensions.can, but the classifiers you’re getting are still not accurateenough. What can you do now? There are two main choices: As a rule, it pays to try the simplest learners first (e.g., naivedesign a better learning algorithm, or gather more data Bayes before logistic regression, k-nearest neighbor before(more examples, and possibly more raw features, subject to support vector machines). More sophisticated learners arethe curse of dimensionality). Machine learning researchers seductive, but they are usually harder to use, because theyare mainly concerned with the former, but pragmatically have more knobs you need to turn to get good results, andthe quickest path to success is often to just get more data. because their internals are more opaque.As a rule of thumb, a dumb algorithm with lots and lots ofdata beats a clever one with modest amounts of it. (After Learners can be divided into two major types: those whoseall, machine learning is all about letting data do the heavy representation has a fixed size, like linear classifiers, andlifting.) those whose representation can grow with the data, like deci- sion trees. (The latter are sometimes called non-parametricThis does bring up another problem, however: scalability. learners, but this is somewhat unfortunate, since they usu-In most of computer science, the two main limited resources ally wind up learning many more parameters than paramet-are time and memory. In machine learning, there is a third ric ones.) Fixed-size learners can only take advantage of soone: training data. Which one is the bottleneck has changed much data. (Notice how the accuracy of naive Bayes asymp-from decade to decade. In the 1980’s it tended to be data. totes at around 70% in Figure 2.) Variable-size learners canToday it is often time. Enormous mountains of data are in principle learn any function given sufficient data, but inavailable, but there is not enough time to process it, so it practice they may not, because of limitations of the algo-goes unused. This leads to a paradox: even though in prin- rithm (e.g., greedy search falls into local optima) or compu-ciple more data means that more complex classifiers can be tational cost. Also, because of the curse of dimensionality,learned, in practice simpler classifiers wind up being used, no existing amount of data may be enough. For these rea-because complex ones take too long to learn. Part of the sons, clever algorithms—those that make the most of theanswer is to come up with fast ways to learn complex classi- data and computing resources available—often pay off infiers, and indeed there has been remarkable progress in this the end, provided you’re willing to put in the effort. Theredirection (e.g., [11]). is no sharp frontier between designing learners and learn- ing classifiers; rather, any given piece of knowledge could bePart of the reason using cleverer algorithms has a smaller encoded in the learner or learned from data. So machinepayoff than you might expect is that, to a first approxima- learning projects often wind up having a significant compo-tion, they all do the same. This is surprising when you nent of learner design, and practitioners need to have someconsider representations as different as, say, sets of rules expertise in it [12].
  7. 7. In the end, the biggest bottleneck is not data or CPU cycles, those produced by (say) bagging or boosting: the latter arebut human cycles. In research papers, learners are typically fairly even, while the former are extremely skewed, to thecompared on measures of accuracy and computational cost. point where the single highest-weight classifier usually dom-But human effort saved and insight gained, although harder inates, making BMA effectively equivalent to just selectingto measure, are often more important. This favors learn- it [8]. A practical consequence of this is that, while modelers that produce human-understandable output (e.g., rule ensembles are a key part of the machine learning toolkit,sets). And the organizations that make the most of ma- BMA is seldom worth the trouble.chine learning are those that have in place an infrastructurethat makes experimenting with many different learners, datasources and learning problems easy and efficient, and wherethere is a close collaboration between machine learning ex-perts and application domain ones. 11. SIMPLICITY DOES NOT IMPLY ACCURACY10. LEARN MANY MODELS, NOT JUST Occam’s razor famously states that entities should not be ONE multiplied beyond necessity. In machine learning, this is of-In the early days of machine learning, everyone had their fa- ten taken to mean that, given two classifiers with the samevorite learner, together with some a priori reasons to believe training error, the simpler of the two will likely have thein its superiority. Most effort went into trying many varia- lowest test error. Purported proofs of this claim appear reg-tions of it and selecting the best one. Then systematic em- ularly in the literature, but in fact there are many counter-pirical comparisons showed that the best learner varies from examples to it, and the “no free lunch” theorems imply itapplication to application, and systems containing many dif- cannot be true.ferent learners started to appear. Effort now went into try-ing many variations of many learners, and still selecting just We saw one counter-example in the previous section: modelthe best one. But then researchers noticed that, if instead ensembles. The generalization error of a boosted ensem-of selecting the best variation found, we combine many vari- ble continues to improve by adding classifiers even after theations, the results are better—often much better—and at training error has reached zero. Another counter-example islittle extra effort for the user. support vector machines, which can effectively have an infi- nite number of parameters without overfitting. Conversely,Creating such model ensembles is now standard [1]. In the the function sign(sin(ax)) can discriminate an arbitrarilysimplest technique, called bagging, we simply generate ran- large, arbitrarily labeled set of points on the x axis, evendom variations of the training set by resampling, learn a though it has only one parameter [23]. Thus, contrary to in-classifier on each, and combine the results by voting. This tuition, there is no necessary connection between the numberworks because it greatly reduces variance while only slightly of parameters of a model and its tendency to overfit.increasing bias. In boosting, training examples have weights,and these are varied so that each new classifier focuses on A more sophisticated view instead equates complexity withthe examples the previous ones tended to get wrong. In the size of the hypothesis space, on the basis that smallerstacking, the outputs of individual classifiers become the in- spaces allow hypotheses to be represented by shorter codes.puts of a “higher-level” learner that figures out how best to Bounds like the one in the section on theoretical guaranteescombine them. above might then be viewed as implying that shorter hy- potheses generalize better. This can be further refined byMany other techniques exist, and the trend is toward larger assigning shorter codes to the hypothesis in the space thatand larger ensembles. In the Netflix prize, teams from all we have some a priori preference for. But viewing this asover the world competed to build the best video recom- “proof” of a tradeoff between accuracy and simplicity is cir-mender system (http://netflixprize.com). As the competi- cular reasoning: we made the hypotheses we prefer simplertion progressed, teams found that they obtained the best by design, and if they are accurate it’s because our prefer-results by combining their learners with other teams’, and ences are accurate, not because the hypotheses are “simple”merged into larger and larger teams. The winner and runner- in the representation we chose.up were both stacked ensembles of over 100 learners, andcombining the two ensembles further improved the results. A further complication arises from the fact that few learnersDoubtless we will see even larger ones in the future. search their hypothesis space exhaustively. A learner with a larger hypothesis space that tries fewer hypotheses from itModel ensembles should not be confused with Bayesian model is less likely to overfit than one that tries more hypothesesaveraging (BMA). BMA is the theoretically optimal approach from a smaller space. As Pearl [18] points out, the size ofto learning [4]. In BMA, predictions on new examples are the hypothesis space is only a rough guide to what reallymade by averaging the individual predictions of all classifiers matters for relating training and test error: the procedurein the hypothesis space, weighted by how well the classifiers by which a hypothesis is chosen.explain the training data and how much we believe in thema priori. Despite their superficial similarities, ensembles and Domingos [7] surveys the main arguments and evidence onBMA are very different. Ensembles change the hypothesis the issue of Occam’s razor in machine learning. The conclu-space (e.g., from single decision trees to linear combinations sion is that simpler hypotheses should be preferred becauseof them), and can take a wide variety of forms. BMA assigns simplicity is a virtue in its own right, not because of a hy-weights to the hypotheses in the original space according to pothetical connection with accuracy. This is probably whata fixed formula. BMA weights are extremely different from Occam meant in the first place.
  8. 8. 12. REPRESENTABLE DOES NOT IMPLY Many researchers believe that causality is only a convenient LEARNABLE fiction. For example, there is no notion of causality in phys-Essentially all representations used in variable-size learners ical laws. Whether or not causality really exists is a deephave associated theorems of the form “Every function can philosophical question with no definitive answer in sight,be represented, or approximated arbitrarily closely, using but the practical points for machine learners are two. First,this representation.” Reassured by this, fans of the repre- whether or not we call them “causal,” we would like to pre-sentation often proceed to ignore all others. However, just dict the effects of our actions, not just correlations betweenbecause a function can be represented does not mean it can observable variables. Second, if you can obtain experimentalbe learned. For example, standard decision tree learners data (for example by randomly assigning visitors to differentcannot learn trees with more leaves than there are training versions of a Web site), then by all means do so [14].examples. In continuous spaces, representing even simplefunctions using a fixed set of primitives often requires an 14. CONCLUSIONinfinite number of components. Further, if the hypothesis Like any discipline, machine learning has a lot of “folk wis-space has many local optima of the evaluation function, as dom” that can be hard to come by, but is crucial for suc-is often the case, the learner may not find the true function cess. This article summarized some of the most salient items.even if it is representable. Given finite data, time and mem- Of course, it’s only a complement to the more conventionalory, standard learners can learn only a tiny subset of all pos- study of machine learning. Check out http://www.cs.washin-sible functions, and these subsets are different for learners gton.edu/homes/pedrod/class for a complete online machinewith different representations. Therefore the key question is learning course that combines formal and informal aspects.not “Can it be represented?”, to which the answer is often There’s also a treasure trove of machine learning lectures attrivial, but “Can it be learned?” And it pays to try different http://www.videolectures.net. A good open source machinelearners (and possibly combine them). learning toolkit is Weka [24]. Happy learning!Some representations are exponentially more compact than 15. REFERENCESothers for some functions. As a result, they may also re- [1] E. Bauer and R. Kohavi. An empirical comparison ofquire exponentially less data to learn those functions. Many voting classification algorithms: Bagging, boostinglearners work by forming linear combinations of simple ba- and variants. Machine Learning, 36:105–142, 1999.sis functions. For example, support vector machines form [2] Y. Bengio. Learning deep architectures for AI.combinations of kernels centered at some of the training ex- Foundations and Trends in Machine Learning,amples (the support vectors). Representing parity of n bits 2:1–127, 2009.in this way requires 2n basis functions. But using a repre- [3] Y. Benjamini and Y. Hochberg. Controlling the falsesentation with more layers (i.e., more steps between input discovery rate: A practical and powerful approach toand output), parity can be encoded in a linear-size classifier. multiple testing. Journal of the Royal StatisticalFinding methods to learn these deeper representations is one Society, Series B, 57:289–300, 1995.of the major research frontiers in machine learning [2]. [4] J. M. Bernardo and A. F. M. Smith. Bayesian Theory. Wiley, New York, NY, 1994. [5] A. Blumer, A. Ehrenfeucht, D. Haussler, and M. K.13. CORRELATION DOES NOT IMPLY Warmuth. Occam’s razor. Information Processing Letters, 24:377–380, 1987. CAUSATION [6] W. W. Cohen. Grammatically biased learning:The point that correlation does not imply causation is made Learning logic programs using an explicit antecedentso often that it is perhaps not worth belaboring. But, even description language. Artificial Intelligence,though learners of the kind we have been discussing can only 68:303–366, 1994.learn correlations, their results are often treated as repre-senting causal relations. Isn’t this wrong? If so, then why [7] P. Domingos. The role of Occam’s razor in knowledgedo people do it? discovery. Data Mining and Knowledge Discovery, 3:409–425, 1999.More often than not, the goal of learning predictive mod- [8] P. Domingos. Bayesian averaging of classifiers and theels is to use them as guides to action. If we find that beer overfitting problem. In Proceedings of the Seventeenthand diapers are often bought together at the supermarket, International Conference on Machine Learning, pagesthen perhaps putting beer next to the diaper section will 223–230, Stanford, CA, 2000. Morgan Kaufmann.increase sales. (This is a famous example in the world of [9] P. Domingos. A unified bias-variance decompositiondata mining.) But short of actually doing the experiment and its applications. In Proceedings of the Seventeenthit’s difficult to tell. Machine learning is usually applied to International Conference on Machine Learning, pagesobservational data, where the predictive variables are not 231–238, Stanford, CA, 2000. Morgan Kaufmann.under the control of the learner, as opposed to experimental [10] P. Domingos and M. Pazzani. On the optimality of thedata, where they are. Some learning algorithms can poten- simple Bayesian classifier under zero-one loss. Machinetially extract causal information from observational data, Learning, 29:103–130, 1997.but their applicability is rather restricted [19]. On the other [11] G. Hulten and P. Domingos. Mining complex modelshand, correlation is a sign of a potential causal connection, from arbitrarily large databases in constant time. Inand we can use it as a guide to further investigation (for Proceedings of the Eighth ACM SIGKDD Internationalexample, trying to understand what the causal chain might Conference on Knowledge Discovery and Data Mining,be). pages 525–531, Edmonton, Canada, 2002. ACM Press.
  9. 9. [12] D. Kibler and P. Langley. Machine learning as an experimental science. In Proceedings of the Third European Working Session on Learning, London, UK, 1988. Pitman.[13] A. J. Klockars and G. Sax. Multiple Comparisons. Sage, Beverly Hills, CA, 1986.[14] R. Kohavi, R. Longbotham, D. Sommerfield, and R. Henne. Controlled experiments on the Web: Survey and practical guide. Data Mining and Knowledge Discovery, 18:140–181, 2009.[15] J. Manyika, M. Chui, B. Brown, J. Bughin, R. Dobbs, C. Roxburgh, and A. Byers. Big data: The next frontier for innovation, competition, and productivity. Technical report, McKinsey Global Institute, 2011.[16] T. M. Mitchell. Machine Learning. McGraw-Hill, New York, NY, 1997.[17] A. Y. Ng. Preventing “overfitting” of cross-validation data. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 245–253, Nashville, TN, 1997. Morgan Kaufmann.[18] J. Pearl. On the connection between the complexity and credibility of inferred models. International Journal of General Systems, 4:255–264, 1978.[19] J. Pearl. Causality: Models, Reasoning, and Inference. Cambridge University Press, Cambridge, UK, 2000.[20] J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, CA, 1993.[21] M. Richardson and P. Domingos. Markov logic networks. Machine Learning, 62:107–136, 2006.[22] J. Tenenbaum, V. Silva, and J. Langford. A global geometric framework for nonlinear dimensionality reduction. Science, 290:2319–2323, 2000.[23] V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, New York, NY, 1995.[24] I. Witten, E. Frank, and M. Hall. Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, San Mateo, CA, 3rd edition, 2011.[25] D. Wolpert. The lack of a priori distinctions between learning algorithms. Neural Computation, 8:1341–1390, 1996.