This document discusses optimization techniques for predictive analytics. It provides examples of loss functions for linear regression, clustering, and propagating labels across a social graph. It also discusses aligning loss functions with business goals, techniques for finding parameters like gradient descent and convex optimization, and tradeoffs between precision and recall to avoid overfitting.

1. Optimization
A framework for predictive analytics
Matt Gattis, Hunch.com

2. Simple Optimization Problem

3. Machine Learning Example: Linear Regression

4. Machine Learning Anecdote: Clustering
Type Loss function
Hierarchical max/min/mean pairwise differences between observations in each cluster
K-means distance from each observation in a cluster to the centroid of the cluster
Hunch 1) number of K nearest neighbors not in the cluster
2) distance from the furthest observation in the cluster

5. SYNERGY
• Align your loss function with your product goals
• Add terms to your loss function for optimizing business
metrics
Make sure your product designers, business strategists, and
engineers are all on the same page.

6. Techniques to Find Parameters
Step #1: Make your loss function convex

7. Techniques to Find Parameters
• Gradient Descent
• Newton’s method
• Stochastic gradient descent
• Conjugate gradient descent
• Alternating variables
• Simplex Algorithm
Step #2: Convex Optimization

8. Techniques to Find Parameters
• Python: cvxopt / cvxmod
• C/C++: GLPK (GNU Linear Programming Kit)
• Matlab, R, Numpy/Scipy, etc
Step #2: Convex Optimization
>> from cvxmod import *
>> p = problem(minimize(norm2(A*x-b)))
>> p.constr.append(x >= 0.5)
>> p.solve()

9. Example: Propagating Labels Across a Social Graph
?
?

10. Example: Propagating Labels Across a Social Graph
?
?
Solution: [Zhou et al. (2005)]

11. Trade-offs: Not all Errors are Created Equal
Predicted Actual
True Positives + +
False Positive + −
True Negative − −
False Negative − +
Precision is TP : FP (avoid looking dumb)
Recall is TP : FN (surface the most possibilities)

12. Overfitting
Model is trying to explaining noise
• insufficient sample size
• too many arguments
• not enough constraints / error terms