Lightning is a Python library for large-scale machine learning that incorporates recent advances in optimization algorithms. It is compatible with scikit-learn and supports both dense and sparse data as well as structured sparsity penalties. Lightning scales to large datasets using stochastic optimization methods like SGD, SVRG, SDCA, and SAGA. It also efficiently handles large feature spaces using coordinate descent algorithms. The API is similar to scikit-learn but is based on optimization algorithms rather than machine learning models. Lightning is part of the scikit-learn-contrib project.

1. LIGHTNING, A LIBRARY FOR
LARGE-SCALE MACHINE
LEARNING IN PYTHON
,Fabian Pedregosa (1) Mathieu Blondel (2)
(1) Chaire Havas-Dauphine / INRIA, Paris France
(2) NTT Communication Science Laboratories, Kyoto Japan

2. SCIKIT-LEARN: WITH GREAT CODE
COMES GREAT RESPONSABILITY
#lines of code in scikit-learn
Very selective for new algorithms/models.

3. LIGHTNING
Incorporate recent progress in large-scale optimization.
scikit-learn compatible .
scalable on large datasets.
support for dense and sparse input.
emphasis on structured sparsity penalties.
dependencies = Python + Cython + scikit-learn.

4. SCIKIT-LEARN COMPATIBLE
mix lightning with scikit-learn Pipeline, GridSearchCV,
etc.
⟹

5. FROM LARGE DATA TO LARGE
OPTIMIZATION
Big data comes in different flavors.
n{
⎛
⎝
⎜
⎜
⎜
⎜
D
A
T
A
⎞
⎠
⎟
⎟
⎟
⎟
  p
Large sample:
Computer vision, advertising,
etc.
Large dimension:
Biology, neuroscience, etc.

6. LEARNING FROM LARGE SAMPLES
Usual methods (gradient descent, BFGS, etc.):
Pass through the data at each iteration.
Prohibitive for large datasets.
Back to simple methods:
Stochastic gradient descent (Robbins and Monro, 1951).

7. LEARNING FROM LARGE SAMPLES
lighting example, n=100.000
In last 5 years, flurry of
new stochastic methods:
Stochastic variance-
reduced gradient
(SVRG)
Stochastic Dual
Coordinate Ascent
(SDCA)
Stochastic Average
Gradient (SAG/SAGA)
They are all in lightning!

8. LEARNING FROM LARGE FEATURES
Iterate through the columns.
Coordinate Descent-like algorithms.
Very efficient for sparse models.
(Blondel et al. 2013) , multiclass classification with group-lasso penalty

9. STRUCTURED SPARSITY
There's so much more than the Lasso ...
Group sparse penalty.
Total variation.
Trace norm (low rank).

10. API
Similarities and differences with scikit-learn
scikit-learn:
(penalty = 'l1', )LogisticRegression
  loss function
solver='liblinear'
  algorithm
lightning:
(penalty = 'l1', ) CDClassifier
  algorithm
loss='log'
  loss function
API based on algorithms, not models.

11. EXTENSIBILITY
Typical loss and penalties available.
Possible to pass custom loss or penalty function
clf = FistaClassifier(
loss=my_loss,
penalty=my_penalty)
(available for Fista*and SAGA*)

12. FUTURE CHALLENGES
Parallel stochastic methods
(Leblond, Pedregosa, Lacoste-Julien 2016)
Out of core (scale beyond computer memory).

13. SCIKIT-LEARN-CONTRIB
lightning is just the beginning.
Welcome projects that are:
Your browser does not support SVG
scikit-learn compatible.
Documented.
Test coverage > 80%.

14. THANKS FOR YOUR ATTENTION
http://contrib.scikit-learn.org/lightning/
(We're hiring!)