Many updates since the last meetup
v1.8.0 (April 12)
v1.9.0 (May 31)
v1.10.0 (June 28)
Many contributions from the community. Thank you very much!!!!!
(30 PRs from non-PFI/PFNers have been merged since the last meetup in
March)
3
New core features
CaffeFunction improved: support Py3, support the ResNet models
Weight initializer
– Most links start supporting initializer to initialize the parameters
– Sample:
import chainer.links as L, chainer.initializers as I
L.Linear(784, 1000, initialW=I.Normal(0.01),
initialb=I.Constant(0.1))
– Many built-in initializers
– You can also write your own initializer easily
(it is just a function/callable that initializes the elements of a given CPU array)
Support float16 and float64 in many Functions
CuPy Profiling API
4
New core feature (cont.)
ndarray arguments for Functions
– Function now accepts NumPy/CuPy ndarrays as arguments
– It automatically wraps the arrays by Variable
– Users do not have to wrap arrays by Variable manually
5
Many Functions/Links are added
Variable.__getitem__ (F.get_item)
– Variable now supports the basic indexing (cf. NumPy) with backprop
– It supports: integer indexing, slice indexing, newaxis, Ellipsis
– E.g.: slice indexing can be used to crop feature maps
Array manipulation: cast, clip, log1p, expm1, logsumexp, minimum, permutate
NN elements: huber_loss, hard_sigmoid, roi_pooling_2d, StatelessLSTM, Bias,
Scale
There are also many updates on existing Functions/Links (new options,
bug fixes, etc.)
See the recent release notes for the full list of new Functions/Links
https://github.com/pfnet/chainer/releases
6
New CuPy functions
cupy.nonzero
– Enumerates the indices of non-zero elements
– It implements inclusive scan kernel in CUDA
– We are willing to support a wider range of routines that require the scan
kernel (like cumsum)
In parallel, some routines using nonzero are added: cupy.ix_,
cupy.flatnonzero
Profiling API (cupy.cuda.profile, cupy.cuda.profiler) to enable CUDA
profile collection only in a specified range of codes
7
Issue: slow merge of PRs. Solution: more minor releases
As I noted, there are now many PRs coming to the Chainer repository
The current release cycle of minor updates per 6 weeks is too slow
We decided to make minor releases more frequently
Until v1.9.0:
Revision release (without new features) for every 2 weeks
Minor release (with new features) for every 6 weeks
From v1.10.0:
Release for every 2 weeks
Any release can contain new features (we increment the minor version
in that case)
8
Planned big features for upcoming releases (v1.11-12)
Dataset and Trainer (will be explained)
cuDNN RNN support
Theano function support (use a Theano function as a Chainer Function)
Asynchronous to_gpu
Target Link
Dataset
Optimizer
Iterator
Dataset and Trainer
Dataset: Abstraction of iterations over datasets
Trainer: Abstraction of training loops
11
Trainer
Extension
Extension
Extension
Updater Optimizer
Optimizer
Target Link
Target Link
Iterator
Iterator
Dataset
Dataset
We often use only one
optimizer and one
dataset. This diagram
shows a general case.
Trainer
Call the Updater and Extensions for every iteration
Updater
– Fetch a mini-batch using Iterator, and update parameters using Optimizer
– You can customize the update routine
– Built-in updater: StandardUpdater, ParallelUpdater (under review)
(ParallelUpdater provides an easy way of data-parallel learning)
Extension
– It adds an extra routine to the training loop
– Basic extensions are built-in:
Evaluator, LogReport, PrintReport, ProgressBar
snapshot, snapshot_object, ExponentialDecay, LinearShift
dump_graph
– You can write your own extensions
12
Dataset / Iterator
Dataset is just a sequence of data points (a.k.a. examples)
Iterator defines how to iterate over the dataset
Built-in iterators:
– SequentialIterator
– ShuffledIterator
– MultiprocessIterator
(you can easily support multiprocess preprocessing with it)
13
Reporter: easy way to report an observation
Trainer uses Reporter to collect observations
(e.g. loss value, accuracy, activation statistics, etc.)
Example (simple Classifier):
class Classifier(chainer.Chain):
def __init__(self, predictor):
super(Classifier, self).__init__(predictor=predictor)
def __call__(self, x, t):
y = self.predictor(x)
loss = F.softmax_cross_entropy(y, t)
accuracy = F.accuracy(y, t)
chainer.report({‘loss’: loss, ‘accuracy’: accuracy},
observer=self)
return loss
14
Note on Trainer
If your training workflow is very different from standard ones, you can
still write your own training loop
We recommend you to use Trainer for newly-written training scripts
– Most usages are covered by the Trainer
– You can flexibly customize each component of Trainer
16
We are planning the first major version up!
Planning to release it in this autumn (Oct. – Nov.)
It will breaks the backward compatibility
Planned features
Separate CuPy into a separate repository/package
Plugin system and service (we need discussions with you!)
(share Functions/Links/etc. easily with other users without sending PRs)
Backprop as a graph (or gradient of expressions including gradients)
Parameter shape inference (without specifying “input size”)
Make asynchronous CPU/GPU transfer by default
Parameter annotation
17