Sony's deep learning software
"Neural Network Libraries/Console“
and its use cases in Sony
Yoshiyuki Kobayashi – Senior Machine Learning Researcher
Sony Network Communications Inc. / Sony Corporation
GTC 2018

2
Neural Network Libraries Neural Network Console
nnabla.org dl.sony.com
Windows version is now available for free.Open source (Apache 2.0 license)
Deep learning framework with Python API GUI based deep learning IDE
Sony's deep learning software

3
Sony real estate AR Effect Xperia Ear aiboXperia Hello
• Used since 2011 and more than 1,000 users in Sony group.
• Just released in 2017
• Already utilized in many products and services.
2011～
1st gen
framework
2013～
2nd gen
framework
2016～ 3rd gen framework
Neural Network Libraries
2015～ GUI Tool
Neural Network Console
Open Sourced
at Jul. 2017
Released Windows
Version at Aug. 2017
for Free

4
Motivation
Deep learning engineer is
lacking worldwide
Deep Learning is great and demand is explosively growing
• Need to make deep learning research & development more efficient
• Also, human resource development needs to be accelerated
SOLD

5
Introduction of Neural Network Console
https://www.youtube.com/watch?v=-lXjnaUSEtM

6
Application Example #1：Sony Real Estate
Real Estate Price Estimate
Neural Network Libraries is used in Real Estate Price Estimate Engine of Sony Real Estate Corporation. the
Library realizes the solution that statistically estimates signed price in buying and selling real estate, analyzing
massive data with unique algorism developed based on evaluation know-how and knowledge of Sony Real
Estate Corporation. The solution is utilized in various businesses of Sony Real Estate Corporation such
as ”Ouchi Direct”, “Real Estate Property Search Map” and “Automatic Evaluation”.
…
Input
Bunch of features about a real estate
Total floor area
Floor plan
Street address
…
Output Price of a real estate

7
Application Example #2：Xperia Ear
Gesture Sensitivity
The Library is used in an intuitive gesture sensitivity function of Sony Mobile Communications “Xperia Ear”.
Based on data from several sensors embedded in Xperia Ear, you can just use a nod of the head to confirm a
command - answering Yes/No, answering/declining the phone call, cancelling text-to-speech reading of
notifications, skipping/rewinding a song track.
…
Input
Sensor data
Output Head gestures such as Yes and No

8
Application Example #3：aibo
Image Recognition (User Identification, Face Tracking, etc.)
The Library is used to realize image recognition of Sony’s Entertainment Robot "aibo" 『ERS-1000』. In the
image recognition through fish-eye cameras installed at the nose, the Library is actively utilized for user
identification, face tracking, charge stand recognition, generic object recognition, etc. These features and
various inbuilt sensors enable its adaptable behavior.
…
入力
Image
Output
Face, object,
Charge stand,
etc…

9
Application of DL spreading depending on input / output
…Input Output
Function Input Output
Image recognition Image Category of image
Image filtering Source image target image
Speech recognition Speech String
Machine translation (En->Ja) English word string Japanese word string
Chat-bot Input word string String of response
Sensor error detection Sensor signal Abnormality level
Robot control Robot's sensor Robot actuator
…
The possibilities of application are infinite
Function

10
Feature 1. Ideal for deep learning beginners
 Easy to setup
• Just unzip the downloaded file and run the application
• Select “GPU” in Setup window to use NVIDIA GPU
 You can learn deep learning visually
 Trial and error in a short time
Provides shortcut for skill improvement on deep learning
Logistic regression
Multi layer perceptron
Convolutional
Neural Networks

11
Feature 2. Realize very efficient development
 Easy debugging
• Instantly locate the error on GUI
 Easy to deploy
• Created model can be deployed immediately by using Neural Network Libraries
• NNL offers both Python and C++ API
 Automatic structure search
• search for better neural network
structure automatically

12
More features
 Functions
 Management of trial and error history
 Visualization of weights
 Export python code
 Supports …
 Various data as well as images, matrices and vectors
 Various objectives such as classification, detection, signal
processing, regression, etc.
 Multiple inputs and outputs for multi-modal application
 Training of huge neural networks like ResNet-152
 Recurrent neural networks (RNN)
 Generative Adversarial Networks (GAN)
 Semi-supervised learning
 Transfer learning
etc…

13
Neural Network Libraries
 Lightweight C++ core library with Python API
• Flexible for arbitrary NNs
• Cross-platform & Cross-device
• Easy to add a new NN operator
• Portable to C++
 Sophisticated
• Easy-to-write & easy-to-read
• Fast training and inference with CUDA/cuDNN
• Supports both static and dynamic NNs
• Supports distributed training
• Supports fp16 and Tensor Cores
• Resources
• NNabla-Examles
https://github.com/sony/nnabla-examples/
ResNet, GANs, CapsNet, Quantized Nets, …
• Jupyter Notebook Tutorials
https://github.com/sony/nnabla/tree/master/tutorial
https://github/sony/nnabla/
pip install nnabla
import nnabla as nn
import nnabla.functions as F
import nnabla.parametric_functions as PF
x = nnabla.Variable((batch_size, 1, 28, 28))
c1 = PF.convolution(x, 16, (5, 5), name='c1')
c1 = F.relu(F.max_pooling(c1, (2, 2)))
c2 = PF.convolution(c1, 16, (5, 5), name='c2')
c2 = F.relu(F.max_pooling(c2, (2, 2)))
f3 = F.relu(PF.affine(c2, 50, name='f3'))
y = PF.affine(f3, 50, name='f4')
t = Variable((batch_size, 1))
loss = F.mean(F.softmax_cross_entropy(y, t))
…
Suitable for both research and production use

14
Neural Network Libraries Neural Network Console
Sony is providing an integrated development environment with hopes
that a wider range of developers and researchers will build on its programs,
and with the aim of contributing to the development of society.
nnabla.org dl.sony.com
Windows version is now available for free.Open source (Apache 2.0 license)
Deep learning framework with Python API GUI based deep learning IDE
Sony's deep learning software

16
Appendix

17
Tutorial
Image classification (hand-writing digits recognition)
…
…
…
…
…
…
…
Classifier
（Neural Network）
Input : image Output : Classification result
Take a typical application “Hand-writing digits recognition” as an example
2
Number of input neuron is
size of input data
28x28 pixel
784 Number of output neural is number of
categories of target data
10

18
Work required to create a image classifier
…
…
…
…
…
…
…
Classifier
（Neural Network）
Input : Image Output :
Classification result
2
1. Prepare dataset
Prepare as much data as possible
Each data includes image and its class index
2. Design neural network structure
You can use sample project
3. Train designed neural network using
prepared dataset
Basically you can create image classifier with these three steps
… 「0」
… 「1」
… 「2」
… 「3」
… 「4」
… 「5」
… 「6」
… 「7」
… 「8」
… 「9」

19
1. Prepare dataset
• Create dataset CSV files according to a predetermined format
x:image y:label
./training/5/0.png 5
./training/0/1.png 0
./training/4/2.png 4
./training/1/3.png 1
./training/9/4.png 9
./training/2/5.png 2
./training/1/6.png 1
…
File name of hand-writing digit class Index
It can be created with Excel or simple script

20
2. Design neural network structure
• Convolutional Neural Networks • Put layers as it is by drag & drop,
and edit properties of layers
Add layers by drag &
drop
Edit property of layer
You can design the network structure very easily and intuitively
5 x 5 Convolution
Activation
2 x 2 Pooling
3 x 3 Convolution
Activation
2 x 2 Pooling
Affine
Activation
Fully connected
Softmax
Categorical Loss

21
3. Train designed neural network using prepared dataset
• Training starts just by pushing the training button
When the training is complete, an image classifier is obtained
loss
training iteration