PyData NYC by Akira Shibata

Akira Shibata
Akira ShibataChief Data Scientist
Putting Together 
World's Best Data Processing Research 
with Python 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 
Akira Shibata, PhD 
Shiroyagi Corporation
Who am I 
Akira Shibata, PhD. 
TW: @punkphysicist 
CEO, Shiroyagi Corporation (shiroyagi.co.jp) 
Kamelio: Personalised News Curation 
Kamect: Contents Discovery Platform 
2004 - 2010: 
Data Scientist @ NYU 
Statistical data modelling @ LHC, CERN 
2010 - 2013 
Boston Consulting Group 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 2
Copyright 2014 Shiroyagi Corporation. All rights reserved. 3
Statistical modelling of Physics data 
Confirmatory: 
Highly theory driven model building 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 4
Telling discovery from noise 
The model tells you the expected uncertainty 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 5
Copyright 2014 Shiroyagi Corporation. All rights reserved. 6
Copyright 2014 Shiroyagi Corporation. All rights reserved. 7
Copyright 2014 Shiroyagi Corporation. All rights reserved. 8
Copyright 2014 Shiroyagi Corporation. All rights reserved. 9
Copyright 2014 Shiroyagi Corporation. All rights reserved. 10
Kamelio 
“Deep Learning” 
“Internet of 
Things” 
“Medical IT” 
“Global Strategy” 
Collects news through >3M 
topics to chose from 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 11
3 
“Cats” 
“Anime” 
“Cats reaction to sighting 
dogs for the first time” 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 12
Python puts all our tools together 
Image in Detect 
regions 
Object 
recog. Scoring Cropping 
0 1 2 3 4 
Matlab 
+Scipy 
C++ 
+Libraries 
Numpy PIL 
IPython and Python script 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 13
Our approach is 
heavily influenced by 
Berkeley Vision and 
Learning Center 
Acknowledgement 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 14
Detect 
regions 
0 1 2 3 4 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 15
Region detection: Telling where to look at 
How do we find regions to feed into object recognition? 
Default strategy was to look at the center 
1 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 16
Exhaustive windows -> segmentation 
Search over position, 
scale, aspect ratio 
Grouping parts of 
image at different scales 
Exhaustive search far too time inefficient 
for use with Deep Learning 
1 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 17
1 Region detection: in practice 
Install Malab and Selective Search algorithm 
from author 
Run matlab as subprocess 
pid = subprocess.Popen(shlex.split(mc), stdout=open('/dev/null', 
'w'), cwd=script_dirname) 
matlab -nojvm -r "try; selective_search({‘image_file.jpg’}, 
‘output.mat'); catch; exit; end; exit” 
1 
2 
3 
Import output using scipy.io 
all_boxes = list(scipy.io.loadmat(‘output.mat')['all_boxes'][0]) 
subtractor = np.array((1, 1, 0, 0))[np.newaxis, :] 
all_boxes = [boxes - subtractor for boxes in all_boxes] 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 18
1 Region detection: proposals generated 
~200 proposals generated per image 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 19
Object 
recog. 
0 1 2 3 4 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 20
Object recognition 
Deep blue beat Kasparov at chess in 1997… 
2 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 21
2 Deep Learning: Damn good at it 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 22
2 Convoluted Neural Network 
… 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 23
Caffe: open R-CNN framework under rapid dev. 
C++/CUDA with Python wrapper 
2 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 24
Pre-trained models published 
We used 200-category object recog. model 
developed for 2013 ImageNet Challenge 
2 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 25
2 Object recognition: in practice 
Install a bunch of libraries and Caffe 
CUDA, Boost, OpenCV, BLAS… 
Import wrapper and configure 
MODEL_FILE=‘models/bvlc_…_ilsvrc13/deploy.prototxt’ 
PRETRAINED_FILE = ‘models/…/bvlc_…_ilsvrc13.caffemodel’ 
MEAN_FILE = 'caffe/imagenet/ilsvrc_2012_mean.npy' 
detector = caffe.Detector(MODEL_FILE, PRETRAINED_FILE, 
mean=np.load(MEAN_FILE), raw_scale=255, channel_swap=[2,1,0]) 
1 
2 
3 
Pass found regions for object detection 
self.detect_windows(zip(image_fnames, windows_list)) 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 26
2 Object recognition: Result 
Obj Score 
0 domestic cat 1.03649377823 
1 domestic cat 0.0617411136627 
2 domestic cat -0.097744345665 
3 domestic cat -0.738470971584 
4 chair -0.988844156265 
5 skunk -0.999914288521 
6 tv or monitor -1.00460898876 
7 rubber eraser -1.01068615913 
8 chair -1.04896986485 
9 rubber eraser -1.09035253525 
10 band aid -1.09691572189 
Takes minutes to detect all windows 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 27
2 Object recognition: Result 
Obj Score 
0 person 0.126184225082 
1 person 0.0311727523804 
2 person -0.0777613520622 
3 neck brace -0.39757412672 
4 person -0.415030777454 
5 drum -0.421649754047 
6 neck brace -0.481261610985 
7 tie -0.649109125137 
8 neck brace -0.719438135624 
9 face powder -0.789100408554 
10 face powder -0.838757038116 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 28
Scoring 
0 1 2 3 4 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 29
3 Scoring 
1 For every pixel, sum up score from all detections 
for 
i 
in 
xrange(len(detec0ons)): 
arr[ymin:ymax, 
xmin:xmax] 
+= 
math.exp(score) 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 30
Score heatmap 
We used 200-cat object recognition model 
developed for 2013 ImageNet Challenge 
3 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 31
Cropping 
0 1 2 3 4 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 32
4 Cropping 
Generate all possible crop areas 
while 
y+hws 
<= 
h: 
while 
x+hws 
<= 
w: 
window_locs 
= 
np.vstack((window_locs, 
[x, 
y, 
x+hws, 
y+hws])) 
Find the crop that encloses the highest point of 
interest in the centre 
for 
i, 
window_loc 
in 
enumerate(window_locs): 
x1, 
y1, 
x2, 
y2 
= 
window_loc 
if 
max_val 
!= 
np.max(arr_con[y1:y2, 
x1:x2]): 
scores[i]=np.nan 
else: 
scores[i] 
= 
((x1+x2)/2.-­‐xp)**2+ 
((y1+y2)/2.-­‐yp)**2 
1 
2 
3 
Crop and save! 
img_pil 
= 
Image.open(fn) 
crop_area=map(lambda 
x: 
int(x), 
window_locs[scores.argmax()]) 
img_crop 
= 
img_pil.crop(crop_area) 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 33
4 Finally 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 34
Future improvements 
Fast face/human 
detection 
Aspect detection: 
square or rectangle? 
Object weighting 
Magnification 
Unseen object 
Copyright 2014 Shiroyagi Corporation. All rights reserved. 35
1 of 35

Recommended

Faster R-CNN: Towards real-time object detection with region proposal network... by
Faster R-CNN: Towards real-time object detection with region proposal network...Faster R-CNN: Towards real-time object detection with region proposal network...
Faster R-CNN: Towards real-time object detection with region proposal network...Universitat Politècnica de Catalunya
25.2K views38 slides
【DL輪読会】Unpaired Image Super-Resolution Using Pseudo-Supervision by
【DL輪読会】Unpaired Image Super-Resolution Using Pseudo-Supervision【DL輪読会】Unpaired Image Super-Resolution Using Pseudo-Supervision
【DL輪読会】Unpaired Image Super-Resolution Using Pseudo-SupervisionDeep Learning JP
933 views33 slides
物体検出の歴史(R-CNNからSSD・YOLOまで) by
物体検出の歴史(R-CNNからSSD・YOLOまで)物体検出の歴史(R-CNNからSSD・YOLOまで)
物体検出の歴史(R-CNNからSSD・YOLOまで)HironoriKanazawa
1.6K views53 slides
【ECCV 2016 BNMW】Human Action Recognition without Human by
【ECCV 2016 BNMW】Human Action Recognition without Human【ECCV 2016 BNMW】Human Action Recognition without Human
【ECCV 2016 BNMW】Human Action Recognition without HumanHirokatsu Kataoka
2.9K views15 slides
200612_BioPackathon_ss by
200612_BioPackathon_ss200612_BioPackathon_ss
200612_BioPackathon_ssSatoshi Kume
2.7K views25 slides
Object detection with Tensorflow Api by
Object detection with Tensorflow ApiObject detection with Tensorflow Api
Object detection with Tensorflow ApiArwinKhan1
1.2K views30 slides

More Related Content

What's hot

CVPR2016を自分なりにまとめてみた by
CVPR2016を自分なりにまとめてみたCVPR2016を自分なりにまとめてみた
CVPR2016を自分なりにまとめてみたHiroshi Fukui
4.8K views27 slides
Action Recognitionの歴史と最新動向 by
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向Ohnishi Katsunori
8.7K views40 slides
[AI07] Revolutionizing Image Processing with Cognitive Toolkit by
[AI07] Revolutionizing Image Processing with Cognitive Toolkit[AI07] Revolutionizing Image Processing with Cognitive Toolkit
[AI07] Revolutionizing Image Processing with Cognitive Toolkitde:code 2017
689 views62 slides
FPT17: An object detector based on multiscale sliding window search using a f... by
FPT17: An object detector based on multiscale sliding window search using a f...FPT17: An object detector based on multiscale sliding window search using a f...
FPT17: An object detector based on multiscale sliding window search using a f...Hiroki Nakahara
1.5K views28 slides
【DL輪読会】ViT + Self Supervised Learningまとめ by
【DL輪読会】ViT + Self Supervised Learningまとめ【DL輪読会】ViT + Self Supervised Learningまとめ
【DL輪読会】ViT + Self Supervised LearningまとめDeep Learning JP
4K views52 slides
モデルアーキテクチャ観点からの高速化2019 by
モデルアーキテクチャ観点からの高速化2019モデルアーキテクチャ観点からの高速化2019
モデルアーキテクチャ観点からの高速化2019Yusuke Uchida
17.7K views90 slides

What's hot(20)

CVPR2016を自分なりにまとめてみた by Hiroshi Fukui
CVPR2016を自分なりにまとめてみたCVPR2016を自分なりにまとめてみた
CVPR2016を自分なりにまとめてみた
Hiroshi Fukui4.8K views
Action Recognitionの歴史と最新動向 by Ohnishi Katsunori
Action Recognitionの歴史と最新動向Action Recognitionの歴史と最新動向
Action Recognitionの歴史と最新動向
Ohnishi Katsunori8.7K views
[AI07] Revolutionizing Image Processing with Cognitive Toolkit by de:code 2017
[AI07] Revolutionizing Image Processing with Cognitive Toolkit[AI07] Revolutionizing Image Processing with Cognitive Toolkit
[AI07] Revolutionizing Image Processing with Cognitive Toolkit
de:code 2017689 views
FPT17: An object detector based on multiscale sliding window search using a f... by Hiroki Nakahara
FPT17: An object detector based on multiscale sliding window search using a f...FPT17: An object detector based on multiscale sliding window search using a f...
FPT17: An object detector based on multiscale sliding window search using a f...
Hiroki Nakahara1.5K views
【DL輪読会】ViT + Self Supervised Learningまとめ by Deep Learning JP
【DL輪読会】ViT + Self Supervised Learningまとめ【DL輪読会】ViT + Self Supervised Learningまとめ
【DL輪読会】ViT + Self Supervised Learningまとめ
Deep Learning JP4K views
モデルアーキテクチャ観点からの高速化2019 by Yusuke Uchida
モデルアーキテクチャ観点からの高速化2019モデルアーキテクチャ観点からの高速化2019
モデルアーキテクチャ観点からの高速化2019
Yusuke Uchida17.7K views
Real Time Human Posture Detection with Multiple Depth Sensors by Wassim Filali
Real Time Human Posture Detection with Multiple Depth SensorsReal Time Human Posture Detection with Multiple Depth Sensors
Real Time Human Posture Detection with Multiple Depth Sensors
Wassim Filali2.1K views
用 Python 玩 LHC 公開數據 by Yuan CHAO
用 Python 玩 LHC 公開數據用 Python 玩 LHC 公開數據
用 Python 玩 LHC 公開數據
Yuan CHAO3.6K views
【CVPR 2020 メタサーベイ】Video Analysis and Understanding by cvpaper. challenge
【CVPR 2020 メタサーベイ】Video Analysis and Understanding【CVPR 2020 メタサーベイ】Video Analysis and Understanding
【CVPR 2020 メタサーベイ】Video Analysis and Understanding
cvpaper. challenge719 views
Ice: lightweight, efficient rendering for remote sensing images by otb
Ice: lightweight, efficient rendering for remote sensing imagesIce: lightweight, efficient rendering for remote sensing images
Ice: lightweight, efficient rendering for remote sensing images
otb833 views
Deep Learning Cases: Text and Image Processing by Grigory Sapunov
Deep Learning Cases: Text and Image ProcessingDeep Learning Cases: Text and Image Processing
Deep Learning Cases: Text and Image Processing
Grigory Sapunov9.4K views
Orfeo ToolBox workshop at FOSS4G Europe 2015 by otb
Orfeo ToolBox workshop at FOSS4G Europe 2015Orfeo ToolBox workshop at FOSS4G Europe 2015
Orfeo ToolBox workshop at FOSS4G Europe 2015
otb1.3K views
Extended Co-occurrence HOG with Dense Trajectories for Fine-grained Activity ... by Hirokatsu Kataoka
Extended Co-occurrence HOG with Dense Trajectories for Fine-grained Activity ...Extended Co-occurrence HOG with Dense Trajectories for Fine-grained Activity ...
Extended Co-occurrence HOG with Dense Trajectories for Fine-grained Activity ...
Hirokatsu Kataoka2.2K views
kaggle NFL 1st and Future - Impact Detection by Kazuyuki Miyazawa
kaggle NFL 1st and Future - Impact Detectionkaggle NFL 1st and Future - Impact Detection
kaggle NFL 1st and Future - Impact Detection
Александр Заричковый "Faster than real-time face detection" by Fwdays
Александр Заричковый "Faster than real-time face detection"Александр Заричковый "Faster than real-time face detection"
Александр Заричковый "Faster than real-time face detection"
Fwdays2.6K views
Deep Learning in the Wild with Arno Candel by Sri Ambati
Deep Learning in the Wild with Arno CandelDeep Learning in the Wild with Arno Candel
Deep Learning in the Wild with Arno Candel
Sri Ambati1.6K views
Pragmatic Remote Sensing - IGARSS 2010 by otb
Pragmatic Remote Sensing - IGARSS 2010Pragmatic Remote Sensing - IGARSS 2010
Pragmatic Remote Sensing - IGARSS 2010
otb2.2K views
Machine learning quality for production by yusuke shibui
Machine learning quality for productionMachine learning quality for production
Machine learning quality for production
yusuke shibui150 views

Similar to PyData NYC by Akira Shibata

Challenges of Deep Learning in Computer Vision Webinar - Tessellate Imaging by
Challenges of Deep Learning in Computer Vision Webinar - Tessellate ImagingChallenges of Deep Learning in Computer Vision Webinar - Tessellate Imaging
Challenges of Deep Learning in Computer Vision Webinar - Tessellate ImagingAdhesh Shrivastava
21 views23 slides
Introduction to Crab - Python Framework for Building Recommender Systems by
Introduction to Crab - Python Framework for Building Recommender SystemsIntroduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender SystemsMarcel Caraciolo
2.7K views25 slides
DS LAB MANUAL.pdf by
DS LAB MANUAL.pdfDS LAB MANUAL.pdf
DS LAB MANUAL.pdfBuilders Engineering College
58 views84 slides
20181212 Queensland AI Meetup by
20181212 Queensland AI Meetup20181212 Queensland AI Meetup
20181212 Queensland AI MeetupAdam Craven
453 views72 slides
IRJET- Object Detection in an Image using Convolutional Neural Network by
IRJET- Object Detection in an Image using Convolutional Neural NetworkIRJET- Object Detection in an Image using Convolutional Neural Network
IRJET- Object Detection in an Image using Convolutional Neural NetworkIRJET Journal
50 views3 slides
Machine_learning_internship_report_facemaskdetection.pptx by
Machine_learning_internship_report_facemaskdetection.pptxMachine_learning_internship_report_facemaskdetection.pptx
Machine_learning_internship_report_facemaskdetection.pptxpratikpatil862906
16 views13 slides

Similar to PyData NYC by Akira Shibata(20)

Challenges of Deep Learning in Computer Vision Webinar - Tessellate Imaging by Adhesh Shrivastava
Challenges of Deep Learning in Computer Vision Webinar - Tessellate ImagingChallenges of Deep Learning in Computer Vision Webinar - Tessellate Imaging
Challenges of Deep Learning in Computer Vision Webinar - Tessellate Imaging
Introduction to Crab - Python Framework for Building Recommender Systems by Marcel Caraciolo
Introduction to Crab - Python Framework for Building Recommender SystemsIntroduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender Systems
Marcel Caraciolo2.7K views
20181212 Queensland AI Meetup by Adam Craven
20181212 Queensland AI Meetup20181212 Queensland AI Meetup
20181212 Queensland AI Meetup
Adam Craven453 views
IRJET- Object Detection in an Image using Convolutional Neural Network by IRJET Journal
IRJET- Object Detection in an Image using Convolutional Neural NetworkIRJET- Object Detection in an Image using Convolutional Neural Network
IRJET- Object Detection in an Image using Convolutional Neural Network
IRJET Journal50 views
Machine_learning_internship_report_facemaskdetection.pptx by pratikpatil862906
Machine_learning_internship_report_facemaskdetection.pptxMachine_learning_internship_report_facemaskdetection.pptx
Machine_learning_internship_report_facemaskdetection.pptx
Using Deep Learning for Computer Vision Applications by Farshid Pirahansiah
Using Deep Learning for Computer Vision ApplicationsUsing Deep Learning for Computer Vision Applications
Using Deep Learning for Computer Vision Applications
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin... by Edureka!
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
PyTorch Python Tutorial | Deep Learning Using PyTorch | Image Classifier Usin...
Edureka!2.4K views
Fernando Arnaboldi - Exposing Hidden Exploitable Behaviors Using Extended Dif... by Codemotion
Fernando Arnaboldi - Exposing Hidden Exploitable Behaviors Using Extended Dif...Fernando Arnaboldi - Exposing Hidden Exploitable Behaviors Using Extended Dif...
Fernando Arnaboldi - Exposing Hidden Exploitable Behaviors Using Extended Dif...
Codemotion210 views
Pytorch kr devcon by jaewon lee
Pytorch kr devconPytorch kr devcon
Pytorch kr devcon
jaewon lee647 views
Vipul divyanshu documentation on Kinect and Motion Tracking by Vipul Divyanshu
Vipul divyanshu documentation  on Kinect and Motion TrackingVipul divyanshu documentation  on Kinect and Motion Tracking
Vipul divyanshu documentation on Kinect and Motion Tracking
Vipul Divyanshu1.7K views
Crab - A Python Framework for Building Recommendation Systems by Marcel Caraciolo
Crab - A Python Framework for Building Recommendation SystemsCrab - A Python Framework for Building Recommendation Systems
Crab - A Python Framework for Building Recommendation Systems
Marcel Caraciolo9.6K views
SDOBenchmark - a machine learning image dataset for the prediction of solar f... by Roman Bolzern
SDOBenchmark - a machine learning image dataset for the prediction of solar f...SDOBenchmark - a machine learning image dataset for the prediction of solar f...
SDOBenchmark - a machine learning image dataset for the prediction of solar f...
Roman Bolzern289 views
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A... by Databricks
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...
Updates from Project Hydrogen: Unifying State-of-the-Art AI and Big Data in A...
Databricks1.7K views
深層学習フレームワーク概要とChainerの事例紹介 by Kenta Oono
深層学習フレームワーク概要とChainerの事例紹介深層学習フレームワーク概要とChainerの事例紹介
深層学習フレームワーク概要とChainerの事例紹介
Kenta Oono1.6K views
Dl4j in the wild by Adam Gibson
Dl4j in the wildDl4j in the wild
Dl4j in the wild
Adam Gibson1.4K views

More from Akira Shibata

W&B monthly meetup#7 Intro.pdf by
W&B monthly meetup#7 Intro.pdfW&B monthly meetup#7 Intro.pdf
W&B monthly meetup#7 Intro.pdfAkira Shibata
738 views14 slides
20230705 - Optuna Integration (to share).pdf by
20230705 - Optuna Integration (to share).pdf20230705 - Optuna Integration (to share).pdf
20230705 - Optuna Integration (to share).pdfAkira Shibata
103 views15 slides
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdf by
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdfmakoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdf
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdfAkira Shibata
759 views29 slides
LLM Webinar - シバタアキラ to share.pdf by
LLM Webinar - シバタアキラ to share.pdfLLM Webinar - シバタアキラ to share.pdf
LLM Webinar - シバタアキラ to share.pdfAkira Shibata
332 views10 slides
W&B Seminar #4.pdf by
W&B Seminar #4.pdfW&B Seminar #4.pdf
W&B Seminar #4.pdfAkira Shibata
448 views11 slides
Kaggle and data science by
Kaggle and data scienceKaggle and data science
Kaggle and data scienceAkira Shibata
1.1K views29 slides

More from Akira Shibata(20)

W&B monthly meetup#7 Intro.pdf by Akira Shibata
W&B monthly meetup#7 Intro.pdfW&B monthly meetup#7 Intro.pdf
W&B monthly meetup#7 Intro.pdf
Akira Shibata738 views
20230705 - Optuna Integration (to share).pdf by Akira Shibata
20230705 - Optuna Integration (to share).pdf20230705 - Optuna Integration (to share).pdf
20230705 - Optuna Integration (to share).pdf
Akira Shibata103 views
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdf by Akira Shibata
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdfmakoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdf
makoto shing (stability ai) - image model fine-tuning - wandb_event_230525.pdf
Akira Shibata759 views
LLM Webinar - シバタアキラ to share.pdf by Akira Shibata
LLM Webinar - シバタアキラ to share.pdfLLM Webinar - シバタアキラ to share.pdf
LLM Webinar - シバタアキラ to share.pdf
Akira Shibata332 views
Kaggle and data science by Akira Shibata
Kaggle and data scienceKaggle and data science
Kaggle and data science
Akira Shibata1.1K views
Akira shibata at developer summit 2016 by Akira Shibata
Akira shibata at developer summit 2016Akira shibata at developer summit 2016
Akira shibata at developer summit 2016
Akira Shibata4.9K views
PyData.Tokyo Hackathon#2 TensorFlow by Akira Shibata
PyData.Tokyo Hackathon#2 TensorFlowPyData.Tokyo Hackathon#2 TensorFlow
PyData.Tokyo Hackathon#2 TensorFlow
Akira Shibata2.6K views
20150421 日経ビッグデータカンファレンス by Akira Shibata
20150421 日経ビッグデータカンファレンス20150421 日経ビッグデータカンファレンス
20150421 日経ビッグデータカンファレンス
Akira Shibata1.7K views
人工知能をビジネスに活かす by Akira Shibata
人工知能をビジネスに活かす人工知能をビジネスに活かす
人工知能をビジネスに活かす
Akira Shibata3.6K views
LHCにおける素粒子ビッグデータの解析とROOTライブラリ(Big Data Analysis at LHC and ROOT) by Akira Shibata
LHCにおける素粒子ビッグデータの解析とROOTライブラリ(Big Data Analysis at LHC and ROOT)LHCにおける素粒子ビッグデータの解析とROOTライブラリ(Big Data Analysis at LHC and ROOT)
LHCにおける素粒子ビッグデータの解析とROOTライブラリ(Big Data Analysis at LHC and ROOT)
Akira Shibata6.7K views
PyData Tokyo Tutorial & Hackathon #1 by Akira Shibata
PyData Tokyo Tutorial & Hackathon #1PyData Tokyo Tutorial & Hackathon #1
PyData Tokyo Tutorial & Hackathon #1
Akira Shibata13.3K views
20141127 py datatokyomeetup2 by Akira Shibata
20141127 py datatokyomeetup220141127 py datatokyomeetup2
20141127 py datatokyomeetup2
Akira Shibata1.5K views
The LHC Explained by CNN by Akira Shibata
The LHC Explained by CNNThe LHC Explained by CNN
The LHC Explained by CNN
Akira Shibata641 views
Analysis Software Development by Akira Shibata
Analysis Software DevelopmentAnalysis Software Development
Analysis Software Development
Akira Shibata826 views
Top Cross Section Measurement by Akira Shibata
Top Cross Section MeasurementTop Cross Section Measurement
Top Cross Section Measurement
Akira Shibata815 views
Analysis Software Benchmark by Akira Shibata
Analysis Software BenchmarkAnalysis Software Benchmark
Analysis Software Benchmark
Akira Shibata732 views

Recently uploaded

Ethical issues associated with Genetically Modified Crops and Genetically Mod... by
Ethical issues associated with Genetically Modified Crops and Genetically Mod...Ethical issues associated with Genetically Modified Crops and Genetically Mod...
Ethical issues associated with Genetically Modified Crops and Genetically Mod...PunithKumars6
22 views20 slides
Artificial Intelligence Helps in Drug Designing and Discovery.pptx by
Artificial Intelligence Helps in Drug Designing and Discovery.pptxArtificial Intelligence Helps in Drug Designing and Discovery.pptx
Artificial Intelligence Helps in Drug Designing and Discovery.pptxabhinashsahoo2001
118 views22 slides
MILK LIPIDS 2.pptx by
MILK LIPIDS 2.pptxMILK LIPIDS 2.pptx
MILK LIPIDS 2.pptxabhinambroze18
7 views15 slides
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdf by
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdfMODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdf
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdfKerryNuez1
21 views5 slides
PRINCIPLES-OF ASSESSMENT by
PRINCIPLES-OF ASSESSMENTPRINCIPLES-OF ASSESSMENT
PRINCIPLES-OF ASSESSMENTrbalmagro
11 views12 slides
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl... by
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...GIFT KIISI NKIN
17 views31 slides

Recently uploaded(20)

Ethical issues associated with Genetically Modified Crops and Genetically Mod... by PunithKumars6
Ethical issues associated with Genetically Modified Crops and Genetically Mod...Ethical issues associated with Genetically Modified Crops and Genetically Mod...
Ethical issues associated with Genetically Modified Crops and Genetically Mod...
PunithKumars622 views
Artificial Intelligence Helps in Drug Designing and Discovery.pptx by abhinashsahoo2001
Artificial Intelligence Helps in Drug Designing and Discovery.pptxArtificial Intelligence Helps in Drug Designing and Discovery.pptx
Artificial Intelligence Helps in Drug Designing and Discovery.pptx
abhinashsahoo2001118 views
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdf by KerryNuez1
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdfMODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdf
MODULE-9-Biotechnology, Genetically Modified Organisms, and Gene Therapy.pdf
KerryNuez121 views
PRINCIPLES-OF ASSESSMENT by rbalmagro
PRINCIPLES-OF ASSESSMENTPRINCIPLES-OF ASSESSMENT
PRINCIPLES-OF ASSESSMENT
rbalmagro11 views
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl... by GIFT KIISI NKIN
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...
Synthesis and Characterization of Magnetite-Magnesium Sulphate-Sodium Dodecyl...
GIFT KIISI NKIN17 views
A Ready-to-Analyze High-Plex Spatial Signature Development Workflow for Cance... by InsideScientific
A Ready-to-Analyze High-Plex Spatial Signature Development Workflow for Cance...A Ready-to-Analyze High-Plex Spatial Signature Development Workflow for Cance...
A Ready-to-Analyze High-Plex Spatial Signature Development Workflow for Cance...
InsideScientific43 views
Light Pollution for LVIS students by CWBarthlmew
Light Pollution for LVIS studentsLight Pollution for LVIS students
Light Pollution for LVIS students
CWBarthlmew5 views
Experimental animal Guinea pigs.pptx by Mansee Arya
Experimental animal Guinea pigs.pptxExperimental animal Guinea pigs.pptx
Experimental animal Guinea pigs.pptx
Mansee Arya13 views
Open Access Publishing in Astrophysics by Peter Coles
Open Access Publishing in AstrophysicsOpen Access Publishing in Astrophysics
Open Access Publishing in Astrophysics
Peter Coles725 views
"How can I develop my learning path in bioinformatics? by Bioinformy
"How can I develop my learning path in bioinformatics?"How can I develop my learning path in bioinformatics?
"How can I develop my learning path in bioinformatics?
Bioinformy21 views
Connecting communities to promote FAIR resources: perspectives from an RDA / ... by Allyson Lister
Connecting communities to promote FAIR resources: perspectives from an RDA / ...Connecting communities to promote FAIR resources: perspectives from an RDA / ...
Connecting communities to promote FAIR resources: perspectives from an RDA / ...
Allyson Lister34 views
RemeOs science and clinical evidence by PetrusViitanen1
RemeOs science and clinical evidenceRemeOs science and clinical evidence
RemeOs science and clinical evidence
PetrusViitanen135 views
How to be(come) a successful PhD student by Tom Mens
How to be(come) a successful PhD studentHow to be(come) a successful PhD student
How to be(come) a successful PhD student
Tom Mens460 views
Guinea Pig as a Model for Translation Research by PervaizDar1
Guinea Pig as a Model for Translation ResearchGuinea Pig as a Model for Translation Research
Guinea Pig as a Model for Translation Research
PervaizDar111 views
Pollination By Nagapradheesh.M.pptx by MNAGAPRADHEESH
Pollination By Nagapradheesh.M.pptxPollination By Nagapradheesh.M.pptx
Pollination By Nagapradheesh.M.pptx
MNAGAPRADHEESH15 views
CSF -SHEEBA.D presentation.pptx by SheebaD7
CSF -SHEEBA.D presentation.pptxCSF -SHEEBA.D presentation.pptx
CSF -SHEEBA.D presentation.pptx
SheebaD711 views

PyData NYC by Akira Shibata

  • 1. Putting Together World's Best Data Processing Research with Python Copyright 2014 Shiroyagi Corporation. All rights reserved. Akira Shibata, PhD Shiroyagi Corporation
  • 2. Who am I Akira Shibata, PhD. TW: @punkphysicist CEO, Shiroyagi Corporation (shiroyagi.co.jp) Kamelio: Personalised News Curation Kamect: Contents Discovery Platform 2004 - 2010: Data Scientist @ NYU Statistical data modelling @ LHC, CERN 2010 - 2013 Boston Consulting Group Copyright 2014 Shiroyagi Corporation. All rights reserved. 2
  • 3. Copyright 2014 Shiroyagi Corporation. All rights reserved. 3
  • 4. Statistical modelling of Physics data Confirmatory: Highly theory driven model building Copyright 2014 Shiroyagi Corporation. All rights reserved. 4
  • 5. Telling discovery from noise The model tells you the expected uncertainty Copyright 2014 Shiroyagi Corporation. All rights reserved. 5
  • 6. Copyright 2014 Shiroyagi Corporation. All rights reserved. 6
  • 7. Copyright 2014 Shiroyagi Corporation. All rights reserved. 7
  • 8. Copyright 2014 Shiroyagi Corporation. All rights reserved. 8
  • 9. Copyright 2014 Shiroyagi Corporation. All rights reserved. 9
  • 10. Copyright 2014 Shiroyagi Corporation. All rights reserved. 10
  • 11. Kamelio “Deep Learning” “Internet of Things” “Medical IT” “Global Strategy” Collects news through >3M topics to chose from Copyright 2014 Shiroyagi Corporation. All rights reserved. 11
  • 12. 3 “Cats” “Anime” “Cats reaction to sighting dogs for the first time” Copyright 2014 Shiroyagi Corporation. All rights reserved. 12
  • 13. Python puts all our tools together Image in Detect regions Object recog. Scoring Cropping 0 1 2 3 4 Matlab +Scipy C++ +Libraries Numpy PIL IPython and Python script Copyright 2014 Shiroyagi Corporation. All rights reserved. 13
  • 14. Our approach is heavily influenced by Berkeley Vision and Learning Center Acknowledgement Copyright 2014 Shiroyagi Corporation. All rights reserved. 14
  • 15. Detect regions 0 1 2 3 4 Copyright 2014 Shiroyagi Corporation. All rights reserved. 15
  • 16. Region detection: Telling where to look at How do we find regions to feed into object recognition? Default strategy was to look at the center 1 Copyright 2014 Shiroyagi Corporation. All rights reserved. 16
  • 17. Exhaustive windows -> segmentation Search over position, scale, aspect ratio Grouping parts of image at different scales Exhaustive search far too time inefficient for use with Deep Learning 1 Copyright 2014 Shiroyagi Corporation. All rights reserved. 17
  • 18. 1 Region detection: in practice Install Malab and Selective Search algorithm from author Run matlab as subprocess pid = subprocess.Popen(shlex.split(mc), stdout=open('/dev/null', 'w'), cwd=script_dirname) matlab -nojvm -r "try; selective_search({‘image_file.jpg’}, ‘output.mat'); catch; exit; end; exit” 1 2 3 Import output using scipy.io all_boxes = list(scipy.io.loadmat(‘output.mat')['all_boxes'][0]) subtractor = np.array((1, 1, 0, 0))[np.newaxis, :] all_boxes = [boxes - subtractor for boxes in all_boxes] Copyright 2014 Shiroyagi Corporation. All rights reserved. 18
  • 19. 1 Region detection: proposals generated ~200 proposals generated per image Copyright 2014 Shiroyagi Corporation. All rights reserved. 19
  • 20. Object recog. 0 1 2 3 4 Copyright 2014 Shiroyagi Corporation. All rights reserved. 20
  • 21. Object recognition Deep blue beat Kasparov at chess in 1997… 2 Copyright 2014 Shiroyagi Corporation. All rights reserved. 21
  • 22. 2 Deep Learning: Damn good at it Copyright 2014 Shiroyagi Corporation. All rights reserved. 22
  • 23. 2 Convoluted Neural Network … Copyright 2014 Shiroyagi Corporation. All rights reserved. 23
  • 24. Caffe: open R-CNN framework under rapid dev. C++/CUDA with Python wrapper 2 Copyright 2014 Shiroyagi Corporation. All rights reserved. 24
  • 25. Pre-trained models published We used 200-category object recog. model developed for 2013 ImageNet Challenge 2 Copyright 2014 Shiroyagi Corporation. All rights reserved. 25
  • 26. 2 Object recognition: in practice Install a bunch of libraries and Caffe CUDA, Boost, OpenCV, BLAS… Import wrapper and configure MODEL_FILE=‘models/bvlc_…_ilsvrc13/deploy.prototxt’ PRETRAINED_FILE = ‘models/…/bvlc_…_ilsvrc13.caffemodel’ MEAN_FILE = 'caffe/imagenet/ilsvrc_2012_mean.npy' detector = caffe.Detector(MODEL_FILE, PRETRAINED_FILE, mean=np.load(MEAN_FILE), raw_scale=255, channel_swap=[2,1,0]) 1 2 3 Pass found regions for object detection self.detect_windows(zip(image_fnames, windows_list)) Copyright 2014 Shiroyagi Corporation. All rights reserved. 26
  • 27. 2 Object recognition: Result Obj Score 0 domestic cat 1.03649377823 1 domestic cat 0.0617411136627 2 domestic cat -0.097744345665 3 domestic cat -0.738470971584 4 chair -0.988844156265 5 skunk -0.999914288521 6 tv or monitor -1.00460898876 7 rubber eraser -1.01068615913 8 chair -1.04896986485 9 rubber eraser -1.09035253525 10 band aid -1.09691572189 Takes minutes to detect all windows Copyright 2014 Shiroyagi Corporation. All rights reserved. 27
  • 28. 2 Object recognition: Result Obj Score 0 person 0.126184225082 1 person 0.0311727523804 2 person -0.0777613520622 3 neck brace -0.39757412672 4 person -0.415030777454 5 drum -0.421649754047 6 neck brace -0.481261610985 7 tie -0.649109125137 8 neck brace -0.719438135624 9 face powder -0.789100408554 10 face powder -0.838757038116 Copyright 2014 Shiroyagi Corporation. All rights reserved. 28
  • 29. Scoring 0 1 2 3 4 Copyright 2014 Shiroyagi Corporation. All rights reserved. 29
  • 30. 3 Scoring 1 For every pixel, sum up score from all detections for i in xrange(len(detec0ons)): arr[ymin:ymax, xmin:xmax] += math.exp(score) Copyright 2014 Shiroyagi Corporation. All rights reserved. 30
  • 31. Score heatmap We used 200-cat object recognition model developed for 2013 ImageNet Challenge 3 Copyright 2014 Shiroyagi Corporation. All rights reserved. 31
  • 32. Cropping 0 1 2 3 4 Copyright 2014 Shiroyagi Corporation. All rights reserved. 32
  • 33. 4 Cropping Generate all possible crop areas while y+hws <= h: while x+hws <= w: window_locs = np.vstack((window_locs, [x, y, x+hws, y+hws])) Find the crop that encloses the highest point of interest in the centre for i, window_loc in enumerate(window_locs): x1, y1, x2, y2 = window_loc if max_val != np.max(arr_con[y1:y2, x1:x2]): scores[i]=np.nan else: scores[i] = ((x1+x2)/2.-­‐xp)**2+ ((y1+y2)/2.-­‐yp)**2 1 2 3 Crop and save! img_pil = Image.open(fn) crop_area=map(lambda x: int(x), window_locs[scores.argmax()]) img_crop = img_pil.crop(crop_area) Copyright 2014 Shiroyagi Corporation. All rights reserved. 33
  • 34. 4 Finally Copyright 2014 Shiroyagi Corporation. All rights reserved. 34
  • 35. Future improvements Fast face/human detection Aspect detection: square or rectangle? Object weighting Magnification Unseen object Copyright 2014 Shiroyagi Corporation. All rights reserved. 35