SlideShare a Scribd company logo
Deep learning features and
similarity of movies based on their
video content
Summer Camp - Show Max - Lukáš Lopatovský
Assignment
● Deep learning allows extracting useful features from video
frames. Your task is to apply new deep learning frameworks
to extract features from video frames of selected movies
available in the ShowMax streaming platform.
● Goals:
● Extract deep features from video frames. Explore similar
movies in the space of latent features and adjust the
extraction process in order to create clusters of video assets
(eg. TV episodes).
Residual Networks
● Enable to build deeper (convolutional) neural
network. (State of the art method for the image
recognition.)
Residual Networks
● To enable to build deeper network, the residual
nets use the simple trick. They maintain the
residuum from the previous layer ( so do not
loose the previously known information )
Torch
- Efficient Tensor library (like NumPy) with an
efficient CUDA backend
- Neural Networks package -- build arbitrary
acyclic computation graphs with automatic
differentiation
- fast CUDA and CPU backends
- Good community and industry support - several
hundred community-built and maintained
packages.
● Torch example of ‘nn’ library
What has been done
● The movies were classified using arbitrary
number of picture frames.
● We have used already trained ImageNet FB-
resnet network and own data set trained and
fine-tuned networks to classify movies.
● To detect the object in the image, we have
classify the whole image, as well as we have
made the various crops to get more accurate
predictions. (Cropping showed better results)
Classification output
● By classification of frames in the movie, the
special file is produced (.res). It is in a form to
contain all the important data. It can be later post-
process according to the special needs of the user:
- To create Object detection .srt file.
- To get various cumulative classification results.
- To trace the appearance of the object at the time-
line.
Object classification exmple
Object classification exmple
Object classification exmple
False positive
False positive
False positive
Own datasets
● The network was successfully trained and fine-tuned
from the ResNet network
● However, it showed some problems based from
improper dataset.
- Some categories contain many irrelevant pictures in second half
of the search. (Special case: “The doctor House”)
- The style of the images in the search is often very different to the
style found in the movie. (kitchen, car)
- Movies mostly contain images full of people, so the categories
containing people make false positive prediction. (cinema, theater)
Classification example
⊙ω⊙
Classification example
⊙ω⊙
Object detection
Object detection
Object detection
Object detection
Object detection
Next step
● After the discussion in the company, the
programs were transformed to the easily usable
form.
● The feature vectors of the classification will be
used to find similarities among movies.
Compared to the existing algorithms and if
successful, incorporated into the current
recommendation system.

More Related Content

Similar to Deep learning features and similarity of movies based on their video content

VM: Image analysis and fragmentation
VM: Image analysis and fragmentation VM: Image analysis and fragmentation
VM: Image analysis and fragmentation
EOSC-hub project
 
Inside prototype 16-113
Inside prototype   16-113Inside prototype   16-113
Inside prototype 16-113
Kasun Udayanga
 
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
Unite2017Tokyo
 
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
Unity Technologies Japan K.K.
 
Computer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathonComputer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathon
Aditya Bhattacharya
 
Productionizing Machine Learning - Bigdata meetup 5-06-2019
Productionizing Machine Learning - Bigdata meetup 5-06-2019Productionizing Machine Learning - Bigdata meetup 5-06-2019
Productionizing Machine Learning - Bigdata meetup 5-06-2019
Iulian Pintoiu
 
Python_for_Visual_Effects_and_Animation_Pipelines
Python_for_Visual_Effects_and_Animation_PipelinesPython_for_Visual_Effects_and_Animation_Pipelines
Python_for_Visual_Effects_and_Animation_Pipelines
Russell Darling
 
Mirko Lucchese - Deep Image Processing
Mirko Lucchese - Deep Image ProcessingMirko Lucchese - Deep Image Processing
Mirko Lucchese - Deep Image Processing
MeetupDataScienceRoma
 
Ai use cases
Ai use casesAi use cases
Ai use cases
Sparsh Agarwal
 
TensorFlow Object Detection API
TensorFlow Object Detection APITensorFlow Object Detection API
TensorFlow Object Detection API
Algoscale Technologies Inc.
 
FINAL_Team_4.pptx
FINAL_Team_4.pptxFINAL_Team_4.pptx
FINAL_Team_4.pptx
nitin571047
 
ResearchSpace Collaborative Features
ResearchSpace Collaborative FeaturesResearchSpace Collaborative Features
ResearchSpace Collaborative Features
Barry Norton
 
Introduction to computer vision with Convoluted Neural Networks
Introduction to computer vision with Convoluted Neural NetworksIntroduction to computer vision with Convoluted Neural Networks
Introduction to computer vision with Convoluted Neural Networks
MarcinJedyk
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsDataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problems
PetteriTeikariPhD
 
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realitySynthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
NAVER Engineering
 
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
Amazon Web Services
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
StampedeCon
 
Introduction to computer vision
Introduction to computer visionIntroduction to computer vision
Introduction to computer vision
Marcin Jedyk
 
Team16_Narayana_InstanceSegmentation.pptx
Team16_Narayana_InstanceSegmentation.pptxTeam16_Narayana_InstanceSegmentation.pptx
Team16_Narayana_InstanceSegmentation.pptx
AnimeGuru1
 
Automatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face RecognitionAutomatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face Recognition
vatsal199567
 

Similar to Deep learning features and similarity of movies based on their video content (20)

VM: Image analysis and fragmentation
VM: Image analysis and fragmentation VM: Image analysis and fragmentation
VM: Image analysis and fragmentation
 
Inside prototype 16-113
Inside prototype   16-113Inside prototype   16-113
Inside prototype 16-113
 
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
 
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
【Unite 2017 Tokyo】インスタンシングを用いた美麗なグラフィックの実現方法
 
Computer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathonComputer vision-nit-silchar-hackathon
Computer vision-nit-silchar-hackathon
 
Productionizing Machine Learning - Bigdata meetup 5-06-2019
Productionizing Machine Learning - Bigdata meetup 5-06-2019Productionizing Machine Learning - Bigdata meetup 5-06-2019
Productionizing Machine Learning - Bigdata meetup 5-06-2019
 
Python_for_Visual_Effects_and_Animation_Pipelines
Python_for_Visual_Effects_and_Animation_PipelinesPython_for_Visual_Effects_and_Animation_Pipelines
Python_for_Visual_Effects_and_Animation_Pipelines
 
Mirko Lucchese - Deep Image Processing
Mirko Lucchese - Deep Image ProcessingMirko Lucchese - Deep Image Processing
Mirko Lucchese - Deep Image Processing
 
Ai use cases
Ai use casesAi use cases
Ai use cases
 
TensorFlow Object Detection API
TensorFlow Object Detection APITensorFlow Object Detection API
TensorFlow Object Detection API
 
FINAL_Team_4.pptx
FINAL_Team_4.pptxFINAL_Team_4.pptx
FINAL_Team_4.pptx
 
ResearchSpace Collaborative Features
ResearchSpace Collaborative FeaturesResearchSpace Collaborative Features
ResearchSpace Collaborative Features
 
Introduction to computer vision with Convoluted Neural Networks
Introduction to computer vision with Convoluted Neural NetworksIntroduction to computer vision with Convoluted Neural Networks
Introduction to computer vision with Convoluted Neural Networks
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsDataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problems
 
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed realitySynthesizing pseudo 2.5 d content from monocular videos for mixed reality
Synthesizing pseudo 2.5 d content from monocular videos for mixed reality
 
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
AWS re:Invent 2016: Deep Learning at Cloud Scale: Improving Video Discoverabi...
 
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
Don't Start from Scratch: Transfer Learning for Novel Computer Vision Problem...
 
Introduction to computer vision
Introduction to computer visionIntroduction to computer vision
Introduction to computer vision
 
Team16_Narayana_InstanceSegmentation.pptx
Team16_Narayana_InstanceSegmentation.pptxTeam16_Narayana_InstanceSegmentation.pptx
Team16_Narayana_InstanceSegmentation.pptx
 
Automatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face RecognitionAutomatic Attendace using convolutional neural network Face Recognition
Automatic Attendace using convolutional neural network Face Recognition
 

More from Showmax Engineering

PostgreSQL Terminology
PostgreSQL TerminologyPostgreSQL Terminology
PostgreSQL Terminology
Showmax Engineering
 
Networking fundamentals
Networking fundamentalsNetworking fundamentals
Networking fundamentals
Showmax Engineering
 
Android crash course
Android crash courseAndroid crash course
Android crash course
Showmax Engineering
 
PostgreSQL Monitoring using modern software stacks
PostgreSQL Monitoring using modern software stacksPostgreSQL Monitoring using modern software stacks
PostgreSQL Monitoring using modern software stacks
Showmax Engineering
 
Implementing GraphQL - Without a Backend
Implementing GraphQL - Without a BackendImplementing GraphQL - Without a Backend
Implementing GraphQL - Without a Backend
Showmax Engineering
 
2015 11-12 GoLang Meetup Praha
2015 11-12 GoLang Meetup Praha2015 11-12 GoLang Meetup Praha
2015 11-12 GoLang Meetup Praha
Showmax Engineering
 

More from Showmax Engineering (6)

PostgreSQL Terminology
PostgreSQL TerminologyPostgreSQL Terminology
PostgreSQL Terminology
 
Networking fundamentals
Networking fundamentalsNetworking fundamentals
Networking fundamentals
 
Android crash course
Android crash courseAndroid crash course
Android crash course
 
PostgreSQL Monitoring using modern software stacks
PostgreSQL Monitoring using modern software stacksPostgreSQL Monitoring using modern software stacks
PostgreSQL Monitoring using modern software stacks
 
Implementing GraphQL - Without a Backend
Implementing GraphQL - Without a BackendImplementing GraphQL - Without a Backend
Implementing GraphQL - Without a Backend
 
2015 11-12 GoLang Meetup Praha
2015 11-12 GoLang Meetup Praha2015 11-12 GoLang Meetup Praha
2015 11-12 GoLang Meetup Praha
 

Recently uploaded

Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
Roger Valdez
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
74nqk8xf
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Walaa Eldin Moustafa
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
javier ramirez
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
v7oacc3l
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
AndrzejJarynowski
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
kuntobimo2016
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
Bill641377
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Aggregage
 
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
74nqk8xf
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
g4dpvqap0
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
jitskeb
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
nuttdpt
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 

Recently uploaded (20)

Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
 
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
一比一原版(Chester毕业证书)切斯特大学毕业证如何办理
 
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
 
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series DatabaseThe Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicineIntelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
 
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
Beyond the Basics of A/B Tests: Highly Innovative Experimentation Tactics You...
 
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
 
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
一比一原版(Glasgow毕业证书)格拉斯哥大学毕业证如何办理
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
Experts live - Improving user adoption with AI
Experts live - Improving user adoption with AIExperts live - Improving user adoption with AI
Experts live - Improving user adoption with AI
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 

Deep learning features and similarity of movies based on their video content

  • 1. Deep learning features and similarity of movies based on their video content Summer Camp - Show Max - Lukáš Lopatovský
  • 2. Assignment ● Deep learning allows extracting useful features from video frames. Your task is to apply new deep learning frameworks to extract features from video frames of selected movies available in the ShowMax streaming platform. ● Goals: ● Extract deep features from video frames. Explore similar movies in the space of latent features and adjust the extraction process in order to create clusters of video assets (eg. TV episodes).
  • 3. Residual Networks ● Enable to build deeper (convolutional) neural network. (State of the art method for the image recognition.)
  • 4. Residual Networks ● To enable to build deeper network, the residual nets use the simple trick. They maintain the residuum from the previous layer ( so do not loose the previously known information )
  • 5. Torch - Efficient Tensor library (like NumPy) with an efficient CUDA backend - Neural Networks package -- build arbitrary acyclic computation graphs with automatic differentiation - fast CUDA and CPU backends - Good community and industry support - several hundred community-built and maintained packages.
  • 6. ● Torch example of ‘nn’ library
  • 7. What has been done ● The movies were classified using arbitrary number of picture frames. ● We have used already trained ImageNet FB- resnet network and own data set trained and fine-tuned networks to classify movies. ● To detect the object in the image, we have classify the whole image, as well as we have made the various crops to get more accurate predictions. (Cropping showed better results)
  • 8. Classification output ● By classification of frames in the movie, the special file is produced (.res). It is in a form to contain all the important data. It can be later post- process according to the special needs of the user: - To create Object detection .srt file. - To get various cumulative classification results. - To trace the appearance of the object at the time- line.
  • 15. Own datasets ● The network was successfully trained and fine-tuned from the ResNet network ● However, it showed some problems based from improper dataset. - Some categories contain many irrelevant pictures in second half of the search. (Special case: “The doctor House”) - The style of the images in the search is often very different to the style found in the movie. (kitchen, car) - Movies mostly contain images full of people, so the categories containing people make false positive prediction. (cinema, theater)
  • 23. Next step ● After the discussion in the company, the programs were transformed to the easily usable form. ● The feature vectors of the classification will be used to find similarities among movies. Compared to the existing algorithms and if successful, incorporated into the current recommendation system.