Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

•

2 likes•1,812 views

https://imatge.upc.edu/web/publications/diving-deep-sentiment-understanding-fine-tuned-cnns-visual-sentiment-prediction Campos V, Salvador A, Jou B, Giró-i-Nieto X. Diving Deep into Sentiment: Understanding Fine-tuned CNNs for Visual Sentiment Prediction. In: 1st International Workshop on Affect and Sentiment in Multimedia. Brisbane, Australia: ACM. 2015. Visual media are powerful means of expressing emotions and sentiments. The constant generation of new content in social networks highlights the need of automated visual sentiment analysis tools. While Convolutional Neural Networks (CNNs) have established a new state-of-the-art in several vision problems, their application to the task of sentiment analysis is mostly unexplored and there are few studies regarding how to design CNNs for this purpose. In this work, we study the suitability of fine-tuning a CNN for visual sentiment prediction as well as explore performance boosting techniques within this deep learning setting. Finally, we provide a deep-dive analysis into a benchmark, state-of-the-art network architecture to gain insight about how to design patterns for CNNs on the task of visual sentiment prediction.

Technology

DIVING DEEP INTO SENTIMENT:
UNDERSTANDING FINE-TUNED CNNS
FOR VISUAL SENTIMENT PREDICTION
Víctor Campos Xavier Giró Amaia Salvador Brendan Jou

Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
2

4
Introduction: problem definition
▷ What?
▷ How?

▷ What? Predict the sentiment that an image provokes to a human
▷ How?
5
Introduction: problem definition

▷ What? Predict the sentiment that an image provokes to a human
▷ How?
6
Introduction: problem definition

▷ What? Predict the sentiment that an image provokes to a human
▷ How? Using Convolutional Neural Networks (CNNs)
7
CNN
Introduction: problem definition

Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
10

Related work: low-level descriptors
11
Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October). Analyzing
and predicting sentiment of images on the social web. In Proceedings of the
international conference on Multimedia (pp. 715-718). ACM.
Machajdik, J., & Hanbury, A. (2010, October). Affective image classification
using features inspired by psychology and art theory. In Proceedings of the
international conference on Multimedia (pp. 83-92). ACM.

12
Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using adjective
noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM.
Related work: SentiBank

Related work: CNNs for sentiment prediction
13
You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep
networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).

Outline
1. Introduction
2. Related work
3. Methodology and results
a. Convolutional Neural Networks
b. Datasets
c. Experimental setup and results
4. Conclusions
5. Future work
14

Convolutional Neural Networks
15
Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012

Datasets
17
Flickr Twitter
Authors Borth et al. (2013) You et al. (2015)
Size ~500k 1269
Annotation method Textual tags
5 human
annotators

Datasets
18
Size
Flickr
dataset
Quality of the
annotations
Twitter
5-agree
dataset

Datasets
19
Size
Flickr
dataset
Quality of the
annotations
Twitter
5-agree
dataset

21
ARCHITECTURE
CaffeNet
Experimental setup: CNN

22
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
Experimental setup: CNN

Experimental setup: CNN
23
Pre-trained
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
[Deng’09]

Experimental setup: CNN
24
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
[Deng’09]
DATASET
[You’15]
Twitter 5-agree
+
Fine-tuning
Pre-training

Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
25

Fine-tuning CaffeNet
29
Pre-trained
model

Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
31

Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
34

Layer ablation
35
Raw ablation
2-neuron on top

Experimental setup: outline
1. Fine-tuning CaffeNet
2. Layer by layer analysis
3. Layer ablation
4. Layer addition
39

Layer addition
40
FC8: semantic
information

Layer addition
41
FC8: semantic
information

Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
42

Outline
1. Introduction
2. Related work
3. Methodology and results
4. Conclusions
5. Future work
46

Future work
47
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset

Future work
48
Size
Flickr
dataset
Quality of the
annotations
Twitter
dataset
MVSO
dataset
(†) B. Jou*, T. Chen*, N. Pappas*, M. Redi*, M. Topkara*, and S.-F. Chang. Visual Affect Around the World: A Large-scale Multilingual
Visual Sentiment Ontology. ACM Int'l Conference on Multimedia (MM), 2015.
†

49
Model
ARCHITECTURE
CaffeNet
SOFTWARE
[Jia’14]
DATASET
MVSO [Jou’15]
Future work

Acknowledgements
50
Financial supportTechnical support
Albert Gil Josep Pujal

This document describes research on using convolutional neural networks (CNNs) for visual sentiment prediction. It presents the methodology which includes fine-tuning a pre-trained CNN model on two datasets labeled for sentiment. Experiments are described that analyze the CNN layer-by-layer, ablate layers, and add new layers. The results show that fine-tuning achieves good performance and deeper layers learn sentiment-related concepts. Future work aims to expand the datasets and analyze network receptive fields.

Temporal Activity Detection in Untrimmed Videos with Recurrent Neural Networks

Universitat Politècnica de Catalunya

https://imatge-upc.github.io/activitynet-2016-cvprw/ This thesis explore different approaches using Convolutional and Recurrent Neural Networks to classify and temporally localize activities on videos, furthermore an implementation to achieve it has been proposed. As the first step, features have been extracted from video frames using an state of the art 3D Convolutional Neural Network. This features are fed in a recurrent neural network that solves the activity classification and temporally location tasks in a simple and flexible way. Different architectures and configurations have been tested in order to achieve the best performance and learning of the video dataset provided. In addition it has been studied different kind of post processing over the trained network's output to achieve a better results on the temporally localization of activities on the videos. The results provided by the neural network developed in this thesis have been submitted to the ActivityNet Challenge 2016 of the CVPR, achieving competitive results using a simple and flexible architecture.

Deep Learning for Computer Vision (3/4): Video Analytics @ laSalle 2016

Universitat Politècnica de Catalunya

Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.

Video Analysis (D4L2 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

The document provides an overview of video analysis techniques including recognition, optical flow, and object tracking. For recognition, it discusses approaches using convolutional neural networks like DeepVideo that perform classification on frames. It also covers models using optical flow as input like two-stream networks as well as 3D CNNs like C3D that directly learn spatiotemporal features. For optical flow, it summarizes FlowNet which uses a CNN to learn optical flow end-to-end. And for object tracking, it mentions deep learning methods like MDNet that train domain-specific layers to generalize across sequences.

Welcome (D1L1 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

https://telecombcn-dl.github.io/2017-dlcv/ Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.

Deep Convnets for Video Processing (Master in Computer Vision Barcelona, 2016)

Universitat Politècnica de Catalunya

Video Analysis with Convolutional Neural Networks (Master Computer Vision Bar...

Universitat Politècnica de Catalunya

1. The document discusses analyzing videos with convolutional neural networks (CNNs). It covers techniques like video recognition using CNNs to classify video content at the clip level. 2. DeepVideo and C3D are discussed as approaches for video recognition using CNNs. DeepVideo employs 2D CNNs on multiple video frames while C3D uses 3D CNNs to learn spatiotemporal features directly from video data. 3. Optical flow estimation techniques like DeepFlow are also covered, which uses a deep matching approach to compute dense correspondences between video frames for large displacements.

Closing, Course Offer 17/18 & Homework (D5 2017 UPC Deep Learning for Compute...

Universitat Politècnica de Catalunya

http://imatge-upc.github.io/telecombcn-2016-dlcv/ Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.

Advanced Deep Architectures (D2L6 Deep Learning for Speech and Language UPC 2...

Universitat Politècnica de Catalunya

https://telecombcn-dl.github.io/2017-dlsl/ Winter School on Deep Learning for Speech and Language. UPC BarcelonaTech ETSETB TelecomBCN. The aim of this course is to train students in methods of deep learning for speech and language. Recurrent Neural Networks (RNN) will be presented and analyzed in detail to understand the potential of these state of the art tools for time series processing. Engineering tips and scalability issues will be addressed to solve tasks such as machine translation, speech recognition, speech synthesis or question answering. Hands-on sessions will provide development skills so that attendees can become competent in contemporary data analytics tools.

Learning with Videos (D4L4 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

Deep Learning for Computer Vision (2/4): Object Analytics @ laSalle 2016

Universitat Politècnica de Catalunya

Multimodal Deep Learning (D4L4 Deep Learning for Speech and Language UPC 2017)

Universitat Politècnica de Catalunya

Deep Learning - Convolutional Neural Networks - Architectural Zoo

Christian Perone

This document discusses different convolutional neural network architectures including traditional architectures using convolutional, pooling, and fully connected layers, siamese networks for learning visual similarity, dense prediction networks for tasks like semantic segmentation and image colorization, video classification networks, music recommendation networks, and networks for tasks like object localization, detection, and alignment. It provides examples of specific networks that have been applied to each type of architecture.

SSII2021 [SS2] Deepfake Generation and Detection – An Overview （ディープフェイクの生成と検出）

SSII

This document provides an overview of deepfake generation and detection. It begins with an introduction to the author and their background and research interests. The rest of the document is outlined as follows: definitions of deepfakes, various deepfake generation techniques including face synthesis, manipulation, reenactment and swapping, and an overview of deepfake detection methods including commonly used datasets, image-based and video-based detection approaches.

Intro To Convolutional Neural Networks

Mark Scully

Slides from Portland Machine Learning meetup, April 13th. Abstract: You've heard all the cool tech companies are using them, but what are Convolutional Neural Networks (CNNs) good for and what is convolution anyway? For that matter, what is a Neural Network? This talk will include a look at some applications of CNNs, an explanation of how CNNs work, and what the different layers in a CNN do. There's no explicit background required so if you have no idea what a neural network is that's ok.

AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...

GeeksLab Odessa

23.05.15 Одесса. Impact Hub Odessa. Конференция AI&BigData Lab Артем Чернодуб (Computer Vision Team, ZZ Wolf) "Распознавание изображений методом Lazy Deep Learning в фото-органайзере ZZ Photo" В докладе рассматривается проблема распознавания изображений методами машинного зрения. Проводится краткий обзор существующих подзадач в этой области (детекция обьектов, классификация сцен, ассоциативный поиск в базах изображений, распознавание лиц и др.) и современных методов их решения с акцентом на глубокое обучение (Deep Learning). Подробнее: http://geekslab.co/ https://www.facebook.com/GeeksLab.co https://www.youtube.com/user/GeeksLabVideo

Neural Architectures for Still Images - Xavier Giro- UPC Barcelona 2019

Universitat Politècnica de Catalunya

https://telecombcn-dl.github.io/2019-dlcv/ Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.

Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016

Universitat Politècnica de Catalunya

Deep Learning Explained

Melanie Swan

This document summarizes Melanie Swan's presentation on deep learning. It began with defining key deep learning concepts and techniques, including neural networks, supervised vs. unsupervised learning, and convolutional neural networks. It then explained how deep learning works by using multiple processing layers to extract higher-level features from data and make predictions. Deep learning has various applications like image recognition and speech recognition. The presentation concluded by discussing how deep learning is inspired by concepts from physics and statistical mechanics.

Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

One Perceptron to Rule Them All: Language and Vision

Universitat Politècnica de Catalunya

http://ixa2.si.ehu.es/deep_learning_seminar/ Deep neural networks have boosted the convergence of multimedia data analytics in a unified framework shared by practitioners in natural language and vision. Image captioning, visual question answering or multimodal translation are some of the first applications of a new and exciting field that exploiting the generalization properties of deep neural representations. This talk will provide an overview of how vision and language problems are addressed with deep neural networks, and the exciting challenges being addressed nowadays by the research community.

Deep convnets for global recognition (Master in Computer Vision Barcelona 2016)

Universitat Politècnica de Catalunya

Lecture by Xavier Giro-i-Nieto (UPC) at the Master in Computer Vision Barcelona (March 30, 2016). http://pagines.uab.cat/mcv/ This lecture provides an overview of computer vision analysis of images at a global scale using deep learning techniques. The session is structure in two blocks: a first one addressing end to end learning, and a second one focusing on applications that use off-the-shelf features. Please submit your feedback as comments on the GDrive source slides: https://docs.google.com/presentation/d/1ms9Fczkep__9pMCjxtVr41OINMklcHWc74kwANj7KKI/edit?usp=sharing

Image classification on Imagenet (D1L4 2017 UPC Deep Learning for Computer Vi...

Universitat Politècnica de Catalunya

Deep Learning - Convolutional Neural Networks

Christian Perone

This document provides an agenda for a presentation on deep learning, neural networks, convolutional neural networks, and interesting applications. The presentation will include introductions to deep learning and how it differs from traditional machine learning by learning feature representations from data. It will cover the history of neural networks and breakthroughs that enabled training of deeper models. Convolutional neural network architectures will be overviewed, including convolutional, pooling, and dense layers. Applications like recommendation systems, natural language processing, and computer vision will also be discussed. There will be a question and answer section.

Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC Barcelona

Universitat Politècnica de Catalunya

Il deep learning ed una nuova generazione di AI - Simone Scardapane

Data Driven Innovation

Il deep learning rappresenta una nuova famiglia di tecniche data-driven, che aprono nuovi orizzonti in quello che le macchine possono essere programmate a fare. In pochi anni abbiamo visto automobili che si guidano da sole, robot che imparano a muoversi, campioni di Go sconfitti, e molto altro ancora. Quali sono le sfide tecniche, sociali e scientifiche del prossimo futuro? E, soprattutto, queste tecnologie sono alla portata di tutti? In questo talk daremo una (brevissima) panoramica di queste questioni e delle loro possibili risposte.

Andrew Ng, Chief Scientist at Baidu

Extract Data Conference

This document discusses applications of deep learning, including computer vision, speech recognition, and natural language processing. It notes that deep learning is driven by scale - large neural networks with huge amounts of labeled training data can perform complex tasks like image captioning and question answering. Deep learning techniques are being applied across many domains from mobile devices to data centers and are powering intelligent assistants with capabilities like vision and speech recognition.

Information retrieval dynamic indexing

Nadia Nahar

The document discusses different approaches to dynamic indexing in information retrieval systems. The simplest approach maintains a main index and auxiliary index, with new documents added to the auxiliary index and merged periodically into the main index. An alternative logarithmic merge approach maintains multiple indexes at increasing sizes, merging them in a logarithmic fashion. While more efficient for indexing, the logarithmic approach requires merging multiple indexes for query processing. Large search engines use dynamic indexing with both incremental changes and periodic full rebuilds of the main index.

Applying Machine Learning to Network Security Monitoring - BayThreat 2013

Alex Pinto

Video (at YouTube) - http://bit.ly/19TNSTF Big Data Security Analytics, Data Science and Machine Learning are a few of the new buzzwords that have invaded out industry of late. Most of what we hear are promises of an unicorn-laden, silver-bullet panacea by heavy-handed marketing folks, evoking an expected pushback from the most enlightened members of our community. This talk will help parse what we as a community need to know and understand about these concepts and help understand where the technical details and actual capabilities of those concepts and also where they fail and how they can be exploited and fooled by an attacker. The talk will also share results of the author's current ongoing research (on MLSec Project) of applying machine learning techniques to information secuirty monitoring.

What's hot

Deep Learning for Computer Vision: Video Analytics (UPC 2016)

Universitat Politècnica de Catalunya

Advanced Deep Architectures (D2L6 Deep Learning for Speech and Language UPC 2...

Universitat Politècnica de Catalunya

Learning with Videos (D4L4 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

Deep Learning for Computer Vision (2/4): Object Analytics @ laSalle 2016

Universitat Politècnica de Catalunya

Multimodal Deep Learning (D4L4 Deep Learning for Speech and Language UPC 2017)

Universitat Politècnica de Catalunya

Deep Learning - Convolutional Neural Networks - Architectural Zoo

Christian Perone

SSII2021 [SS2] Deepfake Generation and Detection – An Overview （ディープフェイクの生成と検出）

SSII

Intro To Convolutional Neural Networks

Mark Scully

AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...

GeeksLab Odessa

Neural Architectures for Still Images - Xavier Giro- UPC Barcelona 2019

Universitat Politècnica de Catalunya

Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016

Universitat Politècnica de Catalunya

Deep Learning Explained

Melanie Swan

Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)

Universitat Politècnica de Catalunya

One Perceptron to Rule Them All: Language and Vision

Universitat Politècnica de Catalunya

Deep convnets for global recognition (Master in Computer Vision Barcelona 2016)

Universitat Politècnica de Catalunya

Image classification on Imagenet (D1L4 2017 UPC Deep Learning for Computer Vi...

Universitat Politècnica de Catalunya

Deep Learning - Convolutional Neural Networks

Christian Perone

Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC Barcelona

Universitat Politècnica de Catalunya

Il deep learning ed una nuova generazione di AI - Simone Scardapane

Data Driven Innovation

Andrew Ng, Chief Scientist at Baidu

Extract Data Conference

What's hot (20)

Deep Learning for Computer Vision: Video Analytics (UPC 2016)

Advanced Deep Architectures (D2L6 Deep Learning for Speech and Language UPC 2...

Learning with Videos (D4L4 2017 UPC Deep Learning for Computer Vision)

Deep Learning for Computer Vision (2/4): Object Analytics @ laSalle 2016

Multimodal Deep Learning (D4L4 Deep Learning for Speech and Language UPC 2017)

Deep Learning - Convolutional Neural Networks - Architectural Zoo

SSII2021 [SS2] Deepfake Generation and Detection – An Overview （ディープフェイクの生成と検出）

Intro To Convolutional Neural Networks

AI&BigData Lab. Артем Чернодуб "Распознавание изображений методом Lazy Deep ...

Neural Architectures for Still Images - Xavier Giro- UPC Barcelona 2019

Deep Learning for Computer Vision (1/4): Image Analytics @ laSalle 2016

Deep Learning Explained

Language and Vision (D3L5 2017 UPC Deep Learning for Computer Vision)

One Perceptron to Rule Them All: Language and Vision

Deep convnets for global recognition (Master in Computer Vision Barcelona 2016)

Image classification on Imagenet (D1L4 2017 UPC Deep Learning for Computer Vi...

Deep Learning - Convolutional Neural Networks

Self-supervised Visual Learning 2020 - Xavier Giro-i-Nieto - UPC Barcelona

Il deep learning ed una nuova generazione di AI - Simone Scardapane

Andrew Ng, Chief Scientist at Baidu

Viewers also liked

Information retrieval dynamic indexing

Nadia Nahar

Applying Machine Learning to Network Security Monitoring - BayThreat 2013

Alex Pinto

The Unreasonable Benefits of Deep Learning

indico data

Dan Kuster led a talk at Sentiment Analysis Symposium discussing why businesses should consider adopting deep learning solutions. Key takeaways include simplicity, accuracy, flexibility, and some hacks for working with the tech. About the Session: Machine learning is becoming the tool of choice for analyzing text and image data. While traditional text processing solutions rely on the ability of experts to encode domain knowledge, machine learning models learn this directly from the data. Deep learning is a branch of machine learning that like the human brain quickly learns hierarchical representations of concepts, and it has been key to unlocking state-of-the-art results on a range of text and image classification tasks such as sentiment analysis and beyond. In this session, we will show the impact of a deep learning based approach over NLP and traditional machine learning based methods for text analysis across key dimensions such as accuracy, flexibility, and the amount of required training data. Specifically, we will discuss how deep learning models are now setting the records for state-of-the-art accuracy in sentiment analysis. We will also demonstrate the flexibility of this approach by showing how the features learned by one model can be easily reused in different domains (e.g., handling additional languages, or predicting new categories) to drastically reduce the time to deployment. Finally, we will touch on the ability of this method to handle additional types of data beyond text, e.g, images, for maximum insight.

Elastic Search: Beyond Ordinary Fulltext Search (Webexpo 2011 Prague)

Karel Minarik

The document discusses search and how it works under the hood. It begins with an overview of common search problems and limitations. It then demonstrates how search works by indexing documents into an inverted index of tokens and associated document references. Key steps include analyzing text by splitting, downcasing, and removing stopwords, and then storing the token postings in the index. Search queries can then be executed by looking up token postings in the index. A Ruby example class demonstrates indexing sample documents and searching the generated index.

Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...

Universitat Politècnica de Catalunya

This document provides a summary of a lecture on video analysis with recurrent neural networks (RNNs). It begins with an acknowledgment slide and outline. It then covers RNN concepts like LSTM and GRU units. Applications discussed include activity recognition from video frames and optical flow, temporal activity detection using CNNs and RNNs, and object tracking in video. The document emphasizes how RNNs allow modeling temporal dependencies in video data.

Region-oriented Convolutional Networks for Object Retrieval

Universitat Politècnica de Catalunya

This document describes research on using region-oriented convolutional neural networks for object retrieval. It discusses using local CNNs like CaffeNet, Fast R-CNN, and SDS to extract visual features from object candidates in images. These features are used to match against query descriptors. Pooled regional features are ranked to retrieve relevant shots. Fine-tuning pre-trained networks on larger datasets like COCO can improve retrieval accuracy. Combining global and local approaches through re-ranking provides an additional boost in performance.

12. Indexing and Hashing in DBMS

koolkampus

The document discusses various indexing techniques used to improve data access performance in databases, including ordered indices like B-trees and B+-trees, as well as hashing techniques. It covers the basic concepts, data structures, operations, advantages and disadvantages of each approach. B-trees and B+-trees store index entries in sorted order to support range queries efficiently, while hashing distributes entries uniformly across buckets using a hash function but does not support ranges.

Viewers also liked (7)

Information retrieval dynamic indexing

Applying Machine Learning to Network Security Monitoring - BayThreat 2013

The Unreasonable Benefits of Deep Learning

Elastic Search: Beyond Ordinary Fulltext Search (Webexpo 2011 Prague)

Video Analysis with Recurrent Neural Networks (Master Computer Vision Barcelo...

Region-oriented Convolutional Networks for Object Retrieval

12. Indexing and Hashing in DBMS

Similar to Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

323462348

DrShubhangiDnyaneshw

This document is a 36-page bachelor's thesis written by Duc Minh Luong Nguyen titled "Detect COVID-19 from Chest X-Ray images using Deep Learning". The thesis was submitted to Metropolia University of Applied Sciences in May 2020. It aims to build a deep convolutional neural network to detect COVID-19 using only chest X-ray images. The model achieves an accuracy of 93% at detecting COVID-19 patients versus healthy patients, despite being trained on a small dataset of 115 images for each class.

323462348

Shubhangi Kirange

Age and Gender Classification using Convolutional Neural Network

IRJET Journal

This document describes a study that aims to accurately identify the gender and age range of facial images using convolutional neural networks. It begins with an introduction to age and gender classification and some of the challenges. It then discusses the system analysis and design, including the use of convolutional neural networks and machine learning techniques. The implementation section notes that Python, TensorFlow, Keras and OpenCV will be used to build a convolutional neural network model to detect faces in images and predict age and gender through training on available datasets. The overall goal is to develop an accurate system for age and gender detection from facial images.

Dissertation final report

Smriti Tikoo

This document is a dissertation submitted by Smriti Tikoo for the fulfillment of requirements for a Master's degree in Electronics and Communication Engineering. The dissertation focuses on facial detection using the Viola-Jones algorithm and facial recognition using a Backpropagation Neural Network. The document begins with an introduction that discusses the history and importance of facial recognition. It then covers topics like facial detection techniques, neural networks, and the proposed methodology which involves Viola-Jones for detection and a Backpropagation Neural Network for recognition. The document is organized into chapters that discuss the literature review, proposed methodology, software implementation, results and discussion, and conclusions.

Open-ended Visual Question-Answering

Universitat Politècnica de Catalunya

http://imatge-upc.github.io/vqa-2016-cvprw/ This thesis studies methods to solve Visual Question-Answering (VQA) tasks with a Deep Learning framework.As a preliminary step, we explore Long Short-Term Memory (LSTM) networks used in Natural Language Processing (NLP) to tackle Question-Answering (text based). We then modify the previous model to accept an image as an input in addition to the question. For this purpose, we explore the VGG-16 and K-CNN convolutional neural networks to extract visual features from the image. These are merged with the word embedding or with a sentence embedding of the question to predict the answer. This work was successfully submitted to the Visual Question Answering Challenge 2016, where it achieved a 53,62\% of accuracy in the test dataset. The developed software has followed the best programming practices and Python code style, providing a consistent baseline in Keras for different configurations.

Introduction to Interpretable Machine Learning

Nguyen Giang

This document discusses interpretable machine learning and explainable AI. It begins with definitions of key terms and an overview of interpretable methods. Deep learning models are often treated as "black boxes" that are difficult to interpret. Interpretability can be achieved by using inherently interpretable models like linear models or decision trees, adding attention mechanisms, or interpreting models before, during or after building them. Later sections discuss specific interpretable techniques like understanding data through examples, MMD-Critic for learning prototypes and criticisms, and visualizing convolutional neural networks to understand predictions. The document emphasizes the importance of interpretability and explains several approaches to make machine learning models more transparent to humans.

Recent Advancements in the Field of Deepfake Detection

AIRCC Publishing Corporation

A deepfake is a photo or video of a person whose image has been digitally altered or partially replaced with an image of someone else. Deepfakes have the potential to cause a variety of problems and are often used maliciously. A common usage is altering videos of prominent political figures and celebrities. These deepfakes can portray them making offensive, problematic, and/or untrue statements. Current deepfakes can be very realistic, and when used in this way, can spread panic and even influence elections and political opinions. There are many deepfake detection strategies currently in use but finding the most comprehensive and universal method is critical. So, in this survey we will address the problems of malicious deepfake creation and the lack of universal deepfake detection methods. Our objective is to survey and analyze a variety of current methods and advances in the field of deepfake detection.

Recent Advancements in the Field of Deepfake Detection

AIRCC Publishing Corporation

2015 sibgrapi-contactlenses

David Menotti

The document discusses using convolutional neural networks to detect iris contact lenses in images. Specifically, it aims to discriminate between textured contact lenses, soft transparent contact lenses, and non-contact lens irises. The document provides background on convolutional networks and iris biometrics. It then describes the objectives of detecting contact lenses, databases and architectures used, and presents results of the experiments.

Ijarcce 27

Smriti Tikoo

Face detection and recognition has been prevalent with research scholars and diverse approaches have been incorporated till date to serve purpose. The rampant advent of biometric analysis systems, which may be full body scanners, or iris detection and recognition systems and the finger print recognition systems, and surveillance systems deployed for safety and security purposes have contributed to inclination towards same. Advances has been made with frontal view, lateral view of the face or using facial expressions such as anger, happiness and gloominess, still images and video image to be used for detection and recognition. This led to newer methods for face detection and recognition to be introduced in achieving accurate results and economically feasible and extremely secure. Techniques such as Principal Component analysis (PCA), Independent component analysis (ICA), Linear Discriminant Analysis (LDA), have been the predominant ones to be used. But with improvements needed in the previous approaches Neural Networks based recognition was like boon to the industry. It not only enhanced the recognition but also the efficiency of the process. Choosing Backpropagation as the learning method was clearly out of its efficiency to recognize non linear faces with an acceptance ratio of more than 90% and execution time of only few seconds.

Neural nets jeff_shomaker_7-6-16_

Jeffrey Shomaker

Jeff Shomaker gave a presentation on neural network software and frameworks. He discussed what neural networks are, including that they are loosely based on neurons in the brain. He then covered several uses of neural networks in medicine, finance, security and energy efficiency. The presentation provided overviews of popular neural network frameworks including TensorFlow, Torch, CNTK and Caffe, describing what each is, capabilities, and including video demonstrations. Contact information was provided for further discussion.

Machine Learning.pptx

RehmatUllah46

The thesis discusses detecting facial expressions and key facial coordinates using deep learning techniques. The objectives are to detect key facial coordinates and recognize seven universal facial expressions with high accuracy. The methodology uses the FER2013 dataset and a key facial points dataset, applying preprocessing like resizing and data augmentation. A residual network model is trained on this data, achieving accuracy scores that outperform previous state-of-the-art models in recognizing both facial expressions and key coordinates.

Age and gender detection using deep learning

IRJET Journal

This document summarizes a research paper on age and gender detection using deep learning. It discusses using convolutional neural networks (CNNs) to extract features from images of faces and classify them by age and gender. The methodology section describes using a CNN model trained on labeled image data to learn the features that distinguish ages and genders. It then discusses preprocessing images, detecting faces, extracting features using CNN layers, and classifying images into age and gender categories. The output section notes that testing the trained model on new images in real-time can classify faces by gender and assign them to an age group with results displayed on screen.

Reproducibility in human cognitive neuroimaging: a community-driven data sha...

Nolan Nichols

The document summarizes Nolan Nichols' dissertation defense on a community-driven data sharing framework for integrating and interoperating neuroimaging provenance information. His research aimed to enhance the reusability of neuroimaging data and workflows by advancing data exchange standards that incorporate provenance. Through two phases involving multiple collaborations, he extended existing standards and developed neuroimaging data models and web services to compute and discover provenance from brain imaging workflows in order to improve reproducibility in cognitive neuroimaging research.

Parkinson’s Disease Detection Using Transfer Learning

IRJET Journal

This document presents a study on detecting Parkinson's disease using transfer learning on sketch images. The researchers collected spiral and wave sketch images from healthy and Parkinson's patients. They used pre-trained Inception v3 and ResNet50 models with transfer learning to classify the sketches. Inception v3 achieved 92.83% accuracy in distinguishing between healthy and Parkinson's patient sketches. The study demonstrates the potential of using deep learning and transfer learning on motor symptoms like sketching to help detect Parkinson's disease early. Future work could improve the methodology and datasets to better identify the disease.

FaceDetectionforColorImageBasedonMATLAB.pdf

Anita Pal

The document proposes a method for face detection in color images using MATLAB and a convolutional neural network (CNN) with two layers. It begins with an introduction to face detection technologies and challenges. It then describes the proposed methodology, which involves reading a color image, applying CNN convolutions to detect faces, and fusing the results. Mathematical aspects of CNNs are discussed, including equations for convolutional operations and dimensions. The CNN method combines feature extraction, segmentation, and classification. Finally, the document summarizes how the proposed algorithm was designed in MATLAB to identify faces using a simple learning algorithm and CNNs with fewer layers than traditional networks. The goal is to develop a fast face detection program for color images based on new deep learning standards.

Ijsrdv8 i10424

aissmsblogs

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Yoonho Lee

Introduction to Soft Computing

Aakash Kumar

Soft computing is an approach to building computationally intelligent systems that is tolerant to imprecision and uncertainty. It includes techniques like fuzzy logic, neural networks, and evolutionary computation. These techniques were developed to mimic human-like intelligence by accommodating imprecision and exploiting uncertainty. Soft computing is used to build intelligent systems that can learn and adapt to new environments. Neuro-fuzzy systems combine neural networks and fuzzy logic to create adaptive and knowledge-based intelligent systems.

Data visualisations: drawing actionable insights from science and technology ...

EFSA EU

Similar to Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction (20)

323462348

Age and Gender Classification using Convolutional Neural Network

Dissertation final report

Open-ended Visual Question-Answering

Introduction to Interpretable Machine Learning

Recent Advancements in the Field of Deepfake Detection

2015 sibgrapi-contactlenses

Ijarcce 27

Neural nets jeff_shomaker_7-6-16_

Machine Learning.pptx

Age and gender detection using deep learning

Reproducibility in human cognitive neuroimaging: a community-driven data sha...

Parkinson’s Disease Detection Using Transfer Learning

FaceDetectionforColorImageBasedonMATLAB.pdf

Ijsrdv8 i10424

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Introduction to Soft Computing

Data visualisations: drawing actionable insights from science and technology ...

More from Universitat Politècnica de Catalunya

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

Universitat Politècnica de Catalunya

This document provides an overview of deep generative learning and summarizes several key generative models including GANs, VAEs, diffusion models, and autoregressive models. It discusses the motivation for generative models and their applications such as image generation, text-to-image synthesis, and enhancing other media like video and speech. Example state-of-the-art models are provided for each application. The document also covers important concepts like the difference between discriminative and generative modeling, sampling techniques, and the training procedures for GANs and VAEs.

Deep Generative Learning for All

Universitat Politècnica de Catalunya

The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...

Universitat Politècnica de Catalunya

The document discusses the Vision Transformer (ViT) model for computer vision tasks. It covers: 1. How ViT tokenizes images into patches and uses position embeddings to encode spatial relationships. 2. ViT uses a class embedding to trigger class predictions, unlike CNNs which have decoders. 3. The receptive field of ViT grows as the attention mechanism allows elements to attend to other distant elements in later layers. 4. Initial results showed ViT performance was comparable to CNNs when trained on large datasets but lagged CNNs trained on smaller datasets like ImageNet.

Towards Sign Language Translation & Production | Xavier Giro-i-Nieto

Universitat Politècnica de Catalunya

Machine translation and computer vision have greatly benefited from the advances in deep learning. A large and diverse amount of textual and visual data have been used to train neural networks whether in a supervised or self-supervised manner. Nevertheless, the convergence of the two fields in sign language translation and production still poses multiple open challenges, like the low video resources, limitations in hand pose estimation, or 3D spatial grounding from poses.

The Transformer - Xavier Giró - UPC Barcelona 2021

Universitat Politècnica de Catalunya

Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...

Universitat Politècnica de Catalunya

Open challenges in sign language translation and production

Universitat Politècnica de Catalunya

Machine translation and computer vision have greatly benefited of the advances in deep learning. The large and diverse amount of textual and visual data have been used to train neural networks whether in a supervised or self-supervised manner. Nevertheless, the convergence of the two field in sign language translation and production is still poses multiple open challenges, like the low video resources, limitations in hand pose estimation, or 3D spatial grounding from poses. This talk will present these challenges and the How2✌️Sign dataset (https://how2sign.github.io) recorded at CMU in collaboration with UPC, BSC, Gallaudet University and Facebook. https://imatge.upc.edu/web/publications/sign-language-translation-and-production-multimedia-and-multimodal-challenges-all

Generation of Synthetic Referring Expressions for Object Segmentation in Videos

Universitat Politècnica de Catalunya

https://imatge-upc.github.io/synthref/ Integrating computer vision with natural language processing has achieved significant progress over the last years owing to the continuous evolution of deep learning. A novel vision and language task, which is tackled in the present Master thesis is referring video object segmentation, in which a language query defines which instance to segment from a video sequence. One of the biggest challenges for this task is the lack of relatively large annotated datasets since a tremendous amount of time and human effort is required for annotation. Moreover, existing datasets suffer from poor quality annotations in the sense that approximately one out of ten language expressions fails to uniquely describe the target object. The purpose of the present Master thesis is to address these challenges by proposing a novel method for generating synthetic referring expressions for an image (video frame). This method pro- duces synthetic referring expressions by using only the ground-truth annotations of the objects as well as their attributes, which are detected by a state-of-the-art object detection deep neural network. One of the advantages of the proposed method is that its formulation allows its application to any object detection or segmentation dataset. By using the proposed method, the first large-scale dataset with synthetic referring expressions for video object segmentation is created, based on an existing large benchmark dataset for video instance segmentation. A statistical analysis and comparison of the created synthetic dataset with existing ones is also provided in the present Master thesis. The conducted experiments on three different datasets used for referring video object segmentation prove the efficiency of the generated synthetic data. More specifically, the obtained results demonstrate that by pre-training a deep neural network with the proposed synthetic dataset one can improve the ability of the network to generalize across different datasets, without any additional annotation cost. This outcome is even more important taking into account that no additional annotation cost is involved.

Discovery and Learning of Navigation Goals from Pixels in Minecraft

Universitat Politècnica de Catalunya

Master MATT thesis defense by Juan José Nieto Advised by Víctor Campos and Xavier Giro-i-Nieto. 27th May 2021. Pre-training Reinforcement Learning (RL) agents in a task-agnostic manner has shown promising results. However, previous works still struggle to learn and discover meaningful skills in high-dimensional state-spaces. We approach the problem by leveraging unsupervised skill discovery and self-supervised learning of state representations. In our work, we learn a compact latent representation by making use of variational or contrastive techniques. We demonstrate that both allow learning a set of basic navigation skills by maximizing an information theoretic objective. We assess our method in Minecraft 3D maps with different complexities. Our results show that representations and conditioned policies learned from pixels are enough for toy examples, but do not scale to realistic and complex maps. We also explore alternative rewards and input observations to overcome these limitations. https://imatge.upc.edu/web/publications/discovery-and-learning-navigation-goals-pixels-minecraft

Learn2Sign : Sign language recognition and translation using human keypoint e...

Universitat Politècnica de Catalunya

Peter Muschick MSc thesis Universitat Pollitecnica de Catalunya, 2020 Sign language recognition and translation has been an active research field in the recent years with most approaches using deep neural networks to extract information from sign language data. This work investigates the mostly disregarded approach of using human keypoint estimation from image and video data with OpenPose in combination with transformer network architecture. Firstly, it was shown that it is possible to recognize individual signs (4.5% word error rate (WER)). Continuous sign language recognition though was more error prone (77.3% WER) and sign language translation was not possible using the proposed methods, which might be due to low accuracy scores of human keypoint estimation by OpenPose and accompanying loss of information or insufficient capacities of the used transformer model. Results may improve with the use of datasets containing higher repetition rates of individual signs or focusing more precisely on keypoint extraction of hands.

Intepretability / Explainable AI for Deep Neural Networks

Universitat Politècnica de Catalunya

This document discusses interpretability and explainable AI (XAI) in neural networks. It begins by providing motivation for why explanations of neural network predictions are often required. It then provides an overview of different interpretability techniques, including visualizing learned weights and feature maps, attribution methods like class activation maps and guided backpropagation, and feature visualization. Specific examples and applications of each technique are described. The document serves as a guide to interpretability and explainability in deep learning models.

Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020

Universitat Politècnica de Catalunya

Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.

Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...

Universitat Politècnica de Catalunya

Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020

Universitat Politècnica de Catalunya

Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...

Universitat Politècnica de Catalunya

https://telecombcn-dl.github.io/dlai-2020/ Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks or Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles of deep learning from both an algorithmic and computational perspectives.

Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020

Universitat Politècnica de Catalunya

https://telecombcn-dl.github.io/drl-2020/ This course presents the principles of reinforcement learning as an artificial intelligence tool based on the interaction of the machine with its environment, with applications to control tasks (eg. robotics, autonomous driving) o decision making (eg. resource optimization in wireless communication networks). It also advances in the development of deep neural networks trained with little or no supervision, both for discriminative and generative tasks, with special attention on multimedia applications (vision, language and speech).

Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)

Universitat Politècnica de Catalunya

Giro-i-Nieto, X. One Perceptron to Rule Them All: Language, Vision, Audio and Speech. In Proceedings of the 2020 International Conference on Multimedia Retrieval (pp. 7-8). Tutorial page: https://imatge.upc.edu/web/publications/one-perceptron-rule-them-all-language-vision-audio-and-speech-tutorial Deep neural networks have boosted the convergence of multimedia data analytics in a unified framework shared by practitioners in natural language, vision and speech. Image captioning, lip reading or video sonorization are some of the first applications of a new and exciting field of research exploiting the generalization properties of deep neural representation. This tutorial will firstly review the basic neural architectures to encode and decode vision, text and audio, to later review the those models that have successfully translated information across modalities.

Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...

Universitat Politècnica de Catalunya

This document summarizes image segmentation techniques using deep learning. It begins with an overview of semantic segmentation and instance segmentation. It then discusses several techniques for semantic segmentation, including deconvolution/transposed convolution for learnable upsampling, skip connections to combine predictions from different CNN depths, and dilated convolutions to increase the receptive field without losing resolution. For instance segmentation, it covers proposal-based methods like Mask R-CNN, and single-shot and recurrent approaches as alternatives to proposal-based models.

Curriculum Learning for Recurrent Video Object Segmentation

Universitat Politècnica de Catalunya

https://imatge-upc.github.io/rvos-mots/ Video object segmentation can be understood as a sequence-to-sequence task that can benefit from the curriculum learning strategies for better and faster training of deep neural networks. This work explores different schedule sampling and frame skipping variations to significantly improve the performance of a recurrent architecture. Our results on the car class of the KITTI-MOTS challenge indicate that, surprisingly, an inverse schedule sampling is a better option than a classic forward one. Also, that a progressive skipping of frames during training is beneficial, but only when training with the ground truth masks instead of the predicted ones.

Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020

Universitat Politècnica de Catalunya

Deep neural networks have achieved outstanding results in various applications such as vision, language, audio, speech, or reinforcement learning. These powerful function approximators typically require large amounts of data to be trained, which poses a challenge in the usual case where little labeled data is available. During the last year, multiple solutions have been proposed to leverage this problem, based on the concept of self-supervised learning, which can be understood as a specific case of unsupervised learning. This talk will cover its basic principles and provide examples in the field of multimedia.

More from Universitat Politècnica de Catalunya (20)

Deep Generative Learning for All - The Gen AI Hype (Spring 2024)

Deep Generative Learning for All

The Transformer in Vision | Xavier Giro | Master in Computer Vision Barcelona...

Towards Sign Language Translation & Production | Xavier Giro-i-Nieto

The Transformer - Xavier Giró - UPC Barcelona 2021

Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...

Open challenges in sign language translation and production

Generation of Synthetic Referring Expressions for Object Segmentation in Videos

Discovery and Learning of Navigation Goals from Pixels in Minecraft

Learn2Sign : Sign language recognition and translation using human keypoint e...

Intepretability / Explainable AI for Deep Neural Networks

Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020

Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...

Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020

Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...

Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020

Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)

Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...

Curriculum Learning for Recurrent Video Object Segmentation

Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020

Recently uploaded

PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx

christinelarrosa

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

DianaGray10

Join us to learn how UiPath Apps can directly and easily interact with prebuilt connectors via Integration Service--including Salesforce, ServiceNow, Open GenAI, and more. The best part is you can achieve this without building a custom workflow! Say goodbye to the hassle of using separate automations to call APIs. By seamlessly integrating within App Studio, you can now easily streamline your workflow, while gaining direct access to our Connector Catalog of popular applications. We’ll discuss and demo the benefits of UiPath Apps and connectors including: Creating a compelling user experience for any software, without the limitations of APIs. Accelerating the app creation process, saving time and effort Enjoying high-performance CRUD (create, read, update, delete) operations, for seamless data management. Speakers: Russell Alfeche, Technology Leader, RPA at qBotic and UiPath MVP Charlie Greenberg, host

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Fwdays

This is a session that details how PostgreSQL's features and Azure AI Services can be effectively used to significantly enhance the search functionality in any application. In this session, we'll share insights on how we used PostgreSQL to facilitate precise searches across multiple fields in our mobile application. The techniques include using LIKE and ILIKE operators and integrating a trigram-based search to handle potential misspellings, thereby increasing the search accuracy. We'll also discuss how the azure_ai extension on PostgreSQL databases in Azure and Azure AI Services were utilized to create vectors from user input, a feature beneficial when users wish to find specific items based on text prompts. While our application's case study involves a drug search, the techniques and principles shared in this session can be adapted to improve search functionality in a wide range of applications. Join us to learn how PostgreSQL and Azure AI can be harnessed to enhance your application's search capability.

Principle of conventional tomography-Bibash Shahi ppt..pptx

BibashShahi

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

Chart Kalyan

What is an RPA CoE? Session 2 – CoE Roles

DianaGray10

Christine's Product Research Presentation.pptx

christinelarrosa

"Scaling RAG Applications to serve millions of users", Kevin Goedecke

Fwdays

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Jason Yip

The typical problem in product engineering is not bad strategy, so much as “no strategy”. This leads to confusion, lack of motivation, and incoherent action. The next time you look for a strategy and find an empty space, instead of waiting for it to be filled, I will show you how to fill it in yourself. If you’re wrong, it forces a correction. If you’re right, it helps create focus. I’ll share how I’ve approached this in the past, both what works and lessons for what didn’t work so well.

Astute Business Solutions | Oracle Cloud Partner |

AstuteBusiness

Apps Break Data

Ivo Velitchkov

How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?

Dandelion Hashtable: beyond billion requests per second on a commodity server

Antonios Katsarakis

This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).

What is an RPA CoE? Session 1 – CoE Vision

DianaGray10

Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...

Pitangent Analytics & Technology Solutions Pvt. Ltd

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

LizaNolte

HERE IS YOUR WEBINAR CONTENT! 'Mastering Customer Journey Management with Dr. Graham Hill'. We hope you find the webinar recording both insightful and enjoyable. In this webinar, we explored essential aspects of Customer Journey Management and personalization. Here’s a summary of the key insights and topics discussed: Key Takeaways: Understanding the Customer Journey: Dr. Hill emphasized the importance of mapping and understanding the complete customer journey to identify touchpoints and opportunities for improvement. Personalization Strategies: We discussed how to leverage data and insights to create personalized experiences that resonate with customers. Technology Integration: Insights were shared on how inQuba’s advanced technology can streamline customer interactions and drive operational efficiency.

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

Neo4j

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

Alex Pruden

Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security. Paper Link: https://eprint.iacr.org/2024/257

GNSS spoofing via SDR (Criptored Talks 2024)

Javier Junquera

In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security. This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing. The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

zjhamm304

Essentials of Automations: Exploring Attributes & Automation Parameters

Safe Software

Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they? Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality. You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.

Recently uploaded (20)

PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx

Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors

"NATO Hackathon Winner: AI-Powered Drug Search", Taras Kloba

Principle of conventional tomography-Bibash Shahi ppt..pptx

How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf

What is an RPA CoE? Session 2 – CoE Roles

Christine's Product Research Presentation.pptx

"Scaling RAG Applications to serve millions of users", Kevin Goedecke

[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...

Astute Business Solutions | Oracle Cloud Partner |

Apps Break Data

Dandelion Hashtable: beyond billion requests per second on a commodity server

What is an RPA CoE? Session 1 – CoE Vision

Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...

inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill

Harnessing the Power of NLP and Knowledge Graphs for Opioid Research

zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...

GNSS spoofing via SDR (Criptored Talks 2024)

QA or the Highway - Component Testing: Bridging the gap between frontend appl...

Essentials of Automations: Exploring Attributes & Automation Parameters

Diving deep into sentiment: Understanding fine-tuned CNNs for visual sentiment prediction

1. DIVING DEEP INTO SENTIMENT: UNDERSTANDING FINE-TUNED CNNS FOR VISUAL SENTIMENT PREDICTION Víctor Campos Xavier Giró Amaia Salvador Brendan Jou

2. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 2

3. Introduction: motivation 3

4. 4 Introduction: problem definition ▷ What? ▷ How?

5. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? 5 Introduction: problem definition

6. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? 6 Introduction: problem definition

7. ▷ What? Predict the sentiment that an image provokes to a human ▷ How? Using Convolutional Neural Networks (CNNs) 7 CNN Introduction: problem definition

8. 8 CNN Introduction: example

9. 9 CNN Introduction: example

10. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 10

11. Related work: low-level descriptors 11 Siersdorfer, S., Minack, E., Deng, F., & Hare, J. (2010, October). Analyzing and predicting sentiment of images on the social web. In Proceedings of the international conference on Multimedia (pp. 715-718). ACM. Machajdik, J., & Hanbury, A. (2010, October). Affective image classification using features inspired by psychology and art theory. In Proceedings of the international conference on Multimedia (pp. 83-92). ACM.

12. 12 Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S. F. (2013, October). Large-scale visual sentiment ontology and detectors using adjective noun pairs. In Proceedings of the 21st ACM international conference on Multimedia (pp. 223-232). ACM. Related work: SentiBank

13. Related work: CNNs for sentiment prediction 13 You, Q., Luo, J., Jin, H., & Yang, J. (2015). Robust image sentiment analysis using progressively trained and domain transferred deep networks. In The Twenty-Ninth AAAI Conference on Artificial Intelligence (AAAI).

14. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 14

15. Convolutional Neural Networks 15 Krizhevsky, A.; Sutskever, I. & Hinton, G. E.: ImageNet Classification with Deep Convolutional Neural Networks. In: NIPS., 2012

16. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 16

17. Datasets 17 Flickr Twitter Authors Borth et al. (2013) You et al. (2015) Size ~500k 1269 Annotation method Textual tags 5 human annotators

18. Datasets 18 Size Flickr dataset Quality of the annotations Twitter 5-agree dataset

19. Datasets 19 Size Flickr dataset Quality of the annotations Twitter 5-agree dataset

20. Outline 1. Introduction 2. Related work 3. Methodology and results a. Convolutional Neural Networks b. Datasets c. Experimental setup and results 4. Conclusions 5. Future work 20

21. 21 ARCHITECTURE CaffeNet Experimental setup: CNN

22. 22 ARCHITECTURE CaffeNet SOFTWARE [Jia’14] Experimental setup: CNN

23. Experimental setup: CNN 23 Pre-trained Model ARCHITECTURE CaffeNet SOFTWARE [Jia’14] DATASET [Deng’09]

24. Experimental setup: CNN 24 Model ARCHITECTURE CaffeNet SOFTWARE [Jia’14] DATASET [Deng’09] DATASET [You’15] Twitter 5-agree + Fine-tuning Pre-training

25. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 25

26. Fine-tuning CaffeNet 26

27. Fine-tuning CaffeNet 27

28. Fine-tuning CaffeNet 28

29. Fine-tuning CaffeNet 29 Pre-trained model

30. Fine-tuning CaffeNet 30

31. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 31

32. Layer by layer analysis 32

33. Layer by layer analysis 33

34. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 34

35. Layer ablation 35 Raw ablation 2-neuron on top

36. Layer ablation 36

37. Layer ablation 37

38. Layer ablation 38 ~16M params (~25%)

39. Experimental setup: outline 1. Fine-tuning CaffeNet 2. Layer by layer analysis 3. Layer ablation 4. Layer addition 39

40. Layer addition 40 FC8: semantic information

41. Layer addition 41 FC8: semantic information

42. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 42

43. Conclusions 43 Pre-trained model

44. 44 CNN Conclusions

45. Conclusions 45

46. Outline 1. Introduction 2. Related work 3. Methodology and results 4. Conclusions 5. Future work 46

47. Future work 47 Size Flickr dataset Quality of the annotations Twitter dataset

48. Future work 48 Size Flickr dataset Quality of the annotations Twitter dataset MVSO dataset (†) B. Jou*, T. Chen*, N. Pappas*, M. Redi*, M. Topkara*, and S.-F. Chang. Visual Affect Around the World: A Large-scale Multilingual Visual Sentiment Ontology. ACM Int'l Conference on Multimedia (MM), 2015. †

49. 49 Model ARCHITECTURE CaffeNet SOFTWARE [Jia’14] DATASET MVSO [Jou’15] Future work

50. Acknowledgements 50 Financial supportTechnical support Albert Gil Josep Pujal

51.

52.

53. Data augmentation (oversampling) 53 CNN

54. Data augmentation (oversampling) 54 CNN

55. Data augmentation (oversampling) 55 CNN

56. Data augmentation (oversampling) 56 CNN

57. Data augmentation (oversampling) 57 CNN

58. Data augmentation (oversampling) 58 CNN

59. Data augmentation (oversampling) 59 CNN