Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
•
0 likes•79 views
This document summarizes a paper on Bootstrap Your Own Latent (BYOL), an unsupervised contrastive learning method that does not use negative pairs. BYOL trains a target network to predict the output of an online network using a different data augmentation. The loss is the mean squared error between the predictions. BYOL achieves state-of-the-art performance on several image classification benchmarks without negative pairs by bootstrapping representations from its own augmented views. Ablation studies show BYOL is robust to different augmentations and batch sizes but requires careful tuning of the target network update rate.
Report
Share
Report
Share
Download to read offline
Recommended
AlexNet and so on...
AlexNet 2012 Contributions and Structure with Simple keras Code.
Multi GPU, Pooling, Data Augmentation...
PR-305: Exploring Simple Siamese Representation Learning
SimSiam is a self-supervised learning method that uses a Siamese network with stop-gradient to learn representations from unlabeled data. The paper finds that stop-gradient plays an essential role in preventing the model from collapsing to a degenerate solution. Additionally, it is hypothesized that SimSiam implicitly optimizes an Expectation-Maximization-like algorithm that alternates between updating the network parameters and assigning representations to samples in a manner analogous to k-means clustering.
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021). Exploring Simple Siamese Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15750-15758).
YOLOv4: optimal speed and accuracy of object detection review
YOLOv4 builds upon previous YOLO models and introduces techniques like CSPDarknet53, SPP, PAN, Mosaic data augmentation, and modifications to existing methods to achieve state-of-the-art object detection speed and accuracy while being trainable on a single GPU. Experiments show that combining these techniques through a "bag of freebies" and "bag of specials" approach improves classifier and detector performance over baselines on standard datasets. The paper contributes an efficient object detection model suitable for production use with limited resources.
Neural network learning ability
This guides you through how a neural network learns a new task without forgetting the old learned task.
Lec3 dqn
This document summarizes Deep Q-Networks (DQN), a deep reinforcement learning algorithm that was able to achieve human-level performance on many Atari 2600 games. The key ideas of DQN include using a deep neural network to approximate the Q-function, experience replay to increase data efficiency, and a separate target network to stabilize learning. DQN has inspired many follow up algorithms, including double DQN, dueling DQN, prioritized experience replay, and noisy networks for better exploration. DQN was able to learn human-level policies directly from pixels and rewards for many Atari games using the same hyperparameters and network architecture.
Tianqi Chen, PhD Student, University of Washington, at MLconf Seattle 2017
Tianqi holds a bachelor’s degree in Computer Science from Shanghai Jiao Tong University, where he was a member of ACM Class, now part of Zhiyuan College in SJTU. He did his master’s degree at Changhai Jiao Tong University in China on Apex Data and Knowledge Management before joining the University of Washington as a PhD. He has had several prestigious internships and has been a visiting scholar including: Google on the Brain Team, at Graphlab authoring the boosted tree and neural net toolkit, at Microsoft Research Asia in the Machine Learning Group, and the Digital Enterprise Institute in Galway Ireland. What really excites Tianqi is what processes and goals can be enabled when we bring advanced learning techniques and systems together. He pushes the envelope on deep learning, knowledge transfer and lifelong learning. His PhD is supported by a Google PhD Fellowship.
Abstract summary
Build Scalable and Modular Learning Systems:
Machine learning and data-driven approaches are becoming very important in many areas. There are one factors that drive these successful applications: scalable learning systems that learn the model of interest from large datasets. More importantly, the system needed to be designed in a modular way to work with existing ecosystem and improve users’ productivity environment. In this talk, I will talk about XGBoost and MXNet, two learning scalable and portable systems that I build. I will discuss how we can apply distributed computing, asynchronous scheduling and hardware acceleration to improve these systems, as well as how do they fit into bigger open-source ecosystem of machine learning.
Neural_Programmer_Interpreter
Neural Programmer-Interpreters (NPI) is a neural network model that can learn programs from small amounts of training data and generalize to new situations. It has a core LSTM module that acts as a router between learned programs. In experiments, NPI outperformed sequence-to-sequence models in data efficiency and generalization when learning tasks like adding numbers and sorting arrays. It was also able to learn new programs like finding the maximum value by composing existing programs, without forgetting previous tasks. The authors conclude that NPI is better able to exploit compositional structure in tasks compared to sequence models.
Recommended
AlexNet and so on...
AlexNet 2012 Contributions and Structure with Simple keras Code.
Multi GPU, Pooling, Data Augmentation...
PR-305: Exploring Simple Siamese Representation Learning
SimSiam is a self-supervised learning method that uses a Siamese network with stop-gradient to learn representations from unlabeled data. The paper finds that stop-gradient plays an essential role in preventing the model from collapsing to a degenerate solution. Additionally, it is hypothesized that SimSiam implicitly optimizes an Expectation-Maximization-like algorithm that alternates between updating the network parameters and assigning representations to samples in a manner analogous to k-means clustering.
Exploring Simple Siamese Representation Learning
Chen, X., & He, K. (2021). Exploring Simple Siamese Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 15750-15758).
YOLOv4: optimal speed and accuracy of object detection review
YOLOv4 builds upon previous YOLO models and introduces techniques like CSPDarknet53, SPP, PAN, Mosaic data augmentation, and modifications to existing methods to achieve state-of-the-art object detection speed and accuracy while being trainable on a single GPU. Experiments show that combining these techniques through a "bag of freebies" and "bag of specials" approach improves classifier and detector performance over baselines on standard datasets. The paper contributes an efficient object detection model suitable for production use with limited resources.
Neural network learning ability
This guides you through how a neural network learns a new task without forgetting the old learned task.
Lec3 dqn
This document summarizes Deep Q-Networks (DQN), a deep reinforcement learning algorithm that was able to achieve human-level performance on many Atari 2600 games. The key ideas of DQN include using a deep neural network to approximate the Q-function, experience replay to increase data efficiency, and a separate target network to stabilize learning. DQN has inspired many follow up algorithms, including double DQN, dueling DQN, prioritized experience replay, and noisy networks for better exploration. DQN was able to learn human-level policies directly from pixels and rewards for many Atari games using the same hyperparameters and network architecture.
Tianqi Chen, PhD Student, University of Washington, at MLconf Seattle 2017
Tianqi holds a bachelor’s degree in Computer Science from Shanghai Jiao Tong University, where he was a member of ACM Class, now part of Zhiyuan College in SJTU. He did his master’s degree at Changhai Jiao Tong University in China on Apex Data and Knowledge Management before joining the University of Washington as a PhD. He has had several prestigious internships and has been a visiting scholar including: Google on the Brain Team, at Graphlab authoring the boosted tree and neural net toolkit, at Microsoft Research Asia in the Machine Learning Group, and the Digital Enterprise Institute in Galway Ireland. What really excites Tianqi is what processes and goals can be enabled when we bring advanced learning techniques and systems together. He pushes the envelope on deep learning, knowledge transfer and lifelong learning. His PhD is supported by a Google PhD Fellowship.
Abstract summary
Build Scalable and Modular Learning Systems:
Machine learning and data-driven approaches are becoming very important in many areas. There are one factors that drive these successful applications: scalable learning systems that learn the model of interest from large datasets. More importantly, the system needed to be designed in a modular way to work with existing ecosystem and improve users’ productivity environment. In this talk, I will talk about XGBoost and MXNet, two learning scalable and portable systems that I build. I will discuss how we can apply distributed computing, asynchronous scheduling and hardware acceleration to improve these systems, as well as how do they fit into bigger open-source ecosystem of machine learning.
Neural_Programmer_Interpreter
Neural Programmer-Interpreters (NPI) is a neural network model that can learn programs from small amounts of training data and generalize to new situations. It has a core LSTM module that acts as a router between learned programs. In experiments, NPI outperformed sequence-to-sequence models in data efficiency and generalization when learning tasks like adding numbers and sorting arrays. It was also able to learn new programs like finding the maximum value by composing existing programs, without forgetting previous tasks. The authors conclude that NPI is better able to exploit compositional structure in tasks compared to sequence models.
Multi-class Image Classification using deep convolutional networks on extreme...
This document summarizes research on using deep convolutional networks for multi-class image classification on a large dataset from a product image classification Kaggle competition. The dataset contains over 5 million images across 5270 categories. Several CNN models were tested including ResNet, ResNext, DenseNet, and WideResNet. WideResNet achieved the best results with over 40% accuracy, while ResNext was the slowest. Training the models required significant computing resources and time due to the large dataset size. Future work includes submitting results to the Kaggle competition after more training epochs.
Hanjun Dai, PhD Student, School of Computational Science and Engineering, Geo...
Graph Representation Learning with Deep Embedding Approach:
Graphs are commonly used data structure for representing the real-world relationships, e.g., molecular structure, knowledge graphs, social and communication networks. The effective encoding of graphical information is essential to the success of such applications. In this talk I’ll first describe a general deep learning framework, namely structure2vec, for end to end graph feature representation learning. Then I’ll present the direct application of this model on graph problems on different scales, including community detection and molecule graph classification/regression. We then extend the embedding idea to temporal evolving user-product interaction graph for recommendation. Finally I’ll present our latest work on leveraging the reinforcement learning technique for graph combinatorial optimization, including vertex cover problem for social influence maximization and traveling salesman problem for scheduling management.
Making neural programming architectures generalize via recursion
ICLR 2017 paper, a cool work demonstrating the importance of recursion on neural program. presented at NTHU AI study group 2017
Anima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor,...
Large-scale Machine Learning: Deep, Distributed and Multi-Dimensional:
Modern machine learning involves deep neural network architectures which yields state-of-art performance on multiple domains such as computer vision, natural language processing and speech recognition. As the data and models scale, it becomes necessary to have multiple processing units for both training and inference. Apache MXNet is an open-source framework developed for distributed deep learning. I will describe the underlying lightweight hierarchical parameter server architecture that results in high efficiency in distributed settings.
Pushing the current boundaries of deep learning requires using multiple dimensions and modalities. These can be encoded into tensors, which are natural extensions of matrices. We present new deep learning architectures that preserve the multi-dimensional information in data end-to-end. We show that tensor contractions and regression layers are an effective replacement for fully connected layers in deep learning architectures. They result in significant space savings with negligible performance degradation. These functionalities are available in the Tensorly package with MXNet backend interface for large-scale efficient learning.
Bio: Anima Anandkumar is a principal scientist at Amazon Web Services and a Bren professor at Caltech CMS department. Her research interests are in the areas of large-scale machine learning, non-convex optimization and high-dimensional statistics. In particular, she has been spearheading the development and analysis of tensor algorithms. She is the recipient of several awards such as the Alfred. P. Sloan Fellowship, Microsoft Faculty Fellowship, Google research award, ARO and AFOSR Young Investigator Awards, NSF Career Award, Early Career Excellence in Research Award at UCI, Best Thesis Award from the ACM Sigmetrics society, IBM Fran Allen PhD fellowship, and several best paper awards. She has been featured in a number of forums such as the yourstory, Quora ML session, O’Reilly media, and so on. She received her B.Tech in Electrical Engineering from IIT Madras in 2004 and her PhD from Cornell University in 2009. She was a postdoctoral researcher at MIT from 2009 to 2010, an assistant professor at U.C. Irvine between 2010 and 2016, and a visiting researcher at Microsoft Research New England in 2012 and 2014.
Regularization in deep learning
Presentation in Vietnam Japan AI Community in 2019-05-26.
The presentation summarizes what I've learned about Regularization in Deep Learning.
Disclaimer: The presentation is given in a community event, so it wasn't thoroughly reviewed or revised.
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks. Link : https://arxiv.org/abs/1605.06409
DQN (Deep Q-Network)
The document discusses Deep Q-Network (DQN), which combines Q-learning with deep neural networks to allow for function approximation and solving problems with large state/action spaces. DQN uses experience replay and a separate target network to stabilize training. It has led to many successful variants, including Double DQN which reduces overestimation, prioritized experience replay which replays important transitions more frequently, and dueling networks which separate value and advantage estimation.
Image Classification using deep learning
image classification is a common problem in Artificial Intelligence , we used CIFR10 data set and tried a lot of methods to reach a high test accuracy like neural networks and Transfer learning techniques .
you can view the source code and the papers we read on github : https://github.com/Asma-Hawari/Machine-Learning-Project-
Dl
CNNs are a type of neural network that can analyze visual imagery. They use convolutional and pooling layers to automatically extract image features and classify images. CNNs use convolution operations instead of general matrix multiplication, which allows them to identify important characteristics in images. They have sparse connectivity and parameter sharing that make them effective for tasks like computer vision, NLP, and audio analysis. Transfer learning is commonly used with CNNs to repurpose pre-trained models for new problems by retraining only the final layers.
Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15
Is Machine Learning Code for 100 Rows or a Billion the Same?: We have built an automatically distributed, implicitly parallel data science platform for running large scale machine learning applications. By abstracting away the computer science required to scale machine learning models, The Ufora platform lets data scientists focus on building data science models in simple scripting code, without having to worry about building large-scale distributed systems, their race conditions, fault-tolerance, etc. This automatic approach requires solving some interesting challenges, like optimal data layout for different ML models. For example, when a data scientist says “do a linear regression on this 100GB dataset”, Ufora needs to figure out how to automatically distribute and lay out that data across a cluster of machines in the cluster in order to minimize travel over the wire. Running a GBM against the same dataset might require a completely different layout of that data. This talk will cover how the platform works, in terms of data and thread distribution, how it generates parallel processes out of single-threaded programs, and more.
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
https://github.com/telecombcn-dl/dlmm-2017-dcu
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017
Best Practices for Hyperparameter Optimization:
All machine learning and artificial intelligence pipelines – from reinforcement agents to deep neural nets – have tunable hyperparameters. Optimizing these hyperparameters provides tremendous performance gains, but only if the optimization is done correctly. This presentation will discuss topics including selecting performance criteria, why you should always use cross validation, and choosing between state of the art optimization methods.
How Criteo optimized and sped up its TensorFlow models by 10x and served them...
When you access a web page, bidders such as Criteo must determine in a few dozens of milliseconds if they want to purchase the advertising space on the page. At that moment, a real-time auction takes place, and once you remove all the communication exchange delays, it leaves a handful of milliseconds to compute exactly how much to bid. In the past year, Criteo has put a large amount of effort into reshaping its in-house machine learning stack responsible for making such predictions—in particular, opening it to new technologies such as TensorFlow.
Unfortunately, even for simple logistic regression models and small neural networks, Criteo’s initial TensorFlow implementations saw inference time increase by 100, going from 300 microseconds to 30 milliseconds.
Nicolas Kowalski and Axel Antoniotti outline how Criteo approached this issue, discussing how Criteo profiled its model to understand its bottleneck; why commonly shared solutions such as optimizing TensorFlow build for the target hardware, freezing and cleaning up the model, and using accelerated linear algebra (XLA) ended up being lackluster; and how Criteo rewrote is models from scratch, reimplementing cross-features and hashing functions using low-level TF operations in order to factorize as much as possible all TensorFlow nodes in its model.
20131024
1) The document reports on work done to compute integral images using parallel and serial versions. Testing on a 16x16 input showed the parallel version was much slower, but on a 640x480 input it was slightly faster than the serial version.
2) Profiling showed most time was spent on a column computation function, with memory copies also taking significant time. Memory access was identified as too slow.
3) An accidental error occurred when copying data from a matrix to a one-dimensional array, due to using an incorrect size calculation for memory initialization.
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)Universitat Politècnica de Catalunya
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.Beyond data and model parallelism for deep neural networks
FlexFlow is a deep learning engine that automatically finds optimal parallelization strategies for arbitrary DNN models and hardware configurations. It defines a comprehensive search space called SOAP that includes sample, operator, attribute, and parameter parallelization. FlexFlow uses a Markov Chain Monte Carlo algorithm to search this space. It simulates strategy executions, either through full simulation or more efficient delta simulation, to identify high performing strategies. Evaluation on real-world DNN models and hardware showed FlexFlow outperforms expert-designed strategies and other automated frameworks by achieving up to 3.3x higher training throughput.
A More Scaleable Way of Making Recommendations with MLlib-(Xiangrui Meng, Dat...
This document summarizes the implementation of Alternating Least Squares (ALS) in MLlib to make recommendations at scale. It discusses how MLlib reduces communication cost through a block-to-block approach and compressed storage formats. It also describes optimizations like avoiding garbage collection through specialized code. The ALS algorithm is tested on real-world datasets including Amazon reviews and Spotify music data involving billions of ratings.
Deep Learning for Computer Vision: Optimization (UPC 2016)
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
An overview of gradient descent optimization algorithms
This document provides an overview of various gradient descent optimization algorithms that are commonly used for training deep learning models. It begins with an introduction to gradient descent and its variants, including batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent. It then discusses challenges with these algorithms, such as choosing the learning rate. The document proceeds to explain popular optimization algorithms used to address these challenges, including momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, and Adam. It provides visualizations and intuitive explanations of how these algorithms work. Finally, it discusses strategies for parallelizing and optimizing SGD and concludes with a comparison of optimization algorithms.
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15
GraphMat: Bridging the Productivity-Performance Gap in Graph Analytics: With increasing interest in large-scale distributed graph analytics for machine learning and data mining, more data scientists and developers are struggling to achieve high performance without sacrificing productivity on large graph problems. In this talk, I will discuss our solution to this problem: GraphMat. Using generalized sparse matrix-based primitives, we are able to achieve performance that is very close to hand-optimized native code, while allowing users to write programs using the familiar vertex-centric programming paradigm. I will show how we optimized GraphMat to achieve this performance on distributed platforms and provide programming examples. We have integrated GraphMat with Apache Spark in a manner that allows the combination to outperform all other distributed graph frameworks. I will explain the reasons for this performance and show that our approach achieves very high hardware efficiency in both single-node and distributed environments using primitives that are applicable to many machine learning and HPC problems. GraphMat is open source software and available for download.
Cvpr 2018 papers review (efficient computing)
Overview of CVPR 2018 papers
For mobile DL
NetAdapt, ADC, Quantization and Training of ... (Tensorflow Lite quantization)
Convolutional neural networks 이론과 응용
This document introduces convolutional neural networks (CNNs). It discusses how CNNs extract features using filters and pooling to build up representations of images while reducing the number of parameters. The key operations of CNNs including convolution, nonlinear activation, pooling and fully connected layers are explained. Examples of CNN applications are provided. The evolution of CNNs is then reviewed, from LeNet and AlexNet to VGGNet, GoogleNet, and improvements like ReLU, dropout, and batch normalization that helped CNNs train better and go deeper.
More Related Content
What's hot
Multi-class Image Classification using deep convolutional networks on extreme...
This document summarizes research on using deep convolutional networks for multi-class image classification on a large dataset from a product image classification Kaggle competition. The dataset contains over 5 million images across 5270 categories. Several CNN models were tested including ResNet, ResNext, DenseNet, and WideResNet. WideResNet achieved the best results with over 40% accuracy, while ResNext was the slowest. Training the models required significant computing resources and time due to the large dataset size. Future work includes submitting results to the Kaggle competition after more training epochs.
Hanjun Dai, PhD Student, School of Computational Science and Engineering, Geo...
Graph Representation Learning with Deep Embedding Approach:
Graphs are commonly used data structure for representing the real-world relationships, e.g., molecular structure, knowledge graphs, social and communication networks. The effective encoding of graphical information is essential to the success of such applications. In this talk I’ll first describe a general deep learning framework, namely structure2vec, for end to end graph feature representation learning. Then I’ll present the direct application of this model on graph problems on different scales, including community detection and molecule graph classification/regression. We then extend the embedding idea to temporal evolving user-product interaction graph for recommendation. Finally I’ll present our latest work on leveraging the reinforcement learning technique for graph combinatorial optimization, including vertex cover problem for social influence maximization and traveling salesman problem for scheduling management.
Making neural programming architectures generalize via recursion
ICLR 2017 paper, a cool work demonstrating the importance of recursion on neural program. presented at NTHU AI study group 2017
Anima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor,...
Large-scale Machine Learning: Deep, Distributed and Multi-Dimensional:
Modern machine learning involves deep neural network architectures which yields state-of-art performance on multiple domains such as computer vision, natural language processing and speech recognition. As the data and models scale, it becomes necessary to have multiple processing units for both training and inference. Apache MXNet is an open-source framework developed for distributed deep learning. I will describe the underlying lightweight hierarchical parameter server architecture that results in high efficiency in distributed settings.
Pushing the current boundaries of deep learning requires using multiple dimensions and modalities. These can be encoded into tensors, which are natural extensions of matrices. We present new deep learning architectures that preserve the multi-dimensional information in data end-to-end. We show that tensor contractions and regression layers are an effective replacement for fully connected layers in deep learning architectures. They result in significant space savings with negligible performance degradation. These functionalities are available in the Tensorly package with MXNet backend interface for large-scale efficient learning.
Bio: Anima Anandkumar is a principal scientist at Amazon Web Services and a Bren professor at Caltech CMS department. Her research interests are in the areas of large-scale machine learning, non-convex optimization and high-dimensional statistics. In particular, she has been spearheading the development and analysis of tensor algorithms. She is the recipient of several awards such as the Alfred. P. Sloan Fellowship, Microsoft Faculty Fellowship, Google research award, ARO and AFOSR Young Investigator Awards, NSF Career Award, Early Career Excellence in Research Award at UCI, Best Thesis Award from the ACM Sigmetrics society, IBM Fran Allen PhD fellowship, and several best paper awards. She has been featured in a number of forums such as the yourstory, Quora ML session, O’Reilly media, and so on. She received her B.Tech in Electrical Engineering from IIT Madras in 2004 and her PhD from Cornell University in 2009. She was a postdoctoral researcher at MIT from 2009 to 2010, an assistant professor at U.C. Irvine between 2010 and 2016, and a visiting researcher at Microsoft Research New England in 2012 and 2014.
Regularization in deep learning
Presentation in Vietnam Japan AI Community in 2019-05-26.
The presentation summarizes what I've learned about Regularization in Deep Learning.
Disclaimer: The presentation is given in a community event, so it wasn't thoroughly reviewed or revised.
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks. Link : https://arxiv.org/abs/1605.06409
DQN (Deep Q-Network)
The document discusses Deep Q-Network (DQN), which combines Q-learning with deep neural networks to allow for function approximation and solving problems with large state/action spaces. DQN uses experience replay and a separate target network to stabilize training. It has led to many successful variants, including Double DQN which reduces overestimation, prioritized experience replay which replays important transitions more frequently, and dueling networks which separate value and advantage estimation.
Image Classification using deep learning
image classification is a common problem in Artificial Intelligence , we used CIFR10 data set and tried a lot of methods to reach a high test accuracy like neural networks and Transfer learning techniques .
you can view the source code and the papers we read on github : https://github.com/Asma-Hawari/Machine-Learning-Project-
Dl
CNNs are a type of neural network that can analyze visual imagery. They use convolutional and pooling layers to automatically extract image features and classify images. CNNs use convolution operations instead of general matrix multiplication, which allows them to identify important characteristics in images. They have sparse connectivity and parameter sharing that make them effective for tasks like computer vision, NLP, and audio analysis. Transfer learning is commonly used with CNNs to repurpose pre-trained models for new problems by retraining only the final layers.
Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15
Is Machine Learning Code for 100 Rows or a Billion the Same?: We have built an automatically distributed, implicitly parallel data science platform for running large scale machine learning applications. By abstracting away the computer science required to scale machine learning models, The Ufora platform lets data scientists focus on building data science models in simple scripting code, without having to worry about building large-scale distributed systems, their race conditions, fault-tolerance, etc. This automatic approach requires solving some interesting challenges, like optimal data layout for different ML models. For example, when a data scientist says “do a linear regression on this 100GB dataset”, Ufora needs to figure out how to automatically distribute and lay out that data across a cluster of machines in the cluster in order to minimize travel over the wire. Running a GBM against the same dataset might require a completely different layout of that data. This talk will cover how the platform works, in terms of data and thread distribution, how it generates parallel processes out of single-threaded programs, and more.
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)Universitat Politècnica de Catalunya
https://github.com/telecombcn-dl/dlmm-2017-dcu
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017
Best Practices for Hyperparameter Optimization:
All machine learning and artificial intelligence pipelines – from reinforcement agents to deep neural nets – have tunable hyperparameters. Optimizing these hyperparameters provides tremendous performance gains, but only if the optimization is done correctly. This presentation will discuss topics including selecting performance criteria, why you should always use cross validation, and choosing between state of the art optimization methods.
How Criteo optimized and sped up its TensorFlow models by 10x and served them...
When you access a web page, bidders such as Criteo must determine in a few dozens of milliseconds if they want to purchase the advertising space on the page. At that moment, a real-time auction takes place, and once you remove all the communication exchange delays, it leaves a handful of milliseconds to compute exactly how much to bid. In the past year, Criteo has put a large amount of effort into reshaping its in-house machine learning stack responsible for making such predictions—in particular, opening it to new technologies such as TensorFlow.
Unfortunately, even for simple logistic regression models and small neural networks, Criteo’s initial TensorFlow implementations saw inference time increase by 100, going from 300 microseconds to 30 milliseconds.
Nicolas Kowalski and Axel Antoniotti outline how Criteo approached this issue, discussing how Criteo profiled its model to understand its bottleneck; why commonly shared solutions such as optimizing TensorFlow build for the target hardware, freezing and cleaning up the model, and using accelerated linear algebra (XLA) ended up being lackluster; and how Criteo rewrote is models from scratch, reimplementing cross-features and hashing functions using low-level TF operations in order to factorize as much as possible all TensorFlow nodes in its model.
20131024
1) The document reports on work done to compute integral images using parallel and serial versions. Testing on a 16x16 input showed the parallel version was much slower, but on a 640x480 input it was slightly faster than the serial version.
2) Profiling showed most time was spent on a column computation function, with memory copies also taking significant time. Memory access was identified as too slow.
3) An accidental error occurred when copying data from a matrix to a one-dimensional array, due to using an incorrect size calculation for memory initialization.
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)Universitat Politècnica de Catalunya
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.Beyond data and model parallelism for deep neural networks
FlexFlow is a deep learning engine that automatically finds optimal parallelization strategies for arbitrary DNN models and hardware configurations. It defines a comprehensive search space called SOAP that includes sample, operator, attribute, and parameter parallelization. FlexFlow uses a Markov Chain Monte Carlo algorithm to search this space. It simulates strategy executions, either through full simulation or more efficient delta simulation, to identify high performing strategies. Evaluation on real-world DNN models and hardware showed FlexFlow outperforms expert-designed strategies and other automated frameworks by achieving up to 3.3x higher training throughput.
A More Scaleable Way of Making Recommendations with MLlib-(Xiangrui Meng, Dat...
This document summarizes the implementation of Alternating Least Squares (ALS) in MLlib to make recommendations at scale. It discusses how MLlib reduces communication cost through a block-to-block approach and compressed storage formats. It also describes optimizations like avoiding garbage collection through specialized code. The ALS algorithm is tested on real-world datasets including Amazon reviews and Spotify music data involving billions of ratings.
Deep Learning for Computer Vision: Optimization (UPC 2016)
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
An overview of gradient descent optimization algorithms
This document provides an overview of various gradient descent optimization algorithms that are commonly used for training deep learning models. It begins with an introduction to gradient descent and its variants, including batch gradient descent, stochastic gradient descent (SGD), and mini-batch gradient descent. It then discusses challenges with these algorithms, such as choosing the learning rate. The document proceeds to explain popular optimization algorithms used to address these challenges, including momentum, Nesterov accelerated gradient, Adagrad, Adadelta, RMSprop, and Adam. It provides visualizations and intuitive explanations of how these algorithms work. Finally, it discusses strategies for parallelizing and optimizing SGD and concludes with a comparison of optimization algorithms.
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15
GraphMat: Bridging the Productivity-Performance Gap in Graph Analytics: With increasing interest in large-scale distributed graph analytics for machine learning and data mining, more data scientists and developers are struggling to achieve high performance without sacrificing productivity on large graph problems. In this talk, I will discuss our solution to this problem: GraphMat. Using generalized sparse matrix-based primitives, we are able to achieve performance that is very close to hand-optimized native code, while allowing users to write programs using the familiar vertex-centric programming paradigm. I will show how we optimized GraphMat to achieve this performance on distributed platforms and provide programming examples. We have integrated GraphMat with Apache Spark in a manner that allows the combination to outperform all other distributed graph frameworks. I will explain the reasons for this performance and show that our approach achieves very high hardware efficiency in both single-node and distributed environments using primitives that are applicable to many machine learning and HPC problems. GraphMat is open source software and available for download.
What's hot (20)
Multi-class Image Classification using deep convolutional networks on extreme...
Multi-class Image Classification using deep convolutional networks on extreme...
Hanjun Dai, PhD Student, School of Computational Science and Engineering, Geo...
Hanjun Dai, PhD Student, School of Computational Science and Engineering, Geo...
Making neural programming architectures generalize via recursion
Making neural programming architectures generalize via recursion
Anima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor,...
Anima Anadkumar, Principal Scientist, Amazon Web Services, Endowed Professor,...
R-FCN : object detection via region-based fully convolutional networks
R-FCN : object detection via region-based fully convolutional networks
Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15
Braxton McKee, Founder & CEO, Ufora at MLconf SF - 11/13/15
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)
Optimizing Deep Networks (D1L6 Insight@DCU Machine Learning Workshop 2017)
Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017
Alexandra Johnson, Software Engineer, SigOpt at MLconf ATL 2017
How Criteo optimized and sped up its TensorFlow models by 10x and served them...
How Criteo optimized and sped up its TensorFlow models by 10x and served them...
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)
Deep Learning for Computer Vision: Backward Propagation (UPC 2016)
Beyond data and model parallelism for deep neural networks
Beyond data and model parallelism for deep neural networks
A More Scaleable Way of Making Recommendations with MLlib-(Xiangrui Meng, Dat...
A More Scaleable Way of Making Recommendations with MLlib-(Xiangrui Meng, Dat...
Deep Learning for Computer Vision: Optimization (UPC 2016)
Deep Learning for Computer Vision: Optimization (UPC 2016)
An overview of gradient descent optimization algorithms
An overview of gradient descent optimization algorithms
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15
Narayanan Sundaram, Research Scientist, Intel Labs at MLconf SF - 11/13/15
Similar to Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
Cvpr 2018 papers review (efficient computing)
Overview of CVPR 2018 papers
For mobile DL
NetAdapt, ADC, Quantization and Training of ... (Tensorflow Lite quantization)
Convolutional neural networks 이론과 응용
This document introduces convolutional neural networks (CNNs). It discusses how CNNs extract features using filters and pooling to build up representations of images while reducing the number of parameters. The key operations of CNNs including convolution, nonlinear activation, pooling and fully connected layers are explained. Examples of CNN applications are provided. The evolution of CNNs is then reviewed, from LeNet and AlexNet to VGGNet, GoogleNet, and improvements like ReLU, dropout, and batch normalization that helped CNNs train better and go deeper.
An Introduction to Deep Learning
This document provides an overview of deep learning including why it is used, common applications, strengths and challenges, common algorithms, and techniques for developing deep learning models. In 3 sentences: Deep learning methods like neural networks can learn complex patterns in large, unlabeled datasets and are better than traditional machine learning for tasks like image recognition. Popular deep learning algorithms include convolutional neural networks for image data and recurrent neural networks for sequential data. Effective deep learning requires techniques like regularization, dropout, data augmentation, and hyperparameter optimization to prevent overfitting on training data.
Introduction to Deep Learning
This document provides an overview of deep learning including:
1. Why deep learning performs better than traditional machine learning for tasks like image and speech recognition.
2. Common deep learning applications such as image recognition, speech recognition, and healthcare.
3. Challenges of deep learning like the need for large datasets and lack of interpretability.
[2020 CVPR Efficient DET paper review]
EfficientDet is a single-shot object detector that achieves state-of-the-art accuracy with high efficiency. It introduces a Bi-directional Feature Pyramid Network (BiFPN) that fuses multi-scale features with weighted connections. EfficientDet also uses a compound scaling method to jointly scale the backbone network, BiFPN, prediction network, and input resolution. Experiments on COCO show that EfficientDet outperforms prior models across different resource constraints in terms of accuracy and efficiency for object detection and semantic segmentation tasks.
Deep Learning
This document provides an overview of artificial intelligence and machine learning techniques, including:
1. It defines artificial intelligence and lists some common applications such as gaming, natural language processing, and robotics.
2. It describes different machine learning algorithms like supervised learning, unsupervised learning, reinforced learning, and their applications in areas such as healthcare, finance, and retail.
3. It explains deep learning concepts such as neural networks, activation functions, loss functions, and architectures like convolutional neural networks and recurrent neural networks.
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...Edge AI and Vision Alliance
For the full video of this presentation, please visit:
https://www.embedded-vision.com/platinum-members/embedded-vision-alliance/embedded-vision-training/videos/pages/sep-2019-alliance-vitf-facebook
For more information about embedded vision, please visit:
http://www.embedded-vision.com
Raghuraman Krishnamoorthi, Software Engineer at Facebook, delivers the presentation "Quantizing Deep Networks for Efficient Inference at the Edge" at the Embedded Vision Alliance's September 2019 Vision Industry and Technology Forum. Krishnamoorthi gives an overview of practical deep neural network quantization techniques and tools.Computer vision-nit-silchar-hackathon
The document introduces various computer vision topics including convolutional neural networks, popular CNN architectures, data augmentation, transfer learning, object detection, neural style transfer, generative adversarial networks, and variational autoencoders. It provides overviews of each topic and discusses concepts such as how convolutions work, common CNN architectures like ResNet and VGG, why data augmentation is important, how transfer learning can utilize pre-trained models, how object detection algorithms like YOLO work, the content and style losses used in neural style transfer, how GANs use generators and discriminators, and how VAEs describe images with probability distributions. The document aims to discuss these topics at a practical level and provide insights through examples.
Deeplearning
This document provides an outline and overview of training convolutional neural networks. It discusses update rules like stochastic gradient descent, momentum, and Adam. It also covers techniques like data augmentation, transfer learning, and monitoring the training process. The goal of training a CNN is to optimize its weights and parameters to correctly classify images from the training set by minimizing output error through backpropagation and updating weights.
深度學習在AOI的應用
This document discusses using fully convolutional neural networks for defect inspection. It begins with an agenda that outlines image segmentation using FCNs and defect inspection. It then provides details on data preparation including labeling guidelines, data augmentation, and model setup using techniques like deconvolution layers and the U-Net architecture. Metrics for evaluating the model like Dice score and IoU are also covered. The document concludes with best practices for successful deep learning projects focusing on aspects like having a large reusable dataset, feasibility of the problem, potential payoff, and fault tolerance.
Deep Learning for Computer Vision - PyconDE 2017
This document discusses deep learning for computer vision tasks. It begins with an overview of image classification using convolutional neural networks and how they have achieved superhuman performance on ImageNet. It then covers the key layers and concepts in CNNs, including convolutions, max pooling, and transferring learning to new problems. Finally, it discusses more advanced computer vision tasks that CNNs have been applied to, such as semantic segmentation, style transfer, visual question answering, and combining images with other data sources.
DenseNet Models for Tiny ImageNet Classification
We present two image classification models on the Tiny ImageNet dataset. We built two very different networks
from scratch based on the idea of Densely Connected Convolution Networks. The architecture of the networks is designed based on the image resolution of this specific dataset
and by calculating the Receptive Field of the convolution layers. We also used some non-conventional techniques related to image augmentation and Cyclical Learning Rate to improve the accuracy of our models. The networks are trained under high constraints and low computation resources. We aimed to
achieve top-1 validation accuracy of 60%.
NVIDIA 深度學習教育機構 (DLI): Medical image segmentation using digits
NVIDIA DIGITS 視覺化平台訓練醫療圖像分割
課程目標與介紹
在本實驗中，您將學習如何使用目前熱門的圖像分類神經網絡進行語義分割，如何使用自定義 Python 層擴大至使用熱門的 Caffe 架構，透過此練習將可以熟悉如何傳輸學習和訓練兩個完全卷積網絡 (FCN) 的概念。本課程使用醫療影像作為範例，將圖像使用語義分割 (Semantic Segmentation) 的方式，將圖片中的每一個像素做分類，通常給予不同顏色代表不同類別。透過國際知名的醫療影像競賽 - Cardiac 左心室MRI 圖像分割挑戰，來學習如何使用 NVIDIA DIGITS 平台簡單的學習圖像分割應用。
Introduction to deep learning
This document provides an introduction to deep learning. It discusses key concepts such as neural network layers that process input tensors, common layer types like convolutional and recurrent layers, and how networks are trained using stochastic gradient descent. Examples of deep learning applications that have achieved near-human level performance are also presented, such as image classification and speech recognition. The document then focuses on convolutional neural networks, covering concepts like convolution operations, spatial hierarchies, and max pooling. It concludes with a demonstration of digit and X-ray image classification using Keras and techniques for dealing with overfitting like dropout and data augmentation.
OBDPC 2022
DEEP NEURAL NETWORKS APPLIED TO LOW POWER ONBOARD IMAGE COMPRESSION
Over the past decade, rapid developments in digital technologies and access to space have enabled unprecedented capabilities of monitoring our planet and, more generally, our Universe.
This new space race is pushing for a paradigm shift in order to respond to the ever-increasing challenge of delivering the useful information to the end users. With huge number of satellites, greater spatial and spectral resolutions, higher temporal cadence and shrinking spectrum resources, on-board data reduction becomes not only a cost saving solution but, in many cases also, a key enabling technology to achieve viable missions.
https://atpi.eventsair.com/obpdc2022/
Online video object segmentation via convolutional trident network
발표자: 장원동 (고려대 박사과정)
발표일: 2017.8.
개요:
A semi-supervised online video object segmentation algorithm, which accepts user annotations about a target object at the first frame, will be presented. It propagates the segmentation labels at the previous frame to the current frame using optical flow vectors.
However, the propagation is error-prone. Therefore, I’ve developed the convolutional trident network, which has three decoding branches: separative, definite foreground, and definite background decoders.
Then, the algorithm performs Markov random field optimization based on outputs of the three decoders.
These process is sequentially carried out from the second to the last frames to extract a segment track of the target object.
Experimental results will demonstrate that this algorithm significantly outperforms the state-of-the-art conventional algorithms on the DAVIS benchmark dataset.
SPPNet
The document summarizes the Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition paper. It introduces spatial pyramid pooling (SPP), which allows convolutional neural networks to take variable-sized images as input without resizing. SPP extracts feature maps from convolutional layers and pools them in spatial bins to generate fixed-length outputs. The paper shows SPP improves image classification and object detection accuracy while greatly increasing detection speed compared to networks that require fixed inputs. SPP networks achieve state-of-the-art results on ImageNet, Pascal VOC, and Caltech101 benchmarks.
Week 12 Dimensionality Reduction Bagian 1
This document discusses dimensionality reduction using principal component analysis (PCA). It explains that PCA is used to reduce the number of variables in a dataset while retaining the variation present in the original data. The document outlines the PCA algorithm, which transforms the original variables into new uncorrelated variables called principal components. It provides an example of applying PCA to reduce data from 2D to 1D. The document also discusses key PCA concepts like covariance matrices, eigenvalues, eigenvectors, and transforming data to the principal component coordinate system. Finally, it presents an assignment applying PCA and classification to a handwritten digits dataset.
Deep learning
This document provides an overview of deep learning concepts including neural networks, regression and classification, convolutional neural networks, and applications of deep learning such as housing price prediction. It discusses techniques for training neural networks including feature extraction, cost functions, gradient descent, and regularization. The document also reviews deep learning frameworks and notable deep learning models like AlexNet that have achieved success in tasks such as image classification.
Similar to Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (20)
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...
"Quantizing Deep Networks for Efficient Inference at the Edge," a Presentatio...
NVIDIA 深度學習教育機構 (DLI): Medical image segmentation using digits
NVIDIA 深度學習教育機構 (DLI): Medical image segmentation using digits
Online video object segmentation via convolutional trident network
Online video object segmentation via convolutional trident network
More from Sungchul Kim
PR-343: Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
This document proposes a new semi-supervised learning method called Cross Pseudo Supervision (CPS) for semantic segmentation. CPS trains two segmentation networks simultaneously where each network generates pseudo labels for the other using its own predictions. It incorporates this cross pseudo supervision loss with a CutMix data augmentation technique. The document evaluates CPS on PASCAL VOC 2012 and Cityscapes datasets with limited labeled data, finding it outperforms other semi-supervised segmentation methods and ablations. Qualitative results show CPS with CutMix generates more accurate segmentations than alternatives.
Revisiting the Calibration of Modern Neural Networks
The document discusses calibration in modern neural networks. It finds that some recent model families, like MLP-Mixers and Vision Transformers (ViTs), are both highly accurate and well-calibrated on in-distribution and out-of-distribution image datasets. Temperature scaling can improve calibration in older models but the newest architectures remain better calibrated. Model size and pretraining amount do not fully explain differences in calibration between families. Architecture appears to be a major factor, with non-convolutional models tending to have better calibration properties.
Emerging Properties in Self-Supervised Vision Transformers
The document summarizes the DINO self-supervised learning approach for vision transformers. DINO uses a teacher-student framework where the teacher's predictions are used to supervise the student through knowledge distillation. Two global and several local views of an image are passed through the student, while only global views are passed through the teacher. The student is trained to match the teacher's predictions for local views. DINO achieves state-of-the-art results on ImageNet with linear evaluation and transfers well to downstream tasks. It also enables vision transformers to discover object boundaries and semantic layouts.
Score based Generative Modeling through Stochastic Differential Equations
This document discusses score-based generative modeling using stochastic differential equations (SDEs). It introduces modeling data diffusion as an SDE from the data distribution to a simple prior and generating samples by reversing this diffusion process. It also describes estimating the score (gradient of the log probability density) needed for the reverse process using score matching. Finally, it notes that noise perturbation models like NCSN and DDPM can be viewed as discretizations of specific SDEs called variance exploding and variance preserving SDEs.
Exploring Simple Siamese Representation Learning
This document discusses an unsupervised representation learning method called SimSiam. It proposes that SimSiam can be interpreted as an expectation-maximization algorithm that alternates between updating the encoder parameters and assigning representations to images. Key aspects discussed include how the stop-gradient operation prevents collapsed representations, the role of the predictor network, effects of batch size and batch normalization, and alternatives to the cosine similarity measure. Empirical results show that SimSiam learns meaningful representations without collapsing, and the various design choices affect performance but not the ability to prevent collapsed representations.
Revisiting the Sibling Head in Object Detector
This document describes a method called Task-aware Spatial Disentanglement (TSD) for object detection. TSD uses separate branches to process features for the classification and localization tasks in a spatially disentangled manner. For classification, TSD applies pointwise deformations to the feature map. For localization, it applies proposal-wise translations. This allows each task to process features with spatial sensitivities suitable for their goals, improving performance over methods that do not separate spatial processing for different tasks. TSD achieves state-of-the-art object detection accuracy on standard benchmarks.
Do Wide and Deep Networks Learn the Same Things: Uncovering How Neural Networ...
Do Wide and Deep Networks Learn the Same Things: Uncovering How Neural Network Representations Vary with Width and Depth
Deeplabv1, v2, v3, v3+
This document summarizes the DeepLab models for semantic image segmentation: DeepLab v1 used atrous convolution with VGG-16 as the backbone network. DeepLab v2 improved on this with atrous spatial pyramid pooling and added ResNet-101 as an option. DeepLab v3 removed dense CRFs and introduced multi-grid atrous convolution and bootstrapping. DeepLab v3+ uses an encoder-decoder architecture with Xception or ResNet-101 as the backbone and atrous separable convolutions.
Going Deeper with Convolutions
This document summarizes a paper on GoogLeNet, a convolutional neural network that won the 2014 ILSVRC competition. It discusses how GoogLeNet uses inception layers and network-in-network dimensionality reduction to achieve higher accuracy than AlexNet with 12x fewer parameters. The document outlines GoogLeNet's architectural details, training methodology using stochastic gradient descent, and results in classification and detection tracks at ILSVRC 2014. It also references related work on R-CNN, Network-in-Network, and AlexNet.
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Generalized Intersection over Union: A Metric and A Loss for Bounding Box Reg...
1. The document proposes a Generalized Intersection over Union (GIoU) as a new loss function for bounding box regression that addresses limitations of the standard Intersection over Union (IoU) metric.
2. GIoU extends IoU by also considering the minimum enclosing box for two bounding boxes, not just their intersection. This allows GIoU to better measure the distance between two boxes even if they do not intersect.
3. Experimental results on object detection models like YOLO v3, Faster R-CNN and Mask R-CNN show that using GIoU as the regression loss provides stronger correlation with IoU and improves model performance compared to using IoU directly as the loss.
Panoptic Segmentation
1. This document discusses panoptic segmentation, which aims to perform semantic segmentation and instance segmentation simultaneously.
2. It introduces the panoptic segmentation task format, which assigns each pixel a semantic label and instance ID. It also presents a new evaluation metric called Panoptic Quality (PQ) to evaluate panoptic segmentation.
3. Baseline experiments are discussed that combine existing semantic and instance segmentation models. The results show current machine performance is still below human consistency levels based on a human annotation study. Future work to develop end-to-end panoptic segmentation models is suggested.
On the Variance of the Adaptive Learning Rate and Beyond
This document discusses the variance of adaptive learning rates in stochastic gradient descent optimization. It is hypothesized that the large variance of adaptive learning rates early in training can lead to getting stuck in bad local optima. The document proposes rectifying adaptive learning rates to reduce this initial variance, similarly to how warmup reduces variance but without requiring tuning hyperparameters. Experiments show that rectifying adaptive learning rates outperforms vanilla adaptive methods and warmup in convergence and robustness to learning rate changes.
A Benchmark for Interpretability Methods in Deep Neural Networks
This document proposes two interpretability evaluation frameworks: ROAR (RemOve And Retrain) and KAR (Keep And Retrain). ROAR evaluates feature importance by removing salient features from inputs based on a trained model's saliency map, regenerating training and test data, and retraining the model. KAR keeps salient features and retrains instead of removing. The document introduces common interpretability methods like gradients, integrated gradients, and ensembling methods. It then explains the ROAR and KAR mechanisms and provides example results, concluding the frameworks can validate feature importance estimates and model reliability.
KDGAN: Knowledge Distillation with Generative Adversarial Networks
This document discusses a method called KDGAN that combines knowledge distillation and generative adversarial networks. KDGAN trains a lightweight classifier (student) using knowledge distilled from a larger teacher model and a discriminator that learns the true data distribution. The classifier, teacher, and discriminator are trained together in an adversarial framework where the classifier learns to mimic the teacher's outputs and fool the discriminator, while the discriminator learns to distinguish the classifier's outputs from real data. This equilibrium allows the classifier to learn the true data distribution effectively from limited data with privileged information only available during training. The document outlines the KDGAN training method and framework and provides examples of compressing deep models and performing image tag recommendation.
Designing Network Design Spaces
This document discusses the design of neural network architectures. It introduces RegNet, a design space that uses a simple linear parameterization to generate "regular" networks. The document analyzes the RegNet design space and compares it to existing networks. Key findings include that optimal depth is around 20 blocks, a bottleneck ratio of 1 is best, and a width multiplier of around 2.5 works well. RegNet models achieve state-of-the-art or competitive results across mobile, standard, and full network regimes.
Search to Distill: Pearls are Everywhere but not the Eyes
This document discusses architecture-aware knowledge distillation (AKD), which uses reinforcement learning to find the optimal student network architecture for distilling knowledge from a given teacher model. It introduces AKD and how it guides the search for best architectures under latency constraints. Preliminary experiments show AKD networks achieve state-of-the-art results on ImageNet classification and generalize well to other tasks like face recognition and ensemble learning.
FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stoch...
FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stochastic Inference
Regularizing Class-wise Predictions via Self-knowledge Distillation
- The document proposes a method called Class-wise Self-Knowledge Distillation (CS-KD) to improve the generalization of deep neural networks.
- CS-KD matches the predictive distributions of different samples from the same class, encouraging consistent predictions for samples of the same class.
- Experiments on image classification datasets show CS-KD reduces error rates compared to other regularization methods and self-distillation baselines. It also improves calibration and works well with other techniques.
- By minimizing the KL divergence between predictive distributions of same-class samples, CS-KD enhances generalization and calibration of neural networks.
More from Sungchul Kim (20)
PR-343: Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
PR-343: Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision
Revisiting the Calibration of Modern Neural Networks
Revisiting the Calibration of Modern Neural Networks
Emerging Properties in Self-Supervised Vision Transformers
Emerging Properties in Self-Supervised Vision Transformers
Score based Generative Modeling through Stochastic Differential Equations
Score based Generative Modeling through Stochastic Differential Equations
Do Wide and Deep Networks Learn the Same Things: Uncovering How Neural Networ...
Do Wide and Deep Networks Learn the Same Things: Uncovering How Neural Networ...
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization
Generalized Intersection over Union: A Metric and A Loss for Bounding Box Reg...
Generalized Intersection over Union: A Metric and A Loss for Bounding Box Reg...
On the Variance of the Adaptive Learning Rate and Beyond
On the Variance of the Adaptive Learning Rate and Beyond
A Benchmark for Interpretability Methods in Deep Neural Networks
A Benchmark for Interpretability Methods in Deep Neural Networks
KDGAN: Knowledge Distillation with Generative Adversarial Networks
KDGAN: Knowledge Distillation with Generative Adversarial Networks
Search to Distill: Pearls are Everywhere but not the Eyes
Search to Distill: Pearls are Everywhere but not the Eyes
FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stoch...
FickleNet: Weakly and Semi-supervised Semantic Image Segmentation using Stoch...
Regularizing Class-wise Predictions via Self-knowledge Distillation
Regularizing Class-wise Predictions via Self-knowledge Distillation
Recently uploaded
Low power architecture of logic gates using adiabatic techniques
The growing significance of portable systems to limit power consumption in ultra-large-scale-integration chips of very high density, has recently led to rapid and inventive progresses in low-power design. The most effective technique is adiabatic logic circuit design in energy-efficient hardware. This paper presents two adiabatic approaches for the design of low power circuits, modified positive feedback adiabatic logic (modified PFAL) and the other is direct current diode based positive feedback adiabatic logic (DC-DB PFAL). Logic gates are the preliminary components in any digital circuit design. By improving the performance of basic gates, one can improvise the whole system performance. In this paper proposed circuit design of the low power architecture of OR/NOR, AND/NAND, and XOR/XNOR gates are presented using the said approaches and their results are analyzed for powerdissipation, delay, power-delay-product and rise time and compared with the other adiabatic techniques along with the conventional complementary metal oxide semiconductor (CMOS) designs reported in the literature. It has been found that the designs with DC-DB PFAL technique outperform with the percentage improvement of 65% for NOR gate and 7% for NAND gate and 34% for XNOR gate over the modified PFAL techniques at 10 MHz respectively.
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...University of Maribor
Slides from talk presenting:
Aleš Zamuda: Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapter and Networking.
Presentation at IcETRAN 2024 session:
"Inter-Society Networking Panel GRSS/MTT-S/CIS
Panel Session: Promoting Connection and Cooperation"
IEEE Slovenia GRSS
IEEE Serbia and Montenegro MTT-S
IEEE Slovenia CIS
11TH INTERNATIONAL CONFERENCE ON ELECTRICAL, ELECTRONIC AND COMPUTING ENGINEERING
3-6 June 2024, Niš, SerbiaLiterature Review Basics and Understanding Reference Management.pptx
Three-day training on academic research focuses on analytical tools at United Technical College, supported by the University Grant Commission, Nepal. 24-26 May 2024
ACEP Magazine edition 4th launched on 05.06.2024
This document provides information about the third edition of the magazine "Sthapatya" published by the Association of Civil Engineers (Practicing) Aurangabad. It includes messages from current and past presidents of ACEP, memories and photos from past ACEP events, information on life time achievement awards given by ACEP, and a technical article on concrete maintenance, repairs and strengthening. The document highlights activities of ACEP and provides a technical educational article for members.
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
CHINA’S GEO-ECONOMIC OUTREACH IN CENTRAL ASIAN COUNTRIES AND FUTURE PROSPECT
The rivalry between prominent international actors for dominance over Central Asia's hydrocarbon
reserves and the ancient silk trade route, along with China's diplomatic endeavours in the area, has been
referred to as the "New Great Game." This research centres on the power struggle, considering
geopolitical, geostrategic, and geoeconomic variables. Topics including trade, political hegemony, oil
politics, and conventional and nontraditional security are all explored and explained by the researcher.
Using Mackinder's Heartland, Spykman Rimland, and Hegemonic Stability theories, examines China's role
in Central Asia. This study adheres to the empirical epistemological method and has taken care of
objectivity. This study analyze primary and secondary research documents critically to elaborate role of
china’s geo economic outreach in central Asian countries and its future prospect. China is thriving in trade,
pipeline politics, and winning states, according to this study, thanks to important instruments like the
Shanghai Cooperation Organisation and the Belt and Road Economic Initiative. According to this study,
China is seeing significant success in commerce, pipeline politics, and gaining influence on other
governments. This success may be attributed to the effective utilisation of key tools such as the Shanghai
Cooperation Organisation and the Belt and Road Economic Initiative.
22CYT12-Unit-V-E Waste and its Management.ppt
Introduction- e - waste – definition - sources of e-waste– hazardous substances in e-waste - effects of e-waste on environment and human health- need for e-waste management– e-waste handling rules - waste minimization techniques for managing e-waste – recycling of e-waste - disposal treatment methods of e- waste – mechanism of extraction of precious metal from leaching solution-global Scenario of E-waste – E-waste in India- case studies.
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology towards Sustainable Steelmaking
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
Kinematics 11th jpp- 01. ( Solved ) unacademy namo kaul on 14th may...
A SYSTEMATIC RISK ASSESSMENT APPROACH FOR SECURING THE SMART IRRIGATION SYSTEMS
The smart irrigation system represents an innovative approach to optimize water usage in agricultural and landscaping practices. The integration of cutting-edge technologies, including sensors, actuators, and data analysis, empowers this system to provide accurate monitoring and control of irrigation processes by leveraging real-time environmental conditions. The main objective of a smart irrigation system is to optimize water efficiency, minimize expenses, and foster the adoption of sustainable water management methods. This paper conducts a systematic risk assessment by exploring the key components/assets and their functionalities in the smart irrigation system. The crucial role of sensors in gathering data on soil moisture, weather patterns, and plant well-being is emphasized in this system. These sensors enable intelligent decision-making in irrigation scheduling and water distribution, leading to enhanced water efficiency and sustainable water management practices. Actuators enable automated control of irrigation devices, ensuring precise and targeted water delivery to plants. Additionally, the paper addresses the potential threat and vulnerabilities associated with smart irrigation systems. It discusses limitations of the system, such as power constraints and computational capabilities, and calculates the potential security risks. The paper suggests possible risk treatment methods for effective secure system operation. In conclusion, the paper emphasizes the significant benefits of implementing smart irrigation systems, including improved water conservation, increased crop yield, and reduced environmental impact. Additionally, based on the security analysis conducted, the paper recommends the implementation of countermeasures and security approaches to address vulnerabilities and ensure the integrity and reliability of the system. By incorporating these measures, smart irrigation technology can revolutionize water management practices in agriculture, promoting sustainability, resource efficiency, and safeguarding against potential security threats.
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generation
Recently uploaded (20)
Low power architecture of logic gates using adiabatic techniques
Low power architecture of logic gates using adiabatic techniques
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Presentation of IEEE Slovenia CIS (Computational Intelligence Society) Chapte...
Literature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptx
Manufacturing Process of molasses based distillery ppt.pptx
Manufacturing Process of molasses based distillery ppt.pptx
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
IEEE Aerospace and Electronic Systems Society as a Graduate Student Member
CHINA’S GEO-ECONOMIC OUTREACH IN CENTRAL ASIAN COUNTRIES AND FUTURE PROSPECT
CHINA’S GEO-ECONOMIC OUTREACH IN CENTRAL ASIAN COUNTRIES AND FUTURE PROSPECT
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
ML Based Model for NIDS MSc Updated Presentation.v2.pptx
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
Iron and Steel Technology Roadmap - Towards more sustainable steelmaking.pdf
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
[JPP-1] - (JEE 3.0) - Kinematics 1D - 14th May..pdf
A SYSTEMATIC RISK ASSESSMENT APPROACH FOR SECURING THE SMART IRRIGATION SYSTEMS
A SYSTEMATIC RISK ASSESSMENT APPROACH FOR SECURING THE SMART IRRIGATION SYSTEMS
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning
- 1. 김 성 철
- 2. Contents 1. Introduction 2. Method 3. Experimental evaluation 4. Building intuitions with ablations 5. Conclusion 2
- 3. Introduction • Contrastive Learning • 좋은 representation을 학습하는 것이 computer vision에서 중요한 문제 • Contrastive learning은 state-of-the-art를 기록하고 있음 • Positive pairs (same image)와 가깝게, negative pairs (different images)와는 멀게 • Large batch sizes (SimCLR), memory banks (MoCo), customized mining strategies • Bootstrap Your Own Latent (BYOL) • state-of-the-art contrastive method without negative pairs • Target에 대한 pseudo-label이나 cluster indices이 아닌 representation을 bootstrap • 다른 augmentation을 적용한 online, target network 사용 3
- 5. Method • Motivation • Step 1 → 1.4% Top 1 Acc • Fixed randomly initialized encoder + trainable linear layer • Labeled dataset으로 학습 • Step 2 → 18.8% Top 1 Acc • Random initialized encoder & linear layer로 unlabeled dataset 예측 • 예측된 label을 새로운 fixed randomly initialized encoder + trainable linear layer 사전 학습 • 실제 labeled dataset으로 학습 → 실제 사용하는 online network에 target network가 필요함! (Self-knowledge distillation + Semi-supervised?) 5 https://hoya012.github.io/blog/byol/
- 6. Method • Terminology • 𝜃 : a set of weights of the online network • 𝜉 : a set of weights of the target network 6 Encoder Projector PredictorAugmentation Stop gradient
- 7. Method • Description of BYOL • Target network는 online network가 학습할 regression target을 예측 • Target network의 parameter 𝜉는 online parameter 𝜃의 exponential moving average 𝜉 ⟵ 𝜏𝜉 + 1 − 𝜏 𝜃, 𝜏 ∈ 0,1 • Loss : prediction 𝑞 𝜃 𝑧 𝜃 와 𝑧 𝜉 ′ 의 𝑙2-norm의 mean squared error ℒ 𝜃 BYOL ≜ 𝑞 𝜃 𝑧 𝜃 − ഥ𝑧 𝜉 ′ 2 2 = 2 − 2 ⋅ 𝑞 𝜃 𝑧 𝜃 , 𝑧 𝜉 ′ 𝑞 𝜃 𝑧 𝜃 2 ⋅ 𝑧 𝜉 ′ 2 when 𝑞 𝜃 𝑧 𝜃 ≜ Τ𝑞 𝜃 𝑧 𝜃 𝑞 𝜃 𝑧 𝜃 2 and ഥ𝑧 𝜉 ′ ≜ ൗ𝑧 𝜉 ′ 𝑧 𝜉 ′ 2 • 두 network의 input을 서로 바꾸어 낸 결과로 ሚℒ 𝜃 BYOL 계산 • ℒ 𝜃 BYOL + ሚℒ 𝜃 BYOL를 online network에만 적용 7
- 8. Method • Implementation details • Image augmentations • SimCLR에서 사용한 기법 사용 8
- 9. Method • Implementation details • Architecture • ResNet50 • 4096-dimension MLP (projection) with no batch normalization • 256-dimension prediction layer • Optimization • LARS optimizer • 1000 epochs with warm-up period of 10 epochs • Linear scaled learning rate 0.2 (LearningRate = 0.2 x BatchSize/256) • 1.5 ⋅ 10−6 global weight decay parameter • 𝜏base = 0.996 and 𝜏 ≜ 1 − 1 − 𝜏base ⋅ cos 𝜋𝑘/𝐾 /2 with k the current training step and K the total step • 4096 batch size split over 512 Cloud TPU v3 cores 9
- 10. Experimental evaluation • Linear evaluation on ImageNet 10
- 11. Experimental evaluation • Semi-supervised training on ImageNet 11
- 12. Experimental evaluation • Transfer to other classification tasks 12
- 13. Experimental evaluation • Transfer to other vision tasks 13
- 14. Building intuitions with ablations • Batch size • Batch size가 작아져도 SimCLR보다 성능 하락 폭이 좁음 • Image augmentations • Image augmentation option에 대해 robust 함 14
- 15. Building intuitions with ablations • Bootstrapping • 𝜏 = 0 : target network = online network • 𝜏 = 1 : never updated target network (18.8% Top 1 Acc) • Online network의 weight를 target network에 입히기 위한 적절한 𝜏 설정이 필요함 15
- 16. Building intuitions with ablations • Ablation to contrastive methods • 𝛽 = 1 : SimCLR • No predictor, no target network • 𝛽 = 0 : BYOL • No negative InfoNCE 𝜃 ≜ 2 𝐵 𝑖=1 𝐵 𝑆 𝜃 𝑣𝑖, 𝑣𝑖 ′ − 𝛽 ⋅ 2𝛼 𝐵 𝑖=1 𝐵 ln 𝑗≠𝑖 exp 𝑆 𝜃 𝑣𝑖, 𝑣𝑗 𝛼 + 𝑗 exp 𝑆 𝜃 𝑣𝑖, 𝑣𝑗 ′ 𝛼 𝑆 𝜃 𝑢1, 𝑢2 ≜ 𝜙 𝑢1 , 𝜓 𝑢2 𝜙 𝑢1 2 ⋅ 𝜓 𝑢2 2 16
- 17. Conclusion • Negative pair 없이 representation 학습 • 하지만 역시나 큰 batch size 필요 • 여러 task에서 State-of-the-art • 하지만 이미 SimCLRv2에게 짐.. • Augmentation option에 대해 robust 함 • 그래도 적합한 augmentation을 찾는 것이 필요함 17
- 18. 감 사 합 니 다 18