Generative artificial intelligence (GenAI) has been developing with many incredible achievements like ChatGPT and Bard. Deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to strong points of deep neural network (DNN) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims to synaptic plasticity of human neuron network, fosters generation ability of DGM which produces surprised results with support of statistical flexibility. Two popular approaches in DGM are Variational Autoencoders (VAE) and Generative Adversarial Network (GAN). Both VAE and GAN have their own strong points although they share and imply underline theory of statistics as well as incredible complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. In this research, I try to unify VAE and GAN into a consistent and consolidated model called Adversarial Variational Autoencoders (AVA) in which VAE and GAN complement each other, for instance, VAE is good at generator by encoding data via excellent ideology of Kullback-Leibler divergence and GAN is a significantly important method to assess reliability of data which is realistic or fake. In other words, AVA aims to improve accuracy of generative models, besides AVA extends function of simple generative models. In methodology this research focuses on combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.
Adversarial Variational Autoencoders to extend and improve generative model -...Loc Nguyen
Generative artificial intelligence (GenAI) has been developing with many incredible achievements like ChatGPT and Bard. Deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to strong points of deep neural network (DNN) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims to synaptic plasticity of human neuron network, fosters generation ability of DGM which produces surprised results with support of statistical flexibility. Two popular approaches in DGM are Variational Autoencoders (VAE) and Generative Adversarial Network (GAN). Both VAE and GAN have their own strong points although they share and imply underline theory of statistics as well as incredible complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. In this research, I try to unify VAE and GAN into a consistent and consolidated model called Adversarial Variational Autoencoders (AVA) in which VAE and GAN complement each other, for instance, VAE is a good data generator by encoding data via excellent ideology of Kullback-Leibler divergence and GAN is a significantly important method to assess reliability of data which is realistic or fake. In other words, AVA aims to improve accuracy of generative models, besides AVA extends function of simple generative models. In methodology this research focuses on combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.
This document introduces an R package called PSF that implements a Pattern Sequence based Forecasting (PSF) algorithm for univariate time series forecasting. The PSF algorithm clusters time series data and then predicts future values based on identifying repeating patterns of clusters. The PSF package contains functions that perform the main steps of the PSF algorithm, including selecting the optimal number of clusters, selecting the optimal window size, and making predictions for a given window size and number of clusters. The package aims to promote and simplify the use of the PSF algorithm for time series forecasting.
This document summarizes a research paper that analyzed the co-occurrence problem using MapReduce on AWS datasets. It compared the pairs and stripes approaches for calculating a co-occurrence matrix on different sized datasets and cluster node configurations. The stripes approach had significantly better performance, taking half the time of the pairs approach. Further optimizations like using a combiner and pre-processing the data were suggested to improve efficiency even more.
Comparison of Cost Estimation Methods using Hybrid Artificial Intelligence on...IJERA Editor
Cost estimating at schematic design stage as the basis of project evaluation, engineering design, and cost
management, plays an important role in project decision under a limited definition of scope and constraints in
available information and time, and the presence of uncertainties. The purpose of this study is to compare the
performance of cost estimation models of two different hybrid artificial intelligence approaches: regression
analysis-adaptive neuro fuzzy inference system (RANFIS) and case based reasoning-genetic algorithm (CBRGA)
techniques. The models were developed based on the same 50 low-cost apartment project datasets in
Indonesia. Tested on another five testing data, the models were proven to perform very well in term of accuracy.
A CBR-GA model was found to be the best performer but suffered from disadvantage of needing 15 cost drivers
if compared to only 4 cost drivers required by RANFIS for on-par performance.
This document discusses using a particle swarm optimization algorithm to solve the K-node set reliability optimization problem for distributed computing systems. The K-node set reliability optimization problem aims to maximize the reliability of a subset of k nodes in a distributed computing system while satisfying a specified capacity constraint. It presents the problem formulation and describes particle swarm optimization, a metaheuristic optimization technique inspired by swarm intelligence. The proposed algorithm applies a discrete particle swarm optimization approach to solve the K-node set reliability optimization problem, which is demonstrated on an example distributed computing system topology.
Node classification with graph neural network based centrality measures and f...IJECEIAES
Graph neural networks (GNNs) are a new topic of research in data science where data structure graphs are used as important components for developing and training neural networks. GNN always learns the weight importance of the neighbor for perform message aggregation in which the feature vectors of all neighbors are aggregated without considering whether the features are useful or not. Using such more informative features positively affect the performance of the GNN model. So, in this paper i) after selecting a subset of features to define important node features, we present new graph features’ explanation methods based on graph centrality measures to capture rich information and determine the most important node in a network. Through our experiments, we find that selecting certain subsets of these features and adding other features based on centrality measure can lead to better performance across a variety of datasets and ii) we introduce a major design strategy for graph neural networks. Specifically, we suggest using batch renormalization as normalization over GNN layers. Combining these techniques, representing features based on centrality measures that passed to multilayer perceptron (MLP) layer which is then passed to adjusted GNN layer, the proposed model achieves greater accuracy than modern GNN models.
An Experiment with Sparse Field and Localized Region Based Active Contour Int...CSCJournals
This paper discusses various experiments conducted on different types of Level Sets interactive segmentation techniques using Matlab software, on select images. The objective is to assess the effectiveness on specific natural images, which have complex image composition in terms of intensity, colour mix, indistinct object boundary, low contrast, etc. Besides visual assessment, measures such as Jaccard Index, Dice Coefficient and Hausdorrf Distance have been computed to assess the accuracy of these techniques, between segmented and ground truth images. This paper particularly discusses Sparse Field Matrix and Localized Region Based Active Contours, both based on Level Sets. These techniques were not found to be effective where object boundary is not very distinct and/or has low contrast with background. Also, the techniques were ineffective on such images where foreground object stretches up to the image boundary.
A survey on methods and applications of meta-learning with GNNsShreya Goyal
This survey paper has provided a comprehensive review of works that are a combination of graph neural networks (GNNs) and meta-learning. They have also provided a thorough review, summary of methods, and applications in these categories. The application of meta-learning to GNNs is a growing and exciting field; many graph problems will benefit immensely from the combination of the two approaches.
Adversarial Variational Autoencoders to extend and improve generative model -...Loc Nguyen
Generative artificial intelligence (GenAI) has been developing with many incredible achievements like ChatGPT and Bard. Deep generative model (DGM) is a branch of GenAI, which is preeminent in generating raster data such as image and sound due to strong points of deep neural network (DNN) in inference and recognition. The built-in inference mechanism of DNN, which simulates and aims to synaptic plasticity of human neuron network, fosters generation ability of DGM which produces surprised results with support of statistical flexibility. Two popular approaches in DGM are Variational Autoencoders (VAE) and Generative Adversarial Network (GAN). Both VAE and GAN have their own strong points although they share and imply underline theory of statistics as well as incredible complex via hidden layers of DNN when DNN becomes effective encoding/decoding functions without concrete specifications. In this research, I try to unify VAE and GAN into a consistent and consolidated model called Adversarial Variational Autoencoders (AVA) in which VAE and GAN complement each other, for instance, VAE is a good data generator by encoding data via excellent ideology of Kullback-Leibler divergence and GAN is a significantly important method to assess reliability of data which is realistic or fake. In other words, AVA aims to improve accuracy of generative models, besides AVA extends function of simple generative models. In methodology this research focuses on combination of applied mathematical concepts and skillful techniques of computer programming in order to implement and solve complicated problems as simply as possible.
This document introduces an R package called PSF that implements a Pattern Sequence based Forecasting (PSF) algorithm for univariate time series forecasting. The PSF algorithm clusters time series data and then predicts future values based on identifying repeating patterns of clusters. The PSF package contains functions that perform the main steps of the PSF algorithm, including selecting the optimal number of clusters, selecting the optimal window size, and making predictions for a given window size and number of clusters. The package aims to promote and simplify the use of the PSF algorithm for time series forecasting.
This document summarizes a research paper that analyzed the co-occurrence problem using MapReduce on AWS datasets. It compared the pairs and stripes approaches for calculating a co-occurrence matrix on different sized datasets and cluster node configurations. The stripes approach had significantly better performance, taking half the time of the pairs approach. Further optimizations like using a combiner and pre-processing the data were suggested to improve efficiency even more.
Comparison of Cost Estimation Methods using Hybrid Artificial Intelligence on...IJERA Editor
Cost estimating at schematic design stage as the basis of project evaluation, engineering design, and cost
management, plays an important role in project decision under a limited definition of scope and constraints in
available information and time, and the presence of uncertainties. The purpose of this study is to compare the
performance of cost estimation models of two different hybrid artificial intelligence approaches: regression
analysis-adaptive neuro fuzzy inference system (RANFIS) and case based reasoning-genetic algorithm (CBRGA)
techniques. The models were developed based on the same 50 low-cost apartment project datasets in
Indonesia. Tested on another five testing data, the models were proven to perform very well in term of accuracy.
A CBR-GA model was found to be the best performer but suffered from disadvantage of needing 15 cost drivers
if compared to only 4 cost drivers required by RANFIS for on-par performance.
This document discusses using a particle swarm optimization algorithm to solve the K-node set reliability optimization problem for distributed computing systems. The K-node set reliability optimization problem aims to maximize the reliability of a subset of k nodes in a distributed computing system while satisfying a specified capacity constraint. It presents the problem formulation and describes particle swarm optimization, a metaheuristic optimization technique inspired by swarm intelligence. The proposed algorithm applies a discrete particle swarm optimization approach to solve the K-node set reliability optimization problem, which is demonstrated on an example distributed computing system topology.
Node classification with graph neural network based centrality measures and f...IJECEIAES
Graph neural networks (GNNs) are a new topic of research in data science where data structure graphs are used as important components for developing and training neural networks. GNN always learns the weight importance of the neighbor for perform message aggregation in which the feature vectors of all neighbors are aggregated without considering whether the features are useful or not. Using such more informative features positively affect the performance of the GNN model. So, in this paper i) after selecting a subset of features to define important node features, we present new graph features’ explanation methods based on graph centrality measures to capture rich information and determine the most important node in a network. Through our experiments, we find that selecting certain subsets of these features and adding other features based on centrality measure can lead to better performance across a variety of datasets and ii) we introduce a major design strategy for graph neural networks. Specifically, we suggest using batch renormalization as normalization over GNN layers. Combining these techniques, representing features based on centrality measures that passed to multilayer perceptron (MLP) layer which is then passed to adjusted GNN layer, the proposed model achieves greater accuracy than modern GNN models.
An Experiment with Sparse Field and Localized Region Based Active Contour Int...CSCJournals
This paper discusses various experiments conducted on different types of Level Sets interactive segmentation techniques using Matlab software, on select images. The objective is to assess the effectiveness on specific natural images, which have complex image composition in terms of intensity, colour mix, indistinct object boundary, low contrast, etc. Besides visual assessment, measures such as Jaccard Index, Dice Coefficient and Hausdorrf Distance have been computed to assess the accuracy of these techniques, between segmented and ground truth images. This paper particularly discusses Sparse Field Matrix and Localized Region Based Active Contours, both based on Level Sets. These techniques were not found to be effective where object boundary is not very distinct and/or has low contrast with background. Also, the techniques were ineffective on such images where foreground object stretches up to the image boundary.
A survey on methods and applications of meta-learning with GNNsShreya Goyal
This survey paper has provided a comprehensive review of works that are a combination of graph neural networks (GNNs) and meta-learning. They have also provided a thorough review, summary of methods, and applications in these categories. The application of meta-learning to GNNs is a growing and exciting field; many graph problems will benefit immensely from the combination of the two approaches.
Similarity-preserving hash for content-based audio retrieval using unsupervis...IJECEIAES
Due to its efficiency in storage and search speed, binary hashing has become an attractive approach for a large audio database search. However, most existing hashing-based methods focus on data-independent scheme where random linear projections or some arithmetic expression are used to construct hash functions. Hence, the binary codes do not preserve the similarity and may degrade the search performance. In this paper, an unsupervised similarity-preserving hashing method for content-based audio retrieval is proposed. Different from data-independent hashing methods, we develop a deep network to learn compact binary codes from multiple hierarchical layers of nonlinear and linear transformations such that the similarity between samples is preserved. The independence and balance properties are included and optimized in the objective function to improve the codes. Experimental results on the Extended Ballroom dataset with 8 genres of 3,000 musical excerpts show that our proposed method significantly outperforms state-ofthe-art data-independent method in both effectiveness and efficiency.
Constructing a classification model is important in machine learning for a particular task. A
classification process involves assigning objects into predefined groups or classes based on a
number of observed attributes related to those objects. Artificial neural network is one of the
classification algorithms which, can be used in many application areas. This paper investigates
the potential of applying the feed forward neural network architecture for the classification of
medical datasets. Migration based differential evolution algorithm (MBDE) is chosen and
applied to feed forward neural network to enhance the learning process and the network
learning is validated in terms of convergence rate and classification accuracy. In this paper,
MBDE algorithm with various migration policies is proposed for classification problems using
medical diagnosis.
MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...cscpconf
Constructing a classification model is important in machine learning for a particular task. A
classification process involves assigning objects into predefined groups or classes based on a
number of observed attributes related to those objects. Artificial neural network is one of the
classification algorithms which, can be used in many application areas. This paper investigates
the potential of applying the feed forward neural network architecture for the classification of
medical datasets. Migration based differential evolution algorithm (MBDE) is chosen and
applied to feed forward neural network to enhance the learning process and the network
learning is validated in terms of convergence rate and classification accuracy. In this paper,
MBDE algorithm with various migration policies is proposed for classification problems using
medical diagnosis.
This document provides an overview of deep feedforward networks. It begins with an example of using a network to solve the XOR problem. It then discusses gradient-based learning and backpropagation. Hidden units with rectified linear activations are commonly used. Deeper networks can more efficiently represent functions and generalize better than shallow networks. Architecture design considerations include width, depth, and number of hidden layers. Backpropagation efficiently computes gradients using the chain rule and dynamic programming.
Application of Hybrid Genetic Algorithm Using Artificial Neural Network in Da...IOSRjournaljce
The main purpose of data mining is to extract knowledge from large amount of data. Artificial Neural network (ANN) has already been applied in a variety of domains with remarkable success. This paper presents the application of hybrid model for stroke disease that integrates Genetic algorithm and back propagation algorithm. Selecting a good subset of features, without sacrificing accuracy, is of great importance for neural networks to be successfully applied to the area. In addition the hybrid model that leads to further improvised categorization, accuracy compared to the result produced by genetic algorithm alone. In this study, a new hybrid model of Neural Networks and Genetic Algorithm (GA) to initialize and optimize the connection weights of ANN so as to improve the performance of the ANN and the same has been applied in a medical problem of predicting stroke disease for verification of the results.
Web image annotation by diffusion maps manifold learning algorithmijfcstjournal
Automatic image annotation is one of the most challenging problems in machine vision areas. The goal of this task is to predict number of keywords automatically for images captured in real data. Many methods are based on visual features in order to calculate similarities between image samples. But the computation cost of these approaches is very high. These methods require many training samples to be stored in memory. To lessen thisburden, a number of techniques have been developed to reduce the number
of features in a dataset. Manifold learning is a popular approach to nonlinear dimensionality reduction. In
this paper, we investigate Diffusion maps manifold learning method for webimage auto-annotation task.Diffusion maps
manifold learning method isused to reduce the dimension of some visual features. Extensive experiments and analysis onNUS-WIDE-LITE web image dataset with
different visual featuresshow how this manifold learning dimensionality reduction method can be applied effectively to image annotation.
This document describes six MATLAB projects available from VenSoft Technologies for the 2014-2015 academic year. It provides the titles, abstracts, publishing details, and index terms for each project. The projects involve topics such as sparse unmixing of hyperspectral data, mixed noise removal, subspace matching pursuit, exploiting spectral a priori information, gradient histogram preservation for image denoising, and image set-based collaborative representation for face recognition.
Comparative Analysis of GANs and VAEs in Generating High-Quality Images: A Ca...IRJET Journal
This document compares Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) for generating images from the MNIST dataset of handwritten digits. Both GANs and VAEs were implemented using TensorFlow and Python. The GAN achieved a lower Frechet Inception Distance of 18.24 compared to the VAE's score of 28.56, indicating its generated images more closely resembled the original MNIST data. However, VAEs have simpler latent space sampling than GANs. While both techniques can produce high-quality digit images, GANs may be better for capturing overall data patterns, while VAEs are more suited for modeling the underlying probability distribution.
Found this paper really interesting. It delves into the learning behaviors of Deep Learning Ensembles and compares them Bayesian Neural Networks, which theoretically does the same thing. This answers why Deep Ensembles Outperform
This document proposes a novel framework called smooth sparse coding for learning sparse representations of data. It incorporates feature similarity or temporal information present in data sets via non-parametric kernel smoothing. The approach constructs codes that represent neighborhoods of samples rather than individual samples, leading to lower reconstruction error. It also proposes using marginal regression rather than lasso for obtaining sparse codes, providing a dramatic speedup of up to two orders of magnitude without sacrificing accuracy. The document contributes a framework for incorporating domain information into sparse coding, sample complexity results for dictionary learning using smooth sparse coding, an efficient marginal regression training procedure, and successful application to classification tasks with improved accuracy and speed.
This document summarizes research applying deep learning techniques to predict epigenomic enhancer regions from DNA sequences. Four models - variations of Basset, DeepSea, DanQ, and a custom Greenside-Basset model - were trained on a dataset from the NIH Roadmap Epigenomics Mapping Consortium to label 1000 base pair sequences as active or inactive for 57 cell types. The Basset variation achieved the best balanced accuracy of 72.8% and had the highest precision and F1 scores, though it still weighted negative examples heavily, as precision was not strong. Experimenting with different embeddings, loss functions, and architectures helped narrow the models best suited for this sequence labeling task.
Clustering using kernel entropy principal component analysis and variable ker...IJECEIAES
Clustering as unsupervised learning method is the mission of dividing data objects into clusters with common characteristics. In the present paper, we introduce an enhanced technique of the existing EPCA data transformation method. Incorporating the kernel function into the EPCA, the input space can be mapped implicitly into a high-dimensional of feature space. Then, the Shannon’s entropy estimated via the inertia provided by the contribution of every mapped object in data is the key measure to determine the optimal extracted features space. Our proposed method performs very well the clustering algorithm of the fast search of clusters’ centers based on the local densities’ computing. Experimental results disclose that the approach is feasible and efficient on the performance query.
A Design and Solving LPP Method for Binary Linear Programming Problem Using D...ijistjournal
Molecular computing is a discipline that aims at harnessing individual molecules for computational purposes. This paper presents the applied Mathematical sciences using DNA molecules. The Major achievements are outlined the potential advances and the challenges for the practitioners in the foreseeable future. The Binary Optimization in Linear Programming is an intensive research area in the field of DNA Computing. This paper presents a research on design and implementation method to solve an Binary Linear Programming Problem using DNA computing. The DNA sequences of length directly represent all possible combinations in different boxes. An Hybridization is performed to form double strand molecules according to its length to visualize the optimal solution based on fluorescent material . Here Maximization Problem is converted into DNA computable form and a complementary are found to solve the problem and the optimal solution is suggested as per the constraints stipulated by the problem.
A Design and Solving LPP Method for Binary Linear Programming Problem Using D...ijistjournal
This document describes a method for solving binary linear programming problems using DNA computing. It involves representing the variables and constraints of the problem as DNA strands. All possible combinations of 0s and 1s for the variables are generated as DNA sequences. Constraints are applied by adding complementary strands tagged with fluorescent markers. Strands that satisfy the constraints will hybridize, revealing feasible solutions. The value of the objective function is then calculated for each feasible solution to determine the optimal one. A hypothetical example problem is worked through step-by-step to demonstrate the method.
SOLVING OPTIMAL COMPONENTS ASSIGNMENT PROBLEM FOR A MULTISTATE NETWORK USING ...ijmnct
The document summarizes research on solving the optimal components assignment problem for a multistate network using fuzzy optimization. It discusses how the problem can be formulated as a fuzzy linear program by defining fuzzy membership functions for the objectives of maximizing reliability, minimizing total lead time, and minimizing total cost. The paper then proposes using a genetic algorithm combined with fuzzy linear programming to find component assignments that maximize the fuzzy objective membership degree.
SOLVING OPTIMAL COMPONENTS ASSIGNMENT PROBLEM FOR A MULTISTATE NETWORK USING ...ijmnct
Optimal components assignment problem subject to system reliability, total lead-time, and total cost
constraints is studied in this paper. The problem is formulated as fuzzy linear problem using fuzzy
membership functions. An approach based on genetic algorithm with fuzzy optimization to sole the
presented problem. The optimal solution found by the proposed approach is characterized by maximum
reliability, minimum total cost and minimum total lead-time. The proposed approach is tested on different
examples taken from the literature to illustrate its efficiency in comparison with other previous methods
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals
This document presents a method for using a multi-layered feed-forward neural network (MLFNN) architecture as a bidirectional associative memory (BAM) for function approximation. It proposes applying the backpropagation algorithm in two phases - first in the forward direction, then in the backward direction - which allows the MLFNN to work like a BAM. Simulation results show that this two-phase backpropagation algorithm achieves convergence faster than standard backpropagation when approximating the sine function, demonstrating that the MLFNN architecture is better suited for function approximation when trained this way.
CAUSALITY LEARNING WITH WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKSgerogepatton
This document summarizes a research paper that proposes a new model called DAG-WGAN for causal structure learning. DAG-WGAN combines a Wasserstein generative adversarial network with an auto-encoder architecture. It learns causal structures by generating synthetic data that matches the real data distribution. The model incorporates an acyclicity constraint to ensure the learned causal graphs are directed acyclic graphs. Experimental results show DAG-WGAN performs better than other models at learning causal structures from high-dimensional tabular data, demonstrating the benefit of using the Wasserstein distance metric in the generative process of causal structure learning.
CAUSALITY LEARNING WITH WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKSijaia
Conventional methods for causal structure learning from data face significant challenges due to
combinatorial search space. Recently, the problem has been formulated into a continuous optimization
framework with an acyclicity constraint to learn Directed Acyclic Graphs (DAGs). Such a framework
allows the utilization of deep generative models for causal structure learning to better capture the relations
between data sample distributions and DAGs. However, so far no study has experimented with the use of
Wasserstein distance in the context of causal structure learning. Our model named DAG-WGAN combines
the Wasserstein-based adversarial loss with an acyclicity constraint in an auto-encoder architecture. It
simultaneously learns causal structures while improving its data generation capability. We compare the
performance of DAG-WGAN with other models that do not involve the Wasserstein metric in order to
identify its contribution to causal structure learning. Our model performs better with high cardinality data
according to our experiments.
Conditional mixture model and its application for regression modelLoc Nguyen
Expectation maximization (EM) algorithm is a powerful mathematical tool for estimating statistical parameter when data sample contains hidden part and observed part. EM is applied to learn finite mixture model in which the whole distribution of observed variable is average sum of partial distributions. Coverage ratio of every partial distribution is specified by the probability of hidden variable. An application of mixture model is soft clustering in which cluster is modeled by hidden variable whereas each data point can be assigned to more than one cluster and degree of such assignment is represented by the probability of hidden variable. However, such probability in traditional mixture model is simplified as a parameter, which can cause loss of valuable information. Therefore, in this research I propose a so-called conditional mixture model (CMM) in which the probability of hidden variable is modeled as a full probabilistic density function (PDF) that owns individual parameter. CMM aims to extend mixture model. I also propose an application of CMM which is called adaptive regression model (ARM). Traditional regression model is effective when data sample is scattered equally. If data points are grouped into clusters, regression model tries to learn a unified regression function which goes through all data points. Obviously, such unified function is not effective to evaluate response variable based on grouped data points. The concept “adaptive” of ARM means that ARM solves the ineffectiveness problem by selecting the best cluster of data points firstly and then evaluating response variable within such best cluster. In orther words, ARM reduces estimation space of regression model so as to gain high accuracy in calculation.
Keywords: expectation maximization (EM) algorithm, finite mixture model, conditional mixture model, regression model, adaptive regression model (ARM).
Nghịch dân chủ luận (tổng quan về dân chủ và thể chế chính trị liên quan đến ...Loc Nguyen
Vũ trụ có vật chất và phản vật chất, xã hội có xung đột và hữu hão để phát triển và suy tàn rồi suy tàn và phát triển. Tôi dựa vào đó để biện minh cho một bài viết có tính chất phản động nghịch chuyển thời cuộc nhưng bạn đọc sẽ tự tìm ra ý nghĩa bất ly của các hình thái xã hội. Ngoài ra bài viết này không đi sâu vào nghiên cứu pháp luật, chỉ đưa ra một cách nhìn tổng quan về dân chủ và thể chế chính trị liên quan đến triết học và tôn giáo, mà theo đó đóng góp của bài viết là khái niệm “nương tạm” của tư pháp không thật sự từ bầu cử và cũng không thật sự từ bổ nhiệm.
More Related Content
Similar to Adversarial Variational Autoencoders to extend and improve generative model
Similarity-preserving hash for content-based audio retrieval using unsupervis...IJECEIAES
Due to its efficiency in storage and search speed, binary hashing has become an attractive approach for a large audio database search. However, most existing hashing-based methods focus on data-independent scheme where random linear projections or some arithmetic expression are used to construct hash functions. Hence, the binary codes do not preserve the similarity and may degrade the search performance. In this paper, an unsupervised similarity-preserving hashing method for content-based audio retrieval is proposed. Different from data-independent hashing methods, we develop a deep network to learn compact binary codes from multiple hierarchical layers of nonlinear and linear transformations such that the similarity between samples is preserved. The independence and balance properties are included and optimized in the objective function to improve the codes. Experimental results on the Extended Ballroom dataset with 8 genres of 3,000 musical excerpts show that our proposed method significantly outperforms state-ofthe-art data-independent method in both effectiveness and efficiency.
Constructing a classification model is important in machine learning for a particular task. A
classification process involves assigning objects into predefined groups or classes based on a
number of observed attributes related to those objects. Artificial neural network is one of the
classification algorithms which, can be used in many application areas. This paper investigates
the potential of applying the feed forward neural network architecture for the classification of
medical datasets. Migration based differential evolution algorithm (MBDE) is chosen and
applied to feed forward neural network to enhance the learning process and the network
learning is validated in terms of convergence rate and classification accuracy. In this paper,
MBDE algorithm with various migration policies is proposed for classification problems using
medical diagnosis.
MEDICAL DIAGNOSIS CLASSIFICATION USING MIGRATION BASED DIFFERENTIAL EVOLUTION...cscpconf
Constructing a classification model is important in machine learning for a particular task. A
classification process involves assigning objects into predefined groups or classes based on a
number of observed attributes related to those objects. Artificial neural network is one of the
classification algorithms which, can be used in many application areas. This paper investigates
the potential of applying the feed forward neural network architecture for the classification of
medical datasets. Migration based differential evolution algorithm (MBDE) is chosen and
applied to feed forward neural network to enhance the learning process and the network
learning is validated in terms of convergence rate and classification accuracy. In this paper,
MBDE algorithm with various migration policies is proposed for classification problems using
medical diagnosis.
This document provides an overview of deep feedforward networks. It begins with an example of using a network to solve the XOR problem. It then discusses gradient-based learning and backpropagation. Hidden units with rectified linear activations are commonly used. Deeper networks can more efficiently represent functions and generalize better than shallow networks. Architecture design considerations include width, depth, and number of hidden layers. Backpropagation efficiently computes gradients using the chain rule and dynamic programming.
Application of Hybrid Genetic Algorithm Using Artificial Neural Network in Da...IOSRjournaljce
The main purpose of data mining is to extract knowledge from large amount of data. Artificial Neural network (ANN) has already been applied in a variety of domains with remarkable success. This paper presents the application of hybrid model for stroke disease that integrates Genetic algorithm and back propagation algorithm. Selecting a good subset of features, without sacrificing accuracy, is of great importance for neural networks to be successfully applied to the area. In addition the hybrid model that leads to further improvised categorization, accuracy compared to the result produced by genetic algorithm alone. In this study, a new hybrid model of Neural Networks and Genetic Algorithm (GA) to initialize and optimize the connection weights of ANN so as to improve the performance of the ANN and the same has been applied in a medical problem of predicting stroke disease for verification of the results.
Web image annotation by diffusion maps manifold learning algorithmijfcstjournal
Automatic image annotation is one of the most challenging problems in machine vision areas. The goal of this task is to predict number of keywords automatically for images captured in real data. Many methods are based on visual features in order to calculate similarities between image samples. But the computation cost of these approaches is very high. These methods require many training samples to be stored in memory. To lessen thisburden, a number of techniques have been developed to reduce the number
of features in a dataset. Manifold learning is a popular approach to nonlinear dimensionality reduction. In
this paper, we investigate Diffusion maps manifold learning method for webimage auto-annotation task.Diffusion maps
manifold learning method isused to reduce the dimension of some visual features. Extensive experiments and analysis onNUS-WIDE-LITE web image dataset with
different visual featuresshow how this manifold learning dimensionality reduction method can be applied effectively to image annotation.
This document describes six MATLAB projects available from VenSoft Technologies for the 2014-2015 academic year. It provides the titles, abstracts, publishing details, and index terms for each project. The projects involve topics such as sparse unmixing of hyperspectral data, mixed noise removal, subspace matching pursuit, exploiting spectral a priori information, gradient histogram preservation for image denoising, and image set-based collaborative representation for face recognition.
Comparative Analysis of GANs and VAEs in Generating High-Quality Images: A Ca...IRJET Journal
This document compares Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) for generating images from the MNIST dataset of handwritten digits. Both GANs and VAEs were implemented using TensorFlow and Python. The GAN achieved a lower Frechet Inception Distance of 18.24 compared to the VAE's score of 28.56, indicating its generated images more closely resembled the original MNIST data. However, VAEs have simpler latent space sampling than GANs. While both techniques can produce high-quality digit images, GANs may be better for capturing overall data patterns, while VAEs are more suited for modeling the underlying probability distribution.
Found this paper really interesting. It delves into the learning behaviors of Deep Learning Ensembles and compares them Bayesian Neural Networks, which theoretically does the same thing. This answers why Deep Ensembles Outperform
This document proposes a novel framework called smooth sparse coding for learning sparse representations of data. It incorporates feature similarity or temporal information present in data sets via non-parametric kernel smoothing. The approach constructs codes that represent neighborhoods of samples rather than individual samples, leading to lower reconstruction error. It also proposes using marginal regression rather than lasso for obtaining sparse codes, providing a dramatic speedup of up to two orders of magnitude without sacrificing accuracy. The document contributes a framework for incorporating domain information into sparse coding, sample complexity results for dictionary learning using smooth sparse coding, an efficient marginal regression training procedure, and successful application to classification tasks with improved accuracy and speed.
This document summarizes research applying deep learning techniques to predict epigenomic enhancer regions from DNA sequences. Four models - variations of Basset, DeepSea, DanQ, and a custom Greenside-Basset model - were trained on a dataset from the NIH Roadmap Epigenomics Mapping Consortium to label 1000 base pair sequences as active or inactive for 57 cell types. The Basset variation achieved the best balanced accuracy of 72.8% and had the highest precision and F1 scores, though it still weighted negative examples heavily, as precision was not strong. Experimenting with different embeddings, loss functions, and architectures helped narrow the models best suited for this sequence labeling task.
Clustering using kernel entropy principal component analysis and variable ker...IJECEIAES
Clustering as unsupervised learning method is the mission of dividing data objects into clusters with common characteristics. In the present paper, we introduce an enhanced technique of the existing EPCA data transformation method. Incorporating the kernel function into the EPCA, the input space can be mapped implicitly into a high-dimensional of feature space. Then, the Shannon’s entropy estimated via the inertia provided by the contribution of every mapped object in data is the key measure to determine the optimal extracted features space. Our proposed method performs very well the clustering algorithm of the fast search of clusters’ centers based on the local densities’ computing. Experimental results disclose that the approach is feasible and efficient on the performance query.
A Design and Solving LPP Method for Binary Linear Programming Problem Using D...ijistjournal
Molecular computing is a discipline that aims at harnessing individual molecules for computational purposes. This paper presents the applied Mathematical sciences using DNA molecules. The Major achievements are outlined the potential advances and the challenges for the practitioners in the foreseeable future. The Binary Optimization in Linear Programming is an intensive research area in the field of DNA Computing. This paper presents a research on design and implementation method to solve an Binary Linear Programming Problem using DNA computing. The DNA sequences of length directly represent all possible combinations in different boxes. An Hybridization is performed to form double strand molecules according to its length to visualize the optimal solution based on fluorescent material . Here Maximization Problem is converted into DNA computable form and a complementary are found to solve the problem and the optimal solution is suggested as per the constraints stipulated by the problem.
A Design and Solving LPP Method for Binary Linear Programming Problem Using D...ijistjournal
This document describes a method for solving binary linear programming problems using DNA computing. It involves representing the variables and constraints of the problem as DNA strands. All possible combinations of 0s and 1s for the variables are generated as DNA sequences. Constraints are applied by adding complementary strands tagged with fluorescent markers. Strands that satisfy the constraints will hybridize, revealing feasible solutions. The value of the objective function is then calculated for each feasible solution to determine the optimal one. A hypothetical example problem is worked through step-by-step to demonstrate the method.
SOLVING OPTIMAL COMPONENTS ASSIGNMENT PROBLEM FOR A MULTISTATE NETWORK USING ...ijmnct
The document summarizes research on solving the optimal components assignment problem for a multistate network using fuzzy optimization. It discusses how the problem can be formulated as a fuzzy linear program by defining fuzzy membership functions for the objectives of maximizing reliability, minimizing total lead time, and minimizing total cost. The paper then proposes using a genetic algorithm combined with fuzzy linear programming to find component assignments that maximize the fuzzy objective membership degree.
SOLVING OPTIMAL COMPONENTS ASSIGNMENT PROBLEM FOR A MULTISTATE NETWORK USING ...ijmnct
Optimal components assignment problem subject to system reliability, total lead-time, and total cost
constraints is studied in this paper. The problem is formulated as fuzzy linear problem using fuzzy
membership functions. An approach based on genetic algorithm with fuzzy optimization to sole the
presented problem. The optimal solution found by the proposed approach is characterized by maximum
reliability, minimum total cost and minimum total lead-time. The proposed approach is tested on different
examples taken from the literature to illustrate its efficiency in comparison with other previous methods
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals
This document presents a method for using a multi-layered feed-forward neural network (MLFNN) architecture as a bidirectional associative memory (BAM) for function approximation. It proposes applying the backpropagation algorithm in two phases - first in the forward direction, then in the backward direction - which allows the MLFNN to work like a BAM. Simulation results show that this two-phase backpropagation algorithm achieves convergence faster than standard backpropagation when approximating the sine function, demonstrating that the MLFNN architecture is better suited for function approximation when trained this way.
CAUSALITY LEARNING WITH WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKSgerogepatton
This document summarizes a research paper that proposes a new model called DAG-WGAN for causal structure learning. DAG-WGAN combines a Wasserstein generative adversarial network with an auto-encoder architecture. It learns causal structures by generating synthetic data that matches the real data distribution. The model incorporates an acyclicity constraint to ensure the learned causal graphs are directed acyclic graphs. Experimental results show DAG-WGAN performs better than other models at learning causal structures from high-dimensional tabular data, demonstrating the benefit of using the Wasserstein distance metric in the generative process of causal structure learning.
CAUSALITY LEARNING WITH WASSERSTEIN GENERATIVE ADVERSARIAL NETWORKSijaia
Conventional methods for causal structure learning from data face significant challenges due to
combinatorial search space. Recently, the problem has been formulated into a continuous optimization
framework with an acyclicity constraint to learn Directed Acyclic Graphs (DAGs). Such a framework
allows the utilization of deep generative models for causal structure learning to better capture the relations
between data sample distributions and DAGs. However, so far no study has experimented with the use of
Wasserstein distance in the context of causal structure learning. Our model named DAG-WGAN combines
the Wasserstein-based adversarial loss with an acyclicity constraint in an auto-encoder architecture. It
simultaneously learns causal structures while improving its data generation capability. We compare the
performance of DAG-WGAN with other models that do not involve the Wasserstein metric in order to
identify its contribution to causal structure learning. Our model performs better with high cardinality data
according to our experiments.
Similar to Adversarial Variational Autoencoders to extend and improve generative model (20)
Conditional mixture model and its application for regression modelLoc Nguyen
Expectation maximization (EM) algorithm is a powerful mathematical tool for estimating statistical parameter when data sample contains hidden part and observed part. EM is applied to learn finite mixture model in which the whole distribution of observed variable is average sum of partial distributions. Coverage ratio of every partial distribution is specified by the probability of hidden variable. An application of mixture model is soft clustering in which cluster is modeled by hidden variable whereas each data point can be assigned to more than one cluster and degree of such assignment is represented by the probability of hidden variable. However, such probability in traditional mixture model is simplified as a parameter, which can cause loss of valuable information. Therefore, in this research I propose a so-called conditional mixture model (CMM) in which the probability of hidden variable is modeled as a full probabilistic density function (PDF) that owns individual parameter. CMM aims to extend mixture model. I also propose an application of CMM which is called adaptive regression model (ARM). Traditional regression model is effective when data sample is scattered equally. If data points are grouped into clusters, regression model tries to learn a unified regression function which goes through all data points. Obviously, such unified function is not effective to evaluate response variable based on grouped data points. The concept “adaptive” of ARM means that ARM solves the ineffectiveness problem by selecting the best cluster of data points firstly and then evaluating response variable within such best cluster. In orther words, ARM reduces estimation space of regression model so as to gain high accuracy in calculation.
Keywords: expectation maximization (EM) algorithm, finite mixture model, conditional mixture model, regression model, adaptive regression model (ARM).
Nghịch dân chủ luận (tổng quan về dân chủ và thể chế chính trị liên quan đến ...Loc Nguyen
Vũ trụ có vật chất và phản vật chất, xã hội có xung đột và hữu hão để phát triển và suy tàn rồi suy tàn và phát triển. Tôi dựa vào đó để biện minh cho một bài viết có tính chất phản động nghịch chuyển thời cuộc nhưng bạn đọc sẽ tự tìm ra ý nghĩa bất ly của các hình thái xã hội. Ngoài ra bài viết này không đi sâu vào nghiên cứu pháp luật, chỉ đưa ra một cách nhìn tổng quan về dân chủ và thể chế chính trị liên quan đến triết học và tôn giáo, mà theo đó đóng góp của bài viết là khái niệm “nương tạm” của tư pháp không thật sự từ bầu cử và cũng không thật sự từ bổ nhiệm.
A Novel Collaborative Filtering Algorithm by Bit Mining Frequent ItemsetsLoc Nguyen
Collaborative filtering (CF) is a popular technique in recommendation study. Concretely, items which are recommended to user are determined by surveying her/his communities. There are two main CF approaches, which are memory-based and model-based. I propose a new CF model-based algorithm by mining frequent itemsets from rating database. Hence items which belong to frequent itemsets are recommended to user. My CF algorithm gives immediate response because the mining task is performed at offline process-mode. I also propose another so-called Roller algorithm for improving the process of mining frequent itemsets. Roller algorithm is implemented by heuristic assumption “The larger the support of an item is, the higher it’s likely that this item will occur in some frequent itemset”. It models upon doing white-wash task, which rolls a roller on a wall in such a way that is capable of picking frequent itemsets. Moreover I provide enhanced techniques such as bit representation, bit matching and bit mining in order to speed up recommendation process. These techniques take advantages of bitwise operations (AND, NOT) so as to reduce storage space and make algorithms run faster.
Simple image deconvolution based on reverse image convolution and backpropaga...Loc Nguyen
Deconvolution task is not important in convolutional neural network (CNN) because it is not imperative to recover convoluted image when convolutional layer is important to extract features. However, the deconvolution task is useful in some cases of inspecting and reflecting a convolutional filter as well as trying to improve a generated image when information loss is not serious with regard to trade-off of information loss and specific features such as edge detection and sharpening. This research proposes a duplicated and reverse process of recovering a filtered image. Firstly, source layer and target layer are reversed in accordance with traditional image convolution so as to train the convolutional filter. Secondly, the trained filter is reversed again to derive a deconvolutional operator for recovering the filtered image. The reverse process is associated with backpropagation algorithm which is most popular in learning neural network. Experimental results show that the proposed technique in this research is better to learn the filters that focus on discovering pixel differences. Therefore, the main contribution of this research is to inspect convolutional filters from data.
Technological Accessibility: Learning Platform Among Senior High School StudentsLoc Nguyen
The document discusses technological accessibility among senior high school students. It covers several topics:
- Returning to nature through technology and ensuring technology accessibility is a life skill strengthened by advantages while avoiding problems for youth.
- Solutions to the dilemma of technology accessibility include letting youth develop independently, using security controls, and encouraging compassion.
- Researching career is optional but following passion and balancing positive and negative emotions can contribute to success.
The document discusses how engineering and technology can be used for social impact. It makes three key points:
1. Technology stems from knowledge and science, but assessing the value of knowledge is difficult. However, the importance of technology is clear through its fruits. Education fosters gaining knowledge and leads to technological development.
2. While technology increases wealth, it also widens inequality gaps. However, technology can indirectly address inequality through education by providing universal access to learning and bringing universities to people everywhere.
3. Diversity among countries should be embraced, and late innovations building on existing technologies can be particularly impactful, as was the strategy of a racing champion who overtook opponents in the final round through close observation and
Harnessing Technology for Research EducationLoc Nguyen
The document discusses harnessing technology for research and education. It notes some paradoxes of technology, such as how it can both increase inequality but also help solve it through education. It discusses the future of education being supported by distance learning tools, AI, and virtual/augmented reality. Open universities are seen as an intermediate step towards "home universities" and as a place where students can become lifelong learners and teachers. Understanding, emotion, and compassion are discussed as being more important than just remembering facts. The document ends by discussing connecting different subjects and technologies like a jigsaw puzzle.
Future of education with support of technologyLoc Nguyen
The document discusses the future of education with the support of technology. It notes that while technology deepens inequality by benefiting the rich, it can indirectly address inequality through education by allowing anyone to access knowledge and study from anywhere. Emerging forms of educational support include distance learning using tools like Zoom, artificial intelligence assistants, and virtual/augmented reality. Blended teaching combining different methods is emphasized. Understanding is seen as more important than memorization, and education should foster understanding, emotional development, and compassion through nature-based activities. The document argues technology can help education promote love by connecting various topics through both fusion and by assembling them like a jigsaw puzzle.
The document discusses generative artificial intelligence (AI) and its applications, using the metaphor of a dragon's flight. It introduces digital generative AI models that can generate images, sounds, and motions. As an example, it describes a model that could generate possible orbits or movements for a dragon and tiger in an image along with a bamboo background. The document then discusses how generative AI could be applied creatively in education to generate new learning materials and methods. It argues that while technology and societies are changing rapidly, climate change is a unifying challenge that nations must work together to overcome.
Learning dyadic data and predicting unaccomplished co-occurrent values by mix...Loc Nguyen
Dyadic data which is also called co-occurrence data (COD) contains co-occurrences of objects. Searching for statistical models to represent dyadic data is necessary. Fortunately, finite mixture model is a solid statistical model to learn and make inference on dyadic data because mixture model is built smoothly and reliably by expectation maximization (EM) algorithm which is suitable to inherent spareness of dyadic data. This research summarizes mixture models for dyadic data. When each co-occurrence in dyadic data is associated with a value, there are many unaccomplished values because a lot of co-occurrences are inexistent. In this research, these unaccomplished values are estimated as mean (expectation) of random variable given partial probabilistic distributions inside dyadic mixture model.
Machine learning forks into three main branches such as supervised learning, unsupervised learning, and reinforcement learning where reinforcement learning is much potential to artificial intelligence (AI) applications because it solves real problems by progressive process in which possible solutions are improved and finetuned continuously. The progressive approach, which reflects ability of adaptation, is appropriate to the real world where most events occur and change continuously and unexpectedly. Moreover, data is getting too huge for supervised learning and unsupervised learning to draw valuable knowledge from such huge data at one time. Bayesian optimization (BO) models an optimization problem as a probabilistic form called surrogate model and then directly maximizes an acquisition function created from such surrogate model in order to maximize implicitly and indirectly the target function for finding out solution of the optimization problem. A popular surrogate model is Gaussian process regression model. The process of maximizing acquisition function is based on updating posterior probability of surrogate model repeatedly, which is improved after every iteration. Taking advantages of acquisition function or utility function is also common in decision theory but the semantic meaning behind BO is that BO solves problems by progressive and adaptive approach via updating surrogate model from a small piece of data at each time, according to ideology of reinforcement learning. Undoubtedly, BO is a reinforcement learning algorithm with many potential applications and thus it is surveyed in this research with attention to its mathematical ideas. Moreover, the solution of optimization problem is important to not only applied mathematics but also AI.
Support vector machine is a powerful machine learning method in data classification. Using it for applied researches is easy but comprehending it for further development requires a lot of efforts. This report is a tutorial on support vector machine with full of mathematical proofs and example, which help researchers to understand it by the fastest way from theory to practice. The report focuses on theory of optimization which is the base of support vector machine.
A Proposal of Two-step Autoregressive ModelLoc Nguyen
Autoregressive (AR) model and conditional autoregressive (CAR) model are specific regressive models in which independent variables and dependent variable imply the same object. They are powerful statistical tools to predict values based on correlation of time domain and space domain, which are useful in epidemiology analysis. In this research, I combine them by the simple way in which AR and CAR is estimated in two separate steps so as to cover time domain and space domain in spatial-temporal data analysis. Moreover, I integrate logistic model into CAR model, which aims to improve competence of autoregressive models.
Extreme bound analysis based on correlation coefficient for optimal regressio...Loc Nguyen
Regression analysis is an important tool in statistical analysis, in which there is a demand of discovering essential independent variables among many other ones, especially in case that there is a huge number of random variables. Extreme bound analysis is a powerful approach to extract such important variables called robust regressors. In this research, a so-called Regressive Expectation Maximization with RObust regressors (REMRO) algorithm is proposed as an alternative method beside other probabilistic methods for analyzing robust variables. By the different ideology from other probabilistic methods, REMRO searches for robust regressors forming optimal regression model and sorts them according to descending ordering given their fitness values determined by two proposed concepts of local correlation and global correlation. Local correlation represents sufficient explanatories to possible regressive models and global correlation reflects independence level and stand-alone capacity of regressors. Moreover, REMRO can resist incomplete data because it applies Regressive Expectation Maximization (REM) algorithm into filling missing values by estimated values based on ideology of expectation maximization (EM) algorithm. From experimental results, REMRO is more accurate for modeling numeric regressors than traditional probabilistic methods like Sala-I-Martin method but REMRO cannot be applied in case of nonnumeric regression model yet in this research.
The document proposes a "jagged stock investment" strategy where stocks are repeatedly purchased at intervals where the price is increasing, similar to rises and falls in a saw blade. It shows mathematically that this strategy can achieve equal or higher returns than bank depositing if the average growth rate and interest rate are the same. The strategy models purchasing a share and its replications over time periods where each replication is bought using the profits from the previous rise. It provides formulas to calculate the overall value and return on investment from this jagged stock investment approach.
This document presents a study on minima distribution and its application to explaining the convergence and convergence speed of optimization algorithms. The author proposes weaker conditions for the convergence and monotonicity properties of minima distribution compared to previous work. Specifically, the convergence condition requires the function τ(x) to be positive and non-minimizing everywhere except at minimizers of the target function f(x). The monotonicity condition requires f(x) and the derivative of τ(x) to be inversely proportional. The author then defines convergence speed as the derivative of the integral of f(x) with respect to the optimization iterations and shows it depends on the slope of the derivative of τ(x).
This is chapter 4 “Variants of EM algorithm” in my book “Tutorial on EM algorithm”, which focuses on EM variants. The main purpose of expectation maximization (EM) algorithm, also GEM algorithm, is to maximize the log-likelihood L(Θ) = log(g(Y|Θ)) with observed data Y by maximizing the conditional expectation Q(Θ’|Θ). Such Q(Θ’|Θ) is defined fixedly in E-step. Therefore, most variants of EM algorithm focus on how to maximize Q(Θ’|Θ) in M-step more effectively so that EM is faster or more accurate.
This is the chapter 3 "Properties and convergence of EM algorithm" in my book “Tutorial on EM algorithm”, which focuses on mathematical explanation of the convergence of GEM algorithm given by DLR (Dempster, Laird, & Rubin, 1977, pp. 6-9).
Local optimization with convex function is solved perfectly by traditional mathematical methods such as Newton-Raphson and gradient descent but it is not easy to solve the global optimization with arbitrary function although there are some purely mathematical approaches such as approximation, cutting plane, branch and bound, and interval method which can be impractical because of their complexity and high computation cost. Recently, some evolutional algorithms which are inspired from biological activities are proposed to solve the global optimization by acceptable heuristic level. Among them is particle swarm optimization (PSO) algorithm which is proved as an effective and feasible solution for global optimization in real applications. Although the ideology of PSO is not complicated, it derives many variants, which can make new researchers confused. Therefore, this tutorial focuses on describing, systemizing, and classifying PSO by succinct and straightforward way. Moreover, a combination of PSO and another evolutional algorithm as artificial bee colony (ABC) algorithm for improving PSO itself or solving other advanced problems are mentioned too.
Maximum likelihood estimation (MLE) is a popular method for parameter estimation in both applied probability and statistics but MLE cannot solve the problem of incomplete data or hidden data because it is impossible to maximize likelihood function from hidden data. Expectation maximum (EM) algorithm is a powerful mathematical tool for solving this problem if there is a relationship between hidden data and observed data. Such hinting relationship is specified by a mapping from hidden data to observed data or by a joint probability between hidden data and observed data (showing MLE, EM, and practical EM, hidden info implies the hinting relationship).
The essential ideology of EM is to maximize the expectation of likelihood function over observed data based on the hinting relationship instead of maximizing directly the likelihood function of hidden data (showing the full EM with proof along with two steps).
An important application of EM is (finite) mixture model which in turn is developed towards two trends such as infinite mixture model and semiparametric mixture model. Especially, in semiparametric mixture model, component probabilistic density functions are not parameterized. Semiparametric mixture model is interesting and potential for other applications where probabilistic components are not easy to be specified (showing mixture models).
I raise a question that whether it is possible to backward discover semiparametric EM from semiparametric mixture model. I hope that this question will open a new trend or new extension for EM algorithm (showing the question).
Describing and Interpreting an Immersive Learning Case with the Immersion Cub...Leonel Morgado
Current descriptions of immersive learning cases are often difficult or impossible to compare. This is due to a myriad of different options on what details to include, which aspects are relevant, and on the descriptive approaches employed. Also, these aspects often combine very specific details with more general guidelines or indicate intents and rationales without clarifying their implementation. In this paper we provide a method to describe immersive learning cases that is structured to enable comparisons, yet flexible enough to allow researchers and practitioners to decide which aspects to include. This method leverages a taxonomy that classifies educational aspects at three levels (uses, practices, and strategies) and then utilizes two frameworks, the Immersive Learning Brain and the Immersion Cube, to enable a structured description and interpretation of immersive learning cases. The method is then demonstrated on a published immersive learning case on training for wind turbine maintenance using virtual reality. Applying the method results in a structured artifact, the Immersive Learning Case Sheet, that tags the case with its proximal uses, practices, and strategies, and refines the free text case description to ensure that matching details are included. This contribution is thus a case description method in support of future comparative research of immersive learning cases. We then discuss how the resulting description and interpretation can be leveraged to change immersion learning cases, by enriching them (considering low-effort changes or additions) or innovating (exploring more challenging avenues of transformation). The method holds significant promise to support better-grounded research in immersive learning.
Current Ms word generated power point presentation covers major details about the micronuclei test. It's significance and assays to conduct it. It is used to detect the micronuclei formation inside the cells of nearly every multicellular organism. It's formation takes place during chromosomal sepration at metaphase.
When I was asked to give a companion lecture in support of ‘The Philosophy of Science’ (https://shorturl.at/4pUXz) I decided not to walk through the detail of the many methodologies in order of use. Instead, I chose to employ a long standing, and ongoing, scientific development as an exemplar. And so, I chose the ever evolving story of Thermodynamics as a scientific investigation at its best.
Conducted over a period of >200 years, Thermodynamics R&D, and application, benefitted from the highest levels of professionalism, collaboration, and technical thoroughness. New layers of application, methodology, and practice were made possible by the progressive advance of technology. In turn, this has seen measurement and modelling accuracy continually improved at a micro and macro level.
Perhaps most importantly, Thermodynamics rapidly became a primary tool in the advance of applied science/engineering/technology, spanning micro-tech, to aerospace and cosmology. I can think of no better a story to illustrate the breadth of scientific methodologies and applications at their best.
Travis Hills' Endeavors in Minnesota: Fostering Environmental and Economic Pr...Travis Hills MN
Travis Hills of Minnesota developed a method to convert waste into high-value dry fertilizer, significantly enriching soil quality. By providing farmers with a valuable resource derived from waste, Travis Hills helps enhance farm profitability while promoting environmental stewardship. Travis Hills' sustainable practices lead to cost savings and increased revenue for farmers by improving resource efficiency and reducing waste.
ESPP presentation to EU Waste Water Network, 4th June 2024 “EU policies driving nutrient removal and recycling
and the revised UWWTD (Urban Waste Water Treatment Directive)”
hematic appreciation test is a psychological assessment tool used to measure an individual's appreciation and understanding of specific themes or topics. This test helps to evaluate an individual's ability to connect different ideas and concepts within a given theme, as well as their overall comprehension and interpretation skills. The results of the test can provide valuable insights into an individual's cognitive abilities, creativity, and critical thinking skills
The use of Nauplii and metanauplii artemia in aquaculture (brine shrimp).pptxMAGOTI ERNEST
Although Artemia has been known to man for centuries, its use as a food for the culture of larval organisms apparently began only in the 1930s, when several investigators found that it made an excellent food for newly hatched fish larvae (Litvinenko et al., 2023). As aquaculture developed in the 1960s and ‘70s, the use of Artemia also became more widespread, due both to its convenience and to its nutritional value for larval organisms (Arenas-Pardo et al., 2024). The fact that Artemia dormant cysts can be stored for long periods in cans, and then used as an off-the-shelf food requiring only 24 h of incubation makes them the most convenient, least labor-intensive, live food available for aquaculture (Sorgeloos & Roubach, 2021). The nutritional value of Artemia, especially for marine organisms, is not constant, but varies both geographically and temporally. During the last decade, however, both the causes of Artemia nutritional variability and methods to improve poorquality Artemia have been identified (Loufi et al., 2024).
Brine shrimp (Artemia spp.) are used in marine aquaculture worldwide. Annually, more than 2,000 metric tons of dry cysts are used for cultivation of fish, crustacean, and shellfish larva. Brine shrimp are important to aquaculture because newly hatched brine shrimp nauplii (larvae) provide a food source for many fish fry (Mozanzadeh et al., 2021). Culture and harvesting of brine shrimp eggs represents another aspect of the aquaculture industry. Nauplii and metanauplii of Artemia, commonly known as brine shrimp, play a crucial role in aquaculture due to their nutritional value and suitability as live feed for many aquatic species, particularly in larval stages (Sorgeloos & Roubach, 2021).
The technology uses reclaimed CO₂ as the dyeing medium in a closed loop process. When pressurized, CO₂ becomes supercritical (SC-CO₂). In this state CO₂ has a very high solvent power, allowing the dye to dissolve easily.
Phenomics assisted breeding in crop improvementIshaGoswami9
As the population is increasing and will reach about 9 billion upto 2050. Also due to climate change, it is difficult to meet the food requirement of such a large population. Facing the challenges presented by resource shortages, climate
change, and increasing global population, crop yield and quality need to be improved in a sustainable way over the coming decades. Genetic improvement by breeding is the best way to increase crop productivity. With the rapid progression of functional
genomics, an increasing number of crop genomes have been sequenced and dozens of genes influencing key agronomic traits have been identified. However, current genome sequence information has not been adequately exploited for understanding
the complex characteristics of multiple gene, owing to a lack of crop phenotypic data. Efficient, automatic, and accurate technologies and platforms that can capture phenotypic data that can
be linked to genomics information for crop improvement at all growth stages have become as important as genotyping. Thus,
high-throughput phenotyping has become the major bottleneck restricting crop breeding. Plant phenomics has been defined as the high-throughput, accurate acquisition and analysis of multi-dimensional phenotypes
during crop growing stages at the organism level, including the cell, tissue, organ, individual plant, plot, and field levels. With the rapid development of novel sensors, imaging technology,
and analysis methods, numerous infrastructure platforms have been developed for phenotyping.
Immersive Learning That Works: Research Grounding and Paths ForwardLeonel Morgado
We will metaverse into the essence of immersive learning, into its three dimensions and conceptual models. This approach encompasses elements from teaching methodologies to social involvement, through organizational concerns and technologies. Challenging the perception of learning as knowledge transfer, we introduce a 'Uses, Practices & Strategies' model operationalized by the 'Immersive Learning Brain' and ‘Immersion Cube’ frameworks. This approach offers a comprehensive guide through the intricacies of immersive educational experiences and spotlighting research frontiers, along the immersion dimensions of system, narrative, and agency. Our discourse extends to stakeholders beyond the academic sphere, addressing the interests of technologists, instructional designers, and policymakers. We span various contexts, from formal education to organizational transformation to the new horizon of an AI-pervasive society. This keynote aims to unite the iLRN community in a collaborative journey towards a future where immersive learning research and practice coalesce, paving the way for innovative educational research and practice landscapes.
Cytokines and their role in immune regulation.pptx
Adversarial Variational Autoencoders to extend and improve generative model
1. Adversarial Variational Autoencoders to
extend and improve generative model
Professor Dr. Loc Nguyen, PhD, Postdoc,
Loc Nguyen’s Academic Network, Vietnam
Email: ng_phloc@yahoo.com
Homepage: www.locnguyen.net
Loc Nguyen - AVA
13/09/2023 1
2. Abstract
Generative artificial intelligence (GenAI) has been developing with many incredible achievements
like ChatGPT and Bard. Deep generative model (DGM) is a branch of GenAI, which is preeminent in
generating raster data such as image and sound due to strong points of deep neural network (DNN) in
inference and recognition. The built-in inference mechanism of DNN, which simulates and aims to
synaptic plasticity of human neuron network, fosters generation ability of DGM which produces
surprised results with support of statistical flexibility. Two popular approaches in DGM are
Variational Autoencoders (VAE) and Generative Adversarial Network (GAN). Both VAE and GAN
have their own strong points although they share and imply underline theory of statistics as well as
incredible complex via hidden layers of DNN when DNN becomes effective encoding/decoding
functions without concrete specifications. In this research, I try to unify VAE and GAN into a
consistent and consolidated model called Adversarial Variational Autoencoders (AVA) in which VAE
and GAN complement each other, for instance, VAE is good at generator by encoding data via
excellent ideology of Kullback-Leibler divergence and GAN is a significantly important method to
assess reliability of data which is realistic or fake. In other words, AVA aims to improve accuracy of
generative models, besides AVA extends function of simple generative models. In methodology this
research focuses on combination of applied mathematical concepts and skillful techniques of
computer programming in order to implement and solve complicated problems as simply as possible.
2
Loc Nguyen - AVA
13/09/2023
3. Table of contents
1. Introduction
2. Methodology
3. Experimental results and discussions
4. Conclusions
3
Loc Nguyen - AVA
13/09/2023
4. 1. Introduction
Variational Autoencoders (VAE) and Generative Adversarial Network (GAN) are two popular
approaches for developing deep generative model with support of deep neural network (DNN)
where high capacity of DNN contributes significantly to successes of GAN and VAE. There are
some researches which combined VAE and GAN. Larsen et al. (Larsen, Sønderby, Larochelle,
& Winther, 2016) proposed a traditional combination of VAE and GAN by considering decoder
of VAE as generator of GAN (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1558). They
constructed target optimization function as sum of likelihood function of VAE and target
function of GAN (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1560). This research is
similar to their research (Larsen, Sønderby, Larochelle, & Winther, 2016, p. 1561) except that
the construction optimization functions in two researches are slightly different where the one in
this research does not include target function of GAN according to traditional approach of GAN.
However uncorrelated variables will be removed after gradients are determined. Moreover,
because encoded data z is basically randomized in this research, I do not make a new random z’
to be included into target function of GAN. This research also mentions skillful techniques of
derivatives in backpropagation algorithm.
13/09/2023 Loc Nguyen - AVA 4
5. 1. Introduction
Mescheder et al. (Mescheder, Nowozin, & Geiger, 2017) transformed gain function of VAE including Kullback-Leibler
divergence into gain function of GAN via a so-called real-valued discrimination network (Mescheder, Nowozin, &
Geiger, 2017, p. 2394) related to Nash equilibrium equation and sigmoid function and then, they trained the transformed
VAE by stochastic gradient descent method. Actually, they estimated three parameters (Mescheder, Nowozin, & Geiger,
2017, p. 2395) like this research, but their method focused on mathematical transformation while I focus on skillful
techniques in implementation. In other words, Mescheder et al. (Mescheder, Nowozin, & Geiger, 2017) tried to fuse VAE
into GAN whereas I combine them by mutual and balancing way but both of us try to make unification of VAE and
GAN. Rosca et al. (Rosca, Lakshminarayanan, Warde-Farley, & Mohamed, 2017, p. 4) used a density ratio trick to
convert Kullback-Leibler divergence of VAE into the mathematical form log(x / (1–x)) which is similar to GAN target
function log(x) + log(1–x). Actually, they made a fusion of VAE and GAN like Mescheder et al. did. The essence of their
methods is based on convergence of Nash equilibrium equation. Ahmad et al. (Ahmad, Sun, You, Palade, & Mao, 2022)
combined VAE and GAN separately as featured experimental research. Firstly, they trained VAE and swapped encoder-
decoder network to decoder-encoder network so that output of VAE becomes some useful information which in turn
becomes input of GAN instead that GAN uses random information as input as usual (Ahmad, Sun, You, Palade, & Mao,
2022, p. 6). Miolane et al. (Miolane, Poitevin, & Li, 2020) combined VAE and GAN by summing target functions of
VAE and GAN weighted with regular hyperparameters (Miolane, Poitevin, & Li, 2020, p. 974). Later, they first trained
VAE and then sent output of VAE to input of GAN (Miolane, Poitevin, & Li, 2020, p. 975).
13/09/2023 Loc Nguyen - AVA 5
6. 1. Introduction
In general, this research focuses on incorporating GAN into VAE by skillful techniques
related to both stochastic gradient descent and software engineering architecture, which
neither focuses on purely mathematical fusion nor focuses on experimental tasks. In
practice, many complex mathematical problems can be solved effectively by some skillful
techniques of computer programming. Moreover, the proposed model called Adversarial
Variational Autoencoders (AVA) aims to extend functions of VAE and GAN as a general
architecture for generative model. For instance, AVA will provide encoding function that
GAN does not concern and provide discrimination function that VAE needs to distinguish
fake data from realistic data. The corporation of VAE and GAN in AVA is strengthened by
regular and balance mechanism, which obviously, is natural and like fusion mechanism. In
some cases, it is better than fusion mechanism because both built-in VAE and GAN inside
AVA can uphold their own strong features. Therefore, experiment in this research is not too
serious with large data when I only compare AVA and VAE within small dataset, which aims
to prove the proposed method mentioned in the next section.
13/09/2023 Loc Nguyen - AVA 6
7. 2. Methodology
In this research I propose a method as well as a generative model which incorporate Generative Adversarial Network (GAN)
into Variational Autoencoders (VAE) for extending and improving deep generative model because GAN does not concern
how to code original data and VAE lacks mechanisms to assess quality of generated data with note that data coding is
necessary to some essential applications such as image impression and recognition whereas auditing quality can improve
accuracy of generated data. As a convention, let vector variable x = (x1, x2,…, xm)T and vector variable z = (z1, z2,…, zn)T be
original data and encoded data whose dimensions are m and n (m > n), respectively. A generative model is represented by a
function f(x | Θ) = z, f(x | Θ) ≈ z, or f(x | Θ) → z where f(x | Θ) is implemented by a deep neural network (DNN) whose
weights are Θ, which converts the original data x to the encoded data z and is called encoder in VAE. A decoder in VAE
which converts expectedly the encoded data z back to the original data x is represented by a function g(z | Φ) = x’ where g(z
| Φ) is also implemented by a DNN whose weights are Φ with expectation that the decoded data x’ is approximated to the
original data x as x’ ≈ x. The essence of VAE is to minimize the following loss function for estimating the encoded
parameter Θ and the decoded parameter Φ.
𝑙VAE Θ, Φ =
1
2
𝒙 − 𝒙′ 2
+ KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼 1
Such that:
Θ∗ = argmin
Φ
KL 𝜇 𝒙 , Σ 𝒙 𝑁 0, 𝐼
Φ∗
= argmin
Θ
1
2
𝒙 − 𝒙′ 2
13/09/2023 Loc Nguyen - AVA 7
8. 2. Methodology
Note that ||x – x’|| is Euclidean distance between x and x’ whereas KL(μ(x), Σ(x) | N(0, I)) is Kullback-Leibler
divergence between Gaussian distribution of x whose mean vector and covariance matrix are μ(x) and Σ(x)
and standard Gaussian distribution N(0, I) whose mean vector and covariance matrix are 0 and identity matrix
I.
GAN does not concern the encoder f(x | Θ) = z but it focuses on optimizing the decoder g(z | Φ) = x’ by
introducing a so-called discriminator which is a discrimination function d(x | Ψ): x → [0, 1] from concerned
data x or x’ to range [0, 1] in which d(x | Ψ) can distinguish fake data from real data. In other words, the larger
result the discriminator d(x’ | Ψ) derives, the more realistic the generated data x’ is. Obviously, d(x | Ψ) is
implemented by a DNN whose weights are Ψ with note that this DNN has only one output neuron denoted d0.
The essence of GAN is to optimize mutually the following target function for estimating the decoder
parameter Φ and the discriminator parameter Ψ.
𝑏GAN Φ, Ψ = log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝑔 𝒛 Φ Ψ 2
Such that Φ and Ψ are optimized mutually as follows:
Φ∗
= argmin
Φ
𝑏GAN Φ, Ψ∗
Ψ∗
= argmax
Ψ
𝑏GAN Φ∗
, Ψ
13/09/2023 Loc Nguyen - AVA 8
9. 2. Methodology
The proposed generative model in this research is called Adversarial Variational Autoencoders (AVA) because it
combines VAE and GAN by fusing mechanism in which loss function and balance function are optimized
parallelly. The AVA loss function implies loss information in encoder f(x | Θ), decoder g(z | Φ), discriminator d(x |
Ψ) as follows:
𝑙AVA Θ, Φ, Ψ =
1
2
𝒙 − 𝒙′ 2
+ KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼 + log 1 − 𝑑 𝑔 𝒛 Φ Ψ 3
The balance function of AVA is to supervise the decoding mechanism, which is the GAN target function as follows:
𝑏AVA Φ, Ψ = 𝑏GAN Φ, Ψ = log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝑔 𝒛 Φ Ψ 4
The key point of AVA is that the discriminator function occurs in both loss function and balance function via the
expression log(1 – d(g(z | Φ) | Ψ)), which means that the capacity of how to distinguish fake data from realistic data
by discriminator function affects the decoder DNN. As a result, the three parameters Θ, Φ, and Ψ are optimized
mutually according to both loss function and balance function as follows:
Θ∗
= argmin
Θ
𝑙AVA Θ, Φ∗
, Ψ∗
Φ∗ = argmin
Φ
𝑙AVA Θ∗, Φ, Ψ∗
Ψ∗ = argmax
Ψ
𝑏AVA Φ∗, Ψ
13/09/2023 Loc Nguyen - AVA 9
10. 2. Methodology
Because the encoder parameter Θ is independent from both the decoder parameter Φ and the discriminator parameter Ψ,
its estimate is specified as follows:
Θ∗
= argmin
Φ
KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼
Because the decoder parameter Φ is independent from the encoder parameter Θ, its estimate is specified as follows:
Φ∗
= argmin
Θ
1
2
𝒙 − 𝒙′ 2
+ log 1 − 𝑑 𝑔 𝒛 Φ∗
Ψ∗
Note that the Euclidean distance ||x – x’|| is only dependent on Θ. Because the discriminator tries to increase credible
degree of realistic data and decrease credible degree of fake data, its parameter Ψ has following estimate:
Ψ∗
= argmax
Ψ
log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝑔 𝒛 Φ∗
Ψ
By applying stochastic gradient descent (SDG) algorithm into backpropagation algorithm, these estimates are determined
based on gradients of loss function and balance function as follows:
Θ = Θ − 𝛾∇Θ KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼
Φ = Φ − 𝛾∇Φ
1
2
𝒙 − 𝒙′ 2
+ log 1 − 𝑑 𝑔 𝒛 Φ∗
Ψ∗
Ψ = Ψ + 𝛾∇Ψ log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝑔 𝒛 Φ∗
Ψ
Where γ (0 < γ ≤ 1) is learning rate.
13/09/2023 Loc Nguyen - AVA 10
11. 2. Methodology
Let af(.), ag(.), and ad(.) be activation functions of encoder DNN, decoder DNN, and
discriminator DNN, respectively and so, let af’(.), ag’(.), and ad’(.) be derivatives of these
activation functions, respectively. The encoder gradient regarding Θ is (Doersch, 2016, p. 9):
∇Θ KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼 = 𝜇 𝒙 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙
The decoder gradient regarding Φ is:
∇Φ
1
2
𝒙 − 𝒙′ 2 + log 1 − 𝑑 𝑔 𝒛 Φ∗ Ψ∗ = − 𝒙 − 𝒙′ +
𝑎𝑑
′
𝑑 𝒙′ Ψ∗
1 − 𝑑 𝒙′
Ψ∗ 𝑎𝑔
′ 𝒙′
Where,
𝑔 𝒛 Φ∗ = 𝒙′
The discriminator gradient regarding Ψ is:
∇Ψ log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝒙′ Ψ =
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′ Ψ
13/09/2023 Loc Nguyen - AVA 11
12. 2. Methodology
As a result, SGD algorithm incorporated into backpropagation algorithm for solving AVA is totally determined as
follows:
Θ = Θ − 𝛾 𝜇 𝒙 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙 5
Φ 𝑖 = Φ 𝑖 + 𝛾 𝒙 𝑖 − 𝒙′
𝑖 +
𝑎𝑑
′
𝑑 𝒙′
Ψ∗
1 − 𝑑 𝒙′
Ψ∗ 𝑎𝑔
′
𝒙′
𝑖 6
Ψ = Ψ + 𝛾
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′
Ψ
7
Where notation [i] denotes the ith element in vector. Please pay attention to the derivatives af’(.), ag’(.), and ad’(.) because
they are helpful techniques to consolidate AVA. The reason of two different occurrences of derivatives ad’(d(x’ | Ψ*)) and
ag’(x’) in decoder gradient regarding Φ is nontrivial because the unique output neuron of discriminator DNN is
considered as effect of the output layer of all output neurons in decoder DNN. When weights are assumed to be 1, error
of causal decoder neuron is error of discriminator neuron multiplied with derivative at the decoder neuron and moreover,
the error of discriminator neuron, in turn, is product of its minus bias –d’(.) and its derivative a’d(.).
error 𝒙′
𝑖 = 1 ∗ error 𝑑0 𝑎𝑔
′
𝒙′
𝑖
error 𝑑0 = −𝑑′
𝑑0 𝑎𝑑
′
𝑑0
13/09/2023 Loc Nguyen - AVA 12
Figure 1. Causality effect relationship between
decoder DNN and discriminator DNN
13. 2. Methodology
It is necessary to describe AVA architecture because skillful techniques
cannot be applied into AVA without clear and solid architecture. The
key point to incorporate GAN into VAE is that the error
𝑎𝑑
′
𝑑 𝒙′
Ψ∗
1−𝑑 𝒙′
Ψ∗ of
generated data is included in both decoder and discriminator, besides
decoded data x’ which is output of decoder DNN becomes input of
discriminator DNN.
Φ 𝑖 = Φ 𝑖 + 𝛾 𝒙 𝑖 − 𝒙′
𝑖 +
𝑎𝑑
′
𝑑 𝒙′
Ψ∗
1 − 𝑑 𝒙′
Ψ∗ 𝑎𝑔
′
𝒙′
𝑖
Ψ = Ψ + 𝛾
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′ Ψ
AVA architecture (figure 2) follows an important aspect of VAE where
the encoder f(x | Θ) does not produce directly decoded data z as f(x | Θ)
= z. It actually produces mean vector μ(x) and covariance matrix Σ(x)
belonging to x instead. In this research, μ(x) and Σ(x) are flattened into
an array of neurons output layer of the encoder f(x | Θ).
𝑓 𝒙 Θ =
𝜇 𝒙
Σ 𝒙
→ 𝒛
13/09/2023 Loc Nguyen - AVA 13
Figure 2. AVA architecture
14. 2. Methodology
The actual decoded data z is calculated randomly from μ(x) and Σ(x) along with a random vector r.
𝒛 = 𝜇 𝒙 + Σ 𝒙
1
2𝒓 8
Where r follows standard Gaussian distribution with mean vector 0 and identity covariance matrix I and each
element of (Σ(x))1/2 is squared root of the corresponding element of Σ(x). This is an excellent invention in
traditional literature which made the calculation of Kullback-Leibler divergence much easier without loss of
information.
The balance function bAVA(Φ, Ψ) aims to balance decoding task and discrimination task without partiality
but it can lean forward decoding task for improving accuracy of decoder by including the error of original
data x and decoded data x’ into balance function as follows:
𝑏AVA Φ, Ψ = 𝑏GAN Φ, Ψ −
1
2
𝒙 − 𝒙′ 2
= log 𝑑 𝒙 Ψ + log 1 − 𝑑 𝑔 𝒛 Φ Ψ −
1
2
𝒙 − 𝒙′ 2
9
As a result, the estimate of discriminator parameter Ψ is:
Ψ = Ψ + 𝛾
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′ Ψ
+ 𝑎𝑑
′
𝑑0
𝑖
𝒙 𝑖 − 𝒙′
𝑖 𝑎𝑔
′
𝒙′
𝑖 10
13/09/2023 Loc Nguyen - AVA 14
15. 2. Methodology
In a reverse causality effect relationship in which the unique output
neuron of discriminator DNN is cause of all output neurons of
decoder DNN as shown in figure 3.
Suppose bias of each decoder output neuron is bias[i], error of the
discriminator output neuron error[i] is sum of weighted biases which
is in turn multiplied with derivative at the discriminator output
neuron with note that every weighted bias is also multiplied with
derivative at every decoder output neuron. Suppose all weights are 1,
we have:
error 𝑖 = 𝑎𝑑
′
𝑑0
𝑖
bias 𝑖 𝑎𝑔
′
𝒙′
𝑖
bias 𝑖 = 𝒙 𝑖 − 𝒙′ 𝑖
13/09/2023 Loc Nguyen - AVA 15
Figure 3. Reverse
causality effect
relationship
between
discriminator DNN
and decoder DNN
16. 2. Methodology
Because the balance function bAVA(Φ, Ψ) aims to improve the decoder g(z | Φ), it is possible to improve the encoder f(x
| Θ) by similar technique with note that output of encoder is mean vector μ(x) and covariance matrix Σ(x). In this
research, I propose another balance function BAVA(Θ, Λ) to assess reliability of the mean vector μ(x) because μ(x) is
most important to randomize z and μ(x) is linear. Let D(μ(x) | Λ) be discrimination function for encoder DNN from
μ(x) to range [0, 1] in which D(μ(x) | Λ) can distinguish fake mean μ(x) from real mean μ(x’). Obviously, D(μ(x) | Λ) is
implemented by a so-called encoding discriminator DNN whose weights are Λ with note that this DNN has only one
output neuron denoted D0. The balance function BAVA(Θ, Λ) is specified as follows:
𝐵AVA Θ, Λ = log 𝐷 𝜇 𝒙 Λ + log 1 − 𝐷 𝜇 𝒙′
Λ 11
Note,
𝑔 𝒛 Φ = 𝒙′
AVA loss function is modified with regard to the balance function BAVA(Θ, Λ) as follows:
𝑙AVA Θ, Φ, Ψ, Λ =
1
2
𝒙 − 𝒙′ 2
+ KL 𝜇 𝒙 , Σ 𝒙 𝑁 𝟎, 𝐼 + log 1 − 𝑑 𝒙′
Ψ + log 1 − 𝐷 𝜇 𝒙′
Λ 12
By similar way of applying SGD algorithm, it is easy to estimate the encoding discriminator parameter Λ as follows:
Λ = Λ + 𝛾
𝑎𝐷
′
𝐷 𝜇 𝒙 Λ
𝐷 𝜇 𝒙 Λ
−
𝑎𝐷
′
𝐷 𝜇 𝒙′
Λ
1 − 𝐷 𝜇 𝒙′
Λ
13
Where aD(.) and a’D(.) are activation function of the discriminator D(μ(x) | Λ) and its derivative, respectively.
13/09/2023 Loc Nguyen - AVA 16
17. 2. Methodology
The encoder parameter Θ is consisted of two separated parts
Θμ and ΘΣ because the output of encoder f(x | Θ) is consisted of
mean vector μ(x) and covariance matrix Σ(x).
Θ =
Θ𝜇
ΘΣ
Where,
Θ𝜇 = Θ𝜇 − 𝛾𝜇 𝒙 𝑎𝑓
′
𝒙
ΘΣ = ΘΣ − 𝛾 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙
When the balance function BAVA(Θ, Λ) is included in AVA loss
function, the part Θμ is recalculated whereas the part ΘΣ is kept
intact as follows:
Θ𝜇 = Θ𝜇 − 𝛾 𝜇 𝒙 −
𝑎𝐷
′
𝐷 𝒙′
Λ
1 − 𝐷 𝒙′ Λ
𝑎𝑓
′
𝒙 14
Figure 4 shows AVA architecture with support of assessing
encoder.
13/09/2023 Loc Nguyen - AVA 17
Figure 4. AVA architecture with support of
encoder assessing
18. 2. Methodology
Similarly, the balance function BAVA(Φ, Λ) can lean forward encoding task for improving accuracy of encoder
f(x | Θ) by concerning the error of original mean μ(x) and decoded data mean μ(x’) as follows:
𝐵AVA Φ, Λ = log 𝐷 𝜇 𝒙 Λ + log 1 − 𝐷 𝜇 𝒙′
Λ −
1
2
𝜇 𝒙 − 𝜇 𝒙′ 2
15
Without repeating explanations, the estimate of discriminator parameter Λ is modified as follows:
Λ = Λ + 𝛾
𝑎𝐷
′
𝐷 𝜇 𝒙 Λ
𝐷 𝜇 𝒙 Λ
−
𝑎𝐷
′
𝐷 𝜇 𝒙′
Λ
1 − 𝐷 𝜇 𝒙′
Λ
+ 𝑎𝐷
′
𝐷0
𝑖
𝜇 𝒙 𝑖 − 𝜇 𝒙′
𝑖 𝑎𝑔
′
𝜇 𝒙′
𝑖 16
These variants of AVA are summarized, and their tests are described in the next section.
13/09/2023 Loc Nguyen - AVA 18
19. 3. Experimental results and discussions
In this experiment, AVA is tested with VAE but there are 5 versions of AVA such as AVA1, AVA2, AVA3, AVA4, and AVA5.
Recall that AVA1 is normal version of AVA whose parameters are listed as follows:
Θ = Θ − 𝛾 𝜇 𝒙 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙
Φ 𝑖 = Φ 𝑖 + 𝛾 𝒙 𝑖 − 𝒙′ 𝑖 +
𝑎𝑑
′
𝑑 𝒙′ Ψ∗
1 − 𝑑 𝒙′ Ψ∗ 𝑎𝑔
′ 𝒙′ 𝑖
Ψ = Ψ + 𝛾
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′ Ψ
AVA2 leans forward improving accuracy of decoder DNN by modifying discriminator parameter Ψ as follows:
Θ = Θ − 𝛾 𝜇 𝒙 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙
Φ 𝑖 = Φ 𝑖 + 𝛾 𝒙 𝑖 − 𝒙′
𝑖 +
𝑎𝑑
′
𝑑 𝒙′ Ψ∗
1 − 𝑑 𝒙′
Ψ∗ 𝑎𝑔
′
𝒙′
𝑖
Ψ = Ψ + 𝛾
𝑎𝑑
′
𝑑 𝒙 Ψ
𝑑 𝒙 Ψ
−
𝑎𝑑
′
𝑑 𝒙′
Ψ
1 − 𝑑 𝒙′ Ψ
+ 𝑎𝑑
′
𝑑0
𝑖
𝒙 𝑖 − 𝒙′ 𝑖 𝑎𝑔
′ 𝒙′ 𝑖
13/09/2023 Loc Nguyen - AVA 19
21. 3. Experimental results and discussions
AVA5 is the last one which supports all functions such as decoder supervising, leaning decoder, encoder supervising, and learning
encoder.
Θ𝜇 = Θ𝜇 − 𝛾 𝜇 𝒙 −
𝑎𝐷
′
𝐷 𝒙′ Λ
1 − 𝐷 𝒙′ Λ
𝑎𝑓
′
𝒙 , ΘΣ = ΘΣ − 𝛾 −
1
2
Σ 𝒙
−1
+
1
2
𝐼 𝑎𝑓
′
𝒙
Φ 𝑖 = Φ 𝑖 + 𝛾 𝒙 𝑖 − 𝒙′ 𝑖 +
𝑎𝑑
′
𝑑 𝒙′ Ψ∗
1 − 𝑑 𝒙′ Ψ∗ 𝑎𝑔
′ 𝒙′ 𝑖
Λ
= Λ
+ 𝛾
𝑎𝐷
′
𝐷 𝜇 𝒙 Λ
𝐷 𝜇 𝒙 Λ
−
𝑎𝐷
′
𝐷 𝜇 𝒙′ Λ
1 − 𝐷 𝜇 𝒙′ Λ
+ 𝑎𝑑
′
𝑑0
𝑖
𝒙 𝑖 − 𝒙′ 𝑖 𝑎𝑔
′ 𝒙′ 𝑖 + 𝑎𝐷
′
𝐷0
𝑖
𝜇 𝒙 𝑖 − 𝜇 𝒙′ 𝑖 𝑎𝑔
′ 𝜇 𝒙′ 𝑖
The experiment is performed on a laptop with CPU AMD64 4 processors, 4GB RAM, Windows 10, and Java 15. The dataset is a
set of ten 180x250 images, but convolution layers built in AVA zoom out 3 times smaller due to lack of memory. The four AVA
variants will be evaluated by root mean square error (RMSE) with 19 learning rates γ = 1, 0.9,…, 0.1, 0.09, 0.001 because
stochastic gradient descent (SGD) algorithm is affected by learning rate and the accuracy of AVA varies a little bit within a learning
rate because of randomizing encoded data z in VAE algorithm.
13/09/2023 Loc Nguyen - AVA 21
22. 3. Experimental results and discussions
Given an AVA was trained by 10 images in the dataset, let imageBest be the best image generated by such AVA,
which is compared with the ith image denoted images[i] in dataset and then, RMSE of such AVA and the ith image is
calculated as follows:
RMSE 𝑖 =
1
𝑛𝑖
𝑗
1
2
imageBest 𝑗 − image 𝑖 𝑗
The overall RMSE of such AVA is average RMSE over N=10 test images as follows:
RMSE =
1
𝑁
𝑖
RMSE 𝑖
This means:
RMSE =
1
𝑁
𝑖
1
𝑛𝑖
𝑗
1
2
imageBest 𝑗 − image 𝑖 𝑗 17
Where N is the number of images, N=10 and ni is the number of pixels of the ith image. Obviously, image[i][j]
(imageBest[j]) is the jth pixel of the ith image (the best image). The notation ||.|| denotes norm of pixel. For example,
norm of RGB pixel is 𝑟2 + 𝑔2 + 𝑏2 where r, g, and b are red color, green color, and blue color of such pixel. The
smaller the RMSE is, the better the AVA is.
13/09/2023 Loc Nguyen - AVA 22
25. 3. Experimental results and discussions
From experimental results shown in table 1 and table 2, RMSE means of AVA1, AVA2, AVA3, AVA4,
AVA5, are VAE over all learning rates are 0.1787, 0.1786, 0.1783, 0.1783, 0.1783, and 0.2454,
respectively. Because AVA3, AVA4, and AVA5 result the same best RMSE (0.1783), it is asserted that AVA
with any discriminators in regardless of decoder discriminator (AVA5) or encoder discriminator (AVA3,
AVA4, AVA5) but it seems that encoder discriminator is better than traditional decoder discriminator, for
instance, Larsen et al. (Larsen, Sønderby, Larochelle, & Winther, 2016) focused on decoder discriminator.
Therefore, we check again RMSE standard deviations of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE
which are 0.0028, 0.0017, 0.0019, 0.0011, 0.0012, and 0.0054, respectively. AVA4 results out the best
RMSE standard deviation (0.0011), which implies undoubtful that encoder discriminator is as good as
decoder discriminator at least because it is necessary to test AVA3 with larger dataset and moreover, AVA4
with smallest RMSE standard deviation leans forward encoder mechanism and so, AVA4 is not as fair as
AVA1 without leaning forward decoder mechanism. Note that AVA1 and AVA3 are fairest because they do
not lean any encoder/decoder mechanism. We check again RMSE minimum values of AVA1, AVA2,
AVA3, AVA4, AVA5, and VAE which are 0.1769, 0.177, 0.1765, 0.177, 0.1768, and 0.2366, respectively.
The fair AVA3 gains the best minimum value (0.1765), which asserts again that AVA with encoder
discriminator is as good as decoder discriminator at least with note that biases among these minimum
values is too small to conclude a conclusion of preeminence of encoder discriminator.
13/09/2023 Loc Nguyen - AVA 25
26. 3. Experimental results and discussions
Table 3 and figure 5 show RMSE means, RMSE minimum values, and RMSE
standard deviations of AVA1, AVA2, AVA3, AVA4, AVA5, and VAE.
Table 3. Evaluation of AVAs and VAE
Figure 5. Evaluation of AVAs and VAE
13/09/2023 Loc Nguyen - AVA 26
AVA1 AVA2 AVA3 AVA4 AVA5 VAE
Mean 0.1787 0.1786 0.1783 0.1783 0.1783 0.2454
Minimum 0.1769 0.1770 0.1765 0.1770 0.1768 0.2366
SD 0.0028 0.0017 0.0019 0.0011 0.0012 0.0054
27. 3. Experimental results and discussions
It is concluded from figure 5 that the corporation of GAN and VAE
which produces AVA in this research results out better encoding and
decoding performance of deep generative model when RMSE means,
standard deviation, and minimum value of all AVA are smaller than
the ones of VAE. Moreover, AVA5 which is full of functions
including decoder discriminator, decoder leaning, encoder
discrimination, and encoder leaning but, it does not produce the best
result as expected although is a very good AVA, especially, with
regard to mean (0.1783), minimum value (0.1768), and standard
deviation (0.0012). The reason may be that it is necessary to test
AVA5 with large data. Alternately, in some complex systems, many
constraints can annul mutually or maybe they restrict mutually instead
of reaching the best result or a perfect balance. However, AVA5 in
this research is stabler than other ones because encoder performance
and decoder performance are proportional together, which means that
improvement of encoder is to improve decoder and vice versa.
13/09/2023 Loc Nguyen - AVA 27
Figure 5. Evaluation of AVAs and VAE
28. 4. Conclusions
It is undoubtful that AVA is better than traditional VAE due to the built-in discriminator
function of GAN that assesses reliability of data. I think that VAE and GAN are solid
models in both theory and practice when their mathematical foundation cannot be
changed or transformed but it is still possible to improve them by modifications or
combinations as well as applying them into specific applications where their strong
points are brought into play. In applications related to raster data like image, VAE has a
drawback of consuming much memory because probabilistic distribution represents
entire image whereas some other deep generative models focus on representing product
of many conditional probabilistic distributions for pixels. However, this pixel approach
for modeling pixels by recurrent neural network does not consume less memory but it is
significantly useful to fill in or recover smaller damaged areas in a bigger image. In the
future trend, I try to apply the pixel approach into AVA, for instance, AVA processes a
big image block by block and then, every block is modeled by conditional probability
distribution with recurrent neural network as well as long short-term memory network.
13/09/2023 Loc Nguyen - AVA 28
29. References
1. Ahmad, B., Sun, J., You, Q., Palade, V., & Mao, Z. (2022, January 21). Brain Tumor Classifcation Using a Combination of
Variational Autoencoders and Generative Adversarial Networks. biomedicines, 10(2), 1-19.
doi:10.3390/biomedicines10020223
2. Doersch, C. (2016, January 3). Tutorial on Variational Autoencoders. arXiv preprint. Retrieved from
https://arxiv.org/abs/1606.05908
3. Larsen, A. B., Sønderby, S. K., Larochelle, H., & Winther, O. (2016). Autoencoding beyond pixels using a learned similarity
metric. International conference on machine learning. 48, pp. 1558-1566. New York: JMLR. Retrieved from
http://proceedings.mlr.press/v48/larsen16.pdf
4. Mescheder, L., Nowozin, S., & Geiger, A. (2017). Adversarial Variational Bayes: Unifying Variational Autoencoders and
Generative Adversarial Networks. Proceedings of the 34 th International Conference on Machine. 70, pp. 2391-2400.
Sydney: PMLR. Retrieved from http://proceedings.mlr.press/v70/mescheder17a/mescheder17a.pdf
5. Miolane, N., Poitevin, F., & Li, Y.-T. (2020). Estimation of Orientation and Camera Parameters from Cryo-Electron
Microscopy Images with Variational Autoencoders and Generative Adversarial. Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition Workshops (pp. 970-971). New Orleans: IEEE. Retrieved from
http://openaccess.thecvf.com/content_CVPRW_2020/papers/w57/Miolane_Estimation_of_Orientation_and_Camera_Param
eters_From_Cryo-Electron_Microscopy_Images_CVPRW_2020_paper.pdf
6. Rosca, M., Lakshminarayanan, B., Warde-Farley, D., & Mohamed, S. (2017, October). Variational Approaches for Auto-
Encoding Generative Adversarial Networks. arXiv preprint. Retrieved from https://arxiv.org/abs/1706.04987
13/09/2023 Loc Nguyen - AVA 29
30. Thank you for listening
30
Loc Nguyen - AVA
13/09/2023