1. Motivation: why do we need low-rank tensors
2. Tensors of the second order (matrices)
3. CP, Tucker and tensor train tensor formats
4. Many classical kernels have (or can be approximated in ) low-rank tensor format
5. Post processing: Computation of mean, variance, level sets, frequency
Optimal interval clustering: Application to Bregman clustering and statistica...Frank Nielsen
We present a generic dynamic programming method to compute the optimal clustering of n scalar elements into k pairwise disjoint intervals. This case includes 1D Euclidean k-means, k-medoids, k-medians, k-centers, etc. We extend the method to incorporate cluster size constraints and show how to choose the appropriate k by model selection. Finally, we illustrate and refine the method on two case studies: Bregman clustering and statistical mixture learning maximizing the complete likelihood.
http://arxiv.org/abs/1403.2485
We combined: low-rank tensor techniques and FFT to compute kriging, estimate variance, compute conditional covariance. We are able to solve 3D problems with very high resolution
Optimal interval clustering: Application to Bregman clustering and statistica...Frank Nielsen
We present a generic dynamic programming method to compute the optimal clustering of n scalar elements into k pairwise disjoint intervals. This case includes 1D Euclidean k-means, k-medoids, k-medians, k-centers, etc. We extend the method to incorporate cluster size constraints and show how to choose the appropriate k by model selection. Finally, we illustrate and refine the method on two case studies: Bregman clustering and statistical mixture learning maximizing the complete likelihood.
http://arxiv.org/abs/1403.2485
We combined: low-rank tensor techniques and FFT to compute kriging, estimate variance, compute conditional covariance. We are able to solve 3D problems with very high resolution
Efficient Analysis of high-dimensional data in tensor formatsAlexander Litvinenko
We solve a PDE with uncertain coefficients. The solution is approximated in the Karhunen Loeve/PCE basis. How to compute maximum ? frequency? probability density function? with almost linear complexity? We offer various methods.
Multi-linear algebra and different tensor formats with applications Alexander Litvinenko
A short overview of well-known tensor formats, elliptic PDE with uncertain coefficients, some academic examples of separable functions, post-processing in tensor format
Tucker tensor analysis of Matern functions in spatial statistics Alexander Litvinenko
1. Motivation: improve statistical models
2. Motivation: disadvantages of matrices
3. Tools: Tucker tensor format
4. Tensor approximation of Matern covariance function via FFT
5. Typical statistical operations in Tucker tensor format
6. Numerical experiments
My talk in the International Conference on Computational Finance 2019 (ICCF2019). The talk is about designing new efficient methods for option pricing under the rough Bergomi model.
Error Estimates for Multi-Penalty Regularization under General Source Conditioncsandit
In learning theory, the convergence issues of the regression problem are investigated with
the least square Tikhonov regularization schemes in both the RKHS-norm and the L 2
-norm.
We consider the multi-penalized least square regularization scheme under the general source
condition with the polynomial decay of the eigenvalues of the integral operator. One of the
motivation for this work is to discuss the convergence issues for widely considered manifold
regularization scheme. The optimal convergence rates of multi-penalty regularizer is achieved
in the interpolation norm using the concept of effective dimension. Further we also propose
the penalty balancing principle based on augmented Tikhonov regularization for the choice of
regularization parameters. The superiority of multi-penalty regularization over single-penalty
regularization is shown using the academic example and moon data set.
Hierarchical Deterministic Quadrature Methods for Option Pricing under the Ro...Chiheb Ben Hammouda
Conference talk at the SIAM Conference on Financial Mathematics and Engineering, held in virtual format, June 1-4 2021, about our recently published work "Hierarchical adaptive sparse grids and quasi-Monte Carlo for option pricing under the rough Bergomi model".
- Link of the paper: https://www.tandfonline.com/doi/abs/10.1080/14697688.2020.1744700
A One-Pass Triclustering Approach: Is There any Room for Big Data?Dmitrii Ignatov
An efficient one-pass online algorithm for triclustering of binary data (triadic formal contexts) is proposed. This algorithm is a modified version of the basic algorithm for OAC-triclustering approach, but it has linear time and memory complexities with respect to the cardinality
of the underlying ternary relation and can be easily parallelized in order to be applied for the analysis of big datasets. The results of computer experiments show the efficiency of the proposed algorithm.
We present recent result on the numerical analysis of Quasi Monte-Carlo quadrature methods, applied to forward and inverse uncertainty quantification for elliptic and parabolic PDEs. Particular attention will be placed on Higher
-Order QMC, the stable and efficient generation of
interlaced polynomial lattice rules, and the numerical analysis of multilevel QMC Finite Element discretizations with applications to computational uncertainty quantification.
The peer-reviewed International Journal of Engineering Inventions (IJEI) is started with a mission to encourage contribution to research in Science and Technology. Encourage and motivate researchers in challenging areas of Sciences and Technology.
R package 'bayesImageS': a case study in Bayesian computation using Rcpp and ...Matt Moores
There are many approaches to Bayesian computation with intractable likelihoods, including the exchange algorithm, approximate Bayesian computation (ABC), thermodynamic integration, and composite likelihood. These approaches vary in accuracy as well as scalability for datasets of significant size. The Potts model is an example where such methods are required, due to its intractable normalising constant. This model is a type of Markov random field, which is commonly used for image segmentation. The dimension of its parameter space increases linearly with the number of pixels in the image, making this a challenging application for scalable Bayesian computation. My talk will introduce various algorithms in the context of the Potts model and describe their implementation in C++, using OpenMP for parallelism. I will also discuss the process of releasing this software as an open source R package on the CRAN repository.
Low rank tensor approximation of probability density and characteristic funct...Alexander Litvinenko
Very often one has to deal with high-dimensional random variables (RVs). A high-dimensional RV can be described by its probability density (\pdf) and/or by the corresponding probability characteristic functions (\pcf), or by a function representation. Here the interest is mainly to compute characterisations like the entropy, or
relations between two distributions, like their Kullback-Leibler divergence, or more general measures such as $f$-divergences,
among others. These are all computed from the \pdf, which is often not available directly, and it is a computational challenge to even represent it in a numerically feasible fashion in case the dimension $d$ is even moderately large. It is an even stronger numerical challenge to then actually compute said characterisations in the high-dimensional case.
In this regard, in order to achieve a computationally feasible task, we propose to represent the density by a high order tensor product, and approximate this in a low-rank format.
Computing f-Divergences and Distances of\\ High-Dimensional Probability Densi...Alexander Litvinenko
Talk presented on SIAM IS 2022 conference.
Very often, in the course of uncertainty quantification tasks or
data analysis, one has to deal with high-dimensional random variables (RVs)
(with values in $\Rd$). Just like any other RV,
a high-dimensional RV can be described by its probability density (\pdf) and/or
by the corresponding probability characteristic functions (\pcf),
or a more general representation as
a function of other, known, random variables.
Here the interest is mainly to compute characterisations like the entropy, the Kullback-Leibler, or more general
$f$-divergences. These are all computed from the \pdf, which is often not available directly,
and it is a computational challenge to even represent it in a numerically
feasible fashion in case the dimension $d$ is even moderately large. It
is an even stronger numerical challenge to then actually compute said characterisations
in the high-dimensional case.
In this regard, in order to achieve a computationally feasible task, we propose
to approximate density by a low-rank tensor.
Efficient Analysis of high-dimensional data in tensor formatsAlexander Litvinenko
We solve a PDE with uncertain coefficients. The solution is approximated in the Karhunen Loeve/PCE basis. How to compute maximum ? frequency? probability density function? with almost linear complexity? We offer various methods.
Multi-linear algebra and different tensor formats with applications Alexander Litvinenko
A short overview of well-known tensor formats, elliptic PDE with uncertain coefficients, some academic examples of separable functions, post-processing in tensor format
Tucker tensor analysis of Matern functions in spatial statistics Alexander Litvinenko
1. Motivation: improve statistical models
2. Motivation: disadvantages of matrices
3. Tools: Tucker tensor format
4. Tensor approximation of Matern covariance function via FFT
5. Typical statistical operations in Tucker tensor format
6. Numerical experiments
My talk in the International Conference on Computational Finance 2019 (ICCF2019). The talk is about designing new efficient methods for option pricing under the rough Bergomi model.
Error Estimates for Multi-Penalty Regularization under General Source Conditioncsandit
In learning theory, the convergence issues of the regression problem are investigated with
the least square Tikhonov regularization schemes in both the RKHS-norm and the L 2
-norm.
We consider the multi-penalized least square regularization scheme under the general source
condition with the polynomial decay of the eigenvalues of the integral operator. One of the
motivation for this work is to discuss the convergence issues for widely considered manifold
regularization scheme. The optimal convergence rates of multi-penalty regularizer is achieved
in the interpolation norm using the concept of effective dimension. Further we also propose
the penalty balancing principle based on augmented Tikhonov regularization for the choice of
regularization parameters. The superiority of multi-penalty regularization over single-penalty
regularization is shown using the academic example and moon data set.
Hierarchical Deterministic Quadrature Methods for Option Pricing under the Ro...Chiheb Ben Hammouda
Conference talk at the SIAM Conference on Financial Mathematics and Engineering, held in virtual format, June 1-4 2021, about our recently published work "Hierarchical adaptive sparse grids and quasi-Monte Carlo for option pricing under the rough Bergomi model".
- Link of the paper: https://www.tandfonline.com/doi/abs/10.1080/14697688.2020.1744700
A One-Pass Triclustering Approach: Is There any Room for Big Data?Dmitrii Ignatov
An efficient one-pass online algorithm for triclustering of binary data (triadic formal contexts) is proposed. This algorithm is a modified version of the basic algorithm for OAC-triclustering approach, but it has linear time and memory complexities with respect to the cardinality
of the underlying ternary relation and can be easily parallelized in order to be applied for the analysis of big datasets. The results of computer experiments show the efficiency of the proposed algorithm.
We present recent result on the numerical analysis of Quasi Monte-Carlo quadrature methods, applied to forward and inverse uncertainty quantification for elliptic and parabolic PDEs. Particular attention will be placed on Higher
-Order QMC, the stable and efficient generation of
interlaced polynomial lattice rules, and the numerical analysis of multilevel QMC Finite Element discretizations with applications to computational uncertainty quantification.
The peer-reviewed International Journal of Engineering Inventions (IJEI) is started with a mission to encourage contribution to research in Science and Technology. Encourage and motivate researchers in challenging areas of Sciences and Technology.
R package 'bayesImageS': a case study in Bayesian computation using Rcpp and ...Matt Moores
There are many approaches to Bayesian computation with intractable likelihoods, including the exchange algorithm, approximate Bayesian computation (ABC), thermodynamic integration, and composite likelihood. These approaches vary in accuracy as well as scalability for datasets of significant size. The Potts model is an example where such methods are required, due to its intractable normalising constant. This model is a type of Markov random field, which is commonly used for image segmentation. The dimension of its parameter space increases linearly with the number of pixels in the image, making this a challenging application for scalable Bayesian computation. My talk will introduce various algorithms in the context of the Potts model and describe their implementation in C++, using OpenMP for parallelism. I will also discuss the process of releasing this software as an open source R package on the CRAN repository.
Low rank tensor approximation of probability density and characteristic funct...Alexander Litvinenko
Very often one has to deal with high-dimensional random variables (RVs). A high-dimensional RV can be described by its probability density (\pdf) and/or by the corresponding probability characteristic functions (\pcf), or by a function representation. Here the interest is mainly to compute characterisations like the entropy, or
relations between two distributions, like their Kullback-Leibler divergence, or more general measures such as $f$-divergences,
among others. These are all computed from the \pdf, which is often not available directly, and it is a computational challenge to even represent it in a numerically feasible fashion in case the dimension $d$ is even moderately large. It is an even stronger numerical challenge to then actually compute said characterisations in the high-dimensional case.
In this regard, in order to achieve a computationally feasible task, we propose to represent the density by a high order tensor product, and approximate this in a low-rank format.
Computing f-Divergences and Distances of\\ High-Dimensional Probability Densi...Alexander Litvinenko
Talk presented on SIAM IS 2022 conference.
Very often, in the course of uncertainty quantification tasks or
data analysis, one has to deal with high-dimensional random variables (RVs)
(with values in $\Rd$). Just like any other RV,
a high-dimensional RV can be described by its probability density (\pdf) and/or
by the corresponding probability characteristic functions (\pcf),
or a more general representation as
a function of other, known, random variables.
Here the interest is mainly to compute characterisations like the entropy, the Kullback-Leibler, or more general
$f$-divergences. These are all computed from the \pdf, which is often not available directly,
and it is a computational challenge to even represent it in a numerically
feasible fashion in case the dimension $d$ is even moderately large. It
is an even stronger numerical challenge to then actually compute said characterisations
in the high-dimensional case.
In this regard, in order to achieve a computationally feasible task, we propose
to approximate density by a low-rank tensor.
Computing f-Divergences and Distances of High-Dimensional Probability Density...Alexander Litvinenko
Poster presented on Stochastic Numerics and Statistical Learning: Theory and Applications Workshop in KAUST, Saudi Arabia.
The task considered here was the numerical computation of characterising statistics of
high-dimensional pdfs, as well as their divergences and distances,
where the pdf in the numerical implementation was assumed discretised on some regular grid.
Even for moderate dimension $d$, the full storage and computation with such objects become very quickly infeasible.
We have demonstrated that high-dimensional pdfs,
pcfs, and some functions of them
can be approximated and represented in a low-rank tensor data format.
Utilisation of low-rank tensor techniques helps to reduce the computational complexity
and the storage cost from exponential $\C{O}(n^d)$ to linear in the dimension $d$, e.g.
O(d n r^2) for the TT format. Here $n$ is the number of discretisation
points in one direction, r<n is the maximal tensor rank, and d the problem dimension.
The particular data format is rather unimportant,
any of the well-known tensor formats (CP, Tucker, hierarchical Tucker, tensor-train (TT)) can be used,
and we used the TT data format. Much of the presentation and in fact the central train
of discussion and thought is actually independent of the actual representation.
In the beginning it was motivated through three possible ways how one may
arrive at such a representation of the pdf. One was if the pdf was given in some approximate
analytical form, e.g. like a function tensor product of lower-dimensional pdfs with a
product measure, or from an analogous representation of the pcf and subsequent use of the
Fourier transform, or from a low-rank functional representation of a high-dimensional
RV, again via its pcf.
The theoretical underpinnings of the relation between pdfs and pcfs as well as their
properties were recalled in Section: Theory, as they are important to be preserved in the
discrete approximation. This also introduced the concepts of the convolution and of
the point-wise multiplication Hadamard algebra, concepts which become especially important if
one wants to characterise sums of independent RVs or mixture models,
a topic we did not touch on for the sake of brevity but which follows very naturally from
the developments here. Especially the Hadamard algebra is also
important for the algorithms to compute various point-wise functions in the sparse formats.
Multidimensional integrals may be approximated by weighted averages of integrand values. Quasi-Monte Carlo (QMC) methods are more accurate than simple Monte Carlo methods because they carefully choose where to evaluate the integrand. This tutorial focuses on how quickly QMC methods converge to the correct answer as the number of integrand values increases. The answer may depend on the smoothness of the integrand and the sophistication of the QMC method. QMC error analysis may assumes the integrand belongs to a reproducing kernel Hilbert space or may assume that the integrand is an instance of a stochastic process with known covariance structure. These two approaches have interesting parallels. This tutorial also explores how the computational cost of achieving a good approximation to the integral depends on the dimension of the domain of the integrand. Finally, this tutorial explores methods for determining how many integrand values are needed to satisfy the error tolerance. Relevant software is described.
New data structures and algorithms for \\post-processing large data sets and ...Alexander Litvinenko
In this work, we describe advanced numerical tools for working with multivariate functions and for
the analysis of large data sets. These tools will drastically reduce the required computing time and the
storage cost, and, therefore, will allow us to consider much larger data sets or ner meshes. Covariance
matrices are crucial in spatio-temporal statistical tasks, but are often very expensive to compute and
store, especially in 3D. Therefore, we approximate covariance functions by cheap surrogates in a
low-rank tensor format. We apply the Tucker and canonical tensor decompositions to a family of
Matern- and Slater-type functions with varying parameters and demonstrate numerically that their
approximations exhibit exponentially fast convergence. We prove the exponential convergence of the
Tucker and canonical approximations in tensor rank parameters. Several statistical operations are
performed in this low-rank tensor format, including evaluating the conditional covariance matrix,
spatially averaged estimation variance, computing a quadratic form, determinant, trace, loglikelihood,
inverse, and Cholesky decomposition of a large covariance matrix. Low-rank tensor approximations
reduce the computing and storage costs essentially. For example, the storage cost is reduced from an
exponential O(nd) to a linear scaling O(drn), where d is the spatial dimension, n is the number of
mesh points in one direction, and r is the tensor rank. Prerequisites for applicability of the proposed
techniques are the assumptions that the data, locations, and measurements lie on a tensor (axesparallel)
grid and that the covariance function depends on a distance,...
Response Surface in Tensor Train format for Uncertainty QuantificationAlexander Litvinenko
We apply low-rank Tensor Train format to solve PDEs with uncertain coefficients. First, we approximate uncertain permeability coefficient in TT format, then the operator and then apply iterations to solve stochastic Galerkin system.
In this work we discuss how to compute KLE with complexity O(k n log n), how to approximate large covariance matrices (in H-matrix format), how to use the Lanczos method.
We solve elliptic PDE with uncertain coefficients. We apply Karhunen-Loeve expansion to separate stochastic part from spatial part. The corresponding eigenvalue problem with covariance function is solved via the Hierarchical Matrix technique. We also demonstrate how low-rank tensor method can be applied for high-dimensional problems (e.g., to compute higher order statistical moments) . We provide explicit formulas to compute statistical moments of order k with linear complexity.
Low-rank methods for analysis of high-dimensional data (SIAM CSE talk 2017) Alexander Litvinenko
Overview of our latest works in applying low-rank tensor techniques to a) solving PDEs with uncertain coefficients (or multi-parametric PDEs) b) postprocessing high-dimensional data c) compute the largest element, level sets, TOP5% elelments
Existence of Solutions of Fractional Neutral Integrodifferential Equations wi...inventionjournals
International Journal of Engineering and Science Invention (IJESI) is an international journal intended for professionals and researchers in all fields of computer science and electronics. IJESI publishes research articles and reviews within the whole field Engineering Science and Technology, new teaching methods, assessment, validation and the impact of new technologies and it will continue to provide information on the latest trends and developments in this ever-expanding subject. The publications of papers are selected through double peer reviewed to ensure originality, relevance, and readability. The articles published in our journal can be accessed online.
ON RUN-LENGTH-CONSTRAINED BINARY SEQUENCESijitjournal
A class of binary sequences, constrained with respect to the length of zero runs, is considered.
For such sequences, termed (d, k)-sequences, new combinatorial and computational results
are established. Explicit expressions for enumerating (d, k)-sequences of finite length are
obtained. Efficient computational procedures for calculating the capacity of a (d, k)-code are
given. A simple method for constructing a near-optimal (d, k)-code is proposed. Illustrative
numerical examples demonstrate further the theoretical results.
Saltwater intrusion occurs when sea levels rise and saltwater moves onto the land. Usually, this occurs during storms, high tides, droughts, or when saltwater penetrates freshwater aquifers and raises the groundwater table. Since groundwater is an essential nutrition and irrigation resource, its salinization may lead to catastrophic consequences. Many acres of farmland may be lost because they can become too wet or salty to grow crops. Therefore, accurate modeling of different scenarios of saline flow is essential to help farmers and researchers develop strategies to improve the soil quality and decrease saltwater intrusion effects.
Saline flow is density-driven and described by a system of time-dependent nonlinear partial differential equations (PDEs). It features convection dominance and can demonstrate very complicated behavior.
As a specific model, we consider a Henry-like problem with uncertain permeability and porosity.
These parameters may strongly affect the flow and transport of salt.
Poster to be presented at Stochastic Numerics and Statistical Learning: Theory and Applications Workshop 2024, Kaust, Saudi Arabia, https://cemse.kaust.edu.sa/stochnum/events/event/snsl-workshop-2024.
In this work we have considered a setting that mimics the Henry problem \cite{Simpson2003,Simpson04_Henry}, modeling seawater intrusion into a 2D coastal aquifer. The pure water recharge from the ``land side'' resists the salinisation of the aquifer due to the influx of saline water through the ``sea side'', thereby achieving some equilibrium in the salt concentration. In our setting, following \cite{GRILLO2010}, we consider a fracture on the sea side that significantly increases the permeability of the porous medium.
The flow and transport essentially depend on the geological parameters of the porous medium, including the fracture. We investigated the effects of various uncertainties on saltwater intrusion. We assumed uncertainties in the fracture width, the porosity of the bulk medium, its permeability and the pure water recharge from the land side. The porosity and permeability were modeled by random fields, the recharge by a random but periodic intensity and the thickness by a random variable. We calculated the mean and variance of the salt mass fraction, which is also uncertain.
The main question we investigated in this work was how well the MLMC method can be used to compute statistics of different QoIs. We found that the answer depends on the choice of the QoI. First, not every QoI requires a hierarchy of meshes and MLMC. Second, MLMC requires stable convergence rates for $\EXP{g_{\ell} - g_{\ell-1}}$ and $\Var{g_{\ell} - g_{\ell-1}}$. These rates should be independent of $\ell$. If these convergence rates vary for different $\ell$, then it will be hard to estimate $L$ and $m_{\ell}$, and MLMC will either not work or be suboptimal. We were not able to get stable convergence rates for all levels $\ell=1,\ldots,5$ when the QoI was an integral as in \eqref{eq:integral_box}. We found that for $\ell=1,\ldots 4$ and $\ell=5$ the rate $\alpha$ was different. Further investigation is needed to find the reason for this. Another difficulty is the dependence on time, i.e. the number of levels $L$ and the number of sums $m_{\ell}$ depend on $t$. At the beginning the variability is small, then it increases, and after the process of mixing salt and fresh water has stopped, the variance decreases again.
The number of random samples required at each level was estimated by calculating the decay of the variances and the computational cost for each level. These estimates depend on the minimisation function in the MLMC algorithm.
To achieve the efficiency of the MLMC approach presented in this work, it is essential that the complexity of the numerical solution of each random realisation is proportional to the number of grid vertices on the grid levels.
We investigated the applicability and efficiency of the MLMC approach to the Henry-like problem with uncertain porosity, permeability and recharge. These uncertain parameters were modelled by random fields with three independent random variables. Permeability is a function of porosity. Both functions are time-dependent, have multi-scale behaviour and are defined for two layers. The numerical solution for each random realisation was obtained using the well-known ug4 parallel multigrid solver. The number of random samples required at each level was estimated by calculating the decay of the variances and the computational cost for each level.
The MLMC method was used to compute the expected value and variance of several QoIs, such as the solution at a few preselected points $(t,\bx)$, the solution integrated over a small subdomain, and the time evolution of the freshwater integral. We have found that some QoIs require only 2-3 mesh levels and samples from finer meshes would not significantly improve the result. Other QoIs require more grid levels.
1. Investigated efficiency of MLMC for Henry problem with
uncertain porosity, permeability, and recharge.
2. Uncertainties are modeled by random fields.
3. MLMC could be much faster than MC, 3200 times faster !
4. The time dependence is challenging.
Remarks:
1. Check if MLMC is needed.
2. The optimal number of samples depends on the point (t;x)
3. An advanced MLMC may give better estimates of L and m`.
Density Driven Groundwater Flow with Uncertain Porosity and PermeabilityAlexander Litvinenko
In this work, we solved the density driven groundwater flow problem with uncertain porosity and permeability. An accurate solution of this time-dependent and non-linear problem is impossible because of the presence of natural uncertainties in the reservoir such as porosity and permeability.
Therefore, we estimated the mean value and the variance of the solution, as well as the propagation of uncertainties from the random input parameters to the solution.
We started by defining the Elder-like problem. Then we described the multi-variate polynomial approximation (\gPC) approach and used it to estimate the required statistics of the mass fraction.
Utilizing the \gPC method allowed us
to reduce the computational cost compared to the classical quasi Monte Carlo method.
\gPC assumes that the output function $\sol(t,\bx,\thetab)$ is square-integrable and smooth w.r.t uncertain input variables $\btheta$.
Many factors, such as non-linearity, multiple solutions, multiple stationary states, time dependence and complicated solvers, make the investigation of the convergence of the \gPC method a non-trivial task.
We used an easy-to-implement, but only sub-optimal \gPC technique to quantify the uncertainty. For example, it is known that by increasing the degree of global polynomials (Hermite, Langange and similar), Runge's phenomenon appears. Here, probably local polynomials, splines or their mixtures would be better. Additionally, we used an easy-to-parallelise quadrature rule, which was also only suboptimal. For instance, adaptive choice of sparse grid (or collocation) points \cite{ConradMarzouk13,nobile-sg-mc-2015,Sudret_sparsePCE,CONSTANTINE12,crestaux2009polynomial} would be better, but we were limited by the usage of parallel methods. Adaptive quadrature rules are not (so well) parallelisable. In conclusion, we can report that: a) we developed a highly parallel method to quantify uncertainty in the Elder-like problem; b) with the \gPC of degree 4 we can achieve similar results as with the \QMC method.
In the numerical section we considered two different aquifers - a solid parallelepiped and a solid elliptic cylinder. One of our goals was to see how the domain geometry influences the formation, the number and the shape of fingers.
Since the considered problem is nonlinear,
a high variance in the porosity may result in totally different solutions; for instance, the number of fingers, their intensity and shape, the propagation time, and the velocity may vary considerably.
The number of cells in the presented experiments varied from $241{,}152$ to $15{,}433{,}728$ for the cylindrical domain and from $524{,}288$ to $4{,}194{,}304$ for the parallelepiped. The maximal number of parallel processing units was $600\times 32$, where $600$ is the number of parallel nodes and $32$ is the number of computing cores on each node. The total computing time varied from 2 hours for the coarse mesh to 24 hours for the finest mesh.
We consider a class of density-driven flow problems. We are particularly interested in the problem of the salinization of coastal aquifers. We consider the Henry saltwater intrusion problem with uncertain porosity, permeability, and recharge parameters as a test case.
The reason for the presence of uncertainties is the lack of knowledge, inaccurate measurements,
and inability to measure parameters at each spatial or time location. This problem is nonlinear and time-dependent. The solution is the salt mass fraction, which is uncertain and changes in time. Uncertainties in porosity, permeability, recharge, and mass fraction are modeled using random fields. This work investigates the applicability of the well-known multilevel Monte Carlo (MLMC) method for such problems. The MLMC method can reduce the total computational and storage costs. Moreover, the MLMC method runs multiple scenarios on different spatial and time meshes and then estimates the mean value of the mass fraction.
The parallelization is performed in both the physical space and stochastic space. To solve every deterministic scenario, we run the parallel multigrid solver ug4 in a black-box fashion.
We use the solution obtained from the quasi-Monte Carlo method as a reference solution.
We investigated the applicability and efficiency of the MLMC approach for the Henry-like problem with uncertain porosity, permeability, and recharge. These uncertain parameters were modeled by random fields with three independent random variables. The numerical solution for each random realization was obtained using the well-known ug4 parallel multigrid solver. The number of required random samples on each level was estimated by computing the decay of the variances and computational costs for each level. We also computed the expected value and variance of the mass fraction in the whole domain, the evolution of the pdfs, the solutions at a few preselected points $(t,\bx)$, and the time evolution of the freshwater integral value. We have found that some QoIs require only 2-3 of the coarsest mesh levels, and samples from finer meshes would not significantly improve the result. Note that a different type of porosity may lead to a different conclusion.
The results show that the MLMC method is faster than the QMC method at the finest mesh. Thus, sampling at different mesh levels makes sense and helps to reduce the overall computational cost.
Here the interest is mainly to compute characterisations like the entropy,
the Kullback-Leibler divergence, more general $f$-divergences, or other such characteristics based on
the probability density. The density is often not available directly,
and it is a computational challenge to just represent it in a numerically
feasible fashion in case the dimension is even moderately large. It
is an even stronger numerical challenge to then actually compute said characteristics
in the high-dimensional case.
The task considered here was the numerical computation of characterising statistics of
high-dimensional pdfs, as well as their divergences and distances,
where the pdf in the numerical implementation was assumed discretised on some regular grid.
We have demonstrated that high-dimensional pdfs,
pcfs, and some functions of them
can be approximated and represented in a low-rank tensor data format.
Utilisation of low-rank tensor techniques helps to reduce the computational complexity
and the storage cost from exponential $\C{O}(n^d)$ to linear in the dimension $d$, e.g.\
$O(d n r^2 )$ for the TT format. Here $n$ is the number of discretisation
points in one direction, $r<<n$ is the maximal tensor rank, and $d$ the problem dimension.
Talk presented on this workshop "Workshop: Imaging With Uncertainty Quantification (IUQ), September 2022",
https://people.compute.dtu.dk/pcha/CUQI/IUQworkshop.html
We consider a weakly supervised classification problem. It
is a classification problem where the target variable can be unknown
or uncertain for some subset of samples. This problem appears when
the labeling is impossible, time-consuming, or expensive. Noisy measurements
and lack of data may prevent accurate labeling. Our task
is to build an optimal classification function. For this, we construct and
minimize a specific objective function, which includes the fitting error on
labeled data and a smoothness term. Next, we use covariance and radial AQ1
basis functions to define the degree of similarity between points. The further
process involves the repeated solution of an extensive linear system
with the graph Laplacian operator. To speed up this solution process,
we introduce low-rank approximation techniques. We call the resulting
algorithm WSC-LR. Then we use the WSC-LR algorithm for analysis
CT brain scans to recognize ischemic stroke disease. We also compare
WSC-LR with other well-known machine learning algorithms.
Identification of unknown parameters and prediction of missing values. Compar...Alexander Litvinenko
H-matrix approximation of large Mat\'{e}rn covariance matrices, Gaussian log-likelihoods.
Identifying unknown parameters and making predictions
Comparison with machine learning methods.
kNN is easy to implement and shows promising results.
Computation of electromagnetic fields scattered from dielectric objects of un...Alexander Litvinenko
We develop fast and efficient stochastic methods for characterizing scattering
from objects of uncertain shapes. This is highly needed in the
fields of electromagnetics, optics, and photonics.
The continuation multilevel Monte Carlo (CMLMC) method is
used together with a surface integral equation solver. The
CMLMC method optimally balances statistical errors due to
sampling of the parametric space, and numerical errors due
to the discretization of the geometry using a hierarchy of
discretizations, from coarse to fine. The number of realizations
of finer discretizations can be kept low, with most samples
computed on coarser discretizations to minimize computational
work. Consequently, the total execution time is significantly
reduced, in comparison to the standard MC scheme.
Identification of unknown parameters and prediction with hierarchical matrice...Alexander Litvinenko
We compare four numerical methods for the prediction of missing values in four different datasets.
These methods are 1) the hierarchical maximum likelihood estimation (H-MLE), and three machine learning (ML) methods, which include 2) k-nearest neighbors (kNN), 3) random forest, and 4) Deep Neural Network (DNN).
From the ML methods, the best results (for considered datasets) were obtained by the kNN method with three (or seven) neighbors.
On one dataset, the MLE method showed a smaller error than the kNN method, whereas, on another, the kNN method was better.
The MLE method requires a lot of linear algebra computations and works fine on almost all datasets. Its result can be improved by taking a smaller threshold and more accurate hierarchical matrix arithmetics. To our surprise, the well-known kNN method produces similar results as H-MLE and worked much faster.
Computation of electromagnetic fields scattered from dielectric objects of un...Alexander Litvinenko
Computational tools for characterizing electromagnetic scattering from objects with uncertain shapes are needed in various applications ranging from remote sensing at microwave frequencies to Raman spectroscopy at optical frequencies. Often, such computational tools use the Monte Carlo (MC) method to sample a parametric space describing geometric uncertainties. For each sample, which corresponds to a realization of the geometry, a deterministic electromagnetic solver computes the scattered fields. However, for an accurate statistical characterization the number of MC samples has to be large. In this work, to address this challenge, the continuation multilevel Monte Carlo (\CMLMC) method is used together with a surface integral equation solver.
The \CMLMC method optimally balances statistical errors due to sampling of
the parametric space, and numerical errors due to the discretization of the geometry using a hierarchy of discretizations, from coarse to fine.
The number of realizations of finer discretizations can be kept low, with most samples
computed on coarser discretizations to minimize computational cost.
Consequently, the total execution time is significantly reduced, in comparison to the standard MC scheme.
Computation of electromagnetic fields scattered from dielectric objects of un...Alexander Litvinenko
Computational tools for characterizing electromagnetic scattering from objects with uncertain shapes are needed in various applications ranging from remote sensing at microwave frequencies to Raman spectroscopy at optical frequencies. Often, such computational tools use the Monte Carlo (MC) method to sample a parametric space describing geometric uncertainties. For each sample, which corresponds to a realization of the geometry, a deterministic electromagnetic solver computes the scattered fields. However, for an accurate statistical characterization the number of MC samples has to be large. In this work, to address this challenge, the continuation multilevel Monte Carlo (\CMLMC) method is used together with a surface integral equation solver.
The \CMLMC method optimally balances statistical errors due to sampling of
the parametric space, and numerical errors due to the discretization of the geometry using a hierarchy of discretizations, from coarse to fine.
The number of realizations of finer discretizations can be kept low, with most samples
computed on coarser discretizations to minimize computational cost.
Consequently, the total execution time is significantly reduced, in comparison to the standard MC scheme.
Propagation of Uncertainties in Density Driven Groundwater FlowAlexander Litvinenko
Major Goal: estimate risks of the pollution in a subsurface flow.
How?: we solve density-driven groundwater flow with uncertain porosity and permeability.
We set up density-driven groundwater flow problem,
review stochastic modeling and stochastic methods, use UG4 framework (https://gcsc.uni-frankfurt.de/simulation-and-modelling/ug4),
model uncertainty in porosity and permeability,
2D and 3D numerical experiments.
Simulation of propagation of uncertainties in density-driven groundwater flowAlexander Litvinenko
Consider stochastic modelling of the density-driven subsurface flow in 3D. This talk was presented by Dmitry Logashenko on the IMG conference in Kunming, China, August 2019.
Large data sets result large dense matrices, say with 2.000.000 rows and columns. How to work with such large matrices? How to approximate them? How to compute log-likelihood? determination? inverse? All answers are in this work.
In this paper, we solve a semi-supervised regression
problem. Due to the luck of knowledge about the
data structure and the presence of random noise, the considered data model is uncertain. We propose a method which combines graph Laplacian regularization and cluster ensemble methodologies. The co-association matrix of the ensemble is calculated on both labeled and unlabeled data; this matrix is used as a similarity matrix in the regularization framework to derive the predicted outputs. We use the low-rank decomposition of the co-association matrix to significantly speedup calculations and reduce memory. Two clustering problem examples are presented.
Full version is here https://arxiv.org/abs/1901.03919
Major Goal: estimate risks of the pollution in a subsurface flow.
How? We solve density-driven groundwater flow with uncertain porosity and permeability.
1. We set up density-driven groundwater flow problem
2. Review stochastic modeling and stochastic methods
3. Modeling of uncertainty in porosity and permeability
4. Numerical methods to solve deterministic problem
5. 2D and 3D examples with 0.5-8 Millions mesh points.
Efficient Simulations for Contamination of Groundwater Aquifers under Uncerta...Alexander Litvinenko
1. Solved time-dependent density driven flow problem with uncertain porosity and permeability in 2D and 3D
2. Computed propagation of uncertainties in porosity into the mass fraction.
3. Computed the mean, variance, exceedance probabilities, quantiles, risks.
4. Such QoIs as the number of fingers, their size, shape, propagation time can be unstable
5. For moderate perturbations, our gPCE surrogate results are similar to qMC results.
6. Used highly scalable solver on up to 800 computing nodes,
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
The Roman Empire A Historical Colossus.pdfkaushalkr1407
The Roman Empire, a vast and enduring power, stands as one of history's most remarkable civilizations, leaving an indelible imprint on the world. It emerged from the Roman Republic, transitioning into an imperial powerhouse under the leadership of Augustus Caesar in 27 BCE. This transformation marked the beginning of an era defined by unprecedented territorial expansion, architectural marvels, and profound cultural influence.
The empire's roots lie in the city of Rome, founded, according to legend, by Romulus in 753 BCE. Over centuries, Rome evolved from a small settlement to a formidable republic, characterized by a complex political system with elected officials and checks on power. However, internal strife, class conflicts, and military ambitions paved the way for the end of the Republic. Julius Caesar’s dictatorship and subsequent assassination in 44 BCE created a power vacuum, leading to a civil war. Octavian, later Augustus, emerged victorious, heralding the Roman Empire’s birth.
Under Augustus, the empire experienced the Pax Romana, a 200-year period of relative peace and stability. Augustus reformed the military, established efficient administrative systems, and initiated grand construction projects. The empire's borders expanded, encompassing territories from Britain to Egypt and from Spain to the Euphrates. Roman legions, renowned for their discipline and engineering prowess, secured and maintained these vast territories, building roads, fortifications, and cities that facilitated control and integration.
The Roman Empire’s society was hierarchical, with a rigid class system. At the top were the patricians, wealthy elites who held significant political power. Below them were the plebeians, free citizens with limited political influence, and the vast numbers of slaves who formed the backbone of the economy. The family unit was central, governed by the paterfamilias, the male head who held absolute authority.
Culturally, the Romans were eclectic, absorbing and adapting elements from the civilizations they encountered, particularly the Greeks. Roman art, literature, and philosophy reflected this synthesis, creating a rich cultural tapestry. Latin, the Roman language, became the lingua franca of the Western world, influencing numerous modern languages.
Roman architecture and engineering achievements were monumental. They perfected the arch, vault, and dome, constructing enduring structures like the Colosseum, Pantheon, and aqueducts. These engineering marvels not only showcased Roman ingenuity but also served practical purposes, from public entertainment to water supply.
Welcome to TechSoup New Member Orientation and Q&A (May 2024).pdfTechSoup
In this webinar you will learn how your organization can access TechSoup's wide variety of product discount and donation programs. From hardware to software, we'll give you a tour of the tools available to help your nonprofit with productivity, collaboration, financial management, donor tracking, security, and more.
Ethnobotany and Ethnopharmacology:
Ethnobotany in herbal drug evaluation,
Impact of Ethnobotany in traditional medicine,
New development in herbals,
Bio-prospecting tools for drug discovery,
Role of Ethnopharmacology in drug evaluation,
Reverse Pharmacology.
We all have good and bad thoughts from time to time and situation to situation. We are bombarded daily with spiraling thoughts(both negative and positive) creating all-consuming feel , making us difficult to manage with associated suffering. Good thoughts are like our Mob Signal (Positive thought) amidst noise(negative thought) in the atmosphere. Negative thoughts like noise outweigh positive thoughts. These thoughts often create unwanted confusion, trouble, stress and frustration in our mind as well as chaos in our physical world. Negative thoughts are also known as “distorted thinking”.
How to Create Map Views in the Odoo 17 ERPCeline George
The map views are useful for providing a geographical representation of data. They allow users to visualize and analyze the data in a more intuitive manner.
The Art Pastor's Guide to Sabbath | Steve ThomasonSteve Thomason
What is the purpose of the Sabbath Law in the Torah. It is interesting to compare how the context of the law shifts from Exodus to Deuteronomy. Who gets to rest, and why?
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
2. Objectives of low-rank/sparse data approximations
1. Drastically reduce computing time and memory
requirements. This will reduce energy consumption and CO2
pollution
2. Extract knowledge from large high-dimensional datasets
How?
1. Develop new low-rank/sparse data structures/formats
2. Represent/approximate multidimensional operators and
functions in these data formats
3. Develop (new) linear algebra algorithms for these tensor
formats
1 / 52
3. Used
I Tensor book of W. Hackbusch 2012, and 2 books of Boris and
Venera Khoromskij
I Dissertations of I. Oseledets and M. Espig
I Articles of Tyrtyshnikov et al., De Lathauwer et al., L.
Grasedyck, B. Khoromskij, M. Espig
I Lecture courses and presentations of Boris and Venera
Khoromskij, D. Kressner
I Software
– T. Kolda et al.;
– M. Espig et al.;
– D. Kressner, K. Tobler;
– I. Oseledets et al.
– L. De Lathauwer
2 / 52
4. History (not full list) of using tensor approximations
I Canonical in 1927, Tucker in 1966, Tensor Train in 2010.
I 1997 : signal processing (Lieven De Lathauwer)
I 2005 and later: for computational physics and chemistry -
Hartree–Fock, Schrödinger equations (Hackbusch,
Tyrtyshnikov, Kressner, Espig, Khoromskij(aja), Grasedyck,
Oseledets ...)
I 2007: uncertainty quantification, SPDEs, parametric PDEs
(Nobile, all above)
I ∼2009: spatical statistics
I ∼2015: Machine Learning
3 / 52
5. Challenging applications
I quantum mechanics
I modelling of multi-particle interactions in large molecular
systems such as proteins, biomolecules,
I modelling of large atomic (metallic) clusters,
I stochastic and parametric equations,
I machine learning, data mining and information technologies,
I multidimensional dynamical systems,
I data compression
I financial mathematics,
I analysis of satellite data.
4 / 52
6. Curse of dimensionality
Assume we have nd data. Our aim is to reduce
storage/complexity from O(nd) to O(dn).
If n = 100 and d = 10, then just to store one needs
8 · 10010 ≈ 8 · 1020 = 8 · 108 TeraBytes.
If we assume that a modern computer compares 107 numbers
per second, then the total time for comparison 1020 elements
will be 1013 seconds or ≈ 3 ∗ 105 years. In some chemical
applications we had n = 100 and d = 800.
I how to compute maxima and minima ?
I how to compute level sets, i.e. all elements from an interval
[a,b] ?
I how to compute the number of elements in an interval [a,b] ?
5 / 52
7. Example: Tensors appear in stochastic PDEs
−∇ · (κ(x,ω)∇u(x,ω)) = f(x,ω), x ∈ G ⊂ Rd
where ω ∈ Ω, and U = L2(G).
Write first Karhunen-Loeve Expansion and then for
uncorrelated random variables the Polynomial Chaos
Expansion
u(x,ω) =
K
X
i=1
p
λiϕi(x)ξi(ω) =
K
X
i=1
p
λiϕi(x)
X
α∈J
ξ
(α)
i Hα(θ(ω))
(1)
where ξ
(α)
i is a tensor. Note that α = (α1,α2,...,αM,...) is a
multi-index.
X
α∈J
ξ
(α)
i Hα(θ(ω)) :=
p1
X
α1=1
...
pM
X
αM=1
ξ
(α1,...,αM)
i
M
Y
j=1
hαj
(θj) (2)
The same decomposition for κ(x,ω). 6 / 52
8. Final discretized stochastic PDE
Au = f, where
A:=
Ps
l=1 Ãl ⊗
NM
µ=1 ∆lµ
, Ãl ∈ RN×N, ∆lµ ∈ RRµ×Rµ,
u:=
Pr
j=1 uj ⊗
NM
µ=1 ujµ
, uj ∈ RN, ujµ ∈ RRµ,
f:=
PR
k=1 f̃k ⊗
NM
µ=1 gkµ, f̃k ∈ RN and gkµ ∈ RRµ.
Examples of stochastic Galerkin matrices:
And then solve iteratively with a tensor preconditioner [PhD of E. Zander, 2012]
M Espig, W Hackbusch, A Litvinenko, HG Matthies, P Wähnert, Efficient low-rank approximation of the stochastic Galerkin matrix in tensor formats Computers
Mathematics with Applications 67 (4), 818-829, 2014
Also see E. Ullmann, Chr. Schwab, B. Khoromskij, Schneider, Ballani, Kressner, Tobler and many-many others.
7 / 52
9. Tensor of order 2
Let M := UΣVT ≈ ŨΣ̃ṼT = Mk, k min{n,m}.
(Truncated Singular Value Decomposition).
Denote A := ŨΣ̃ and B := Ṽ, then Mk = ABT.
Storage of A and BT is k(n + m) in contrast to nm for M.
U V
Σ
T
=
M
U
V
Σ
∼
∼ ∼ T
=
M
∼
8 / 52
10. Arithmetic operations with low-rank matrices
Let v ∈ Rm.
Suppose Mk = ABT ∈ Rn×m, A ∈ Rn×k, B ∈ Rm×k is given.
MV product: Mkv = ABTv = (A(BTv)). Cost O(km + kn).
Suppose M
0
= CDT, C ∈ Rn×k and D ∈ Rm×k.
Matrix addition: Mk + M
0
= AnewBT
new, Anew := [A C] ∈ Rn×2k
and Bnew = [B D] ∈ Rm×2k.
Cost of rank truncation from rank 2k to k is O((n + m)k2 + k3).
9 / 52
11. Post-processing: Compute mean and variance
Let W := [v1,v2,...,vm], where vi are vectors (e.g., solution
vectors of a Navier-Stokes equation).
Given tSVD Wk = ABT ≈ W ∈ Rn×m, A := UkSk, B := Vk.
v =
1
m
m
X
i=1
vi =
1
m
m
X
i=1
A · bi = Ab, (3)
C =
1
m − 1
WcWT
c =
1
m − 1
ABT
BAT
=
1
m − 1
AAT
. (4)
Diagonal of C can be computed with the complexity
O(k2(m + n)).
If kW − Wkk ≤ ε, then
a) kv − vkk ≤ 1
√
n
ε,
b) kC − Ckk ≤ 1
m−1ε2.
10 / 52
12. Example from CFD and aerodynamics
Inflow and air-foil shape uncertain.
Data compression achieved by updated SVD:
Made from m = 600 MC Simulations, SVD is updated every 10
samples.
n = 260,000
Updated SVD: Relative errors, memory requirements:
rank k pressure turb. kin. energy memory [MB]
10 1.9e-2 4.0e-3 21
20 1.4e-2 5.9e-3 42
50 5.3e-3 1.5e-4 104
Dense matrix M ∈ R260000×600 costs 1250 MB storage.
1.A. Litvinenko, H.G. Matthies, T.A. El-Moselhy, Sampling and low-rank tensor approximation of the response surface, Monte Carlo and Quasi-Monte Carlo Methods 2012,
535-551, 2013
2. A. Litvinenko, H.G. Matthies, Numerical methods for uncertainty quantification and bayesian update in aerodynamics, Management and Minimisation of Uncertainties
and Errors in Numerical Aerodynamics, pp 265-282, Springer, Berlin, 2013
11 / 52
13. Example: a high-dimensional PDE
−∇2
u = f on G = [0,1]d
with u|∂G = 1,
and the right-hand-side
f(x1,...,xd) ∝
d
X
k=1
d
Y
`=1,`,k
x`(1 − x`).
Solved via finite-difference method with n = 100 grid-points in
each direction.
Tensor u has N = nd entries.
Applications: computing electron density and Hartree potential
of molecules (see Diss. of M. Espig).
12 / 52
14. Comp. time to compute the maximum
d # loc’s.: ≈ years [a] actual time [s]
N = nd inspect. N (see Espig’s diss.)
25 1050 1.6 × 1033 0.16
50 10100 1.6 × 1083 0.42
75 10150 1.6 × 10133 1.16
100 10200 1.6 × 10183 2.58
125 10250 1.6 × 10233 4.97
150 10300 1.6 × 10283 8.56
Assumed 2 × 109 FLOPs/sec. on an 2 GHz CPU.
M. Espig, W. Hackbusch, A. Litvinenko, H.G. Matthies, E. Zander, Iterative algorithms for the post-processing of high-dimensional data Journal of Computational Physics
410, 109396, 2020
13 / 52
15. A tensor is a multi-index array
where multi-indices are used instead of indices.
Let w ∈ RN, N = 1012.
We can reshape it into a matrix W ∈ R106×106
, which is a tensor
of 2nd order.
Or a tensor of 3rd order
R104×104×104
Or a tensor of 6th order
R102×...×102
(6 times).
Or a tensor of 12th order
R10×...×10 (12 times).
14 / 52
16. Definition of tensor of order d
Tensor of order d is a multidimensional array over a d-tuple
index set I = I1 × ··· × Id,
A = [ai1...id
: i` ∈ I`] ∈ RI
, I` = {1,...,n`}, ` = 1,..,d.
A is an element of the linear space
Vn =
d
O
`=1
V`, V` = RI`
equipped with the Euclidean scalar product h·,·i : Vn × Vn → R,
defined as
hA,Bi :=
X
(i1...id)∈I
ai1...id
bi1...id
, forA, B ∈ Vn.
15 / 52
17. Canonical, Tucker and TT tensor formats
Canonical in 1927 (F.L. Hitchcock), Tucker in 1966, TT in 2010.
a) Schema of the CP tensor decomposition of a 3D tensor; b)
Tucker; c) TT decompositions. The waggons denote the TT
cores and each wheel denotes the index iν.
16 / 52
18. Tensor formats: CP, Tucker, TT
A(i1,i2,i3) ≈
r
X
α=1
u1(i1,α)u2(i2,α)u3(i3,α)
A(i1,i2,i3) ≈
X
α1,α2,α3
c(α1,α2,α3)u1(i1,α1)u2(i2,α2)u3(i3,α3)
A(i1,...,id) ≈
X
α1,...,αd−1
G1(i1,α1)G2(α1,i2,α2)...Gd−1(αd−1,id)
Discrete: Gk(ik) is a rk−1 × rk matrix, r1 = rd = 1.
17 / 52
19. Tensor and Matrices
Rank-1 tensor
A = u1 ⊗ u2 ⊗ ... ⊗ ud =:
d
O
µ=1
uµ
Ai1,...,id
= (u1)i1
· ... · (ud)id
Rank-1 tensor A = u ⊗ v is equivalent to rank-1 matrix A = uvT,
where u ∈ Rn, v ∈ Rm,
Rank-k tensor A =
Pk
i=1 ui ⊗ vi, matrix A =
Pk
i=1 uivT
i .
Kronecker product A ⊗ B ∈ Rnm×nm is a block matrix whose ij-th
block is [AijB].
18 / 52
20. Examples (B. Khoromskij’s lecture)
Rank-1: f = exp(f1(x1) + ... + fd(xd)) =
Qd
j=1 exp(fj(xj))
Rank-2: f = sin(
Pd
j=1 xj), since
2i · sin(
Pd
j=1 xj) = ei
Pd
j=1 xj
− e−i
Pd
j=1 xj
Rank-d function f(x1,...,xd) = x1 + x2 + ... + xd can be
approximated by rank-2: with any prescribed accuracy:
f ≈
Qd
j=1(1 + εxj)
ε
−
Qd
j=1 1
ε
+ O(ε), as ε → 0
19 / 52
24. Definitions of CP
Let T :=
Nd
µ=1 Rnµ be the tensor product constructed from
vector spaces (Rnµ,h,iR
nµ) (d ≥ 3).
Tensor representation U is a multilinear map U : P → T , where
parametric space P =
D
ν=1 Pν (d ≤ D).
Further, Pν depends on some representation rank parameter
rν ∈ N.
A standard example of a tensor representation is the canonical
tensor format.
(!!!)We distinguish between a tensor v ∈ T and its tensor format
representation p ∈ P, where v = U(p).
23 / 52
25. r-Terms, Tensor Rank, Canonical Tensor Format
The set Rr of tensors which can be represented in T with
r-terms is defined as
Rr(T ) := Rr :=
r
X
i=1
d
O
µ=1
viµ ∈ T : viµ ∈ Rnµ
. (10)
Let v ∈ T . The tensor rank of v in T is
rank(v) := min{r ∈ N0 : v ∈ Rr}. (11)
Example: The Laplace operator in 3d:
∆3
= ∆1
⊗ I ⊗ I + I ⊗ ∆1
⊗ I + I ⊗ I ⊗ ∆1
24 / 52
26. Definitions of CP
The canonical tensor format is defined by the mapping
Ucp :
d
µ=1
Rnµ×r
→ Rr, (12)
v̂ := (viµ : 1 ≤ i ≤ r, 1 ≤ µ ≤ d) 7→ Ucp(v̂) :=
r
X
i=1
d
O
µ=1
viµ.
25 / 52
27. Properties of CP
Let r1,r2 ∈ N, u ∈ Rr1
and v ∈ Rr2
. We have
(i) hu,viT =
Pr1
j1=1
Pr2
j2=1
Qd
µ=1
D
uj1µ,vj2µ
E
R
nµ. The computational
cost of hu,viT is O
r1r2
Pd
µ=1 nµ
.
(ii) u + v ∈ Rr1+r2
.
(iii) u
30. v can be computed in the canonical
tensor format with r1r2
Pd
µ=1 nµ arithmetic operations.
Let R1 = A1BT
1 , R2 = A2BT
2 be rank-k matrices, then
R1 + R2 = [A1A2][B1B2]T be rank-2k matrix. Rank truncation!
26 / 52
31. Properties of Hadamard product and FT
Let u =
Pk
j=1
Nd
i=1 uji, uji ∈ Rn.
F [d]
(ũ) =
k
X
j=1
d
O
i=1
(Fi (ũji)), where F [d]
=
d
O
i=1
Fi. (13)
Let S = ABT =
Pk1
i=0 aibT
i ∈ Rn×m, T = CDT =
Pk2
j=0 cidT
i ∈ Rn×m
where ai, ci ∈ Rn, bi, di ∈ Rm, k1,k2,n,m 0. Then
F (2)
(S ◦ T) =
k1
X
i=0
k2
X
j=0
F (ai ◦ cj)F (bT
i ◦ dT
j ).
27 / 52
32. Tensor Format
A =
k1
X
i1=1
...
kd
X
id=1
ci1,...,id
· u1
i1
⊗ ... ⊗ ud
id
(14)
Core tensor c ∈ Rk1×...×kd , rank (k1,...,kd).
Nonlinear fixed rank approximation problem:
X = argminX minrank(k1,...,kd)kA − Xk (15)
I Problem is well-posed but not solved
I There are many local minima
I HOSVD (Lathauwer et al.) yields rank
(k1,...,kd)Y : kA − Yk ≤
√
dkA − Xk
I reliable arithmetic, exponential scaling (c ∈ Rk1×k2×...×kd )
28 / 52
33. Example: Canonical rank d, whereas TT rank 2
d-Laplacian over uniform tensor grid. It is known to have the
Kronecker rank-d representation,
∆d = A ⊗IN ⊗...⊗IN +IN ⊗A ⊗...⊗IN +...+IN ⊗IN ⊗...⊗A ∈ RI⊗d⊗I⊗d
(16)
with A = ∆1 = tridiag{−1,2,−1} ∈ RN×N, and IN being the N × N
identity. Notice that for the canonical rank we have rank
kC(∆d) = d, while TT-rank of ∆d is equal to 2 for any dimension
due to the explicit representation
∆d = (∆1 I) ×
I 0
∆1 I
!
× ... ×
I 0
∆1 I
!
×
I
∆1
!
(17)
where the rank product operation ”×” is defined as a regular
matrix product of the two corresponding core matrices, their
blocks being multiplied by means of tensor product.
29 / 52
34. Linear algebra in the CP format
w = u + v =
ru
X
j=1
d
O
ν=1
u
(ν)
j
+
rv
X
k=1
d
O
µ=1
v
(µ)
k
=
ru+rv
X
j=1
d
O
ν=1
w
(ν)
j ,
where w
(ν)
j := u
(ν)
j for j ≤ ru and w
(ν)
j := v
(ν)
j for ru j ≤ ru + rv.
Cost O(1).
The Hadamard product
w = u
37. v
(ν)
k
.
The new rank is generally ru × rv, and the computational cost is
O(ru rvn d) arithmetic operations.
30 / 52
38. The Euclidean inner product
is computed as follows:
hu|vi = h
ru
X
j=1
d
O
ν=1
u
(ν)
j |
rv
X
k=1
d
O
ν=1
v
(ν)
k i =
ru
X
j=1
rv
X
k=1
d
Y
ν=1
hu
(ν)
j |v
(ν)
k i
The computational cost of the inner product is O(ru rv n d).
31 / 52
39. Advantages and disadvantages
Denote k - rank, d-dimension, n = # dofs in 1D:
1. CP: ill-posed approx. alg-m [V. de Silva, L-H. Lim’08], O(dnk),
hard to compute approx.
2. Tucker: reliable arithmetic based on SVD, O(dnk + kd)
3. Hierarchical Tucker: based on SVD, storage O(dnk + dk3),
truncation O(dnk2 + dk4)
4. TT: based on SVD, O(dnk2) or O(dnk3), stable
5. Quantics-TT: O(nd) → O(dlogqn)
32 / 52
40. How to compute the mean value in CP format
Let u =
Pr
j=1
Nd
µ=1 ujµ ∈ Rr, then the mean value u can be
computed as a scalar product
u =
*
r
X
j=1
d
O
µ=1
ujµ
,
d
O
µ=1
1
nµ
1̃µ
+
=
r
X
j=1
d
O
µ=1
D
ujµ,1̃µ
E
nµ
= (18)
=
r
X
j=1
d
Y
µ=1
1
nµ
nµ
X
k=1
ujµ
, (19)
where 1̃µ := (1,...,1)T ∈ Rnµ.
Numerical cost is O
r ·
Pd
µ=1 nµ
.
33 / 52
41. How to compute the variance in CP format
Let u ∈ Rr and
ũ := u − u
d
O
µ=1
1
nµ
1 =
r+1
X
j=1
d
O
µ=1
ũjµ ∈ Rr+1, (20)
then the variance var(u) of u can be computed as follows
var(u) =
hũ,ũi
Qd
µ=1 nµ
=
1
Qd
µ=1 nµ
*
r+1
X
i=1
d
O
µ=1
ũiµ
,
r+1
X
j=1
d
O
ν=1
ũjν
+
=
r+1
X
i=1
r+1
X
j=1
d
Y
µ=1
1
nµ
D
ũiµ,ũjµ
E
.
Numerical cost is O
(r + 1)2 ·
Pd
µ=1 nµ
.
S. Dolgov, B.N. Khoromskij, A. Litvinenko, H.G. Matthies, Computation of the response surface in the tensor train data format, arXiv:1406.2816, 2014
34 / 52
42. Conclusion
We discussed:
I Motivation: why do we need low-rank tensors
I Tensors of the second order (matrices)
I CP, Tucker and tensor train tensor formats
I Many classical kernels have (or can be approximated in )
low-rank tensor format
I Post processing: Computation of mean, variance, level sets,
frequency
35 / 52
43. Tensor Software
Ivan Oseledets et al., Tensor Train toolbox (Matlab),
http://spring.inm.ras.ru/osel
D.Kressner, C. Tobler, Hierarchical Tucker Toolbox (Matlab),
http://www.sam.math.ethz.ch/NLAgroup/htucker toolbox.html
M. Espig, et al
Tensor Calculus library (C): http://gitorious.org/tensorcalculus
36 / 52
45. Two types of tensors
1. Function-related and 2. data-related
Function-related tensors we considered earlier:
Ex1: A d -dimensional function f(x1,...,xd) = sin(x1 + ... + xd)
discretised on an axis-parallel grid).
Ex2: solution of a stochastic PDE.
These tensors are usually given implicitly.
Data-related tensors:
Ex1: user i1 cited work i2 from author i3 published in year i4.
Ex2: a tensor 400 × 480 × 360 × 3 -containing 400 CT images of
size 480 × 360 pixels, 3 colours.
Ex3: disease forecast depends on temperature i1, blood
pressure i2, other blood parameters i3,i4,i5.
These tensors are given explicitly.
38 / 52
46. What is needed for tensor approximation?
1. Does a low-rank tensor approximation always exist?
Yes, it exists always. The question is only what is the rank
(ranks)? In the worst case the rank is huge, of order O(n) (or
O(N)).
2. What is needed for a low-rank tensor approximations?
For function-related tensors:
1. Decay of eigenvalues is crucial.
2. Low-rank tensor approximability and function separability
are strongly connected
3. Smoothness is not necessary. Piecewise smoothness can be
enough.
4. Smoothness is not sufficient
39 / 52
47. How to compute tensor decomposition?
This is not so easy (just give a short list):
1. Factors of CP decomposition: a minimization problem is
solved by a quasi Newton method or ALS (dissertation of M.
Espig, Leipzig 2007)
2. Factors of Tucker decomp.: HOSVD (De Lathauwer,...)
3. TT, Hierarchical Tucker: SVD, QR (Kressner, Tobler,
Hackbusch, Ballani, Grasedyck,...), Cauchy integral formula,
sinc quadrature (Boris and Venera Khoromskij,...)
4. (adaptive) cross methods (Dolgov, Oseledets, Bebendorf,...)
5. Successive rank-1 approximation(A. Nouy)
6. Randomized methods
See more in tensor book of W. Hackbusch’13, 2 books of Boris and Venera Khoromskij, book chapters of A. Nouy and I. Oseledets, two tensor overview papers by B.
Khoromskij’11 and by Kressner/Grasedyck/Tobler’13, tensor dissertations on
https://www.mis.mpg.de/scicomp/phdthesis.de.html
40 / 52
48. Practical exercises Ex1:
Implement in Matlab.
Generate two tensors u and v of order d in the CP tensor
format with tensor ranks ru = 3 and rv = 4 respectively
1. Compute u + v. Which rank has this sum?
2. Compute scalar product hu,vi
3. Apply d-dimensional Fast Fourier transform (FFT) F d to u, i.e.,
w = F du.
4. Generate a full d-dimensional tensor u ∈ Rn×...×n, d times,
n = 2M and apply FFT, i.e., F d(u). Measure the computing
time and the needed memory. Now assume
u ≈
Pr
i=1
Nd
µ=1 ũiµ. Apply F d to the CP representation, i.e.,
F d(
Pr
i=1
Nd
µ=1 uiµ). Measure and compare again the
computational time and the memory requirement. Play with
different M, d, and r.
41 / 52
49. Practical exercise Ex2:
Prove numerically that for the Laplace operator, discretised
with a finite difference on a 3d axis-parallel grid [0,1]3 with a
step size h, hold:
∆3
= ∆1
⊗ I ⊗ I + I ⊗ ∆1
⊗ I + I ⊗ I ⊗ ∆1
,
where ∆1 is the discretised Laplace operator in 1d.
Hint:
1. You may use this Matlab code
https://www.mathworks.com/matlabcentral/fileexchange/
27279-laplacian-in-1d-2d-or-3d to generate ∆3
2. Use Matlab operators kron() and eye() to compute ∆1 ⊗ I ⊗ I
42 / 52
50. Practical exercise Ex3:
Let C and D two quadratic matrices of size n × n and m × m. Let
the eigenvalues of C be µ1,...,µn. Let the eigenvalues of D be
λ1,...,λm. What are eigenvalues of matrix C ⊗ D?
43 / 52
51. Practical exercise Ex4:
Let cov(x,y) = exp−|x−y|2
, where
x = (x1,..,xd), y = (y1,...,yd) ∈ D ∈ Rd, d = 3,
cov(x,y) = exp−|x1−y1|2
⊗exp−|x2−y2|2
⊗exp−|x3−y3|2
.
C = C1 ⊗ ... ⊗ Cd.
Assume that d Cholesky decompositions exist, i.e, Ci = Li · LT
i ,
i = 1..d. Use properties of the Kronecker tensor product to
compute the following tensor product in terms of factors Li, i.e.,
compute L and LT
C1 ⊗ ... ⊗ Cd =: L · LT
.
Are L and LT also low- and upper-triangular matrices?
Generate all needed intermediate matrices and visualize L in
Matlab.
Show that the computational complexity was reduced from
O(N logN), N = nd, to O(dn logn).
44 / 52
52. Practical exercise Ex5:
Assume that inverse matrices C−1
i , i = 1..d, exist. Use
properties of the Kronecker tensor product to compute
(C1 ⊗ ... ⊗ Cd)−1
=? (21)
45 / 52
53. Practical exercise Ex6:
Let C ≈ C̃ =
Pr
i=1
Nd
µ=1 Ciµ, then
diag(C̃) = diag
r
X
i=1
d
O
µ=1
Ciµ
=
r
X
i=1
d
O
µ=1
diag
Ciµ
, (22)
trace(C̃) = trace
r
X
i=1
d
O
µ=1
Ciµ
=
r
X
i=1
d
Y
µ=1
trace(Ciµ). (23)
det(C1 ⊗ C2) = det(C1)n2 · det(C2)n1
logdet(C1 ⊗ C2) = log(det(C1)n2 · det(C2)n1)
= n2 logdetC1 + n1 logdetC2.
logdet(C1 ⊗ C2 ⊗ C3) = n2n3 logdetC1 + n1n3 logdetC2
+ n1n2 logdetC3.
46 / 52
54. Practical exercise Ex7:
Let us solve a 2D Poisson problem with uncertain coefficients,
uncertain boundary conditions and uncertain right-hand side.
Use the following matlab code:
mkdir foo
cd foo
git clone https://github.com/ezander/sglib.git
git clone https://github.com/ezander/sglib-testing.git
run matlab
open in matlab and run two scripts
sglib/startup.m
sglib-testing/startup.m
open and run script
sglib-testing/demo/old/eigel/sample_solve_spde.m
read below...
47 / 52
55. This code will solve the second order elliptic PDE with uncertain
coefficients: settings are listed below (you can play with them!)
2D L-shape domain, N = 557.
KLE terms for q(x,ω) = eκ(x,ω)
: lk = 10,
stoch. dim. mk = 10 and pk = 2,
shifted lognormal distrib. for κ(x,ω),
covκ(x,y) is of the Gaussian type, `x = `y = 0.3.
RHS: lf = 10, mf = 10, pf = 2 and Beta distrib. {4,2} for RVs.
covf(x,y) is of the Gaussian type, `x = `y = 0.6.
Total stoch. dim. mu = mk + mf = 20, |J | = 231
Solution will be the tensor:
u =
231
X
j=1
21
O
µ=1
ujµ ∈ R557
⊗
20
O
µ=1
R3
.
48 / 52
56. Interesting questions to ask
1. How is it difficult to convert one tensor representation into
another?
2. How to truncate T (T (u1 + u2) + u3) or T (u1 + u2 + u3)?
3. Which tensor representation (tensor format) is the best?
4. Does a low-rank representation not exist or I just cannot find
it?
5. Sparse grids or low-rank tensors? or Monte Carlo?
6. Does the notion of tensor eigenvalues exist?
7. What are other differences with usual matrices?
49 / 52
57. What we not discussed in these lectures
Algorithms for basic operations with low-rank matrices and
tensors: randomized compression, alternating optimization,
Riemannian optimization, nuclear norm minimization, adaptive
cross approximation and variants, functions of tensor
Further interesting and promising applications: Image
processing, matrix and tensor completion, model reduction,
solution of large- and extreme-scale linear algebra problems
from various applications (dynamics and control, quantum
computing, ...), tensors in deep learning
See all these topics in lectures of Prof. D. Kressner (EPFL)
https://www5.in.tum.de/wiki/index.php/Low_Rank_
Approximation
50 / 52
58. Literature
1. M. Espig, Dissertation, Leipzig 2008.
2. M. Espig, W. Hackbusch, A regularized Newton method for the
efficient approx. of tensor represented in the c.t. format, MPI
Leipzig 2010
3. H.G. Matthies, Uncertainty Quantification with Stochastic
Finite Elements, Encyclopedia of Computational Mechanics,
Wiley, 2007.
4. B.N. Khoromskij, A. Litvinenko, H.G. Matthies, Application of
hierarchical matrices for computing the Karhunen-Loéve
expansion, Computing 84 (1-2), 49-67, 2009
5. S. Dolgov, B.N. Khoromskij, A. Litvinenko, H.G. Matthies,
Polynomial Chaos Expansion of Random Coefficients and the
Solution of Stochastic Partial Differential Equations in the
Tensor Train Format IAM/ASA J. Uncertainty Quantification 3
(1), 1109-1135, 2015
51 / 52
59. Literature
6. S. Dolgov, A. Litvinenko, D. Liu, Kriging in tensor train data
format, Conf. Proceedings, 3rd International Conference on
Uncertainty Quantification in CSE, pp 309-329,
https://doi.org/10.7712/120219.6343.18651, 2019
7. A. Litvinenko, D. Keyes, V. Khoromskaia, B.N. Khoromskij, H.G.
Matthies, Tucker tensor analysis of Matérn functions in
spatial statistics, Computational Methods in Applied
Mathematics, 19 (1), 101-122, 2019
8. A. Litvinenko, H.G. Matthies, Inverse problems and
uncertainty quantification arXiv preprint:1312.5048, 2013
52 / 52