I do overview of methods for computing KL expansion. For tensor grid one can use high-dimensional Fourier, for non-tensor --- hierarchical matrices.

1. Application of data sparse approximation
techniques for solving SPDE
Alexander Litvinenko
Institut f¨ur Wissenschaftliches Rechnen, Technische Universit¨at Braunschweig,
0531-391-3008, litvinen@tu-bs.de
March 6, 2008

2. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

3. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

4. Stochastic PDE
We consider
− div(κ(x, ω)∇u) = f(x, ω) in D,
u = 0 on ∂D,
with stochastic coefficients κ(x, ω), x ∈ D ⊆ Rd
and ω belongs to the
space of random events Ω.
[Babuˇska, Ghanem, Schwab, Vandewalle, ...].
Methods and techniques:
1. Response surface
2. Monte-Carlo
3. Perturbation
4. Stochastic Galerkin

5. Plan of the solution
1. Discretisation of the determ. operator (FE method).
2. Discretisation of the random fields κ(x, ω), f(x, ω) (KLE).
KLE is computed by the Lanczos method + sparse data techniques.
3. Iterative solving a huge linear system
Total dimension of the SPDE is the product of dimensions of the
determ. and stochastic parts.

6. Covariance functions
The random field requires to specify its spatial correl. structure
covf (x, y) = E[(f(x, ·) − µf (x))(f(y, ·) − µf (y))],
where E is the expectation and µf (x) := E[f(x, ·)].
We classify all covariance functions into three groups:
1. isotropic (directionally independent) and stationary (translation
invariant), i.e.
cov(x, y) = cov(|x − y|),
2. anisotropic (directionally dependent) and stationary, i.e.
cov(x, y) = cov(x − y),
3. instationary, i.e. of a general type.

7. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

8. KLE
The spectral representation of the cov. function is
Cκ(x, y) = ∞
i=0 λi ki(x)ki (y), where λi and ki(x) are the eigenvalues
and eigenfunctions.
The Karhunen-Lo`eve expansion [Loeve, 1977] is the series
κ(x, ω) = µk (x) +
∞
i=1
λi ki (x)ξi (ω), where
ξi (ω) are uncorrelated random variables and ki are basis functions in
L2
(D).
Eigenpairs λi , ki are the solution of
Tki = λi ki, ki ∈ L2
(D), i ∈ N, where.
T : L2
(D) → L2
(D),
(Tu)(x) := D
covk (x, y)u(y)dy.

9. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

10. Computation of eigenpairs by FFT
If the cov. function depends on (x − y) then on a uniform tensor grid
the cov. matrix C is (block) Toeplitz.
Then C can be extended to the circulant one and the decomposition
C =
1
n
F H
ΛF (1)
may be computed like follows. Multiply (1) by F , obtain
F C = ΛF , for the first column we have
F C1 = ΛF1.
Since all entries of F1 are unity, obtain
λ = F C1.
F C1 may be computed very efficiently by FFT [Cooley, 1965] in
O(n log n) FLOPS.
C1 may be represented in a matrix or in a tensor format.

11. Properties of FFT
Lemma: Let C ∈ Rn×m
and C = k
i=0 ai bT
i , where ai ∈ Rn
, bi ∈ Rm
.
Then
F (2)
(C) =
k
i=0
F (1)
(ai )F (1)
(bT
i ). (2)
Lemma: The d-dim. FT F (d)
can be represented as following
F (d)
=
d
i=i
F (1)
= F (1)
⊗ F (1)
. . . ⊗ F (1)
(3)
and the computational complexity of F (d)
is O(dnd
log n), where nd
is
the number of dofs.

12. Discrete eigenvalue problem
After discretisation of the integral equation above, obtain
Wij :=
k,m D
bi (x)bk (x)dxCkm
D
bj (y)bm(y)dy,
Mij =
D
bi (x)bj (x)dx,
and the discrete equation will be
W fh
ℓ = λℓMfh
ℓ , where W := MCM
Approximate C in
◮ a low rank format
◮ the H-matrix format
◮ a sparse tensor format
and use the Lanczos method to compute m largest eigenvalues.

13. H-matrix [Hackbusch et al. 99]
25 11
11 20 12
13
20 11
9 16
13
13
20 11
11 20 13
13 32
13
13
20 8
10 20 13
13 32 13
13
32 13
13 32
13
13
20 11
11 20 13
13 32 13
13
20 10
10 20 12
12 32
13
13
32 13
13 32 13
13
32 13
13 32
13
13
20 11
11 20 13
13 32 13
13
32 13
13 32
13
13
20 9
9 20 13
13 32 13
13
32 13
13 32
13
13
32 13
13 32 13
13
32 13
13 32
13
13
32 13
13 32 13
13
32 13
13 32
Figure: An H-matrix approximates of cov(x, y) = e−2|x−y|
, CH ∈ Rn×n
,
n = 322
. Dense blocks are red and rank-k blocks green, max. rank-k = 13.

14. Construction process
H-matrixvertices
finite elements
cluster tree
block
cluster tree
admissibility
condition
admissible
partitioning
ACAcov. function
a good
preconditioner
fast
arithmetics
Figure: A block cluster tree. The initial matrix is decomposed into blocks and
each block is filled by a low-rank matrix or by a dense matrix.

15. H - Matrices
Comp. complexity is O(kn log n) and storage O(kn log n).
To assemble low-rank blocks use ACA [Bebendorf, Tyrtyshnikov].
Dependence of the computational time and storage requirements of
CH on the rank k, n = 322
.
k time (sec.) memory (MB) C−CH 2
C 2
2 0.04 2e + 6 3.5e − 5
6 0.1 4e + 6 1.4e − 5
9 0.14 5.4e + 6 1.4e − 5
12 0.17 6.8e + 6 3.1e − 7
17 0.23 9.3e + 6 6.3e − 8
The time for dense matrix C is 3.3 sec. and the storage 1.4e + 8 MB.

16. H - Matrices
Let h =
2
i=1 h2
i /ℓ2
i + d2 − d
2
, where hi := xi − yi , i = 1, 2, 3,
ℓi are cov. lengths and d = 1.
exponential cov(h) = σ2
· exp(−h),
The cov. matrix C ∈ Rn×n
, n = 652
.
ℓ1 ℓ2
C−CH 2
C 2
0.01 0.02 3e − 2
0.1 0.2 8e − 3
1 2 2.8e − 6
10 20 3.7e − 9

17. H - matrices + ARPACK → Eigenvalues
Table: Time required for computing m eigenpairs. exponential cov. matrix
C ∈ Rn×
, n = 2572
, ℓ1 = ℓ2 = 0.1. H-matrix comp. time is 26 sec., storage
for C is 1.2 GB.
m 10 20 40 80 160
time (sec.), ℓ1 = ℓ2 = 1 7 16 34 104 449
time (sec.), ℓ1 = ℓ2 = 0.1 35 51 97 194 532
Eigenvalues and the computational error for different covariance
lengths, max. rank k = 20.
ℓ1 = ℓ2 = 1 ℓ1 = ℓ2 = 0.1
i λi Cxi − λi xi 2 λi Cxi − λi xi 2
1 303748 1.6e-4 26006 0.03
2 56358 1.7e-3 21120 0.01
20 1463 2.2e-2 4895 0.2
80 139 4.2e-2 956 0.26
150 68 7e-2 370 0.5

18. Exponential Eigenvalue decay
0 100 200 300 400 500 600 700 800 900 1000
0
100
200
300
400
500
600
700
0 100 200 300 400 500 600 700 800 900 1000
0
1
2
3
4
5
6
7
8
9
10
x 10
4
0 100 200 300 400 500 600 700 800 900 1000
0
200
400
600
800
1000
1200
1400
1600
1800
0 100 200 300 400 500 600 700 800 900 1000
0
0.5
1
1.5
2
2.5
x 10
5
0 100 200 300 400 500 600 700 800 900 1000
0
50
100
150
0 100 200 300 400 500 600 700 800 900 1000
0
0.5
1
1.5
2
2.5
3
3.5
4
x 10
4
Figure: 23 grid 48 × 64 × 40, (left) ℓ1 = 1, ℓ2 = 2, ℓ3 = 1 and (right) ℓ1 = 5,
ℓ2 = 10, ℓ2 = 5. 1st row - Gaussian, 2-nd exponential and 3-rd spherical cov.

19. Sparse tensor decompositions of kernels
cov(x, y) = cov(x − y)
We want to approximate C ∈ RN×N
, N = nd
by
Cr = r
k=1 V 1
k ⊗ ... ⊗ V d
k such that C − Cr ≤ ε.
The storage of C is O(N2
) = O(n2d
) and the storage of Cr is O(rdn2
).
To define V i
k use e.g. SVD.
Approximate all V i
k in the H-matrix format ⇒ HKT format
[Hackbusch, Khoromskij, Tyrtyshnikov].
Assume f(x, y), x = (x1, x2), y = (y1, y2), then the equivalent approx.
problem is f(x1, x2; y1, y2) ≈
r
k=1 Φk (x1, y1)Ψk (x2, y2).

20. Numerical examples of tensor approximations
Gaussian kernel exp{−|x − y|2
} has the Kroneker rank 1.
The exponen. kernel exp{−|x − y|} can be approximated by a tensor
with low Kronecker rank
r 1 2 3 4 5 6 10
C−Cr ∞
C ∞
11.5 1.7 0.4 0.14 0.035 0.007 2.8e − 8
C−Cr 2
C 2
6.7 0.52 0.1 0.03 0.008 0.001 5.3e − 9

21. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

22. Application: covariance of the solution
Let K be the stiffness matrix. For SPDE with stochastic RHS the
eigenvalue problem and spectral decom. look like
Cf fℓ = λℓfℓ, Cf = Φf Λf ΦT
f .
If we only want the covariance
Cu = (K ⊗ K)−1
Cf = (K−1
⊗ K−1
)Cf = K−1
Cf K−T
,
one may with the KLE of Cf = Φf Λf ΦT
f reduce this to
Cu = K−1
Cf K−T
= K−1
Φf ΛΦT
f K−T
.

23. Application: higher order moments
Let operator K be deterministic and
Ku(θ) =
α∈J
Ku(α)
Hα(θ) = ˜f(θ) =
α∈J
f(α)
Hα(θ), with
u(α)
= [u
(α)
1 , ..., u
(α)
N ]T
. Projecting onto each Hα obtain
Ku(α)
= f(α)
.
The KLE of f(θ) is
f(θ) = f +
ℓ
λℓφℓ(θ)fℓ =
ℓ α
λℓφ
(α)
ℓ Hα(θ)fℓ
=
α
Hα(θ)f(α)
,
where f(α)
= ℓ
√
λℓφ
(α)
ℓ fℓ.

24. Application: higher order moments
The 3-rd moment of u is
M
(3)
u = E


α,β,γ
u(α)
⊗ u(β)
⊗ u(γ)
HαHβHγ

 =
α,β,γ
u(α)
⊗u(β)
⊗u(γ)
cα,β,γ,
cα,β,γ := E (Hα(θ)Hβ(θ)Hγ(θ)) = cα,β · γ!, and cα,β are constants
from the Hermitian algebra.
Using u(α)
= K−1
f(α)
= ℓ
√
λℓφ
(α)
ℓ K−1
fℓ and uℓ := K−1
fℓ, obtain
M
(3)
u =
p,q,r
tp,q,r up ⊗ uq ⊗ ur , where
tp,q,r := λpλqλr
α,β,γ
φ
(α)
p φ
(β)
q φ
(γ)
r cα,βγ.

25. Outline
Problem setup
Karhunen-Lo`eve expansion
Data Sparse Techniques
Fast Fourier Transformation (FFT)
Hierarchical Matrices
Sparse tensor approximation
Applications
Conclusion

26. Conclusion
◮ Covariance matrices allow data sparse approximations.
◮ Application of H-matrices
◮ extend the class of covariance functions to work with,
◮ allows non-regular discretisations of the cov. function on large
spatial grids.
◮ Application of sparse tensor product allows computation of k-th
moments.

27. Plans for Feature
1. Apply H-matrix - ARPACK technique for solving SPDEs [M.
Krosche’s software]
2. Further research how to apply sparse KLE for computing
moments and functionals of the solution [DFG]
3. Implement sparse tensor vector product for the Lanczos method
[MPI Leipzig]

28. Thank you for your attention!
Questions?