Large data sets result large dense matrices, say with 2.000.000 rows and columns. How to work with such large matrices? How to approximate them? How to compute log-likelihood? determination? inverse? All answers are in this work.

1. Center for Uncertainty
Quantification
C
Q
Likelihood Approximation With Parallel Hierarchical Matrices
For Large Spatial Datasets
A. Litvinenko (RWTH), R. Kriemann (MIS MPG), Y. Sun, M. Genton, D. Keyes (KAUST)
litvinenko@uq.rwth-aachen.de
HIERARCHICAL LIKELIHOOD APPROXIMATION
Goal: To improve estimation of unknown statistical parameters in a spatial soil moisture field.
How ?: By reducing linear algebra cost from O(n3
) to O(n log n).
Let Z be a mean-zero, stationary and isotropic
Gaussian process with a Matérn covariance at n
irregularly spaced locations.
Let Z = (Z(s1), ..., Z(sn))T
∼ N(0, C(θ)),
θ ∈ Rq
is an unknown parameter vector of inter-
est, where
cov(Z(si), Z(sj)) = C(h, θ) =
=
2σ2
Γ(ν)
h
2
ν
Kν
h
, θ = (σ2
, ν, )T
is the Matérn covariance function
with h := si − sj .
The MLE of θ is obtained by maximizing the
Gaussian log-likelihood function:
L(θ) = −
n
2
log(2π)−
1
2
log |C(θ)|−
1
2
Z C(θ)−1
Z.
We approximate C ≈ C in the H-matrix format
with cost and storage O(kn log n), k n. Obtain
a cheap approximation L(θ) ≈ ˜L(θ; k).
Types of matrices
Moisture field modeled by Matérn random field
-84 -82 -80 -78 -76 -74
latitude
34
36
38
40
42
longitude
-3
-2
-1
0
1
2
3
rank k
3 7 9
ℓ
0.02
0.025
0.03
0.035
0.04
0.045
0.05
0.055
0.06
128 64 32 16 8 4 2
n, samples in thousands
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
0.75
ν
Boxplots for simulated data. (left) vs. different
H-matrix ranks k = {3, 7, 9}; ν = 0.5, = 0.0334,
σ2
= 1. (right) ν for n = {128, 64, 32, 16, 8, 4, 2} × 1000,
( ∗
, ν∗
, σ∗
) = (0.0864, 0.5, 1.0).
PARALLEL HIERARCHICAL MATRICES (HACKBUSCH, KRIEMANN’05)
Advantages to approximate C by C: H-approximation is cheap; storage and matrix-vector product
cost O(kn log n); LU and inverse cost O(k2
n log2
n); efficient parallel implementations exists.54
54
86
86
68
67 67
20
19 112
28 26
15 24
112
112 123
26 107
19 17
107
107
82
67 67
25
25
25
32 66
20 20
28 24
25 26
16 25
66
66 73
44 63
30 23
63
63 67
29
15 67
14 27
22 18
69
60 60
25 60
25 25
79
63 63
24
14
27 63
19 29
16 21
23 30
64
64 79
19 66
23 29
66
66 66
24
11 65
21 52
28 35
65
65 70
112 112
19
24
35
21 112
19 8
23 35
18 15
12 20
79
71 71
115 115
33 115
23 29
127
125
66
66 67
24
25 125
27 25
25 30
125
125
81
64 64
17 125
26 19
125
125
65
65 73
17
35
22
14 65
20 30
25 26
25 21
15 28
65
65 82
21 70
22 49
70
60 60
30
29 60
25 16
17 13
71
64 64
15 61
14 20
61
61 70
19
20
18 59
23 23
17 25
21 25
59
59 85
119 119
16 119
23 21
70
62 62
120 120
12
17
16
13
20 120
14 17
25 23
17 24
13 20
8 12
80
67 67
19 65
11 15
65
64 64
17 122
18 15
122
122 122
34
23 112
11 26
24 23
113
113
74
74 84
21 104
17 23
104
104 120
28
24
18 120
29 22
26 31
18 27
78
57 57
125 125
28 125
19 30
62
62 77
122 122
23
32 121
15 15
26 33
122
122 123
25 123
24 19
70
63 63
29 63
24 18
75
65 65
18
28
35
33
28 65
18 24
24 35
16 14
19 21
12 20
78
66 66
124 124
18 123
21 16
123
123
66
66 70
18
27 127
29 19
21 23
127
127
77
63 63
21 123
11 32
123
123
61
61 82
28
26
16 123
26 30
19 31
18 26
123
113 113
22 113
16 25
73
59 59
29 59
26 29
76
76 79
26
16
19 62
20 19
28 23
24 26
62
62 73
18 69
41 20
69
65 66
28 123
25 12
123
26 26
54
54
86
86
68
67 67
20
39 112
48 73
15 39
112
112 123
63 107
56 65
107
107
82
67 67
25
50
67
58 66
59 58
71 76
50 75
16 41
66
66 73
63 63
62 66
63
63 67
84
58 67
54 55
75 76
69
60 60
59 60
56 55
79
63 63
65
70
58 63
59 57
67 69
63 80
64
64 79
59 66
54 56
66
66 66
82
57 65
60 66
79 83
65
65 70
112 112
19
43
76
82 112
73 72
62 83
37 51
12 32
79
71 71
115 115
87 115
78 77
127
127
66
66 67
63
73 125
70 77
59 75
125
125
81
64 64
76 125
78 84
125
125
65
65 73
48
71
79
56 65
61 67
75 82
59 71
43 61
65
65 82
59 70
54 60
70
60 60
80
52 60
48 54
67 52
71
64 64
54 61
61 55
61
61 70
69
74
48 59
52 63
67 67
64 78
59
59 85
119 119
64 119
65 74
70
62 62
120 120
12
29
45
47
59 120
51 58
53 69
44 52
25 40
8 20
80
67 67
46 65
50 56
65
64 64
67 122
62 61
122
122 122
69
71 113
63 71
58 71
113
113
74
74 84
64 104
59 72
104
104 120
55
69
63 120
66 80
66 74
43 60
78
57 57
125 125
81 125
69 78
62
62 77
122 122
68
77 122
62 65
64 81
122
122 123
76 123
71 80
70
63 63
63 63
56 55
75
65 65
34
56
73
85
70 65
63 60
76 93
55 61
45 54
27 40
78
66 66
124 124
68 123
63 71
123
123
66
66 70
61
69 127
68 74
60 74
127
127
77
63 63
73 123
63 74
123
123
61
61 82
57
72
66 123
67 88
63 74
45 61
123
113 113
72 113
65 75
73
59 59
62 59
59 57
76
76 79
69
68
52 62
54 64
72 83
62 74
62
62 73
58 69
62 58
69
66 66
83 123
26 26
123
26 26
0.05 0.1 0.15 0.2
ℓ
0.8
1
1.2
1.4
1.6
1.8
−˜L/n
2000
4000
8000
16000
32000
64000
128000
0.3 0.4 0.5 0.6 0.7
ν
0.8
1
1.2
1.4
1.6
1.8
−˜L/n
2000
4000
8000
16000
32000
64000
128000
0 0.5 1 1.5 2 2.5
ℓ, ν = 0.325, σ2
= 0.98
10 -10
10 -8
10 -6
10 -4
10 -2
10 0
error
1e-4
1e-6
(1st) Matérn H-matrix approximations for moisture example, n = 8000, ε = 10−3
, = 0.64, ν = 0.325,
σ2
= 0.98, 29.3MB vs 488.3MB for dense, set up time 0.4 sec.; (2nd) H-Cholesky factor ˜L, with accuracy
in each block ε = 10−8
, 4.8 sec., storage 52.8 MB.; (3rd) Dependence of the negative log-likelihood − ˜L/n,
on n = {2,000, ..., 128,000} and on the parameters and ν (4th), in log-log scale for the soil moisture data
example; (5th) error I − (˜L˜L )−1 ˜C 2 vs. (ν = 0.325, σ2 = 0.98 fixed), ε = {10−4
, 10−6
}.
NUMERICAL EXAMPLES
H-matrix approximation, ν = 0.5, domain G = [0, 1]2
, C(0.25,0.75) 2 = {212, 568}, n = 16049.
k KLD C − C 2 CC−1
− I 2
= 0.25 = 0.75 = 0.25 = 0.75 = 0.25 = 0.75
10 2.6e-3 0.2 7.7e-4 7.0e-4 6.0e-2 3.1
50 3.4e-13 5e-12 2.0e-13 2.4e-13 4e-11 2.7e-9
Computing time and number of iterations for maximization of log-likelihood L(θ; k), n = 66049.
k size, GB C, set up time, s. compute L, s. maximizing, s. # iters
10 1 7 115 1994 13
20 1.7 11 370 5445 9
dense 38 42 657 ∞ -
Moisture data. We used adaptive rank arithmetics with ε = 10−4
for each block of C and ε = 10−8
for
each block of C−1
.
Computing time and storage cost for parallel H-matrix approximation; number of cores is 40, ν = 0.325, = 0.64,
σ2 = 0.98. H-matrix accuracy in each sub-block for both ˜C and ˜L is 10−5
.
n ˜C ˜L˜L
time size kB/dof time size I − (˜L˜L )−1 ˜C 2
sec. MB sec. MB
512,000 52.0 3410 7.0 77.4 4150 3.4 · 10−2
1,000,000 103 7070 7.4 187 8830 6.6 · 10−2
2,000,000 227 14720 7.7 471 18970 1.3 · 10−1
ℓ
0 0.2 0.4 0.6 0.8 1−L
×10 4
-1
0
1
2
3
4
5
6
nugget 0.01
nugget 0.005
nugget 0.001
ℓ
0 0.1 0.2 0.3 0.4 0.5
−L
-1000
-800
-600
-400
-200
0
200
400
600
800
1000
nugget 0.01
nugget 0.005
nugget 0.001
(1st) −L vs. with nuggets {0.01, 0.005, 0.001} for Gaussian covariance, n = 2000, k = 14, σ2
= 1;
(2nd) zoomed; (3rd) Identification ˆν vs. H-accuracy, ( ∗
, ν∗
, σ∗2
) = (0.0864, 0.5, 1.0); (4th) functional
boxplots σ2
2ν vs. H-accuracy; 30 replicates.
REFERENCES AND ACKNOWLEDGEMENTS
[1] A. Litvinenko, Y. Sun, M.G. Genton, D. Keyes, HLIBCov: Parallel Hierarchical Matrix Approximation of Large Covariance Matrices and Likelihoods with Applications in Parameter Identification, J. MethodsX
Elsevier, (arXiv:1709.08625), 2019
[2] A. Litvinenko, Y. Sun, M.G. Genton, D. Keyes, Likelihood Approximation With Hierarchical Matrices For Large Spatial Datasets, Computational Statistics & Data Analysis 137, 115-132, 2019
[3] M. Espig, W. Hackbusch, A. Litvinenko, H.G. Matthies, E. Zander, Post-Processing of High-Dimensional Data arXiv:1906.05669, submitted to J. SIMODS SIAM, 2019
[4] S. Dolgov, A. Litvinenko, D. Liu, Kriging in Tensor Train data format, arXiv:1904.09668, 2019
[5] A. Litvinenko, D. Keyes, V. Khoromskaia, B.N. Khoromskij, H.G. Matthies, Tucker tensor analysis of Matérn functions in spatial statistics, Computational Methods in Applied Mathematics 19 (1), 101-122,
2019
[6] B. N. KHOROMSKIJ, A. LITVINENKO, H. G. MATTHIES, Application of hierarchical matrices for computing the Karhunen-Loéve expansion, Computing, Vol. 84, Issue 1-2, pp 49-67, 2008.
[7] W. NOWAK, A. LITVINENKO, Kriging and spatial design accelerated by orders of magnitude: combining low-rank covariance approximations with FFT-techniques, J. Mathematical Geosciences, Vol. 45, N4, pp
411-435, 2013.
Work supported by SRI-UQ and ECRC, KAUST. Thanks to Ronald Kriemann for HLIBPro .