ADI for Tensor Structured Equations

83rd GAMM Annual Scientiﬁc Conference
Darmstadt, 28 March 2012

ADI for Tensor Structured Equations
Thomas Mach and Jens Saak

Max Planck Institute for Dynamics of Complex Technical Systems
Computational Methods in Systems and Control Theory

MAX PLANCK INSTITUTE
FOR DYNAMICS OF COMPLEX
TECHNICAL SYSTEMS
MAGDEBURG

Max Planck Institute Magdeburg Thomas Mach, Jens Saak, Tensor-ADI 1/24

ADI ADI for Tensors Numerical Results and Shifts Conclusions

Classic ADI [Peaceman/Rachford ’55]

Developed to solve linear systems related to Poisson problems

−∆u = f in Ω ⊂ Rd , d = 1, 2
u=0 on ∂Ω.

uniform grid size h, centered diﬀerences, d = 1,

⇒ ∆1,h u = h2 f
 
2 −1
−1 2 −1 
 
∆1,h =
 .. .. .. .

 . . . 
 −1 2 −1
−1 2




Developed to solve linear systems related to Poisson problems

−∆u = f in Ω ⊂ Rd , d = 1, 2
u=0 on ∂Ω.

uniform grid size h, 5-point diﬀerence star, d = 2,

⇒ ∆2,h u = h2 f
   
K −I 4 −1
−I K −I  −1 4 −1 
   
∆2,h =
 .. .. ..  and K = 
  .. .. .. .

 . . .   . . . 
 −I K −I   −1 4 −1
−I K −1 4




Observation
∆2,h = (∆1,h ⊗ I ) + (I ⊗ ∆1,h ).
=:H =:V




Observation
∆2,h = (∆1,h ⊗ I ) + (I ⊗ ∆1,h ).
=:H =:V

˜
Solve ∆2,h u = h2 f =: f exploiting structure in H and V .




Observation
∆2,h = (∆1,h ⊗ I ) + (I ⊗ ∆1,h ).
=:H =:V

˜
Solve ∆2,h u = h2 f =: f exploiting structure in H and V .

For certain shift parameters perform
˜
(H + pi I ) ui+ 1 = (pi I − V ) ui + f ,
2
˜
(V + pi I ) ui+1 = (pi I − H) ui+ 1 + f ,
2

until ui is good enough.



ADI and Lyapunov Equations [Wachspress ’88]

Lyapunov Equation
FX + XF T = −GG T




Lyapunov Equation
FX + XF T = −GG T

Vectorized Lyapunov Equation
(I ⊗ F ) + (F ⊗ I ) vec(X ) = −vec(GG T )
=:HF =:VF

Same structure ⇒ apply ADI




Lyapunov Equation
FX + XF T = −GG T

Vectorized Lyapunov Equation
(I ⊗ F ) + (F ⊗ I ) vec(X ) = −vec(GG T )
=:HF =:VF

Same structure ⇒ apply ADI

(F + pi I ) Xi+ 1 = −GG T − Xi F T − pi I
2

(F + pi I ) Xi+1 = −GG T − Xi+ 1 F T − pi I
T
2



Generalizing Matrix Equations

∆2,h vec(X ) = vec(B)
I ⊗ ∆1,h + ∆1,h ⊗ I vec(X ) = vec(B)

=H =V =u =f

∆µa a
Xa c

+ = Ba c
c ∆µc
Xa c



Generalizing Matrix Equations

∆4,h vec(X ) = vec(B)
I ⊗ I ⊗ I ⊗ ∆1,h + I ⊗ I ⊗ ∆1,h ⊗ I + I ⊗ ∆1,h ⊗ I ⊗ I + ∆1,h ⊗ I ⊗ I ⊗ I vec(X ) = vec(B)

=H =V =R =Q =u =f

∆µa a
Xabcd + Xabcd
∆µb b
+ = Babcd
c ∆µc
Xabcd + Xabcd
d ∆µd



Generalizing ADI

I ⊗ ∆1,h + ∆1,h ⊗ I vec(X ) = vec(B)

=H =V =u =f

(H + I ⊗ pi,1 I )Xi+ 1 = (pi,1 I − V )Xi + B
2
(V + pi,2 I ⊗ I )Xi+ 1 = (pi,2 I − H)Xi+ 1 + B
2 2



Generalizing ADI

I ⊗ ∆1,h + ∆1,h ⊗ I vec(X ) = vec(B)

=H =V =u =f

(H + I ⊗ pi,1 I )Xi+ 1 = (pi,1 I − V )Xi + B
2
(V + pi,2 I ⊗ I )Xi+ 1 = (pi,2 I − H)Xi+ 1 + B
2 2

I ⊗ I ⊗ I ⊗ ∆1,h + I ⊗ I ⊗ ∆1,h ⊗ I + I ⊗ ∆1,h ⊗ I ⊗ I + ∆1,h ⊗ I ⊗ I ⊗ I vec(X ) = vec(B)

=H =V =R =Q =u =f

(H + I ⊗ I ⊗ I ⊗ pi,1 I )Xi+ 1 = (pi,1 I − V − R − Q)Xi +B
4
(V + I ⊗ I ⊗ pi,2 I ⊗ I )Xi+ 1 = (pi,2 I − H − R − Q)Xi+ 1 +B
2 4
(R + I ⊗ pi,3 I ⊗ I ⊗ I )Xi+ 3 = (pi,3 I − H − V − Q)Xi+ 1 +B
4 2
(Q + pi,4 I ⊗ I ⊗ I ⊗ I )Xi+1 = (pi,4 I − H − V − R)Xi+ 3 +B
4



Goal

Solve AX = B

A = I ⊗ I ⊗ · · · ⊗ I ⊗ I ⊗ A1 +
I ⊗ I ⊗ · · · ⊗ I ⊗ A2 ⊗ I +
... +
Ad ⊗ I ⊗ · · · ⊗ I ⊗ I ⊗ I

B is given in tensor train decomposition
⇒ X is sought in tensor train decomposition.



Tensor Trains [Oseledets, Tyrtyshnikov ’09]

r1 ,...,rd−1
T (i1 , i2 , . . . , id ) = G1 (i1 , α1 )G2 (α1 , i2 , α2 )
α1 ,...,αd−1 =1

· · · Gj (αj−1 , ij , αj ) · · ·
Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id ).

G1 (i1 , α1 ) α1 G2 (α1 , i2 , α2 ) α2 ··· Gd (αd−1 , id )




Tensor trains are
computable, and
d
require only O(dnr 2 ) storage, with TT-rank r and T ∈ Rn .

Canonical representation

T (i1 , i2 , . . . , id ) = G1 (i1 , α) · · · Gd (id , α)
α

Tucker decomposition

T (i1 , i2 , . . . , id ) = C (α1 , . . . , αd )G1 (i1 , α1 ) · · · Gd (id , αd )
α1 ,...,αd




(I ⊗ · · · ⊗ I ⊗ A1 ) T




(I ⊗ · · · ⊗ I ⊗ A1 ) T

A1 (β, i1 )

i1





(I ⊗ · · · ⊗ I ⊗ A1 ) T

A1 (β, i1 )

i1


T (i1 , i2 , . . . , id ) ×1 A1 = A1 β,i1 G1 (i1 , α1 )G2 (α1 , i2 , α2 )
α1 ,...,αd−1

· · · Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id )



(I ⊗ · · · ⊗ I ⊗ A1 ) T

˜
= G1 (β, α1 ) = A1 G1
A1 (β, i1 )

i1


T (i1 , i2 , . . . , id ) ×1 A1 = A1 β,i1 G1 (i1 , α1 )G2 (α1 , i2 , α2 )
α1 ,...,αd−1

· · · Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id )



(I ⊗ · · · ⊗ I ⊗ A1 ) −1 T

˜
= G1 (β, α1 ) = A1 G1
A1 (β, i1 )

i1


T (i1 , i2 , . . . , id ) ×1 A1 −1 = A1 −1 β,i1 G1 (i1 , α1 )G2 (α1 , i2 , α2 )
α1 ,...,αd−1

· · · Gd−1 (αd−2 , id−1 , αd−1 )Gd (αd−1 , id )


Algorithm

Input: {A1 , . . . , Ad }, tensor train B, accuracy
Output: tensor train X , with AX = B
forall j ∈ {1, . . . , d} do
(0)
Xj := zeros(n, 1, 1)
end
while r (i) > do
Choose shift pi
forall k ∈ {1, . . . , d} do
d
×j Aj ×k (Ak + pi I )−1
k k−1 k−1
X (i+ d ) := B +pi X (i+ d ) − X (i+ d )

j=1
j=k
end
end



Algorithm
r (i) := B
forall j ∈ {1, . . . , d} do
Output: tensor train X , with AX(i) B =
r (i) := r − Xi ×j Aj
forall j ∈ {1, . . . , d} do
(0) end
end
while r (i) > do
Choose shift pi
forall k ∈ {1, . . . , d} do
d
×j Aj ×k (Ak + pi I )−1
k k−1 k−1
X (i+ d ) := B +pi X (i+ d ) − X (i+ d )

j=1
j=k
end
end



Algorithm

Output: tensor train X , with AX = B
forall j ∈ {1, . . . , d} do
(0)
end (I ⊗ I ⊗ · · · ⊗ I ⊗ Aj ⊗ I ⊗ · · · ⊗ I ) Xi+ k−1
d
while r (i) > do
Choose shift pi
forall k ∈ {1, . . . , d} do
d
×j Aj ×k (Ak + pi I )−1
k k−1 k−1
X (i+ d ) := B +pi X (i+ d ) − X (i+ d )

j=1
j=k
end
end



Eigenvalues

A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I
St´phanos’ theorem:
e

⇒ λi (A) = λi1 (A1 ) + λi2 (A2 ) + · · · + λid (Ad ),
d−1
with i = i1 + i2 n1 + · · · + id nj .
j=1



Eigenvalues

A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I
St´phanos’ theorem:
e

⇒ λi (A) = λi1 (A1 ) + λi2 (A2 ) + · · · + λid (Ad ),
d−1
with i = i1 + i2 n1 + · · · + id nj .
j=1

d
AX = B ⇔ X ×j Aj = B
j=1

A is regular ⇔ λi (A) = 0 ∀i ⇐ Ai Hurwitz ∀i



Lemma

Lemma [Grasedyck ’04]
The tensor equation
d
j=1 X ×j Aj = B

with Ak Hurwitz ∀k has the solution
∞
X =− 0 B ×1 exp(A1 t) ×2 · · · ×d exp(Ad t)dt

Z (t) = B ×1 exp(A1 t) ×2 · · · ×d exp(Ad t)
d ∞
˙
Z (t) = Z (t) ×j Aj Z (∞) − Z (0) = ˙
Z (t)dt,
j=1 0

d ∞
0−B = Z (t)dt ×j Aj
j=1 0



Theorem
Theorem
{A1 , . . . , Ad } ⇒ A, Λ(A) ⊂ [−λmax , −λmin ] ⊕ ı [−µ, µ] ⊂ C− .
Let k ∈ N and use the quadrature points and weights:
√
hst := √k , tj := log e jhst + 1 + e 2jhst , wj := √ hst−2jh .
π
1+e st

Then the solution X can be approximated by
r1 ,...,rd−1
˜
X (i1 , i2 , . . . , id ) = − H1 (i1 , α1 ) · · · Hd (αd−1 , id ),
α1 ,...,αd−1 =1

2tj
2wj Ap
with Hp (αp−1 , ip , αp ) := k
j=−k λmin βp e
λmin
ip ,βp
Gp (αp−1 , βp , αp )
with the approximation error
2µλ−1 +1 √
(λI − 2A/λmin )−1
min
˜
X −X ≤ Cst −π k
πλmin e dΓ λ B 2.
2 π
Γ 2
extending [Grasedyck ’04] (X and B of low Kronecker rank) to low TT-rank


Approximation Accuracy
Storage in 104 ·Double

constant truncation error 10−2

i
8 tightened truncation error

Truncation Error
6 10−8
4
10−14
2

10−20
0 5 10 15 20 25 30
Iteration



Example: Laplace – Ai = ∆1, 11
1

Ai = ∆1, 1
11

B = 0 0 ... 0 1

Shifts:
pi := e1 (∗1 ) + . . . + ed (∗d ) — random chosen eigenvalue



Numerical Results – Ai = ∆1, 11
1

d t in s residual mean(#it)
2 3.887 e−01 7.015 e−10 112.8
5 5.398 e+00 7.467 e−10 45.8
8 6.007 e+00 6.936 e−10 12.8
10 3.662 e+00 7.685 e−10 6.8
25 3.142 e+01 2.437 e−10 5.0
50 2.268 e+02 2.049 e−10 5.0
75 7.192 e+02 4.036 e−10 5.0
100 1.700 e+03 1.864 e−10 5.0
150 5.538 e+03 1.801 e−10 5.0
200 1.280 e+04 1.472 e−10 5.0
250 2.499 e+04 1.816 e−10 5.0
300 4.298 e+04 2.535 e−10 5.0
500 1.952 e+05 2.039 e−10 5.0


1

sparse dense
d TADI MESS Penzl’s sh. lyap
2 0.310 0.0006 0.024 0.003 0.0003 0.0005
4 3.130 0.1695 0.011 0.049 6.331 0.012
6 8.147 — 0.076 0.094 — 7.17
8 5.458 — 5.863 1.097 — 13 698.2
10 5.306 — 3 445.523 249.464 — —



1

105

104
Computation Time in s

103

102
Tensor ADI
101 sparse
MESS
100
Penzl’s shifts
10−1 dense
lyap
10−2
10 100 300
Dimension d


Single Shift and Convergence

A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I
We assume Λ(Ak ) ⊂ R− .

Error Propagation, Single Shift
p− λk + λl λk
 
d d
k k
G1 2 ≤ max = 1 −  .
λk ∈Λ(Ak ), p + λl p + λl
k=1,...,d l=0 l=0

If G1 2 < 1, then the ADI iteration converges.
p < 0 and p > −∞




A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I

p− λk + λl λk
 
d d
k k
G1 2 ≤ max = 1 −  .
k=1,...,d l=0 l=0

p < 0 and p > −∞
d
p < λi (A) = k=1 λk (Ak ) ∀i




A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I

p− λk + λl λk
 
d d
k k
G1 2 ≤ max = 1 −  .
k=1,...,d l=0 l=0

p < 0 and p > −∞
d
p < λi (A) = k=1 λk (Ak ) ∀i
d−2
Lyapunov case (Ak = A0 ∀k): p < 2 λmin (A0 )




A = I ⊗ · · · ⊗ I ⊗ A1 + I ⊗ · · · ⊗ I ⊗ A2 ⊗ I + . . . + Ad ⊗ I ⊗ · · · ⊗ I

p− λk + λl λk
 
d d
k k
G1 2 ≤ max = 1 −  .
k=1,...,d l=0 l=0

p < 0 and p > −∞
d
p < λi (A) = k=1 λk (Ak ) ∀i
2−2
Lyapunov case (Ak = A0 ∀k): p < 2 λmin (A0 ) =0



Shifts

Min-Max-Problem
d
pi,k − j=k λj
min max
{p1,1 ,...,p ,d }⊂C λk ∈Λ(Ak ) ∀k pi,k + λk
i=0 k=0



Shifts

Min-Max-Problem
d
pi,k − j=k λj
min max
i=0 k=0

Min-Max-Problem, Lyapunov case (Ak = A0 ∀k, A0 Hurwitz)
d
pi,k − j=k λj
min max
{p1,1 ,...,p ,d }⊂C λk ∈Λ(A0 ) ∀k pi,k + λk
i=0 k=0



Shifts

Min-Max-Problem
d
pi,k − j=k λj
min max
i=0 k=0

d
pi − j=k λj
min max
{p1 ,...,p }⊂C λk ∈Λ(A0 ) ∀k pi + λk
i=0 k=0



Shifts

Min-Max-Problem
d
pi,k − j=k λj
min max
i=0 k=0

d
pi − j=k λj
min max
{p1 ,...,p }⊂C λk ∈Λ(A0 ) ∀k pi + λk
i=0 k=0

λk = λ0 ∀k
Penzl’s idea: {p1 , . . . , p } ⊂ (d − 1)Λ(A0 )



Random Example

seed := 1;
R := rand(10);
R := R + R ;
R := R − λmin + 0.1;
A0 = −R;
Λ(A0 ) = {−0.1000, −0.2250, −1.1024, −1.7496, −2.0355,
−2.4402, −3.1330, −3.3961, −3.9347, −11.9713}

⇒ The random shifts do not lead to convergence.

p0 = λ10 (A0 )(d − 1)
p1 = λ9 (A0 )(d − 1)
p2 = λ8 (A0 )(d − 1)


Numerical Results – Ai = −R

d t in s residual #it
2 2.7673 9.1353 e−09 219.0
5 7.8942 9.6503 e−09 98.0
8 18.9964 9.8650 e−09 84.0
10 18.4739 7.5746 e−09 58.0
15 27.5661 5.0619 e−09 40.0
20 32.2409 4.9971 e−09 32.0
25 40.2462 5.1732 e−09 29.0
50 76.3225 7.4093 e−09 14.0
75 159.6627 3.2629 e−09 10.0
100 436.6120 9.1137 e−09 11.0



Numerical Results – Ai = −R

d t in s tdmrg in s #it
2 2.7673 0.0148 219.0
5 7.8942 2.5576 98.0
8 18.9964 5.4536 84.0
10 18.4739 5.5852 58.0
15 27.5661 6.3068 40.0
20 32.2409 7.4044 32.0
25 40.2462 8.3371 29.0
50 76.3225 11.8840 14.0
75 159.6627 18.0581 10.0
100 436.6120 28.8515 11.0



Conclusions and Outlook

We have seen
a generalization of the ADI method,
capable of solving tensor Lyapunov and Sylvester equations,
producing solutions of low TT-rank.




We have seen

Open questions:
more sophisticated shift strategies and
why is the dmrg solver so much faster?




We have seen

Open questions:
more sophisticated shift strategies and
why is the dmrg solver so much faster?

Thank you for your attention.


ADI for Tensor Structured Equations

Recommended

Recommended

More Related Content

What's hot

What's hot (8)

Viewers also liked

Viewers also liked (8)

Similar to ADI for Tensor Structured Equations

Similar to ADI for Tensor Structured Equations (11)

More from Thomas Mach

More from Thomas Mach (8)

Recently uploaded

Recently uploaded (20)

ADI for Tensor Structured Equations