Speed-Up Solving Systems via composition of their clans

1. Dmitry A. Zaitsev
http://member.acm.org/~daze
Composition of Clans for
Solving Linear Systems on
Parallel Architectures
Odessa State Environmental University
http://www.odeku.edu.ua

2. Speed-Up Solving Systems
on 16 Nodes
2

3. Key Features
• Speed-up solving (especially Diophantine)
systems of linear algebraic equations
• Sparse systems of specific form, namely
“well decomposable into clans”
• Concept of a sign forms clans of equations
• Applicable to other algebraic structures
with sign
3

4. Form of Obtained Matrix
4

5. Divide and Sway
• Decompose a given system into its clans
• Solve a system for each clan
• Solve a system of clans composition
• Or collapse the decomposition graph
solving a system for each contracted edge
• Obtain a result in feasible time
5

6. Algebraic structure
Numbers Structure Methods Complexity
Complex field a) reduction: LU, QR;
б) iteration methods
O(n3)
Real
Integer ring Normal forms:
Hermite, Smith
O(n4)
Nonnegative
integer
monoid Methods of Toudic
(Silva) and Contejean
O(2n)

7. Real-life matrices (systems)
• Matrix Market -
https://math.nist.gov/MatrixMarket
• The SuiteSparse Matrix Collection
https://sparse.tamu.edu
• Model Checking Contest – Petri net
models https://mcc.lip6.fr

8. Basic software
Structure Type Dense Sparse
Field LAPACK UMFPACK
Ring 4ti2 ParAd+4ti2
Monoid 4ti2 ParAd

9. A Clan – Transitive Closure
of Nearness Relation
Two equations are near if they contain the same
variable having nonzero coefficients of the same sign
9

10. Decomposition into Clans
10

11. Systems and Directed Bipartite Graphs
Equation –
transition (rectangle)
Variable –
place (circle)
Positive sign –
incoming arc of
a place
Negative sign –
outgoing arc of
a place
11

12. Decomposition Graph
x1, x2, x3, x4, x5, x11, x12, x13, x15, x18
x1, x10, x17 x6, x8, x16
x7, x9, x19
12

13. Collapse of Decomposition Graph
I. II.
13

14. Decomposition into Clans as
Matrix Reordering
• Clan – subset of equations
• Decomposition into clans – reordering of
rows
• Linear complexity in the number of nonzero
elements
• Classification of variables into contact and
internal (on clans) reorders columns
• Combination of a block-column and a block
diagonal matrices
14

15. Systems of Equations (Inequalities)
bxA 
its general solution
yGxx 
)(...)()()( 21 xLxLxLxS m
),0()(  xaxL i
i
Consider a system as a predicate
}{ iL
15

16. Relations on the Set of Equations
Relation of nearness: ,ji LL 
:Xxk  ,0, ,, kjki aa )()( ,, kjki asignasign 
Statement. The relation of nearness is reflexive and symmetric.
Relation of clan: ji LL 
:,...,, 21 klll LLL jlli LLLL k
 ...1
Theorem. The relation of clan is an equivalence relation
(reflexive, symmetric, and transitive).
Corollary. Relation of clan defines a partition of the set of equations;
an element of this partition is called a clan.
16

17. Classification of Variables
Variables of a clan:
}0:,{)( ,  ik
j
kii
jj
aCLXxxCXX
Internal variables of a clan:
),( j
i CXx  l
i Xx :, jlCl

j
X

j
X
Contact variables: 0
X
:, lj
CC ,j
i Xx  l
i Xx 
Contact variables of a clan: j
X

,jjj
XXX



 jj
XX



Theorem. A contact variable belongs to two clans exactly entering
one clan with sign plus and the other clan with sign minus.
17

18. Decomposition of System Matrix
18

19. Composition of Clans
1. Solve the system separately for each clan:
,0 jj
xA ,jjj
AAA


j
j
j
x
x
x 


jjj
yGx 
2. Solve a system of composition of clans for contact variables:
ll
i
jj
i yGyG  :0 yFor zRy 
3. Recover sought solutions:
,yGx  ,
000
000
000
22
11
T
kk
GJ
GJ
GJ
G




 ,zRGx 
19

20. General Solutions Obtained via
Composition of Clans
Theorem 1. A general solution of homogeneous system is:
,zHx  RGH 
Theorem 2. A general solution of heterogeneous system is:
,zHyx  ,yGxy  RGH 
Statement. Speed-up of computations is about:
)2( pq
O 
( )
( )
M q
k nz k M p  
For exponential methods – exponential speed-up:
20

21. Example: Decomposition into Clans
0000011000
1100010000
1120010000
2011000010
0011000010
0100001100
0100201100
0001100001
0001100201









A
},,,{ 6521
1
LLLLC  },,,,{ 98743
2
LLLLLC 
},,,,,,{ 72110863
1
xxxxxxxX  },,,,,,{ 95410863
2
xxxxxxxX 
},,,{ 10863
210
xxxxXXX 

},,{ 721
1
xxxX 

},,{ 954
2
xxxX 

21

22. Example: Renumeration of
Equations and Variables
 95472110863nx
 987436521nL
0110000000
1100001000
1100000200
1010000001
1010000021
0001102100
0001100100
0001010010
0001010012









A
22

23. Example: Solution of Systems for Clans
,
1110000
0101100
1100011
1
T
G 
,
1110000
10011112
T
G 
T
yyyy ),,( 1
3
1
2
1
1
1

T
yyy ),( 2
2
2
1
2

23

24. Example: Solution of System for Contact
Variables











.0
,0
,0
,0
2
1
1
2
2
1
1
2
2
1
1
1
2
1
1
1
yy
yy
yy
yy
T
R
10000
00100
01011

24

25. T
G
1110000000
1000000000
0001110000
0000101100
0001100011

T
G
1110000000
1000001111
0001110000
0000100000
0001100000
or
T
GRH
1110000000
0001110000
1001201111

Example: Composition of Source System
Solution
25

26. Sequential Contraction of Graphs as a
Scheme of Solving System
Graph of system decomposition into its clans: ),,( WEVG 
},{vV  Cv  vertices correspond to clans
VVE  edges connect clans having common contact variables
))()(())()((: 12210
21 CxOCxICxOCxIXxEvv 
)()(:  EVW  weight function;

u
uvwvw ),()()(vw number of clan variables;
),( uvw number of contact variables;
k
d
V
d
V
d
V
GGGGG
k
k
 ...210
2
2
1
1
Collapse of graph:
26

27. Collapse of Subgraphs
27

28. Edge collapse of graph
C4C5
C8 C1
C2
C3C6
C7
7
5
2
7
2
6
2
4
4
3
4
6
3
2
8
C3C7, 4
C4C5
C8 C1
C2
C3,7C6
2
7
5
7
2
4
3
2
8
9
6
C1C3, 5
C4C5
C8 C1,3,7
C2
C6
8
7
9
2
4
3
2
8
7
C1C5, 2
C4
C8 C1,3,7
C2
C6
14
15
11
4
3
7
C4
C8 C1,3,5,7
C2
C6
7
15
14
4
3
11
6
6
C1C2, 15
C4
C8 C1,2,3,5,7
C6
7
14
4
14 C1C4, 14
C8 C1,2,3,4,5,7
C6
11
14 C1C8, 11
C1,2,3,4,5,7,8
C6
14
C1C6,14 C1,2,3,4,5,6,7,8
d=15
Collapse width 15 – dimension of systems.
28

29. An Exhaustive Search of Edge Collapse
29

30. A Partial Lattice of Collapse
iiiiiii
eeeeeee 21
1
31
1
3
1
31  
tpp ii   11 t - number of triangles- number of edges
Statement. Each edge on a step of a collapse is a sum of some
edges of the source graph.
30

31. Comparing
Heuristic Strategies of
Edge Collapse
(maximal, random,
and minimal edge)
31

32. Comparison of Collapse Strategy for Random Graphs
32

33. Software
• Deborah – decomposition into clans, 2004
• Adriana – solving a homogenous system
via (a) simultaneous or (b) sequential
composition of clans, 2005
• ParAd – solving a homogenous system via
(a) simultaneous or (b) parallel-sequential
composition of clans on parallel
architectures, 2017
33

34. Composition of Clans on Parallel
Architectures
34

35. Parallel-sequential Composition of
Clans
35

36. ParAd – Parallel Adriana
36

37. Protocols of Master-Worker
Communication
(a) Sending system (b) Receiving solution
37

38. Master-Worker Basic
Communication Model
38

39. Parallel-Sequential Composition
Communication Model
39

40. Run ParAd
• Run with mpirun
>mpirun -n 5 ./ParAd -c -r zsolve tcp.spm tcp-pi.spm
>mpirun -n 10 ./ParAd -s -t -d 1 tcp.spm tcp-ti.spm
• Run with Slurm
>srun -N 10 ./ParAd -s -t -d 1 tcp.spm tcp-ti.spm
• SPM – simple sparse matrix format:
i j a[i][j]
• Check decomposability (Matrix Market Format)
>toclans lp_cre_d.mtx
40

41. Aggregation of Clans for
Workload Balancing
• A clan is a sum of minimal clans
• The maximal clan size restricts granulation
• Many small clans lead to heavy
communication load
• Balancing: create clans having size close to
the maximal
• Key: -a val
• Aggregation steeds-up about 20%
41

42. Load balancing
• Dynamic on demand – appoint a clan to
a free node (version 1.1)
• Static – aggregate minimal clans into big
clans according to the number of
available computing nodes
• Hybrid – pre-aggregation to equal size
and then dynamic scheduling clans to
nodes (version 1.2)

43. Aggregation idea

44. Hypertorus switching grid,
2D, 2x2 model

45. Decomposition of hypertorus 4D, 3x3

46. Aggregation by METIS into 7 clans

47. Extra speed-up because of
aggregation

48. Solving Systems over Real Numbers
• How to solve a linear system for a non-square
matrix (what software to use)?
• A variant: LAPACK, SVD
• A problem – accumulation of errors
• Preference to simultaneous composition
• Clans with SVD speed-up about 2 times on 16
nodes
48

49. Conclusions
• Composition of clans speeds-up solving linear
systems of equations
• Decomposition into clans is linear in the
number of nonzero elements
• Many application area matrices are
decomposable into clans
• Aggregation of clans provides load-balancing
and gives additional speed-up
• The best total speed-up obtained is 180 times
on 16 nodes with 20 cores each
49

50. Basic references
• Zaitsev D.A., Tomov S., Dongarra J. Solving Linear
Diophantine Systems on Parallel Architectures, IEEE
Transactions on Parallel and Distributed Systems, 30(05),
2019, 1158-1169.
• Zaitsev D.A. Sequential composition of linear systems’
clans, Information Sciences, Vol. 363, 2016, 292–307.
• Zaitsev D.A. Compositional analysis of Petri nets,
Cybernetics and Systems Analysis, Volume 42, Number 1
(2006), 126-136.
• Zaitsev D.A. Decomposition of Petri Nets, Cybernetics
and Systems Analysis, Volume 40, Number 5 (2004), 739-
746.
50

51. Directions for future work
• A library that implements composition of
clans independently from data types and
solvers
• Multi-core implementation of decomposition
• Multi-core implementation of sparse matrix
multiplication
• Use GPU
• Solve heterogeneous systems and inequalities
http://member.acm.org/~daze
51