Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
Loading in …5
×

# Parallel Computing 2007: Bring your own parallel application

2,877 views

Published on

Discussion of parallel implementation of some datamining applications that might be relevant on multicore systems.

Published in: Technology, Education
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Interesantes aplicaciones de optimizacion

Are you sure you want to  Yes  No
Your message goes here

### Parallel Computing 2007: Bring your own parallel application

1. 1. Parallel Computing 2007: Bring your own parallel application February 26-March 1 2007 Geoffrey Fox Community Grids Laboratory Indiana University 505 N Morton Suite 224 Bloomington IN [email_address]
2. 2. Intel’s Application Stack Discussed here Rest mainly classic parallel computing
3. 3. K-Means <ul><li>The diagrams come from Wikipedia </li></ul><ul><li>Take N data points x in some space (can be relatively abstract such as space of chemical properties) </li></ul><ul><li>We want to cluster into c components based on distance in space </li></ul><ul><li>Algorithm assumes you have a guess c k for cluster centers k=1..c </li></ul><ul><li>Associate each of N points with one and only one cluster by minimizing distance to the c k </li></ul><ul><li>Replace c k by the centroid of points associated with it </li></ul><ul><li>Iterate algorithm </li></ul>
4. 4. Problem used later in deterministic annealing version of K-Means
5. 5. K-Means illustrated Again, the centers are moved to the centroids of the corresponding associated points. Now, the association is shown in more detail, once the centroids have been moved. Centers have been associated with the points and have been moved to the respective centroids Shows the initial randomized centers and a number of points a) b) c) d)
6. 6. Parallel K-Means <ul><li>This algorithm is data parallel over N points x </li></ul><ul><li>Assign N/N proc points to each of N proc processors; no ordering needed in simple algorithm </li></ul><ul><li>Broadcast initial cluster centers c k to each processor </li></ul><ul><li>Each processor independently calculates nearest c k for each data point it is responsible before </li></ul><ul><li>Further it calculates partial sums for c centroids and error estimates (used for convergence) </li></ul><ul><li>{Sums over all points} are {Sums over processors (sums over all points in given processor)} </li></ul><ul><li>Apply MPI_Allreduce for global sums with (same) c results placed in each processor </li></ul><ul><li>All processors calculate new c k and iterate </li></ul>
7. 7. MPI Parallel Divkmeans clustering of PubChem AVIDD Linux cluster, 5,273,852 structures (Pubchem compound collection, Nov 2005) David Wild Indiana
8. 8. Performance of Parallel K-Means <ul><li>There is an an amount of distance calculation that is proportional to ( n =N/N proc )*c for c clusters and N points on N proc processors </li></ul><ul><li>There is the global sum calculation proportional to c log 2 N proc </li></ul><ul><li>So overhead f comm is log 2 N proc t comm / n t calc </li></ul><ul><li>Appearance of log 2 N proc is quite common as global sums over used </li></ul><ul><ul><li>That’s why MPI has MPI_Allreduce with hope it can be optimized on whatever network is available </li></ul></ul><ul><ul><li>Notice these MPI collectives are often not optimized and rarely used except by Marine Corps </li></ul></ul><ul><li>Note this problem has information dimension 1 </li></ul>
9. 9. Find Maximum of a distributed array TEST <ul><li>ALLREDUCE can do many reductions typically after user has done reduction internally to each processor </li></ul>
10. 10. ALLREDUCE on a multicore chip <ul><li>On a shared memory machine, one can use a different strategy by “transposing” the decomposition so that in global reduction you parallelize over c (the number of) centers not over geometric spatial decomposition </li></ul><ul><ul><li>Each core sums over contributions to a given center </li></ul></ul><ul><li>Computational Complexity is Max(1, c/N proc ) * Dimension of vector x </li></ul><ul><li>Distributed version is c log 2 N proc * Dimension of vector x </li></ul>
11. 11. Transposing Partial Sums <ul><li>Let result of parallel computation by partial sum C( i,k ) for Processor i calculating centroid k </li></ul><ul><li>1 ≤ i ≤ N proc and 1 ≤ k ≤ c </li></ul><ul><li>Take special case c = N proc = 4 </li></ul>C(1,1) C(1,2) C(1,3) C(1,4) 1 C(2,1) C(2,2) C(2,3) C(2,4) 2 C(3,1) C(3,2) C(3,3) C(3,4) 3 C(4,1) C(4,2) C(4,3) C(4,4) 4 Calculate Partial Sums locally 1 2 3 4 C(1,1)+C(2,1)+C(3,1)+C(4,1) C(1,2)+C(2,2)+C(3,2)+C(4,2) C(1,3)+C(2,3)+C(3,3)+C(4,3) C(1,4)+C(2,4)+C(3,4)+C(4,4) Transpose and sum along rows in each processor to get 100% efficiency MPI Solution cannot transpose for free and so uses a tree in this direction
12. 12. Continuing the Intel Homework Set
13. 13. Clustering by Deterministic Annealing <ul><li>One can refine this by using multi scale methods and anneal system in position resolution (Gurewitz and Rose) </li></ul>
14. 14. Deterministically find cluster centers y j using “mean field approximation” – could use slower Monte Carlo
15. 16. Annealing avoids local minima
16. 18. Deterministic Annealing <ul><li>Method does not need to assume a number of clusters </li></ul><ul><li>See K. Rose , &quot;Deterministic Annealing for Clustering, Compression, Classification, Regression, and Related Optimization Problems,&quot; Proceedings of the IEEE, vol. 80, pp. 2210-2239, November 1998 </li></ul><ul><li>Parallelization is similar to ordinary K-Means as we are calculating global sums which are decomposed into local averages and then summed over components calculated in each processor </li></ul><ul><li>I found it interesting that clustering (and K-Means) very important in Chemical Informatics for finding related compounds </li></ul><ul><ul><li>Field does not seem to know about these multi-resolution methods </li></ul></ul>
17. 19. Frequent Itemsets Mining <ul><li>We have a transaction database TDB whose records T i are a set of items {i 1 ,i 2 …..i m } </li></ul><ul><li>The i k are items from a source vocabulary {s 1 … s N } and we wish to find frequently occurring itemsets {s A , s B …} based on number of times this itemset appears in any order in a transaction </li></ul><ul><li>I looked at two algorithms – Apriori and Frequent Pattern Growth </li></ul><ul><li>Apriori focuses on the itemsets searching from smallest to largest systematically </li></ul><ul><ul><li>Natural for short transactions and small vocabularies </li></ul></ul><ul><li>Frequent Pattern Growth focuses on transactions after re-ordering them in order of item frequency </li></ul><ul><ul><li>Superior for finding long itemsets </li></ul></ul><ul><ul><li>Effectively generates a new (compact) database with re-ordered items </li></ul></ul>
18. 20. Parallel Frequent Itemsets Mining <ul><li>Parallelize by partitioning transaction database and calculating independently frequent patterns from each partition </li></ul><ul><li>Use global reduction to accumulate itemset counts from each partition </li></ul><ul><li>Now global reduction is summing counts over candidate patterns and goes together with a pruning to only consider patterns with an occurrence > than some threshold </li></ul><ul><li>This pruning is not easy to do before global sums (in spite of claims of at least one paper) </li></ul><ul><li>The “ transposed multicore ” ALLREDUCE would be a good strategy </li></ul>
19. 21. Transposing Partial Itemset Counts <ul><li>Let result of parallel computation by partial sum C( i,k ) for Processor i counting occurrences of itemset k </li></ul><ul><li>1 ≤ i ≤ N proc and 1 ≤ k ≤ c </li></ul><ul><li>Take unrealistic special case c = N proc = 4 </li></ul>MPI Solution cannot transpose for free and so uses a tree in this direction Multicore Algorithm Distributed MPI_ALLREDUCE C(1,1) C(1,2) C(1,3) C(1,4) 1 C(2,1) C(2,2) C(2,3) C(2,4) 2 C(3,1) C(3,2) C(3,3) C(3,4) 3 C(4,1) C(4,2) C(4,3) C(4,4) 4 Calculate Partial Sums locally 1 2 3 4 C(1,1)+C(2,1)+C(3,1)+C(4,1) C(1,2)+C(2,2)+C(3,2)+C(4,2) C(1,3)+C(2,3)+C(3,3)+C(4,3) C(1,4)+C(2,4)+C(3,4)+C(4,4) Transpose and sum along rows in each processor to get 100% efficiency
20. 22. (Mixed) Integer Programming <ul><li>We are solving an optimization problem such as minimize f(x) = C T x (for linear programming) </li></ul><ul><li>Subject to constraints (which are also linear for linear programming) such as A T 1 x = b 1 or A T 2 x  0 </li></ul><ul><li>With constraints that some (mixed case) or all the elements of x are integers (possibly 0 or 1) </li></ul><ul><li>The non integer problem is soluble by Simplex method or by interior point methods (Karmarkar) in polynomial time </li></ul><ul><li>The integer programming problem is NP complete </li></ul>
21. 23. Integer Programming Parallelization <ul><li>Typically one does not parallelize the linear program solver but rather runs this sequentially and instead parallelizes a branch and bound (or cut) search over possible solutions in NP complete case </li></ul><ul><ul><li>e.g. search over integer choices for x </li></ul></ul><ul><li>The hard integer programming problem consists of Divide space into subspaces Find upper and lower bounds on f(x) in each subspace If lower bound on f(x) in a subspace is greater than current minimum of upper bounds of f(x) in other subspaces (i.e. upper bound of f(x) in any subspace), then one can prune this subspace </li></ul><ul><li>If a subspace is still active and upper bound > lower bound , then further divide it into subspaces and iterate process </li></ul><ul><li>Parallelism comes from “ data parallelism ” over subspaces which is suitable for thread based systems </li></ul><ul><li>There is typically important shared knowledge such as current minimum upper bound and other information from one subspace that can be re-used by others </li></ul><ul><ul><li>Shared (in memory) database for performance </li></ul></ul>
22. 24. Computer Chess I <ul><li>Games like computer chess are a special case of the general branch and bound strategy </li></ul><ul><li>The space is the set of all moves where N moves by white and black is 2N plys ; at each ply there are roughly 35 legal moves so complexity is 35 2N </li></ul><ul><li>Evaluation of of one set of moves to depth 2N is completed by evaluating the final position f( x ; x is set of moves) by rules reflecting chess wisdom and summarized by a number (Queen=10, Pawn =1 etc.) </li></ul><ul><li>Deep Blue parallelized the calculation of f( x ) but here we explore subspace parallelization </li></ul><ul><li>We follow work done at Caltech using a 512 node nCUBE which competed as WAYCOOL with poor reliability and results in 1987 and 1988 ACM Computer Chess Championships </li></ul>
23. 25. Computer Chess II <ul><li>The upper-lower bound approach is replaced by a minimax principle </li></ul><ul><li>Assume f( x ) positive is good for white; then at each move white looks at each subspace spawned from the white move and chooses the one with the largest f( x ) </li></ul><ul><li>In evaluating the subspace we assume that each stage, the side on move makes the best choice </li></ul><ul><ul><li>White always maximizes f( x ) at her move and black minimizes f(x) at his move </li></ul></ul><ul><li>Of course as N is finite and evaluation function approximate, this is not precise but it gets better and better the larger N is </li></ul><ul><li>Note human players tend to use more pattern recognition and less brute force evaluation </li></ul><ul><ul><li>Computer games are unimaginative but have fewer errors </li></ul></ul>
24. 26. Computer Chess III <ul><li>Pruning is illustrated below; as it is advantageous to get (if white is to move) to get a large (good) value of f(x) as early as possible, one sorts moves at each node and looks at the most plausible first </li></ul><ul><ul><li>This reduces effective branching ratio from 35 to 6 </li></ul></ul>4 4 -1 -7 -17 White Maximizes Black Minimizes The dotted lines show subspaces that never need to be searched ; this requires that one have done a complete depth search at first subspaces looked at 4 29 13 -1 5 2 -7 3 15 -11 -10 -17 5
25. 27. Computer Chess IV <ul><li>Threads were spawned in groups of 4 in Caltech example at different depths of tree and project achieved a speed up of over a 100 and the larger # plys N gets the more parallelism there will be </li></ul>Increasing search depth
26. 28. Computer Chess V <ul><li>We have subsets of threads (4 in this example) synchronizing on node minimax value </li></ul><ul><li>This is a global variable and there are (as in other branch and bound) very important performance gains from a shared position database </li></ul><ul><li>This allows scores to be stored for positions and re-used </li></ul><ul><li>In chess there are many transpositions leading to identical positions </li></ul><ul><ul><li>1 e4 e5 2 Nf3 Nc6 is identical to (less usual) 1 Nf3 Nc6 2 e4 e5 </li></ul></ul><ul><li>There was only a few percent overhead for a distributed database on Caltech distributed memory implementation </li></ul><ul><ul><li>Queuing of update requests ensured no errors from multiple threads accessing same location </li></ul></ul><ul><li>Multicore architecture should be excellent for this and other large branch and bound and related search algorithms as support shared databases and fast thread synchronization </li></ul><ul><li>Note that in Deep Fritz vs. Vladimir Kramnik (human world champion) in November 2006, the program ran on a personal computer containing two Intel Core 2 Duo CPUs, capable of evaluating 8 million positions per second , and searching to an average depth of 17 to 18 ply in the middlegame. Deep Fritz won 4-2 </li></ul>
27. 29. Wikipedia SVM Example <ul><li>We are finding optimal hyperplane splitting two samples </li></ul><ul><li>Samples are training set </li></ul><ul><li>Normal w to splitting hyperplane given by w =  i =1 n y i  i x i </li></ul><ul><li>Two samples denoted by crosses y i =1 or circles y i = -1 </li></ul>
28. 30. Support Vector Machines SVM I <ul><li>These divide sets by (in simplest case) hyperplanes into two in an optimal least squares fashion </li></ul><ul><li>Minimize f(  ) = 0.5  T G  -  i =1 n  i </li></ul><ul><li>Subject to  i =1 n y i  i = 0 and 0 ≤  i ≤ C </li></ul><ul><li>With G ij = y i y j K( x i , x j ) for Kernel K </li></ul><ul><li>This is a training problem where we have a total of n data points from two populations with y i = +1 for first and = -1 for second </li></ul><ul><li>K( x i , x j ) = x i . x j is simplest case when division is by a hyperplane in space in which x is a vector but Gaussian forms are often used K = exp(- constant  x i - x j  2 ) </li></ul><ul><li>G is an n by n dense matrix (n is number of data points) </li></ul><ul><li>This is a a quadratic programming QP problem </li></ul>
29. 31. Support Vector Machines SVM II <ul><li>Differentiating wrt  gives linear equations that must solved iteratively to satisfy inequality constraints </li></ul><ul><li>The solver matrix G is both large (10 6 by 10 6 ) and can be dense and this requires large storage space which often exceeds available memory </li></ul><ul><li>As in much quadratic programming one can use conjugate gradient solution methods as this identifies systematically the important directions in space (roughly large eigenvalues of positive definite symmetric matrix G) </li></ul><ul><li>There are several papers on parallel SVM but I did not see substantial use of parallel implementations </li></ul><ul><li>There were two approaches </li></ul><ul><ul><li>Either solve the matrix problems in parallel or </li></ul></ul><ul><ul><li>Split up dataset and solve multiple subproblems </li></ul></ul>
30. 32. Support Vector Machines SVM III <ul><li>Solve the matrix problems in parallel </li></ul><ul><ul><li>Interestingly one does not solve full G but iterates up from smaller (~150 by 150) problems and so data parallelism does not exploit size n </li></ul></ul><ul><ul><li>Need more reliable SVM solvers for large matrices? </li></ul></ul><ul><li>Split up dataset and solve multiple subproblems – Scalable! </li></ul><ul><ul><li>Here the difficulty is that essentially you have changed algorithm and it is not clear how best to combine solution of subproblems </li></ul></ul><ul><ul><li>But original SVM is full of heuristics (choice of K) so other heuristics may be allowed! </li></ul></ul><ul><li>Note whereas multicore appears especially attractive for search problems, it is not so clear for SVM </li></ul><ul><ul><li>Multicore does not address huge size of matrix G </li></ul></ul><ul><ul><li>High performance matrix solvers are available for distributed memory machines </li></ul></ul><ul><ul><li>I suspect there are better “approximate” SVM solvers that will do well on multicore and reduce dimension of G but this is research </li></ul></ul>
31. 33. Some Parallelization Results from “Parallel Software for Training Large Scale Support Vector Machines on Multiprocessor Systems” This paper reviews much previous work Super linear speedup in (a) due to extra memory