Sparse matrix computations in MapReduce

1. ICME MapReduce Workshop! April 29 – May 1, 2013! ! David F. Gleich! Computer Science! Purdue University David Gleich · Purdue 1 ! Website www.stanford.edu/~paulcon/icme-mapreduce-2013 Paul G. Constantine! Center for Turbulence Research! Stanford University MRWorkshop

2. Goals Learn the basics of MapReduce & Hadoop Be able to process large volumes of data from science and engineering applications … help enable you to explore on your own! David Gleich · Purdue 2 MRWorkshop

3. Workshop overview Monday! Me! Sparse matrix computations in MapReduce! Austin Benson Tall-and-skinny matrix computations in MapReduce Tuesday! Joe Buck Extending MapReduce for scientiﬁc computing! Chunsheng Feng Large scale video analytics on pivotal Hadoop Wednesday! Joe Nichols Post-processing CFD dynamics data in MapReduce ! Lavanya Ramakrishnan Evaluating MapReduce and Hadoop for science David Gleich · Purdue 3 MRWorkshop

4. Sparse matrix computations in MapReduce! David F. Gleich! Computer Science! Purdue University David Gleich · Purdue 4 Slides online soon! Code https://github.com/dgleich/mapreduce-matrix-tutorial MRWorkshop

5. How to compute with big matrix data ! A tale of two computers 224k Cores 10 PB drive 1.7 Pﬂops 7 MW Custom ! interconnect! $104 M 80k cores! 50 PB drive ? Pﬂops ? MW GB ethernet $?? M 625 GB/core! High disk to CPU 45 GB/core High CPU to disk 5 ORNL 2010 Supercomputer! Google’s 2010? ! Data computer! David Gleich · Purdue MRWorkshop

6. My data computers 6 Nebula Cluster @ Sandia CA! 2TB/core storage, 64 nodes, 256 cores, GB ethernet Cost $150k These systems are good for working with enormous matrix data! ICME Hadoop @ Stanford! 3TB/core storage, 11 nodes, 44 cores, GB ethernet Cost $30k David Gleich · Purdue MRWorkshop

7. My data computers 7 Nebula Cluster @ Sandia CA! 2TB/core storage, 64 nodes, 256 cores, GB ethernet Cost $150k These systems are good for working with enormous matrix data! ICME Hadoop @ Stanford! 3TB/core storage, 11 nodes, 44 cores, GB ethernet Cost $30k ^ but not great, David Gleich · Purdue some ^ MRWorkshop

8. By 2013(?) all Fortune 500 companies will have a data computer David Gleich · Purdue 8 MRWorkshop

9. How do you program them? 9 David Gleich · Purdue MRWorkshop

10. MapReduce and! Hadoop overview 10 David Gleich · Purdue MRWorkshop

11. MapReduce is designed to solve a different set of problems from standard parallel libraries 11 David Gleich · Purdue MRWorkshop

12. The MapReduce programming model Input a list of (key, value) pairs Map apply a function f to all pairs Reduce apply a function g to ! all values with key k (for all k) Output a list of (key, value) pairs 12 David Gleich · Purdue MRWorkshop

13. Computing a histogram ! A simple MapReduce example 13 Input! ! Key ImageId Value Pixels Map(ImageId, Pixels) for each pixel emit" Key = (r,g,b)" Value = 1 Reduce(Color, Values) emit" Key = Color Value = sum(Values) Output! ! Key Color Value ! # of pixels David Gleich · Purdue 5 15 10 9 3 17 5 10 1 1 1 1 Map Reduce 1 1 1 1 1 1 1 1 1 1 1 1 shufﬂe MRWorkshop

14. Many matrix computations are possible in MapReduce Column sums are easy ! Input Key (i,j) Value Aij Other basic methods ! can use common parallel/out-of-core algs! Sparse matrix-vector products y = Ax Sparse matrix-matrix products C = AB 14 Reduce(j,Values) emit Key = j, Value = sum(Values) David Gleich · Purdue Map((i,j), val) emit" Key = j, Value = val A11 A12 A13 A14 A21 A22 A23 A24 A31 A32 A33 A34 A41 A42 A43 A44 (3,4) -> 5 (1,2) -> -6.0 (2,3) -> -1.2 (1,1) -> 3.14 … “Coordinate storage” MRWorkshop

15. Many matrix computations are possible in MapReduce Column sums are easy ! Input Key (i,j) Value Aij Other basic methods ! can use common parallel/out-of-core algs! Sparse matrix-vector products y = Ax Sparse matrix-matrix products C = AB 15 Reduce(j,Values) emit Key = j, Value = sum(Values) David Gleich · Purdue Map((i,j), val) emit" Key = j, Value = val A11 A12 A13 A14 A21 A22 A23 A24 A31 A32 A33 A34 A41 A42 A43 A44 (3,4) -> 5 (1,2) -> -6.0 (2,3) -> -1.2 (1,1) -> 3.14 … “Coordinate storage” Beware of un-thoughtful ideas MRWorkshop

16. Why so many limitations? 16 David Gleich · Purdue MRWorkshop

17. The MapReduce programming model Input a list of (key, value) pairs Map apply a function f to all pairs Reduce apply a function g to ! all values with key k (for all k) Output a list of (key, value) pairs Map function f must be side-effect free! All map functions run in parallel Reduce function g must be side-effect free! All reduce functions run in parallel 17 David Gleich · Purdue MRWorkshop

18. A graphical view of the MapReduce programming model David Gleich · Purdue 18 data Map data Map data Map data Map key value key value key value key value key value key value () Shuffle key value value dataReduce key value value value dataReduce key value dataReduce MRWorkshop

19. Data scalability The idea ! Bring the computations to the data MR can schedule map functions without moving data. 1 M M R R M M M Maps Reduce Shufﬂe 2 3 4 5 1 2 M M 3 4 M M 5 M 19 David Gleich · Purdue MRWorkshop

20. After waiting in the queue for a month and ! after 24 hours of ﬁnding eigenvalues, one node randomly hiccups. heartbreak on node rs252 David Gleich · Purdue 20 MRWorkshop

21. Fault tolerant Redundant input helps make maps data-local Just one type of communication: shufﬂe M M R R M M Input stored in triplicate Map output! persisted to disk! before shufﬂe Reduce input/! output on disk David Gleich · Purdue 21 MRWorkshop

22. Fault injection 10 100 1000 1/Prob(failure) – mean number of success per failure Timetocompletion(sec) 200 100 No faults (200M by 200) Faults (800M by 10) Faults (200M by 200) No faults ! (800M by 10) With 1/5 tasks failing, the job only takes twice as long. David Gleich · Purdue 22 MRWorkshop

23. Data scalability The idea ! Bring the computations to the data MR can schedule map functions without moving data. 1 M M R R M M M Maps Reduce Shufﬂe 2 3 4 5 1 2 M M 3 4 M M 5 M 23 David Gleich · Purdue MRWorkshop

24. Computing a histogram ! A simple MapReduce example 24 Input! ! Key ImageId Value Pixels Map(ImageId, Pixels) for each pixel emit" Key = (r,g,b)" Value = 1 Reduce(Color, Values) emit" Key = Color Value = sum(Values) Output! ! Key Color Value ! # of pixels David Gleich · Purdue 5 15 10 9 3 17 5 10 1 1 1 1 Map Reduce 1 1 1 1 1 1 1 1 1 1 1 1 shufﬂe The entire dataset is “transposed” from images to pixels. This moves the data to the computation! (Using a combiner helps to reduce the data moved, but it cannot always be used) MRWorkshop

25. Hadoop and MapReduce are bad systems for some matrix computations. David Gleich · Purdue 25 MRWorkshop

26. How should you evaluate a MapReduce algorithm? Build a performance model! Measure the worst mapper Usually not too bad Measure the data moved Could be very bad Measure the worst reducer Could be very bad David Gleich · Purdue 26 MRWorkshop

27. Tools I like hadoop streaming dumbo mrjob hadoopy C++ David Gleich · Purdue 27 MRWorkshop

28. Tools I don’t use but other people seem to like … pig java hbase mahout Eclipse Cassandra David Gleich · Purdue 28 MRWorkshop

29. hadoop streaming the map function is a program! (key,value) pairs are sent via stdin! output (key,value) pairs goes to stdout the reduce function is a program! (key,value) pairs are sent via stdin! keys are grouped! output (key,value) pairs goes to stdout David Gleich · Purdue 29 MRWorkshop

30. mrjob from a wrapper around hadoop streaming for map and reduce functions in python class MRWordFreqCount(MRJob): def mapper(self, _, line): for word in line.split(): yield (word.lower(), 1) def reducer(self, word, counts): yield (word, sum(counts)) if __name__ == '__main__': MRWordFreqCount.run() David Gleich · Purdue 30 MRWorkshop

31. How can Hadoop streaming possibly be fast? Iter 1 QR (secs.) Iter 1 Total (secs.) Iter 2 Total (secs.) Overall Total (secs.) Dumbo 67725 960 217 1177 Hadoopy 70909 612 118 730 C++ 15809 350 37 387 Java 436 66 502 Synthetic data test 100,000,000-by-500 matrix (~500GB) Codes implemented in MapReduce streaming Matrix stored as TypedBytes lists of doubles Python frameworks use Numpy+Atlas Custom C++ TypedBytes reader/writer with Atlas New non-streaming Java implementation too David Gleich (Sandia) All timing results from the Hadoop job tracker C++ in streaming beats a native Java implementation. 16/22MapReduce 2011 David Gleich · Purdue 31 Example available from github.com/dgleich/mrtsqr! for veriﬁcation mrjob could be faster if it used typedbytes for intermediate storage see https://github.com/Yelp/mrjob/pull/447 MRWorkshop

32. Code samples and short tutorials at github.com/dgleich/mrmatrix github.com/dgleich/mapreduce-matrix-tutorial David Gleich · Purdue 32 MRWorkshop

33. Matrix-vector product David Gleich · Purdue 33 Ax = y yi = X k Aik xk A x Follow along! mapreduce-matrix-tutorial! /codes/smatvec.py! MRWorkshop

34. Matrix-vector product David Gleich · Purdue 34 Ax = y yi = X k Aik xk A x A is stored by row $ head samples/smat_5_5.txt ! 0 0 0.125 3 1.024 4 0.121! 1 0 0.597! 2 2 1.247! 3 4 -1.45! 4 2 0.061! x is stored entry-wise ! $ head samples/vec_5.txt! 0 0.241! 1 -0.98! 2 0.237! 3 -0.32! 4 0.080! Follow along! mapreduce-matrix-tutorial! /codes/smatvec.py! MRWorkshop

35. Matrix-vector product! (in pictures) David Gleich · Purdue 35 Ax = y yi = X k Aik xk A x A x Input Map 1! Align on columns! Reduce 1! Output Aik xk! keyed on row i A x Reduce 2! Output sum(Aik xk)! y MRWorkshop

36. Matrix-vector product! (in pictures) David Gleich · Purdue 36 Ax = y yi = X k Aik xk A x A x Input Map 1! Align on columns! def joinmap(self, key, line):! vals = line.split()! if len(vals) == 2:! # the vector! yield (vals[0], # row! (float(vals[1]),)) # xi! else:! # the matrix! row = vals[0]! for i in xrange(1,len(vals),2):! yield (vals[i], # column! (row, # i,Aij! float(vals[i+1])))! MRWorkshop

37. Matrix-vector product! (in pictures) David Gleich · Purdue 37 Ax = y yi = X k Aik xk A x A x Input Map 1! Align on columns! Reduce 1! Output Aik xk! keyed on row i A x def joinred(self, key, vals):! vecval = 0. ! matvals = []! for val in vals:! if len(val) == 1:! vecval += val[0]! else:! matvals.append(val) ! for val in matvals:! yield (val[0], val[1]*vecval)! Note that you should use a secondary sort to avoid reading both in memory MRWorkshop

38. Matrix-vector product! (in pictures) David Gleich · Purdue 38 Ax = y yi = X k Aik xk A x A x Input Map 1! Align on columns! Reduce 1! Output Aik xk! keyed on row i A x Reduce 2! Output sum(Aik xk)! y def sumred(self, key, vals):! yield (key, sum(vals))! MRWorkshop

39. Move the computations to the data? Not really! David Gleich · Purdue 39 A x A x Input Map 1! Align on columns! Reduce 1! Output Aik xk! keyed on row i A x Reduce 2! Output sum(Aik xk)! y Copy data once, now aligned on column Copy data again, align on row MRWorkshop

40. Matrix-matrix product David Gleich · Purdue 40 A B AB = C Cij = X k Aik Bkj Follow along! mapreduce-matrix-tutorial! /codes/matmat.py! MRWorkshop

41. Matrix-matrix product David Gleich · Purdue 41 A B AB = C Cij = X k Aik Bkj A is stored by row $ head samples/smat_10_5_A.txt ! 0 0 0.599 4 -1.53! 1! 2 2 0.260! 3! 4 0 0.267 1 0.839 B is stored by row $ head samples/smat_5_5.txt ! 0 0 0.125 3 1.024 4 0.121! 1 0 0.597! 2 2 1.247! Follow along! mapreduce-matrix-tutorial! /codes/matmat.py! MRWorkshop

42. Matrix-matrix product ! (in pictures) David Gleich · Purdue 42 A B AB = C Cij = X k Aik Bkj A Map 1! Align on columns! B Reduce 1! Output Aik Bkj! keyed on (i,j) A B Reduce 2! Output sum(Aik Bkj)! C MRWorkshop

43. Matrix-matrix product ! (in code) David Gleich · Purdue 43 A B AB = C Cij = X k Aik Bkj A Map 1! Align on columns! B def joinmap(self, key, line):! mtype = self.parsemat()! vals = line.split()! row = vals[0]! rowvals = ! [(vals[i],float(vals[i+1])) ! for i in xrange(1,len(vals),2)]! if mtype==1:! # matrix A, output by col! for val in rowvals:! yield (val[0], (row, val[1]))! else:! yield (row, (rowvals,))! MRWorkshop

44. Matrix-matrix product ! (in code) David Gleich · Purdue 44 A B AB = C Cij = X k Aik Bkj A Map 1! Align on columns! B Reduce 1! Output Aik Bkj! keyed on (i,j) A B def joinred(self, key, line):! # load the data into memory ! brow = []! acol = []! for val in vals:! if len(val) == 1:! brow.extend(val[0])! else:! acol.append(val)! ! for (bcol,bval) in brow:! for (arow,aval) in acol:! yield ((arow,bcol),aval*bval)! MRWorkshop

45. Matrix-matrix product ! (in pictures) David Gleich · Purdue 45 A B AB = C Cij = X k Aik Bkj A Map 1! Align on columns! B Reduce 1! Output Aik Bkj! keyed on (i,j) A B Reduce 2! Output sum(Aik Bkj)! C def sumred(self, key, vals):! yield (key, sum(vals))! MRWorkshop

46. Why is MapReduce so popular? if (root) {! PetscInt cur_nz=0;! unsigned char* root_nz_buf;! unsigned int *root_nz_buf_i,*root_nz_buf_j;! double *root_nz_buf_v;! PetscMalloc((sizeof(unsigned int)*2+sizeof(double))*root_nz_bufsize,root_nz_buf);! PetscMalloc(sizeof(unsigned int)*root_nz_bufsize,root_nz_buf_i);! PetscMalloc(sizeof(unsigned int)*root_nz_bufsize,root_nz_buf_j);! PetscMalloc(sizeof(double)*root_nz_bufsize,root_nz_buf_v);! ! unsigned long long int nzs_to_read = total_nz;! ! while (send_rounds 0) {! // check if we are near the end of the file! // and just read that amount! size_t cur_nz_read = root_nz_bufsize;! if (cur_nz_read nzs_to_read) {! cur_nz_read = nzs_to_read;! }! PetscInfo2(PETSC_NULL, reading %i non-zeros of %llin, cur_nz_read, nzs_to_read);! 600 lines of gross code in order to load a sparse matrix into memory, streaming from one processor. MapReduce offers a better alternative David Gleich · Purdue 46 MRWorkshop

47. Thoughts on a better system Default quadruple precision Matrix computations without indexing Easy setup of MPI data jobs David Gleich · Purdue 47                                        Initial data load of any MPI job Compute task MRWorkshop

48. Double-precision ﬂoating point was designed for the era where “big” was 1000-10000 David Gleich · Purdue 48 MRWorkshop

49. Error analysis of summation s = 0; for i=1 to n: s = s + x[i] A simple summation formula has ! error that is not always small if n is a billion David Gleich · Purdue 49 ﬂ(x + y) = (x + y)(1 + ) ﬂ( X i xi ) X i xi  nµ X i |xi | µ ⇡ 10 16 MRWorkshop

50. If your application matters then watch out for this issue. Use quad-precision arithmetic or compensated summation instead. David Gleich · Purdue 50 MRWorkshop

51. Compensated Summation “Kahan summation algorithm” on Wikipedia s = 0.; c = 0.; for i=1 to n: y = x[i] – c t = s + y c = (t – s) – y s = t David Gleich · Purdue 51 Mathematically, c is always zero. On a computer, c can be non-zero The parentheses matter! ﬂ(csum(x)) X i xi  (µ + nµ2 ) X i |xi | µ ⇡ 10 16 MRWorkshop

52. Summary MapReduce is a powerful but limited tool that has a role in the future of computational math. … but it should be used carefully! See Austin’s talk next! David Gleich · Purdue 52 MRWorkshop Code samples and short tutorials at github.com/dgleich/mrmatrix github.com/dgleich/mapreduce-matrix-tutorial

53. David Gleich · Purdue 53 MRWorkshop

Sparse matrix computations in MapReduce

Sparse matrix computations in MapReduce

Recommended

More Related Content

What's hot

What's hot(20)

Viewers also liked

Viewers also liked(20)

Similar to Sparse matrix computations in MapReduce

Similar to Sparse matrix computations in MapReduce(20)

More from David Gleich

More from David Gleich(20)

Recently uploaded

Recently uploaded(20)

Sparse matrix computations in MapReduce