Large Scale Math with Hadoop MapReduce

Large Scale Math with Hadoop MapReduce Tsz-Wo (Nicholas) Sze, PhD Hadoop Summit June 29, 2011 1

Who am I?• Hortonworks Software Engineer• Apache Hadoop PMC Member• Mathematician Interests: Distributed Computing Algorithms Number Theory 2

Agenda • Introduction • Integer Multiplication • MapReduce-FFT • MapReduce-Sum • MapReduce-SSA • A New World Record • The “Machine” Behind the ComputationTsz-Wo Sze, Hadoop Summit 2011 3

Typical Hadoop Applications Major applications of Hadoop include • Search and crawling • Text processing • Machine learning • ...Tsz-Wo Sze, Hadoop Summit 2011 5

Typical Hadoop Applications Major applications of Hadoop include • Search and crawling • Text processing • Machine learning • ... But not yet commonly used in scientiﬁc or mathematical applications. Why?Tsz-Wo Sze, Hadoop Summit 2011 6

Why Not Math? No MapReduce math libraries available, and More fundamentally, MapReduce math algorithms are not well studied.Tsz-Wo Sze, Hadoop Summit 2011 7

Existing Library Really no MapReduce Math Library? Not exactly.Tsz-Wo Sze, Hadoop Summit 2011 8

Existing Library Really no MapReduce Math Library? Not exactly. Apache Mahout • A machine learning library. • Includes packages for matrix operations.Tsz-Wo Sze, Hadoop Summit 2011 9

Existing Library Really no MapReduce Math Library? Not exactly. Apache Mahout • A machine learning library. • Includes packages for matrix operations. Apache Hama (Incubation) • A matrix computational package.Tsz-Wo Sze, Hadoop Summit 2011 10

Computational Intensive Problems (1) Integer Factoring • a.k.a. breaking RSA cryptosystem Given N , e and c, compute m such that     e c ≡ m (mod N ),       where N is a product of two primes. • a 768-bit RSA modulus was factored1 in 2009 1 Kleinjung et al., Factorization of a 768-bit RSA modulus, CRYPTO 2010.Tsz-Wo Sze, Hadoop Summit 2011 11

Computational Intensive Problems (2) Solving PDEs (Partial Diﬀerential Equations) • Fluid dynamics • Electromagnetism • Financial analysis • ... (Two-dimensional Turbulence, courtesy of Y.K. Tsang)Tsz-Wo Sze, Hadoop Summit 2011 12

Computational Intensive Problems (3) Finding complex zeros of Riemann Zeta function ∞ 1 ζ(s) = for s ∈ C, (s) > 1 n=1 ns and then analytically continued to all s = 1.Tsz-Wo Sze, Hadoop Summit 2011 13

Computational Intensive Problems (3) Finding complex zeros of Riemann Zeta function ∞ 1 ζ(s) = for s ∈ C, (s) > 1 n=1 ns and then analytically continued to all s = 1. • Disprove Riemann Hypothesis (RH) Then, you will get $1,000,000 dollars2. However, RH is unlikely to be false. 2 See http://www.claymath.org/millennium/Riemann_Hypothesis/.Tsz-Wo Sze, Hadoop Summit 2011 14

Computational Intensive Problems (3) Finding complex zeros of Riemann Zeta function ∞ 1 ζ(s) = for s ∈ C, (s) > 1 n=1 ns and then analytically continued to all s = 1. • Disprove Riemann Hypothesis (RH) Then, you will get $1,000,000 dollars. However, RH is unlikely to be false. • More likely: Obtain more evidents which support RH.Tsz-Wo Sze, Hadoop Summit 2011 15

Computational Intensive Problems (4) Computing π Latest world records: • Five trillion decimal digits (August 2010) by Alexander Yee & Shigeru Kondo3 3 See http://www.numberworld.org/misc_runs/pi-5t/announce_en.htmlTsz-Wo Sze, Hadoop Summit 2011 16

Computational Intensive Problems (4) Computing π Latest world records: • Five trillion decimal digits (August 2010) by Alexander Yee & Shigeru Kondo • The two quadrillionth bits (July 2010) by Tsz-Wo Sze & the Yahoo! Cloud Computing Team4 4 See http://developer.yahoo.net/blogs/hadoop/2010/09/two_quadrillionth_bit_pi.htmlTsz-Wo Sze, Hadoop Summit 2011 17

Missing Functionalities Fast Fourier Transform (FFT) – the basic rountine behind many algorithms. Arbitrary Precision Arithmetic Integer functions Floating-point functions Complex functions ...Tsz-Wo Sze, Hadoop Summit 2011 18

Why Integer Multiplication? There exist fast algorithms. Many applications • Division • Logarithm • Trigonometric functions • ...Tsz-Wo Sze, Hadoop Summit 2011 20

Prerequisite of Algorithms D.J. Bernstein, Fast multiplication and its applications, ANTS 2008.Tsz-Wo Sze, Hadoop Summit 2011 21

Integer Multiplication Algorithms Na¨ O(N 2) ıve, Karatsuba, O(N log2 3) = O(N 1.585) Toom-Cook, O(N log(2D−1)/ log D ) If D = 3, then O(N log 5/ log 3) = O(N 1.465) FFT-based algorithms O(N log N · · · )Tsz-Wo Sze, Hadoop Summit 2011 22

FFT-based Algorithms Basic FFT, O(N log N log log N log log log N · · · ) Sch¨nhage-Strassen, O(N log N log log N ) o Nussbaumer, O(N log N log log N ) log∗ N F¨rer, O(N (log N )2 u ) log∗ N De-Kurur-Saha-Saptharishi, O(N (log N )2 )Tsz-Wo Sze, Hadoop Summit 2011 23

Convolution By the convolution theorem, a × b = dft−1(dft(a) ∗ dft(b)), where × denotes the convolution operator , ∗ denotes componentwise multiplication, dft( · ) denotes discrete Fourier transform.Tsz-Wo Sze, Hadoop Summit 2011 24

Sch¨nhage-Strassen Algorithm o (SSA) Represent integers as polynomials. Then, compute convolution with DFTs modulo an integer5. 5 It has the form 2n + 1 and is called the Sch¨nhage-Strassen modulas. oTsz-Wo Sze, Hadoop Summit 2011 25

SSA Steps Step 1: two DFTs, def ˆ def dft(b); ˆ a = dft(a) and b= Step 2: componentwise multiplication, def ˆ ˆ p = a ∗ b; ˆ Step 3: a DFT inverse, −1 p = dft (ˆ ); p Step 4: normalization.Tsz-Wo Sze, Hadoop Summit 2011 26

Calculating DFTs DFT can be calculated by a family of algorithms called Fast Fourier Transform (FFT).Tsz-Wo Sze, Hadoop Summit 2011 27

FFT Family Recursive-FFT Parallel-FFT Cooley-Tukey (decimation-in-time) Gentleman-Sande (decimation-in-frequency) Danielson-Lanczos Ping-pong FFT ...Tsz-Wo Sze, Hadoop Summit 2011 28

Data Model (1) Need a data model which allows accessing terabit integers eﬃciently. An integer x is represented as a D-dimensional tuple x = (xD−1, xD−2, . . . , x0).Tsz-Wo Sze, Hadoop Summit 2011 29

Data Model (2) Write D = IJ. where I and J are powers of two. Deﬁne J-dimensional tuples (i) def x = (x(J−1)I+i, x(J−2)I+i, . . . , xi) for 0 ≤ i < I.Tsz-Wo Sze, Hadoop Summit 2011 30

Data Model (3) Then,     x(0) x(J−1)I x(J−2)I . . . x0  (1)    x   x(J−1)I+1 x(J−2)I+1 . . . x1    . = . . ... .   .   . . .  x(I−1) x(J−1)I+(I−1) x(J−2)I+(I−1) . . . xI−1 We call it the (I, J)-format of x.Tsz-Wo Sze, Hadoop Summit 2011 31

Data Model (4) Each x(i) is a sequence of J records. Each record is a key-value pair. Record # <Key, Value> 0 < i, xi > 1 < J + i, xJ+i > . . . . J −1 < (J − 1)I + i, x(J−1)I+i >Tsz-Wo Sze, Hadoop Summit 2011 32

Data Model (5) Thus, an integer is stored as I SequenceFiles in HDFS, each SequenceFile contains J records.Tsz-Wo Sze, Hadoop Summit 2011 33

Parallel-FFT Steps Step 1: I inner DFTs with J-point, a(i) = dft(a(i)); Step 2: componentwise shifting, def zjI+i = ζ ij a(i)j ; Step 3: transposition, [j] def z = (zjI+(I−1), zjI+(I−2), . . . , zjI ); Step 4: J outer DFTs with I-point, [j] def z = dft(z[j]).Tsz-Wo Sze, Hadoop Summit 2011 34

MapReduce Model Input Map1 Map2 Map3 Map4 Shuffle Reduce1 Reduce2 Reduce3 Reduce4 OutputTsz-Wo Sze, Hadoop Summit 2011 35

MapReduce-FFT Input Inner FFT1 Inner FFT2 Inner FFT3 Inner FFT4Transposition (by shuffle) Outer FFT1 Outer FFT2 Outer FFT3 Outer FFT4 Output Tsz-Wo Sze, Hadoop Summit 2011 36

Data Locality The FFT transposition, which is traditionally dif- ﬁcult in preserving locality, becomes trivial in MapReduce.Tsz-Wo Sze, Hadoop Summit 2011 37

MapReduce-FFT (1) Map function: (k1, v1) −→ list k2, v2 Algorithm 1 (Forward FFT, Mapper). (f.m.1) read key i, value a(i); (f.m.2) calculate a J-point DFT; (f.m.3) componentwise multiply; (f.m.4) for 0 ≤ j < J, emit key j, value (i, zjI+i).Tsz-Wo Sze, Hadoop Summit 2011 38

MapReduce-FFT (2) Reduce function: (k2, list v2 ) −→ list k3, v3 . Algorithm 2 (Forward FFT, Reducer). (f.r.1) receive key j, list [(i, zjI+i)]0≤i<I ; (f.r.2) calculate an I-point DFT; (f.r.3) write key j, value z[j].Tsz-Wo Sze, Hadoop Summit 2011 39

Normalization Normalization can be viewed as a summation of three integers.Tsz-Wo Sze, Hadoop Summit 2011 40

Summation Integer summation can be done by (1) componentwise summation, (2) carry evaluation and then (3) parallel carrying.Tsz-Wo Sze, Hadoop Summit 2011 41

MapReduce Model Input Map1 Map2 Map3 Map4 Shuffle Reduce1 Reduce2 Reduce3 Reduce4 OutputTsz-Wo Sze, Hadoop Summit 2011 42

MapReduce-Sum Input Summation1 Summation2 Summation3 Summation4 Carry Evaluation(modified shuffle) Carrying1 Carrying2 Carrying3 Carrying4 Output Tsz-Wo Sze, Hadoop Summit 2011 43

Job 1: Componwise Summation Input Summation1 Summation2 Summation3 Summation4 Output A map-only job.Tsz-Wo Sze, Hadoop Summit 2011 44

Job 2: Carrying Input Carry Evaluation Carrying1 Carrying2 Carrying3 Carrying4 OutputTsz-Wo Sze, Hadoop Summit 2011 45

MapReduce-SSA two concurrent forward FFT jobs; a backward FFT job with componentwise multiplication and splitting ; a componentwise summation map-only job; a carrying job6. 6 It is possible to combine the last two jobs if we modify the shuﬄe process in MapReduce [.next].Tsz-Wo Sze, Hadoop Summit 2011 46

Prototype Implementation DistMpMult – distributed multi-precision multiplication DistFft – distributed FFT DistCompSum – distributed componentwise summation DistCarrying – distributed carrying Open source – available at https://issues.apache.org/jira/browse/MAPREDUCE-2471Tsz-Wo Sze, Hadoop Summit 2011 47

Cluster Conﬁguration A shared cluster: Apache Hadoop 0.20 1350 nodes 6 GB memory per node 2 map tasks & 1 reduce task per node Imposed a limitation on the aggregated memory usage of individual jobs.Tsz-Wo Sze, Hadoop Summit 2011 48

Running Time Actual running time for 236 ≤ N ≤ 240. 11.5 t is the elapsed time in seconds 11 10.5 10 9.5 log(t) 9 8.5 8 7.5 7 32 33 34 35 36 37 38 39 40 log(N)Tsz-Wo Sze, Hadoop Summit 2011 49

What is π? π is a mathematical constant such that, for any circle, circumference C π= = . diameter dTsz-Wo Sze, Hadoop Summit 2011 51

What is π? π is a mathematical constant such that, for any circle, circumference C π= = . diameter d We have π = 3.244Tsz-Wo Sze, Hadoop Summit 2011 52

What is π? π is a mathematical constant such that, for any circle, circumference C π= = . diameter d We have π = 3.244 (in hexadecimal )Tsz-Wo Sze, Hadoop Summit 2011 53

Decimal, Hexadecimal & Binary Representing π in diﬀerent bases π = 3.1415926535 8979323846 2643383279 ... = 3.243F6A88 85A308D3 13198A2E ... = 11.00100100 00111111 01101010 ... Bit position is counted after the radix point. e.g., the eight bits starting at the ninth bit position are 00111111 in binary or 3F in hexadecimal.Tsz-Wo Sze, Hadoop Summit 2011 54

A New World Record Yahoo! Cloud Computing (July 2010) • Machines: Idle slices of 1000-node clusters Each node has two quad-core 1.8-2.5 GHz CPUs • Duration: 23 days • CPU time: 503 years • Veriﬁcation: 582 years CPU timeTsz-Wo Sze, Hadoop Summit 2011 55

A New World Record Bit values (in hexadecimal) 0E6C1294 AED40403 F56D2D76 4026265B CA98511D 0FCFFAA1 0F4D28B1 BB5392B8Tsz-Wo Sze, Hadoop Summit 2011 56

A New World Record Bit values (in hexadecimal) 0E6C1294 AED40403 F56D2D76 4026265B CA98511D 0FCFFAA1 0F4D28B1 BB5392B8 (256 bits) The ﬁrst bit position: 1,999,999,999,999,997 (= 2 · 1015 − 3) The last bit position: 2,000,000,000,000,252 (= 2·1015 +252) The two quadrillionth (2 · 1015th) bit is 0.Tsz-Wo Sze, Hadoop Summit 2011 57

BBC News (16 Sep 2010) Pi record smashed as team ﬁnds two-quadrillionth digit http://www.bbc.co.uk/news/technology-11313194Tsz-Wo Sze, Hadoop Summit 2011 58

NewScientist (17 Sep 2010) New pi record exploits Yahoo’s computers http://www.newscientist.com/article/dn19465-new-pi-record-exploits-yahoos-com htmlTsz-Wo Sze, Hadoop Summit 2011 59

Other News Coverage New Pi Record Exploits Yahoo’s Computers http://cacm.acm.org/news/99207-new-pi-record-exploits-yahoos-computers The Yahoo! boﬃn scores pi’s two quadrillionth bit http://www.theregister.co.uk/2010/09/16/pi_record_at_yahoo Pi calculation more than doubles old record http://www.radionz.co.nz/news/world/57128/pi-calculation-more-than-doubles-ol Hadoop used to calculate Pi’s two quadrillionth bit http://www.zdnet.co.uk/blogs/mapping-babel-10017967/hadoop-used-to-calculate-Tsz-Wo Sze, Hadoop Summit 2011 60

Yahoo! researcher breaks Pi record in ﬁnding the two-quadrillionth digit http://www.engadget.com/2010/09/17/yahoo-researcher-breaks-pi-record-in-findi Nicholas Sze of Yahoo Finds Two-Quadrillionth Digit of Pi http://science.slashdot.org/story/10/09/16/2155227/Nicholas-Sze-of-Yahoo-Find The 2,000,000,000,000,000th digit of the mathematical constant pi discovered http://news.gather.com/viewArticle.action?articleId=281474978525563 Researcher Shatters Pi Record by Finding Two-Quadrillionth Digit http://www.maximumpc.com/article/news/researcher_shatters_pi_record_finding_ two-quadrillionth_digitTsz-Wo Sze, Hadoop Summit 2011 61

A bigger slice of pi http://radar.oreilly.com/2010/09/strata-week-grabbing-a-slice.html 2 Quadrillionth digit of PI is found: Scientist celebration in worldwide Pandemonium http://engforum.pravda.ru/showthread.php?296242-2-Quadrillionth-digit-of-PI-i And the number is...0 http://www.hexus.net/content/item.php?item=26505 Pi Record Smashed as Team Finds Two- Quadrillionth Digit http://hardocp.com/news/2010/09/16/pi_record_smashed_as_team_finds_twoquadril digitTsz-Wo Sze, Hadoop Summit 2011 62

Yahoo Engineer Calculates Two Quadrillionth Bit Of Pi http://www.webpronews.com/topnews/2010/09/17/yahoo-engineer-calculates-two-qu A Cloud Computing Milestone: Yahoo! Reaches the 2 Quadrillionth Bit of Pi http://www.readwriteweb.com/cloud/2010/09/a-cloud-computing-milestone-ya. php Yahoo researcher Nicolas Sze determines the 2,000,000,000,000,000th digit of the mathematical constant pi http://www.thaindian.com/newsportal/sci-tech/yahoo-researcher-nicolas-sze-det 100430278.html ...Tsz-Wo Sze, Hadoop Summit 2011 63

Computing π How to compute the nth bits of π?Tsz-Wo Sze, Hadoop Summit 2011 64

Computing π How to compute the nth bits of π? Let’s ignore this question in this talk ... and focus on:Tsz-Wo Sze, Hadoop Summit 2011 65

Computing π How to compute the nth bits of π? Let’s ignore this question in this talk ... and focus on: How to execute such huge computation?Tsz-Wo Sze, Hadoop Summit 2011 66

Map- & Reduce-side Computations Developed a generic framework to execute tasks on either the map-side or the reduce-side. Applications deﬁne two functions: • partition(c, m): partition the computation c into m parts. • compute(c): execute the computation cTsz-Wo Sze, Hadoop Summit 2011 67

Map-side Job Contains multiple mappers and zero reducers • A PartitionInputFormat partitions c into m parts • Each part is executed by a mapperTsz-Wo Sze, Hadoop Summit 2011 68

Reduce-side Job Contains a mapper and multiple reducers • A SingletonInputFormat launches a PartitionMapper • An Indexer launches m reducers.Tsz-Wo Sze, Hadoop Summit 2011 69

Abstract Machine (1) Machine – an abstract base class allows abstract Runner(s) to execute MachineComputable tasks. Machine subclasses • Map Side Machine m100t3: 100 maps with 3 threads each. • Reduce Side Machine r50t2: 50 reduces with 2 threads each.Tsz-Wo Sze, Hadoop Summit 2011 70

Abstract Machine (2) More Machine subclasses • Mix Machine – chooses Map-/Reduce-side jobs according to the cluster status. x-m200t1-r100t2-5: either launch a job with 200 maps with 1 thread each; or a job with 100 reduces with 2 thread each. • Alternation Machine – alternates Map-side and Reduce-side jobs in a regular pattern. a-m200t1-r100t2-mrr: submit a map job, then a reduce job, then another reduce job and repeat this pattern. • Null Machine – does nothing for testing.Tsz-Wo Sze, Hadoop Summit 2011 71

Utilizing The Idle Slices Monitor cluster status • Submit a map-side (or reduce-side) job if there are suﬃcient available map (or reduce) slots. Small jobs • Hold resource only for a short period of time Interruptible & resumable • can be interrupted at any time by simply killing the running jobsTsz-Wo Sze, Hadoop Summit 2011 72

Running The JobsTsz-Wo Sze, Hadoop Summit 2011 73

The Implementation Main programs: DistBbp – a program to submit jobs. DistSum – distributed summation. Open source – available at https://issues.apache.org/jira/browse/MAPREDUCE-1923Tsz-Wo Sze, Hadoop Summit 2011 74

The World Record Computation 35,000 MapReduce jobs, each job either has: • 200 map tasks with one thread each, or • 100 reduce tasks with two threads each. Each thread computes 200,000,000 terms • ∼45 minutes. Submit up to 60 concurrent jobs The entire computation took: • 23 days of real time and 503 CPU yearsTsz-Wo Sze, Hadoop Summit 2011 75

Referneces • [1] Tsz-Wo Sze. Sch¨nhage-Strassen Algorithm with MapReduce for Mul- o tiplying Terabit Integers. Symbolic-Numeric Computation 2011, to ap- pear. Preprint available at http://people.apache.org/~szetszwo/ ssmr20110430.pdf • [2] Tsz-Wo Sze. The Two Quadrillionth Bit of Pi is 0! Distributed Computation of Pi with Apache Hadoop. In IEEE 2nd International Conference on Cloud Computing Technology and Science (CloudCom), pages 727-732, 2010. (Earlier versions available at http://arxiv.org/ abs/1008.3171)Tsz-Wo Sze, Hadoop Summit 2011 76

Thank you!Tsz-Wo Sze, Hadoop Summit 2011 77

http://abjkk.posterous.com	19
http://paper.li	6
http://us-w1.rockmelt.com	2
http://twitter.com	2
http://www.slideshare.net	1
http://posterous.com	1
https://twitter.com	1
http://www.techgig.com	1
http://tweetedtimes.com	1
https://blackboardtest.sscc.edu	1

Large Scale Math with Hadoop MapReduce

by Hortonworks on Jul 02, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

10 Embeds 35

Statistics

Large Scale Math with Hadoop MapReduce — Presentation Transcript