The document discusses the limitations of traditional Hadoop and the innovative solutions offered by MapR, including features such as snapshots and cluster-wide file access through NFS. It emphasizes the benefits of using Singular Value Decomposition (SVD) in combination with a hybrid architecture for efficient feature extraction and visualization in R. Overall, it highlights the potential for scalability and interoperability in data processing, particularly for large datasets.

1. RHadoop and MapR

2. The bad old days (i.e. now)Hadoop is a siloHDFS isn’t a normal file systemHadoop doesn’t really like C++R is limitedOne machine, one memory spaceIsn’t there any way we can just get along?

3. The white knightMapR changes thingsLots of new stuff like snapshots, NFSAll you need to know, you already knowNFS provides cluster wide file accessEverything works the way you expectPerformance high enough to use as a message bus

4. Example, out-of-core SVDSVD provides compressed matrix formBased on sum of rank-1 matrices≈++ ?±±

5. More on SVDSVD provides a very nice basis

6. And a nifty approximation property

7. Also known as …Latent Semantic IndexingPCAEigenvectors

8. An application, approximate translationTranslation distributes over concatenationBut counting turns concatenation into additionThis means that translation is linear!ish

9. ish

10. Traditional computationProducts of A are dominated by large singular values and corresponding vectorsSubtracting these dominate singular values allows the next ones to appearLanczos method, generally Krylov sub-space

11. But …

12. The gotchaIteration in Hadoop is deathHuge process invocation costsLose all memory residency of dataTotal lost cause

13. Randomness to the rescueTo save the day, run all iterations at the same time==A

14. In Rlsa = function(a, k, p) { n = dim(a)[1] m = dim(a)[2] y = a %*% matrix(rnorm(m*(k+p)), nrow=m)y.qr = qr(y) b = t(qr.Q(y.qr)) %*% ab.qr = qr(t(b))svd = svd(t(qr.R(b.qr))) list(u=qr.Q(y.qr) %*% svd$u[,1:k], d=svd$d[1:k], v=qr.Q(b.qr) %*% svd$v[,1:k])}

15. Not good enough yetLimited to memory sizeAfter memory limits, feature extraction dominates

16. Hybrid architectureMap-reduceSide-dataVia NFSFeatureextractionanddownsamplingInputDatajoinSequentialSVD

17. Hybrid architectureMap-reduceSide-dataVia NFSFeatureextractionanddownsamplingInputDatajoinRVisualizationSequentialSVD

18. Randomness to the rescueTo save the day again, use blocks===

19. Hybrid architectureMap-reduceFeature extractionanddown samplingVia NFSMap-reduceRVisualizationBlock-wiseparallelSVD

20. ConclusionsInter-operability allows massively scalabilityPrototyping in R not wastedMap-reduce iteration not needed for SVDFeasible scale ~10^9 non-zeros or more