Downloaded 74 times
Uploaded byGreg Silin
Stream Collections - Scala Days
This document discusses using streaming collections to process large amounts of data stored in Amazon S3. It describes how Nitro uses Play Iteratees to build asynchronous streams for operations like counting, extracting data, and cleanup. These streams are then abstracted as Scala collections for simple operations like map, filter, and count. Examples are given of using streams to clean files and extract data by date. The benefits of this approach for processing billions of objects across many documents are discussed.
More Related Content
Similar to Stream Collections - Scala Days
Stream Collections - Scala Days
- 1. Streams as ScalaCollections S3 Scala Client with Play Iteratees and Composable Operations Greg Silin Platform Engineer @gregsbriefs www.github.com/nitro/streamcollections ScalaDays 2015
- 2. Agenda • Reactive atNitro • Smart Documents at Scale • Motivation for Streaming Collections • Building Streams with Iteratees • Streams as Scala Collections • Applications • Questions
- 3. The Old Way Create& Prepare On the Desktop Print Document Sign Printed Document Scan Into Computer
- 4. Knowledge workers spendapproximately 11+ hours a week creating and managing documents The New Way Create Prepare Sign (Anywhere)
- 5. Nitro accelerates theway businesses create, prepare, and sign documents. Anytime and anywhere. Smarter Documents for EveryoneTM
- 6. Reactive Systems atNitro react to user expectations <- responsive react to state changes <- message driven react to variable load <- elastic react to failure <- resilient
- 7. Smart Documents atScale multiple pages and formats per document
- 8. Smart Documents atScale Each action results in a new document version render sign approve ...
- 9. Smart Documents atScale documents / second * versions / document * pages / version = billions of objects in S3
- 10. Smart Documents atScale millions of new document uploads a day 100MM+/day document state changes resulting in 10x messages billions of objects in S3
- 11. Motivation for StreamingCollections counting copying extracting cleanup become non-trivial at scale
- 12. Motivation for StreamingCollections 1 percent error margin = 10M objects That’s money for the business
- 13. How? How do wetraverse the data?
- 14. How? Command line toolsdon’t provide flexibility / scale
- 15. How? Can’t load everythingin memory Command line tools don’t provide flexibility / scale
- 16. How? Can’t load everythingin memory Need some batched solution Command line tools don’t provide flexibility / scale
- 17. How? Amazon S3 SDKhas a Java key iterator
- 18. How? Amazon S3 SDKhas a Java key iterator
- 19. How? ... Amazon S3 SDKhas a Java key iterator
- 20. But we areScala engineers! How?
- 21. How? Streaming is anatural fit Amazon SDK has a Java key iterator
- 22. How? Streaming is anatural fit We are reactive Amazon SDK has a Java key iterator
- 23. How? Streaming is anatural fit Amazon SDK has a Java key iterator Thus asynchronous streams We are reactive
- 24. How? Streaming is anatural fit Amazon SDK has a Java key iterator Thus asynchronous streams We are reactive Can’t over-parallelize
- 25. What Streams? Enter PlayIteratees Enumerator - Source Enumeratee - Transformer Iteratee - Consumer / Sink
- 26. Building Streams withIteratees Why Play Iteratees?
- 27. Building Streams withIteratees Why Play Iteratees? Most mature technology at the time
- 28. Building Streams withIteratees Why Play Iteratees? Most mature technology at the time Production Experience
- 29. Building Streams withIteratees Play Iteratees via a counting example
- 30. Building Streams withIteratees Enumerator = Source
- 31. Building Streams withIteratees Enumeratee = Transformer
- 32. Building Streams withIteratees Iteratee = Sink / Reduce
- 33. Building Streams withIteratees Tying things together...
- 34. Building Streams withIteratees Can this be simplified?
- 35. Streams as ScalaCollections We are all familiar with Scala collections
- 36. Streams as ScalaCollections We are all familiar with Scala collections map filter foreach grouped count
- 37. Streams as ScalaCollections Can reason about iteratee streams as a collection
- 38. Streams as ScalaCollections Can now redo our grouped & count example
- 39. Streams as ScalaCollections Can now redo our grouped & count example
- 40. Streams as ScalaCollections With the internals hidden, my counting code becomes simple
- 41. Streams as ScalaCollections - Examples Cleaning up files
- 42. Streams as ScalaCollections - Examples Extract data by date
- 43. Streams as ScalaCollections - Applications Can extend this model onto other data sources We don’t have to stop at S3 ➔ Relational DB ➔ ElasticSearch ➔ HBase / Cassandra ➔ Spark
- 44. "Much of mywork has come from being lazy." - John Backus Quoted in the IBM employee magazine Think in 1979 (http://en.wikiquote.org/wiki/John_Backus)
- 45. What We Learned Iterateesare good for traversing large volume of data Programming iteratees can get a bit tricky Scaling ain’t easy Stream Collections abstraction makes streams simple
- 46. Future of Streamsas Scala Collections Continue developing a reactive S3 Client In use in Nitro Production Introduce other stream implementations (akka streams, etc)
- 47. www.github.com/nitro/streamcollections Contributors: www.github.com/gregsilin / @gregsbriefs www.github.com/mkolod/ @marekinfo Open Sourcing Are you interested? We welcome collaborators!
- 48. San Francisco ScalaDays 2015 • Nitro is a Gold sponsor • Meet us at our community booth sfscala.org: • Wed: Scala D’Ehs meetup @ Stock in Trade • Thu: unconference @ Galvanize • Thu evening: Spark Notebook & Rapture @ Nitro • Fri: free Shapeless training @ Nitro
- 49.
- 50.