Beam Me Up: Voyaging into Big Data
•
0 likes•384 views
More engineering organizations than ever are dealing with big data. The long times required to process big datasets slow down development cycles and delay analysis. Apache Beam pipelines distribute processing across many workers, reducing the time it takes to transform large datasets. Creating an effective Beam pipeline requires following best practices and using the specialized data structures Beam introduces. In this talk, I’ll share strategies and lessons learned from scaling Apache Beam pipelines to handle ever-increasing workloads.
Recommended
Recommended
More Related Content
Similar to Beam Me Up: Voyaging into Big Data
Similar to Beam Me Up: Voyaging into Big Data (20)
More from Michele Titolo
More from Michele Titolo (20)
Beam Me Up: Voyaging into Big Data
- 1. Beam Me Up: Voyaging Into Big Data Michele Titolo Senior Software Engineer, Square @micheletitolo
- 2. @micheletitolo Big Data Is All Around Us
- 3. @micheletitolo Apache Beam Is One Of The Available Tools
- 7. @micheletitolo ‣ What is Beam ‣ How to build a pipeline ‣ Tips and Gotchas What We Will Cover
- 8. What Is Beam
- 12. Time 1 2 3
- 14. @micheletitolo Sits On Top Of Or Adjacent To Other Tools
- 21. @micheletitolo
- 22. @micheletitolo
- 25. @micheletitolo 1 2 3 0 0> 1
- 26. @micheletitolo 1 2 3 > 1
- 27. @micheletitolo 1 2 3 > 1
- 30. @micheletitolo Pipelines Are Defined Solely In Code
- 35. @micheletitolo Use A Small Dataset To Test
- 36. @micheletitolo Run And Test Locally
- 39. @micheletitolo PCollections Are Kind Of Like Arrays
- 43. @micheletitolo Inputs And Outputs Must Serialize To Disk
- 44. @micheletitolo KV: one key hash
- 49. @micheletitolo Most Of The Code Is In These
- 51. @micheletitolo Process PCollection 1 Element at a Time
- 60. @micheletitolo 1 2 3 PCollection DoFn PCollection PCollection PCollection Pipeline https://beam.apache.org/get-started/wordcount-example/ DoFn DoFn
- 61. Tips And Gotchas
- 67. @micheletitolo A B C3 B C2 Time A B C1 A
- 68. @micheletitolo 3 B C2 Time A B C1 A RESHUFFLE D E F
- 73. 1 2 3
- 74. 1 2 3
- 75. 1 2
- 78. @micheletitolo : { } : { }: { } : { } : { } Partition : { }{ }: : { }{ }: : { }{ }: : { }{ }: : { }{ }:
- 80. @micheletitolo Intermediate State Goes Away After Finish
- 81. @micheletitolo
- 83. @micheletitolo Multiple Of The Same Pipeline Can Be Running
- 84. In Summary
- 85. @micheletitolo Beam Is A General Purpose Tool
- 87. @micheletitolo Easy To Get Started
- 90. • https://unsplash.com/photos/MShiKyjGhck • https://unsplash.com/photos/DByY8MbE9OE • https://unsplash.com/photos/fR47SivxkSM • https://unsplash.com/photos/m3TYLFI_mDo Photo Credits