Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache Samza - New features in the upcoming Samza release 0.10.0

2,552 views

Published on

These slides were presented as a part of the Bay Area Samza Meetup on October 13, 2015.

Published in: Engineering
  • Be the first to comment

Apache Samza - New features in the upcoming Samza release 0.10.0

  1. 1. Apache Samza 0.10.0 What’s coming up in the next Samza release LinkedIn Navina R Committer @ Apache Samza
  2. 2. New Features in Samza 0.10.0  Dynamic Configuration & Control ◦ Coordinator Stream ◦ Broadcast Stream  Host affinity in Samza  New Consumer: Kinesis  New Producers: Kinesis, HDFS, ElasticSearch  Upgraded RocksDB
  3. 3. Dynamic Configuration & Control 1. Coordinator Stream 2. Broadcast Stream
  4. 4. How does Config work today? Job Config RM AM C0 C1 C2 Submit Job Cfg via cmd line Cfg via cmd line Job deployment in Yarn:  Job is localized to the Resource Manager (RM)  RM allocates a container for the Application Master (AM) and passes the config parameters as command- line arguments to the run- am script  Similarly, AM passes config to the containers on allocation Checkpoint Stream
  5. 5. Problems Job Config RM AM C0 C1 C2 Submit Job Cfg via cmd line Cfg via cmd line  Escaping / Unescaping quotes is cumbersome (SAMZA-700)  Limits the number of arguments that can be set through shell command line (SAMZA-337, SAMZA-333)  Dynamic config change not possible. Every config change requires a job re- submission (restart) (SAMZA-348)  Handle system config like checkpoints differently than user-defined config (SAMZA- 348)Checkpoint Stream
  6. 6. Solution: Coordinator Stream RM AM C0 C1 C2 Submit Job JC Coordinator Stream Config requested via HTTP Coordinator Stream (CS)  Single partition  Log-compacted  Each job has its own CS Job Coordinator (JC)  Exposes HTTP end- point for containers to query for Job Model  Bootstraps from CS and then, continues consumption from CS Samza job deployment using Job Coordinator & Coorindator Stream Bootstraps config from stream
  7. 7. Data in Coordinator Stream Coordinator Stream (CS) contains: ◦ Checkpoints for the input streams  Containers periodically write to checkpoints to CS, instead of a separate checkpoint topic ◦ Task-to-changelog partition mapping ◦ Container Locality Info (required for Host Affinity)  Containers write their location (machine-name) to CS ◦ User-defined configuration  Entire configuration is written to the CS when the job is started ◦ Migration related messages
  8. 8. Coordinator Stream: Benefits RM AM C0 C1 C2 Submit Job JC Coordinator Stream Config requested via HTTP  Config can be easily serialized / deserialized  Checkpoints & user- defined configs are stored similarly  Config change can be made by writing to the CS*  JC can be used to coordinate job execution* * Work In Progress Samza job deployment using Job Coordinator & Coorindator Stream Bootstraps config from stream
  9. 9. Coordinator Stream: Tools / Migration Tools:  Command-line tool to write config changes to coordinator stream Migration:  JobRunner in 0.10.0 automatically migrates checkpoints and changelog mappings in 0.9.1 to Coordinator Stream in 0.10.0
  10. 10. Broadcast Stream Stream consumed by all Tasks in the job
  11. 11. Motivation  Dynamically configure job behavior  Acts a custom control channel for an application
  12. 12. A typical input stream Task- 0 Task- 1 Task- 2 Task- 3 MyInputStream Partition-0 Partition-1 Partition-2 Partition-3 task.inputs = $system-name.$stream-name task.inputs = kafka.MyInputStream One stream partition consumed only by one task
  13. 13. Broadcast Stream Task- 0 Task- 1 Task- 2 Task- 3 MyInputStream Partition-0 Partition-1 Partition-2 Partition-3 MyBroadcastStream Partition-0 task.inputs = $system-name.$stream-name task.global.inputs = $system-name.$stream-name#$partition- number task.inputs = kafka.MyInputStream task.global.inputs = kafka.MyBroadcastStream#0 One stream partition consumed only by ALL partitions
  14. 14. Broadcast Stream Task- 0 Task- 1 Task- 2 Task- 3 MyInputStream Partition-0 Partition-1 Partition-2 Partition-3 MyBroadcastStream Partition-0 Partition-1 Partition-2 task.inputs = $system-name.$stream-name task.global.inputs = $system-name.$stream-name#[$partition- range] task.inputs = kafka.MyInputStream task.global.inputs = kafka.MyBroadcastStream#[0-1]
  15. 15. Host-Affinity Making Samza aware of container locality
  16. 16. A Stateful Job P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-A Host-B Host-C Changelog Stream Stable State
  17. 17. Fault Tolerance in a Stateful Job P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-A Host-B Host-C Changelog Stream Task-0 & Task-1 running on the container in Host-A fail
  18. 18. Fault Tolerance in a Stateful Job P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Changelog Stream Yarn allocates the tasks to a container on a different host!
  19. 19. Fault Tolerance in a Stateful Job P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C : : 0 1 159 : : 0 1 82 Local state restored by consuming the changelog from the earliest offset!
  20. 20. Fault Tolerance in a Stateful Job P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C After restored, job continues with input processing – Back to Stable State!
  21. 21. Problems P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C  State stores are not persisted if the container fails ◦ Tasks need to restore the state stores from the change-log before continuing with input processing  Samza AppMaster is not aware of host locality for a container ◦ Container gets relocated to a new host  Excessive start-up times when a job is restarted
  22. 22. Motivation  During upgrades and job failures, ◦ Local state built in the task is lost ◦ Samza is not aware of the container locality ◦ Job start-up time is large (hours)  Job is no longer “near-realtime”  Multiple stateful jobs starting up at the same time will DDoS kafka – saturating the Kafka clusters
  23. 23. Solution: Host Affinity in Samza  Host Affinity – ability of Samza to allocate a container to the same machine across job restarts/deployments  Host affinity is best-effort ◦ Cluster load may vary ◦ Machine may be non-responsive ◦ Container should shutdown cleanly
  24. 24. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Coordinator Stream Container-0 -> Host-E Container-1 -> Host-B Container-2 -> Host-C Persist container locality in Coordinator Stream
  25. 25. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Coordinator Stream Container-0 -> Host-E Container-1 -> Host-B Container-2 -> Host-C Task-0 & Task-1 running on the container in Host-E fail
  26. 26. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Coordinator Stream Container-0 -> Host-E Container-1 -> Host-B Container-2 -> Host-C AM JC Tasks failed, but local state stores remain!
  27. 27. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Coordinator Stream Container-0 -> Host-E Container-1 -> Host-B Container-2 -> Host-C AM JC RM Ask: Host-E Allocate: Host-E Job Coordinator is aware of container locality!
  28. 28. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C : : 0 1 159 : : 0 1 82 Coordinator Stream Container-0 -> Host-E Container-1 -> Host-B Container-2 -> Host-C Container-0 -> Host-E State store does not have to be restored from the earliest offset!
  29. 29. Host Affinity in Samza P0 P1 P2 P3 Task-0 Task-1 Task-2 Task-3 P0 P1 P2 P3 Host-E Host-B Host-C Job back to Stable state pretty quickly!
  30. 30. Host Affinity in Samza  Enable host-affinity ◦ yarn.samza.host-affinity.enabled=true  Enable continuous scheduling in Yarn  Useful for stateful jobs  Does not affect stateless jobs
  31. 31. Upgraded RocksDB
  32. 32. Upgraded RocksDB  New RocksDb JNI 3.13.1+ version supports TTL  Impact: ◦ Removes the need to write customized code to delete expired records
  33. 33. New Features in Samza 0.10.0  Dynamic Configuration & Control ◦ Coordinator Stream ◦ Broadcast Stream  Host affinity in Samza  New Consumer: Kinesis  New Producers: Kinesis, HDFS, ElasticSearch  Upgraded RocksDB
  34. 34. Thanks!  Expected release date – Nov 2015  Thanks to all the contributors!  Contact Us: ◦ Mailing List – dev@samza.apache.org ◦ Twitter - #samza, @samzastream
  35. 35. Questions?

×