Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2017: Tzu-Li (Gordon) Tai - Managing State in Apache Flink

711 views

Published on

Over the past year, we've seen users build entire event-driven applications such as social networks on top of Apache Flink (Drivetribe.com), elevating the importance of state management in Flink to a whole new level. Users are placing more and more data as state in Flink, using it as a replacement to conventional databases. With such mission-critical data entrusted to Flink, we need to provide similar capabilities of a database to users. One of such capabilities is being flexible in how data is persisted and represented. Specifically, how can I change how my state is serialized and stored, or even the schema of my state, as business logic changes over time? In this talk, we'll provide details on the latest state management features in Flink that allows users to do exactly that. We'll talk about how Flink manages the state for you, how it provides flexibility to the user to adapt to evolving state serialization formats and schemas, and the best practices when working with it.

Published in: Data & Analytics

Flink Forward Berlin 2017: Tzu-Li (Gordon) Tai - Managing State in Apache Flink

  1. 1. Tzu-Li (Gordon) Tai Flink Committer / PMC Member tzulitai@apache.org @tzulitai Managing State in Apache Flink
  2. 2. Original creators of Apache Flink® Providers of dA Platform 2, including open source Apache Flink + dA Application Manager
  3. 3. Users are placing more and more of their most valuable assets within Flink: their application data
  4. 4. @ Complete social network implemented using event sourcing and CQRS (Command Query Responsibility Segregation)
  5. 5. Classic tiered architecture Streaming architecture database layer compute layer compute + stream storage and snapshot storage (backup) application state
  6. 6. Streams and Snapshots 6 Version control Exactly-once guarantees Job reconfiguration Asynchronous, pipelined State Snapshots
  7. 7. … want to ensure that users can fully entrust Flink with their state, as well as good quality of life with state management
  8. 8. State management FAQs ... ● State declaration best practices? ● How is my state serialized and persisted? ● Can I adapt how my state is serialized? ● Can I adapt the schema / data model of my state? 8
  9. 9. Recap: Flink’s Managed State 9
  10. 10. Flink Managed State 10 ... ... Your Code Your Code ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>( “my-value-state”, MyPojo.class ); ValueState<MyPojo> state = getRuntimeContext().getState(desc); MyPojo p = state.value(); ... state.update(...) ...
  11. 11. Flink Managed State 11 ... ... Your Code Your Code ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>( “my-value-state”, MyPojo.class ); ValueState<MyPojo> state = getRuntimeContext().getState(desc); MyPojo p = state.value(); ... state.update(...) ... unique identifier for this managed state
  12. 12. Flink Managed State 12 ... ... Your Code Your Code ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>( “my-value-state”, MyPojo.class ); ValueState<MyPojo> state = getRuntimeContext().getState(desc); MyPojo p = state.value(); ... state.update(...) ... type information for the state
  13. 13. Flink Managed State 13 ... ... Your Code Your Code ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>( “my-value-state”, MyPojo.class ); ValueState<MyPojo> state = getRuntimeContext().getState(desc); MyPojo p = state.value(); ... state.update(...) ... Local State Backend registers state
  14. 14. Flink Managed State 14 ... ... Your Code Your Code ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>( “my-value-state”, MyPojo.class ); ValueState<MyPojo> state = getRuntimeContext().getState(desc); MyPojo p = state.value(); ... state.update(...) ... Local State Backend read write
  15. 15. Flink Managed State (II) 15 State State State State State State State DFS Checkpoints - Pipelined checkpoint barriers flow through the topology - Operators asynchronously backup their state in checkpoints on a DFS checkpointed state
  16. 16. Flink Managed State (II) 16 State State State State State State State DFS Restore - Each operator is assigned their corresponding file handles, from which they restore their state checkpointed state
  17. 17. State Serialization Behaviours 17 State State State State State State State DFS JVM Heap backed state backends (MemoryStateBackend, FsStateBackend) ⇒ lazy serialization + eager deserialization checkpointed state
  18. 18. State Serialization Behaviours 18 State State State State State State State DFS JVM Heap backed state backends (MemoryStateBackend, FsStateBackend) ⇒ lazy serialization + eager deserialization Java object read / writes checkpointed state
  19. 19. State Serialization Behaviours 19 State State State State State State State DFS JVM Heap backed state backends (MemoryStateBackend, FsStateBackend) ⇒ lazy serialization + eager deserialization Java object read / writes serialize on chckpts / savepoints Checkpoint checkpointed state
  20. 20. State Serialization Behaviours 20 State State State State State State State DFS JVM Heap backed state backends (MemoryStateBackend, FsStateBackend) ⇒ lazy serialization + eager deserialization deserialize to objects on restore Restore checkpointed state
  21. 21. State Serialization Behaviours 21 State State State State State State State DFS Out-of-core state backends (RocksDBStateBackend) ⇒ eager serialization + lazy deserialization checkpointed state
  22. 22. State Serialization Behaviours 22 State State State State State State State DFS Out-of-core state backends (RocksDBStateBackend) ⇒ eager serialization + lazy deserialization De-/serialize on every local read / write checkpointed state
  23. 23. State Serialization Behaviours 23 State State State State State State State DFS Out-of-core state backends (RocksDBStateBackend) ⇒ eager serialization + lazy deserialization De-/serialize on every local read / write exchange of state bytes Checkpoint checkpointed state
  24. 24. State Serialization Behaviours 24 State State State State State State State DFS Out-of-core state backends (RocksDBStateBackend) ⇒ eager serialization + lazy deserialization De-/serialize on every local read / write exchange of state bytes Restore checkpointed state
  25. 25. State Declaration 25 Matters of State (I)
  26. 26. State Access Optimizations 26 ValueStateDescriptor<Map<String, MyPojo>> desc = new ValueStateDescriptor<>(“my-value-state”, MyPojo.class); ValueState<Map<String, MyPojo>> state = getRuntimeContext().getState(desc); ● Flink supports different state “structures” for efficient state access for various patterns Map<String, MyPojo> map = state.value(); map.put(“someKey”, new MyPojo(...)); state.update(map); X don’t do
  27. 27. State Access Optimizations 27 MapStateDescriptor<String, MyPojo> desc = new MapStateDescriptor<>(“my-value-state”, String.class, MyPojo.class); MapState<String, MyPojo> state = getRuntimeContext().getMapState(desc); ● Flink supports different state “structures” for efficient state access for various patterns MyPojo pojoVal = state.get(“someKey”); state.put(“someKey”, new MyPojo(...)); optimized for random key access
  28. 28. Declaration Timeliness 28 ● Try registering state as soon as possible • Typically, all state declaration can be done in the “open()” lifecycle of operators
  29. 29. Declaration Timeliness 29 public class MyStatefulMapFunction extends RichMapFunction<String, String> { private static ValueStateDescriptor<MyPojo> DESC = new ValueStateDescriptor<>(“my-pojo-state”, MyPojo.class); @Override public String map(String input) { MyPojo pojo = getRuntimeContext().getState(DESC).value(); … } }
  30. 30. Declaration Timeliness 30 public class MyStatefulMapFunction extends RichMapFunction<String, String> { private static ValueStateDescriptor<MyPojo> DESC = new ValueStateDescriptor<>(“my-pojo-state”, MyPojo.class); private ValueState<MyPojo> state; @Override public void open(Configuration config) { state = getRuntimeContext().getState(DESC); } @Override public String map(String input) { MyPojo pojo = state.value(); … } }
  31. 31. Eager State Declaration 31 ● Under discussion with FLIP-22 ● Allows the JobManager to have knowledge on declared states of a job (under discussion, release TBD)
  32. 32. Eager State Declaration 32 (under discussion, release TBD) public class MyStatefulMapFunction extends RichMapFunction<String, String> { @KeyedState( stateId = “my-pojo-state”, queryableStateName = “state-query-handle” ) private MapState<String, MyPojo> state; @Override public String map(String input) { MyPojo pojo = state.get(“someKey”); state.put(“someKey”, new MyPojo(...)); } }
  33. 33. State Serialization 33 Matters of State (II)
  34. 34. How is my state serialized? 34 ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>(“my-value-state”, MyPojo.class); ● Provided state type class info is analyzed by Flink’s own serialization stack, producing a tailored, efficient serializer ● Supported types: • Primitive types • Tuples, Scala Case Classes • POJOs (plain old Java objects) ● All non-supported types fallback to using Kryo for serialization
  35. 35. Avoid Kryo for State Serde 35 ● Kryo is generally not recommended for use on persisted data • Unstable binary formats • Unfriendly for data model changes to the state ● Serialization frameworks with schema evolution support is recommended: Avro, Thrift, etc. ● Register custom Kryo serializers for your Flink job that uses these frameworks
  36. 36. Avoid Kryo for State Serde 36 StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // register the serializer included with // Apache Thrift as the standard serializer for your type env.registerTypeWithKryoSerializer(MyCustomType, TBaseSerializer.class); ● Also see: https://goo.gl/oU9NxJ
  37. 37. Custom State Serialization 37 ● Instead of supplying type information to be analyzed, directly provide a TypeSerializer public class MyCustomTypeSerializer extends TypeSerializer<MyCustomType> { ... } ValueStateDescriptor<MyCustomType> desc = new ValueStateDescriptor<>(“my-value-state”, new MyCustomTypeSerializer());
  38. 38. State Migration 38 Matters of State (III)
  39. 39. State migration / evolution 39 ● Upgrading to more efficient serialization schemes for performance improvements ● Changing the schema / data model of state types, due to evolving business logic
  40. 40. Upgrading State Serializers 40 ● The new upgraded serializer would either be compatible, or not compatible. If incompatible, state migration is required. ● Disclaimer: as of Flink 1.3, upgraded serializers must be compatible, as state migration is not yet an available feature. ValueStateDescriptor<MyCustomType> desc = new ValueStateDescriptor<>(“my-value-state”, new UpgradedSerializer<MyCustomType>()); ValueStateDescriptor<MyPojo> desc = new ValueStateDescriptor<>(“my-value-state”, MyPojo.class); // modified MyPojo type Case #2: New custom serializer Case #1: Modified state types, resulting in different Flink-generated serializers
  41. 41. Upgrading State Serializers (II) 41 ● On restore, new serializers are checked against the metadata of the previous serializer (stored together with state data in savepoints) for compatibility. ● All Flink-generated serializers define the metadata to be persisted. Newly generated serializers at restore time are reconfigured to be compatible.Savepoint state data serializer config metadataField “A” Field “B” Old serializer at time of savepoint Field “B” Field “A” New serializer at restore time
  42. 42. Upgrading State Serializers (II) 42 ● On restore, new serializers are checked against the metadata of the previous serializer (stored together with state data in savepoints) for compatibility. ● All Flink-generated serializers define the metadata to be persisted. Newly generated serializers at restore time are reconfigured to be compatible.Savepoint state data serializer config metadataField “A” Field “B” Old serializer at time of savepoint Field “B” Field “A” New serializer at restore time reconfigure
  43. 43. Upgrading State Serializers (II) 43 ● On restore, new serializers are checked against the metadata of the previous serializer (stored together with state data in savepoints) for compatibility. ● All Flink-generated serializers define the metadata to be persisted. Newly generated serializers at restore time are reconfigured to be compatible.Savepoint state data serializer config metadataField “A” Field “B” Old serializer at time of savepoint Field “A” Field “B” New serializer at restore time
  44. 44. Upgrading State Serializers (III) 44 ● Custom serializers must define the metadata to write: TypeSerializerConfigSnapshot ● Also define logic for compatibility checks public class MyCustomTypeSerializer extends TypeSerializer<MyCustomType> { ... TypeSerializerConfigSnapshot snapshotConfiguration(); CompatibilityResult<MyCustomType> ensureCompatibility( TypeSerializerConfigSnapshot configSnapshot); }
  45. 45. Closing 45
  46. 46. TL;DR 46 ● The Flink community takes state management in Flink very seriously ● We try to make sure that users will feel comfortable in placing their application state within Flink and avoid any kind of lock-in. ● Upcoming changes / features to expect related to state management: Eager State Declaration & State Migration
  47. 47. 4 Thank you! @tzulitai @ApacheFlink @dataArtisans
  48. 48. We are hiring! data-artisans.com/careers

×