Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Apache flink 1.0.0 overview

432 views

Published on

Apache Flink Overview presented by Jamie Grier in Flink Meetup at MaR Headquarters. https://community.mapr.com/docs/DOC-1503/edit?draftID=3848

Published in: Technology
  • Be the first to comment

Apache flink 1.0.0 overview

  1. 1. What’s new in Apache FlinkTM 1.0 Kostas Tzoumas @kostas_tzoumas
  2. 2. Flink 1.0 • March 8, 2016 • First release in 1.x.y series • Initiates backwards compatibility for selected APIs • More than 64 contributors • More than 450 JIRAs resolved 2
  3. 3. Flink 1.0: major features • Out of core state • Savepoints • CEP library • Improved monitoring & Kafka 0.9 support 3
  4. 4. Interface stability 4
  5. 5. Out of core state 5
  6. 6. Out of core state • Alternative to in-memory state • Powered by RocksDB instances in Flink TMs • Enabled by using the RocksDBStateBackend • State limited by disk space only • State checkpoints save RocksDB databases in reliable store 6
  7. 7. Savepoints 7
  8. 8. Production deployments • Maintaining stateful applications in production settings comes with its own challenges • Failures, code upgrades, cluster maintenance, … • Streaming jobs cannot be simply stopped and restarted 8
  9. 9. Reminder: fault tolerance • At least once, at most once, exactly once • Flink guarantees exactly-once processing • Flink guarantees end to end exactly-once with selected sources and sinks • e.g., Kafka —> Flink —> HDFS
  10. 10. How? Checkpoints • Flink guarantees fault tolerance by regularly taking checkpoints of the application state without ever stopping the execution • At failure, input stream is rewinded to the logical time of the last checkpoint 10
  11. 11. Introducing savepoints • A savepoint is a Flink checkpoint that (1) is taken by the user, (2) is accessible externally, and (3) never expires • Command line save & resume interface • Save: flink savepoint <JobID> • Resume: flink run -s <path/to/savepoint> <jobJar> 11
  12. 12. Savepoints and versions • A savepoint saves a version of a stateful application at a well-defined time • E.g.: take snapshots of one application at well-defined times 12
  13. 13. “Like git for state” • Branch off from savepoints creating a tree of running application versions 13
  14. 14. Essential for production deployments • Application code upgrades • Flink version upgrades • Maintenance, migration, debugging • What-if simulations • A/B testing • Time travel 14
  15. 15. Complex Event Processing 15
  16. 16. FlinkCEP • What is Complex Event Processing? • A catch-all term • In our context: easily detect patterns in streams 16
  17. 17. 17
  18. 18. Pattern API 18
  19. 19. 19
  20. 20. 20
  21. 21. Other features in 1.0 • Support for Kafka 0.9 API (and hence MapR Streams) • Monitoring console: job submission, checkpoint statistics, detecting bottlenecks • See http://flink.apache.org/news/2016/03/08/release- 1.0.0.html 21
  22. 22. Closing 22
  23. 23. Summary • Flink 1.0: Initiating backwards compatibility and pushing the envelope even further for production streaming deployments 23
  24. 24. What’s next • SQL • Dynamic scaling (+ savepoints) • Hybrid in-memory/out-of-core state backend • Query-able state • Support for Apache Mesos • More connectors and sinks (Kinesis, Cassandra, …) 24
  25. 25. Join the community • Follow: @ApacheFlink, @dataArtisans • Read: flink.apache.org/blog, data-artisans.com/blog • Subscribe: (news | dev | user)@flink.apache.org

×