Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Serverless Event Streaming with Pulsar Functions

151 views

Published on

As data continues to evolve, there are more and more requirements for calculating data. From servers to VMS to containers to serverless, the computing framework is constantly evolving with the needs of users. How to provide users with a fast, easy-to-deploy computing framework has become a question for everyone to think about. In this talk, we will introduce how Pulsar provides a powerful computing power based on the serverless computing architecture. Apache Pulsar is a new generation of cloud messaging system and real-time processing platform. The message system is closely related to the real-time computing platform, but it is often loosely deployed and managed separately. Pulsar Functions, as the computing component of Pulsar, is a fusion and innovation of the message and computing platform in the serverless direction. Pulsar Functions provides support for multiple languages such as Go, Python, and Java; and runtimes such as Thread, Process, and Kubernetes. This provides great flexibility for user-written, run, and deploy features. Let users only use logic that cares about real calculations, without complicated configuration or management; it is easier to build a message-triggered flow platform.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Serverless Event Streaming with Pulsar Functions

  1. 1. Serverless Event Streaming with Pulsar Functions Xiaolong Ran (@wolf4j1) 09/12/2019
  2. 2. ● Go programmer ● PingCAP > Bitmain > StreamNative ● Pulsar committer ○ pulsar-client-go ○ Go Functions ○ … ● https://github.com/wolfstudy Who am I
  3. 3. ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases Agenda
  4. 4. ● What is Apache Pulsar? ● Event Stream - Pulsar view on Data ● When Event Streaming meets serverless ○ Programming Model ○ Architecture ○ Use cases Agenda
  5. 5. A Pulsar view on Data
  6. 6. Pulsar View - Topic
  7. 7. Pulsar View - Partition
  8. 8. Pulsar View - Segment
  9. 9. Pulsar View - Event Stream
  10. 10. Event Stream is the right foundation for your data
  11. 11. M(essaging), S(torage), P(rocessing)
  12. 12. MSP - Interactive Queries
  13. 13. MSP - Stream & Batch Processing
  14. 14. MSP - What is next? What is next?
  15. 15. When Event Streaming meets Serverless
  16. 16. Introduce Pulsar Functions
  17. 17. Pulsar Functions ● A serverless framework for data processing ● Lightweight computation ● Event-first, Stream-first ● Support both stateless and stateful computation ● Multi languages ● Multi runtimes ● SDK-less & SDK
  18. 18. Function Elements ● Input Topics ● Output Topics ● Function ● State ● Log Topics
  19. 19. API - Native Java / Python / Go Function Golang Python Java
  20. 20. API - Function Context ● Logger ● State ● Metrics ● Security / Secrets ● ...
  21. 21. Context - Logger
  22. 22. Context - State
  23. 23. Context - State ● Global Managed State * ● Mutable by functions and admin-cli ● Queryable by functions and admin-cli ● State are stored at storage layer ● State are implemented using streams + snapshot
  24. 24. Context - State API ● Key/Value State API ○ putState ○ getState ● Counter State API ○ getCounter ○ incrCounter
  25. 25. Context - Metrics ● API - recordMetric(String metricName, double value) ● Exposed in prometheus format ● Collected by prometheus
  26. 26. Flexible Runtime ● Colocate with Broker - Thread & Process ● Managed Function Workers - Thread & Process ● External Schedulers - Container ○ Kubernetes
  27. 27. Colocate with Brokers
  28. 28. Managed Function Workers
  29. 29. External Schedulers - Kubernetes
  30. 30. Event Routing ● Events are routed to different partitions ● Leverage Pulsar’s MessageRouter ● Existing MessageRouters ○ Round-Robin ○ SinglePartition ○ Hash (Murmur32) ● Customize MessageRouter
  31. 31. Auto load balancing ● Pulsar Functions use Pulsar’s auto-balancing mechanism on consumers ● Shared Subscription ○ Load is distributed among function instances (consumers) evenly ○ More function instances provides more processing capability ● Failover Subscription ○ Load is distributed among function instances (consumers) by partitions ○ The number of function instances is limited by the number of partitions
  32. 32. Integrated with pulsar-admin CLI
  33. 33. Pulsar Functions Architecture
  34. 34. Usage Pattern
  35. 35. Usage Pattern Simple Event Processing with Pulsar Functions (by David Kjerrumgaard)
  36. 36. Content Routing
  37. 37. Message Filtering
  38. 38. Transformation
  39. 39. Alert and thresholds
  40. 40. Complex Event Processing Pipelines
  41. 41. Pulsar Functions at EMQ
  42. 42. EMQ ● IoT Messaging Provider ● Open source software company ● Users ○ HPE ○ Ericsson ○ Huawei ○ China Mobile ○ China UnionPay ○ ...
  43. 43. ActorCloud ● Open Source IoT Platform
  44. 44. ActorCloud Streaming
  45. 45. Summary
  46. 46. Pulsar Functions Summary ● A serverless approach to do event streaming ● Flexible, lightweight, easy to understand and use ● Event-first, Stream-first ● Stateless + Stateful (*) ● Flexible runtime and data locality ● Functions can be orchestrated to do complex processing ○ Workflow, DAG, Iterations, Graph, and ...
  47. 47. Pulsar Functions Roadmap ● More languages support ● Function Orchestration ● Managed state vs Local state ● Large state ● Transactional Processing ● ...
  48. 48. Community ● Twitter: @apache_pulsar , @streamnativeio ● Mailing Lists: dev@pulsar.apache.org users@pulsar.apache.org ● Slack: https://apache-pulsar.slack.com/ ● Github: ○ https://github.com/apache/pulsar ○ https://github.com/apache/bookkeeper ● Documentation: https://pulsar.apache.org ● Medium https://medium.com/streamnative
  49. 49. Pulsar in Action https://www.manning.com/books/ pulsar-in-action The first Pulsar book by David Kjerrumgaard
  50. 50. Thanks!

×