Successfully reported this slideshow.
Your SlideShare is downloading. ×

Building an Asynchronous Application Framework with Python and Pulsar - Pulsar Summit SF 2022

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad

Check these out next

1 of 26 Ad

Building an Asynchronous Application Framework with Python and Pulsar - Pulsar Summit SF 2022

Download to read offline

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

This talk describes Klaviyo’s internal messaging system, an asynchronous application framework built around Pulsar that provides a set of high-quality tools for building business-critical asynchronous data flows in unreliable environments. This framework includes: a pulsar ORM and schema migrator for topic configuration; a retry/replay system; a versioned schema registry; a consumer framework oriented around preventing message loss and in hostile environments while maximizing observability; an experimental “online schema change” for topics; and more. Development of this system was informed by lessons learned during heavy use of datastores like RabbitMQ and Kafka, and frameworks like Celery, Spark, and Flink. In addition to the capabilities of this system, this talk will also cover (sometimes painful) lessons learned about the process of converting a heterogenous async-computing environment onto Pulsar and a unified model.

Advertisement
Advertisement

More Related Content

More from StreamNative (20)

Advertisement

Building an Asynchronous Application Framework with Python and Pulsar - Pulsar Summit SF 2022

  1. 1. Building an Asynchronous Application Framework with Python and Pulsar FEBRUARY 9 2022 Pulsar Summit Zac Bentley Lead Site Reliability Engineer Boston, MA
  2. 2. 2 2022 © Klaviyo Confidential The Problem What We Built Challenges What Worked Well What’s Next? 01 02 03 04 05
  3. 3. 3 2022 © Klaviyo Confidential Segmentation Reviews Retail POS Social Surveys Referrals Logistics Shipping Customer service Loyalty On site personalization Forms Ecommerce Order confirmation SMS Email
  4. 4. Existing Architecture
  5. 5. 5 2022 © Klaviyo Confidential Problems Reliability Scalability Ownership/Process Architectural
  6. 6. 6 2022 © Klaviyo Confidential Problems Reliability RabbitMQ has reliability issues when pushed too hard. “Backpressure will find you” Deep queues behave poorly. Lots of outages and firefighting. Scalability Ownership/Process Architectural
  7. 7. 7 2022 © Klaviyo Confidential Problems Reliability Scalability Scaling RabbitMQ is intrusive: application code has to be aware of topology changes at every level. Geometry changes are painful. Scale-out doesn’t bring reliability/redundancy benefits. Ownership/Process Architectural
  8. 8. 8 2022 © Klaviyo Confidential Problems Reliability Scalability Ownership/Process Individual team ownership is expensive in: - Roadmap time. - Hiring/onboarding capacity. - Coordination. Per-team ownership creates redundant expertise. Architectural
  9. 9. 9 2022 © Klaviyo Confidential Problems Reliability Scalability Ownership/Process Architectural Celery is pretty hostile to SOA. Ordered consuming: not possible. Processing more >1 message at a time: not possible. Pub/sub: difficult. Replay/introspection: not possible.
  10. 10. Existing API: Producers from app.tasks import mytask # Synchronous call: mytask("arg1", "arg2", kwarg1=SomeObject()) # Asynchronous call: mytask.apply_async(args=("arg1", "arg2"), kwargs={"kwarg1": SomeObject()}) @celery.task(acks_late=True) def mytask(arg1, arg2, kwarg1=None): ... @celery.task(acks_late=True) def mytask2(*args, **kwargs): ... Existing API: Consumer Workload Declaration
  11. 11. 11 2022 © Klaviyo Confidential Problems Reliability Scalability Ownership/Process Architectural
  12. 12. 02 What We Built 1. Platform Services: a team 2. Pulsar: a broker deployment 3. StreamNative: a support relationship 4. Chariot: an asynchronous application framework
  13. 13. ORM for Pulsar Interactions for tenant in Tenant.search(name="standalone"): if tenant.allowed_clusters == ["standalone"]: ns = Namespace( tenant=tenant, name="mynamespace", acknowledged_quota=AcknowledgedMessageQuota(age=timedelta(minutes=10)), ) ns.create() topic = Topic( namespace=ns, name="mytopic" ) topic.create() subscription = Subscription( topic=topic, name="mysubscription", type=SubscriptionType.KeyShared, ) subscription.create() assert Subscription.get(name="mysubscription") == subscription consumer = subscription.consumer(name="myconsumer").connect() while True: message = consumer.receive() consumer.acknowledge(message)
  14. 14. Declarative API for Schema Management & Migrations from klaviyo_schema.registry.teamname.data.payload_pb2 import PayloadProto my_topic = ChariotTopic( name="demo", durability=Durability.DURABILITY_REDUNDANT, max_message_size="1kb", max_producers=100, max_consumers=10, publish_rate_limits=( RateLimit( messages=1000, period="1m", actions=[RateLimitAction.RATE_LIMIT_ACTION_BLOCK], ), ), thresholds=( Threshold( kind=ThresholdKind.THRESHOLD_KIND_UNACKNOWLEDGED, size="200mb", actions=[ThresholdAction.THRESHOLD_FAIL_PUBLISH], ), ), consumer_groups=(ConsumerGroup(name="demo-consumer-group", type=SubscriptionType.KeyShared),), payload=RegisteredPayloadFromClass(payload_class=PayloadProto), )
  15. 15. Existing API: Producers from app.tasks import mytask # Synchronous call: mytask("arg1", "arg2", kwarg1=SomeObject()) # Asynchronous call: mytask.apply_async(args=("arg1", "arg2"), kwargs={"kwarg1": SomeObject()}) @celery.task(acks_late=True) def mytask(arg1, arg2, kwarg1=None): ... @celery.task(acks_late=True) def mytask2(*args, **kwargs): ... Existing API: Consumer Workload Declaration
  16. 16. New API: Producers class DemoExecutor(AsynchronousExecutor): @lifecycle_method(timeout=timedelta(seconds=10)) async def on_executor_shutdown_requested(self): ... @lifecycle_method(timeout=timedelta(seconds=10)) async def on_executor_shutdown(self): ... @lifecycle_method(timeout=timedelta(seconds=10)) async def on_executor_startup(self): ... @lifecycle_method(timeout=timedelta(seconds=10)) async def on_message_batch(self, messages: Sequence[PayloadProto]): for idx, msg in enumerate(messages): if idx % 2 == 0: await self.chariot_ack(msg) else: await self.chariot_reject(msg) from klaviyo_schema.registry.teamname.data.payload_pb2 import PayloadProto from klaviyo_schema.registry.teamname.topics import my_topic await my_topic.send(PayloadProto(...)) New API: Consumer Workload Declaration
  17. 17. Back-Of-Queue Retries class DemoExecutor(AsynchronousExecutor): @lifecycle_method(timeout=timedelta(seconds=10)) @requeue_retry( batch_predicate=retry_on_exception_type( RetryException, retry_log_level=logging.INFO ), message_predicate=retry_until_approximate_attempt_count(10), delay=wait_exponential(max=timedelta(seconds=5)) + timedelta(seconds=1), ) async def on_message_batch(self, messages: Sequence[PayloadProto]): raise RetryException("Expected retry") ~> chariot worker start --topic demo --consumer-group democg --parallel 10 --start-executors-lazily --executor-class app.executors.demo:DemoExecutor --message-batch-assignment-behavior AnyKeyToAnyExecutor ~> chariot worker start --topic demo --consumer-group democg --parallel 10 --start-executors-lazily --executor-class app.executors.demo:DemoExecutor --message-batch-assignment-behavior NoOverlapBestEffortKeyExecutorAffinity --message-batch-flush-after-items 1000 --message-batch-flush-after-time 10sec Custom Batching and “Steering” for Parallel Execution without Reordering
  18. 18. 18 2022 © Klaviyo Confidential Problems Solutions Reliability Scalability Ownership/Process Architectural
  19. 19. 19 2022 © Klaviyo Confidential Solutions Reliability To become a user is to express the enforced maximum workload you’ll run. Pulsar’s redundancy helps weather outages. Deep backlogs are usable because reads aren’t always writes. Scalability Ownership/Process Architectural
  20. 20. 20 2022 © Klaviyo Confidential Solutions Reliability Scalability The “CEO” (Central Expert Owner) can scale out pulsar to respond to demand. Teams express scalability need in the form of elevated rate limits or partition counts. Consultation with the community and StreamNative is invaluable. Ownership/Process Architectural
  21. 21. 21 2022 © Klaviyo Confidential Solutions Reliability Scalability Ownership/Process Teams own producers/consumers. Teams submit their contracts, in the form of schema PRs, to the broker owners. Schema changes and backwards compatibility aren’t simple but they are now predictable. Architectural
  22. 22. 22 2022 © Klaviyo Confidential Solutions Reliability Scalability Ownership/Process Architectural Many new patterns are now on the table: - Pub-sub - Ordered consume - Batched consumption + out-of-order acks - Deduplication/debouncing Reading topics at rest improves visibility. Async interaction with the same stream from multiple codebases is now possible.
  23. 23. 03 Challenges ● Distribution as a library/framework rather than an application ● Python/C++ Pulsar client maturity ● Combining advanced broker features surfaced bugs ● Forking consumer daemons + threaded clients + async/await style is a costly combination ● Expectation management ● The “gap ledger” ● Management API quality
  24. 24. 04 What Worked Well Process: ● Support from above ● Managed rollout speed ● Solving 2025’s problems, not 2022’s ● “Steel-thread” style focus on specific use-cases ● Willingness to commit to bring work in-house and start fresh where it made sense Technology: ● Declarative schemas for messages and dataflows ● Schema registry as code rather than a SPOF ● Managed Pulsar allows us to learn with less pain ● Isolating user code from consumer code improves reliability
  25. 25. 05 What’s Next? Near Term: ● Manage internal adoption ● Scale to meet annual shopping holidays’ needs ● Start work on a “publish gateway” for connection pooling, circuit breaking, etc. Long Term: ● Online schema changes ● Key-local state ● Complex workflow support ● Make our work available to the community
  26. 26. klaviyo.com/careers zac@klaviyo.com

×