MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB
•
5 likes•3,205 views
This document provides an overview of Google Cloud Pub/Sub including key concepts, benefits, and usage examples. It discusses why message queues are important, early players in the space, and how Cloud Pub/Sub provides scalability, flexibility and reliability. The document also summarizes Spotify's migration from their old event delivery system to using Cloud Pub/Sub and the results of producer load and consumer stability tests.
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB
Similar to MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB (20)
More from Tu Pham
More from Tu Pham (20)
Recently uploaded
Recently uploaded (20)
MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB
- 2. WHO AM I PHAM PHUONG TU Co-Founder of DYNO (Online email marketing SaaS) Data engineer at MOG Corp (previous: VC Corp) Cloud lover, data lover
- 3. FACTS Google: 1.5 B monthly active device. Whatsapp: 1 B monthly active users, 1.6 billion daily photo message Facebook: 1.55 B monthly active users Spotify: 100M monthly active users, moving from Kafka to Google Cloud Pub Sub.
- 4. WHY MESSAGE QUEUE SO IMPORTANT Traditional database is not the right approach for: Balancing workload Asynchronous workflows Data stream Reliability improvement Refreshing distributed caches
- 6. WHAT IS CLOUD PUB/SUB A PaaS message queue with Scalability Flexibility Reliability
- 7. WHAT IS CLOUD PUB/SUB
- 8. PUB / SUB CONCEPTS AND FLOW
- 10. BENEFITS AND FEATURES Unified messaging Global presence Flexible delivery options Data reliability
- 11. BENEFITS AND FEATURES End-to-end reliability Data security and protection Flow control Simplicity
- 12. GCLOUD SDK Setup • curl https://sdk.cloud.google.com | bash • exec -l $SHELL • gcloud init • gcloud components: list / install / update Commands: gcloud alpha pubsub • topics: create / list / publish / list-subscription / delete • subscriptions: create / list / ack / modify-ack-deadline / modify-push-config / delete
- 13. API CLIENTS
- 14. CREATE CLIENT
- 15. CREATE TOPIC
- 16. LIST TOPIC
- 17. PUSH AND PULL
- 18. PUBLISH MESSAGE
- 19. SAMPLE MESSAGE
- 23. RETRY POLICY Push: If the push endpoint returns an error code, messages are retried for up to 7 days. Pull: Messages that are not acknowledged become available again for pulling when the acknowledgement deadline expires, up to 7 days after publication time.
- 25. SPOTIFY EVENT DELIVERY - OLD SYSTEM
- 26. SPOTIFY EVENT DELIVERY – NEW SYSTEM
- 29. MAKE MORE FUN
- 30. DEMO AppEngine & Cloud Pub/Sub • Sample: https://goo.gl/ifMPE6 • Gihub:https://github.com/phamptu/gcp-next2016- pubsub-demo
- 31. Q&A
- 32. RESOURCES • Gcloud SDK https://cloud.google.com/sdk/ • Identity platform https://developers.google.com/identity/ • API client https://developers.google.com/api-client-library/ • API explore https://developers.google.com/apis-explorer/ • Cloud API https://console.cloud.google.com/apis/library?project=<project_id> • Cloud PubSub https://cloud.google.com/pubsub/overview • Blog https://cloudplatform.googleblog.com/ • Youtube https://www.youtube.com/user/googlecloudplatform
Editor's Notes
- How huge system run
- At the simplest level, a message queue is a way for applications and discrete components to send messages between one another in order to reliably communicate. Message queues are typically (but not always) ‘brokers’ that facilitate message passing by providing a protocol or interface which other services can access. Balancing workloads in network clusters. For example, a large queue of tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances. Implementing asynchronous workflows. For example, an order processing application can place an order on a topic, from which it can be processed by one or more workers. Distributing event notifications. For example, a service that accepts user signups can send notifications whenever a new user registers, and downstream services can subscribe to receive notifications of the event. Refreshing distributed caches. For example, an application can publish invalidation events to update the IDs of objects that have changed. Logging to multiple systems. For example, a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on. Data streaming from various processes or devices. For example, a residential sensor can stream data to backend servers hosted in the cloud. Reliability improvement. For example, a single-zone Compute Engine service can operate in additional zones by subscribing to a common topic, to recover from failures in a zone or region.
- Opensource and in house solution
- Release: 03/2015 Google also uses the Pub/Sub API for its own products. Cloud Pub/Sub is designed to provide “at least once” delivery by storing copies of messages in multiple zones to ensure that subscribers can receive messages as swiftly as possible. All message data is encrypted and protected on the wire and at rest.
- 1 - A publisher application creates a topic in the Google Cloud Pub/Sub service and sends messages to the topic. A message contains a payload and optional attributes that describe the payload content. 2 - Messages are persisted in a message store until they are delivered and acknowledged by subscribers. 3 - The Pub/Sub service forwards messages from a topic to all of its subscriptions, individually. Each subscription receives messages either by Pub/Sub pushing them to the subscriber's chosen endpoint, or by the subscriber pulling them from the service. 4 - The subscriber receives pending messages from its subscription and acknowledges each one to the Pub/Sub service. 5 - When a message is acknowledged by the subscriber, it is removed from the subscription's queue of messages.
- At google: 1.5 million devices activated every day
- Unified messaging: Durability and low-latency delivery in a single product Global presence: Connect services located anywhere in the world Flexible delivery options: Both push- and pull-style subscriptions supported Data reliability: Replicated storage and guaranteed at-least-once message delivery
- End-to-end reliability: Explicit application-level acknowledgement Data security and protection: Encryption of data on the wire and at rest Flow control: Dynamic rate limiting implemented by the Pub/Sub system Simplicity: Easy-to-use REST/JSON API
- gcloud components list gcloud components install COMPONENT_ID gcloud components update
- Authentication The Google APIs client libraries handle much of the authentication process for you. The details of the authentication process, or "flow," for OAuth 2.0 vary somewhat depending on what kind of application you're using with Google Cloud Pub/Sub. (For detailed information about flows for various types of applications, see Google's OAuth 2.0 documentation.) Since most Pub/Sub messaging operations occur offline, with no human intervention, the most typical flow for a Google Cloud Pub/Sub application is server-to-server, using a service account. Your application signs the authorization request with the secret key to obtain a web access token that accompanies every call to the API. Google App Engine and Google Compute Engine use service accounts under the hood, and simplify the authentication process. Authorization scope Create a Cloud Pub/Sub client Authorization scope When you request access using OAuth 2.0, your application specifies the authorization scope information for Google Cloud Pub/Sub. You can use either of the following scopes: https://www.googleapis.com/auth/pubsub https://www.googleapis.com/auth/cloud-platform Note that to make any kind of call to Google Cloud Pub/Sub, you must use one of these scopes. These scopes are a prerequisite for making calls; they do not confer permissions. A principal's capabilities are determined by the access controls configured for the principal (that is, the authenticated user or service account).
- The general flow Define a topic and send a request to the Pub/Sub system to create it. Create a message, and set data and optional attributes on it. Send a request to the Pub/Sub Server to publish the message to the desired topic.
- By default, a maximum of 100 results are returned per query; you can specify an alternate value up to 1,000 in the pageSize query parameter.
- HIGH LIGHT: Use push delivery for low traffic, low-latency messaging. Use pull delivery to send data from App Engine to large numbers of subscribers.
- When using JSON, message data must be base64-encoded, and can be a maximum of 10MB after encoding. (The Java client libraries provide methods for Base64-encoding data.) Note that the message payload must not be empty; it must contain either a non-empty data field, or at least one attribute. Also, though Pub/Sub usually delivers messages in order of publication, this is not guaranteed; it is possible for subscriptions to receive messages out of order. For this reason, we suggest that you include sequence information in the message payload or attribute so that subscribers that need in-order messaging can implement logic to do so.
- Delivery contract For the most part Pub/Sub delivers each message once, and in the order in which it was published. However, once-only and in-order delivery are not guaranteed: it may happen that a message is delivered more than once, and out of order. Therefore, your subscriber should be idempotent when processing messages, and, if necessary, able to handle messages received out of order. If ordering is important, we recommend that the publisher of the topic to which you subscribe include some kind of sequence information in the message; see this page for a full discussion on message ordering. Messages that are not acknowledged, are retried indefinitely for up to seven days.
- Is the number of seconds after delivery, during which the subscriber must acknowledge the receipt of a pull or push message. If a subscriber does not respond with an explicit acknowledge (for a pull subscriber) or with a success response code (for a push subscriber) by this deadline, the server will attempt to resend the message. By default this deadline is 10 seconds, but you can set a custom value when you create a subscription. Pull subscribers can also modify the deadline on the fly -- on a per-message basis -- to allow for shorter or longer time to process a given message. The following example shows the effect of modifying the ack deadline for a subscription that is configured with a deadline of 60 seconds:
- In a push subscription, the Pub/Sub server sends a request to the subscriber application, at a preconfigured endpoint. The subscriber's HTTP response serves as an implicit acknowledgement: a success response indicates that the message has been succesfully processed and the Pub/Sub system can delete it from the subscription; a non-success response indicates that the Pub/Sub server should resend it (implicit "nack"). To ensure that subscribers can handle the message flow, the Pub/Sub dynamically adjusts the flow of requests and uses an algorithm to rate-limit retries. SLOW START ALGORITHM To regulate the rate of push message delivery, Google Cloud Pub/Sub uses a slow-start algorithm per subscription. With slow-start, Google Cloud Pub/Sub starts by sending a single message at a time, and doubles up with each successful delivery, until it reaches the maximum number of concurrent messages outstanding. Any time there is a delivery failure, the number of messages outstanding allowed for the subscription is halved.
- In a pull subscription, the subscribing application explicitly calls the API pull method, which requests delivery of a message in the subscription queue. The Pub/Sub server responds with the message (or an error if the queue is empty), and an ack ID. The subscriber then explicitly calls the acknowledge method, using the returned ack ID, to acknowledge receipt. You can specify push or pull delivery when you create a subscription and you can change it any time afterwards (however, a subscription can only use one type of delivery at a time).
- Our event delivery system is one of the foundational pieces of Spotify’s data infrastructure. In Spotify’s case, all data needs to be delivered to a centrally located Hadoop cluster. Spotify servers, from which we collect data, are located in multiple data centers on two continents. Bandwidth between our data centers is a scarce resource. When the system was built, one of the missing features from Kafka 0.7 was the ability of the Kafka Broker cluster to behave as a reliable persistent storage. This influenced a major design decision to not keep persistent state between the producer of data, Kafka Syslog Producer, and Hadoop. An event is considered reliably persisted only when it gets written to a file on HDFS. => If Hadoop is down the whole event delivery system stalls.
- The missing piece here is an event delivery system or queue that implements reliable transport of events and the persistence of undelivered messages in the queue. Choosing Google Pub/Sub over Kafka, including a Spotify-scale load test of 2,000,000 messages per second. Real load – 700 000 messages per second. Cloud Pub/Sub came with extra goodies: + Global availability—as a global service, Pub/Sub is available in all Google Cloud Zones; transferring data between our data center wouldn’t be through our normal internet provider but would use underlying Google network. + A simple REST API—if we didn’t like client library Google provided, we can easily write our own. + Operational responsibility was handled by someone else—there was no need to create a capacity model or deployment strategy, or to set up monitoring and alerting.
- We used a simple load generator to send mock traffic through the Event Service to Pub/Sub. The generated traffic was routed through two Pub/Sub topics with a ratio of 7:3. To push 2M messages per second, we ran the Event Service on 29 machines. Enabling batching and compression on the Event Service machines resulted in ~1Gbps of network traffic towards Pub/Sub.
- In our test, we created a subscription, then one hour later we started to consume data. We consumed data in batches of 1000 messages. The median end-to-end latency we measured during the test period—including backlog recovery—was around 20 seconds. We did not observe any lost messages whatsoever during the test period.
- Backbone of everything
- + App engine + Pubsub + Improve each group chat / direct chat in one seperated topic App Engine Modules (or just "Modules" hereafter) let developers factor large applications into logical components that can share stateful services and communicate in a secure fashion. A deployed module behaves like a microservice. By using multiple modules you can deploy your app as a set of microservices, which is a popular design pattern. For example, an app that handles customer requests might include separate modules to handle other tasks, such as: + API requests from mobile devices + Internal, admin-like requests + Backend processing such as billing pipelines and data analysis