Recommended
More Related Content
What's hot
What's hot (20)
Similar to Thinking in Streaming - Twitter Streaming API
Similar to Thinking in Streaming - Twitter Streaming API (20)
Recently uploaded
Recently uploaded (20)
Thinking in Streaming - Twitter Streaming API
- 3. Turtles All The Way Down • Your client ≅ Our server • Gather Events • Parse JSON • Match on Predicates • Route to Consumers
- 4. Properties • Offered • At Least Once • Roughly Sorted (K-Sorted) • Desired • Exactly Once • Sorted
- 5. Plan • Over-deliver Ensure At Least Once • De-duplicate Unordered Exactly Once • Sort Ordered Exactly Once
- 6. Why At Least Once? • Exactly Once impractical across streams • Clients must handle reconnect over-delivery • Reuse this capability • Mask upstream failures • Relax server restart issues
- 7. Why At Least Once? • Exactly Once impractical across streams • Clients must handle reconnect over-delivery • Reuse this capability • Mask upstream failures • Relax server restart issues
- 8. Why At Least Once? • Exactly Once impractical across streams • Clients must handle reconnect over-delivery • Reuse this capability to • Mask upstream failures • Relax server restart issues
- 10. Startup • Prefetch from peer to populate circular buffer • Go multi-user • Consume Kestrel backlog - duplicates between: • Buffer and backlog • Previous connection and backlog • Steady State: Exactly Once Delivery
- 11. Startup • Prefetch from peer to populate circular buffer • Go multi-user • Consume Kestrel backlog - duplicates between: • Buffer and backlog • Previous connection and backlog • Steady State: Exactly Once Delivery
- 12. Upstream Failure • Cascaded source fails • Fail over to next peer • Over-request to avoid loss • Steady State: Exactly Once Delivery
- 13. Client Over-delivery • Use Count Parameter after fast reconnect • Deep backfill from REST API • Client offline offline for a while • User first issues new query • Overlap connections slightly
- 14. De-Duplication
- 15. Infinite Streams • De-duplicating a randomly ordered infinite stream requires infinite time and storage • Sorting? Ditto • I have neither infinite time nor storage
- 16. Roughly Sorted • A sequence α is k-sorted IFF ∀ i, r, 1 ≤ i ≤ r ≤ n, i ≤ r - k implies aᵢ ≤ aᵣ • Strictly sorted is 0-sorted. • Transpose two adjacent values in a 0-sorted sequence, becomes 1-sorted. • K For the firehose?
- 17. Firehose K 500k IDs Sunday Night Monday Peak 100.0% 3356 2507 99.99% 650 509 99.90% 232 271 99.00% 143 160 90.00% 36 45 50.00% 5 7 Average 14 17 Noisy
- 18. Firehose K 500k IDs Sunday Night Monday Peak 100.0% 3356 2507 99.99% 650 509 99.90% 232 271 99.00% 143 160 90.00% 36 45 50.00% 5 7 Average 14 17 Noisy
- 19. Pessimist’s K • In theory could be hours & millions of events • Practically, if current and stale queues exist: • We’ll flush the stale queues before exposing • You’ll never know this happened • If all queues stale: • We’ll deliver the backlog • K remains reasonable
- 20. Unordered De-duplication • Create two HashSets: Primary, Secondary, each preallocated to size K • New event is duplicate if ID exists in Primary • Add new ID to both HashSets • When Primary.size > K / 2 Primary.clear Swap Primary & Secondary
- 21. Unordered De-duplication • Bounded memory consumption • O(n) behavior • Low latency • Emit first tweet • Discard subsequent duplicates • Cheaper than de-duplication by sorting? Probably depends on K
- 22. Ordered & De-duplicated • Insertion sort and de-duplicate by ID into a decreasing order list • While length > K, remove sorted tail
- 23. Ordered & De-duplicated • O(n) --- O(n * K) • Bounded memory consumption • Induces latency of K • Assumes average items not very unsorted • K is usually large to handle the outliers
- 24. Routing Events • By Keyword or by UserId • Add predicates to HashMap • Apply events to Map • Query holds private predicate set for later Map removal • O(n)
- 25. Reliability • Decompose • Decouple • Monitor
- 26. Monitoring • What to look at? • Latency • Throughput • Errors • Alerting
- 27. Horizontal Scale • Firehose keeps Growing. • Eventually Firehose stream will become impractical. • Partition the Firehose into N streams.
Editor's Notes
- There is a lot of symmetry in what the Streaming API servers do and what your streaming clients do. In both cases we’re gathering events, parsing them, and farming them out to various consumers. The issues are similar at all processing points in the stream.
- We present a stream of events that is roughly sorted by created at time. This means that the events are mostly in created at time order, but not exactly so. We’ve designed our system to publish each event at least once -- which means none are lost, but there may, at times, be duplicates. I’ll discuss why our streams have these properties. Also, you’ll probably want to display or process tweets exactly once -- none missing and none duplicated. You might also want to present them sorted, or you might be OK with a rough sorting. I’ll go over two algorithms for converting what the API offers into the stream that you want.
- The basic plan is to over deliver events and then de-duplicate them to provide an exactly once quality of service. One technique is to just de-duplicate with set logic, the other is to sort and de-duplicate. There are trade offs with each.
- First, let’s see why the Streaming API offers events at least once. It would be nice if we could offer everything transactionally, that is, exactly once. But, it’s impractical to synchronize this state across client reconnections. For example, it’s unlikely that you’ll reconnect to the same server.
- Also, event streams aren’t strictly ordered, so we wouldn’t know what to deliver. We’d have to coordinate a large vector of sent events between servers. And, clients would have to transactionally acknowledge all events received. This is quite impractical at scale unless we sorted streams, but this would introduce latency. We’ll see why sorting induces latency later.
- Yet, first and foremost, we want a very low latency experience. And, we want a simple programming model for clients. So, we assume that clients can over-request when reconnecting, and post process to get the required stream properties. Once we make this fundamental assumption, we can reuse this to also handle the internal data loss risk as well.
- Our Streaming API server is called Hosebird. Hosebird receives events from the rest of the Twitter system through Kestrel message queues. Two hosebird processes in each cluster read transactionally from Kestrel. The rest of the servers in a cluster cascade via Streaming HTTP.
- When a hosebird server starts, it prefetches events from a peer to pre-populate its circular buffers. These buffers are used to support the count parameter, which allows some historical back fill on streaming queries. Count allows your stream to start back a few minutes, then catch up and transition to real time streaming.
- This startup prefetching creates a window where you might see the same event twice, if you are unlucky enough to connect to a very recently restarted server. The backlog read from kestrel will contain some of the same events that were prefetched into the buffer. The backlog may also have events that you read on your last connection. You might have to suffer through a minute or so of duplicates as the backlog is processed and displaces the prefetched events in the circular buffer. Outside of this restart case, during steady state processing, we deliver each event exactly once on fanout servers.
- When a cascaded server has its source Hosebird restart, say during a deploy, the server needs to quickly fail over to another source. A gap in the stream would be introduced during the failure, detection and reconnection window. We cover this gap by requesting some back-fill from the new source. This causes a short period of duplicated events. During steady state processing, however, we deliver each event exactly once on cascaded servers.
- Your client should use these same techniques on reconnect. Over request with the count parameter if the connection was momentarily lost. If the client has been disconnected for an extended period, you’ll have to back fill from the REST API. When you need to make a predicate change, you can create a new connection, wait for the first event to arrive, then disconnect the old connection. This should generally produce an at least once stream.
- Let’s talk about de-duplication on your end.
- A finite stream looks a lot like a relational database table -- a finite relation. We’re used to thinking about finite relations. But, a stream appears as an infinite relation, you can’t ever read to the end. Also, since we want very low latency, we can’t wait to read to the end. We have to present results immediately.
- A roughly sorted sequence is mostly sorted, where no element is more than K positions away from its strictly sorted position. At Twitter, we talk about K sorted things all the time. K this, K that. Nothing is strictly ordered. We have relaxed various legs of the CAP theorem to make our distributed system feasible. We’ve never had strictly ordered event processing. Tweets are applied to your timelines in a rough ordering. On the REST API, we sort the vector before we present it to you, but it’s very loose behind the scenes. Likewise, events show up in the Streaming API roughly sorted by created at time.
- Here are two samples from the status firehose. I took five hundred thousand status ids, and did an insertion sort into a reverse sorted list. The most recent id at the head, the oldest status at the tail. These distributions show the number of list elements traversed before finding the sorted insertion point. So, the average and median number of hops are pretty small. The hundred percent case, the worst case, shows a much larger K.
- Assuming about 600 events per second on this stream, back when I took this sample, we can see that events show up as much as 5 seconds out of order. Close comparison of the distributions shows that they’re very noisy. If you took many samples, they’d all have a different shape. Having an idea of K helps us tune our de-duplication algorithms.
- Daily operational issues cause K to grow beyond 5 seconds now and then. It’s hard to say what a good upper bound for a display client should be. Something around a few minutes would cover most issues we’ve had over the last six months. A long-term storage client might want to assume a K of a few hours or a day or so. In the unlikely event that something goes really wrong with the system, we’ll make a judgement call on recovery. We’ll probably bias towards delivering the backlog, but, if there’s a partial failure, we’ll keep your K in mind.
- Now that we have a handle on K we can think about de-duplication. An infinite, but roughly sorted, stream can be de duplicated with some set logic. The key is efficiently aging out irrelevant set members. One way is to keep two hashes, and alternately clear them. You don’t have to do any fancy tracking of items, and off the shelf HashSets will work just fine. The union of the two sets contain at least K items and allow deduplication of a K sorted sequence.
- Given the Firehose K, you don’t even need all that much space to de-duplicate. Please don’t resort to using mySQL primary keys to de-duplicate streams. Unnecessary. The nice thing here is that we can emit events as they arrive and throw away late arriving dups. We don’t need to add any latency.
- On the other hand, if we want a sorted and deduplicated stream, we have to do a little more work. Given the Firehose K distribution, doing an insertion sort isn’t the worst thing. Most events don’t need to traverse too deeply into the list. Elements dequeued from the tail of the list are sorted and deduplicated.
- This algorithm does, however introduce a latency of K. We can’t emit a sorted event unless we have at least K elements to examine. Still, this is quite practical to do in memory. You can plow through a lot of ids per second even in a scripting language like Ruby.
- Now that we have a de-duplicated stream, we need to route it to consumers. This can be done very cheaply by registering every consumer’s predicates in a HashMap. If, say, you are displaying columns of search results, like TweetDeck, you can have each column register its keywords in the HashMap. Each new event is applied to the HashMap, and routed to all consumers easily. Duplicates can arise, as a given column may have several OR predicates that match. Hosebird uses a generational de-duplication scheme to solve this. This scheme is the degenerate case of the sorted algorithm above. Each client stream maintains just the primary key of the last event. If the same id is presented twice in a row, it can be discarded.
- Break things up into components. Host components in separate processes. Measure what happens between components. Use (reliable) queues between components.