The document discusses the development of eBay's auction platform utilizing serverless and event-driven architectures with Kafka. It covers the importance of treating events as the core of data interaction, focusing on how to manage streams and integrate serverless functions for processing events effectively. Key takeaway points include the need for both event sourcing and microservices principles to ensure scalability and flexibility in a high-transaction environment.

1. Serverless and Streaming
Building ‘ebay’ by turning the database inside out
Neil Avery, @avery_neil
Office of the CTO
Kafka Summit, San Francisco, October 2018

2. - Building the auction platform
- Event driven architectures on Kafka
- FaaS for streaming
- Event driven patterns
- Putting it in action
Agenda

3. Building the Auction platform

4. Top down approach:
● What does it look like?
● What qualities?
● How many items?
● How many users?
The grand plan: what are we building?
users
the world
selling
buying
the sketch

5. A stack view
from stateful to stateless
{...}
{...}
{...}
FaaSKafka cluster
stream
processin
g
Kafka Streams
KSQL

6. Streaming on the cloud
Think: Streaming First

7. Stream processors versus FaaS processors
FaaS - GCP Fn, Azure Fn, AWS Lambda etc
- Stateless
- Out of band (generally)
- Edge (in band)
- Ad hoc
- Super-elastic
Kafka Streams & KSQL
- Stateful (and stateless)
- In band
- Dataflow - control plane
- Not ad hoc (except KSQL)
- Elastic
Think: data-model --> {...}Think: fire and forget -->

8. Event driven architectures with Kafka

9. Events
What is an event?
FACT!
SOMETHING
HAPPENED!

10. Events
A Sale An Invoice A Trade A Customer
Experience

11. Events
Why do you care?
Loose coupling, autonomy, evolvability, scalability, resilience, traceability, replayability
BEING EVENT-FIRST CHANGES HOW YOU
THINK ABOUT WHAT YOU ARE BUILDING
...more importantly...

12. Events versus commands
Events:
- Fact
- Something happen
- Immutable
- When
- Behavioural
Commands:
- Intent
- Contract
- Do something
- Coupling
- Structural
EVENTS ARE USED TO MODEL BEHAVIOUR IN A DOMAIN
EVENT-SOURCING CAPTURES THAT BEHAVIOUR

13. What is a company?
A business is a series of events and reacting to those events.

14. All your data is a stream of events

15. Events and data pipelines: Events as data
Databases Databases
Customer
Data Updates
Unified 360
Merged Customer
Profiles
“Who bought what”
Events

16. Events, Streams, Partitions, Tables
producers brokers
KSQL
Kafka streams
consumer
Stream
processor
KSQL
Kafka streams
consumer
Stream
processor
stream processors
CDC streams

17. Turning the database inside out
• Model the transaction log as a Stream
• Use stream processors to materialize tables
• Scale out using many stream processors
• Record every fact in a log (Kafka!)
• Replay every fact from the log
Kafka: the log
MKleppmann2015:
Turningthedatabaseinsideout
Stream
processor
Stream
processor
Stream
processor
Stream
processor
materialized viewmaterialized viewmaterialized viewMaterialized view

18. You need both tables and streams

19. Streaming correctness (stateful)
• Preservation of order
• Support late arriving data
• Tolerance for out-of-sequence data
• Support machine failure scenarios
• Exactly once --- plus the ability to reason about time
{sensor:100}
{sensor:150}
{sensor:190}
Stream
processor
Resilient state storage
{delta:50}
{delta:50}
{delta:40}

20. Streaming correctness (stateless)
Stream
processor
{sensor:100}
{sensor:150}
{sensor:190} good stream
bad stream

21. Data Modelling
Think events not commands,
-> streams of events,
-> series of streams to model the domain
{
user: 100
type: bid
item: 389
cat: bikes/mtb
region: dc-east
}
/bikes/mtb by item-id
/bikes/ by dc-east‘-’item-id
- Keyspace
- Throughput (events per sec)
- Throughput (historic)
- Parallelism
- Replicas
- Retention
- Data evolution

22. {
user: 100
type: bid
item: 389
cat: bikes/mtb
region: dc-east
}
Data Modelling
/bikes/mtb by item-id
- Keyspace
- Throughput (events per sec)
- Throughput (historic)
- Parallelism
- Replicas
- Retention
- Data evolution
key#
partition
topic
key space

23. Stateless versus Stateful
Stateful
- Window
- Aggregate
- Join (stream-table, stream-stream etc)
Think: remembers events!
Stateless
- Filtering
- Transform
- Projection
Think: fire and forget!
/stream-1
/stream-2
join
Stream
processor

24. Winning with Event-driven streaming
architectures
the secret sauce
the log, event sourcing, source of truth,
CQRS, event collaboration, replayability,
at-least once, exactly one, evolutionary
architectures, data-virtualization and
more

25. FaaS and Stream processing
{...}

26. What is FaaS?
- Bring your own compute
- Elastic
- Pay per use
- Stateless
- Cloud native
Any language, concurrent, sync or async
{...}

27. FaaS apps
1. Single site, events in any order
Async chained, stateless
2. Non-stream oriented, non-time critical
3. Edge processing (in or out)
4. Enrich incoming events (stateless etl)
or
outgoing - email users, enrich image, ad-hoc requests
{...}

28. FaaS BUTs
- Sync chaining FaaS anti-pattern
- Complicated on different platforms
- Really really granular - like micronano-services
- Not cross cloud interoperable yet (without kafka or svrless fwk)
- Testability sucks
- Automation sucks
{...}
… but …
FaaS requires the event-driven paradigm in order to be successful.
You cannot throw away 25 years of event-driven legacy and go back to rpc (fail!)
It’s not ‘event driven’ unless you use event-sourcing

29. FaaS and event-sourcing
{...}
{...}
{...}
Recap:
- Events are behavioural
- FaaS is meant to be event driven
- Event-sourcing ensure replayability + others
the log the log
Event sourcing:
How did we get here?

30. FaaS and streaming correctness
{...}
FaaS qualities
- Elastic
- Concurrency
- Sync or async
- Short-lived
{...}
{...}
{...}
bad stream

31. FaaS for Stream processing
{...}
Needs...
- throughput
- ordering
- concurrency
- async
Works when...
- throughput per stream >
invocation
- stateless
- not-historic processing
- async
Otherwise...
- pin streams to individual faas
processors
- only 1 faas per stream
- problematic for scale :(

32. FaaS and the auction platform
- Enrich users on signup (address validation)
- Geo-enrich items on placement (city, state, lat-lon cell identification)
- Notify users on item sold or reserve not met
- Perform analytics on auction when item ‘completes’`
- Notify user of items-of-interest from their history when browsing
- Ad-placement analytics (watched items, interested items, users-purchased
- Monte-carlo auction simulations to guide users and calc item trending scores
In-band but edge
Out of band, edge
ad-hoc

33. Patterns

34. Patterns for Infrastructure
1. Ops: (Observability, instrumentation, metrification) = monitoring patterns, dead-letter-
queues, error-queues, audit-logs, application-logging, data-lineage
2. Stream-based: Worker Queue (compute grid or faas), event-backbone
3. Data-based: K/V store, Queryable Data fabric,
Data virtualization (via connect cdc streams) → Data Fabric
4. FaaS: fire and forget, event sourcing, unit-of-work

35. Patterns for Infrastructure: Ops
Observability, instrumentation, metrification
dead-letter-queues, error-queues, audit-
logs, application-logging, data-lineage
Stream
processor
/dead-letter/bid/region/processor
/auction/region/bid
/audit/sec-ops/category/region/
/ops/logs/category/region/
Ops Queues
/ops/metrics/elastic_search

36. Events as a backbone
“digital nervous system”
Dept 2 Dept 3 Dept 4Dept 1
Patterns for Infrastructure: Stream based
Digital nervous system
W
A
R
N
IN
G
:
A
void
ESB
anti-patterns

37. Patterns for Infrastructure: Stream based
Worker Queue (compute grid or faas)
Stream
processor
/priority-10
/priority-9
/priority-8
worker
worker
worker
worker
workerworker
Worker Queue
millions of atomic events with differing priority

38. Patterns for Infrastructure: Stream based
Payment processing flow
DEBIT
Check
[from]
/payment-inflight
[from, to]
Payment processing
millions of concurrent payments
CREDIT
[to]
Balance
conf’d
Balance
in-flight
/payment-complete
[from, to]
/payment
*from, to
/payment
from, *to
/payment
from, to

39. Patterns for Infrastructure: Data
K/V store, Queryable Data fabric,
Stream
processor
/stream-1
/stream-2
/stream-3
worker
worker
worker
worker
worker
Clients
Queryable data fabric
Stream
processor
query
interactive
query

40. Patterns for Infrastructure: Data
Data virtualization (via connect cdc streams) → Data Fabric
Stream
processor
/items
/users
/bids
worker
worker
worker
worker
worker
Client
requests
Kafka connect: data virtualization
Stream
processor
items
users
bids

41. Patterns for Infrastructure: FaaS
FaaS
Connector/items
Fire-and-forget
{...}
{...}
{...}
cloud-events
FaaS
W
ORK
IN
PROGRESS
AWS Lambda connector

42. Patterns for Infrastructure: FaaS
FaaS
Connecto
r
/items
Event-sourcing
{...}
{...}
{...}
cloud-events
FaaS
W
O
RK
IN
PRO
G
RESS
/item-analytics

43. Patterns for Infrastructure: FaaS
/items
Unit-of-work
{...}
{...}
{...}
‘N’
cloud-events
FaaS
(calculation)
W
O
RK
IN
PRO
G
RESS
/item-analytics
{...}
FaaS
(aggregate)

44. Building the Auction Platform

45. Auction functionality
Search, Bid, Item-Complete, marketplace analytics
● Search: Find an item to bid against
● Bid: Compete against others to win the auction
● Item-Complete: When the time has expired tell users to pay or relist
● Marketplace analytics: Educate users in order to drive competition

46. Auction functionality: BID
search, bid, item-complete, analytics
Auction items
Items being placed
1.
Bids against item ‘duran duran’
stream-table
join against ‘duran duran’
2.
Item ‘duran duran’
3.
CDC Stream
{ item:duran res: 100 bid: 125 buyer: michael }
{ item:duran res: 100 bid: 120 buyer: nick }
{ item:duran res: 100 bid: 121 buyer: damian }
{ item:duran res: 100 bid: 115 buyer: andy }
{ item:duran res: 100 bid: 110 buyer: andy }
KSQL: SELECT * from bidding-stream
where item=’duran duran’
4.

47. Auction functionality: SEARCH
search, bid, item-complete, analytics
4. KStream: Interactive query
3. Table: Materialized view
topic: /auction/records/80sSearch: Identify set of topics
1.
2. Locate KTables

48. Auction functionality: ITEM COMPLETE
search, bid, Item-complete, analytics
1. Stream processor runs ‘future’ on
local-state of item-bids
auction items
3. Rejoined
removes item
4. Stream triggers FaaS
complete item processor
bid-history
Stream table join
5. FaaS Notify all bidders
bid-notifications
2.
Rewrite ‘completed’ item status
to retrigger join
item status

49. Auction functionality: ANALYTICS
search, bid, item-complete, analytics
How is the item trending? Banding on condition (new, as-new, used, worn, for-parts)
Indicative pricing bands: reserve versus final bid?
What’s the usual bid-offer spread?
Frequency of sale?
Percentiles?
{...}
{...}
{...}
{ bid:100; sold:1000;}
/bid-history
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
{
stream-lib.TDigest(values[])
}
Calculating percentiles using the ‘unit-of-work’ pattern
{
digest.merge(digest)
}
{...}

50. Auction functionality: SYSTEM
Stream processing and FaaS
/auction/items
{...}
{ item:100;}
Stream
processor
1. Items
{ bid:100;}
{...} Stream
processor
/auction/bids
2. Bidding
{...}
notify bidders & seller
{ item:100; offer:101}
/auction/bids/history
{...}
analytics for bidders
3. Processing

51. Wrapping it up!

52. Key takeaways:
● Model: Events as the API, model the use-cases, model for scale, evolve the
data-model : DDD
● As Streams: Streams are the database, tables materialized views, architect
for evolutionary apps by using events
● App Infra: Build patterns to underpin higher order models (metrics, k/v etc)
● FaaS: Ad-hoc and edge stream processing or pinned processors

53. Key takeaways:
Event-first forces you to think about behaviour of the system
Event sourcing captures that behaviour

55. The future
● Stream processing: more powerful = KStreams, KSQL, UDFs
● Streaming platform: more accessible, complete experience, cross-cloud
● Closer affinity: between Streaming and FaaS for streaming processing
● FaaS: CNCF working group: CloudEvents, Event function flow

56. Thank You!
Check out my ‘serverless and stream processing’ blog via twitter: @avery_neil
As of September 2018, Ebay announced that they were replatforming onto Kafka
Kafka Summit - New York (April 2) - CFP soon
Kafka Summit - London (May 13-14)
Rate this talk on the Summit-App ;)

57. 60Confidential
Kafka Summit SF 2018 - Download the Mobile App
● Search App store for “Kafka Summit”
● https://guidebook.com/g/kafkasummitsf
● See speakers and schedules
● Personalize your agenda
● Rate speakers and sessions!
● Network with fellow attendees
● Share comments and photos on social wall
● Turn on notifications to receive up to date info