A journey of Spotify's ad technology system's full stack overhaul presented in "AYAWA" track at Qcon London 2017.

1. Ad Serving at Spotify Scale
A journey of incremental full stack overhaul
Kinshuk Mishra, Director of Engineering
kinshuk@spotify.com
@_kinshukmishra

2. A lucky mistake

3. Expected consequences

4. Sarcastic empathy

5. Some valuable feedback

6. The unintended consequence
Artist engagement
for exposed users went up

7. The unintended consequence
Promising insights about
content promotion use-case

8. The unintended consequence
Confirmation that the
ad server is a powerful
messaging platform

9. Why should you care?

10. Introduction
Ad
technology
stack
Architecture
Evolution

11. Introduction
Ad
technology
stack
Architecture
Evolution

12. What I do
● Founded ads engineering team at Spotify in 2011
● Build all things ads engineering - team & software
● Major focus areas :
○ Ad delivery (Backend and Web)
○ Multi-platform native ads (Client Platform)
○ Ad performance (ML and Data)

13. 3 noteworthy things

14. Full stack refactor
Evolution at scale
Pragmatic choices

15. 100,000,000+
MAU

16. 50,000,000+
Subscribers

17. 30,000,000+
Songs

18. 2,000,000,000+
Playlists

19. $5,000,000,000+
Revenue paid to rightsholders

20. 60 Markets

21. Platform Ubiquity

22. Freemium business model

23. Ad

27. Introduction
Ad
technology
stack
Architecture
Evolution

28. Beauty of Ad Server
Relevancy Pacing Unique View Sequence Optimization

29. Complexity of Ad tech ecosystem

30. In essence it is pretty simple
Client
User Profile database
Ad Server
Campaign Management
Portal
Billing/
Reporting
Ad campaign
database
Data
Collection
System

31. Spotify Ads infrastructure in 2011
Edge
Service
Desktop
Log
Delivery
HDFS
User
Profile
Batch
Basic Ad Server
Campaign
Management
Billing/
Reporting

32. Spotify Ads infrastructure in 2017
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

33. Multi-platform clients
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

34. Data collection
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

35. Intelligence
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

36. Ad Delivery
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

37. Demand fulfillment
iOS
Edge
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Optimization
Modeling
Self-Serve Portal
Creative
Generation
Payments
Billing/
Reporting

38. Now you know too
Ad server is a powerful
messaging platform

39. Introduction
Ad
technology
stack
Architecture
Evolution

40. Architecture overhaul is hard
● While keeping the business running
● While innovating on new products
● When you should have done it yesterday

41. Why did Spotify evolve Ads
architecture?

42. Future needs
●
● Growth in scale
● Emergence of new client platforms
● Cheap cloud computing
● New products to meet business objectives
● Technical debt

43. The 3 stories

44. Fixing the legacy mess
Story 1

45. Original ad server design
Edge Service
Router
hash(userid)
Ad server ring with
partitions
Ad server instance
Memcache
Memcache
Memcache
Memcache
Campaign DB
User DB
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication

46. Problems

47. Stateful service with faulty
persistence

48. Cache as a data store

49. Service cluster as a hashed ring

50. Ad decisioning in Client

51. Batch Client-Server Calls

52. Fix strategy

53. Fix strategy tactic

54. Isolate refactor to one system at
a time

55. The ad server transition
Edge
Service
Log
Delivery
HDFS
User
Profile
Batch
Smart Ad
Server
Campaign
Management
Billing/
Reporting
Ad Server
Proxy
(routing) Basic Ad
Server
Gradual transition from basic to smart ad
serving
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch
communication

56. After the ad server transition
Proxy
Service
Log
Delivery
HDFS
User
Profile
Batch
Campaign
Management
Billing/
Reporting
Smart Ad Server
Desktop
Rendering Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch
communication

57. Lean, mean and fast
Story 2

58. Division of responsibilities
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering
Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication
Ad
decisioning
Ad fetch
orchestration
Client context
Ad Trigger & Render
Before After

59. Problems

60. Thick Clients

61. Logic duplication

62. Tightly coupled monolith

63. Fix strategy

64. Reduce State Management

65. Break monolith into services

66. Isolate platform independent
logic into a lib

67. Fix tactic

68. Design your systems to be
master of one thing

69. Remember division of responsibilities?
Desktop iOS
Android
Ads SDK
Desktop
Web
Rendering
Ad trigger
decisioning
Ads
Ranking
Ads
Caching
Ad batching & fetch communication
Ad
decisioning
Ad fetch
orchestration
Client context
Ad Trigger & Render
BAD GOOD

70. Multiplatform Client design
iOS
Proxy
Service
Android
Ads SDK
Desktop
Web
Chromecast/
Playstation/
FireTV
Ad
Aggregation
Service
Log
Delivery
GCS
User
Profile
Targeting
Service
DMP
Stream Batch
Ad Server
Decision Delivery
Ad
Exchanges
Campaign
Management
Modeling
Self-Serve Service
Creative
Generation
Payments
Billing/
Reporting

71. Knowledge is power,
Unreliable data is your enemy
Story 3

72. Event
Stream
Historical
ETL1 ETL2 ETL3
UserEntity1(attribute1, attribute2) UserEntity1(attribute1, attribute3) UserEntity1(attribute1, attribute3’)

74. Problems

75. Duplicate, undiscoverable and
fragmented datasets

76. Metric inaccuracy

77. Overloaded Data Infra

78. Fix strategy

79. Focus on reliable and timely log
delivery

81. Data engineering with SLA

82. Dataset canonicalization

83. Some useful lessons learnt from
architectural overhaul

84. Test with minimal impact radius

85. Mistakes are inevitable

86. Speed up build decisions

87. Think for tomorrow, Solve for today

88. Thank You!
kinshuk@spotify.com
@_kinshukmishra