An online sports apparel powerhouse, Fanatics has to power an always-on scalable ecommerce site listing wares for over 1,000 vendors, and to track the shopping carts of millions of demanding sports fans. Find out how Fanatics uses Scylla for order capture, how they handle abandonments, perform data mining and more. Learn their “practical engineering” lessons, and discover how Fanatics wrote their own management tool for cluster automation and autoscaling on AWS.

1. Scylla at Fanatics
Niraj Konathi, Director of Platforms Engineering

2. Presenter
Niraj Kothari, Director of Platforms Engineering
Niraj Kothari has more than fifteen years of experience in
build platforms and frameworks. He has worked at
companies including NBCi, eBay/Paypal, and now at
Fanatics Inc

3. Over $2B In Sales Through
A Multichannel Approach For
More Than 300 Global Partners
Hot Market And Replica Jersey
Rights To Meet Today’s
On-Demand Culture
Licensing Rights Across
All Major Leagues And Numerous
NCAA Partners Programs
Event Retail And In-Venue
Retail Rights With Top Leagues,
Teams And Properties

4. E-COMMERCE
STRATEGIC RETAIL
INTERNATIONAL
WHOLESALE
OUR CORE
RIGHTS AND VERTICALITY
SCALE / INFRASTRUCTURE
BRAND
INVESTING HEAVILY TO BUILD COMPREHENSIVE PLATFORM TO REVOLUTIONIZE THE FAN EXPERIENCE

5. More than 300 Partners Worldwide

6. The Shift to Mobile
■ Driven By Fan Passion
50% of all Fanatics Sales come through mobile and that
spikes to nearly 80% around big events and championships.
Compared to just above 30% for retail industry.
■ Mobile First Company
Fanatics is a mobile-first company. Every site enhancement
and update is first tested on mobile device before being
optimized across platforms.
■ Investment
Fanatics will invest $100m in mobile and technology
this year
Platform Traffic from Mobile
70%
Platform Revenue from Mobile
50%
MOBILE FIRST DEVELOPMENT PHILOSOPHY

7. 40Offices & Facilities
10K+Employees at Peak
VISITS TO OUR PLATFORM
401M
MARKETING KEYWORDS
600M
RETAIL & EVENT VENUES
200+
REVENUE GENERATED FROM
MOBILE DEVICES
57%

8. 325KUnique Products
1MTotal SKUs
HOT MARKET EVENTS
363
UNITS SHIPPED “DTC”
30M+
UNITS MANUFACTURED
45M

9. 9
Cassandra Usage
■ Used as a primary store for
● Order Capture
● Cart mutations (for analytic purposes as well)
● Accounts/User
● Loyalty
● Promo
● Order Management (downstream)
● Order Visibility
● Ratelimiting

10. 10
Issues seen Cassandra
■ Large wide rows for Cart mutations
● Forcing a large cluster size
● Causing frequent GC pauses
● CPU spikes when compacting these wide rows
● Resulting into increase in number of timeouts
■ Due to larger cluster size
● Increase in EC2 cost
● Increase in TCO
● Longer maintenance window

11. 11
Move to Scylla
■ Moved Cart mutation use case to Scylla resulting into the
following
● Cut Cassandra cluster by more than 50%
● Drastically reduced EC2 cost
● Reduced TCO
● Almost 0 timeouts
● Happier customers/application teams

12. 12
Managing Scylla
■ Automation
● One can deploy a new stack in minutes using 2 slack commands
● Node2Node/Client2Node encryption
● Use of ENI for Cluster/seed node discovery
● ASG + ASG Hooks
● Home grown tool that we use for rolling restarts and other maintenance task
● Scylla-Mon for monitoring (hooked with PD)
● Scylla-Manager for data repairs

13. 13
Use of ENI (Elastic Network Interface)
■ Seed node attaches to an ENI and broadcasts ENI’s static IP
■ Non-seed joins based on ENI IP during bootstrap
■ No manual updates needed when a seed node goes down and new
seed node comes up (through ASG)
■ When a new seed node comes up, it will attach to same ENI

14. 14
Commando
■ UI based tool like “OpsCenter”
■ Tool to manage state-full and stateless applications
■ Rest service to take requests
■ Agent waiting for the command
■ Mutual TLS based communication
■ Can be used for
● Rolling restarts
● Gathering logs from numerous VMs in one place based on DNS
● Applying patches to state-full systems
■ Uses Kafka for Pub/Sub mechanism
■ Uses RDS for persistence

15. Thank you Stay in touch
Any questions?
Niraj Kothari
nkothari@fanatics.com
@NirajRKothari