This document discusses building a micro-league fantasy football system that simulates a full season in under a minute. It describes ingesting play-by-play data from an API and simulating plays at 300 per second. It also outlines the architecture using Amazon Web Services instances and windowed streaming to process the data in real-time blocks.

1. Fantasy Football
Fisher
Paul Singman
Insight Data Engineering Fellow
October 2016

2. Motivation
● Current offerings:
○ Season long
○ Daily
● What’s missing?
○ Micro-leagues
1 Day Full SeasonSub 1 minute

3. Data Simulation and Ingestion
• Play-by-play files obtained from
SportRadar API
• Plays simulated at rate of 300 /s
{"player_name": "Marcus Mariota",
"timestamp": "2016-09-29_02:19:23",
"touchdown": 0, "yards": 9, "position": "QB"}
Example Play

4. Fantasy Football Fisher Architecture
Instances Cost Total
2 x 4
m4.large
$0.12 /hr $.96 /hr

5. Use of Windowed Streaming

6. Use of Windowed Streaming
30 sec 30 sec

7. Why Windowed Streaming?

8. Why Windowed Streaming?
○ Receiver creates a blocks of data every Block Interval (200ms)
■ No. of tasks = Batch Interval / Block Interval
○ Ideally No. of tasks = (No. of cores in cluster - 1) * 2 - to - 3
■ 3 cores means 6-9 tasks of data per batch
■ Batch Interval of 1.5s is ideal

9. • Bachelor of Science in Stats
from Penn
● Shelf full of O’Reilly books
• Jr Data Engineer experience
at early-stage startup
(Mighty)
• Enjoy movies,
backgammon, and rooftop
yoga