This document describes a pipeline for ingesting StarCraft II match data using Apache Kafka and Avro serialization. The pipeline ingests both streaming and batch data from API requests into Kafka topics, then serializes the data with Avro for storage in HDFS. The document also provides background on the presenter, Guang Yang, including his education and StarCraft II ranking.

1. StarGazer:
!
Map Analytics in StarCraft II
Guang Yang

14. (Demo)

15. Pipeline
15
Ingestion
Streaming
Batch
Data-store
API/Front-end

16. Ingestion
16
API
AWS EC2 Node
AWS EC2 Node
AWS EC2 Node
AWS EC2 Node
Serialize
via Avro
Kafka Topics:
!
- Stream
!
- Batch
Example API Request: http://api.ggtracker.com/api/v1/matches/3529593.json

17. Serialization
via Avro
- reinforces schema
!
- splittable on HDFS
!
- backward compatible
!
- saves space (binary)

18. Pipeline
18
Ingestion
Streaming
Batch
Data-store
API/Front-end

19. Batch / Streaming
19

20. Guang Yang
- B.A. in Computational and Applied Mathematics (Rice University)
- M.S. in Industrial Engineering & Operations Research (UC Berkeley)
- Got into Diamond League as Terran without making any Siege Tanks
- Email: gy8@berkeley.edu
- GitHub: github.com/gy8
20