This session will be a detailed recount of the design, implementation, and launch of the next-generation Shutterstock Data Platform, with strong emphasis on conveying clear, understandable learnings that can be transferred to your own organizations and projects. This platform was architected around the prevailing use of Kafka as a highly-scalable central data hub for shipping data across your organization in batch or streaming fashion. It also relies heavily on Avro as a serialization format and a global schema registry to provide structure that greatly improves quality and usability of our data sets, while also allowing the flexibility to evolve schemas and maintain backwards compatibility. As a company, Shutterstock has always focused heavily on leveraging open source technologies in developing its products and infrastructure, and open source has been a driving force in big data more so than almost any other software sub-sector. With this plethora of constantly evolving data technologies, it can be a daunting task to select the right tool for your problem. We will discuss our approach for choosing specific existing technologies and when we made decisions to invest time in home-grown components and solutions. We will cover advantages and the engineering process of developing language-agnostic APIs for publishing to and consuming from the data platform. These APIs can power some very interesting streaming analytics solutions that are easily accessible to teams across our engineering organization. We will also discuss some of the massive advantages a global schema for your data provides for downstream ETL and data analytics. ETL into Hadoop and creation and maintenance of Hive databases and tables becomes much more reliable and easily automated with historically compatible schemas. To complement this schema-based approach, we will cover results of performance testing various file formats and compression schemes in Hadoop and Hive, the massive performance benefits you can gain in analytical workloads by leveraging highly optimized columnar file formats such as ORC and Parquet, and how you can use good old fashioned Hive as a tool for easily and efficiently converting exiting datasets into these formats. Finally, we will cover lessons learned in launching this platform across our organization, future improvements and further design, and the need for data engineers to understand and speak the languages of data scientists and web, infrastructure, and network engineers.

1. Building a Next-gen Data Platform
And Leveraging OSS Ecosystem for Easy Wins
Sean Quigley
Shutterstock

2. Myself
● Applying logic in varied domains
○ Physics and Economics in University
○ Quant Finance
○ Data Science in Ad Tech
○ Data Engineering
● Contact
○ squigley@shutterstock.com
○ https://github.com/seanpquig
○ https://twitter.com/s_quigls
○ https://www.linkedin.com/in/seanpquig

3. Shutterstock
● Global technology company
● High-quality licensed content for businesses, marketing/media
agencies
● 90M+ images, and 4M+ videos, music too
● 1.4M active customers in 150 countries
● Sell 5 images per second
● Data Infrastructure
○ Multi-petabyte YARN cluster
■ Hadoop, Hive, Spark, Oozie, Flink (POC)
○ Messaging + streaming solutions powered by Kafka
■ APIs for data production and consumption

4. Common Systems Lifecycle

5. Legacy Data Pipelines
● Logs (application and Nginx)
○ Flume
● Logs (Apache)
○ MariaDB
● Behavioral events
○ ZeroMQ + custom interfaces
○ JSON (messy)
● Time-series:
○ StatsD + CollectD + Graphite/Grafana
● Hadoop, Hive, and ETL:
○ Custom, home-grown jobs
○ Manual Hive DDL

6. This has become a bit of a Mess

7. Time to build
something new

8. Shutterstock Data Platform (SDP)
● End-to-end service pipeline
● Logs, user behavior/actions, click streams, time-series monitoring
● Ingestion and production of data
● Consumption of feeds by variable consumers
● Streaming and Batch
● ETL to long-term storage

9. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

10. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

11. Apache Kafka
● Pub-sub system modeled as a distributed commit log
● Highly scalable and performant
● Zookeeper for distributed coordination
● Topics are partitioned and replicated
● Consumers pull messages via a log offset per partition

12. Apache Kafka
Like a Streaming
Hadoop Cluster!
*from http://hortonworks.com/apache/kafka/

13. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

14. Apache Avro (Overview)
● Data serialization format
● Compact, fast, binary format
● Rich data structures
● Everything revolves around schemas

15. Apache Avro (Example)
{
type: record,
name: User,
fields: [
{
name: first_name,
type: string
},
{
name: age,
type: [null, int],
default: null
}
]
}

16. Confluent Platform
● Confluent, Inc.
● “Stream Data Platform”
● Strong influence on design of our Data Platform

17. Confluent Schema Registry (Overview)
● RESTful interface for storing and receiving Avro schemas
● Provides various compatibility settings for schema
evolution
● Confluent Kafka serializers

18. Confluent Schema Registry (Architecture)
*from http://docs.confluent.io/2.0.0/schema-registry/docs/design.html

19. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

20. SDP REST API (Intro)
● Language Agnostic protocol for producing Avro events
into Kafka
● But Confluent tries to solve this problem.
● Why not use their REST proxy?

21. SDP REST API vs. Confluent REST proxy
● Example event JSON
○ {“name”: “bill”, “age”: 27}
● To send this to the Confluent REST proxy:
○ {"value_schema": "{"type": "record", "name": "User", "fields":
[{"name": "name", "type": "string"}, {"name": "age", "type":
["null", "int"], "default": null}]}", "records": [{"value": {"name":
"bill", "age": {"int": 27}}}]}
○ {"value_schema": 41, "records": [{"value": {"name": "bill", "age":
{"int": 27}}}]}

22. SDP REST API (Overview)
● Written in Scala
● Clients send valid JSON
● JSON -> Avro schema inference and conversion
● All schema logic is fully recursive, so it works with
arbitrarily nested data.

23. SDP REST API (Features)
● Balance between ease of use and data structure
● Flexibility for evolution of schema
○ Easy to add and remove field
○ Some type evolutions permitted
● Schema maintains a historical record
● Error tracking and debugging tools for clients
○ Message UUIDs
○ Error topics in Kafka
○ Receive Timestamps

24. SDP REST API (Performance)
● 1st design iteration
○ SLOW: ~100 msg/s per CPU
● Optimizations
○ Be LAZY
○ Directly populate Avro bytes
○ In-memory cache of schemas on API nodes
○ Specialized data structures
● Led to performance of ~2000-3000 msg/s per CPU (20-
30X speedup)

25. Other APIs and Tools
● Consumer Service
○ Wraps the Kafka Consumer API in WebSocket
protocol
○ Support for group IDs
■ Consumer groups
■ Scale consumption horizontally
● Carbon API
○ Graphite-format time-series data in Kafka
● Clients
○ Node + Java producer/consumer clients

26. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

27. Camus (Overview)
● Specialized MapReduce Job for Kafka -> Hadoop ETL
● Open source
● Configuration driven

28. Camus (Architecture)
*from http://docs.confluent.io/2.0.0/camus/docs/design.html

29. Hive ETL (Overview)
● Hive DDL and DML that wrapped in Python scripts
● Scheduled via Oozie
● Schema-based approach really pays off here
○ Automated table management
○ Schema Evolution

30. Hive ETL (Example)
● Get latest historically compatible schema
○ schema.registry.net/subjects/topic_name-value/versions/latest
● Update avro table schema
○ ALTER TABLE topic_name
SET TBLPROPERTIES ('avro.schema.literal' = '{...}')

31. Hive ETL (What format?)
● Problem with Avro in Hive is that it is SLOW
● Let’s convert to something else!
○ Columnar (ORC, Parquet)
○ Easy OSS win!

32. ORC Format

33. Hive ETL (Columnar Conversion)
● Hive makes format conversion VERY EASY
● CREATE TABLE new_table STORED AS ORC
● INSERT … SELECT * FROM original_table
● We build a Python lib for wrapping this
○ Hive-format-converter
○ Supports schema evolution

34. 9 Shutterstock Data Design principles
● Extremely scalable
● Structured data
● Language-agnostic protocols for data production/consumption
● Fault-tolerant
● Automation
● Unification of disparate, fractured pipelines
● Error tracking and debugging
● Well-monitored
● Leverage latest from the OSS community and minimize DIY

35. Monitoring
● Nothing super sexy
● CodaHale/Dropwizard metrics is great with JVM
○ Know your metric types
■ Counters, meters, timers, gauges
● New Relic
● Icinga
● StatsD, CollectD
● Grafana, Graphite
● Health checks on APIs

36. Lessons learned
● Need for data engineers to speak different languages
○ Networking, Infrastructure, and Ops
○ Frontend and Web
○ Backend
○ Data Scientists and Business Analysts
● Data is UBIQUITOUS

37. Lessons learned
Logs tier →
Apps tier →

38. Lessons learned
● Data quality and usability should be a priority for all
○ Need to communicate and partner w teams
■ Product
■ Engineering
● Strike balance between standards and flexibility for clients
○ Too little => sloppy, hard-to-manage data
○ Too much => slows down and annoys teams

39. Lessons learned
● In a perfect world, teams have perfectly defined interfaces
● Perfect worlds do not exist
● Take the time to understand other teams’ code/systems
● Leads to better solutions influenced by diverse viewpoints

40. Future Work and Possibilities
● Admin UI
○ Self-service topic creation
○ Finer-grained schema control
● More robust offset management in Consumer API
● Streaming as a Service

41. Remember This?

42. Predicting future re-design
● Momentum of Kafka Ecosystem protects us partially
● Kafka Connect looks promising!
○ Framework for copying to/from Kafka
○ Looks to solve common pain points
● NiFi promising too!
○ Could potentially replace ingestion pieces
● More continuous spectrum of structure/flexibility tradeoff