SlideShare a Scribd company logo
Content Delivery
At Aviary
MongoDb User Group 11/19/13
Aviary
Photo-Editing
SDK & Apps
Fully-Baked UI
Configurable, High-Quality Tools
Over 5,000 Partners
Over 50 Million Monthly Users
Over 4 Billion Photos Edited
iOS, Android, Web, Windows, Server
J
Who Are We?
Nir

Jack

Lead Serverside

Director of

Engineer

Engineering

Likes:
●
●
●

Automated deployment
Big-O notation
Brainteasers

Hates:
●

Cilantro

Likes:
●
●
●

Parallelizing processes
DRY code
Seltzer

Hates:
●

Food after the sell-by date
Content
Effects

Frames

Stickers

Messages

J
The “Fat Tiny” Situation
We want to dynamically deliver the Fat Tiny
stickers to our users

How can we do that?
J
The CDS
Aviary’s Content Delivery System
Version 1
● Static files on S3
● Excess data sent
● Data stored in MySQL
● New features meant new code

N
The “Fat Tiny” Situation
We want to display the Fat Tiny sticker pack only
to Picstitch users in Japan who use iOS 7

How can we do that?
N
CDS V2
Version 2 Overview
Stack
Global
Users

Akamai

AWS Cloud Formation
AWS Elastic LoadBalancer (ELB)
Node.js
AWS Elastic Cloud Computing (EC2)
Ubuntu
Bash
MongoDb
JSON
JSON Schema

Content
Delivery
Network

Load Balancer

API
Servers
Database
Cluster
Management
System

N
Why MongoDB?
● Great with Node (JSON-based)
● Schema-less is easy to change/query
● Read-heavy
● Relatively small data set

N
How It Works
A Behind-the-Scenes Look
Delivered Types
● Manifest JSON
○ Content Set
○ Content Versions

● Content JSON
○ Content Metadata
○ All Assets

N
The “Fat Tiny” Situation
We want to manage the Fat Tiny sticker pack
as a single entity, but we want to deliver it to
each device in its own
optimal format.
How can we do that?
N
Response Formatting Model
Content Entry

Response Formats

Responses

JSON document describing
content item

JSON documents defining mappings
from entry to responses

Actual JSON responses
delivered to devices

J
Content Formats
● JSON schema

{
"type":"object",
"properties":{
"metadata": {

● Added properties

"type": "object",
"properties":{
"displayName": {
"type": "string",

● Custom types

}
}
},

● Validates entries

"icon": {
"type": "object",
"customType": "image"
"properties":{
"path": {
"type": "string",
"required": true,

J
Response Formats
● JSON schema

{
"type":"object",
"properties":{
"identifier": {

● dataKey property

"type": "string",
"dataKey": "identifier"
},

● Defines response

"name": {
"type": "string",
"dataKey": "metadata.displayName"
},

structure

"iconImagePath": {
"type": "string",
"dataKey": "icon.path-100"

● Maps content

},
"items": {
"type": "array",
"dataKey": "items"

J
Content Deployment
1. Insert/Update CMS Entry
"identifier": "com.aviary.stickers.234fe"

2. Find Response Formats
"type":"object",

"id": "com.aviary.stickers.234fe",

"properties":{

"metadata": {
"displayName": "Hats"

"name": "Hats",

"id": {
"type": "string",

},

"dataKey": "identifier"

"icon": {
"path": "cds/hats/icon.png"

"iconImagePath": "cds/hats/icon100.png"
"stickers": [
{

},

"identifier": "1"

"name": {

"path-100": "cds/hats/icon100.png"

"type": "string",

},

"imageUrl": "cds/hats/1.png"

"dataKey": "metadata.displayName"

"items": [

},

{

"type": "string",

"imageUrl": "cds/hats/1.png"
}

}
],

"iconImagePath": {

"identifier": "1"

]

3. Generate+Insert Responses

"versionKey": "e4532fd342"

"dataKey": "icon.path-100"
},
"stickers": {
"type": "array",
"dataKey": "items"

J
Manifests
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"
}
],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]

J
Manifest Deployment
Using Aggregate to Find the Newest Versions

Manifest with Correct Version Keys

{
$match:{

"stickers": [
{

formatId:{$in:formatIds},

"id": "com.aviary.stickers.234fe",

identifier:{$in:identifiers}

"versionKey": "e4532fd342"

}
}

},
],

{
$sort: { _id: -1 }

"effects": [
{

},

"id": "com.aviary.effects.25435",

{

"versionKey": "fd4324323"

$group: {
}

_id: "$identifier",
versionKey:{$first:"$versionKey"}

]

}
}

J
Scopes and Targeting
Deployed Manifests Have Scopes

End Users Have Scope Parameters

Manifest 1
"targetingScope": {
"apiKey": "abc",
"country": ["JP"]
},

/manifest?

"formattingScope": {
"platform": "ios",
"minOsVersion": "7.0.0"
}

apiKey=abc&
country=JP&
language=ja&
platform=ios&

Manifest 2

osVersion=7.2.0

"targetingScope": {
"apiKey": "def",
},
"formattingScope": {
"platform": "android",
"minOsVersion": "6.0.0"
}

N
API Servers
Scope parameters are converted into queries
Manifest 1
"targetingScope": {
db.manifest.find({

"apiKey": "abc",

"apiKey": {$in: ["abc", null]},
/manifest?

country=JP&

"formattingScope": {

"platform": {$in: ["ios", null]},
7002000

"platform": "ios",

"minOsVersion": {$lte: 7002000}

language=ja&

osVersion=7.2.0

},

"language": {$in: ["ja", null]},

apiKey=abc&

platform=ios&

"country": ["JP"]

"country": {$in: ["JP", null]},

"minOsVersion": "7.0.0"

}).sort({

}

"apiKey": -1,

Manifest 2

"language": -1,

"targetingScope": {

"country": -1,

"apiKey": "abc",

"minOsVersion": -1,

},

"platform": -1,

"formattingScope": {

"_id": -1

"platform": "ios",

}).limit(1)

"minOsVersion": "6.0.0"
}

N
Versioned Content
Received Manifests Contain VersionKeys
"stickers": [
{
"id": "com.aviary.stickers.234fe",
"versionKey": "e4532fd342"
},
{
"id": "com.aviary.stickers.fed34",
"versionKey": "c54532343d"

db.content.findOne({
/content?
versionKey=e4532fd342

}

"versionKey": “e4532fd342”
});

],
"frames": [
{
"id": "com.aviary.frames.25435",
"versionKey": "fd4324323"
}
]

N
Response Caching
db.manifests.find({

/manifest?

"apiKey": {$in: ["abc", null]},

apiKey=abc&

"country": {$in: ["JP", null]},

country=JP&

"language": {$in: ["ja", null]},

language=ja&

"platform": {$in: ["ios", null]},

platform=ios&

"minOsVersion": {$gte: 7002000}

osVersion=7.2.0

}).sort({
…,
"_id": -1
}).limit(1)

db.cachedManifests.findOne({
"url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios"
})

N
PAULA
The CDS Management Console
Auto-generated UI

J
Other Mongo Usage
● PAULA permissions in user objects
users collection
{
"name": "nir",
"email": "nir@aviary.com",
"permissions": [
"content",
"dev",
"admin",
"partying"
]
}

● Integration tests interact with schemaless db willy nilly
N
Conclusion
The Takeaway
The Facts
● Built and deployed in 3 months
● Very few struggles with MongoDB
● Seamless management
● Graceful scaling from 0 to over 20M MAUs
● Happy serverside engineers
J
The Future
● Targeted Translations
● Granular User Targeting
● PAULA for the masses

N
Questions?
Comments also welcome
nir@aviary.com

jack@aviary.com

…and by the way, WE’RE HIRING!

More Related Content

Similar to MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson

RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
contest-theta360
 
Engineering the New LinkedIn Profile
Engineering the New LinkedIn ProfileEngineering the New LinkedIn Profile
Engineering the New LinkedIn Profile
Josh Clemm
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScript
wesley chun
 
Build 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive CardsBuild 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive Cards
Windows Developer
 
Getting started with titanium
Getting started with titaniumGetting started with titanium
Getting started with titanium
Naga Harish M
 
Using Google (Cloud) APIs
Using Google (Cloud) APIsUsing Google (Cloud) APIs
Using Google (Cloud) APIs
wesley chun
 
The Rise of NoSQL
The Rise of NoSQLThe Rise of NoSQL
The Rise of NoSQL
Arnd Kleinbeck
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Shirshanka Das
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystem
Yael Garten
 
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Frédéric Harper
 
Getting started with Appcelerator Titanium
Getting started with Appcelerator TitaniumGetting started with Appcelerator Titanium
Getting started with Appcelerator Titanium
Techday7
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
Microsoft Tech Community
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
Microsoft Tech Community
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB Stitch
MongoDB
 
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CAAppcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Jeff Haynie
 
iPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumiPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumAxway Appcelerator
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
wesley chun
 
Fixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blockingFixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blocking
Igalia
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Jeado Ko
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점 Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
WebFrameworks
 

Similar to MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson (20)

RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
RICOH THETA x IoT Developers Contest : Cloud API Seminar (2nd installation)
 
Engineering the New LinkedIn Profile
Engineering the New LinkedIn ProfileEngineering the New LinkedIn Profile
Engineering the New LinkedIn Profile
 
Exploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScriptExploring Google (Cloud) APIs with Python & JavaScript
Exploring Google (Cloud) APIs with Python & JavaScript
 
Build 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive CardsBuild 2017 - B8002 - Introducing Adaptive Cards
Build 2017 - B8002 - Introducing Adaptive Cards
 
Getting started with titanium
Getting started with titaniumGetting started with titanium
Getting started with titanium
 
Using Google (Cloud) APIs
Using Google (Cloud) APIsUsing Google (Cloud) APIs
Using Google (Cloud) APIs
 
The Rise of NoSQL
The Rise of NoSQLThe Rise of NoSQL
The Rise of NoSQL
 
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystemStrata 2016 - Architecting for Change: LinkedIn's new data ecosystem
Strata 2016 - Architecting for Change: LinkedIn's new data ecosystem
 
Architecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystemArchitecting for change: LinkedIn's new data ecosystem
Architecting for change: LinkedIn's new data ecosystem
 
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
Windows 8 Pure Imagination - 2012-11-24 - Getting your HTML5 game Windows 8 r...
 
Getting started with Appcelerator Titanium
Getting started with Appcelerator TitaniumGetting started with Appcelerator Titanium
Getting started with Appcelerator Titanium
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
 
Microsoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needsMicrosoft Graph: Connect to essential data every app needs
Microsoft Graph: Connect to essential data every app needs
 
Building Your First App with MongoDB Stitch
Building Your First App with MongoDB StitchBuilding Your First App with MongoDB Stitch
Building Your First App with MongoDB Stitch
 
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CAAppcelerator iPhone/iPad Dev Con 2010 San Diego, CA
Appcelerator iPhone/iPad Dev Con 2010 San Diego, CA
 
iPhone/iPad Development with Titanium
iPhone/iPad Development with TitaniumiPhone/iPad Development with Titanium
iPhone/iPad Development with Titanium
 
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & morePower your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
Power your apps with Gmail, Google Drive, Calendar, Sheets, Slides & more
 
Fixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blockingFixing Gaps. Strengthening the Chromium platform for content blocking
Fixing Gaps. Strengthening the Chromium platform for content blocking
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
 
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점 Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
Angular를 활용한 웹 프론트단 개발과 2.0에서 달라진점
 

More from Hakka Labs

Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)
Hakka Labs
 
DataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchDataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series search
Hakka Labs
 
DataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data ScienceDataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data Science
Hakka Labs
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
Hakka Labs
 
DataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at InstacartDataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at Instacart
Hakka Labs
 
DataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scaleDataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scale
Hakka Labs
 
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataDataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
Hakka Labs
 
DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale
Hakka Labs
 
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQDataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
Hakka Labs
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
Hakka Labs
 
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...
Hakka Labs
 
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at PinterestDataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
Hakka Labs
 
DataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineeringDataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineering
Hakka Labs
 
DataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data StructuresDataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data Structures
Hakka Labs
 
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkDataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
Hakka Labs
 
DataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with OurselvesDataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with Ourselves
Hakka Labs
 
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityDataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
Hakka Labs
 
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
Hakka Labs
 
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedInDataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
Hakka Labs
 
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopDataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL Workshop
Hakka Labs
 

More from Hakka Labs (20)

Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)Always Valid Inference (Ramesh Johari, Stanford)
Always Valid Inference (Ramesh Johari, Stanford)
 
DataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series searchDataEngConf SF16 - High cardinality time series search
DataEngConf SF16 - High cardinality time series search
 
DataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data ScienceDataEngConf SF16 - Data Asserts: Defensive Data Science
DataEngConf SF16 - Data Asserts: Defensive Data Science
 
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast DataDatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
DatEngConf SF16 - Apache Kudu: Fast Analytics on Fast Data
 
DataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at InstacartDataEngConf SF16 - Recommendations at Instacart
DataEngConf SF16 - Recommendations at Instacart
 
DataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scaleDataEngConf SF16 - Running simulations at scale
DataEngConf SF16 - Running simulations at scale
 
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor DataDataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
DataEngConf SF16 - Deriving Meaning from Wearable Sensor Data
 
DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale
 
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQDataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
DataEngConf SF16 - BYOMQ: Why We [re]Built IronMQ
 
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
DataEngConf SF16 - Unifying Real Time and Historical Analytics with the Lambd...
 
DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...DataEngConf SF16 - Three lessons learned from building a production machine l...
DataEngConf SF16 - Three lessons learned from building a production machine l...
 
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at PinterestDataEngConf SF16 - Scalable and Reliable Logging at Pinterest
DataEngConf SF16 - Scalable and Reliable Logging at Pinterest
 
DataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineeringDataEngConf SF16 - Bridging the gap between data science and data engineering
DataEngConf SF16 - Bridging the gap between data science and data engineering
 
DataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data StructuresDataEngConf SF16 - Multi-temporal Data Structures
DataEngConf SF16 - Multi-temporal Data Structures
 
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using SparkDataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
DataEngConf SF16 - Entity Resolution in Data Pipelines Using Spark
 
DataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with OurselvesDataEngConf SF16 - Beginning with Ourselves
DataEngConf SF16 - Beginning with Ourselves
 
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High DeliverabilityDataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
DataEngConf SF16 - Routing Billions of Analytics Events with High Deliverability
 
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
DataEngConf SF16 - Tales from the other side - What a hiring manager wish you...
 
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedInDataEngConf SF16 - Methods for Content Relevance at LinkedIn
DataEngConf SF16 - Methods for Content Relevance at LinkedIn
 
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopDataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL Workshop
 

Recently uploaded

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
SOFTTECHHUB
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
James Anderson
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Vladimir Iglovikov, Ph.D.
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
Neo4j
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
Alpen-Adria-Universität
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
DianaGray10
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
Kumud Singh
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Alex Pruden
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Neo4j
 

Recently uploaded (20)

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...
 
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIEnchancing adoption of Open Source Libraries. A case study on Albumentations.AI
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI
 
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
Video Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the FutureVideo Streaming: Then, Now, and in the Future
Video Streaming: Then, Now, and in the Future
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
Mind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AIMind map of terminologies used in context of Generative AI
Mind map of terminologies used in context of Generative AI
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
 
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
 

MongoDB and Content Delivery at Aviary by Nir Zicherman and Jack Sisson

  • 2. Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 5,000 Partners Over 50 Million Monthly Users Over 4 Billion Photos Edited iOS, Android, Web, Windows, Server J
  • 3. Who Are We? Nir Jack Lead Serverside Director of Engineer Engineering Likes: ● ● ● Automated deployment Big-O notation Brainteasers Hates: ● Cilantro Likes: ● ● ● Parallelizing processes DRY code Seltzer Hates: ● Food after the sell-by date
  • 5. The “Fat Tiny” Situation We want to dynamically deliver the Fat Tiny stickers to our users How can we do that? J
  • 6. The CDS Aviary’s Content Delivery System
  • 7. Version 1 ● Static files on S3 ● Excess data sent ● Data stored in MySQL ● New features meant new code N
  • 8. The “Fat Tiny” Situation We want to display the Fat Tiny sticker pack only to Picstitch users in Japan who use iOS 7 How can we do that? N
  • 9. CDS V2 Version 2 Overview
  • 10. Stack Global Users Akamai AWS Cloud Formation AWS Elastic LoadBalancer (ELB) Node.js AWS Elastic Cloud Computing (EC2) Ubuntu Bash MongoDb JSON JSON Schema Content Delivery Network Load Balancer API Servers Database Cluster Management System N
  • 11. Why MongoDB? ● Great with Node (JSON-based) ● Schema-less is easy to change/query ● Read-heavy ● Relatively small data set N
  • 12. How It Works A Behind-the-Scenes Look
  • 13. Delivered Types ● Manifest JSON ○ Content Set ○ Content Versions ● Content JSON ○ Content Metadata ○ All Assets N
  • 14. The “Fat Tiny” Situation We want to manage the Fat Tiny sticker pack as a single entity, but we want to deliver it to each device in its own optimal format. How can we do that? N
  • 15. Response Formatting Model Content Entry Response Formats Responses JSON document describing content item JSON documents defining mappings from entry to responses Actual JSON responses delivered to devices J
  • 16. Content Formats ● JSON schema { "type":"object", "properties":{ "metadata": { ● Added properties "type": "object", "properties":{ "displayName": { "type": "string", ● Custom types } } }, ● Validates entries "icon": { "type": "object", "customType": "image" "properties":{ "path": { "type": "string", "required": true, J
  • 17. Response Formats ● JSON schema { "type":"object", "properties":{ "identifier": { ● dataKey property "type": "string", "dataKey": "identifier" }, ● Defines response "name": { "type": "string", "dataKey": "metadata.displayName" }, structure "iconImagePath": { "type": "string", "dataKey": "icon.path-100" ● Maps content }, "items": { "type": "array", "dataKey": "items" J
  • 18. Content Deployment 1. Insert/Update CMS Entry "identifier": "com.aviary.stickers.234fe" 2. Find Response Formats "type":"object", "id": "com.aviary.stickers.234fe", "properties":{ "metadata": { "displayName": "Hats" "name": "Hats", "id": { "type": "string", }, "dataKey": "identifier" "icon": { "path": "cds/hats/icon.png" "iconImagePath": "cds/hats/icon100.png" "stickers": [ { }, "identifier": "1" "name": { "path-100": "cds/hats/icon100.png" "type": "string", }, "imageUrl": "cds/hats/1.png" "dataKey": "metadata.displayName" "items": [ }, { "type": "string", "imageUrl": "cds/hats/1.png" } } ], "iconImagePath": { "identifier": "1" ] 3. Generate+Insert Responses "versionKey": "e4532fd342" "dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items" J
  • 19. Manifests "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" } ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] J
  • 20. Manifest Deployment Using Aggregate to Find the Newest Versions Manifest with Correct Version Keys { $match:{ "stickers": [ { formatId:{$in:formatIds}, "id": "com.aviary.stickers.234fe", identifier:{$in:identifiers} "versionKey": "e4532fd342" } } }, ], { $sort: { _id: -1 } "effects": [ { }, "id": "com.aviary.effects.25435", { "versionKey": "fd4324323" $group: { } _id: "$identifier", versionKey:{$first:"$versionKey"} ] } } J
  • 21. Scopes and Targeting Deployed Manifests Have Scopes End Users Have Scope Parameters Manifest 1 "targetingScope": { "apiKey": "abc", "country": ["JP"] }, /manifest? "formattingScope": { "platform": "ios", "minOsVersion": "7.0.0" } apiKey=abc& country=JP& language=ja& platform=ios& Manifest 2 osVersion=7.2.0 "targetingScope": { "apiKey": "def", }, "formattingScope": { "platform": "android", "minOsVersion": "6.0.0" } N
  • 22. API Servers Scope parameters are converted into queries Manifest 1 "targetingScope": { db.manifest.find({ "apiKey": "abc", "apiKey": {$in: ["abc", null]}, /manifest? country=JP& "formattingScope": { "platform": {$in: ["ios", null]}, 7002000 "platform": "ios", "minOsVersion": {$lte: 7002000} language=ja& osVersion=7.2.0 }, "language": {$in: ["ja", null]}, apiKey=abc& platform=ios& "country": ["JP"] "country": {$in: ["JP", null]}, "minOsVersion": "7.0.0" }).sort({ } "apiKey": -1, Manifest 2 "language": -1, "targetingScope": { "country": -1, "apiKey": "abc", "minOsVersion": -1, }, "platform": -1, "formattingScope": { "_id": -1 "platform": "ios", }).limit(1) "minOsVersion": "6.0.0" } N
  • 23. Versioned Content Received Manifests Contain VersionKeys "stickers": [ { "id": "com.aviary.stickers.234fe", "versionKey": "e4532fd342" }, { "id": "com.aviary.stickers.fed34", "versionKey": "c54532343d" db.content.findOne({ /content? versionKey=e4532fd342 } "versionKey": “e4532fd342” }); ], "frames": [ { "id": "com.aviary.frames.25435", "versionKey": "fd4324323" } ] N
  • 24. Response Caching db.manifests.find({ /manifest? "apiKey": {$in: ["abc", null]}, apiKey=abc& "country": {$in: ["JP", null]}, country=JP& "language": {$in: ["ja", null]}, language=ja& "platform": {$in: ["ios", null]}, platform=ios& "minOsVersion": {$gte: 7002000} osVersion=7.2.0 }).sort({ …, "_id": -1 }).limit(1) db.cachedManifests.findOne({ "url": "/manifest?apiKey=abc&country=JP&language=ja&osVersion=7.2.0&platform=ios" }) N
  • 27. Other Mongo Usage ● PAULA permissions in user objects users collection { "name": "nir", "email": "nir@aviary.com", "permissions": [ "content", "dev", "admin", "partying" ] } ● Integration tests interact with schemaless db willy nilly N
  • 29. The Facts ● Built and deployed in 3 months ● Very few struggles with MongoDB ● Seamless management ● Graceful scaling from 0 to over 20M MAUs ● Happy serverside engineers J
  • 30. The Future ● Targeted Translations ● Granular User Targeting ● PAULA for the masses N