Handling Changes to Your Server-Side Data Model

Evan Coyne Maloney Principal Engineer
Mobile
Gracefully handling changes
to your server-side data model

Mobile
Things change.
…sometimes more often than we’d like.

Mobile
Handling change on the Web
• No app review process

• “Go live” with the push of a button

• Immediate deployment to production

• All users get changes right away

• No need to test “on device”

• Won’t spend hours in provisioning hell

• Easy to perform automated testing on the hardware you’ll
be using in production
Handling change Web vs. iOS

Mobile
Handling change on iOS
• Mandatory app review

• Can take a week or more

• Must test “on device”

• If you only test in the simulator, you don’t really know
what your code is doing

• How does it perform?

• It is exhibiting device-only crashing?

Mobile
Web vs. iOS
Web iOS
Fully automated testing Partially manual testing
Push a button and go live
Going live requires Apple’s permission,
which in turn requires waiting
Can fix stuff right away
Being able to fix stuff right away requires
careful planning and lots of work

(…and even then you can’t fix everything)

Mobile
These diﬀerences
have consequences
Your iOS apps can become an anchor

on your server-side environment

Mobile
Chirper
Handling change An Example
A hypothetical social network
• Unidirectional relationships between users (following)

• Encourages very short messages; optimized for mobile input

• Allows followers to tag a message to categorize it

Mobile
Chirper
A hypothetical social network
• Launched by an early-stage startup with limited resources

• Built a simple system to launch quickly

• Having trouble scaling

• Need to bring hosting and bandwidth costs down

Mobile
Chirper’s Message Feed
• The feed contains all messages that:

• Are new to the user

• Were delivered previously, but have newly-applied tags
[{
"message_id": 10271972,
"message": "Is there going to be an encore? I really hope they play Penelope!",
"author_id": 5375,
"tags": ["Pinback", "concert"]
}]
• When the Chirper app launches, it retrieves the updated message feed from
the server

• Messages in the feed arrive within a JSON array:

Mobile
Not the most eﬃcient
• Any time anyone tags a message, the entire message will be
re-sent in the feed update, even though previously-received
messages are already stored in the client app

• Tags themselves can add a lot of redundant data, especially
in heavily active feeds with long tag names

• Chirper has decided to refactor their feeds to be more
eﬃcient

Mobile
An example scenario
1. Someone adds a new tag to an old message
2. Two new messages arrive

Mobile
How it looks in the current feed
[{
"message_id": 10271972,
"content": "Is there going to be an encore? I really hope they play Penelope!",
"author_id": 5375,
"tags": ["Pinback", "concert", "live music"],
"is_tag_update": true
},
{
"message_id": 10272003,
"content": "I like the live Penelope so much better than the album version.",
"author_id": 6022,
"tags": ["Pinback", "concert", "live music"]
},
{
"message_id": 10272012,
"content": "They played that last night. Expect June or Grey Machine tonight.",
"author_id": 4216,
}]
503 bytes (compact JSON)

Mobile
A refactored feed
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"author_id": 6022
},
"10272012": {
"author_id": 4216
}
}
}
344 bytes (compact JSON)

Mobile
Success?
• The response size is now about ⅔ of what it was

• Savings will be even better under many scenarios
Great, but…
These savings won’t lower Chirper’s bandwidth costs until:

1. They build and submit a new app

2. Apple approves it

3. A critical mass of users update to the latest version

Mobile
The root of the problem
Handling change Designing for change
• The client app and the server must always agree on the
structure of the data

• On the client side, this knowledge has to be compiled in to the
application binary

• Once the app has been submitted, it can’t be modiﬁed

• But, we can design a way around this!

Mobile
How we build server-driven apps today
1. The server maintains knowledge of the data necessary to
drive the app

2. The app requests data, and receives a data structure in
response

3. The app will “ask questions of the data” to do its work

4. This requires having knowledge of the data structure

Mobile
How we could build server-driven apps
1. The server maintains knowledge of the data necessary to drive
the app

2. The app requests data, and receives in response:

• The data itself

• A set of instructions for how to extract values from the data structure

3. The app will still “ask questions of the data” to do its work, but it
will use the server-provided instructions to do it — the client no
longer needs advance knowledge of the server’s data model!

Mobile
1. The exact structure of the
data

2. What questions it needs
answered from the data

3. How to translate those
questions into procedures for
extracting answers from the
data
The Client needs to know: The Server needs to know:
data
Rethink where we require knowledge

Mobile
Rethink where we require knowledge
1. What questions it needs
answered from the data
data

2. The set of questions each
app version needs answered
from the data

3. How to translate those
questions into procedures for
extracting answers from the
data
The Client needs to know: The Server needs to know:

Mobile
How can we do this?

Mobile
Introducing
Mockingbird Expressions

Mobile
• Text strings that specify values within an iOS application’s
runtime environment
• Objective-C and Swift variables

• Server-side data models

• Device information

• Screen size, system clock

• OS-level information

• iOS version, system permissions, user settings

• Provided by the Mockingbird Data Environment library
Mockingbird Expressions An Introduction
What are they?

Mobile
What do they look like?
Variables Functions
The expression: The expression:
$foo ^currentTime()
Returns the object bound to the Mockingbird variable
name “foo”.
Returns an NSDate object containing the current time.
$dict[key].propVal ^formatLongDate(^currentTime())
Selects the object associated with key from the
NSDictionary bound to the name “dict”, and returns
the value of its propVal property.
Returns an NSString containing the current date in the
format “August 11, 2014”.
$UIDevice.model ^filter($users|$item.firstName == Jill)
Equivalent to the Objective-C code:

[[UIDevice currentDevice] model]
Returns a subset of the collection $users containing
only the items whose firstName property equals “Jill”.

Mobile
We can use
to describe how to extract
answers from data

Mobile
1. What are the IDs of the new messages?

2. What are the IDs of the messages with newly-
applied tags?

3. What is the content of a given message?

4. Who is the author of a given message?

5. Which tags are associated with a given message?
Mockingbird Expressions A Use Case
Questions Chirper asks its data

Mobile
[{
"message_id": 10271972,
"author_id": 5375,
},
{
"message_id": 10272003,
"author_id": 6022,
},
{
"message_id": 10272012,
"author_id": 4216,
}]
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"content": "I like the live Penelope so mu
"author_id": 6022
},
"10272012": {
"content": "They played that last night. E
"author_id": 4216
}
}
}
1. What are the IDs of the new messages?
^list(^filter($feed|!$item.is_tag_update)|$item.message_id)Old feed
New feed^list($feed.messages|$key)

Mobile
2. What are the IDs of the messages with newly-applied tags?
[{
"message_id": 10271972,
"author_id": 5375,
},
{
"message_id": 10272003,
"author_id": 6022,
},
{
"message_id": 10272012,
"author_id": 4216,
}]
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"author_id": 6022
},
"10272012": {
"author_id": 4216
}
}
}
^list(^filter($feed|$item.is_tag_update)|$item.message_id)
^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])
Old feed
New feed

Mobile
3. What is the content of a given message?
[{
"message_id": 10271972,
"author_id": 5375,
},
{
"message_id": 10272003,
"author_id": 6022,
},
{
"message_id": 10272012,
"author_id": 4216,
}]
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"author_id": 6022
},
"10272012": {
"author_id": 4216
}
}
}
^associate($feed|$item.message_id|$item.content)[$id]
$feed.messages[$id].content
Old feed
New feed

Mobile
4. Who is the author of a given message?
[{
"message_id": 10271972,
"author_id": 5375,
},
{
"message_id": 10272003,
"author_id": 6022,
},
{
"message_id": 10272012,
"author_id": 4216,
}]
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"author_id": 6022
},
"10272012": {
"author_id": 4216
}
}
}
^associate($feed|$item.message_id|$item.author_id)[$id]
$feed.messages[$id].author_id
Old feed
New feed

Mobile
5. Which tags are associated with a given message?
[{
"message_id": 10271972,
"author_id": 5375,
},
{
"message_id": 10272003,
"author_id": 6022,
},
{
"message_id": 10272012,
"author_id": 4216,
}]
{
"message_tags": {
"10271972": [ 0, 1, 2 ],
"10272003": [ 0, 1, 2 ],
"10272012": [ 0, 1, 2 ]
},
"messages": {
"10272003": {
"author_id": 6022
},
"10272012": {
"author_id": 4216
}
}
}
^associate($feed|$item.message_id|$item.tags)[$id]
^list($feed.message_tags[$id]|$feed.tags[$item])
Old feed
New feed

Mobile
{
"new_message_ids": "^list(^filter($feed|!$item.is_tag_update)|$item.message_id)",
"newly_tagged_message_ids": "^list(^filter($feed|$item.is_tag_update)|$item.message_id)",
"content_for_message_with_id": "âssociate($feed|$item.message_id|$item.content)[$id]",
"author_id_for_message_with_id": "âssociate($feed|$item.message_id|$item.author_id)[$id]",
"tags_for_message_with_id": "âssociate($feed|$item.message_id|$item.tags)[$id]"
}
Old feed
We can now separate out the
expressions needed to query the data
{
"new_message_ids": "^list($feed.messages|$key)",
"newly_tagged_message_ids": "^filter(^list($feed.message_tags|$key)|!$feed.messages[$item])",
"content_for_message_with_id": "$feed.messages[$id].content",
"author_id_for_message_with_id": "$feed.messages[$id].author_id",
"tags_for_message_with_id": "^list($feed.message_tags[$id]|$feed.tags[$item])"
}
New feed

Mobile
This architecture now relies on

“model/query decoupling”
wherein
The ability to query a set of data
is separated from
knowledge of that data’s structure

Mobile
How would this work in practice?

Mobile
1. The server maintains a schema version for the data model it uses

• Whenever a non-backwards-compatible schema change is introduced, the
schema version increments

2. Any time the schema version changes, the server publishes a
corresponding set of query expressions for that schema version

3. When an app fetches new data, it checks the schema version
returned by the server. If the app doesn’t have the query expressions
for that version, it can get them from the server

• Apps will cache query expressions

• Apps can include a compiled-in set of query expressions to avoid having to
download them the ﬁrst time an app launches
Mockingbird Expressions Code Samples
Model/Query Decoupling
using Mockingbird Expressions

Mobile
1. When we retrieve feed data from the server, we expose it to the
variable space:
Populating the
Mockingbird Data Environment
NSData* json = ... // Unicode text containing JSON structure from server
id feedData = [NSJSONSerialization JSONObjectWithData:json options:0 error:nil];
[[MBVariableSpace instance] setVariable:@"feed" value:feedData];
The expression $feed now refers to the value of feedData
2. We load the query expressions associated with the current schema into a
dictionary and expose it to the variable space:
NSDictionary* queries = ... // query names -> expressions
[[MBVariableSpace instance] setVariable:@"queries" value:queries];
The expression $queries now refers to the NSDictionary queries

Mobile
3. Query for new messages:
Querying the data
NSString* query = [MBExpression asString:@"$queries.new_message_ids"];
NSArray* newMessageIDs = [MBExpression asObject:query];
The array newMessageIDs contains two elements:
( 10272003, 10272012 )
4. Query for messages with updated tags:
NSString* query = [MBExpression asString:@"$queries.newly_tagged_message_ids"];
NSArray* newlyTaggedMessageIDs = [MBExpression asObject:query];
The array newlyTaggedMessageIDs contains one element:
( 10271972 )
5. Get the content of a message given a message ID:
[[MBVariableSpace instance] setVariable:@"id" value:@(10272003)];
NSString* query = [MBExpression asString:@"$queries.content_for_message_with_id"];
NSString* content = [MBExpression asNumber:query];
The string content is “I like the live Penelope so much better than the album version.”

Mobile
Querying the data
7. Get the tags associated with a given message:
NSString* query = [MBExpression asString:@"$queries.tags_for_message_with_id"];
NSArray* tags = [MBExpression asObject:query];
The array tags contains three elements:
( "Pinback", "concert", "live music" )
6. Get the author ID of a message given a message ID:
NSString* query = [MBExpression asString:@"$queries.author_id_for_message_with_id"];
NSNumber* authorID = [MBExpression asNumber:query];
The NSNumber authorID contains the integer value 5375.

Mobile
What can you use this for?

Mobile
1. No App Store update required to introduce support for new
schemas
• New sets of query expressions can be downloaded to add immediate support for new
schemas

2. Maintain legacy support while evolving your services
• You won’t need to run multiple versions of your backend just to maintain legacy support

3. Simplify your backend codebase
• Avoid spaghetti code — no need for version-check conditionals littered throughout your
server-side codebase

4. A/B testing made easy
• Want to support multiple schemas simultaneously? No problem! Just switch between sets of
query expressions. This makes server-driven A/B testing easy.

Mockingbird Expressions Use Cases
Beneﬁts

Mobile
Mockingbird Availability
Mockingbird Data Environment availability
The Mockingbird Data Environment is part of the
Mockingbird Library open-source project from Gilt Groupe.
Coming Fall 2014

Mobile
Mockingbird Data Environment availability
Available today!
Coming Fall 2014
The Mockingbird Data Environment is part of the
Mockingbird Library open-source project from Gilt Groupe.

Mobile
Mockingbird Toolbox
open source project
https://github.com/gilt/mockingbird-toolbox
available at:

Handling Changes to Your Server-Side Data Model

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Handling Changes to Your Server-Side Data Model

Similar to Handling Changes to Your Server-Side Data Model (20)

More from Gilt Tech Talks

More from Gilt Tech Talks (11)

Recently uploaded

Recently uploaded (20)

Handling Changes to Your Server-Side Data Model