YouGov uses MongoDB to store semi-structured survey response data across a globally distributed sharded cluster. They implement tag-aware sharding to partition data by region and leverage migration managers to update schemas across versions. This allows them to provide low-latency reads, scale throughput, and support dynamic surveys worldwide through MongoDB-as-a-Service.

1. FROM THE POLLS TO
THE TROLLS
SEEING WHAT THE WORLD THINKS AT
YOUGOV

2. YouGov conducts highly dynamic surveys that rely on
MongoDB's rich support for semi-structured data.
YouGov uses tag-aware sharding to provide
MongoDB-as-a-Service (MongoDBaaS) for a global
enterprise.
MongoDB is a joy to use and an effective
multipurpose persistence store for large-scale
applications.

3. ABOUT US

4. ABOUT YOUGOV
community
opinions & habits
smart analysis
insights and prediction
today.yougov.com

6. ABOUT YOUGOV DEV
38 developers in 9 countries
primarily leverage Python and open-source
high interactivity and integration
streamlined deployment

7. ABOUT JASON
<3 Python
studied web data (semi-structured)
motivated to unfetter data

8. TWO CHALLENGES

9. CHALLENGE: SURVEY COLLECTION
highly dynamic surveys
programmed by users
semi-structured data
high throughput
increasing throughput

10. CHALLENGE: GLOBAL OPERATION

11. FLEXIBLE DATA MODEL
store survey responses as document
(automatic) schema migration

12. MIGRATION MANAGER
between driver and application
schema version
incrementing integer
migrates on load
untouched on save

13. MIGRATION MANAGER

22. IMPLEMENTATION (1)
class Manager:
"""
A manager for facilitating the registration of migration functions
and applying those migrations to documents.
"""
version_attribute_name = 'version'
_upgrade_funcs = set()
def __init__(self, target_version):
self.target_version = target_version
@classmethod
def register(cls, func):
"""
Decorate a migration function with this method
to make it available for migrating cases.

23. IMPLEMENTATION (2)
class Manager:
def migrate_doc(self, doc):
"""
Migrate the doc from its current version to the target version
and return it.
"""
orig_ver = doc.get(self.version_attribute_name, 0)
funcs = self._get_migrate_funcs(orig_ver, self.target_version)
for func in funcs:
func(self, doc)
doc[self.version_attribute_name] = func.target
return doc
@classmethod
def _get_migrate_funcs(cls, orig_version, target_version):
direction = 1 if target_version > orig_version else -1
versions = range(orig_version, target_version + direction, direction)

24. IMPLEMENTATION (3)
@Manager.register
def v6_to_7(manager, case):
"""
V7 adds the survey_id and user_id fields
"""
ident = CaseIdent(case.get('ident', ''))
case.setdefault('user_id', ident.user_id)
case.setdefault('survey_id', ident.survey_id)
@Manager.register
def v7_to_6(manager, case):
ident = CaseIdent.from_parts(
case.pop('user_id', None),
case.pop('survey_id', None),
)
case['ident'] = str(ident)

25. CAUTION: FIELD RENAME
queries
indexes

26. MITIGATION
support both fields
maintain dual indexes
backfill migration

27. OTHER CONSIDERATIONS
single-document design
forward-compatibility
reversible operations

28. IMPLEMENTATION (4)
@Manager.register
def v10_to_11(manager, case):
"""
Remove legacy attributes.
"""
case.pop('survey_name', None)
case.pop('survey_version', None)
case.pop('ident', None)
https://github.com/jaraco/jaraco.mongodb

29. GLOBAL SHARDED CLUSTER

30. SHARDING STRATEGY
2 datacenters separated by 5,400 miles
partitions survey data via tag-aware sharding
provides Database-as-a-Service (DBaaS)

31. INTERVIEW DATA MODEL

36. INTERVIEW LIFE CYCLE

37. INTERVIEW DATA SCHEMA
{
"region": "EMEA",
"survey_id": "rfc1274",
"user_id": 216074,
"responses": {
"favourite_drink": "chocolate milk",
// ...
}
}

38. SHARD KEY SELECTION

39. OOPS!

40. LET'S TRY THAT AGAIN...

41. MUCH BETTER!

42. CAUTION: BROADCAST OPERATIONS

43. TARGETED OPERATIONS

44. TWO DATACENTERS
London (a.k.a. "EMEA")
Palo Alto, CA (a.k.a. "US")

45. DEPLOYING SHARDS

47. GLOBAL SHARDED CLUSTER

49. TAG-AWARE SHARDING (1)
// connect to mongos
// switch to admin database
use admin
// Step 1: Add shards
db.runCommand({addShard: "ldn-rs1/ldn-host1", name: "LDN1"});
db.runCommand({addShard: "ldn-rs2/ldn-host4", name: "LDN2"});
db.runCommand({addShard: "ldn-rs3/ldn-host7", name: "LDN3"});
db.runCommand({addShard: "pa-rs1/pa-host1", name: "PA1"});
db.runCommand({addShard: "pa-rs2/pa-host4", name: "PA2"});

50. TAG-AWARE SHARDING (2)
// Step 2: Add shard tags
sh.addShardTag("LDN1", "EMEA");
sh.addShardTag("LDN2", "EMEA");
sh.addShardTag("LDN3", "EMEA");
sh.addShardTag("PA1", "US");
sh.addShardTag("PA2", "US");

51. TAG-AWARE SHARDING (3)
// Step 3: Enable sharding
sh.enableSharding("yougov");
sh.shardCollection(
"yougov.interviews", // collection namespace
{"region": 1, "survey_id": 1} // shard key pattern
);

52. TAG-AWARE SHARDING (4)
// Step 4: Tag each shard key range
sh.addTagRange(
"yougov.interviews", // collection namespace
{"region": "EMEA", "survey_id": MinKey}, // min value
{"region": "EMEA", "survey_id": MaxKey}, // max value
"EMEA" // tag
);
sh.addTagRange(
"yougov.interviews", // collection namespace
{"region": "US", "survey_id": MinKey}, // min value
{"region": "US", "survey_id": MaxKey}, // max value
"US" // tag
);

53. (MONGO) DATABASE-AS-A-SERVICE
mongos instance running on each app server
applications connect to mongodb://localhost/

54. READ PREFERENCE "NEAREST"
provides low-latency reads when using geographically
distributed replica sets
from pymongo import MongoClient, ReadPreference
# set read preference on the client
mongo = MongoClient('localhost',
read_preference=ReadPreference.NEAREST)
# ...or on an individual database
mongo = MongoClient('localhost')
db = mongo.get_database('dragoman',
read_preference=ReadPreference.NEAREST)

55. DATABASE CREATION
def create_db_in_shard(db_name, shard, client=None):
"""
In a sharded cluster, create a database in a particular shard.
"""
client = client or pymongo.MongoClient()
# flush the router config to ensure it's not stale
res = client.admin.command('flushRouterConfig')
if not res.get('ok'):
raise RuntimeError("unable to flush router config")
if shard not in get_ids(client.config.shards):
raise ValueError(nf("Unknown shard {shard}"))
if db_name in get_ids(client.config.databases):
raise ValueError("database already exists")
# MongoDB doesn't have a 'create database' command, so insert an
# item into a collection and then drop the collection.
client[db_name].foo.insert({'foo': 1})
client[db_name].foo.drop()

56. MULTI-DIMENSIONAL PARTITIONING
still partition for archival
managed programmatically
seek additional layers of partitioning

57. PARTITIONING MOTIVES
locality
distribution (size)
distribution (performance)
storage

58. PREDICTIONS
migrate with whimsy
all apps in global cluster

59. WISHES
Supply shard hint, independent from query
Generalized custom shard logic, independent from data in
records
Oplog-only replica for creating new replicas
Embedded solution (sqlite)
SERVER-11991
Server-14539

60. CONCLUSIONS
If you have dynamically changing data, choose a
schemaless database. Take advantage of the schemaless
aspects and migrate with whimsy.
When your company is successful and international, use
tag-aware sharding to achieve partitioning of your data
without losing control, but let MongoDB manage the
abstraction.

61. QUESTIONS?
https://bit.ly/polls2trolls