Mesa is Google's data warehousing system that powers their advertising business. It provides an atomic, consistent, available, near real-time and scalable data store. Mesa stores metadata on BigTable and data on Colossus. It handles trillions of queries per day and millions per second across tens of thousands of machines globally. Mesa uses a multi-versioned data model with pre-aggregation and supports multiple indexes.

1. Google Mesa
Sameer Tiwari
Hadoop Architect, Pivotal Inc.
stiwari@pivotal.io @sameertech Aug 12, 2014

2. What is Mesa?
● Geo-Replicated, Near Real-Time, Scalable
Data Warehousing for Google’s Internet
Advertising Business.
● Ok so what is it really?
o Its an Atomic, Consistent, Available, Near Real
Time, Scalable Store

3. Salient features
● DW for Ad serving at Google
● Metadata on BigTable
● Data on Colossus
● Trillions of Queries/day, Millions/second
● Support Multiple indexes
● Runs on tens of thousands machines across
geos

4. Data Model
● Table are specified by Table Schemas
● Table Schema by, Key and Value Space
o K, V are sets
o Each is represented as column tuples
o Specifies an aggregation function
● Each Col stored separately
● For consistency updates are multi-versioned
and batched for throughput
● Data is amenable to aggregation

5. Data Model
● Pre-aggregates data into Deltas (no
repeated row keys/delta) and applies a
version
● Compaction is multi-level
● A Controller handles updates/ maintenance,
works with BigTable

6. Controller
● 4 sub-systems
o Updates
o Compaction
o Checksum
o Schema change
● Does not do any work, only schedules it

7. Storage and Indexes
- AO, log structured, read-only
- Rows organized as compressed row-blocks
- Indexes have starting entry of the row-block
- Naive lookup
- Binary Search on index to find row-
blocks
- Binary Search on the row-blocks

8. Query sub system
● Limited Query engine with Filtering/Predicate
● Used by higher level systems
Dremel/MySQL
● Has multiple stateless Query Servers
● Works on both the BigTable and Colossus
● Provides nice sharding and LB mechanism
● Groups similar queries to a subset of
Servers

9. Multi Datacenter Deployment
● Tables are multi-versioned
o (Serve old data while new is in-progress)
● Committer is stateless and sends updates to
multiple Datacenters
o Built on top of versionsDB. - Globally replicated and
consistent store build on top of distributed Paxos.
● Data goes async across Mesa instances
● Only Metadata is sync-repl using Paxos-
versionsDB

10. Optimizations
● Delta pruning - similar to Filter pushdown
● Resume-Key, Key per data block
o Data is returned a block at a time, so if a
QueryServer dies, another one can pick it up.
● Parallelizing workloads: Uses MR to shard
o While writing delta, Mesa sample row-keys which is
used to figure out the right number of
Mappers/Reducers.
o The workers are the same 4 workers scheduled by
the Controller

11. Optimizations
● Schema changes - two techniques
o Create, Copy, Replay and delete - Expensive
o Link and add default values - This is used in Mesa
● New Instances of Mesa use P2P
mechanisms to come up and online.

12. Handling Data Corruption
● Mesa runs on ~50K boxes
● Online - During updates.
o Fact: Each Mesa instance is logically same but
physically may differ in deltas
o Check chksums of indexes/data
o Row-order, key-range, aggregate values should be
same, across instances
● Offline
o Run global chksums of all indexes

13. Reference
http://static.googleusercontent.com/media/rese
arch.google.com/en/us/pubs/archive/42851.pdf