2. What am I talking about?
Integrating Mongo into a mission-critical system
• EnerNOC provides a 24/ 7 system to help ensure electric grid stability
• How do we introduce Mongo to help meet scalability needs?
• How do we do so in a way that ensures stability?
2
3. Background
We provide stability to the electric grid.
How we do it:
1. Instability on the grid (peaking load, transmission line failure)
2. Grid operator sends a “demand response” signal to us
3. We send a signal to our customers (large electric consumers)
4. Customers reduce their energy usage
5. ??? We monitor customer telemetry in near-realtime
6. Profit!
3
4. Background
• We’ve experienced tremendous growth
• Software (esp. database) struggles to keep up
• Hundreds of tables
• Hibernate Inefficient queries
• Our solution? Make everything a stored proc
• Better, except…
• A few tables take up 99% of our storage
• It’s still slow
• Looking at $$millions in hardware cost to scale
4
5. What’s worse than a database that won’t scale?
Two databases that won’t scale.
5
7. Planning for Mongo
• Bucketing
• Pre-allocated stubbed out documents
• Hashed IDs for data locality
pid = 1234 # natural key
hash_pid = pid.to_s.rjust(15,'0').reverse!.to_i
_id = {
:p => hash_pid,
:s => Date.new
}
7
8. The Software Architect’s Haiku
Avoid vendor lock-in
Mongo might solve scaling problems
But increase dependencies
8
9. What did we do?
Create a Data Management Service
• Common HTTP interface for data in, data out
• Hides ID hashing & bucketing
• Interface for on-the-fly aggregation
• Caching behind the service
• Native client libraries – Java, Ruby, R
9
What I’m not talking about – High availabilityQuestion:Who uses Mongo in production today?Who has more than one front-end or app server hitting their mongo instance?
So clearly our current architecture is not sclable
So we can’t just add a second database and expect that to solve our problems. Even if Mongo can solve our problems, we need to put some thought into this to make sure we do it right.
Basically we followed known best-practices. Sharding is something you have to plan for so your ID is right.
Our buckets have a predictable size from the beginning, so it’s easy to stub them outPrevents fragmentationThe ID hashing ensures balanced shards
So even when we know Mongo is right and it will scale, adding it as the backend to every app is messy.Now all apps have logic for when to ask RDBMS and when to ask Mongo. Increased coupling == bad!
So we made a common front-end API and built that separate from the services for the underlying data sources.
Here’s why – we wanted a common layer for authentication and auditing.
Why is that important? Because when there are multiple apps hitting the services, we want a single point of entry.Auditing and SLA trackingAnd API versioning!
