Learn how RelateIQ takes advantage of MongoDB's tooling and oplog to drive near-realtime features and infrastructure for their customers. In this session we will cover how we exploit Morphia to make at-rest encryption of critical data both DRY and easy. We will also be covering our consumption of MongoDB's unsung hero, the oplog, which drives a significant portion of our back-end computing systems. While both projects are simple enough to be the product of Hackdays (a time-honored tradition at RelateIQ), they've blossomed into critical components of our infrastructure.
6. At-Rest Encryption
@preLoad
Called before mapping the datastore object to the
entity (POJO); the DBObject is passed as an
argument (you can add/remove/change values)
You have
● The DBObject
● A class definition (reflection!)
You don’t have
● That POJO with all the
convenience methods
@preSave
Called right before DBCollection.save() is called.
Changes made to the entity (POJO) will not be
persisted; the DBObject can be passed as an
argument (you can add/remove/change values)
You have
● The original POJO, including class
definitions and convenience
methods
7. At-Rest Encryption
SadPojo.java
public class SadPojo {
public String clientId;
public String secret;
};
SadDAO.java
public class SadDAO
extends BasicDAO<SadPojo> {
@Inject
public SadDAO(Datastore ds) {
super(SadPojo.class, ds);
}
};
8. At-Rest Encryption
Command Line
> db.SadPojo.findOne()
{
"_id": ObjectId("5445dd2b7830e8d4dff439d9"),
"clientId": "Dr. No",
"secret": "What follows are my evil plans to take over the
world..."
}
9. SketchyPojo.java
public class SketchyPojo {
public String clientId;
public String secretKey;
public String secret;
@PreSave
public void preSave(DBObject obj) {
obj.put(“secret”, encryptWithSecret(secretKey, secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
At-Rest Encryption (Take One)
14. At-Rest Encryption (Take Two)
LessSketchyDAO.java
public class LessSketchyDAO extends BasicDAO<LessSketchyPojo> {
@Inject
public LessSketchyDAO(Datastore ds) {
super(LessSketchyPojo.class, ds);
}
@PreSave
public void preSave(LessSketchyPojo pojo, DBObject obj) {
obj.put(“secret”, encryptWithSecret(pojo.secretKey, pojo.secret);
}
@PreLoad
public void preLoad(DBObject obj) {
try {
obj.put(“secret”, decryptWithSecret(obj.get(“secretKey”), obj.get(“secret”));
} catch (EncryptionException e) {}
}
};
15. At-Rest Encryption (Take Three)
HappyPojo.java
public class HappyPojo {
@EncryptionScope public String clientId;
@EncryptAtRest public String secret;
};
18. At-Rest Encryption (Take Three)
MongoModule.java
public class MongoModule extends AbstractModule {
@Provides
public Morphia providesMorphia(CrypterFactory crypterFactory) {
Morphia morphia = new Morphia();
...
morphia.getMapper().addInterceptor(
new EncryptAtRestInterceptor(crypterFactory));
return morphia;
};
19. Encrypt-At-Rest like the boss
Key Takeaways
1. Don’t Repeat Yourself.
2. Morphia’s lifecycle handling is incredibly useful, when used properly.
3. Don’t Repeat Yourself!
4. Encryption is a scary word, but we can make it as natural as annotating a POJO.
5. Use OSS to DRY your code
https://github.com/relateiq/EncryptAtRestInterceptor
1. Pull requests appreciated!
21. Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database for
analytics and data warehousing?
● Better, how about we transform
documents and feed them back into
the system!?
22. Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
23. Golden Horde
Lifecycle Methods are Awesome
● So why don’t we use it to ETL
changes to the database?
Okay… awesome-”ish”
● It only takes one guy with CLI
access, or a buggy service.
Oplog to the rescue!
● We can rely on the same system
that Mongo uses internally for
tracking and updating repls.
Who We Are
I’m a DevOps guys these days, and my passion is turning engineering productivity up to 11.
Core part of RelateIQ culture is the Hack Day, and I thought I would take this opportunity to talk to you about some of the mongo-specific projects that I’ve had the privilege of working on over the last couple of years.
Don’t need to sell MongoDB!
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Lots of DRY code, tricky to remember, need to remember the *actual* DBObject key for every encrypted value
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
OSS EncryptAtRest
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
OSS EncryptAtRest
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
OSS EncryptAtRest
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments, and
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments
Asynchronous deployment cycles for iOS, Android, Web and API requires a common communication
A persistent queue like kafka requires that messages be “backwards compatible” to withstand deployments
Evolution in thinking-style presentation.
Application-level code failed to capture developers changing things from the command line
Hack Day —> Iteration One —> Kafka —> Awesome-sauce
There were some bumps along the road, but we stuck with it and it was totally worth it.
Bumps in the road
Document-level locking (finally!)
Write-consistency quorum