The document discusses best practices for data management in MongoDB, focusing on data validation, redaction, and quality of service (QoS) features. It outlines the architecture for implementing a validation and post-query operation model using Java, alongside various techniques for managing and optimizing data access. Additionally, it provides examples of how to integrate these features with a web service, emphasizing an organized and robust approach to data governance.

1. Buzz Moschetti
Enterprise Architect
buzz.moschetti@mongodb.com
@buzzmoschetti
Bulletproof
Data Management

2. 2
Part 3 In The Data Management Series
Validating Data
Software Best Practices
Safe Leverage
From Relational
To MongoDB
Conquering
Data Proliferation
Bulletproof
Data Management
ç
Ω
Part
1
Part
2
Part
3

3. 3
Congratulations! At this Point You’ve:
• Created a Data Design
• Migrated Data
• Built a PoC or maybe an App
• Explored Operations

4. 4
The Next Stage: Defend & Leverage!
• Document Validation
• Redaction
• Quality Of Service

5. 5
MongoDB Doesn’t Have These Things
• Document Validation
• Redaction
• Quality Of Service

6. 6
What Can We Do Today?

7. Write Some Code!
1. Focus on interfaces
2. Design for change
3. Keep application, data access layer,
data management logic, and database
i/o well-factored
4. Minimize compile-time binding

8. 8
Starting Point: The Data Access Layer
MongoDB
Java Driver
Data Access
Layer
Application
class DataAccessLayer {
private String authenicatedID;
private String effectiveID;
private Role role;
init() {
MongoClient mc = new MongoClient (args);
DB db = mc.getDB(args);
}
List getTransactions(Map predicate) {
Map mql = doWhateverYouNeed(predicate);
DBCollection coll = db.get(“TX”);
DBCursor c = coll.find(mql);
while(c.hasNext()) {
Map raw = (Map) c.getNext();
Map morphed = myMorphingLogic(raw);
list.add(morphed);
}
return list;
}
}

9. 9
Document Validation

10. 10
A Query Filters Outbound Data
{$and:[{“name”:”buzz”},{“prefs”:{$exists:true}}]

11. 11
How About Using It To Filter Inbounds?
{$and:[{“name”:”buzz”},{“prefs”:{$exists:true}}]}

12. 12
$exists And $type Already in MQL
{“name”:{$type:2}}
{$or:[{“age”:{$exists:false}}, {“age”:{$type:16}} ]}
{$and: [
{$name: {$type:2}},
{$or:[
{$and:[{"weight”:{$type:16}}, {"height":{$type:16}}]}
,{$and:[{"weight”:{$exists:0}}, {"height":{$exists:0}}]}
]}
])
Ensure “name” exists (because not null) and is a string:
“age” optional but if exists must be a 32bit integer:
“name” required as string and weight and height both
required integers or both not present:

13. 13
… And MQL Goes Way Beyond…
{$or:[
{$and:[
{“name”: {$type:2}},
{“numClues”: {$gt: 0}}, {“numClues”:{$type:16}},
{“birthday”: {$type: 9}},
{“hiredate”: {$type: 9}},
{$or: [{“prefs”:{$exists:false}},
{“prefs”:{$type:3}} ]
}
]
},
{“name”: {$exists:false}}
]
}

14. 14
A New MQL Validator Module Emerges
class MQLValidator {
ValidationResult validate(Map MQL, Map data)
}
MongoDB
Java Driver
Data Access
Layer
Application
Validator NOT inline to MongoDB driver
• Interface too big to create a façade
• Beware of “tall stacks”
MQLValidator

15. 15
MongoDB
DB Engine
Migrating Capability into MongoDB
MongoDB
Java Driver
MQLValidator
Java
Data Access
Layer
MongoDB
DB Engine
MongoDB
Java Driver
MQLValidator
Java
Data Access
Layer
• Coming in v3.2!
• Investment in validation design preserved
• Validation enforceable through ALL drivers
and languages
MongoDB
Python Driver
Application Application

16. 16
Code For The Future…Today
class DataAccessLayer {
someWriteOperation(Map data) {
if(ValidationEnabledInMongoDBengine) {
collection.insert(data); // Not yet
} else {
Map mql = getMQL(); // we’ll see this shortly!
// {$or:[{“age”:{$exists:false}},
// {“age”:{$type:16}}]}
ValidationResult vr = MQLValidator.validate(mql,data);
if(vr.ok()) {
collection.insert(data);
}
}
}
}

17. 17
But What About Today?
MQLValidator

18. 18
Temporary Filling: PQL
• (P)refix (Q)uery (L)anguage
• Database independent filter of Maps
• Similar to MQL
• 450 lines of Java
• moschetti.org/rants/PQL.html
PQL

19. 19
Bridge MQL to PQL
class MQLValidator {
private PQLFilter pqlfilter;
validate(Map mql, Map data) {
boolean rc;
if(MQLValidationAvailableAsLibrary) {
rc = ActualMongoDBMQL.validate(mql, data);
} else {
Map pqlfilter = convertMQLtoPQL(mql);
// {or:[{“null”: “age”},
// {“type”: {“age”: “INT”}}]}
rc = pqlfilter.eval(pql, data);
}
return rc;
}
Map convertMQLtoPQL(Map mql) { // ~200 lines }
}

20. 20
No PQL? No Problem
class MQLValidator {
validate(Map mql, Map data) {
boolean rc;
if(MQLValidationAvailableAsLibrary) {
rc = ActualMongoDBMQL.validate(mql, data);
} else {
SomeType yo = convertMQLtoYourThing(mql);
rc = YourFilter(yo, data);
}
return rc;
}
SomeType convertMQLtoYourThing(Map mql) { . . . }
}

21. 21
MQL Is Easy To Navigate
{$or:[
{$and:[
{“name”: {$type:2}},
{“numClues”: {$gt: 0}}, {“numClues”:{$type:16}},
{“birthday”: {$type: 9}},
{“hiredate”: {$type: 9}},
{$or: [{“prefs”:{$exists:false}},
{“prefs”:{$type:3}} ]
}
]
},
{“name”: {$exists:false}}
]
}
• “Walk”, not “parse”
• Operators distinct from operands
• Operands are native type (e.g. Date)

22. 22
Where Do Validations Come From?
The Database!

23. 23
The Validations Collection
> db.validations.find()
{
“collectionName”: “product”,
“validations”: [
{ “name”: “simple”, “type”: “MQL”, “expr”:
{__$or:[{“age”:{__$exists:false}},
{“age”:{__$type:16}} ]}
]
}
{
“collectionName”: “transaction”,
“validations”: [
{ “name”: “frontOffice”, “type”: “MQL”, “expr”:
{ … lots of MQL here …}
}
]
}

24. 24
Various “Levels” of Validation
> db.validations.find()
{
“collectionName”: “foo”,
“defaultValidation”: “initialSetup”,
“validations”: [
{“name”: “initialSetup”, …},
{“name”: “frontOffice”, …},
{“name”: “middleOffice”, …},
{“name”: “backOffice”, …}
]
}

25. 25
Multiple Types: Schema By Example
> db.validations.find()
{
“collectionName”: “foo”,
“validations”: [
{ “name”: “simple”,
“type”: “SBE”,
“expr”:
{ “name!”: “string”,
“age”: “integer”,
“petNames”: [ “string” ],
“bday!”: “date”
}
}
]
}

26. 26
The Stack So Far
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Application
ValidatorDBUtils
ValidatorDBUtils populates an MQLValidator object from MongoDB
PQLFilter

27. 27
Representative Example
class DataAccessLayer {
MQLValidator vv = new MQLValidator(); // NOT DB dependent!
init() {
DB db = mongoClient.getDB( ”mydb" );
ValidatorDBUtils.populate(vv, db); // db.validations
}
someWriteOperation(Map data) {
if(ValidationEnabledInMongoDBengine) {
collection.insert(data); // Not yet
} else {
String vn = “appropriateValidationRulesName”;
ValidationResult vr = vv.validate(collname, vn, data))
if(vr.ok()) {
collection.insert(data);
}
}
}
}

28. 28
Redaction

29. 29
Concept: Post Query Operations (PQO)
{ ssn: { $hash: model }, birthdate: null }
{$and:[{“name”:”buzz”},{“prefs”:{$exists:true}}]

30. 30
Adopt MQL-like behavior
{“ssn”:null}
{“address”: “XXXX”}
{“ssn”: { $substitute: “ssnmodel” }}
Remove field by setting to null
Redact address with fixed value
Substitute SSN with a different, correct, consistent value
{“counterparty”: { $hash: “MD5” }}
Hash counterparty name to consistent value

31. 31
A New PostQuery Module Emerges
class PostQuery {
process(Map data, Map operations)
}
PostQuery
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Application
ValidatorDBUtils
PQLFilter

32. 32
Where Do PQOs Come From?
The Database!

33. 33
The Postquery Collection
> db.postquery.find()
{
“collectionName”: “product”,
“operations”: [
{ “name”: “basicPI”, “type”: “PQO”, “expr”:
{“ssn”:null}
}
]
}
{
“collectionName”: “customerIndo”,
“operations”: [
{ “name”: “personalData”, “type”: “PQO”,
“expr”:
{ … lots of PQO here …}
}
]
}

34. 34
The Stack Is Getting Rich
PostQuery
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Application
ValidatorDBUtils
PQLFilter
PQODBUtils

35. 35
Representative Example
class DataAccessLayer {
MQLValidator vv = new MQLValidator(); // NOT DB dependent!
PostQuery pp = new PostQuery();
init() {
DB db = mongoClient.getDB( ”mydb" );
ValidatorDBUtils.populate(vv, db);
PQODBUtils.populate(pp, db);
}
someWriteOperation(Map data) { … }
someReadOperation(Map pred) {
Map mql = convertToMQL(pred);
Map data = collection.find(mql);
String pqon = mapRoleToRulesName();
pp.process(collname, pqon, data); // in place update
return data;
}
}

36. 36
Quality of Service

37. 37
QOS In Action
class DataAccessLayer {
someReadOperation(Map pred) {
if(qos.blackout(“someReadOperation”))
throw QOSOperationDenied;
Map mql = convertToMQL(pred);
int ms = qos.getMaxTime(“someReadOperation”, role);
Map data =
collection.find(mql).maxTime(ms,TimeUnit.MILLISECONDS);
String pqon = “appropriatePQORulesName”;
pp.process(collname, pqon, data); // in place update
return data;
}
}

38. 38
Where Do We Store QOS Values?
The Database!

39. 39
The QOS Collection
> db.qos.find()
{
“collectionName”: “product”,
“qos”: [
{ “function”: “someReadOperation”,
“rule”: “std”,
“maxtime”: 250 },
{ “function”: “someReadOperation”,
“rule”: “reporting”,
“blackout”: { “start”: “08:00”, “end”: “17:00”},
“maxtime”: 2000}
“ … ”
}
]
}

40. 40
QOSDBUtils
Coming Together…
PostQuery
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Application
ValidatorDBUtils
PQLFilter
PQODBUtils
QOS

41. 41
Representative Example
class DataAccessLayer {
MQLValidator vv = new MQLValidator(); // NOT DB dependent!
PostQuery pp = new PostQuery();
QOS qs = new QOS();
init() {
DB db = mongoClient.getDB( ”mydb" );
ValidatorDBUtils.populate(vv, db);
PQODBUtils.populate(pp, db);
QOSDBUtils.populate(qs, db);
}
someReadOperation(Map pred) {
Map mql = convertToMQL(pred);
String role = getRole(); // somehow
int maxms = qs.getMaxTime(“someReadOperation”, role);
Map data = collection.find(mql).maxtTime(maxms, tu);
String pqon = “appropriatePQORulesName”;
pp.process(collname, pqon, data); // in place update
return data;
}
}

42. 42
QOSDBUtils
A Highly Leveragable Investment
PostQuery
MQLValidator
Data Access
Layer 1
Application1
ValidatorDBUtils
PQLFilter
PQODBUtils
QOS
Application2
Data Access
Layer 2
Application3
Application4
Data Access
Layer 3
Application5
Application6
Reusable For ALL Data Access Layer Logic

43. 43
Not Just Java? Not A Problem
DAL operations have little or no state…
Data and MQL and diagnostics easily
and losslessly converted to and from
JSON…
Can you say … Web Service!

44. 44
A Really Nice Stack
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Java
Application
ValidatorDBUtils
PQLFilter
HTTP Endpoint
python
Application
curl
JSON over HTTP(S)
JSON<->Java Maps
QOSDBUtils
PostQuery
PQODBUtils
QOS

45. 45
What Can We Do?

46. 46
Secure Access To Redacted Data for Testing
$ curl –o contacts.json
-H X-Portal-Id:testID
-H X-Portal-PW:thePassword
https://refdata:8080/customers?op=find&predicate=‘{“n
ame.last”: “Jones”}’
$ head -1 contacts.json
{ “name”: { “first”:”Bob”, “last”:”Jones” },
“location”:”NA-EAST”, “ssn”: “000-00-0000”,
“hiredate”: {“$date”, “2015-04-22T17:04:54.580-0400”}}
$ mongoimport –-host testHost –d testdb –c contacts
contacts.json
15 items inserted

47. 47
Get It Programmatically, Too
// This JSON parser observes MongoDB type metadata
// conventions e.g. {“$date”, “2015-04-22T17:04:54.580-0400”}
import com.mongodb.util.JSON
getData() {
String url = "https://refdata:8080/customers…”
URLConnection con = new URL(url).openConnection();
InputStream response = con.getInputStream();
BufferedReader in = new BufferedReader(response);
String doc;
while((doc = in.readLine()) != null) {
Map data = JSON.parse(doc);
// data.ssn = “000-00-0000”
// date.hiredate = java.util.Date 2015-04 …
// data.name.first = “Bob”;
}
}

48. 48
Robust, Validated Data Ingest
$ curl –d @trades.json
-H X-Portal-Id:prodadm
-H X-Portal-PW:thePassword
https://refdata:8080trades?op=load
-o response.json
$ cat response.json
{ “assignedBatchID”: “B123”,
“numItemsExamined”: 13245,
“numItemsInserted”: 13242,
“numItemsRejected”: 3,
“errors”: [ { type: “valfail”, rule: “front … ],
“batchMD5”: “e19c1283c925b3206685ff522acfe3e6”
}

49. 49
Concept: The control_ Collection
> show collections
books
control_
customer
firms
> db.control_.find()
{
“collectionName”: “product”,
“qos”: [ … ],
“validations”: [ … ]
“operations”: [… ]
}
• Single namespace for capabilities
• Easier to add new capabilities
• Tighter (therefore better) security/entitlement

50. 50
Validation, QOS, and PQO via Web Services
MongoDB
Java Driver
MQLValidator
Data Access
Layer
Java
Application
ValidatorHTTPUtils
PQLFilter
python
Application
curl
JSON over HTTP(S)
QOSHTTPUtils
PostQuery
PQOHTTPUtils
QOS
HTTP Service
JSON<->Java Maps

51. 51
Are We Excited Yet?
Contact me or MongoDB for
• Beta program for 3.2 features
• Access to MQLValidator, PQO,
and other Java resources

52. Questions & Answers

54. 54
Concept: DataProvider
public interface DataProvider {
init();
fetch(String collection, Map mql);
insert(String collection, Map data);
update(String collection, Map mql, Map newData);
}
Class MongoProvider implements DataProvider { … }
Class RESTfulProvider implements DataProvider { … }

55. 55
The RESTful Provider
class RESTfulProvider implements DataProvider {
init() { // setup HTTP machine:port endpoint
fetch(String collection, Map mql) {
String jsonstr = JSONUtils.toJSON(mql);
String url = construct(collection, jsonstr);
// url is:
http://machine:port/collectionName?op=find&mql=‘{“produc
t”:”cleanser”,”expires”: {$gt: {$date: “20200101”}}}’
HTTPResponse res = call(url);
Map data = JSONUtils.fromJSON(res.getContent());
}
}