Users demand applications which integrate intelligently with data from mobile, social media and cloud sources. A system of engagement meets these expectations by applying data and analytics drawn from an array of master systems. The enormous scale and performance required overwhelm relational approaches, but we can use MongoDB to meet the challenge. We'll learn to capture and transmit data changes among disparate systems, expose batch data as interactive operational queries and build systems with strong division of concerns, agility and flexibility

1. Building a System of Engagement
with MongoDB
{ name: ‘Bryan Reinero’,
twitter: ‘@blimpyacht’,
email: ‘bryan@mongdb.com’
web: ‘mongodb.com/bryan’ }

2. 2
Systems of Engagement
• Real-Time
• Context-Aware
• Encourage
Interaction
• Embedded in
Business Systems

3. 3
Systems of Engagement
• Context rich and User
Relevant Interactions
• Integrates data from many
systems
• Integrates Analytics

4. 4
The Importance of Engagement
• 74% Consumer would respond positively to companies that
understand them
• 57% Would recommend the company
• 29% Would make additional purchases
2013 Experian study

5. 5
• 84% Would walk away from a non-responsive company
• 45% from a company that contacted them when they had asked
not to
• 52% Would leave a company that tries to sell them something
they already said they weren't interested in
The Importance of Engagement
2013 Experian study

6. 6
How is the
application of value
to the user?

7. 7

8. 8
Department of Veterans Affairs
20+ million Veterans in the US today
Doctors need a single view of a patient’s health record.
What happens when a patient has to change their address?

9. Requirements
• High performance requirements
• Increasingly large datasets
• High Availability

10. 10
Flexible Schema
{ patient: 8675309,
format: “NCQA”,
date: ISODate("2015-09-29T15:48:44.901Z”),
procedure: {
name: “umbilicoplasty”
},…
},
{ patient: 8675309,
format: “LTCv1”,
date: ISODate("2015-09-29T15:48:44.901Z"),
medications: [
"Zolpidem",
"Chlorpheniramine”,
"Chlorhexidine"
],…
}

11. 11
Flexible Schema
{ patient: 8675309,
format: “NCQA”,
date: ISODate("2015-09-29T15:48:44.901Z”),
procedure: {
name: “umbilicoplasty”
},…
},
{ patient: 8675309,
format: “LTCv1”,
date: ISODate("2015-09-29T15:48:44.901Z"),
medications: [
"Zolpidem",
"Chlorpheniramine”,
"Chlorhexidine"
],…
}
Common primary key
Flexible Schema

12. 12
Flexible Schema
Common primary key
Discriminator
{ patient: 8675309,
format: “NCQA”,
date: ISODate("2015-09-29T15:48:44.901Z”),
procedure: {
name: “umbilicoplasty”
},…
},
{ patient: 8675309,
format: “LTCv1”,
date: ISODate("2015-09-29T15:48:44.901Z"),
medications: [
"Zolpidem",
"Chlorpheniramine”,
"Chlorhexidine"
],…
}

13. 13
Flexible Schema
{ patient: 8675309,
format: “NCQA”,
date: ISODate("2015-09-29T15:48:44.901Z”),
procedure: {
name: “umbilicoplasty”
},…
},
{ patient: 8675309,
format: “LTCv1”,
date: ISODate("2015-09-29T15:48:44.901Z"),
medications: [
"Zolpidem",
"Chlorpheniramine”,
"Chlorhexidine"
],…
}
Common primary key
Discriminator
Polymorphic entities

14. 14
• Wearable Devices
• Embedded
Systems
• Internet of Things

15. 15
Customer Single View
• Understand customer
relationships
• Improves customer
experience
• Develops effective
customer marketing
• Improves product

16. 16
Architecture
Systems of Engagement
DataServices
Systems of Record
Master Data
Raw Data
Integrated Data
…
ETL
record
record
record

17. 17
Aggregating a Single View
Single customer
VIEW

18. 18
Aggregating a Single View
Common Data
Source Metadata
Source Data A
Source Data B

19. 19
Aggregating a Single View
Common Data
Source Metadata
Source Data A
Source Data B
{
_id: <hash>,
address: {
num: 860,
street: “Grove”,
city: “San Francisco”,
state: “CA”,
zip:
}
}

20. 20
Aggregating a Single View
Common Data
Source Metadata
Source Data A
Source Data B
{
sources: [
{
source: “URI”,
updated: ISODate(),
},
…
]
}

21. 21
Aggregating a Single View
Common Data
Source Metadata
Source Data A
Source Data B
Shopping Cart,
Purchase history,
Prescriptions,
Medical History,

22. The Point of Engagement

23. 23
The Scavenger Hunt App
Users create
scavenger hunts by
“pinning” waypoints

24. 24
The Checkpoint Document
{
_id: ObjectId(),
user: UUID,
huntId: UUID,
timestamp: ISODate(),
geometry: {
type: "Point",
coordinates: [125.6, 10.1]
}
}

25. 25
The Checkpoint Document
{
_id: ObjectId(),
user: UUID,
huntId: UUID,
timestamp: ISODate(),
geometry: {
type: "Point",
coordinates: [125.6, 10.1]
}
}
db.checkpoints.ensureIndex(
timestamp: -1,
geometry: “2dsphere”
);

26. 26
The Checkpoint Document
db.checkpoints.find(
{
date: {$gt: <10mins ago> }
geometry: {
$near:{
{ type : "Point" ,
coordinates :
[ -173, 40.7 ]
},
$maxDistance : 100 }
}});
{
_id: ObjectId(),
user: UUID,
huntId: UUID,
timestamp: ISODate(),
geometry: {
type: "Point",
coordinates: [125.6, 10.1]
}
}

27. 27
The Scavenger Hunt App
Business Requirements
• Location Based
Targeting
• Enable Social
Interactions
• Recommendations

28. 28
Waypoint
{ _id: ObjectId(),
name: "Doug’s Coffee",
desc: "The best brew",
offers: [
”Morning rush hour discount",
”Order ahead with our app",
”Start your digital frequent flyer
card"
],
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1] }
};

29. 29
Waypoint
{ _id: ObjectId(),
name: "Doug’s Coffee",
desc: "The best brew",
offers: [
”Morning rush hour discount",
”Order ahead with our app",
”Start your digital frequent flyer
card"
],
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1] }
};

30. 30
Waypoint
{ _id: ObjectId(),
name: "Doug’s Coffee",
desc: "The best brew",
offers: [
”Morning rush hour discount",
”Order ahead with our app",
”Start your digital frequent flyer
card"
],
"geometry": {
"type": "Point",
"coordinates": [125.6, 10.1] }
};
Geospacial Index:
ensureIndex(
{ geometry: “2dsphere” }
)

31. 31
Geo Targeting

32. 32
Geo Targeting
{
"type": "Polygon",
"coordinates" : [
[
[ -73.969581, 40.760331 ],
[ -73.974487, 40.762245 ],
[ -73.977692, 40.763598],
[ -73.979508, 40.761269 ],
[ -73.982364, 40.762358 ],
[ -73.983692, 40.760497 ],
[ -73.972821, 40.755861 ],
[ -73.969581, 40.760331 ]
]
]
} Defines a business service area

33. 33
$geoIntersects
{
$geoIntersects: {
$geometry: {
"type": "Point",
"coordinates": [
-73.975010,
40.760071
]
}
}
}

34. 34
Defining Service Areas
Data Driven Decisions
• Find users within a given area
• Intersections with multiple service areas
• Fraud detection
• Where are my competitors located

35. Social Interactions

36. 36
Social Interactions
Requirements
• Allow users to follow one
another
• Allow users to exchange
messages
• Send users notifications

37. 37
Social Interactions
Eratosthenes
Democritus
Hypatia
Shemp
Euripides
Graph models
Personal Relationships
Recommendation Engines

38. 38
Social Interactions
Followers Collection
{ follower: ‘Shemp’, followed: ‘Euripides’},
{ follower:‘Shemp’, followed: ’Eratosthenes”},
{ follower: “Eratosthenes’, followed: ‘Shemp’ },
…
Eratosthenes
Democritus
Hypatia
Shemp
Euripides

39. 39
Social Interactions
Eratosthenes
Democritus
Hypatia
Shemp
Euripides
Followers Collection
{ follower: ‘Shemp’, followed: ‘Euripides’},
{ follower:‘Shemp’, followed: ’Eratosthenes”},
{ follower: “Eratosthenes’, followed: ‘Shemp’ },
…
! (Euripides -> Shemp )

40. 40
Social Interactions
db.followers.find( { follower:‘Shemp’ } );
Followers Collection
{ follower: ‘Shemp’, followed: ‘Euripides’},
{ follower:‘Shemp’, followed: ’Eratosthenes”},
{ follower: “Eratosthenes’, followed: ‘Shemp’ },
…

41. Notifications

42. 42
Personal Timeline / Hotlist
{
_id: UUID,
user: ”Democritus",
hotList" : [
{
message: "New scavenger hunt tomorrow!",
url: "http://bit.ly/1hKn9ff",
date" : ISODate()
},
{
message: "Get 50% off at Toga City",
url: "http://bit.ly/1KnlFHQ",
date: ISODate()
}
],
atime: ISODate("20150313T04:38:43.606Z")
}

43. 43
Personal Timeline / Hotlist
Notifications of highest
user relevance
{
_id: UUID,
user: ”Democritus",
hotList" : [
{
message: "New scavenger hunt tomorrow!",
url: "http://bit.ly/1hKn9ff",
date" : ISODate()
},
{
message: "Get 50% off at Toga City",
url: "http://bit.ly/1KnlFHQ",
date: ISODate()
}
],
atime: ISODate("20150313T04:38:43.606Z")
}

44. 44
Write to Bucket
Parameters
• User
db.user.update(
{"user" : "Democritus"},
{$push: {
hotList: {
$each:
[
{ o: 10, <MESSAGE> },
{ o: 7, <MESSAGE> },
…
],
$sort: { o: -1 },
$slice: 50 }
}}
);

45. 45
Write to Bucket
Parameters
• user
• $push append to end of hotList array
db.user.update(
{"user" : "Democritus"},
{$push: {
hotList: {
$each:
[
{ o: 10, <MESSAGE> },
{ o: 7, <MESSAGE> },
…
],
$sort: { o: -1 },
$slice: 50 }
}}
);

46. 46
Write to Bucket
Parameters
• user
• $push append to end of hotList array
• $each list of message elements
db.user.update(
{"user" : "Democritus"},
{$push: {
hotList: {
$each:
[
{ o: 10, <MESSAGE> },
{ o: 7, <MESSAGE> },
…
],
$sort: { o: -1 },
$slice: 50 }
}}
);

47. 47
Write to Bucket
Parameters
• user
• $push append to end of hotList array
• $each message in array
• $sort the resulting array in descending order
db.user.update(
{"user" : "Democritus"},
{$push: {
hotList: {
$each:
[
{ o: 10, <MESSAGE> },
{ o: 7, <MESSAGE> },
…
],
$sort: { o: -1 },
$slice: 50 }
}}
);

48. 48
Write to Bucket
Parameters
• user
• $push append to end of hotList array
• $each message in array
• $sort the resulting array in descending order
• $slice include only the first 50 elements
db.user.update(
{"user" : "Democritus"},
{$push: {
hotList: {
$each:
[
{ o: 10, <MESSAGE> },
{ o: 7, <MESSAGE> },
…
],
$sort: { o: -1 },
$slice: 50 }
}}
);

49. Analytics

50. 50
Data Management
Offline Processing
Analytics
Data Warehousing
OLTP
Application
Fine grained Operations

51. 51
https://github.com/mongodb/mongo-hadoop

52. 52
Data Services
ETLDatabase
HDFS

53. 53
ETL
HDFS
Hadoop / Spark
Connector
Data Services

54. 54
Analytics
Systems of Engagement
DataServices
Data Processing
Integration, Analytics, etc.
Systems of Record
Master Data
Raw Data
Integrated Data
…
ETL
record
record
record

55. 55
MongoDB as an Operational Store
Application Server
Other DBMS
& Legacy systems

56. Data Capture

57. 57
Capture Data Changes
Systems of Engagement
DataServices
Data Processing
Integration, Analytics, etc.
Systems of Record
Master Data
Raw Data
Integrated Data
…
ETL
Bus
Apache Kafka
record
record
record

58. 58
Many Complexities to Tackle
• Data modeling
• Data Extraction (ETL)
• Change Data Capture (CDC)
• Data Governance
• Data Lineage
• Security

59. 59
How is the application of
value to the user?

60. Thanks!
{ name: ‘Bryan Reinero’,
twitter: ‘@blimpyacht’,
email: ‘bryan@mongodb.com’
web: ‘mongodb.com/bryan’ }