The document discusses implementing cryptographically verifiable change history for documents stored in MongoDB. It proposes storing a full copy of documents upon any change, with each copy stored as a new document in a separate collection from the current documents. A cryptographic hash would be stored with each document copy to allow verification that the document has not been manipulated. This implementation aims to provide an auditable change history with detection of any data tampering or manipulation.

1. @
#MDBlocal
Salman Baset
Product Security
salman_baset
Using MongoDB Transactions to Implement
Cryptographically Verifiable Change History

2. #MDBLocal
Safe Harbor

3. #MDBLocal
Drug Development
Long and complex process
End goal: drug released is “safe” for usage.
Preclinical* Clinical (trials)*
Stage 1 Stage 2 Stage 2 Stage N Phase I Phase II Phase III Phase N
** https://www.cleardata.com/wp-content/uploads/2018/01/GxP_Brochure_FINAL.pdf
Icons made by Freepik from www.flaticon.com
* Illustration of
preclinical or clinical
timelines.

4. #MDBLocal
Drug Development
Long and complex process
End goal: drug released is “safe” for usage.
Preclinical* Clinical (trials)*
Stage 1 Stage 2 Stage 2 Stage N Phase I Phase II Phase III Phase N
** https://www.cleardata.com/wp-content/uploads/2018/01/GxP_Brochure_FINAL.pdf
Icons made by Freepik from www.flaticon.com
Good Practices (GxP) regulations for drug development (US, FDA), and include**
● Traceability: reconstruct development history of drug
● Data integrity: avoid data manipulation and breaches
* Illustration of
preclinical or clinical
timelines.

5. #MDBLocal
Drug Development
Long and complex process
End goal: drug released is “safe” for usage.
Preclinical* Clinical (trials)*
Stage 1 Stage 2 Stage 2 Stage N Phase I Phase II Phase III Phase N
** https://www.cleardata.com/wp-content/uploads/2018/01/GxP_Brochure_FINAL.pdf
Icons made by Freepik from www.flaticon.com
Good Practices (GxP) regulations for drug development (US, FDA), and include**
● Traceability: reconstruct development history of drug
● Data integrity: avoid data manipulation and breaches
* Illustration of
preclinical or clinical
timelines.
Could you as a developer record drug development history in an easy way for reconstruction?
Could you as a developer implement measures (e.g., cryptographic verification) for detecting
data manipulation?

6. #MDBLocal
Banking Transaction
1. Visit ATM.
2. Check balance.
3. Withdraw $20.
4. Get a paper receipt of
remaining $80 balance.
$100
$80
$80
Icons made by ultimatearm and Vectors Market from www.flaticon.com
ATM

7. #MDBLocal
Banking Transaction
1. Visit ATM.
2. Check balance.
3. Withdraw $20.
4. Get a paper receipt of
remaining $80 balance.
$100
$80
$80
1. Visit bank website.
2. Check balance.
3. Transfer $20.
4. Get an email of transaction
and screenshot of remaining
balance.
$100
$80
Icons made by ultimatearm and Vectors Market from www.flaticon.com
ATM Online

8. #MDBLocal
Banking Transaction
1. Visit ATM.
2. Check balance.
3. Withdraw $20.
4. Get a paper receipt of
remaining $80 balance.
$100
$80
$80
1. Visit bank website.
2. Check balance.
3. Transfer $20.
4. Get an email of transaction
and screenshot of remaining
balance.
$100
$80
Icons made by ultimatearm and Vectors Market from www.flaticon.com
ATM Online
Are two transactions equivalent?
Are two “confirmations” equivalent?

9. #MDBLocal
Banking Transaction
1. Visit ATM.
2. Check balance.
3. Withdraw $20.
4. Get a paper receipt of
remaining $80 balance.
$100
$80
$80
1. Visit bank website.
2. Check balance.
3. Transfer $20.
4. Get an email of transaction
and screenshot of remaining
balance.
$100
$80
Icons made by ultimatearm and Vectors Market from www.flaticon.com
ATM Online
Are two transactions equivalent?
Are two “confirmations” equivalent?
Could there be a digital receipt (with cryptographic verification) of your transaction and
balance when you logout of the website?

10. #MDBLocal
Outline - Using MongoDB Transactions to Implement
Cryptographically Verifiable Change History
• Goals
• What is change history?
• Demo of a proof-of-concept
• What is cryptographically verifiable change history?
• Discussion and feedback

11. #MDBLocal
Two Goals
(1) Describe
• Using MongoDB Transactions to Implement Cryptographically Verifiable
Change History
• while preserving MongoDB MQL and API interface

12. #MDBLocal
Two Goals
(1) Describe
• Using MongoDB Transactions to Implement Cryptographically Verifiable
Change History
• while preserving MongoDB MQL and API interface
(2) Feedback and discussion
• Would a similar functionality in a new product address and simplify your data
integrity needs?
• Would you rather see enhancements in existing auditing?
• Help us build the roadmap!
• Send an email to customer-security-working-group@mongodb.com with the subject “JOIN” to
to get involved, or fill out the survey: https://www.surveymonkey.com/r/3RNLJ2L

13. #MDBLocal
What is Change History?
Informally
- History of change for an entity, going back to some “reference origin”

14. #MDBLocal
Why is Change History Needed?
• To meet application data integrity requirements (e.g., drug
development, crypto exchanges, HR)
• To meet information security and compliance requirements (e.g., did
someone tamper with the stored data)
These requirements are not mutually exclusive.

15. #MDBLocal
Why is Change History Needed? (focus of this talk)
• To meet application data integrity requirements (e.g., drug
development, crypto exchanges, HR)
• To meet information security and compliance requirements

16. #MDBLocal
What is Change History in a Database Context?
• Change = CUD (create, update, delete)

17. #MDBLocal
What is Change History in a Database Context?
• Change = CUD (create, update, delete)
• What about read?

18. #MDBLocal
Application Change History != Database Auditing
• Audit logs are stored in files
• Not easy to get document change history from audit files
• e.g., getDocHistory(_id)

19. #MDBLocal
Example Scenario for Change History
Icon credit flaticon.com
● A (financial) website is required to show user all changes they have made to their
account (e.g., address or name updates). The most recent user information should be
visible within each site web page.
● The financial website stores all historical information in MongoDB
1. Update my info
4. Updated
2. Update in DB
3. Updated
Financial
website

20. #MDBLocal
Example Scenario for Change History
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York",
"State": "NY"
}
1. User passes their information to a website,
which stores it as MongoDB document
2. User updates their address
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL"
}
3. User wants to see previous updates to its
address (e.g., its a credit application)
How to implement it?

21. #MDBLocal
Options for Implementing Change History
Relevant links
http://www.askasya.com/post/trackversions/
http://www.askasya.com/post/revisitversions/
• Many, e.g.,
• Store a full copy of the document upon change
• Within the same document
• As a new document in the same collection
• As a new document in a separate collection
• Store deltas
} may need atomic updates (e.g.,
MongoDB Transaction)

22. #MDBLocal
Options for Implementing Change History
• Many, e.g.,
• Store a full copy of the document upon change
• Within the same document
• Document size limitation*, query complexity (for historic data)
• As a new document in the same collection
• As a new document in a separate collection
• Store deltas
* https://docs.mongodb.com/manual/reference/limits/

23. #MDBLocal
Options for Implementing Change History
• Many, e.g.,
• Store a full copy of the document upon change
• Within the same document
• As a new document in the same collection
• query complexity and speed, e.g., for current version. how to link current version
with previous version? multiple documents to update?
• As a new document in a separate collection
• Store deltas

24. #MDBLocal
Options for Implementing Change History
• Many, e.g.,
• Store a full copy of the document upon change
• Within the same document
• As a new document in the same collection
• As a new document in a separate collection (RECOMMENDED)
• how to link current version with previous version? multiple documents to update?
• Store deltas

25. #MDBLocal
Demo POC
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
"_cryptoevidence": ***,
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York,
"State": "NY",
"_cryptoevidence": ***,
}
Update
document
Insert
db.customers.insert_one_ledger({“_id”: doc_id, “First name”: “Jane”, “Last name”: “Hill”, “State”:
“IL”, ...})
doc_id = db.customers.insert_one_ledger({“First name”: “Jane”, “Last name”: “Hill”, “State”, “NY”, ...})
Source code of demo available @ https://github.com/salmanbaset/mdb_ledger
Verify
document
db.customers.verify_one_ledger({“_id”: doc_id})
Delete
document
db.customers.delete_one_ledger({“_id”: doc_id})
customers customers_history

26. #MDBLocal
DIY: Use separate collections for current and historical
data
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York",
"State": "NY",
"Timestamp": TS
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": $$now,
}
db.col.find_one_and_update()
db.col_history.find()
Explicit separate
collection maintenance
in application code.
Update db.col_history.insert_one()
db.col.find()
● programmer_code.update({"First name": "Jane", "Last name": "Hill"}, { $set: {"Address": "1
Michigan Avenue", "City": "Chicago", "State": "IL"}})
○ Transaction
■ olddoc = db.col.findOneAndUpdate()
■ db.col_history.insert_one(olddoc)
● programmer_code.find({"First name": "Jane", "Last name": "Hill"})
○ db.col.find()
● programmer_code.find_history({"First name": "Jane", "Last name": "Hill"})
○ db.col_history.find()
Find
history
Current
doc
New
doc

27. #MDBLocal
Want something more familiar? Create a collection as a
‘ledger’, and use existing driver functions as is (1/3)*
● db.create_collection({flags: ledger})
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York",
"State": "NY",
"Timestamp": TS
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
}
Current
doc
New
doc
* This is an idea. Discuss towards the end.

28. #MDBLocal
Want something more familiar? Create a collection as
‘ledger’, and use existing driver functions as is (2/3)*
● db.create_collection({flags: ledger})
● db.col.update_one({"First name": "Jane", "Last name": "Hill"}, { $set: {"Address": "1
● Michigan Avenue", "City": "Chicago", "State": "IL"}}) - If updating an existing document, it inserts
the existing document in a “history” collection and updates the existing document.
● db.col.find_one({"First name": "Jane", "Last name": "Hill"})
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York",
"State": "NY",
"Timestamp": TS
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
}
Current
doc
New
doc
* This is an idea. Discuss towards the end.

29. #MDBLocal
Want something more familiar? Create a collection as
‘ledger’, and use existing driver functions as is (3/3)*
● db.create_collection({flags: ledger})
● db.col.update_one({"First name": "Jane", "Last name": "Hill"}, { $set: {"Address": "1
● Michigan Avenue", "City": "Chicago", "State": "IL"}}) - If updating an existing document, it inserts the
existing document in a “history” collection and updates the existing document.
● db.col.find_one({"First name": "Jane", "Last name": "Hill"})
● db.col.find_one_history({"First name": "Jane", "Last name": "Hill"}) <= NEW FUNCTION
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York",
"State": "NY",
"Timestamp": TS
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
}
Current
doc
New
doc
* This is an idea. Discuss towards the end.

30. #MDBLocal
Are we done?
• If all you wanted was to store and maintain change history for a
document in MongoDB, the above operations might all be you need.
• Use MongoDB Transactions to Implement Change History
• Talk title: Use MongoDB Transactions to Implement Cryptographically
Verifiable Change History

31. #MDBLocal
What is Cryptographic Verification and Why is it Needed?
Informally
• If a single document has been inserted, updated, or deleted, a “user”
wants to verify that the document (or its history) has not been
manipulated (e.g., upon retrieval).
Benefits
• Enhanced data integrity for applications.

32. #MDBLocal
Example Scenario for Cryptographic Verification
Icon made by Dave Gandy from flaticon.com
● Steps 1-5 (above picture): A (financial) user adds or updates their information in the
website. The website returns crypto-evidence of their information (e.g., in the form of QR
code) to the end user, which the user stores in their mobile wallet.
● On subsequent logins, the user can use the QR code to validate that the information it
provided previously matches the one returned by the website.
1. Update my info
4. Updated and
returns QR code
of crypto evidence
as a digital receipt
2. Update in DB
3. Updated. crypto
evidence of
inserted information
5. stored
in digital
wallet

33. #MDBLocal
Who is the user, what is the manipulation?
Who is the “user”?
• Developer: uses MongoDB for storing and retrieving documents
• Application user: uses a website to store and retrieve information.
The website uses MongoDB.
Who can manipulate the change history?
• DBA
• Developer
• Application user
• Malicious actor
• ...

34. #MDBLocal
Manipulation Example
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "ID",
"Timestamp": now,
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New Jersey,
"State": "NY",
}
Manipulated HistoryManipulated Current

35. #MDBLocal
What is Cryptographic Verification? - Example using
Hashing
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
"_cryptoevidence": ***,
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York,
"State": "NY",
"_cryptoevidence": ***,
}
HistoryCurrent
"_cryptoevidence" = SHA256(JSON.stringify({
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York,
"State": "NY",
}))
"_cryptoevidence" = SHA256(JSON.stringify({
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
}) + )
Implementation detail: Keys in JSON document should be sorted before crypto evidence computation (e.g., hashing)

36. #MDBLocal
When can Cryptographic Verification be performed?
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
"_cryptoevidence": ***,
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York,
"State": "NY",
"_cryptoevidence": ***,
}
Upon read
Upon insert
find_history_ledger({“First name”: “Jane”, “Last name”: “Hill”})
verify_history({“First name”: “Jane”, “Last name”: “Hill”})
insert_ledger({“First name”: “Jane”, “Last name”: “Hill”, ...}) - fail inserts if tampering detected (e.g., in
previous document version or all previous document versions)

37. #MDBLocal
Is Cryptographic Verification over a Single Document
History?
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Michigan Avenue",
"City": "Chicago",
"State": "IL",
"Timestamp": now,
"_cryptoevidence": ***,
}
{
"First name": "Jane",
"Last name": "Hill",
"Address": "1 Broadway",
"City": "New York,
"State": "NY",
"_cryptoevidence": ***,
}
Upon read
Upon insert
find_history_ledger({“First name”: “Jane”, “Last name”: “Hill”})
verify_history({“First name”: “Jane”, “Last name”: “Hill”})
insert_ledger({“First name”: “Jane”, “Last name”: “Hill”})
Not necessarily. Examples of possible options
Verify only the last version
Verify entire document history
Verify all collection documents (excluding history)
Verify all collection documents and their history

38. #MDBLocal
Conclusion
Using MongoDB Transactions to Implement Cryptographically Verifiable
Change History
• while preserving MongoDB interface that developers love
Feedback and discussion
• Would a similar functionality in a new product address and simplify your data
integrity needs?
• Would you rather see enhancements in existing auditing?
• Help us build the roadmap!
• Send an email to customer-security-working-group@mongodb.com with the subject “JOIN”
to be added to our security focused customer working group.

39. #MDBlocal
Using Transactions to
Implement Cryptographically
Verifiable Change History in
MongoDB
https://www.surveymonkey.com/r/3RNLJ2L
Every session you rate enters you into a drawing for a $250
Visa gift card, sponsored by