MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

2. Atlas Data Lake Technical Deep-Dive
Himanshumali, Solutions Architect-APAC, MongoDB

3. Himanshumali

4. State of Affairs
Why are we building this?
• Businesses have a humongous amount of data
o IDC predicts that by 2025 global data will reach 175 Zettabytes
o 49% of it will reside in the public cloud.
• Cloud storage is cost-effective
• Stored data is hard to operationalize

5. A New Service Offered by MongoDB Atlas
Atlas Data Lake allows you to...
• Access long-term data
• Query long-term data
• Analyze long-term data

6. Product Guidelines
Every product has requirements!
• Look and act like MongoDB
• Access customer’s data securely
• Handle queries over vast amounts of data
• Handle long-running queries
• Efficient use of resources

7. Emulating MongoDB

8. Behaviour
• Communicate with our existing drivers. Written in Go.
• Implemented a TCP server, using mongo-go-driver’s wire protocol package
• Implemented commands for a read-only server
• Used the server’s aggregation engine

9. Security
Must have the same security as MongoDB.
• Users configured in Atlas
• Implemented MongoDB’s security model
• Authentication
• Authorization
• Require the use of TLS + SNI
• SNI = Server Name Indicator

10. Control and Configuration

11. Controlled by Customer
Customers have complete control
• Provide us with an IAM Role
• Configure your buckets
• Configure your users in Atlas

12. Configuration
Customers control their data layout.
• Stores
• Databases, Collections
• DataSources

13. {
“stores” : [{
s3 : {
name: <string>,
bucket: <string>,
region: <string>,
prefix: <string>
}}
],
“databases” : {
“<database_name>” : {
“<collection_name>” : [
{
“store” : “<string>”
“definition” : “<string>”
}]
}}
}
Configuration: Store

14. Configuration (S3 Bucket): ent-archive
/archive/customers
- a-m.json
- n-z.json
/archive/invoices
- 2019
- 1.parquet
- 2.parquet
- 2018
- 1.parquet
- 2017.json.gz
- 2016.json.gz

15. s3 : {
name: "ent-archive",
bucket: "ent-archive",
region: "us-east-1",
prefix: "/archive/"
}
Configuration: Store

16. databases : {
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: “ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}
}
Configuration: Data

17. history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
cluster: "my-cluster",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)

18. Configuration: File Formats
Each file has a format
• BSON (gzipped)
• JSON (gzipped)
• Avro (gzipped)
• CSV/TSV (gzipped)
• Parquet

19. Processing
MQL : Distributed MQL
• Parse
• Distribute & Parallelize

20. Architecture

21. Architecture

22. { $match: { year: { $gt: 2000 } } }
{ $limit: 10 }
Map:
{ $match: { year: { $gt: 2000 } } }
{ $limit: 10 }
Reduce:
{ $limit: 10 }
Query Example: $limit

23. { $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "amount" },
totalAvg_count: { $sum: 1 }} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum",
"$totalAvg_count"] } } }
Query Example: $group

24. Use Cases
Scenario
• Customer storing devices data in Atlas for last 2 years
• Actively requires on last 2 years of data
• Might also need to query older data in case of customer request

25. Future

26. Future
More supported MongoDB operators.
• $out
• $merge
• $graphLookup
• Geo operators
• Full Text Search

27. Future
Optimizations!
• Indexes
• Statistics

28. Future
More File Formats!
• ORC
• Excel
• PDF

29. Future
Integrations!
• Microsoft Azure
• Google Cloud

30. Thank You!