MongoDB Atlas Data Lake is a new service offered by MongoDB Atlas. Many organizations store long term, archival data in cost-effective storage like S3, GCP, and Azure Blobs. However, many of them do not have robust systems or tools to effectively utilize large amounts of data to inform decision making. MongoDB Atlas Data Lake is a service allowing organizations to analyze their long-term data to discover a wealth of information about their business. This session will take a deep dive into the features that are currently available in MongoDB Atlas Data Lake and how they are implemented. In addition, we'll discuss future plans and opportunities and offer ample Q&A time with the engineers on the project.

1. #MDBlocal
Atlas Data Lake
Technical Deep-Dive
CHICAGO
Craig Wilson, Senior Staff Engineer, MongoDB

2. #MDBLocal
State of Affairs
Businesses have a humongous amount of data
• IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the
public cloud.
Cloud storage is cost-effective
Cloud storage is hard to operationalize

3. #MDBLocal
A New Service Offered by MongoDB Atlas
Access long-term data
Query long-term data
Analyze long-term data

4. #MDBLocal
Requirements
Look and act like MongoDB
Access customer’s data securely
Handle queries over vast amounts of data
Handle long-running queries
Efficient use of resources

5. Emulating MongoDB

6. #MDBLocal
Language
Must be able to communicate with our drivers
Written in Go
Implemented a TCP server
Used mongo-go-driver’s wireprotocol package
Used mongo-go-driver's bson package

7. #MDBLocal
Security
Must have the same security as MongoDB
Users configured in Atlas
Implemented MongoDB’s security model
Require the use of TLS + SNI(Server Name Indicator)

8. #MDBLocal
Behavior
Must behave like MongoDB
Implemented commands for a read-only server
Used the server’s aggregation engine

9. Customer’s Data

10. #MDBLocal
Security: Customers
Customers have complete control
Provide us with an IAM Role
Configure your buckets
Configure your users in Atlas

11. #MDBLocal
Security: Atlas
Atlas controls access to your data
Storage of IAM Role
Temporary Credentials

12. #MDBLocal
Configuration
Customers control their data layout
Stores
Databases, Collections
DataSources
CollectionCollection
Store Store
DataSource DataSource
DataSource

13. #MDBLocal
Configuration: File Formats
• BSON (gzipped)
• JSON (gzipped)
• Avro (gzipped)
• CSV/TSV (gzipped)
• Parquet
• XLSX

14. Configuration (S3 Bucket): ent-archive
/archive/customers
- a-m.json
- n-z.json
/archive/invoices
- 2019
- 1.parquet
- 2.parquet
- 2018
- 1.parquet
- 2017.json.gz
- 2016.json.gz

15. Configuration: Store
s3 : {
name: "ent-archive",
bucket: "ent-archive",
region: "us-east-1",
prefix: "/archive/"
}

16. Configuration: Data
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}

17. history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}.json.gz
}, {
store: "atlas",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)

18. Queries

19. #MDBLocal
Processing
MQL à Distributed MQL
Parse
Parallelize
Distribute

20. #MDBLocal
Architecture
Atlas
Control
Control
Plane
Compute
Plane
Data
Plane
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer
DataLake
Frontend
DataLake
Agent
Load Balancer
Load Balancer

21. #MDBLocal
Architecture
Atlas
Control
Control
Plane
Compute
Plane
Data
Plane
DataLake
Frontend
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent
DataLake
Agent

22. Query Example: $limit
Map:
{ $match: { year: { $gt: 2000 } } }
{ $limit: 10 }
Reduce:
{ $limit: 10 }
{ $match: { year: { $gt: 2000 } } }
{ $limit: 10 }

23. Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "$amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }

24. Future

25. #MDBLocal
Future
More supported MongoDB operators.
$out
$merge
Geo operators
Full Text Search

26. #MDBLocal
Future
Optimizations
Indexes
Statistics

27. #MDBLocal
Future
File Formats
• ORC
• PDF
Compression
• Bzip2
• Snappy
• LZMA
• LZO
• Zstd

28. #MDBLocal
Future
Integrations
Atlas
Microsoft Azure
Google Cloud

29. #MDBLocal
Hiring
Lots to do
mongodb.com/careers

30. Craig Wilson
Senior Staff Engineer, MongoDB

31. Our Developer focused talks
are back on the road!
Find one near you
At your MongoDB.local, you’ll learn technologies, tool, and best practices
That make it easy for you to build data-driven applications without distraction.

32. THANK YOU