Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Nic Cottrell, MongoDB France
MongoDB cluster design:
from Redundancy to GDPR
@niccottrell
nic-cottrell
niccottrell
Who am I?
§ I am currently a Technical Services Engineer.
§ I was recently a Consul;ng Engineer.
§ Before that I was a So>...
Why are you here?
You know about databases and growth but need to be sure
MongoDB can scale while maintaining data localit...
What will you learn?
By the end of this talk, you will know about:
• Data at scale in multiple physical locations.
• Best ...
Core Principle There should be no single point
of failure in the system.According to me
European
Odyssey
1
2
3
4
5
✈
✈
Replication and
Sharding
There are two important ways to
scale a MongoDB database:
1. Replicate the same data.
2. Shard (o...
Replication
London
Paris Frankfurt
PRIMARY
SECONDARY
SECONDARY
PRIMARY
DOWN
Each node stores:
• Data, Indexes
(Encrypted o...
Sharding
PRI
SEC SEC
PR I
SEC SEC
PRI
SEC SEC
Shard 1 Shard 2 Shard 3
Application
server
mongos
Config
servers
France Spai...
Datacenters
and
Geographic
Distribu3on
Let’s talk more about why you
need multiple datacenters
Replication and Sharding
London
Paris Frankfurt
PRIMARY
SECONDARY
SECONDARY
Virginia
Ohio California
PRIMARY
SECONDARY
SEC...
Features
Defend
Some key features:
• Access Control
• Firewalls, bindIp
• Passwords / x509
• Pseudonymization
• Encryption...
Regulation (EU)
2016/679
The three Ps:
• permission
• privacy
• protec3on
aka GDPR
The two Ts:
• transfer
• transit
SEEK L...
Use cases !
Don’t do any of these!
“Mobile data”
the wrong way
!
“Mobile” Sharding
PRIMARY
SECONDARY
SECONDARY
PRIMARY
SECONDARY
SEC O N D A RY
PRIMARY
SECONDARY
SECONDARY
North Atlantic ...
How’d it do?
• Fast recovery !
• Data has redundant copies
"
• Robust, performant #
• GDPR $
PRIMARY
SECONDARY
SEC O N D A...
What did we learn in the Netherlands?
In order for data to be available and secure:
• Data needs to be in data centers (pr...
High Availability
the wrong way
!
The setup
• Each node has its own auto-
scaling group, each with a fixed IP
• A Lambda function checks health
and trigger ...
The setup
How’d it do?
!
• Easy to perform post-
mortem
• Fast recovery !
• Data has redundant copies
"
• Robust, performant "
• GDP...
What did we learn in Germany?
• MongoDB is already fault-tolerant.
• With standard topology we can always take down a node...
Multiple NICs
More network cards = Less
points of failure ?
No!
!
What did we learn in Italy?
MongoDB is designed for commodity hardware
• Designed to add redundancy and automatic failure ...
Best cases !
Please remember this bit!
Minimum (viable) topology
eu-west-2a
PR IM A RY
SECONDARY SECONDARY
London
!
eu-west-2b eu-west-2c
2-datacenter topology (on-prem)
Data center 1
PRIMARY
SECONDARY ARBITER
Data center 2 Cloud
Write concern:
Majority
SECOND...
Geo-
Sharding
!
Let’s put it all together:
• Scalability
• Redundancy
• High availability
• Data sovereignty
Region-level redundancy
London
Paris Frankfurt
PR IM A RY
SECONDARY
SECONDARY
Virginia
Ohio California
PRIMARY
SECONDARY
S...
Example topology
Config servers contain metadata
and shard key values
Each shard contains data for
countries in that regio...
Atlas
! • AWS, Azure, GCP
• Secures your data with individual
VPCs
• Data encrypted with your key
• Best practices for hig...
Atlas Global Clusters
• Atlas can automate
regional clusters for
you.
• Focus on
performance (low
latency) not really
comp...
What did we learn?
By now I hope you all agree:
§ For high-availability, we need data in at least three copies of the data...
What’s next?
Here’s some other talks that might be interesting:
Tutorial: Hands-on with Realm Mobile Database
§ Alexander ...
Questions?
Reach me at nic.c@mongodb.com
or @niccottrell
GDPR Resources
GDPR: Frequently Asked Questions
Website
§ How does MongoDB help my organizaAon comply with the GDPR?
§ How does MongoDB A...
GDPR: Impact to Your Data Management Landscape
Whitepaper
How MongoDB Can Help Meet GDPR Requirements
§ Discover
§ Defend
...
GDPR and « The right to be forgotten ».
Checklist
Some key issues to keep in mind:
§ MongoDB Atlas backups can be deleted ...
Pseudonymization with MongoDB Views
Blog post
About using Views for access control and auditing
§ https://www.mongodb.com/...
Extras
Audit log examples
Failed authentication
{"atype":"authenticate","ts":{"$date":"2017-02-
14T14:11:29.975+0100"},"local":{"...
Redacted log examples
Original log
2017-06-09T13:35:23.446-0400 I COMMAND [conn1] command internal.clients
command: insert...
Upcoming SlideShare
Loading in …5
×

of

MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 1 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 2 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 3 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 4 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 5 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 6 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 7 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 8 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 9 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 10 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 11 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 12 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 13 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 14 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 15 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 16 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 17 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 18 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 19 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 20 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 21 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 22 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 23 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 24 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 25 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 26 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 27 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 28 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 29 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 30 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 31 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 32 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 33 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 34 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 35 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 36 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 37 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 38 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 39 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 40 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 41 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 42 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 43 MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR Slide 44
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

1 Like

Share

Download to read offline

MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR

Download to read offline

No matter if you're thinking of migrating to MongoDB, or need to meet legal requirements for an existing on-prem cluster, this talk has you covered. We start with the basics of replication and sharding and quickly scale up, covering everything you need to know to control your data, and keep it safe from unexpected data loss or downtime - a well-designed MongoDB cluster should have no single point of failure. Learn how others are “stretching” what’s possible but why you shouldn't! I'll present real-world examples from my life in the field in Europe and beyond.

Related Books

Free with a 30 day trial from Scribd

See all

MongoDB World 2019: MongoDB Cluster Design: From Redundancy to GDPR

  1. 1. Nic Cottrell, MongoDB France MongoDB cluster design: from Redundancy to GDPR @niccottrell nic-cottrell niccottrell
  2. 2. Who am I? § I am currently a Technical Services Engineer. § I was recently a Consul;ng Engineer. § Before that I was a So>ware Engineer. § However, I also completed an MBA. § It’s complicated. ! " # $
  3. 3. Why are you here? You know about databases and growth but need to be sure MongoDB can scale while maintaining data locality and secure flows. Your organization has specific requirements that data be on-prem, encrypted, controlled and backed up with zero-tolerance for downtime.
  4. 4. What will you learn? By the end of this talk, you will know about: • Data at scale in multiple physical locations. • Best (and worst) practices for topology design. • How to take control of your data geographically. • Controlling sensitive data in distributed databases. • Avoiding any single point of failure. • How to compare on-prem with MongoDB Atlas.
  5. 5. Core Principle There should be no single point of failure in the system.According to me
  6. 6. European Odyssey 1 2 3 4 5 ✈ ✈
  7. 7. Replication and Sharding There are two important ways to scale a MongoDB database: 1. Replicate the same data. 2. Shard (or “partition”) the data into subsets. Not the same thing!
  8. 8. Replication London Paris Frankfurt PRIMARY SECONDARY SECONDARY PRIMARY DOWN Each node stores: • Data, Indexes (Encrypted on Enterprise) • MongoDB logs New connecJons AuthenJcaJon aKempts • Audit logs (Enterprise only) • FTDC metrics • System logs
  9. 9. Sharding PRI SEC SEC PR I SEC SEC PRI SEC SEC Shard 1 Shard 2 Shard 3 Application server mongos Config servers France Spain Italy
  10. 10. Datacenters and Geographic Distribu3on Let’s talk more about why you need multiple datacenters
  11. 11. Replication and Sharding London Paris Frankfurt PRIMARY SECONDARY SECONDARY Virginia Ohio California PRIMARY SECONDARY SECONDARY Sydney Hong Kong Mumbai PR IM A RY SECONDARY SECONDARY European Union Data North America Data Asia-Pacific Data mongos mongos mongos
  12. 12. Features Defend Some key features: • Access Control • Firewalls, bindIp • Passwords / x509 • Pseudonymization • Encryption • Connections with TLS • At rest with rotated keys Detect • Monitoring and Reporting redactClientLogData • Auditing See more on the blog series: GDPR: Impact to Your Data Management Landscape New encryption capabilities in MongoDB 4.2: A deep dive into protecting sensitive workloads Kenn White, MongoDB - Tuesday, 3pm Discover • Compass to explore data • Automatic data retention with TTL indexes
  13. 13. Regulation (EU) 2016/679 The three Ps: • permission • privacy • protec3on aka GDPR The two Ts: • transfer • transit SEEK LEGAL ADVICE Also HIPAA, PCI-DSS, CCPA (California), PIPEDA (Canada) etc.
  14. 14. Use cases ! Don’t do any of these!
  15. 15. “Mobile data” the wrong way !
  16. 16. “Mobile” Sharding PRIMARY SECONDARY SECONDARY PRIMARY SECONDARY SEC O N D A RY PRIMARY SECONDARY SECONDARY North Atlantic Data Mediterranean Data Central Atlantic Data mongos ❌ Config
  17. 17. How’d it do? • Fast recovery ! • Data has redundant copies " • Robust, performant # • GDPR $ PRIMARY SECONDARY SEC O N D A RY North Atlantic Data
  18. 18. What did we learn in the Netherlands? In order for data to be available and secure: • Data needs to be in data centers (preferably 3+) • Large oplogs can be used to let nodes that _do_ go offline to catch up without a initial sync. For more details about Mobile databases, check out § Hands-on with Realm Mobile Database Today 1:00pm - 2:45pm !
  19. 19. High Availability the wrong way !
  20. 20. The setup • Each node has its own auto- scaling group, each with a fixed IP • A Lambda function checks health and trigger failover • When a host was considered failed, it was rebuilt from scratch • All packages and config were rebuilt with CloudFormation • Requires a initial sync each time
  21. 21. The setup
  22. 22. How’d it do? ! • Easy to perform post- mortem • Fast recovery ! • Data has redundant copies " • Robust, performant " • GDPR #
  23. 23. What did we learn in Germany? • MongoDB is already fault-tolerant. • With standard topology we can always take down a node for maintenance, upgrades or debugging. • The mongodb.log and FTDC data are invaluable to diagnose crashes or slowdowns. • We want to avoid initial syncs where ever possible by leaving the dbPath intact. • Monitoring and alerting tools are still recommended. !
  24. 24. Multiple NICs More network cards = Less points of failure ? No! !
  25. 25. What did we learn in Italy? MongoDB is designed for commodity hardware • Designed to add redundancy and automatic failure for simple, standard hardware. • Adding extra complexity from the DevOps side can interfere with MongoDB health and failover. • Our documentation addresses many edge cases. • Reach out to MongoDB Support before you try something “advanced” in production. !
  26. 26. Best cases ! Please remember this bit!
  27. 27. Minimum (viable) topology eu-west-2a PR IM A RY SECONDARY SECONDARY London ! eu-west-2b eu-west-2c
  28. 28. 2-datacenter topology (on-prem) Data center 1 PRIMARY SECONDARY ARBITER Data center 2 Cloud Write concern: Majority SECONDARY SECO N D A RY
  29. 29. Geo- Sharding ! Let’s put it all together: • Scalability • Redundancy • High availability • Data sovereignty
  30. 30. Region-level redundancy London Paris Frankfurt PR IM A RY SECONDARY SECONDARY Virginia Ohio California PRIMARY SECONDARY SEC O N D A RY Sydney Hong Kong Mumbai PRIMARY SECONDARY SECONDARY European Union North America Asia-Pacific mongos mongos mongos
  31. 31. Example topology Config servers contain metadata and shard key values Each shard contains data for countries in that region. With the balancer disabled, no data is transferred from that region. Each web applica>on limits which country codes it can process
  32. 32. Atlas ! • AWS, Azure, GCP • Secures your data with individual VPCs • Data encrypted with your key • Best practices for high availability, fault tolerance • Automates security upgrades • MongoDB SOC 2 Security Type II report available • Backups all in one region Fully automated MongoDB in the Cloud
  33. 33. Atlas Global Clusters • Atlas can automate regional clusters for you. • Focus on performance (low latency) not really compliance right now.
  34. 34. What did we learn? By now I hope you all agree: § For high-availability, we need data in at least three copies of the data, preferably in separate physical loca>ons. § MongoDB provides a good solu>ons to distribute terabytes of data sharded by workload or geopoli>cal requirements. § You can s>ll have a single database, but keep customer data separated.
  35. 35. What’s next? Here’s some other talks that might be interesting: Tutorial: Hands-on with Realm Mobile Database § Alexander Stigsen, Realm/MongoDB § Today, 1:00pm - 2:45pm § https://sched.co/PULz Using the New Security Features in MongoDB 4.2 § Tuesday • 12:45pm - 1:30pm § Kevin Albertson, MongoDB § https://sched.co/PwAP New Encryption Capabilities in MongoDB 4.2: A Deep Dive into Protecting Sensitive Workloads § Kenn White, MongoDB § Tuesday • 3:00pm - 3:45pm § https://sched.co/OJqV
  36. 36. Questions? Reach me at nic.c@mongodb.com or @niccottrell
  37. 37. GDPR Resources
  38. 38. GDPR: Frequently Asked Questions Website § How does MongoDB help my organizaAon comply with the GDPR? § How does MongoDB Atlas help me comply with the GDPR? § What commitments does MongoDB make with respect to the GDPR? § hEps://www.mongodb.com/cloud/trust/compliance/gdpr
  39. 39. GDPR: Impact to Your Data Management Landscape Whitepaper How MongoDB Can Help Meet GDPR Requirements § Discover § Defend § Detect § https://webassets.mongodb.com/mongodb_gdpr.pdf
  40. 40. GDPR and « The right to be forgotten ». Checklist Some key issues to keep in mind: § MongoDB Atlas backups can be deleted and re-synced at any time § Reference users in a consistent manner to make it easier to find and delete any historical/log documents by user ID
  41. 41. Pseudonymization with MongoDB Views Blog post About using Views for access control and auditing § https://www.mongodb.com/blog/post/pseudonymization-with-mongodb- views-the-solution-for-gdpr-and-game-of-thrones-spoilers
  42. 42. Extras
  43. 43. Audit log examples Failed authentication {"atype":"authenticate","ts":{"$date":"2017-02- 14T14:11:29.975+0100"},"local":{"ip":"127.0.1.1","port":27017},"remote":{"ip":" 127.0.0.1","port":42634},"users":[],"roles":[],"param":{"user":"root","db":"adm in","mechanism":"SCRAM-SHA-1"},"result":18} Failed insert (due to auth) {"atype":"authCheck","ts":{"$date":"2017-02- 14T14:15:49.161+0100"},"local":{"ip":"127.0.1.1","port":27017},"remote":{"ip":" 127.0.0.1","port":42636},"users":[{"user":"antun","db":"admin"}],"roles":[{"rol e":"read","db":"admin"}],"param":{"command":"insert","ns":"test.orders","args": {"insert":"orders","documents":[{"_id":{"$oid":"58a3030507bd5e3486b1220d"},"id" :1.0,"item":"paper clips"}],"ordered":true}},"result":13}
  44. 44. Redacted log examples Original log 2017-06-09T13:35:23.446-0400 I COMMAND [conn1] command internal.clients command: insert { documents: [ { _id: ObjectId('593adc5b99001b7d119d0c97'), name: "Joe", PII: " Sensitive Information" } ], ... Redacted log 2017-06-09T13:45:18.599-0400 I COMMAND [conn1] command internal.clients command: insert { insert: "###", documents: [ { _id: "###", name: "###", PII: "###" } ], }
  • nicholascottrell

    Aug. 7, 2019

No matter if you're thinking of migrating to MongoDB, or need to meet legal requirements for an existing on-prem cluster, this talk has you covered. We start with the basics of replication and sharding and quickly scale up, covering everything you need to know to control your data, and keep it safe from unexpected data loss or downtime - a well-designed MongoDB cluster should have no single point of failure. Learn how others are “stretching” what’s possible but why you shouldn't! I'll present real-world examples from my life in the field in Europe and beyond.

Views

Total views

531

On Slideshare

0

From embeds

0

Number of embeds

17

Actions

Downloads

8

Shares

0

Comments

0

Likes

1

×