Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

1. What’s new with MongoDB & Atlas, for you. Sigfrido “Sig” Narváez Executive Solution Architect, MongoDB @signarvaez DataConLA 2022

2. Agenda MongoDB Developer Data Platform For Platform Developers For Data Engineers & Scientists For Mobile Developers For DevOps & Security Engineers What’s new with MongoDB & Atlas

4. Traditional relational strengths Expressive query language and secondary indexes Enterprise security and management ACID transactions Additional NoSQL and big data strengths and innovations Native data structures and idiomatic drivers Laptop, your data center, multi-cloud Scale and distribution Mission critical and general purpose

5. Time Series collections Clustered indexes Window functions Live resharding Client-Side FLE KMIP & cloud KMS Atlas Serverless (preview) Atlas Search fast facets, function scores, synonyms Long running snapshot reads Sharded $lookup & $graphlookup Majority write concern default 4x faster initial sync Schema validation diagnostics New MongoDB Shell GA Resumable index builds Rewritten Swift driver Rewritten C# LINQ provider and .NET Analyzer PyMongoArrow API New accumulator operators Charts on Data Lake x509 certiﬁcate rotation Auditing ++ Atlas K8S controller Ops Manager migration wizard Doc-Level Concurrency RAFT / Fast Failover $lookup Ops Manager Compression ≤50 replicas Aggregation ++ Encrypted and In-Memory storage engines BI Connector Compass 2015 2016 2017 2018 2019 2020 2021-2022 3.0 & 3.2 WiredTiger (Acquisition + Integration) MongoDB Atlas ACID (Transactions) Stable API & Rapid Releases Views Graph Processing Zones ++Aggregation ++ Auto-balancing ++ Linearizable Reads Decimal Intra-cluster Compression Log Redaction Spark Connector ++ BI Connector ++ 3.4 Change Streams Retryable Writes Schema Validation Expressive $lookUp Query Expressivity Causal Consistency Consistent Sharded Secondary Reads Query Advisor End to End Compression WiredTiger 1m+ Collections MongoDB BI Connector ++ R Driver Charts (post GA) Atlas X-Region Replication Atlas Auto Storage Scaling 3.6 Replica set transactions Atlas global clusters 40% Faster Shard Migrations Atlas HIPAA Atlas LDAP Atlas Audit Atlas Enc. Storage Engine Atlas Backup Snapshots Type Conversions Snapshot Reads Non-Blocking Sec. Reads SHA-2 & TLS 1.1+ Compass Agg Pipeline Builder Compass Export to Code Free Monitoring Cloud Service Ops Manager K8s Beta 4.0 Distributed Transactions Client-Side Field Level Encryption Materialized Views Wildcard Indexes Global PIT Reads Large Transactions Mutable Shard Key Values Atlas Data Lake (Beta) Atlas Auto Scaling (Beta) Atlas Search (Beta) Multi-CAs Expressive Updates Apache Kafka Connector MongoDB Charts GA Retryable Reads & Writes New Index Builds 10x Faster stepDown Storage Node Watchdog Zstandard Compression 4.2 Union Custom Agg Expressions Reﬁnable Shard Keys Compound Hashed Shard Keys Mirrored Reads Hedged Reads Resumable Initial Sync Time-Based Oplog Retention Simultaneous Indexing Hidden Indexes Streaming Replication Global Read/Write Concerns Rust & Swift Drivers GA TLS 1.3 & Faster Client Auth OCSP Stapling Kerberos Utility Atlas Online Archive Auto-Scaling Schema Recommendations AWS IAM Auth & Atlas x509 Federated Queries Multi-cloud clusters 4.4 5.x The evolution of MongoDB Multi-Cloud 2022-2023 6.x Queryable encryption (preview) Atlas Serverless GA Column store indexes Cluster-to-cluster sync Independent analytic nodes scaling Flexible sync GA Asymmetric sync preview Kotlin and Flutter SDKs for Realm $lookup perf improvements Atlas CLI GA Atlas Data API GA Secondary index on measurements for time series Time series read perf + Atlas Data Lake with fully managed storage Atlas Data Federation Atlas Charts dashboard embedding Atlas SQL interface Encrypted audit log Change streams improvements Announcement of relational to MongoDB migrator Developer Data Platform

6. MongoDB Atlas: global cloud database Intelligent performance optimization Simpliﬁed data architecture Best-in-class security and operations Global and multi-cloud reach 95+ Regions

7. Application Application Application Application RDBMS RDBMS RDBMS RDBMS Key-Value DB Graph DB Wide Column Document DB Search Engine Search Engine Mobile DB Edge DB Enterprise Data Warehouse ETL ETL Caching Layer Caching Layer Sync ETL ETL Search Engine Time Series DB ETL Sync Data Lake ETL ETL ETL ETL ETL MongoDB’s Developer Data Platform aims to Reduce Technology Sprawl

8. Application D Application C Application B Application A Application D

9. Accelerate innovation with One interface. For any application. Anywhere. The MongoDB Atlas Developer Data Platform

11. { "_id" : ObjectId("5ad88534e3632e1a35a58d00"), "name" : { "first" : "John", "last" : "Doe" }, "address" : [ { "location" : "work", "address" : { "street" : "16 Hatfields", "city" : "London", "postal_code" : "SE1 8DJ"}, "geo" : { "type" : "Point", "coord" : [ -0.109081, 51.5065752]}}, + {...} ], "dob" : ISODate("1977-04-01T05:00:00Z"), "retirement_fund" : NumberDecimal("1292815.75") } Contrasting data models Tabular (Relational) Data Model Related data split across multiple records and tables Document Data Model Related data contained in a single, rich document

12. Basics: Replication & Sharding Primary Secondary Secondary Replication Application Driver MongoS Primary Secondary Secondary Shard 1 Primary Secondary Secondary Shard 2 Primary Secondary Secondary Shard N MongoS ••• ••• ••• Application High availability Replica sets Horizontal scalability Sharded Clusters 3 Sharding Strategies: Range, Hash & Zone collection-level Application Driver Application

13. Tunable Consistency Highest Data Safety Financial Transactions, PII, etc. writeConcern: majority readConcern: majority Lowest-latency Reads eCommerce & Media Catalogs, Content, etc readPreference: nearest Fastest Ingestion with tolerance for loss IoT, Viewer/Gamer Telemetry writeConcern: 0 readConcern: local

14. ACID Transactions try (ClientSession clientSession = client.startSession()) { clientSession.startTransaction(); collectionA.insertOne(clientSession, docOne); collectionA.insertOne(clientSession, docTwo); collectionB.updateOne(clientSession, findQry, updateStmt); collectionC.deleteOne(clientSession, findDelQry); clientSession.commitTransaction(); }

15. Schema Design Patterns: Subset Subset Pattern You want to display dependent information, however only part of it. 2+ strong entities. •The rest of the data is fetched only if needed •Examples: ‒ The Cast of a Movie ‒ Last 10 Movies an Actor has starred in https://www.mongodb.com/blog/post/ building-with-patterns-a-summary

16. Schema Design Patterns: Subset Subset Pattern You want to display dependent information, however only part of it. 2+ strong entities. •The rest of the data is fetched only if needed •Examples: ‒ The Cast of a Movie ‒ Last 10 Movies an Actor has starred in https://github.com/snarvaez/ mdb-distributedtx-subset https://www.mongodb.com/blog/post/ building-with-patterns-a-summary Multi-Cloud Distributed ACID Transaction

17. DEMO

18. Database driver and query language Search driver and query language Architectural complexity Difficult to keep data in sync between two separate systems. Requires its own systems and skills Operational overhead More to provision, secure, upgrade, patch, back up, monitor, scale, etc. Lower developer productivity Developers need to use different query APIs for database and search and coordinate schema changes Application Database Cluster Data Sync Search Cluster Pain of bolting on a search engine to a database

19. Application Same driver and query API Higher developer productivity Build database and search features using the same query API Fully managed platform Get the security, performance, and reliability of Atlas Simpliﬁed data architecture Automatic data synchronization, even as your data and schema changes Atlas Database Atlas Search MongoDB Atlas MongoDB Atlas: integrated database and search

20. Atlas Search Architecture Atlas Search offers the features needed to ﬁne-tune your search results to help users ﬁnd what they need, including: • Fuzzy search • Autocomplete • Filters and facets • Synonyms • Multi-data type support • Multi-language support • Highlighting • Custom scoring • Rich query language • More-Like-This • Dense Vector Search (Q3’22)

21. Just like that! Local Atlas Search Query: db.getMongo().setReadPref("nearest"); db.movies.aggregate([ { "$search": { "compound": { "must": [{ "text": { "query": arg, "path": ["plot","fullplot"], "fuzzy": { "maxEdits": 1, "prefixLength": 1, "maxExpansions": 256 }}}], "should": [{ "text": { "query": arg, "path": "title", "fuzzy": { "maxEdits": 1, "prefixLength": 1, "maxExpansions": 256 }}}] }} }]);

22. At the end of the day, building applications boils down to accessing your data from the database from your app in a scalable and seamless manner

23. The core Atlas Application Services… Database and Scheduled Events can be processed by serverless functions Data Triggers Out-of-the-box schema-based API with ability to customize with custom resolvers GraphQL API Out-of-the-box ﬂexible-based API with ability to customize with HTTPS Endpoints Data API Seamless mobile client sync without the REST APIs or conﬂict resolution Device Sync

24. …are supercharged by the supporting services Email Password, API Key, Custom Function, JWT Authentication - help provide identities to the client and IP Access list Auth/Security Deﬁne what data can and cannot be accessed Rules/Permissions Access the core services from native client libraries (React Native, Web Browser, iOS, Android, .NET, Flutter) SDKs Can be run in the context of Triggers, APIs, or from SDKs Serverless Functions

25. Integrate all of Atlas into your apps via a single endpoint Atlas GraphQL API GraphQL API Serverless Compute Layer over HTTPS Database Data Lake Frontends Federate into a larger supergraph Microservices and 3rd party services Search

26. DEMO

28. The best way to visualize MongoDB data Atlas Charts Fast Easy Powerful Built for the Document Model No Data Movement or Duplication Integrated into MongoDB Atlas Share & Collaborate Drive Actionable Insights Create Engaging Data Experiences

29. Atlas Charts in 2022 Dashboard Embedding Allow users to embed a full dashboard rather than a single chart Q1 2022 Q1 2022 H1 2022 2022 2022 Query Library Reuse queries in multiple charts Streamlined Data Sources Create charts without explicitly creating a data source Charts API Scheduled and Async Queries 2022 Auto suggest Charts for new users 2022 Enterprise Sharing

30. Interactive Filtering!

31. ● Model their application data as Documents ● Use MQL and the Aggregation Framework ● Want to query multiple operational databases with SQL-based BI Tools ● No time to learn or switch to MQL RESULT: Productivity RESULT: Friction Developers SQL Users Challenge: MongoDB + SQL Users

32. Atlas SQL Interface Visualize data using Named Connectors for BI tools Use ODBC/JDBC drivers for other SQL connections Use the Atlas SQL interface to unlock access to Atlas data from SQL based tools and SQL centric users so that we can embrace the ecosystem of people that want access to data in our platform. BI Tools ODBC/JDBC Drivers Atlas SQL provides a ﬁrst class SQL experience for querying Atlas data Atlas SQL Interface is currently in preview Atlas SQL interface Connect to the Atlas SQL interface from the most popular BI tools

33. Step 3: Add Named Connector -or- Connect via JDBC/ODBC Driver Atlas SQL Drivers Atlas Atlas SQL Interface BI or Analytics Tool Atlas SQL Connectors Step 1: Enable SQL Interface Step 2: Download Driver Atlas SQL Connectors & Drivers Atlas SQL Interface Leverages Atlas Data Federation as the query engine Use MongoSQL, a MongoDB-Native SQL-92 compatible dialect How does it work? MongoDB ODBC MongoDB JDBC Tableau Connector PowerBI Connector

34. Atlas Data Federation • Federate queries across Atlas and AWS S3 without data movement or transformation to power real-time applications. • Seamlessly aggregate data across 1+ sources. Gather insight, and enrich data without complex data pipelines. • Convert MongoDB data to a columnar ﬁle format and output to S3 to be consumed by downstream teams for enabling operational insights and modern app experiences.

35. Control Plane MongoDB Atlas Control {Data Federation Endpoint} Data Plane Atlas Data Federation architecture Application & Services Apps • Analytic Engines • Data Products MongoDB Compass • Data Exploration MongoDB SQL Support • Business Analytics • Reporting Data Science Tools • Statistics • Machine Learning DRIVERS: Javascript, Perl, Python, C, C++, Java, Ruby, Go, Scala, R MONGODB SHELL Compute Plane Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Data Federation computer node Region 1 Region 2 Atlas Clusters HTTP Store BYO AWS S3 or Atlas Data Lake 1+ Any Atlas Cluster Type 1+

36. Atlas Data Lake is a combination of several pieces of new functionality: ● Automated Data Extraction ● Partition Level Indexing ● MongoDB Native Analytic File Format ● Object Storage This set of functionality provides you a workload isolated view of your cluster data in economical object storage with high performance for analytic type queries. Atlas Data Lake A fully managed, high performance, analytic storage option in Atlas, powered by cost-effective cloud object storage.

37. PARTITION INDEXES Write Metadata Write Data Atlas CLOUD DATABASE AWS S3 (Future) A B C A B C A B C Enriched Extraction, Partitioning & Auto-Balancing ANALYTIC FILE FORMAT

38. What is in a Partition Index? ● Unique Values or Ranges ● Sum ● Count ● Average ● ... Improve query performance ● Improve point query performance by ﬁnding partitions faster. ● Additionally, certain aggregate queries like Counts and Sums are “covered” by the index. FEDERATED ENDPOINT PARTITION INDEXES COLUMNAR PARTITIONS Note: All components of Atlas Data Lake are fully managed solutions that exist in the Atlas Infrastructure. { name: [‘ben’, ‘john’, ‘james’] age: 34-75 } { name: [‘prem’, ‘sally’] age: 23-75 } { name: [‘tifani’, ‘joe’] age: 23-75 } A B C A B C A B C Partition Indexes

39. MongoDB Spark Connector • Massively parallel processing, machine learning and streaming at scale • Process data “in place” avoiding the latency of ETL • Aggregation pre-ﬁltering with secondary indexing to only select data that’s required • Read from secondaries isolate analytics workload from business-critical operations • Shard aware for data locality SQL Machine Learning Streaming Graph Scala, Java, Python, R APIs MongoDB Spark Connector MongoDB Spark Connector

40. MongoDB Kafka Connector Sink: Source: MongoDB Database MongoDB Connector topicA topicB topicC Kafka Cluster Writes documents to DB collection Receives events from Kafka Topic(s) MongoDB Database MongoDB Connector Kafka Cluster Receives documents from DB collection Writes events to Kafka Topics(s) topicA topicB topicC Change Streams Built in partnership with Conﬂuent

41. Analytical Nodes Analytics Consumers Primary Secondary Secondary Secondary {Analytics Node} Transactional Applications Secondary {Analytics Node} Rich MongoDB Query API and distributed architecture allows you to run both Transactions and Analytics on the same cluster with no resource contention Higher Tier (RAM / CPU) for demanding analytical & ML

42. Atlas → Atlas On-prem → Atlas On-prem → On-prem On-prem cluster Atlas cluster Atlas cluster Atlas cluster On-prem cluster On-prem cluster Atlas → On-Prem Atlas cluster On-prem cluster Cluster-to-Cluster Sync

44. Real-time experiences at scale require complex sync & conflict resolution Data synchronization JSON NATIVE OBJECT ORM MSG QUEUE BACKEND OBJECT REST CALL TO API PUSH SERVICE PUSH NOTIFICATION COMPLEX RETRY & ERROR HANDLING POTENTIAL APP CRASH OFFLINE? WRITE CONFLICTS? INVALID DATA? SERVER ERROR? TIMED OUT? Backend database WRITE CONFLICTS? Mobile devices with local data

45. The easiest way to build and scale mobile apps Atlas Device Sync Out-of-the-box device-cloud synchronization Atlas The leading Developer Data Platform on the cloud Realm Mobile-optimized data store Lightning fast Super lightweight No ORM needed No DAO needed Dynamic, ﬂexible sync options Granular permissions Built-in conﬂict resolution Built-in error handling

46. Supported frameworks and languages Kotlin/Kotlin Multiplatform Swift/SwiftUI .NET React Native Node.js / Javascript Dart/Flutter (beta) Xamarin Unity Java Ionic MAUI

47. Provides a single data layer across all platforms WatchOS Mobile devices (iOS, Android) Tablets (iOS, Android, Windows) Laptops, Desktops POS Systems IoT (connected cars, wearables, equipment, etc.) Atlas Device Sync Atlas

48. How Device Sync works: dynamic queries Customer Store Warehouse Delivery.date == 01-01-2022 Inventory.store == 101 category == 'jeans' AND size == 8 AND price < 40 Atlas Device Sync MongoDB Atlas Atlas Device Sync Atlas Device Sync

49. How Device Sync works: hierarchical permissions VP Manager Sales Atlas Device Sync MongoDB Atlas Atlas Device Sync Atlas Device Sync Sales.team == 'NA-US' Sales.team == 'US-WEST' Sales.owner == 'Bill Buckner'

50. How Device Sync works: document-level permissions Banker READ: ALL FIELDS WRITE: ALL FIELDS MongoDB Atlas Atlas Device Sync Trades Support Bank User Support READ: ALL FIELDS WRITE: SOME FIELDS End User READ: ALL FIELDS WRITE: PROFILE ONLY

51. { "_id": { "$oid": "60e51a64d0ff67f998bc1732" }, "name": "Jane Leaf", "address": { "street": "Bedford Ave" "city": "New York" } "billing": { "creditCard": "1234123412341234" "balance": "20" } "vitals": [{ "visitDate": "May62022” "bpm": "80” "bloodPressure": "120/80" ]} "prescriptions":[{ "name": "tylenol” "quantity": "20”} ]} } How Device Sync works: field-level permissions Patient { "_id": { "$oid": "60e51a64d0ff67f998bc1732" }, "name": "Jane Leaf", "address": { "street": "Bedford Ave" "city": "New York" } "billing": { "creditCard": "1234123412341234" "balance": "20" } } Billing { "_id": { "$oid": "60e51a64d0ff67f998bc1732" }, "name": "Jane Leaf", "vitals": [{ "visitDate": "May62022” "bpm": "80” "bloodPressure": "120/80" ]} "prescriptions":[{ "name": "tylenol” "quantity": "20”} ]} } Doctor ID Address Billing Vitals Prescription

53. Business Continuity ● 3 AZ’s default ● 3 Clouds: AWS, Google, Azure ● Cross-Region, Cross-Cloud & Multi-Cloud ● Automatic-failover ● Continuous Backup ○ Second-resolution RPO <<< ● Auto Scale Compute & Storage ● Atlas Serverless

54. Automation, APM & Integrations ● K8S Operator ● Hashicorp Terraform & Vault ● AWS CloudFormation ● REST API & CLI ● Built-in Metrics, Alerts, Performance Advisor ● DataDog, Prometheus, PagerDuty, etc.

55. DEMO Migration to Atlas ● Live Migrate (hosted by Atlas) ● Push Migrate (from OpsMgr or Cloud Mgr) -Enterprise customers - NEW!! ● Cluster to Cluster Replication - NEW!!! ● mongomirror (hosted by you)

56. MongoDB Security Assurance HITRUST https://www.mongodb.com/cloud/trust

57. What’s so hard about “right to erasure”? Deleting user data from the database is easy • But what about user data stored in backups and logs... how do you delete that? • Risk of archived data being restored and used With Client-side FLE, you simply destroy the encryption key for the user’s data, and their PII is unusable and unrecoverable: in the DB, in backups, in logs… ...everywhere

58. Network (In-Flight) Storage (At-Rest) Memory (In-Use) Encryption Scope TLS ❌ ❌ All data transmitted over the network Volume and Storage Engine ❌ ❌ All data stored in the database cluster Client-Side FLE Speciﬁc ﬁelds containing sensitive data Addressing different threat models End-to-end encryption for defense in depth

59. How Client-side FLE works

60. Accelerate innovation with One interface. For any application. Anywhere. The MongoDB Atlas Developer Data Platform

61. db.runCommand(“Thank you!!”); db.QnA.find({Q:“answer”}).explain(); Sig Narváez Executive Solution Architect Aug 13 2022 Sigfrido “Sig” Narváez Executive Solution Architect, MongoDB @signarvaez

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

You are reading a preview.

Check these out next

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

More Related Content

More from Data Con LA (20)

Recently uploaded (20)