Migrating from MySQL to MongoDB
•
1 like•655 views
MongoDB was chosen to replace an existing MySQL system due to its high performance for worldwide access across multiple locations, simple usage without needing a DBA, and ability to do real-time reporting without impacting production. MongoDB uses a document-oriented model and simple querying of JSON, supports availability and automatic failover between replica sets, and was considered a better fit than alternative NoSQL databases for the project's requirements without needing to implement complex sharding.
Report
Share
Report
Share
Download to read offline
Recommended
MongoDB : Scaling, Security & Performance
MongoDB provides high performance for write-intensive workloads, with the ability to handle 80,000 inserts per second on a single node. It offers easy replication and high availability. For large datasets, MongoDB scales horizontally using a sharding architecture with query routers, config servers, and shards. MongoDB ensures data security using encryption, access control, and authentication features.
MongoDB Administration ~ Kevin Hanson
The document provides an overview of MongoDB administration topics including:
- Supported platforms for MongoDB
- Common MongoDB administration tools like mongod, mongo, and mongodump
- How to set up replication with replica sets for high availability and data redundancy
- Popular tools for MongoDB monitoring like munin, ganglia, nagios, and cacti
- How to take backups with mongodump or filesystem snapshots
- The importance of indexing queries for performance and avoiding table scans
No sql
Amazon DynamoDB is a scalable and distributed NoSQL database that scales horizontally and seamlessly over hundreds of servers. It has flexible schemas and automatic data replication but has limitations on row and query sizes. HBase is an open-source, distributed, scalable big data store that provides real-time random access to big data and is highly scalable and reliable but has single points of failure. MongoDB is a scalable, schema-less document database ideal for flexible schemas with easy horizontal scaling and high availability but has higher data sizes than relational databases and less querying flexibility. Redis is an in-memory data structure store with no query language and offers fast data access but lacks security and recoverability of relational databases.
What every developer should know about database scalability, PyCon 2010
The document discusses the challenges of scaling databases and provides strategies for improving scalability. It covers:
1) The two main types of database operations that need to be scaled are reads and writes. Common "band-aid" approaches that don't actually scale the database are discussed.
2) Scaling reads can be improved through caching, partitioning caches, and replication. Scaling writes is more difficult and requires approaches like write-back caching, asynchronous replication, and data partitioning.
3) Partitioning data through sharding or vertical partitioning distributes the load across multiple database nodes. Consistent hashing is presented as a method to determine which server owns each data key. The challenges of non-transparent partitioning approaches are also
NoSQL
This document provides an introduction to NoSQL databases. It discusses that NoSQL is a non-relational approach to data storage that does not rely on fixed schemas and provides better scalability than traditional relational databases. Specific NoSQL examples mentioned include document databases like CouchDB and MongoDB, as well as key-value stores like Redis and Cassandra. The document outlines some of the characteristics and usage of these NoSQL solutions.
NoSQL
The document discusses NoSQL databases and their advantages compared to SQL databases. It defines NoSQL as any database that is not relational and describes the main categories of NoSQL databases - key-value stores, document databases, wide column stores like BigTable, and graph databases. It also covers common use cases for different NoSQL databases and examples of companies using NoSQL technologies like MongoDB, Cassandra, and HBase.
MongoDB Administration 20110922
This document summarizes a presentation about MongoDB best practices and lessons learned. The presentation covers: an introduction to MongoDB and how it compares to SQL databases; supported platforms and drivers; common use cases and misuses; best practices including configuration options, indexing, and operations; lessons learned around updates, sharding, and embedding data; and resources for learning more including books, training, forums, and future MongoDB releases.
HBaseCon 2015- HBase @ Flipboard
My presentation on how we use HBase at Flipboard, given at HBaseCon in San Francisco on May 7th, 2015.
Recommended
MongoDB : Scaling, Security & Performance
MongoDB provides high performance for write-intensive workloads, with the ability to handle 80,000 inserts per second on a single node. It offers easy replication and high availability. For large datasets, MongoDB scales horizontally using a sharding architecture with query routers, config servers, and shards. MongoDB ensures data security using encryption, access control, and authentication features.
MongoDB Administration ~ Kevin Hanson
The document provides an overview of MongoDB administration topics including:
- Supported platforms for MongoDB
- Common MongoDB administration tools like mongod, mongo, and mongodump
- How to set up replication with replica sets for high availability and data redundancy
- Popular tools for MongoDB monitoring like munin, ganglia, nagios, and cacti
- How to take backups with mongodump or filesystem snapshots
- The importance of indexing queries for performance and avoiding table scans
No sql
Amazon DynamoDB is a scalable and distributed NoSQL database that scales horizontally and seamlessly over hundreds of servers. It has flexible schemas and automatic data replication but has limitations on row and query sizes. HBase is an open-source, distributed, scalable big data store that provides real-time random access to big data and is highly scalable and reliable but has single points of failure. MongoDB is a scalable, schema-less document database ideal for flexible schemas with easy horizontal scaling and high availability but has higher data sizes than relational databases and less querying flexibility. Redis is an in-memory data structure store with no query language and offers fast data access but lacks security and recoverability of relational databases.
What every developer should know about database scalability, PyCon 2010
The document discusses the challenges of scaling databases and provides strategies for improving scalability. It covers:
1) The two main types of database operations that need to be scaled are reads and writes. Common "band-aid" approaches that don't actually scale the database are discussed.
2) Scaling reads can be improved through caching, partitioning caches, and replication. Scaling writes is more difficult and requires approaches like write-back caching, asynchronous replication, and data partitioning.
3) Partitioning data through sharding or vertical partitioning distributes the load across multiple database nodes. Consistent hashing is presented as a method to determine which server owns each data key. The challenges of non-transparent partitioning approaches are also
NoSQL
This document provides an introduction to NoSQL databases. It discusses that NoSQL is a non-relational approach to data storage that does not rely on fixed schemas and provides better scalability than traditional relational databases. Specific NoSQL examples mentioned include document databases like CouchDB and MongoDB, as well as key-value stores like Redis and Cassandra. The document outlines some of the characteristics and usage of these NoSQL solutions.
NoSQL
The document discusses NoSQL databases and their advantages compared to SQL databases. It defines NoSQL as any database that is not relational and describes the main categories of NoSQL databases - key-value stores, document databases, wide column stores like BigTable, and graph databases. It also covers common use cases for different NoSQL databases and examples of companies using NoSQL technologies like MongoDB, Cassandra, and HBase.
MongoDB Administration 20110922
This document summarizes a presentation about MongoDB best practices and lessons learned. The presentation covers: an introduction to MongoDB and how it compares to SQL databases; supported platforms and drivers; common use cases and misuses; best practices including configuration options, indexing, and operations; lessons learned around updates, sharding, and embedding data; and resources for learning more including books, training, forums, and future MongoDB releases.
HBaseCon 2015- HBase @ Flipboard
My presentation on how we use HBase at Flipboard, given at HBaseCon in San Francisco on May 7th, 2015.
MongoDb - Details on the POC
MongoDB is a document-oriented, schema-free, scalable, high-performance, open-source database that bridges the gap between key-value stores and traditional relational databases. MongoDB uses a document-oriented data model where data is stored in documents that map to programming language data types, which reduces the need for joins. It provides high performance through an absence of joins and support for indexing of embedded documents and arrays.
Starting with MongoDB
This document provides an overview and introduction to MongoDB, including:
- MongoDB is a non-relational, document-oriented database with dynamic schemas and stores data in JSON-like documents rather than tables.
- It is well suited for prototyping, quick development refactoring, non-relational data, log data and metadata. Related data and needing joins may not be the best use case.
- Installation and configuration process varies by operating system but involves downloading packages and creating a data directory with correct permissions.
- The MongoDB command line interface provides commands to view and manage databases, collections, documents and perform basic operations like create, read, update and delete.
No sql - { If and Else }
This document provides an overview of NoSQL databases, including:
1) NoSQL databases differ from relational databases by being schemaless and not using joins. They were created to address limitations of relational databases.
2) NoSQL databases can be categorized as key-value, column-oriented, document-based or graph-based. MongoDB is presented as an example document-based NoSQL database.
3) The CAP theorem states that a distributed system cannot simultaneously provide consistency, availability, and partition tolerance. Pros and cons of NoSQL databases are discussed.
Mongo DB
- Mongo DB is an open-source document database that provides high performance, a rich query language, high availability through clustering, and horizontal scalability through sharding. It stores data in BSON format and supports indexes, backups, and replication.
- Mongo DB is best for operational applications using unstructured or semi-structured data that require large scalability and multi-datacenter support. It is not recommended for applications with complex calculations, finance data, or those that scan large data subsets.
- The next session will provide a security and replication overview and include demonstrations of installation, document creation, queries, indexes, backups, and replication and sharding if possible.
MongoDB Introduction - Document Oriented Nosql Database
- MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas, horizontal scaling with sharding and replication, and high performance.
- It is used by many large companies and is a good fit for huge, unstructured data or when dynamic schemas are needed. However, it does not support transactions or have the same data integrity features as relational databases.
- MongoDB can store data in flexible document formats, allows ad hoc queries, and supports common operations like insert, update, delete. Features include replication, sharding, indexing, geospatial queries and MapReduce.
Cassandra database design best practises
1. Key considerations for Cassandra database design include distributing storage without a single point of failure, linear scalability, and understanding the tradeoffs of consistency models.
2. The document provides tips such as using time-to-live columns, wide rows for efficiency, tunable consistency levels, modeling one-to-many and many-to-many relationships, and indexing strategies.
3. Best practices are outlined like limiting collection sizes, modeling time series data, and avoiding indexes on high-cardinality or frequently updated columns.
MongoDB
MongoDB is a document-oriented NoSQL database that stores data as JSON-like documents. It is schema-less, scales easily, supports dynamic queries on documents, and stores data in BSON format. MongoDB is good for high write loads, high availability, large and changing datasets. Installation is simple, and it supports replication and sharding for availability and scaling. Data can be embedded or referenced between documents. Indexes and text search are supported. Programming involves JavaScript and MongoDB methods.
MongoDB DOC v1.5
This document provides information about MongoDB, including:
- MongoDB is a cross-platform document-oriented database that provides high performance, high availability, and easy scalability.
- Data is stored in MongoDB in the form of JSON-like documents with dynamic schemas, instead of using fixed table schemas as in SQL-based databases.
- Relationships between documents can be modeled either by embedding one document inside another or by storing references between separate documents.
MongoDB Replication and Sharding
The document provides information on MongoDB replication and sharding. Replication allows for redundancy and increased data availability by synchronizing data across multiple database servers. A replica set consists of a primary node that receives writes and secondary nodes that replicate the primary. Sharding partitions data across multiple machines or shards to improve scalability and allow for larger data sets and higher throughput. Sharded clusters have shards that store data, config servers that store metadata, and query routers that direct operations to shards.
MongoDB for the SQL Server
This document provides an overview of MongoDB, a NoSQL document database, comparing its features to SQL databases. It demonstrates how to setup MongoDB, import and query data, create indexes, and connect to MongoDB from C#. Key features covered include MongoDB's document model with dynamic schemas, indexing, aggregation capabilities, and scaling through replication and sharding.
Open source Technology
Open source technologies allow anyone to view, modify, and distribute source code freely. The key characteristics of open source are that it is free to use and modify. Anyone can improve open source code by adding new functionality. As more people contribute code, the potential uses of open source software grow beyond what the original creator intended. To be a web developer requires a passion for learning and skills with technologies like HTML, PHP, Linux, Apache, MySQL, and PHP (LAMP stack). Caching and NoSQL databases like MongoDB can improve performance of dynamic web applications.
Zarafa Scaling & Performance
This document discusses several improvements to scaling and performance in Zarafa 7.0, including using a hybrid row and column data storage model, removing unused read flags from the index, incremental tracking of message counters, deferred writes to reduce I/O, and leveraging native asynchronous I/O in MySQL 5.5. It also describes using the Mail Store Relocator for bidirectional data synchronization within and between clusters.
Optimizing InnoDB bufferpool usage
The document discusses strategies for optimizing InnoDB buffer pool usage in MySQL/MariaDB databases to decrease I/O load. It explains that concentrating frequently accessed data into fewer pages results in more efficient caching. Specific techniques recommended include increasing record density through smaller records, removing unnecessary indexes, and improving data locality through clustered primary keys on commonly queried fields. The use of random identifiers like GUIDs is discouraged as it reduces data locality.
Cassandra Day Atlanta 2015: Introduction to Apache Cassandra & DataStax Enter...
This is a crash course introduction to Cassandra. You'll step away understanding how it's possible to to utilize this distributed database to achieve high availability across multiple data centers, scale out as your needs grow, and not be woken up at 3am just because a server failed. We'll cover the basics of data modeling with CQL, and understand how that data is stored on disk. We'll wrap things up by setting up Cassandra locally, so bring your laptops.
Zarafa SummerCamp 2012 - Tips & tricks for running Zarafa is larger scale env...
The document discusses the InnoDB buffer pool in MySQL and provides recommendations for optimizing a Zarafa installation in a larger environment. It explains that the InnoDB buffer pool acts as both a read and write cache. It flushes dirty pages to disk when MySQL is idle, max_dirty_pages_pct is reached, or transaction logs are full. An example configuration is given. It also provides general recommendations for optimizing aspects like kernels, storage, reverse proxy configuration, LDAP, and more.
Mongo db
MongoDB is a popular NoSQL database. This presentation was delivered during a workshop.
First it talks about NoSQL databases, shift in their design paradigm, focuses a little more on document based NoSQL databases and tries drawing some parallel from SQL databases.
Second part, is for hands-on session of MongoDB using mongo shell. But the slides help very less.
At last it touches advance topics like data replication for disaster recovery and handling big data using map-reduce as well as Sharding.
Hardware Provisioning for MongoDB
1) Two case studies are presented of companies provisioning MongoDB clusters correctly and incorrectly. A Spanish bank stored 6 months of logs (18TB total) in MongoDB and sized their cluster to handle a 4TB working set. An online retailer moved their product catalog to MongoDB and deployed a single replica set large enough to hold all 240GB of data to meet their performance needs.
2) A software company deployed a replica set incorrectly by overspending on physical servers and underprovisioning EC2 instances, causing the EC2 instances to be a bottleneck. Another company did not provision enough RAM for their workload.
3) Key lessons are to understand performance needs up front, get help from MongoDB, conduct proof of concepts to
What Every Developer Should Know About Database Scalability
Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955
Ndb cluster 80_ycsb_mem
NDB Cluster 8.0 was benchmarked using YCSB, the de facto cloud benchmark. The benchmark showed that:
1) NDB Cluster achieved the highest throughput of any distributed in-memory transactional database, scaling linearly as data nodes were added.
2) Increasing the number of rows in the cluster from 300M to 600M rows showed no impact on performance or latency.
3) Performance was optimized for latency versus throughput by adjusting load generators and the number of data manager threads per data node.
Hdfs internals
Hadoop uses large 64MB blocks by default to store file data in HDFS for improved performance. The namenode manages file metadata and knows which datanodes store each block. Datanodes store and retrieve blocks as requested by clients. The secondary namenode helps manage the namenode metadata but cannot replace it in case of failure. Writing files involves breaking them into blocks and storing replicas across datanodes, while reading locates blocks and retrieves their data.
MongoDB 2.4 and spring data
This document provides an introduction and agenda for a presentation on MongoDB 2.4 and Spring Data. The presentation will include a quick introduction to NoSQL and MongoDB, an overview of Spring Data's MongoDB support including configuration, templates, repositories and queries, and details on metadata mapping, aggregation functions, GridFS file storage and indexes in MongoDB.
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
We'll be covering some aspects of our architecture, highlighting differences between MongoDB and Cassandra. We'll go in depth to explain why Cassandra is a better choice for our general purpose Application Platform (SHIFT) as well as our Media Buying Analytics tool (the SHIFT Media Manager). We'll be going over common design patterns people might be familiar with coming from a background with MongoDB and highlight how Cassandra would be used as a better alternative. We'll also touch more on cqlengine which is nearing feature completeness as the Cassandra object mapper for Python.
More Related Content
What's hot
MongoDb - Details on the POC
MongoDB is a document-oriented, schema-free, scalable, high-performance, open-source database that bridges the gap between key-value stores and traditional relational databases. MongoDB uses a document-oriented data model where data is stored in documents that map to programming language data types, which reduces the need for joins. It provides high performance through an absence of joins and support for indexing of embedded documents and arrays.
Starting with MongoDB
This document provides an overview and introduction to MongoDB, including:
- MongoDB is a non-relational, document-oriented database with dynamic schemas and stores data in JSON-like documents rather than tables.
- It is well suited for prototyping, quick development refactoring, non-relational data, log data and metadata. Related data and needing joins may not be the best use case.
- Installation and configuration process varies by operating system but involves downloading packages and creating a data directory with correct permissions.
- The MongoDB command line interface provides commands to view and manage databases, collections, documents and perform basic operations like create, read, update and delete.
No sql - { If and Else }
This document provides an overview of NoSQL databases, including:
1) NoSQL databases differ from relational databases by being schemaless and not using joins. They were created to address limitations of relational databases.
2) NoSQL databases can be categorized as key-value, column-oriented, document-based or graph-based. MongoDB is presented as an example document-based NoSQL database.
3) The CAP theorem states that a distributed system cannot simultaneously provide consistency, availability, and partition tolerance. Pros and cons of NoSQL databases are discussed.
Mongo DB
- Mongo DB is an open-source document database that provides high performance, a rich query language, high availability through clustering, and horizontal scalability through sharding. It stores data in BSON format and supports indexes, backups, and replication.
- Mongo DB is best for operational applications using unstructured or semi-structured data that require large scalability and multi-datacenter support. It is not recommended for applications with complex calculations, finance data, or those that scan large data subsets.
- The next session will provide a security and replication overview and include demonstrations of installation, document creation, queries, indexes, backups, and replication and sharding if possible.
MongoDB Introduction - Document Oriented Nosql Database
- MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas, horizontal scaling with sharding and replication, and high performance.
- It is used by many large companies and is a good fit for huge, unstructured data or when dynamic schemas are needed. However, it does not support transactions or have the same data integrity features as relational databases.
- MongoDB can store data in flexible document formats, allows ad hoc queries, and supports common operations like insert, update, delete. Features include replication, sharding, indexing, geospatial queries and MapReduce.
Cassandra database design best practises
1. Key considerations for Cassandra database design include distributing storage without a single point of failure, linear scalability, and understanding the tradeoffs of consistency models.
2. The document provides tips such as using time-to-live columns, wide rows for efficiency, tunable consistency levels, modeling one-to-many and many-to-many relationships, and indexing strategies.
3. Best practices are outlined like limiting collection sizes, modeling time series data, and avoiding indexes on high-cardinality or frequently updated columns.
MongoDB
MongoDB is a document-oriented NoSQL database that stores data as JSON-like documents. It is schema-less, scales easily, supports dynamic queries on documents, and stores data in BSON format. MongoDB is good for high write loads, high availability, large and changing datasets. Installation is simple, and it supports replication and sharding for availability and scaling. Data can be embedded or referenced between documents. Indexes and text search are supported. Programming involves JavaScript and MongoDB methods.
MongoDB DOC v1.5
This document provides information about MongoDB, including:
- MongoDB is a cross-platform document-oriented database that provides high performance, high availability, and easy scalability.
- Data is stored in MongoDB in the form of JSON-like documents with dynamic schemas, instead of using fixed table schemas as in SQL-based databases.
- Relationships between documents can be modeled either by embedding one document inside another or by storing references between separate documents.
MongoDB Replication and Sharding
The document provides information on MongoDB replication and sharding. Replication allows for redundancy and increased data availability by synchronizing data across multiple database servers. A replica set consists of a primary node that receives writes and secondary nodes that replicate the primary. Sharding partitions data across multiple machines or shards to improve scalability and allow for larger data sets and higher throughput. Sharded clusters have shards that store data, config servers that store metadata, and query routers that direct operations to shards.
MongoDB for the SQL Server
This document provides an overview of MongoDB, a NoSQL document database, comparing its features to SQL databases. It demonstrates how to setup MongoDB, import and query data, create indexes, and connect to MongoDB from C#. Key features covered include MongoDB's document model with dynamic schemas, indexing, aggregation capabilities, and scaling through replication and sharding.
Open source Technology
Open source technologies allow anyone to view, modify, and distribute source code freely. The key characteristics of open source are that it is free to use and modify. Anyone can improve open source code by adding new functionality. As more people contribute code, the potential uses of open source software grow beyond what the original creator intended. To be a web developer requires a passion for learning and skills with technologies like HTML, PHP, Linux, Apache, MySQL, and PHP (LAMP stack). Caching and NoSQL databases like MongoDB can improve performance of dynamic web applications.
Zarafa Scaling & Performance
This document discusses several improvements to scaling and performance in Zarafa 7.0, including using a hybrid row and column data storage model, removing unused read flags from the index, incremental tracking of message counters, deferred writes to reduce I/O, and leveraging native asynchronous I/O in MySQL 5.5. It also describes using the Mail Store Relocator for bidirectional data synchronization within and between clusters.
Optimizing InnoDB bufferpool usage
The document discusses strategies for optimizing InnoDB buffer pool usage in MySQL/MariaDB databases to decrease I/O load. It explains that concentrating frequently accessed data into fewer pages results in more efficient caching. Specific techniques recommended include increasing record density through smaller records, removing unnecessary indexes, and improving data locality through clustered primary keys on commonly queried fields. The use of random identifiers like GUIDs is discouraged as it reduces data locality.
Cassandra Day Atlanta 2015: Introduction to Apache Cassandra & DataStax Enter...
This is a crash course introduction to Cassandra. You'll step away understanding how it's possible to to utilize this distributed database to achieve high availability across multiple data centers, scale out as your needs grow, and not be woken up at 3am just because a server failed. We'll cover the basics of data modeling with CQL, and understand how that data is stored on disk. We'll wrap things up by setting up Cassandra locally, so bring your laptops.
Zarafa SummerCamp 2012 - Tips & tricks for running Zarafa is larger scale env...
The document discusses the InnoDB buffer pool in MySQL and provides recommendations for optimizing a Zarafa installation in a larger environment. It explains that the InnoDB buffer pool acts as both a read and write cache. It flushes dirty pages to disk when MySQL is idle, max_dirty_pages_pct is reached, or transaction logs are full. An example configuration is given. It also provides general recommendations for optimizing aspects like kernels, storage, reverse proxy configuration, LDAP, and more.
Mongo db
MongoDB is a popular NoSQL database. This presentation was delivered during a workshop.
First it talks about NoSQL databases, shift in their design paradigm, focuses a little more on document based NoSQL databases and tries drawing some parallel from SQL databases.
Second part, is for hands-on session of MongoDB using mongo shell. But the slides help very less.
At last it touches advance topics like data replication for disaster recovery and handling big data using map-reduce as well as Sharding.
Hardware Provisioning for MongoDB
1) Two case studies are presented of companies provisioning MongoDB clusters correctly and incorrectly. A Spanish bank stored 6 months of logs (18TB total) in MongoDB and sized their cluster to handle a 4TB working set. An online retailer moved their product catalog to MongoDB and deployed a single replica set large enough to hold all 240GB of data to meet their performance needs.
2) A software company deployed a replica set incorrectly by overspending on physical servers and underprovisioning EC2 instances, causing the EC2 instances to be a bottleneck. Another company did not provision enough RAM for their workload.
3) Key lessons are to understand performance needs up front, get help from MongoDB, conduct proof of concepts to
What Every Developer Should Know About Database Scalability
Replication. Partitioning. Relational databases. Bigtable. Dynamo. There is no one-size-fits-all approach to scaling your database, and the CAP theorem proved that there never will be. This talk will explain the advantages and limits of the approaches to scaling traditional relational databases, as well as the tradeoffs made by the designers of newer distributed systems like Cassandra. These slides are from Jonathan Ellis's OSCON 09 talk: http://en.oreilly.com/oscon2009/public/schedule/detail/7955
Ndb cluster 80_ycsb_mem
NDB Cluster 8.0 was benchmarked using YCSB, the de facto cloud benchmark. The benchmark showed that:
1) NDB Cluster achieved the highest throughput of any distributed in-memory transactional database, scaling linearly as data nodes were added.
2) Increasing the number of rows in the cluster from 300M to 600M rows showed no impact on performance or latency.
3) Performance was optimized for latency versus throughput by adjusting load generators and the number of data manager threads per data node.
Hdfs internals
Hadoop uses large 64MB blocks by default to store file data in HDFS for improved performance. The namenode manages file metadata and knows which datanodes store each block. Datanodes store and retrieve blocks as requested by clients. The secondary namenode helps manage the namenode metadata but cannot replace it in case of failure. Writing files involves breaking them into blocks and storing replicas across datanodes, while reading locates blocks and retrieves their data.
What's hot (20)
MongoDB Introduction - Document Oriented Nosql Database
MongoDB Introduction - Document Oriented Nosql Database
Cassandra Day Atlanta 2015: Introduction to Apache Cassandra & DataStax Enter...
Cassandra Day Atlanta 2015: Introduction to Apache Cassandra & DataStax Enter...
Zarafa SummerCamp 2012 - Tips & tricks for running Zarafa is larger scale env...
Zarafa SummerCamp 2012 - Tips & tricks for running Zarafa is larger scale env...
What Every Developer Should Know About Database Scalability
What Every Developer Should Know About Database Scalability
Similar to Migrating from MySQL to MongoDB
MongoDB 2.4 and spring data
This document provides an introduction and agenda for a presentation on MongoDB 2.4 and Spring Data. The presentation will include a quick introduction to NoSQL and MongoDB, an overview of Spring Data's MongoDB support including configuration, templates, repositories and queries, and details on metadata mapping, aggregation functions, GridFS file storage and indexes in MongoDB.
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
We'll be covering some aspects of our architecture, highlighting differences between MongoDB and Cassandra. We'll go in depth to explain why Cassandra is a better choice for our general purpose Application Platform (SHIFT) as well as our Media Buying Analytics tool (the SHIFT Media Manager). We'll be going over common design patterns people might be familiar with coming from a background with MongoDB and highlight how Cassandra would be used as a better alternative. We'll also touch more on cqlengine which is nearing feature completeness as the Cassandra object mapper for Python.
Scaling MongoDB - Presentation at MTP
This document discusses scaling MongoDB. It covers key concepts like sharding and replication which allow horizontal and high availability scaling. It describes types of scaling like cluster, performance, and data scaling. Implementation involves architecture with mongod, mongos, and config servers. Choosing an appropriate shard key is important, and depends on the application's data model, queries, and requirements. Scaling MongoDB effectively requires a ground-up approach with the application designed with scaling in mind.
JasperWorld 2012: Reinventing Data Management by Max Schireson
Max Schireson is the president of 10gen, the makers of MongoDB. In his talk, he discusses how database assumptions have changed with trends like big data, cloud computing, and faster hardware. He challenges some traditional relational database assumptions, such as the need for transactions, normal forms, and SQL queries. He argues different use cases require different database approaches, and that new database technologies will find niches alongside traditional relational databases.
MongoDB
This document discusses MongoDB and provides information on why it is useful, how it works, and best practices. Specifically, it notes that MongoDB is a noSQL database that is easy to use, scalable, and supports high performance and availability. It is well-suited for flexible schemas, embedded documents, and complex relationships. The document also covers topics like BSON, CRUD operations, indexing, map reduce, transactions, replication, and sharding in MongoDB.
MongoDB Replication fundamentals - Desert Code Camp - October 2014
MongoDB uses replication to provide high availability and redundancy. The document discusses MongoDB replication fundamentals including replica sets, oplogs, and reading from secondary nodes. It provides an overview of primary/secondary roles in replica sets, how writes are logged to oplogs, and how secondaries replicate by reading the primary's oplog. It also covers read preference settings and write concerns in MongoDB replication.
MongoDB Replication fundamentals - Desert Code Camp - October 2014
MongoDB uses replication to provide high availability and scalability. The primary node accepts all write operations and logs the changes to the oplog (operation log), which secondary nodes replicate from to stay in sync. The oplog contains entries for all insert, update, and delete operations with metadata like the timestamp and operation. It acts like a rolling queue to support replication as older entries are overwritten. Replication allows increased read capacity and redundancy for disaster recovery.
MongoDB Internals
MongoDB is a document-oriented NoSQL database that uses flexible schemas and provides high performance, high availability, and easy scalability. It uses either MMAP or WiredTiger storage engines and supports features like sharding, aggregation pipelines, geospatial indexing, and GridFS for large files. While MongoDB has better performance than Cassandra or Couchbase according to benchmarks, it has limitations such as a single-threaded aggregation and lack of joins across collections.
NoSQL and MongoDB
The document provides an agenda for a two-day training on NoSQL and MongoDB. Day 1 covers an introduction to NoSQL concepts like distributed and decentralized databases, CAP theorem, and different types of NoSQL databases including key-value, column-oriented, and document-oriented databases. It also covers functions and indexing in MongoDB. Day 2 focuses on specific MongoDB topics like aggregation framework, sharding, queries, schema-less design, and indexing.
Best Practices for NoSQL Workloads on Amazon EC2 and Amazon EBS - February 20...
Learn how to optimize your NoSQL database on AWS for cost, efficiency, and scale. NoSQL databases are great for modern datasets that require simplicity in design, handle structured and unstructured data, scale horizontally, and offer finer control over availability. With AWS, you have options for running NoSQL on Amazon EC2 with Amazon EBS or on Amazon DynamoDB. This webinar will dive deep into best practices and architectural considerations for designing and managing NoSQL databases like Cassandra, MongoDB, CouchDB, and Aerospike on EC2 and EBS. We will share best practices around instance and volume selection, provide performance tuning hints, and describe cost optimization techniques.
Learning Objectives:
• Learn about common NoSQL database options and use cases for Cassandra, MongoDB, CouchDB, and Aerospike
• Review best practices around architecting on AWS for different NoSQL databases
• Understand the cost vs. performance of different Amazon EC2 instances and Amazon EBS volumes
mongodb tutorial
mongodb 소개 및 basic tutorial 자료입니다.
- 데이터베이스의 진화
- nosql의 등장배경 및 역할
- mycq 개발 경험
- mongodb 기능 및 구조
2010 mongo berlin-scaling
Eliot Horowitz discusses various techniques for scaling MongoDB deployments, including optimization of schema design, indexes, hardware configuration, embedding documents, and different replication architectures like replica sets and sharding. The key techniques for scaling reads are to optimize schemas and indexes, use replica sets to distribute reads to slaves, and scale by adding more slaves. For writes, sharding allows scaling by partitioning data across multiple shard clusters.
MongoDB Knowledge Shareing
This document provides an overview of MongoDB including:
- MongoDB is an open-source document database that is schemaless and document-oriented.
- It has advantages like rich querying, horizontal scalability, high availability, and flexibility in schemas.
- The document includes information on MongoDB's data model, querying capabilities, indexing, availability through replication, and scaling through sharding.
- Case studies are presented showing how companies like Mailbox, Visual China, and Youku use MongoDB for applications processing large amounts of data.
The Care + Feeding of a Mongodb Cluster
This document summarizes best practices for scaling MongoDB deployments. It discusses Behance's use of MongoDB for their activity feed, including moving from 40 nodes with 250M documents on ext3 to 60 nodes with 400M documents on ext4. It covers topics like sharding, replica sets, indexing, maintenance, and hardware considerations for large MongoDB clusters.
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL is becoming the relational database of choice. An important factor in the rising popularity of Postgres is the extension APIs that allow developers to improve any database module’s behavior. As a result, Postgres users have access to hundreds of extensions today.
In this talk, we're going to first describe extension APIs. Then, we’re going to present four popular Postgres extensions, and demo their use.
* PostGIS turns Postgres into a spatial database through adding support for geographic objects.
* HLL & TopN add approximation algorithms to Postgres. These algorithms are used when real-time responses matter more than exact results.
* pg_partman makes managing partitions in Postgres easy. Through partitions, Postgres provide 5-10x higher performance for time-series data.
* Citus transforms Postgres into a distributed database. To do this, Citus shards data, performs distributed deadlock detection, and parallelizes queries.
Finally, we’ll conclude with why we think Postgres sets the way forward for relational databases.
PostgreSQL is becoming the relational database of choice. One important factor in the rising popularity of Postgres are the extension APIs. These APIs allow developers to extend any database sub-module’s behavior for higher performance, security, or new functionality. As a result, Postgres users have access to over a hundred extensions today, and more to come in the future.
In this talk, I’m going to first describe PostgreSQL’s extension APIs. These APIs are unique to Postgres, and have the potential to change the database landscape. Then, we’re going to present the four most popular Postgres extensions, show the use cases where they are applicable, and demo their usage.
PostGIS turns Postgres into a spatial database. It adds support for geographic objects, allowing location queries to be run in SQL.
HyperLogLog (HLL) & TopN add approximation algorithms to Postgres. These sketch algorithms are used in distributed systems when real-time responses to queries matter more than exact results.
pgpartman makes creating and managing partitions in Postgres easy. Through careful partition management with pgpartman, Postgres offers 5-10x higher write and query performance for time-series data.
Citus transforms Postgres into a distributed database. Citus transparently shards and replicates data, performs distributed deadlock detection, and parallelizes queries.
After demoing these popular extensions, we’ll conclude with why we think the monolithic relational database is dying and how Postgres sets a path for the future. We’ll end the talk with a Q&A.
Scaling with mongo db (with notes)
This document discusses MongoDB and scaling strategies when using MongoDB. It begins with an overview of MongoDB's architecture, data model, and operations. It then describes some early performance issues encountered with MongoDB including issues with durability settings, queries locking servers, and updates moving documents. The document recommends strategies for scaling such as adding more RAM, partitioning data through sharding, and monitoring replication delay closely for disaster recovery.
Hardware Provisioning
This document discusses hardware provisioning best practices for MongoDB. It covers key concepts like bottlenecks, working sets, and replication vs sharding. It also presents two case studies where these concepts were applied: 1) For a Spanish bank storing logs, the working set was 4TB so they provisioned servers with at least that much RAM. 2) For an online retailer storing products, testing found the working set was 270GB, so they recommended a replica set with 384GB RAM per server to avoid complexity of sharding. The key lessons are to understand requirements, test with a proof of concept, measure resource usage, and expect that applications may become bottlenecks over time.
MongoDB: An Introduction - July 2011
Introduction to MongoDB presented to the Dallas Big Data meetup http://www.dfwbigdata.org/events/17759701/
2011 mongo sf-scaling
This document summarizes Eliot Horowitz's presentation on practical scaling and sharding in MongoDB. The key points covered include:
1) Horizontal scaling is needed to scale beyond the limits of vertical scaling in the cloud. Replica sets allow scaling reads but not writes.
2) Sharding distributes write load across shards, keeps working data set in RAM, enables consistent reads, and allows capacity to increase with no downtime.
3) Sharding design goals are to scale linearly, increase capacity non-disruptively, be transparent to applications, and have low administration overhead.
Similar to Migrating from MySQL to MongoDB (20)
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
Cassandra Community Webinar: From Mongo to Cassandra, Architectural Lessons
JasperWorld 2012: Reinventing Data Management by Max Schireson
JasperWorld 2012: Reinventing Data Management by Max Schireson
MongoDB Replication fundamentals - Desert Code Camp - October 2014
MongoDB Replication fundamentals - Desert Code Camp - October 2014
MongoDB Replication fundamentals - Desert Code Camp - October 2014
MongoDB Replication fundamentals - Desert Code Camp - October 2014
Best Practices for NoSQL Workloads on Amazon EC2 and Amazon EBS - February 20...
Best Practices for NoSQL Workloads on Amazon EC2 and Amazon EBS - February 20...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
Recently uploaded
20240607 QFM018 Elixir Reading List May 2024
Everything I found interesting about the Elixir programming ecosystem in May 2024
Best 20 SEO Techniques To Improve Website Visibility In SERP
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Programming Foundation Models with DSPy - Meetup Slides
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Video Streaming: Then, Now, and in the Future
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Full-RAG: A modern architecture for hyper-personalization
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
TrustArc Webinar - 2024 Global Privacy Survey
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
UiPath Test Automation using UiPath Test Suite series, part 5
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Building Production Ready Search Pipelines with Spark and Milvus
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Uni Systems Copilot event_05062024_C.Vlachos.pdf
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Recently uploaded (20)
Best 20 SEO Techniques To Improve Website Visibility In SERP
Best 20 SEO Techniques To Improve Website Visibility In SERP
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Full-RAG: A modern architecture for hyper-personalization
Full-RAG: A modern architecture for hyper-personalization
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
UiPath Test Automation using UiPath Test Suite series, part 5
UiPath Test Automation using UiPath Test Suite series, part 5
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
GraphSummit Singapore | Enhancing Changi Airport Group's Passenger Experience...
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
Migrating from MySQL to MongoDB
- 1. Migrating from MySQL to MongoDB Based on a true story
- 3. Starting Point Requirements for replacement: • Store data • High performance o worldwide access o multiple physical locations o read heavy application • Simple to use o No DBA... • MI o real-time reporting without affecting production
- 4. • Availability and Partition Tolerance o from CAP o by default Consistency not Availability... • Document-Orientated o called from OOPHP • Simple querying using JSON o benefit over CouchDB which needs MapReduce • Simple multi-node setup • MongoLab.com o web interface for prototyping So why MongoDB?
- 5. Read Preference Defaults to primary (strongly consistent) Other options allow eventual consistency primaryPreferred secondary secondaryPreferred
- 6. New Architecture Single primary, multiple read members
- 7. What are the other members? One Delayed member provides a rolling backup by maintaining a dataset, delay is limited by the size of the oplog One Hidden member provides access for MI (and also used for dumping in this scenario)
- 8. Automatic Failover Primary is FUBAR, new primary is elected
- 9. Automatic Failover Had experience of this in production. It's magic. Remaining replica set members will vote and elect a new primary, which will start taking writes. (PHP extension issue) Replica set members that have been out of the cluster will recover automatically on rejoining o If you've gone past the oplog window then it's a manual process
- 10. Sharding, and why we didn't • Sharding divides the data set and distributes the data over multiple servers, or shards • Necessary when o data exceeds max for single instance (based on disk space) o the Working Set exceeds available RAM o single instance can’t deal with writes (would have to be a lot) • We had none of these and Sharding adds complexity
- 11. Sharding Architecture http://docs.mongodb.org/manual/core/sharded-cluster-architectures-production/
- 12. Sharding Availability Each Shard would need a Replica Set containing at least 3 members to achieve the same level of availability as our non-sharded architecture. In addition if one or two of the Config Servers are unavailable chunk migration and chunk splitting are suspended
- 13. DataBase Design • Schemaless means that the structure and the content of the “record” are held for each “row” o long key names add up o we used single chars and a lookup table • Collections o Can think of them as “buckets” rather than tables o Contain similar records by design but it’s not enforced
- 14. Management Information • Different conceptually in NoSQL databases due to the way they are queried, returning JSON • Traditional working through a query tool is less intuitive (although MongoVUE is quite good) • This lead us to replace our SQL script -> Excel -> Graph process (yuck) • MapReduce, powerful but a steep learning curve