MyRocks is an open source LSM based MySQL database, created by Facebook. This slides introduce MyRocks overview and how we deployed at Facebook, as of 2017.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark Summit
What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet.
At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it.
We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
In this talk, we'll walk through RocksDB technology and look into areas where MyRocks is a good fit by comparison to other engines such as InnoDB. We will go over internals, benchmarks, and tuning of MyRocks engine. We also aim to explore the benefits of using MyRocks within the MySQL ecosystem. Attendees will be able to conclude with the latest development of tools and integration within MySQL.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
The Parquet Format and Performance Optimization OpportunitiesDatabricks
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of FacebookThe Hive
This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...Spark Summit
What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet.
At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it.
We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).
Optimizing MariaDB for maximum performanceMariaDB plc
When it comes to optimizing the performance of a database, DBAs have to look at everything from the OS to the network. In this session, MariaDB Enterprise Architect Manjot Singh shares best practices for getting the most out of MariaDB. He highlights recommended OS settings, important configuration and tuning parameters, options for improving replication and clustering performance and features such as query result caching.
In this talk, we'll walk through RocksDB technology and look into areas where MyRocks is a good fit by comparison to other engines such as InnoDB. We will go over internals, benchmarks, and tuning of MyRocks engine. We also aim to explore the benefits of using MyRocks within the MySQL ecosystem. Attendees will be able to conclude with the latest development of tools and integration within MySQL.
Meta/Facebook's database serving social workloads is running on top of MyRocks (MySQL on RocksDB). This means our performance and reliability depends a lot on RocksDB. Not just MyRocks, but also we have other important systems running on top of RocksDB. We have learned many lessons from operating and debugging RocksDB at scale.
In this session, we will offer an overview of RocksDB, key differences from InnoDB, and share a few interesting lessons learned from production.
The Parquet Format and Performance Optimization OpportunitiesDatabricks
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.
This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
Cosco: An Efficient Facebook-Scale Shuffle ServiceDatabricks
Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroDatabricks
Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.
There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.
Migrating from InnoDB and HBase to MyRocks at FacebookMariaDB plc
Facebook created a new storage engine called MyRocks to optimize space and write performance, and recently migrated both UDB (a database for social activities, and our biggest in production) and Facebook Messenger to MyRocks. In this session, Yoshinori Matsunobu of Facebook talks about the challenges, benefits and lessons learned by migrating these applications from InnoDB to MyRocks.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
Performance Tuning RocksDB for Kafka Streams' State Stores (Dhruba Borthakur,...confluent
RocksDB is the default state store for Kafka Streams. In this talk, we will discuss how to improve single node performance of the state store by tuning RocksDB and how to efficiently identify issues in the setup. We start with a short description of the RocksDB architecture. We discuss how Kafka Streams restores the state stores from Kafka by leveraging RocksDB features for bulk loading of data. We give examples of hand-tuning the RocksDB state stores based on Kafka Streams metrics and RocksDB’s metrics. At the end, we dive into a few RocksDB command line utilities that allow you to debug your setup and dump data from a state store. We illustrate the usage of the utilities with a few real-life use cases. The key takeaway from the session is the ability to understand the internal details of the default state store in Kafka Streams so that engineers can fine-tune their performance for different varieties of workloads and operate the state stores in a more robust manner.
This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah
Apache Kafka is a new breed of messaging system built for the "big data" world. Coming out of LinkedIn (and donated to Apache), it is a distributed pub/sub system built in Scala. It has been an Apache TLP now for several months with the first Apache release imminent. Built for speed, scalability, and robustness, Kafka should definitely be one of the data tools you consider when designing distributed data-oriented applications.
The talk will cover a general overview of the project and technology, with some use cases, and a demo.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
Cosco: An Efficient Facebook-Scale Shuffle ServiceDatabricks
Cosco is an efficient shuffle-as-a-service that powers Spark (and Hive) jobs at Facebook warehouse scale. It is implemented as a scalable, reliable and maintainable distributed system. Cosco is based on the idea of partial in-memory aggregation across a shared pool of distributed memory. This provides vastly improved efficiency in disk usage compared to Spark's built-in shuffle. Long term, we believe the Cosco architecture will be key to efficiently supporting jobs at ever larger scale. In this talk we'll take a deep dive into the Cosco architecture and describe how it's deployed at Facebook. We will then describe how it's integrated to run shuffle for Spark, and contrast it with Spark's built-in sort-based shuffle mechanism and SOS (presented at Spark+AI Summit 2018).
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroDatabricks
Zstandard is a fast compression algorithm which you can use in Apache Spark in various way. In this talk, I briefly summarized the evolution history of Apache Spark in this area and four main use cases and the benefits and the next steps:
1) ZStandard can optimize Spark local disk IO by compressing shuffle files significantly. This is very useful in K8s environments. It’s beneficial not only when you use `emptyDir` with `memory` medium, but also it maximizes OS cache benefit when you use shared SSDs or container local storage. In Spark 3.2, SPARK-34390 takes advantage of ZStandard buffer pool feature and its performance gain is impressive, too.
2) Event log compression is another area to save your storage cost on the cloud storage like S3 and to improve the usability. SPARK-34503 officially switched the default event log compression codec from LZ4 to Zstandard.
3) Zstandard data file compression can give you more benefits when you use ORC/Parquet files as your input and output. Apache ORC 1.6 supports Zstandardalready and Apache Spark enables it via SPARK-33978. The upcoming Parquet 1.12 will support Zstandard compression.
4) Last, but not least, since Apache Spark 3.0, Zstandard is used to serialize/deserialize MapStatus data instead of Gzip.
There are more community works to utilize Zstandard to improve Spark. For example, Apache Avro community also supports Zstandard and SPARK-34479 aims to support Zstandard in Spark’s avro file format in Spark 3.2.0.
Migrating from InnoDB and HBase to MyRocks at FacebookMariaDB plc
Facebook created a new storage engine called MyRocks to optimize space and write performance, and recently migrated both UDB (a database for social activities, and our biggest in production) and Facebook Messenger to MyRocks. In this session, Yoshinori Matsunobu of Facebook talks about the challenges, benefits and lessons learned by migrating these applications from InnoDB to MyRocks.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
The Hive Think Tank: Rocking the Database World with RocksDBThe Hive
Dhruba Borthakur, Facebook
Dhruba Borthakur is an engineer at Facebook. He has been one of the founding engineer of RocksDB, an open-source key-value store optimized for storing data in flash and main-memory storage. He has been one of the founding architects of the Apache Hadoop Distributed File System and has been instrumental in scaling Facebook's Hadoop cluster to multiples of petabytes. Dhruba has contributed code to the Apache HBase project. Earlier, he contributed to the development of the Andrew File System (AFS). He has an M.S. in Computer Science from the University of Wisconsin, Madison and a B.S. in Computer Science BITS, Pilani, India.
Database as a Service on the Oracle Database Appliance PlatformMaris Elsins
Speaker: Marc Fielding, Co-speaker: Maris Elsins.
Oracle Database Appliance provides a robust, highly-available, cost-effective, and surprisingly scalable platform for database as a service environment. By leveraging Oracle Enterprise Manager's self-service features, databases can be provisioned on a self-service basis to a cluster of Oracle Database Appliance machines. Discover how multiple ODA devices can be managed together to provide both high availability and incremental, cost-effective scalability. Hear real-world lessons learned from successful database consolidation implementations.
Introduction to TokuDB v7.5 and Read Free ReplicationTim Callaghan
TokuDB v7.5 introduced Read Free Replication, allowing MySQL slaves to run with virtually no read IO. This presentation discusses how Fractal Tree indexes work, what they enable in TokuDB, and they allow TokuDB to uniquely offer this replication innovation.
Vote NO for MySQL - Election 2012: NoSQL. Researchers predict a dark future for MySQL. Significant market loss to come. Are things that bad, is MySQL falling behind? A look at NoSQL, an attempt to identify different kinds of NoSQL stores, their goals and how they compare to MySQL 5.6. Focus: Key Value Stores and Document Stores. MySQL versus NoSQL means looking behind the scenes, taking a step back and looking at the building blocks.
This is from a 2 hour talk introducing in-memory databases. First a look at traditional RDBMS architecture and some of it's limitations, then a look at some in-memory products and finally a closer look at OrigoDB, the open source in-memory database toolkit for NET/Mono.
The Apache CouchDB project is a NoSQL database that enables users to store unstructured data with high availability and partition tolerance.
As an example of a geo big data problem, this presentation will show how to process and store the results of a large image classification in geojson format into CouchDB and the potential problems that can arise of
* Database availability
* Multiple data types
* Sharding of large geospatial data across multiple database nodes
* Querying geospatial data efficiently using complex polygons
Cloudant supports the Apache CouchDB project and recently donated code to enable clustering within CouchDB. This presentation will also include details on advanced GIS on top of NoSQL.
Parse has one of the world's largest and most complicated MongoDB deployments, with millions of collections, tens of millions of indexes, and hundreds of thousands of different workload types. In this talk we'll cover our experiences evaluating all three major storage engines — mmapv1, WiredTiger, and RocksDB, sharing benchmarks and compression rates and and fun anecdotes from all three. Igor Canadi will do a deeper dive into RocksDB — what is RocksDB, what is LSM storage, and what are a few of the similarities and differences between RocksDB and WiredTiger. We will then talk about why we are pursuing RocksDB as our primary workload storage engine moving forward, and share some thoughts about the future of MongoDB storage engine innovation.
Making the case for write-optimized database algorithms / Mark Callaghan (Fac...Ontico
Write-optimized database algorithms have been available in NoSQL products for many years. With MyRocks, the RocksDB storage engine for MySQL, we are using a write-optimized algorithm for a SQL DBMS. This talk will explain why we created MyRocks and how to compare write-optimized algorithms with the ubiquitous B-Tree in terms of read, write and space efficiency. A large MySQL deployment at Facebook is in the process of migrating from InnoDB to MyRocks. With RocksDB storage engines for MySQL and MongoDB and the Vinyl storage engine for Tarantool we think it is likely that you can consider a write-optimized database engine in the next few years.
Slides for a talk.
Talk abstract:
In the dark of the night, if you listen carefully enough, you can hear databases cry. But why? As developers, we rarely consider what happens under the hood of widely used abstractions such as databases. As a consequence, we rarely think about the performance of databases. This is especially true to less widespread, but often very useful NoSQL databases.
In this talk we will take a close look at NoSQL database performance, peek under the hood of the most frequently used features to see how they affect performance and discuss performance issues and bottlenecks inherent to all databases.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Hivelance Technology
Cryptocurrency trading bots are computer programs designed to automate buying, selling, and managing cryptocurrency transactions. These bots utilize advanced algorithms and machine learning techniques to analyze market data, identify trading opportunities, and execute trades on behalf of their users. By automating the decision-making process, crypto trading bots can react to market changes faster than human traders
Hivelance, a leading provider of cryptocurrency trading bot development services, stands out as the premier choice for crypto traders and developers. Hivelance boasts a team of seasoned cryptocurrency experts and software engineers who deeply understand the crypto market and the latest trends in automated trading, Hivelance leverages the latest technologies and tools in the industry, including advanced AI and machine learning algorithms, to create highly efficient and adaptable crypto trading bots
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Why React Native as a Strategic Advantage for Startup Innovation.pdfayushiqss
Do you know that React Native is being increasingly adopted by startups as well as big companies in the mobile app development industry? Big names like Facebook, Instagram, and Pinterest have already integrated this robust open-source framework.
In fact, according to a report by Statista, the number of React Native developers has been steadily increasing over the years, reaching an estimated 1.9 million by the end of 2024. This means that the demand for this framework in the job market has been growing making it a valuable skill.
But what makes React Native so popular for mobile application development? It offers excellent cross-platform capabilities among other benefits. This way, with React Native, developers can write code once and run it on both iOS and Android devices thus saving time and resources leading to shorter development cycles hence faster time-to-market for your app.
Let’s take the example of a startup, which wanted to release their app on both iOS and Android at once. Through the use of React Native they managed to create an app and bring it into the market within a very short period. This helped them gain an advantage over their competitors because they had access to a large user base who were able to generate revenue quickly for them.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
Worried about document security while sharing them in Salesforce? Fret no more! Here are the top-notch security standards XfilesPro upholds to ensure strong security for your Salesforce documents while sharing with internal or external people.
To learn more, read the blog: https://www.xfilespro.com/how-does-xfilespro-make-document-sharing-secure-and-seamless-in-salesforce/
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
Your Digital Assistant.
Making complex approach simple. Straightforward process saves time. No more waiting to connect with people that matter to you. Safety first is not a cliché - Securely protect information in cloud storage to prevent any third party from accessing data.
Would you rather make your visitors feel burdened by making them wait? Or choose VizMan for a stress-free experience? VizMan is an automated visitor management system that works for any industries not limited to factories, societies, government institutes, and warehouses. A new age contactless way of logging information of visitors, employees, packages, and vehicles. VizMan is a digital logbook so it deters unnecessary use of paper or space since there is no requirement of bundles of registers that is left to collect dust in a corner of a room. Visitor’s essential details, helps in scheduling meetings for visitors and employees, and assists in supervising the attendance of the employees. With VizMan, visitors don’t need to wait for hours in long queues. VizMan handles visitors with the value they deserve because we know time is important to you.
Feasible Features
One Subscription, Four Modules – Admin, Employee, Receptionist, and Gatekeeper ensures confidentiality and prevents data from being manipulated
User Friendly – can be easily used on Android, iOS, and Web Interface
Multiple Accessibility – Log in through any device from any place at any time
One app for all industries – a Visitor Management System that works for any organisation.
Stress-free Sign-up
Visitor is registered and checked-in by the Receptionist
Host gets a notification, where they opt to Approve the meeting
Host notifies the Receptionist of the end of the meeting
Visitor is checked-out by the Receptionist
Host enters notes and remarks of the meeting
Customizable Components
Scheduling Meetings – Host can invite visitors for meetings and also approve, reject and reschedule meetings
Single/Bulk invites – Invitations can be sent individually to a visitor or collectively to many visitors
VIP Visitors – Additional security of data for VIP visitors to avoid misuse of information
Courier Management – Keeps a check on deliveries like commodities being delivered in and out of establishments
Alerts & Notifications – Get notified on SMS, email, and application
Parking Management – Manage availability of parking space
Individual log-in – Every user has their own log-in id
Visitor/Meeting Analytics – Evaluate notes and remarks of the meeting stored in the system
Visitor Management System is a secure and user friendly database manager that records, filters, tracks the visitors to your organization.
"Secure Your Premises with VizMan (VMS) – Get It Now"
Code reviews are vital for ensuring good code quality. They serve as one of our last lines of defense against bugs and subpar code reaching production.
Yet, they often turn into annoying tasks riddled with frustration, hostility, unclear feedback and lack of standards. How can we improve this crucial process?
In this session we will cover:
- The Art of Effective Code Reviews
- Streamlining the Review Process
- Elevating Reviews with Automated Tools
By the end of this presentation, you'll have the knowledge on how to organize and improve your code review proces
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
2. Who am I
▪ Was a MySQL consultant at MySQL (Sun, Oracle) for 4 years
▪ Joined Facebook in March 2012
▪ MySQL 5.1 -> 5.6 upgrade
▪ Fast master failover without losing data
▪ Partly joined HBase Production Engineering team in 2014.H1
▪ Started a research project to integrate RocksDB and MySQL from
2014.H2, with MySQL Engineering Team and MariaDB
▪ Started MyRocks production deployment since 2016.H2
3. Agenda
▪ MySQL at Facebook
▪ Issues in InnoDB
▪ RocksDB and MyRocks overview
▪ Production Deployment
4. “Main MySQL Database” at Facebook
▪ Storing Social Graph
▪ Massively Sharded
▪ Petabytes scale
▪ Low latency
▪ Automated Operations
▪ Pure Flash Storage (Constrained by space, not by CPU/IOPS)
5. H/W trends and limitations
▪ SSD/Flash is getting affordable, but MLC Flash is still a bit expensive
▪ HDD: Large enough capacity but very limited IOPS
▪ Reducing read/write IOPS is very important -- Reducing write is harder
▪ SSD/Flash: Great read iops but limited space and write endurance
▪ Reducing space is higher priority
8. InnoDB Issue (3) -- Compression
Uncompressed
16KB page
Row
Row
Row
Compressed
to 5KB
Row
Row
Row
Using 8KB space
on storage
Row
Row
Row
0~4KB => 4KB
4~8KB => 8KB
8~16KB => 16KB
9. RocksDB
▪ http://rocksdb.org/
▪ Forked from LevelDB
▪ Key-Value LSM (Log Structured Merge) persistent store
▪ Embedded
▪ Data stored locally
▪ Optimized for fast storage
▪ LevelDB was created by Google
▪ Facebook forked and developed RocksDB
▪ Used at many backend services at Facebook, and many external large services
▪ Needs to write C++ or Java code to access RocksDB
20. Reducing Space/Write Amplification
Append Only Prefix Key Encoding Zero-Filling metadata
WAL/
Memstore
Row
Row
Row
Row
Row
Row
sst
id1 id2 id3
100 200 1
100 200 2
100 200 3
100 200 4
id1 id2 id3
100 200 1
2
3
4
key value seq id flag
k1 v1 1234561 W
k2 v2 1234562 W
k3 v3 1234563 W
k4 v4 1234564 W
Seq id is 7 bytes in RocksDB. After compression,
“0” uses very little space
key value seq id flag
k1 v1 0 W
k2 v2 0 W
k3 v3 0 W
k4 v4 0 W
21. LSM Compaction Algorithm -- Level
▪ For each level, data is sorted by key
▪ Read Amplification: 1 ~ number of levels (depending on cache -- L0~L3 are usually cached)
▪ Write Amplification: 1 + 1 + fanout * (number of levels – 2) / 2
▪ Space Amplification: 1.11
▪ 11% is much smaller than B+Tree’s fragmentation
22. Read Penalty on LSM
MemTable
L0
L1
L2
L3
RocksDBInnoDB
SELECT id1, id2, time FROM t WHERE id1=100 AND id2=100 ORDER BY time DESC LIMIT 1000;
Index on (id1, id2, time)
Branch
Leaf
Range Scan with covering index is
done by just reading leaves sequentially,
and ordering is guaranteed (very efficient)
Merge is needed to do range scan with
ORDER BY
(L0-L2 are usually cached, but in total it needs
more CPU cycles than InnoDB)
24. Column Family
Query atomicity across different key spaces.
▪ Column families:
▪ Separate MemTables and SST files
▪ Share transactional logs
WAL
(shared)
CF1
Active MemTable
CF2
Active MemTable
CF1
Active MemTable
CF1
Read Only
MemTable(s)
CF1
Active MemTable
CF2
Read Only
MemTable(s)
File File File
CF1 Persistent Files
File File File
CF2 Persistent Files
CF1
Compaction
CF2
Compaction
25. What is MyRocks
▪ MySQL on top of RocksDB (RocksDB storage engine)
▪ Taking both LSM advantages and MySQL features
▪ LSM advantage: Smaller space and lower write amplification
▪ MySQL features: SQL, Replication, Connectors and many tools
▪ Open Source, distributed from MariaDB and Percona as well
MySQL Clients
InnoDB RocksDB
Parser
Optimizer
Replication
etc
SQL/Connector
MySQL
http://myrocks.io/
26. MyRocks features
▪ Clustered Index (same as InnoDB)
▪ Transactions, including consistency between binlog and RocksDB
▪ Faster data loading, deletes and replication
▪ Dynamic Options
▪ TTL
▪ Crash Safety
▪ Online logical and binary backup
27. Performance (LinkBench)
▪ Space Usage
▪ QPS
▪ Flash writes per query
▪ HDD
▪ http://smalldatum.blogspot.com/2016/01/myrocks-vs-innodb-with-linkbench-over-7.html
32. Getting Started
▪ Downloading MySQL with MyRocks support (MariaDB, Percona Server)
▪ Configuring my.cnf
▪ Installing MySQL and initializing data directory
▪ Starting mysqld
▪ Creating and manipulating some tables
▪ Shutting down mysqld
▪ https://github.com/facebook/mysql-5.6/wiki/Getting-Started-with-MyRocks
33. my.cnf (minimal configuration)
▪ It’s generally not recommended mixing multiple transactional storage engines within the same
instance
▪ Not transactional across engines, Not tested enough
▪ Add “allow-multiple-engines” in my.cnf, if you really want to mix InnoDB and MyRocks
[mysqld]
rocksdb
default-storage-engine=rocksdb
skip-innodb
default-tmp-storage-engine=MyISAM
collation-server=latin1_bin (or utf8_bin, binary)
log-bin
binlog-format=ROW
34. Creating tables
▪ “ENGINE=RocksDB” is a syntax to create RocksDB tables
▪ Setting “default-storage-engine=RocksDB” in my.cnf is fine too
▪ It is generally recommended to have a PRIMARY KEY, though MyRocks allows tables without
PRIMARY KEYs
▪ Tables are automatically compressed, without any configuration in DDL. Compression
algorithm can be configured via my.cnf
CREATE TABLE t (
id INT PRIMARY KEY,
value1 INT,
value2 VARCHAR (100),
INDEX (value1)
) ENGINE=RocksDB COLLATE latin1_bin;
36. MyRocks Goals
InnoDB in main database
90%
SpaceIOCPU
Machine limit
15%20%
MyRocks in main database
45%
SpaceIOCPU
Machine limit
12%18%
18%
12%
45%
37. MyRocks goals (more details)
▪ Smaller space usage
▪ 50% compared to compressed InnoDB at FB
▪ Better write amplification
▪ Can use more affordable flash storage
▪ Fast, and small enough CPU usage with general purpose workloads
▪ Large enough data that don’t fit in RAM
▪ Point lookup, range scan, full index/table scan
▪ Insert, update, delete by point/range, and occasional bulk inserts
▪ Same or even smaller CPU usage compared to InnoDB at FB
▪ Make it possible to consolidate 2x more instances per machine
38. Facebook MySQL related teams
▪ Production Engineering (SRE/PE)
▪ MySQL Production Engineering
▪ Data Performance
▪ Software Engineering (SWE)
▪ RocksDB Engineering
▪ MySQL Engineering
▪ MyRocks
▪ Others (Replication, Client, InnoDB, etc)
39. Relationship between MySQL PE and SWE
▪ SWE does MySQL server upgrade
▪ Upgrading MySQL revision with several FB patches
▪ Hot fixes
▪ SWE participates in PE oncall
▪ Investigating issues that might have been caused by MySQL server codebase
▪ Core files analysis
▪ Can submit diffs each other
▪ “Hackaweek” to work on shorter term tasks for a week
40. MyRocks migration -- Technical Challenges
▪ Initial Migration
▪ Creating MyRocks instances without downtime
▪ Loading into MyRocks tables within reasonable time
▪ Verifying data consistency between InnoDB and MyRocks
▪ Continuous Monitoring
▪ Resource Usage like space, iops, cpu and memory
▪ Query plan outliers
▪ Stalls and crashes
41. MyRocks migration -- Technical Challenges (2)
▪ When running MyRocks on master
▪ RBR (Row based binary logging)
▪ Removing queries relying on InnoDB Gap Lock
▪ Robust XA support (binlog and RocksDB)
42. Creating first MyRocks instance without downtime
▪ Picking one of the InnoDB slave instances, then starting logical dump
and restore
▪ Stopping one slave does not affect services
Master (InnoDB)
Slave1 (InnoDB) Slave2 (InnoDB) Slave3 (InnoDB) Slave4 (MyRocks)
Stop & Dump & Load
43. Faster Data Loading
Normal Write Path in MyRocks/RocksDB
Write Requests
MemTableWAL
Level 0 SST
Level 1 SST
Level max SST
….
Flush
Compaction
Compaction
Faster Write Path
Write Requests
Level max SST
“SET SESSION rocksdb_bulk_load=1;”
Original data must be sorted by primary key
44. Data migration steps
▪ Dst) Create table … ENGINE=ROCKSDB; (creating MyRocks tables with proper column families)
▪ Dst) ALTER TABLE DROP INDEX; (dropping secondary keys)
▪ Src) STOP SLAVE;
▪ mysqldump –host=innodb-host --order-by-primary | mysql –host=myrocks-host –init-
command=“set sql_log_bin=0; set rocksdb_bulk_load=1”
▪ Dst) ALTER TABLE ADD INDEX; (adding secondary keys)
▪ Src, Dst) START SLAVE;
45. Data Verification
▪ MyRocks/RocksDB is relatively new database technology
▪ Might have more bugs than robust InnoDB
▪ Ensuring data consistency helps avoid showing conflicting
results
46. Verification tests
▪ Index count check between primary key and secondary keys
▪ If any index is broken, it can be detected
▪ SELECT ‘PRIMARY’, COUNT(*) FROM t FORCE INDEX (PRIMARY)
UNION SELECT ‘idx1’, COUNT(*) FROM t FORCE INDEX (idx1)
▪ Can’t be used if there is no secondary key
▪ Index stats check
▪ Checking if “rows” show SHOW TABLE STATUS is not far different from actual row count
▪ Checksum tests w/ InnoDB
▪ Comparing between InnoDB instance and MyRocks instance
▪ Creating a transaction consistent snapshot at the same GTID position, scan, then compare
checksum
▪ Shadow correctness check
▪ Capturing read traffics
47. Shadow traffic tests
▪ We have a shadow test framework
▪ MySQL audit plugin to capture read/write queries from production instances
▪ Replaying them into shadow master instances
▪ Shadow master tests
▪ Client errors
▪ Rewriting queries relying on Gap Lock
▪ gap_lock_raise_error=1, gap_lock_write_log=1
49. Promoting MyRocks as a master
Master (MyRocks)
Slave1 (InnoDB) Slave2 (InnoDB) Slave3 (InnoDB) Slave4 (MyRocks)
50. Crash Safety
▪ Crash Safety makes operations much easier
▪ Just restarting failed instances can restart replication
▪ No need to rebuild entire instances, as long as data is there
▪ Crash Safe Slave
▪ Crash Safe Master
▪ 2PC (binlog and WAL)
51. Master crash safety settings
sync-binlog rocksdb-flush-log-at-
trx-commit
rocksdb-enable-
2pc
rocksdb-wal-
recovery-mode
No data loss on
unplanned machine
reboot
1 (default) 1 (default) 1 (default) 1 (default)
No data loss on
mysqld crash &
recovery
0 2 1 2
No data loss if
always failover
0 2 0 2
52. Notable issues fixed during migration
▪ Lots of “Snapshot Conflict” errors
▪ Because of implementation differences in MyRocks (PostgreSQL Style snapshot isolation)
▪ Setting tx-isolation=READ-COMMITTED solved the problem
▪ Slaves stopped with I/O errors on reads
▪ We switched to make MyRocks abort on I/O errors for both reads and writes
▪ Index statistics bugs
▪ Inaccurate row counts/length on bulk loading -> fixed
▪ Cardinality was not updated on fast index creation -> fixed
▪ Crash safety bugs and some crash bugs in MyRocks/RocksDB
53. Preventing stalls
▪ Heavy writes cause lots of compactions, which may cause stalls
▪ Typical write stall cases
▪ Online schema change
▪ Massive data migration jobs that write lots of data
▪ Use fast data loading whenever possible
▪ InnoDB to MyRocks migration can utilize this technique
▪ Can be harder for data migration jobs that write into existing tables
54. When write stalls happen
▪ Estimated number of pending compaction bytes exceeded X bytes
▪ soft|hard_pending_compaction_bytes, default 64GB
▪ Number of L0 files exceeded level0_slowdown|stop_writes_trigger (default 10)
▪ Number of unflushed number of MemTables exceeded
max_write_buffer_number (default 4)
▪ All of these incidents are written to LOG as WARN level
▪ All of these options apply to each column family
55. What happens on write stalls
▪ Soft stalls
▪ COMMIT takes longer time than usual
▪ Total estimated written bytes to MemTable is capped to
rocksdb_delayed_write_rate, until slowdown conditions are cleared
▪ Default is 16MB/s (previously 2MB/s)
▪ Hard stalls
▪ All writes are blocked at COMMIT, until stop conditions are cleared
56. Mitigating Write Stalls
▪ Speed up compactions
▪ Use faster compression algorithm (LZ4 for higher levels, ZSTD in the bottommost)
▪ Increase rocksdb_max_background_compactions
▪ Reduce total bytes written
▪ But avoid using too strong compression algorithm on upper levels
▪ Use more write efficient compaction algorithm
▪ compaction_pri=kMinOverlappingRatio
▪ Delete files slowly on Flash
▪ Deleting too many large files cause TRIM stalls on Flash
▪ MyRocks has an option to throttle sst file deletion speed
▪ Binlog file deletions should be slowed down as well
57. Monitoring
▪ MyRocks files
▪ SHOW ENGINE ROCKSDB STATUS
▪ SHOW ENGINE ROCKSDB TRANSACTION STATUS
▪ LOG files
▪ information_schema tables
▪ sst_dump tool
▪ ldb tool
59. SHOW GLOBAL STATUS
mysql> show global status like 'rocksdb%';
+---------------------------------------+-------------+
| Variable_name | Value |
+---------------------------------------+-------------+
| rocksdb_rows_deleted | 216223 |
| rocksdb_rows_inserted | 1318158 |
| rocksdb_rows_read | 7102838 |
| rocksdb_rows_updated | 1997116 |
....
| rocksdb_bloom_filter_prefix_checked | 773124 |
| rocksdb_bloom_filter_prefix_useful | 308445 |
| rocksdb_bloom_filter_useful | 10108448 |
....
60. Kernel VM allocation stalls
▪ RocksDB has limited support for O_DIRECT
▪ Too many buffered writes may cause “VM allocation stalls” on
older Linux kernels
▪ Some queries may take more than 1s because of this
▪ Using more %sys
▪ Linux Kernel 4.6 is much better for handling buffered writes
▪ For details: https://lwn.net/Articles/704739/
62. Summary
▪ We started migrating from InnoDB to MyRocks in our main database
▪ We run both master and slaves in production
▪ Major motivation was to save space
▪ Online data correctness check tool helped to find lots of data integrity
bugs and prevented from deploying inconsistent instances in production
▪ Bulk loading greatly reduced compaction stalls
▪ Transactions and crash safety are supported
63. (c) 2009 Facebook, Inc. or its licensors. "Facebook" is a registered trademark of Facebook, Inc.. All rights reserved. 1.0