Immutable Data Structures
•Download as PPTX, PDF•
0 likes•265 views
宇
宇 傅The document discusses various immutable data structures including stacks, trees, vector tries, hash array mapped tries (HAMTs), log-structured merge trees (LSM-Trees), and their applications in programming and databases like Git. It provides code examples of implementing immutable data structures in Java to demonstrate how they can be used to make code thread-safe and avoid issues like null pointer exceptions.
Report
Share
Report
Share
Recommended
Update on Raptor - understanding usage information for e-resources - Dr Rhys ...
Raptor is a free to use, open source software suite generally designed to enable accounting within event-based systems, and specifically aimed at accounting within systems that handle authentication events such as the Shibboleth IdP, OpenAthens LA, and EZproxy. It has been developed by Cardiff University with support from JISC. This session will give an update on Raptor, including talking about the UK federation Raptor aggregator for federation-wide statistics.
The Cassandra Distributed Database
This document summarizes Cassandra, an open source distributed database management system designed to handle large amounts of data across many commodity servers. It discusses Cassandra's history, key features like tunable consistency levels and support for structured and indexed columns. Case studies describe how companies like Digg, Twitter, Facebook and Mahalo use Cassandra to handle terabytes of data and high transaction volumes. The roadmap outlines upcoming releases that will improve features like compaction, management tools, and support for dynamic schema changes.
Redis: An introduction
This document provides an overview of Redis, including:
- Redis is an in-memory key-value store that also supports data types like lists, sets, sorted sets, and hashes. It provides persistence and is open source.
- Redis has features like master-slave replication, persistence, high performance, support for many languages, transactions, pub/sub functionality, and sharding.
- It supports common data structures like lists, sets, sorted sets, and hashes that can be used to solve problems like leaderboards, caching of frequently accessed data, and counting metrics.
Outside The Box With Apache Cassnadra
Cassandra presentation given at the 3rd annual Palmetto Open Source Software Conference (POSSCON 2010).
Ruby and Distributed Storage Systems
This document discusses using Ruby for distributed storage systems. It describes components like Bigdam, which is Treasure Data's new data ingestion pipeline. Bigdam uses microservices and a distributed key-value store called Bigdam-pool to buffer data. The document discusses designing and testing Bigdam using mocking, interfaces, and integration tests in Ruby. It also explores porting Bigdam-pool from Java to Ruby and investigating Ruby's suitability for tasks like asynchronous I/O, threading, and serialization/deserialization.
PostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw twenty random slides and explain you why such practices are bad and how to avoid running into them.
Intro to Big Data and NoSQL
This document provides an introduction to big data and NoSQL databases. It begins with an introduction of the presenter. It then discusses how the era of big data came to be due to limitations of traditional relational databases and scaling approaches. The document introduces different NoSQL data models including document, key-value, graph and column-oriented databases. It provides examples of NoSQL databases that use each data model. The document discusses how NoSQL databases are better suited than relational databases for big data problems and provides a real-world example of Twitter's use of FlockDB. It concludes by discussing approaches for working with big data using MapReduce and provides examples of using MongoDB and Azure for big data.
SQL Server 2014 Memory Optimised Tables - Advanced
Hekaton is large piece of kit, this session will focus on the internals of how in-memory tables and native stored procedures work and interact – Database structure: use of File Stream, backup/restore considerations in HA and DR as well as Database Durability, in-memory table make up: hash and range indexes, row chains, Multi-Version Concurrency Control (MVCC). Design considerations and gottcha’s to watch out for.
The session will be demo led.
Note: the session will assume the basics of Hekaton are known, so it is recommended you attend the Basics session.
Recommended
Update on Raptor - understanding usage information for e-resources - Dr Rhys ...
Raptor is a free to use, open source software suite generally designed to enable accounting within event-based systems, and specifically aimed at accounting within systems that handle authentication events such as the Shibboleth IdP, OpenAthens LA, and EZproxy. It has been developed by Cardiff University with support from JISC. This session will give an update on Raptor, including talking about the UK federation Raptor aggregator for federation-wide statistics.
The Cassandra Distributed Database
This document summarizes Cassandra, an open source distributed database management system designed to handle large amounts of data across many commodity servers. It discusses Cassandra's history, key features like tunable consistency levels and support for structured and indexed columns. Case studies describe how companies like Digg, Twitter, Facebook and Mahalo use Cassandra to handle terabytes of data and high transaction volumes. The roadmap outlines upcoming releases that will improve features like compaction, management tools, and support for dynamic schema changes.
Redis: An introduction
This document provides an overview of Redis, including:
- Redis is an in-memory key-value store that also supports data types like lists, sets, sorted sets, and hashes. It provides persistence and is open source.
- Redis has features like master-slave replication, persistence, high performance, support for many languages, transactions, pub/sub functionality, and sharding.
- It supports common data structures like lists, sets, sorted sets, and hashes that can be used to solve problems like leaderboards, caching of frequently accessed data, and counting metrics.
Outside The Box With Apache Cassnadra
Cassandra presentation given at the 3rd annual Palmetto Open Source Software Conference (POSSCON 2010).
Ruby and Distributed Storage Systems
This document discusses using Ruby for distributed storage systems. It describes components like Bigdam, which is Treasure Data's new data ingestion pipeline. Bigdam uses microservices and a distributed key-value store called Bigdam-pool to buffer data. The document discusses designing and testing Bigdam using mocking, interfaces, and integration tests in Ruby. It also explores porting Bigdam-pool from Java to Ruby and investigating Ruby's suitability for tasks like asynchronous I/O, threading, and serialization/deserialization.
PostgreSQL worst practices, version PGConf.US 2017 by Ilya Kosmodemiansky
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw twenty random slides and explain you why such practices are bad and how to avoid running into them.
Intro to Big Data and NoSQL
This document provides an introduction to big data and NoSQL databases. It begins with an introduction of the presenter. It then discusses how the era of big data came to be due to limitations of traditional relational databases and scaling approaches. The document introduces different NoSQL data models including document, key-value, graph and column-oriented databases. It provides examples of NoSQL databases that use each data model. The document discusses how NoSQL databases are better suited than relational databases for big data problems and provides a real-world example of Twitter's use of FlockDB. It concludes by discussing approaches for working with big data using MapReduce and provides examples of using MongoDB and Azure for big data.
SQL Server 2014 Memory Optimised Tables - Advanced
Hekaton is large piece of kit, this session will focus on the internals of how in-memory tables and native stored procedures work and interact – Database structure: use of File Stream, backup/restore considerations in HA and DR as well as Database Durability, in-memory table make up: hash and range indexes, row chains, Multi-Version Concurrency Control (MVCC). Design considerations and gottcha’s to watch out for.
The session will be demo led.
Note: the session will assume the basics of Hekaton are known, so it is recommended you attend the Basics session.
Alluxio - Scalable Filesystem Metadata Services
This talk was presented by Alluxio's top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup.
This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.
Big Data (NJ SQL Server User Group)
This document provides a summary of a presentation on Big Data and NoSQL databases. It introduces the presenters, Melissa Demsak and Don Demsak, and their backgrounds. It then discusses how data storage needs have changed with the rise of Big Data, including the problems created by large volumes of data. The presentation contrasts traditional relational database implementations with NoSQL data stores, identifying five categories of NoSQL data models: document, key-value, graph, and column family. It provides examples of databases that fall under each category. The presentation concludes with a comparison of real-world scenarios and which data storage solutions might be best suited to each scenario.
HBase in Practice
HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
HBase in Practice
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Why databases cry at night
Slides for a talk.
Talk abstract:
In the dark of the night, if you listen carefully enough, you can hear databases cry. But why? As developers, we rarely consider what happens under the hood of widely used abstractions such as databases. As a consequence, we rarely think about the performance of databases. This is especially true to less widespread, but often very useful NoSQL databases.
In this talk we will take a close look at NoSQL database performance, peek under the hood of the most frequently used features to see how they affect performance and discuss performance issues and bottlenecks inherent to all databases.
Concurrency and Multithreading Demistified - Reversim Summit 2014
Life as a software engineer is so exciting! Computing power continue to rise exponentially, software demands continue to rise exponentially as well, so far so good. The bad news are that in the last decade the computing power of single threaded application remains almost flat.
If you decide to continue ignoring concurrency and multi-threading the gap between the problems you are able to solve and your hardware capabilities will continue to rise. In this session we will discuss different approaches for taming the concurrency beast, such as shared mutability,shared immutability and isolated mutability actors, STM, etc we will discuss the shortcomings and the dangers of each approach and we will compare different programming languages and how they choose to tackle/ignore concurrency.
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
My keynote presentation about how we developed FiloDB, a distributed, Prometheus-compatible time series database, productionized it at Apple and scaled it out to handle a huge amount of operational data, based on the stack of Kafka, Cassandra, Scala/Akka.
What's new with Apache Tika?
A presentation from ApacheCon Europe 2015 / Apache Big Data Europe 2015
Apache Tika detects and extracts metadata and text from a huge range of file formats and types. From Search to Big Data, single file to internet scale, if you've got files, Tika can help you get out useful information!
Apache Tika has been around for nearly 10 years now, and in that time, a lot has changed. Not only has the number of formats supported gone up and up, but the ways of using Tika have expanded, and some of the philosophies on the best way to handle things have altered with experience. Tika has gained support for a wide range of programming languages to, and more recently, Big-Data scale support, and ways to automatically compare effects of changes to the library.
Whether you're an old-hand with Tika looking to know what's hot or different, or someone new looking to learn more about the power of Tika, this talk will have something in it for you!
DataEngConf: Parquet at Datadog: Fast, Efficient, Portable Storage for Big Data
By Doug Daniels (Director of Engineering, Data Dog)
At Datadog, we collect hundreds of billions of metric data points per day from hosts, services, and customers all over the world. In addition charting and monitoring this data in real time, we also run many large-scale offline jobs to apply algorithms and compute aggregations on the data. In the past months, we’ve migrated our largest data sets over to Apache Parquet—an efficient, portable columnar storage format
Large Scale Graph Analytics with JanusGraph
The past few years have seen an enormous growth in the popularity of graph databases, but what exactly is a graph database and how can I use one to gain deeper insights from my data?
In this session we will introduce JanusGraph, a highly scalable, transactional graph database with flexible backend storage options such as Apache HBase, Apache Cassandra, and Oracle Berkeley DB. We will begin with a brief introduction to graph databases and data models, common use cases, and the benefits of a relationship centric approach to analytics. We will follow with a more technical dive into the features and deployment options of JanusGraph, including accessing the graph with the Apache Tinkerpop API stack, manipulating it with the Blueprints API, and querying the graph with the Gremlin query language. Finally, we will look at how JanusGraph integrates with other technologies like Apache Spark as part of an overall analytics architecture.
Large Scale Graph Analytics with JanusGraph
This document provides an overview of large scale graph analytics and JanusGraph. It discusses graph databases and their use cases. JanusGraph is presented as an open source graph database that can scale to billions of vertices and edges across multiple storage backends like HBase, Cassandra and Bigtable. It uses the TinkerPop framework and Gremlin query language. JanusGraph supports ACID transactions, external indices, and evolving schemas. Example graph queries are demonstrated using the Gremlin console.
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
Microservices can provide terabytes of data in microseconds by mapping data from SQL databases into in-memory key-value stores and column key stores within JVMs. This is done through periodic synchronization of changed data from databases into memory and mapping the in-memory data into fast access structures. The in-memory data is then exposed through Java Stream and REST APIs to microservices for high performance querying and analysis of large datasets. This architecture allows microservices to quickly share access to large datasets and restart rapidly by reloading from the synchronized persistent stores.
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
By leveraging memory-mapped files, Speedment and the Chronicle Engine supports large Java maps that easily can exceed the size of your server’s RAM.Because the Java maps are mapped onto files, these maps can be shared instantly between several microservice JVMs and new microservice instances can be added, removed, or restarted very quickly. Data can be retrieved with predictable ultralow latency for a wide range of operations. The solution can be synchronized with an underlying database so that your in-memory maps will be consistently “alive.” The mapped files can be tens of terabytes, which has been done in real-world deployment cases, and a large number of micro services can share these maps simultaneously. Learn more in this session.
Introduction to libre « fulltext » technology
The presentation will be based on my personal experience on SQLite, MySQL and Zend Search ; on workshops I’ve attended (PostgreSQL) and on tests conducted under my supervision (PostgreSQL, MySQL, Sphinx, Lucene, Xapian). It will cover an exhaustive overview of existing techniques, from the most basic to the more advanced, and will lead to a comparative table of the existing technology.
Scalable and High available Distributed File System Metadata Service Using gR...
Alluxio Community Office Hour
Apr 7, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speaker: Bin Fan
Alluxio (alluxio.io) is an open-source data orchestration system that provides a single namespace federating multiple external distributed storage systems. It is critical for Alluxio to be able to store and serve the metadata of all files and directories from all mounted external storage both at scale and at speed.
This talk shares our design, implementation, and optimization of Alluxio metadata service (master node) to address the scalability challenges. Particularly, we will focus on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc. As a result of the combined above techniques, Alluxio 2.0 is able to store at least 1 billion files with a significantly reduced memory requirement, serving 3000 workers and 30000 clients concurrently.
In this Office Hour, we will go over how to:
- Metadata storage challenges
- How to combine different open source technologies as building blocks
- The design, implementation, and optimization of Alluxio metadata service
Rust All Hands Winter 2011
Rust is a new systems programming language designed for safety, concurrency, and speed. It uses a borrow checker to ensure memory safety and avoid use-after-free bugs, and enables safe concurrency through message passing and ownership rules. The Rust team has been redesigning and implementing the language over the past year based on lessons learned from writing the compiler in Rust. Their goals are to have a memory-safe, concurrent language that achieves performance comparable to C++. They plan to use Rust for the parallel engine of the Servo browser project.
Search at Twitter: Presented by Michael Busch, Twitter
Twitter processes over 500 million tweets per day and more than 2 billion search queries per day. The company uses a search architecture based on Lucene with custom extensions. This includes an in-memory real-time index optimized for concurrency without locks, and a schema-based document factory. Future work includes support for parallel index segments and additional Lucene features.
Building Big Data Streaming Architectures
In this talk we depict how to build streaming architectures connecting to Kafka with different technologies. Includes links to Github code samples.
UNIT I Introduction to NoSQL.pptx
NoSQL databases provide an alternative to traditional relational databases that is well-suited for large datasets, high scalability needs, and flexible, changing schemas. NoSQL databases sacrifice strict consistency for greater scalability and availability. The document model is well-suited for semi-structured data and allows for embedding related data within documents. Key-value stores provide simple lookup of data by key but do not support complex queries. Graph databases effectively represent network-like connections between data elements.
Hbase schema design and sizing apache-con europe - nov 2012
The document provides an overview of HBase schema design and cluster sizing notes. It discusses HBase architecture including tables, regions, distribution, and compactions. It emphasizes the importance of schema design, including using intelligent keys, denormalization, and duplication to overcome limitations. The document also covers techniques like salting keys, hashing vs sequential keys, and examples of schema design for applications like mail inbox and Facebook insights. It stresses designing for the use case and avoiding hotspotting when sizing clusters.
More Related Content
Similar to Immutable Data Structures
Alluxio - Scalable Filesystem Metadata Services
This talk was presented by Alluxio's top contributor and PMC Maintainer Calvin Jia at the Alluxio bay area Meetup.
This talk shares our design, implementation and optimization of Alluxio metadata service to address the scalability challenges, focusing on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc.
Big Data (NJ SQL Server User Group)
This document provides a summary of a presentation on Big Data and NoSQL databases. It introduces the presenters, Melissa Demsak and Don Demsak, and their backgrounds. It then discusses how data storage needs have changed with the rise of Big Data, including the problems created by large volumes of data. The presentation contrasts traditional relational database implementations with NoSQL data stores, identifying five categories of NoSQL data models: document, key-value, graph, and column family. It provides examples of databases that fall under each category. The presentation concludes with a comparison of real-world scenarios and which data storage solutions might be best suited to each scenario.
HBase in Practice
HBase is a distributed, column-oriented database that stores data in tables divided into rows and columns. It is optimized for random, real-time read/write access to big data. The document discusses HBase's key concepts like tables, regions, and column families. It also covers performance tuning aspects like cluster configuration, compaction strategies, and intelligent key design to spread load evenly. Different use cases are suitable for HBase depending on access patterns, such as time series data, messages, or serving random lookups and short scans from large datasets. Proper data modeling and tuning are necessary to maximize HBase's performance.
HBase in Practice
From: DataWorks Summit 2017 - Munich - 20170406
HBase hast established itself as the backend for many operational and interactive use-cases, powering well-known services that support millions of users and thousands of concurrent requests. In terms of features HBase has come a long way, overing advanced options such as multi-level caching on- and off-heap, pluggable request handling, fast recovery options such as region replicas, table snapshots for data governance, tuneable write-ahead logging and so on. This talk is based on the research for the an upcoming second release of the speakers HBase book, correlated with the practical experience in medium to large HBase projects around the world. You will learn how to plan for HBase, starting with the selection of the matching use-cases, to determining the number of servers needed, leading into performance tuning options. There is no reason to be afraid of using HBase, but knowing its basic premises and technical choices will make using it much more successful. You will also learn about many of the new features of HBase up to version 1.3, and where they are applicable.
Why databases cry at night
Slides for a talk.
Talk abstract:
In the dark of the night, if you listen carefully enough, you can hear databases cry. But why? As developers, we rarely consider what happens under the hood of widely used abstractions such as databases. As a consequence, we rarely think about the performance of databases. This is especially true to less widespread, but often very useful NoSQL databases.
In this talk we will take a close look at NoSQL database performance, peek under the hood of the most frequently used features to see how they affect performance and discuss performance issues and bottlenecks inherent to all databases.
Concurrency and Multithreading Demistified - Reversim Summit 2014
Life as a software engineer is so exciting! Computing power continue to rise exponentially, software demands continue to rise exponentially as well, so far so good. The bad news are that in the last decade the computing power of single threaded application remains almost flat.
If you decide to continue ignoring concurrency and multi-threading the gap between the problems you are able to solve and your hardware capabilities will continue to rise. In this session we will discuss different approaches for taming the concurrency beast, such as shared mutability,shared immutability and isolated mutability actors, STM, etc we will discuss the shortcomings and the dangers of each approach and we will compare different programming languages and how they choose to tackle/ignore concurrency.
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
My keynote presentation about how we developed FiloDB, a distributed, Prometheus-compatible time series database, productionized it at Apple and scaled it out to handle a huge amount of operational data, based on the stack of Kafka, Cassandra, Scala/Akka.
What's new with Apache Tika?
A presentation from ApacheCon Europe 2015 / Apache Big Data Europe 2015
Apache Tika detects and extracts metadata and text from a huge range of file formats and types. From Search to Big Data, single file to internet scale, if you've got files, Tika can help you get out useful information!
Apache Tika has been around for nearly 10 years now, and in that time, a lot has changed. Not only has the number of formats supported gone up and up, but the ways of using Tika have expanded, and some of the philosophies on the best way to handle things have altered with experience. Tika has gained support for a wide range of programming languages to, and more recently, Big-Data scale support, and ways to automatically compare effects of changes to the library.
Whether you're an old-hand with Tika looking to know what's hot or different, or someone new looking to learn more about the power of Tika, this talk will have something in it for you!
DataEngConf: Parquet at Datadog: Fast, Efficient, Portable Storage for Big Data
By Doug Daniels (Director of Engineering, Data Dog)
At Datadog, we collect hundreds of billions of metric data points per day from hosts, services, and customers all over the world. In addition charting and monitoring this data in real time, we also run many large-scale offline jobs to apply algorithms and compute aggregations on the data. In the past months, we’ve migrated our largest data sets over to Apache Parquet—an efficient, portable columnar storage format
Large Scale Graph Analytics with JanusGraph
The past few years have seen an enormous growth in the popularity of graph databases, but what exactly is a graph database and how can I use one to gain deeper insights from my data?
In this session we will introduce JanusGraph, a highly scalable, transactional graph database with flexible backend storage options such as Apache HBase, Apache Cassandra, and Oracle Berkeley DB. We will begin with a brief introduction to graph databases and data models, common use cases, and the benefits of a relationship centric approach to analytics. We will follow with a more technical dive into the features and deployment options of JanusGraph, including accessing the graph with the Apache Tinkerpop API stack, manipulating it with the Blueprints API, and querying the graph with the Gremlin query language. Finally, we will look at how JanusGraph integrates with other technologies like Apache Spark as part of an overall analytics architecture.
Large Scale Graph Analytics with JanusGraph
This document provides an overview of large scale graph analytics and JanusGraph. It discusses graph databases and their use cases. JanusGraph is presented as an open source graph database that can scale to billions of vertices and edges across multiple storage backends like HBase, Cassandra and Bigtable. It uses the TinkerPop framework and Gremlin query language. JanusGraph supports ACID transactions, external indices, and evolving schemas. Example graph queries are demonstrated using the Gremlin console.
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
Microservices can provide terabytes of data in microseconds by mapping data from SQL databases into in-memory key-value stores and column key stores within JVMs. This is done through periodic synchronization of changed data from databases into memory and mapping the in-memory data into fast access structures. The in-memory data is then exposed through Java Stream and REST APIs to microservices for high performance querying and analysis of large datasets. This architecture allows microservices to quickly share access to large datasets and restart rapidly by reloading from the synchronized persistent stores.
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
By leveraging memory-mapped files, Speedment and the Chronicle Engine supports large Java maps that easily can exceed the size of your server’s RAM.Because the Java maps are mapped onto files, these maps can be shared instantly between several microservice JVMs and new microservice instances can be added, removed, or restarted very quickly. Data can be retrieved with predictable ultralow latency for a wide range of operations. The solution can be synchronized with an underlying database so that your in-memory maps will be consistently “alive.” The mapped files can be tens of terabytes, which has been done in real-world deployment cases, and a large number of micro services can share these maps simultaneously. Learn more in this session.
Introduction to libre « fulltext » technology
The presentation will be based on my personal experience on SQLite, MySQL and Zend Search ; on workshops I’ve attended (PostgreSQL) and on tests conducted under my supervision (PostgreSQL, MySQL, Sphinx, Lucene, Xapian). It will cover an exhaustive overview of existing techniques, from the most basic to the more advanced, and will lead to a comparative table of the existing technology.
Scalable and High available Distributed File System Metadata Service Using gR...
Alluxio Community Office Hour
Apr 7, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speaker: Bin Fan
Alluxio (alluxio.io) is an open-source data orchestration system that provides a single namespace federating multiple external distributed storage systems. It is critical for Alluxio to be able to store and serve the metadata of all files and directories from all mounted external storage both at scale and at speed.
This talk shares our design, implementation, and optimization of Alluxio metadata service (master node) to address the scalability challenges. Particularly, we will focus on how to apply and combine techniques including tiered metadata storage (based on off-heap KV store RocksDB), fine-grained file system inode tree locking scheme, embedded state-replicate machine (based on RAFT), exploration and performance tuning in the correct RPC frameworks (thrift vs gRPC) and etc. As a result of the combined above techniques, Alluxio 2.0 is able to store at least 1 billion files with a significantly reduced memory requirement, serving 3000 workers and 30000 clients concurrently.
In this Office Hour, we will go over how to:
- Metadata storage challenges
- How to combine different open source technologies as building blocks
- The design, implementation, and optimization of Alluxio metadata service
Rust All Hands Winter 2011
Rust is a new systems programming language designed for safety, concurrency, and speed. It uses a borrow checker to ensure memory safety and avoid use-after-free bugs, and enables safe concurrency through message passing and ownership rules. The Rust team has been redesigning and implementing the language over the past year based on lessons learned from writing the compiler in Rust. Their goals are to have a memory-safe, concurrent language that achieves performance comparable to C++. They plan to use Rust for the parallel engine of the Servo browser project.
Search at Twitter: Presented by Michael Busch, Twitter
Twitter processes over 500 million tweets per day and more than 2 billion search queries per day. The company uses a search architecture based on Lucene with custom extensions. This includes an in-memory real-time index optimized for concurrency without locks, and a schema-based document factory. Future work includes support for parallel index segments and additional Lucene features.
Building Big Data Streaming Architectures
In this talk we depict how to build streaming architectures connecting to Kafka with different technologies. Includes links to Github code samples.
UNIT I Introduction to NoSQL.pptx
NoSQL databases provide an alternative to traditional relational databases that is well-suited for large datasets, high scalability needs, and flexible, changing schemas. NoSQL databases sacrifice strict consistency for greater scalability and availability. The document model is well-suited for semi-structured data and allows for embedding related data within documents. Key-value stores provide simple lookup of data by key but do not support complex queries. Graph databases effectively represent network-like connections between data elements.
Hbase schema design and sizing apache-con europe - nov 2012
The document provides an overview of HBase schema design and cluster sizing notes. It discusses HBase architecture including tables, regions, distribution, and compactions. It emphasizes the importance of schema design, including using intelligent keys, denormalization, and duplication to overcome limitations. The document also covers techniques like salting keys, hashing vs sequential keys, and examples of schema design for applications like mail inbox and Facebook insights. It stresses designing for the use case and avoiding hotspotting when sizing clusters.
Similar to Immutable Data Structures (20)
Concurrency and Multithreading Demistified - Reversim Summit 2014
Concurrency and Multithreading Demistified - Reversim Summit 2014
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
FiloDB: Reactive, Real-Time, In-Memory Time Series at Scale
DataEngConf: Parquet at Datadog: Fast, Efficient, Portable Storage for Big Data
DataEngConf: Parquet at Datadog: Fast, Efficient, Portable Storage for Big Data
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
JavaOne2016 - Microservices: Terabytes in Microseconds [CON4516]
Scalable and High available Distributed File System Metadata Service Using gR...
Scalable and High available Distributed File System Metadata Service Using gR...
Search at Twitter: Presented by Michael Busch, Twitter
Search at Twitter: Presented by Michael Busch, Twitter
Hbase schema design and sizing apache-con europe - nov 2012
Hbase schema design and sizing apache-con europe - nov 2012
More from 宇 傅
The Volcano/Cascades Optimizer
The document describes the Volcano/Cascades query optimizer. It uses dynamic programming to efficiently search the large space of possible query execution plans. The optimizer represents queries as logical and physical operators connected by transformation and implementation rules. It explores the logical plan space and then builds physical plans by applying these rules. The search is guided by estimating physical operator costs. The optimizer memoizes partial results to avoid redundant work. This approach allows finding optimal execution plans in a principled way that scales to complex queries and optimizer extensions.
PelotonDB - A self-driving database for hybrid workloads
PelotonDB is a self-driving database that uses a hybrid storage layout to support both OLTP and OLAP workloads. It uses logical tiles to decouple storage management from query execution. Tuples can be stored in either a narrow storage model, dense storage model, or flexible storage model depending on how "hot" the data is. The system continuously monitors queries and reorganizes the physical data layout in the background to optimize for the workload over time using k-means clustering. It employs MVCC for concurrency control and minimizes overhead during data reorganization by only modifying versioning metadata. An evaluation using the ADAPT benchmark shows PelotonDB can adapt the storage layout to improve performance.
The Case for Learned Index Structures
The document discusses learned index structures that use machine learning models instead of traditional B-trees. It introduces the Recursive Model Index (RMI) which uses machine learning models at different levels or stages of the index to map keys to positions. The RMI was able to achieve up to 3x faster performance and an order of magnitude smaller size compared to B-trees on various test datasets. It also discusses using machine learning models for point indexes and bloom filters to improve lookup performance and reduce space requirements compared to traditional implementations.
Spark and Spark Streaming
Spark and Spark Streaming can process streaming data using a technique called Discretized Streams (D-Streams) that divides the data into small batch intervals. This allows Spark to provide fault tolerance through checkpointing and recovery of state across intervals. Spark Streaming also introduces the concept of "exactly-once" processing semantics through checkpointing and write ahead logs. Spark Structured Streaming builds on these concepts and adds SQL support and watermarking to allow incremental processing of streaming data.
Functional Programming in Java 8
This document discusses functional programming concepts in Java 8 such as lambda expressions, streams, parallelization, and asynchronization. It covers lambda expressions as a syntactic sugar for anonymous classes, closures that capture variables from enclosing scopes, and functional interfaces. It also covers stream operations like map, filter, flatMap, and collect, parallelization using stream.parallel(), and asynchronization using CompletableFuture and RxJava. It reminds readers about functional programming concepts like pure functions, currying, and the Y-combinator.
Data Streaming Algorithms
Streaming algorithms aim to process massive data streams with very limited memory. They can approximate calculations like minimum, maximum, average and quantiles in one pass over the data using hashing, sketching and other techniques. Key algorithms discussed include HyperLogLog for cardinality, Flajolet-Martin sketch for distinct items, Count-Min sketch for heavy hitters, and Frugal streaming for order statistics like median and quantiles using only logarithmic memory. T-Digest is also summarized as a way to find quantiles in streaming data using a balanced binary tree of centroids.
Golang 101
The document provides an introduction to the Go programming language. Some key points:
- Go was designed by Google to handle large scale software development with thousands of developers and machines. It focuses on concurrency, simplicity, and reliability.
- Some core features of Go include goroutines for lightweight threads, channels for communication between goroutines, and interfaces for abstraction.
- Go aims to compile quickly to machine code for performance and packages into single binaries for easy deployment.
- The document demonstrates building a basic web server in Go and encourages hands-on learning through its tour and examples.
Docker Container: isolation and security
Linux namespaces and control groups (cgroups) provide isolation and security for Docker containers. Namespaces isolate processes by PID, network, IPC, mount points, etc. Cgroups limit CPU, memory, storage resources. Capabilities and security models like seccomp, AppArmor further harden containers by dropping privileges and blocking risky syscalls. Together, these mechanisms isolate containers and applications from hosts, other containers, and external attacks while still allowing resource sharing through the kernel.
Paxos and Raft Distributed Consensus Algorithm
This document summarizes Paxos and Raft, two distributed consensus algorithms. It provides background on Leslie Lamport, who developed Paxos, and describes the roles, quorums, phases, and concepts of the Paxos algorithm. It then introduces Raft as being equivalent to Paxos in fault tolerance and performance but designed to be easier to understand. Raft decomposes the problem into subproblems like leader election, log replication, and safety. The document concludes by pointing to demonstrations of how Paxos and Raft work.
More from 宇 傅 (12)
PelotonDB - A self-driving database for hybrid workloads
PelotonDB - A self-driving database for hybrid workloads
Recently uploaded
How to write a program in any programming language
How to write a program in any programming language
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
Malibou Pitch Deck For Its €3M Seed Round
French start-up Malibou raised a €3 million Seed Round to develop its payroll and human resources
management platform for VSEs and SMEs. The financing round was led by investors Breega, Y Combinator, and FCVC.
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM 🐰
E-commerce Development Services- Hornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
UI5con 2024 - Bring Your Own Design System
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Presentation from Ghazal Aakel and Eric Jochum at the GSD Community Stage Meetup on June 6th, 2024
Microservice Teams - How the cloud changes the way we work
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
SMS API Integration in Saudi Arabia| Best SMS API Service
Discover the benefits and implementation of SMS API integration in the UAE and Middle East. This comprehensive guide covers the importance of SMS messaging APIs, the advantages of bulk SMS APIs, and real-world case studies. Learn how CEQUENS, a leader in communication solutions, can help your business enhance customer engagement and streamline operations with innovative CPaaS, reliable SMS APIs, and omnichannel solutions, including WhatsApp Business. Perfect for businesses seeking to optimize their communication strategies in the digital age.
Oracle 23c New Features For DBAs and Developers.pptx
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
When it is all about ERP solutions, companies typically meet their needs with common ERP solutions like SAP, Oracle, and Microsoft Dynamics. These big players have demonstrated that ERP systems can be either simple or highly comprehensive. This remains true today, but there are new factors to consider, including a promising new contender in the market that’s Odoo. This blog compares Odoo ERP with traditional ERP systems and explains why many companies now see Odoo ERP as the best choice.
What are ERP Systems?
An ERP, or Enterprise Resource Planning, system provides your company with valuable information to help you make better decisions and boost your ROI. You should choose an ERP system based on your company’s specific needs. For instance, if you run a manufacturing or retail business, you will need an ERP system that efficiently manages inventory. A consulting firm, on the other hand, would benefit from an ERP system that enhances daily operations. Similarly, eCommerce stores would select an ERP system tailored to their needs.
Because different businesses have different requirements, ERP system functionalities can vary. Among the various ERP systems available, Odoo ERP is considered one of the best in the ERp market with more than 12 million global users today.
Odoo is an open-source ERP system initially designed for small to medium-sized businesses but now suitable for a wide range of companies. Odoo offers a scalable and configurable point-of-sale management solution and allows you to create customised modules for specific industries. Odoo is gaining more popularity because it is built in a way that allows easy customisation, has a user-friendly interface, and is affordable. Here, you will cover the main differences and get to know why Odoo is gaining attention despite the many other ERP systems available in the market.
GreenCode-A-VSCode-Plugin--Dario-Jurisic
Presentation about a VSCode plugin from Dario Jurisic at the GSD Community Stage meetup
Requirement Traceability in Xen Functional Safety
Using OpenFastTrace to trace requirements in Xen functional safety
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Presented at NLJUG's J-Spring 2024.
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Mobile app Development Services | Drona Infotech
Drona Infotech is one of the Best Mobile App Development Company In Noida Maintenance and ongoing support. mobile app development Services can help you maintain and support your app after it has been launched. This includes fixing bugs, adding new features, and keeping your app up-to-date with the latest
Visit Us For :
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
Recently uploaded (20)
How to write a program in any programming language
How to write a program in any programming language
All you need to know about Spring Boot and GraalVM
All you need to know about Spring Boot and GraalVM
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
Odoo ERP Vs. Traditional ERP Systems – A Comparative Analysis
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
J-Spring 2024 - Going serverless with Quarkus, GraalVM native images and AWS ...
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
UI5con 2024 - Keynote: Latest News about UI5 and it’s Ecosystem
Immutable Data Structures
- 1. Immutable Data Structures Eric Fu 2018-06-15
- 2. Agenda • Stack • Tree • Vector Trie • HAMT • Git • LSM-Tree • Practice
- 3. Immutable Stack A Clean Example for Starters
- 5. Stack
- 6. Stack
- 7. Stack
- 8. Stack
- 10. Tree
- 11. Tree
- 12. RB-Tree
- 13. RB-Tree
- 14. Vector Trie Immutable Array List
- 15. Trie Also called: • Dictionary tree • Radix tree • Prefix tree • Digital search tree
- 16. Trie (compressed) • With prefix only "A", "to", "tea", "ted" "ten", "i", "in", "inn"
- 17. Vector Trie
- 19. Vector Trie - Update
- 20. Vector Trie - Push
- 21. Vector Trie – Push (1/3)
- 22. Vector Trie – Push (2/3)
- 23. Vector Trie – Push (3/3)
- 24. Vector Trie – Pop (1/3)
- 25. Vector Trie – Pop (2/3)
- 26. Vector Trie – Pop (3/3)
- 28. Go Efficient • More branches
- 29. Go Efficient • More branches • Align to binary bits
- 30. Go Efficient • More branches – 32 • Align to binary bits – 5 bits
- 31. Hash Array Mapped Trie (HAMT) Immutable Hash Table
- 32. Rethink Vector Trie • Must the index be continuous (0, 1, 2...) ? • Actually not! • Furthermore: hash(anything) -> integer
- 33. HAMT - Naive • Contains {0x00, 0xff}
- 34. HAMT – Bitmap Compressed • Compress each node with bitmap
- 35. HAMT - Pruned • Leave out unnecessary nodes
- 37. Git
- 38. How Git works? • Git stored all versions of all directories and files • A Git commit is a snapshot of root directory • A Git branch/tag is a reference to some commit
- 52. That's it!
- 59. More Trees
- 60. Insights • Modify in memory • Read merged results • Newly merged blocks are appended
- 61. Modern LSM-Tree Implements • Memtable • Immutable Memtable • Level 0 (overlapped) • Level 1 • Level 2 • .... Memory Disk Write Read 61
- 62. Immutable in Practice In Java
- 63. A Familiar Example • Java's string vs. CPP's string
- 64. Easy to understand • Same name, same value Mutable Request Immutable Request:
- 66. Always thread-safe • Immutable to programming as MVCC to database • Lock or Snapshot
- 67. Other Pros • Friendly for caching • Atomic Failure • Avoid null-pointer exception
- 68. Java Example 1
- 69. Java Example 2 • Requirement: cache the results of factorizing • Two variable with setter? • Two atomic variable?
- 70. Java Example 2 (cont.)
- 71. Java Example 2 (cont.)
- 72. Thanks! Q&A