Seastar @ SF/BA C++UG
•
2 likes•704 views
Seastar is a C++ asynchronous programming framework that allows for multi-domain async programming across networking, storage I/O, and multi-core communications. It uses an event-driven model where each logical core runs a task scheduler independently. Logical cores communicate through queues. Seastar is applicable for workloads with high I/O to compute ratios, high concurrency needs, and distributed applications. It provides futures/promises abstractions and rich APIs for tasks like HTTP servers, RPC, and distributed databases.
Report
Share
Report
Share
Download to read offline
Recommended
Seastar @ NYCC++UG
Seastar is a C++ asynchronous programming framework that allows for multi-domain async programming across networking, storage I/O, and inter-core communications. It uses an event-driven model where each logical core runs a task scheduler independently. Cores communicate through queues and each core owns its own data in a "shared nothing" architecture. Seastar provides futures/promises abstractions and composable APIs for networking, storage, and more. It is applicable for high concurrency workloads involving disk and network I/O like distributed databases and object stores.
Seastar at Linux Foundation Collaboration Summit
We have developed a new framework, Seastar, for high-throughput server applications, along with a key-value store capable of millions of transactions per second. Seastar, which runs on OSv and Linux, is completely asynchronous and based on shared-nothing data structures that eliminate costly locking between CPUs. SeaStar is event-driven and supports writing non-blocking, asynchronous server code in a straightforward manner that facilitates debugging and reasoning about performance.
Seastar / ScyllaDB, or how we implemented a 10-times faster Cassandra
Nadav Har'El from ScyllaDB team, on the path for 10x Cassandra performance.
Seastar / ScyllaDB intro at the generalist engineer meetup
Developing Scylla Applications: Practical Tips
This document provides tips for developing applications with ScyllaDB. It recommends limiting request concurrency by using semaphores or driver configuration options. It discusses strategies for retrying requests, including ensuring idempotency and introducing delays through exponential backoff. It also recommends using prepared statements, the right load balancing policy, and testing retry strategies before production.
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
A solid backup strategy is a DBA's bread and butter. Cassandra's nodetool snapshot makes it easy to back up the SSTable files, but there remains the question of where to put them and how. Knewton's backup strategy uses Ansible for distributed backups and stores them in S3.
Unfortunately, it's all too easy to store backups that are essentially useless due to the absence of a coherent restoration strategy. This problem proved much more difficult and nuanced than taking the backups themselves. I will discuss Knewton's restoration strategy, which again leverages Ansible, yet I will focus on general principles and pitfalls to be avoided. In particular, restores necessitated modifying our backup strategy to generate cluster-wide metadata that is critical for a smooth automated restoration. Such pitfalls indicate that a restore-focused backup design leads to faster and more deterministic recovery.
About the Speaker
Joshua Wickman Database Engineer, Knewton
Dr. Joshua Wickman is currently part of the database team at Knewton, a NYC tech company focused on adaptive learning. He earned his PhD at the University of Delaware in 2012, where he studied particle physics models of the early universe. After a brief stint teaching college physics, he entered the New York tech industry in 2014 working with NoSQL, first with MongoDB and then Cassandra. He was certified in Cassandra at his first Cassandra Summit in 2015.
Gnocchi v3 brownbag
This document summarizes the author's experience optimizing Gnocchi, an open source time-series database, to store metrics for hundreds of thousands of resources over many months. The author describes improving performance by adding Ceph storage nodes, tuning Ceph configurations, minimizing I/O operations, and improving the storage format. Benchmark results show the new version achieves 50% higher write throughput, 40-60% faster computation times, 30-60% better overall performance, and 30-40% fewer operations. Usage hints are also provided to help optimize for different use cases.
Openstack Scheduler and Scalability Issue
The nova scheduler determines where to run virtual machine instances in OpenStack. It uses filters and weights to identify the best compute host from available information. An instance request is fulfilled by the scheduler selecting a host, informing the conductor, and having the compute node launch the instance. For large clouds, a horizontally scalable scheduler that uses flavor-based queues and avoids the database may improve performance. A Scheduler-as-a-Service project is also planned to provide a generic scheduler for other OpenStack components.
Anatomy of an action
A key feature when monitoring and debugging any Cloud infrastructure is to provide the ability to trace, track, and collate all the individual, discrete steps that compose an event. A typical resource action in OpenStack is often a combination of smaller tasks -- which given the distributed nature of OpenStack -- can fail at unpredictable points in the workflow. By collecting the appropriate events, operators can view all events within Ceilometer, filter on a failed action and trace back the history of related events to spot anomalies or errors. In this talk, we provide an overview of the recent enhancements made in Ceilometer to support the collection of event notifications from OpenStack services. We will describe: how events are processed, transformed and stored in Ceilometer; how you can derive metrics from events; and how it’s possible to track the events of a resource and analyse where errors occur.
Recommended
Seastar @ NYCC++UG
Seastar is a C++ asynchronous programming framework that allows for multi-domain async programming across networking, storage I/O, and inter-core communications. It uses an event-driven model where each logical core runs a task scheduler independently. Cores communicate through queues and each core owns its own data in a "shared nothing" architecture. Seastar provides futures/promises abstractions and composable APIs for networking, storage, and more. It is applicable for high concurrency workloads involving disk and network I/O like distributed databases and object stores.
Seastar at Linux Foundation Collaboration Summit
We have developed a new framework, Seastar, for high-throughput server applications, along with a key-value store capable of millions of transactions per second. Seastar, which runs on OSv and Linux, is completely asynchronous and based on shared-nothing data structures that eliminate costly locking between CPUs. SeaStar is event-driven and supports writing non-blocking, asynchronous server code in a straightforward manner that facilitates debugging and reasoning about performance.
Seastar / ScyllaDB, or how we implemented a 10-times faster Cassandra
Nadav Har'El from ScyllaDB team, on the path for 10x Cassandra performance.
Seastar / ScyllaDB intro at the generalist engineer meetup
Developing Scylla Applications: Practical Tips
This document provides tips for developing applications with ScyllaDB. It recommends limiting request concurrency by using semaphores or driver configuration options. It discusses strategies for retrying requests, including ensuring idempotency and introducing delays through exponential backoff. It also recommends using prepared statements, the right load balancing policy, and testing retry strategies before production.
Cassandra Backups and Restorations Using Ansible (Joshua Wickman, Knewton) | ...
A solid backup strategy is a DBA's bread and butter. Cassandra's nodetool snapshot makes it easy to back up the SSTable files, but there remains the question of where to put them and how. Knewton's backup strategy uses Ansible for distributed backups and stores them in S3.
Unfortunately, it's all too easy to store backups that are essentially useless due to the absence of a coherent restoration strategy. This problem proved much more difficult and nuanced than taking the backups themselves. I will discuss Knewton's restoration strategy, which again leverages Ansible, yet I will focus on general principles and pitfalls to be avoided. In particular, restores necessitated modifying our backup strategy to generate cluster-wide metadata that is critical for a smooth automated restoration. Such pitfalls indicate that a restore-focused backup design leads to faster and more deterministic recovery.
About the Speaker
Joshua Wickman Database Engineer, Knewton
Dr. Joshua Wickman is currently part of the database team at Knewton, a NYC tech company focused on adaptive learning. He earned his PhD at the University of Delaware in 2012, where he studied particle physics models of the early universe. After a brief stint teaching college physics, he entered the New York tech industry in 2014 working with NoSQL, first with MongoDB and then Cassandra. He was certified in Cassandra at his first Cassandra Summit in 2015.
Gnocchi v3 brownbag
This document summarizes the author's experience optimizing Gnocchi, an open source time-series database, to store metrics for hundreds of thousands of resources over many months. The author describes improving performance by adding Ceph storage nodes, tuning Ceph configurations, minimizing I/O operations, and improving the storage format. Benchmark results show the new version achieves 50% higher write throughput, 40-60% faster computation times, 30-60% better overall performance, and 30-40% fewer operations. Usage hints are also provided to help optimize for different use cases.
Openstack Scheduler and Scalability Issue
The nova scheduler determines where to run virtual machine instances in OpenStack. It uses filters and weights to identify the best compute host from available information. An instance request is fulfilled by the scheduler selecting a host, informing the conductor, and having the compute node launch the instance. For large clouds, a horizontally scalable scheduler that uses flavor-based queues and avoids the database may improve performance. A Scheduler-as-a-Service project is also planned to provide a generic scheduler for other OpenStack components.
Anatomy of an action
A key feature when monitoring and debugging any Cloud infrastructure is to provide the ability to trace, track, and collate all the individual, discrete steps that compose an event. A typical resource action in OpenStack is often a combination of smaller tasks -- which given the distributed nature of OpenStack -- can fail at unpredictable points in the workflow. By collecting the appropriate events, operators can view all events within Ceilometer, filter on a failed action and trace back the history of related events to spot anomalies or errors. In this talk, we provide an overview of the recent enhancements made in Ceilometer to support the collection of event notifications from OpenStack services. We will describe: how events are processed, transformed and stored in Ceilometer; how you can derive metrics from events; and how it’s possible to track the events of a resource and analyse where errors occur.
Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red Hat
Audience Level
Intermediate
Synopsis
Hypercoverged Compute, Network and Storage is ready for production workloads – where it makes sense.
Whether you’re a telecommunications carrier, service provider or enterprise; implementing Network Function Virtualisation (NFV), focusing on specific known workloads or simply a dev / test cloud – deploying a hypercoverged OpenStack cloud makes a lot of sense.
Come along and discover which workloads fit a hyperconverged architecture, see examples and look into the very near future and learn how OpenStack is truly ready to serve your every need.
Speaker Bio
Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.
Cassandra at Glogster
Roman Komkov presented on Cassandra at Glogster. Glogster EDU is an online platform used by 19 million users for presentation and learning, generating over 45,000 new projects daily. Glogster has used Cassandra since 2011 as its primary database, starting with 0.6-0.8 and upgrading over time. Currently it uses a 5 node cluster storing 600GB of data. Upgrades brought challenges around migration, repairs, and a data loss incident that took 10 days to repair after decommissioning an old datacenter without proper hints handling. Lessons included increasing repair windows, enabling parallel repairs, and regularly scheduling backups and repairs.
1027 predictive models in 10 seconds, by David Pardo Villaverde, Corunet
This document discusses using ClickHouse to run 1027 predictive models on 250 million records in under 4 seconds. It shows how ClickHouse was able to import CSV data and perform over 1000 queries to test different predictive models on a large dataset in a very short period of time without much configuration. The document concludes that ClickHouse is a powerful tool for running massive predictive modeling workloads at scale with good performance out of the box.
OSMC 2017 | Icinga 2 + Director, flexible Thresholds with Ansible by Kevin H...
An introduction to the utilization of flexible Thresholds and how to adapt your checks to an ever changing Environment.
Central points:
– Where can I use flexible Thresholds
– What are the advantages and disadvantages
– Is it worth it?
Code for the earth OCP APAC Tokyo 2013-05
OCP APAC Engineering Workshop Tokyo Day2 (2013-5-28)
- Application Framework (eco PHP)
- Measurement of Runtime Efficiency
Tuning Speculative Retries to Fight Latency (Michael Figuiere, Minh Do, Netfl...
The Cassandra architecture shines at ensuring a very high availability of data even while nodes are failing or are overloaded. On the other hand, query latency will often rise during these events, especially on the higher percentiles. Many improvements have been made to reduce this effect over the past years. This talk will focus on one in particular: Speculative Retries. Introduced in Cassandra 2.0 on the server side and in the Java Driver 3.0 on the client side, this strategy remains complex to fully understand and to finely tune. This talk will deep dive into theoretical and practical aspects of Speculative Retries, showing the effect of tuning strategies with ad-hoc benchmarks.
About the Speakers
Michael Figuiere Cloud Platform Engineer, Netflix
Michael is a senior software engineer at Netflix where he works on improving the cloud storage infrastructure. He previously worked at Apple and DataStax where he worked for several years on creating Drivers and Developer Tools for Cassandra. At ease with both enterprise applications and lower level technologies, he specializes in distributed architectures and topics such as databases, search engines, and cloud.
Minh Do Senior Distributed Engineer, Netflix
Minh Do has been working at Netflix for the last several years to run, patch, and troubleshoot Cassandra on both server and client sides, and is also a co-creator of Dynomite project. Prior to Netflix, at Tango, he spearheaded its Big Data pipeline system from the ground using Spark/Hadoop. Before that, at Qualys, he built a distributed queue system that bridges traffics between all major components. He has passion in distributed system, machine learning/deep learning, and data storages.
VPC by Default時代のアクセス制御
This document discusses VPC by default and AWS services. It provides examples of how EC2 and S3 can communicate through a VPC using security groups and NACLs. It also discusses using VPC for hybrid use cases with on-premise systems through IPsec VPN or Direct Connect. Finally, it highlights how VPC allows integration between different AWS services.
DataStax: Backup and Restore in Cassandra and OpsCenter
Cassandra and OpsCenter has a range of backup and restore topics. I will start with a basic overview of Cassandra backup/restore, walking through the operational steps to provide the understanding required to perform an on disk backup and restore. Expanding on this overview, I'll cover the limitations (including schema requirements) and their impact on the restore process. Further, I'll discuss commit log archiving and point in time restore operations. After covering the underlying operations, I'll wrap up with a discussion of how OpsCenter automates this process and leverages S3.
Altinity Cluster Manager: ClickHouse Management for Kubernetes and Cloud
Webinar. August 21, 2019
By Robert Hodges and Altinity Engineering Team
Simplified management is a prerequisite for running any data warehouse at scale. Altinity is developing a new web-based console for ClickHouse called the Altinity Cluster Manager. It's now in beta and offers simplified operation of ClickHouse installations for users. In this webinar we introduce the ACM and demonstrate use on Kubernetes as well as Amazon Web Services. Attendees are welcome to sign up as beta testers and provide feedback. Please join us to see the future of Clickhouse management!
Move Over, Rsync
This document summarizes a presentation about the advantages of using ZFS for data replication compared to rsync. It provides several examples showing ZFS performing full dataset replications between 289x and 1,148x faster than rsync due to its use of copy-on-write snapshots and incremental block-level replication. It also briefly explains how copy-on-write snapshots and incremental replication work at a high-level in ZFS to provide such performance advantages over rsync. Additional context is provided about data services that offer ZFS-based replication for $60 or less per terabyte per month.
Gnocchi v4 (preview)
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
Virtual training Intro to the Tick stack and InfluxEnterprise
In this webinar, we will provide an introduction to the components of the TICK Stack and a review the features of InfluxEnterprise and InfluxCloud. We also demo how to install the TICK stack.
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
Demi Ben-Ari gave a presentation about dumping time series data using Apache Spark. The presentation covered an overview of Spark, the structure of maritime location and metadata being collected over time, and the initial challenges of missing data and late arrivals causing holes in the data. It described the evolution of solutions from MongoDB to Cassandra and optimizations made to improve write speeds from 40 minutes to under 5 minutes. The fastest solution was to write the raw data to S3 and have a separate process aggregate it into Cassandra for serving.
Speeding up R with Parallel Programming in the Cloud
Presented to eRum (Budapest), May 2018
There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe the doAzureParallel package, a backend to the "foreach" package that automates the process of spawning a cluster of virtual machines in the Azure cloud to process iterations in parallel. This will include an example of optimizing hyperparameters for a predictive model using the "caret" package.
Migrating Existing Open Source Machine Learning to Azure
By David Smith. Presented at Microsoft Build (Seattle), May 7 2018.
Your data scientists have created predictive models using open-source tools, proprietary software, or some combination of both, and now you are interested in lifting and shifting those models to the cloud. In this talk, I'll describe how data scientists can transition their existing workflows — while using mostly the same tools and processes — to train and deploy machine learning models based on open source frameworks to Azure. I'll provide guidance on keeping connections to data sources up-to-date, evaluating and monitoring models, and deploying applications that make use of those models.
Using ansible to core os & kubernetes clusters
Slides for my talk on how to set up a coreos / kubernetes cluster on baremetal at the ansible meetup berlin at 04/28/2016 in berlin.
Antoine Coetsier - billing the cloud
Antoine Coetsier - billing the cloud. Presentation from CloudStack European User Group, Thursday, April 19, London.
Build a Complex, Realtime Data Management App with Postgres 14!
Congratulations: you've been selected to build an application that will manage reservations for rooms!
On the surface, this sounds simple, but you are building a system for managing a high traffic reservation web page, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the website checking to see what availability each room has.
Fortunately, PostgreSQL is prepared for this! And even better, we will be using Postgres 14 to make the problem even easier!
We will explore the following PostgreSQL features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges / Multirnages
Indexes such as:
* GiST
* Common Table Expressions and Recursion (though multiranges will make things easier!)
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all users made possible by the innovation of PostgreSQL!
Taking Your Database Beyond the Border of a Single Kubernetes Cluster
The document discusses expanding an Apache Cassandra database deployment from a single Kubernetes cluster across multiple clusters. It notes some challenges with networking, address collisions, and service discovery that must be addressed when deploying a database across cluster boundaries. Potential solutions discussed include using overlay networks, managed DNS, Istio, Cilium, and customizing Kubernetes DNS configurations to help route traffic and discover services between clusters. The presenters aim to demonstrate how to deploy a multi-cluster Cassandra database managed by the K8ssandra operator.
Apache MXNet Distributed Training Explained In Depth by Viacheslav Kovalevsky...
Distributed training is a complex process that does more harm than good if it not setup correctly.
https://www.bigdataspain.org/2017/talk/apache-mxnet-distributed-training-explained-in-depth
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
Scylla: 1 Million CQL operations per second per server
My Cassandra Summit 2015 presentation introducing Scylla, an open source NoSQL implementation compatible with Apache Cassandra, but 10 times faster.
De-animated
http://scylladb.com
ScyllaDB @ Apache BigData, may 2016
Scylla is a new open source NoSQL database that is compatible with Apache Cassandra but provides significantly higher performance through a redesign that takes advantage of modern hardware. Scylla is capable of over 1.8 million operations per second per node with predictable low latencies. It uses an architecture with shard-per-core and reactor programming that avoids locks and threads for near-linear scaling. Scylla also has its own efficient unified cache and I/O scheduler that maximize throughput and allow it to outperform Cassandra on benchmarks by an order of magnitude. Scylla is fully compatible with Cassandra and aims to build an open source community around ongoing core database improvements.
More Related Content
What's hot
Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red Hat
Audience Level
Intermediate
Synopsis
Hypercoverged Compute, Network and Storage is ready for production workloads – where it makes sense.
Whether you’re a telecommunications carrier, service provider or enterprise; implementing Network Function Virtualisation (NFV), focusing on specific known workloads or simply a dev / test cloud – deploying a hypercoverged OpenStack cloud makes a lot of sense.
Come along and discover which workloads fit a hyperconverged architecture, see examples and look into the very near future and learn how OpenStack is truly ready to serve your every need.
Speaker Bio
Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.
Cassandra at Glogster
Roman Komkov presented on Cassandra at Glogster. Glogster EDU is an online platform used by 19 million users for presentation and learning, generating over 45,000 new projects daily. Glogster has used Cassandra since 2011 as its primary database, starting with 0.6-0.8 and upgrading over time. Currently it uses a 5 node cluster storing 600GB of data. Upgrades brought challenges around migration, repairs, and a data loss incident that took 10 days to repair after decommissioning an old datacenter without proper hints handling. Lessons included increasing repair windows, enabling parallel repairs, and regularly scheduling backups and repairs.
1027 predictive models in 10 seconds, by David Pardo Villaverde, Corunet
This document discusses using ClickHouse to run 1027 predictive models on 250 million records in under 4 seconds. It shows how ClickHouse was able to import CSV data and perform over 1000 queries to test different predictive models on a large dataset in a very short period of time without much configuration. The document concludes that ClickHouse is a powerful tool for running massive predictive modeling workloads at scale with good performance out of the box.
OSMC 2017 | Icinga 2 + Director, flexible Thresholds with Ansible by Kevin H...
An introduction to the utilization of flexible Thresholds and how to adapt your checks to an ever changing Environment.
Central points:
– Where can I use flexible Thresholds
– What are the advantages and disadvantages
– Is it worth it?
Code for the earth OCP APAC Tokyo 2013-05
OCP APAC Engineering Workshop Tokyo Day2 (2013-5-28)
- Application Framework (eco PHP)
- Measurement of Runtime Efficiency
Tuning Speculative Retries to Fight Latency (Michael Figuiere, Minh Do, Netfl...
The Cassandra architecture shines at ensuring a very high availability of data even while nodes are failing or are overloaded. On the other hand, query latency will often rise during these events, especially on the higher percentiles. Many improvements have been made to reduce this effect over the past years. This talk will focus on one in particular: Speculative Retries. Introduced in Cassandra 2.0 on the server side and in the Java Driver 3.0 on the client side, this strategy remains complex to fully understand and to finely tune. This talk will deep dive into theoretical and practical aspects of Speculative Retries, showing the effect of tuning strategies with ad-hoc benchmarks.
About the Speakers
Michael Figuiere Cloud Platform Engineer, Netflix
Michael is a senior software engineer at Netflix where he works on improving the cloud storage infrastructure. He previously worked at Apple and DataStax where he worked for several years on creating Drivers and Developer Tools for Cassandra. At ease with both enterprise applications and lower level technologies, he specializes in distributed architectures and topics such as databases, search engines, and cloud.
Minh Do Senior Distributed Engineer, Netflix
Minh Do has been working at Netflix for the last several years to run, patch, and troubleshoot Cassandra on both server and client sides, and is also a co-creator of Dynomite project. Prior to Netflix, at Tango, he spearheaded its Big Data pipeline system from the ground using Spark/Hadoop. Before that, at Qualys, he built a distributed queue system that bridges traffics between all major components. He has passion in distributed system, machine learning/deep learning, and data storages.
VPC by Default時代のアクセス制御
This document discusses VPC by default and AWS services. It provides examples of how EC2 and S3 can communicate through a VPC using security groups and NACLs. It also discusses using VPC for hybrid use cases with on-premise systems through IPsec VPN or Direct Connect. Finally, it highlights how VPC allows integration between different AWS services.
DataStax: Backup and Restore in Cassandra and OpsCenter
Cassandra and OpsCenter has a range of backup and restore topics. I will start with a basic overview of Cassandra backup/restore, walking through the operational steps to provide the understanding required to perform an on disk backup and restore. Expanding on this overview, I'll cover the limitations (including schema requirements) and their impact on the restore process. Further, I'll discuss commit log archiving and point in time restore operations. After covering the underlying operations, I'll wrap up with a discussion of how OpsCenter automates this process and leverages S3.
Altinity Cluster Manager: ClickHouse Management for Kubernetes and Cloud
Webinar. August 21, 2019
By Robert Hodges and Altinity Engineering Team
Simplified management is a prerequisite for running any data warehouse at scale. Altinity is developing a new web-based console for ClickHouse called the Altinity Cluster Manager. It's now in beta and offers simplified operation of ClickHouse installations for users. In this webinar we introduce the ACM and demonstrate use on Kubernetes as well as Amazon Web Services. Attendees are welcome to sign up as beta testers and provide feedback. Please join us to see the future of Clickhouse management!
Move Over, Rsync
This document summarizes a presentation about the advantages of using ZFS for data replication compared to rsync. It provides several examples showing ZFS performing full dataset replications between 289x and 1,148x faster than rsync due to its use of copy-on-write snapshots and incremental block-level replication. It also briefly explains how copy-on-write snapshots and incremental replication work at a high-level in ZFS to provide such performance advantages over rsync. Additional context is provided about data services that offer ZFS-based replication for $60 or less per terabyte per month.
Gnocchi v4 (preview)
early benchmarks on pre-release Gnocchi v4. includes benchmark comparison between all-ceph v3.x driver versus all-ceph v4 driver. also, shows benchmark using redis+ceph deployment.
Virtual training Intro to the Tick stack and InfluxEnterprise
In this webinar, we will provide an introduction to the components of the TICK Stack and a review the features of InfluxEnterprise and InfluxCloud. We also demo how to install the TICK stack.
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
Demi Ben-Ari gave a presentation about dumping time series data using Apache Spark. The presentation covered an overview of Spark, the structure of maritime location and metadata being collected over time, and the initial challenges of missing data and late arrivals causing holes in the data. It described the evolution of solutions from MongoDB to Cassandra and optimizations made to improve write speeds from 40 minutes to under 5 minutes. The fastest solution was to write the raw data to S3 and have a separate process aggregate it into Cassandra for serving.
Speeding up R with Parallel Programming in the Cloud
Presented to eRum (Budapest), May 2018
There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe the doAzureParallel package, a backend to the "foreach" package that automates the process of spawning a cluster of virtual machines in the Azure cloud to process iterations in parallel. This will include an example of optimizing hyperparameters for a predictive model using the "caret" package.
Migrating Existing Open Source Machine Learning to Azure
By David Smith. Presented at Microsoft Build (Seattle), May 7 2018.
Your data scientists have created predictive models using open-source tools, proprietary software, or some combination of both, and now you are interested in lifting and shifting those models to the cloud. In this talk, I'll describe how data scientists can transition their existing workflows — while using mostly the same tools and processes — to train and deploy machine learning models based on open source frameworks to Azure. I'll provide guidance on keeping connections to data sources up-to-date, evaluating and monitoring models, and deploying applications that make use of those models.
Using ansible to core os & kubernetes clusters
Slides for my talk on how to set up a coreos / kubernetes cluster on baremetal at the ansible meetup berlin at 04/28/2016 in berlin.
Antoine Coetsier - billing the cloud
Antoine Coetsier - billing the cloud. Presentation from CloudStack European User Group, Thursday, April 19, London.
Build a Complex, Realtime Data Management App with Postgres 14!
Congratulations: you've been selected to build an application that will manage reservations for rooms!
On the surface, this sounds simple, but you are building a system for managing a high traffic reservation web page, so we know that a lot of people will be accessing the system. Therefore, we need to ensure that the system can handle all of the eager users that will be flooding the website checking to see what availability each room has.
Fortunately, PostgreSQL is prepared for this! And even better, we will be using Postgres 14 to make the problem even easier!
We will explore the following PostgreSQL features:
* Data types and their functionality, such as:
* Data/Time types
* Ranges / Multirnages
Indexes such as:
* GiST
* Common Table Expressions and Recursion (though multiranges will make things easier!)
* Set generating functions and LATERAL queries
* Functions and the PL/PGSQL
* Triggers
* Logical decoding and streaming
We will be writing our application primary with SQL, though we will sneak in a little bit of Python and using Kafka to demonstrate the power of logical decoding.
At the end of the presentation, we will have a working application, and you will be happy knowing that you provided a wonderful user experience for all users made possible by the innovation of PostgreSQL!
Taking Your Database Beyond the Border of a Single Kubernetes Cluster
The document discusses expanding an Apache Cassandra database deployment from a single Kubernetes cluster across multiple clusters. It notes some challenges with networking, address collisions, and service discovery that must be addressed when deploying a database across cluster boundaries. Potential solutions discussed include using overlay networks, managed DNS, Istio, Cilium, and customizing Kubernetes DNS configurations to help route traffic and discover services between clusters. The presenters aim to demonstrate how to deploy a multi-cluster Cassandra database managed by the K8ssandra operator.
Apache MXNet Distributed Training Explained In Depth by Viacheslav Kovalevsky...
Distributed training is a complex process that does more harm than good if it not setup correctly.
https://www.bigdataspain.org/2017/talk/apache-mxnet-distributed-training-explained-in-depth
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
What's hot (20)
Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red Hat
Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red Hat
1027 predictive models in 10 seconds, by David Pardo Villaverde, Corunet
1027 predictive models in 10 seconds, by David Pardo Villaverde, Corunet
OSMC 2017 | Icinga 2 + Director, flexible Thresholds with Ansible by Kevin H...
OSMC 2017 | Icinga 2 + Director, flexible Thresholds with Ansible by Kevin H...
Tuning Speculative Retries to Fight Latency (Michael Figuiere, Minh Do, Netfl...
Tuning Speculative Retries to Fight Latency (Michael Figuiere, Minh Do, Netfl...
DataStax: Backup and Restore in Cassandra and OpsCenter
DataStax: Backup and Restore in Cassandra and OpsCenter
Altinity Cluster Manager: ClickHouse Management for Kubernetes and Cloud
Altinity Cluster Manager: ClickHouse Management for Kubernetes and Cloud
Virtual training Intro to the Tick stack and InfluxEnterprise
Virtual training Intro to the Tick stack and InfluxEnterprise
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
S3, Cassandra or Outer Space? Dumping Time Series Data using Spark - Demi Ben...
Speeding up R with Parallel Programming in the Cloud
Speeding up R with Parallel Programming in the Cloud
Migrating Existing Open Source Machine Learning to Azure
Migrating Existing Open Source Machine Learning to Azure
Using ansible to core os & kubernetes clusters
Using ansible to core os & kubernetes clusters
Build a Complex, Realtime Data Management App with Postgres 14!
Build a Complex, Realtime Data Management App with Postgres 14!
Taking Your Database Beyond the Border of a Single Kubernetes Cluster
Taking Your Database Beyond the Border of a Single Kubernetes Cluster
Apache MXNet Distributed Training Explained In Depth by Viacheslav Kovalevsky...
Apache MXNet Distributed Training Explained In Depth by Viacheslav Kovalevsky...
Viewers also liked
Scylla: 1 Million CQL operations per second per server
My Cassandra Summit 2015 presentation introducing Scylla, an open source NoSQL implementation compatible with Apache Cassandra, but 10 times faster.
De-animated
http://scylladb.com
ScyllaDB @ Apache BigData, may 2016
Scylla is a new open source NoSQL database that is compatible with Apache Cassandra but provides significantly higher performance through a redesign that takes advantage of modern hardware. Scylla is capable of over 1.8 million operations per second per node with predictable low latencies. It uses an architecture with shard-per-core and reactor programming that avoids locks and threads for near-linear scaling. Scylla also has its own efficient unified cache and I/O scheduler that maximize throughput and allow it to outperform Cassandra on benchmarks by an order of magnitude. Scylla is fully compatible with Cassandra and aims to build an open source community around ongoing core database improvements.
Scylla Summit 2016: Keynote - Big Data Goes Native
This document discusses Scylla, a new database that aims to improve upon existing databases. It notes several key differences in Scylla's architecture that allow it to be faster and more scalable than other databases, including its use of techniques like log-structured merge trees, lock-free design, and asynchronous programming. The document also outlines Scylla's value proposition as the fastest database with the best high availability and ease of management compared to other options.
Scylla Summit 2016: ScyllaDB, Present and Future
Where is Scylla now and where is it going? ScyllaDB's CTO Avi Kivity outlines the 3 ScyllaDB Commitments, and gives an overview of the ScyllaDB road map.
Scylla Summit 2016: Scylla at Samsung SDS
Learn about the challenges, solution and technical validation for Scylla in Samsung SDS. There are also use cases and the future plan.
Scylla Summit 2016: Using ScyllaDB for a Microservice-based Pipeline in Go
How do you handle the continuous transformation and refinement of billions of entities with some sort of reliability and performance? In this talk, Henrik will describe how Scylla enabled him and his team to create a pipelined solution using a series of microservices written in Go communicating with each other using Nats. You’ll hear about the mistakes and learnings they had along the way as they built the services that led to the great performance and stability they are experiencing today.
Viewers also liked (6)
Scylla: 1 Million CQL operations per second per server
Scylla: 1 Million CQL operations per second per server
Scylla Summit 2016: Keynote - Big Data Goes Native
Scylla Summit 2016: Keynote - Big Data Goes Native
Scylla Summit 2016: Using ScyllaDB for a Microservice-based Pipeline in Go
Scylla Summit 2016: Using ScyllaDB for a Microservice-based Pipeline in Go
Similar to Seastar @ SF/BA C++UG
Back to the future with C++ and Seastar
Seastar is an open source framework that provides highly scalable and asynchronous distributed applications. It uses a shared-nothing architecture with no locks or threads to achieve linear scaling across cores. Applications built on Seastar can handle millions of connections and I/O operations in parallel. It uses an asynchronous programming model based on promises and futures with zero-copy networking and disk I/O for high performance.
Programming Sideways: Asynchronous Techniques for Android
Android apps need to respond fast, support highly parallel execution and multi component architecture.
Learn some tricks of the trade for these problems!
as presented at www.mobileconference.it (2013 edition)
JS everywhere 2011
The document discusses the future of server-side JavaScript. It covers various Node.js frameworks and libraries that support both synchronous and asynchronous programming styles. CommonJS aims to provide interoperability across platforms by implementing synchronous proposals using fibers. Examples demonstrate how CommonJS allows for synchronous-like code while maintaining asynchronous behavior under the hood. Benchmarks show it has comparable performance to Node.js. The author advocates for toolkits over frameworks and continuing development of common standards and packages.
Server side JavaScript: going all the way
Node has captured the attention of early adopters by clearly differentiating itself as being asynchronous from the ground up while remaining accessible. Now that server side JavaScript is at the cutting edge of the asynchronous, real time web, it is in a much better position to establish itself as the go to language for also making synchronous, CRUD webapps and gain a stronger foothold on the server.
This talk covers the current state of server side JavaScript beyond Node. It introduces Common Node, a synchronous CommonJS compatibility layer using node-fibers which bridges the gap between the different platforms. We look into Common Node's internals, compare its performance to that of other implementations such as RingoJS and go through some ideal use cases.
Behind modern concurrency primitives
The document discusses modern concurrency primitives like threads, thread pools, coroutines, and schedulers. It covers why asynchronous programming with async/await is preferred over traditional threading. It also discusses challenges like sharing data across threads and blocking on I/O calls. Some solutions covered include using thread pools with dedicated I/O threads, work stealing, and introducing interruption points in long-running tasks.
Behind modern concurrency primitives
During this talk we'll cover the theory and practical implementation behind most common patterns in modern multi-threaded programming. How our everyday libraries and frameworks optimize use of operating system resources for maximum efficiency. We'll also try to understand differences between various approaches and what tradeoffs do they infer. Finally we'll take a look at how they are supported by various compilers and runtimes.
Binary Studio Academy: Concurrency in C# 5.0
Lecture for Binary Studio Academy (.NET group) by Max Goncharuk (.NET developer at Binary Studio)
academy.binary-studio.com
Stackless Python In Eve
Stackless Python is used extensively in EVE Online to provide cooperative multitasking. It allows tasks to be split across multiple tasklets that run concurrently without preemption. Channels are used to synchronize tasklets in a way similar to coroutines. This approach allows EVE's massive multiplayer server to handle over 120,000 concurrent users on a single database shard using Stackless Python's lightweight tasklets and channels for communication and synchronization.
Writing robust Node.js applications
This document provides an introduction and overview of a Node.js tutorial presented by Tom Hughes-Croucher. The tutorial covers topics such as building scalable server-side code with JavaScript using Node.js, debugging Node.js applications, using frameworks like Express.js, and best practices for deploying Node.js applications in production environments. The tutorial includes exercises for hands-on learning and demonstrates tools and techniques like Socket.io, clustering, error handling and using Redis with Node.js applications.
[OpenStack Days Korea 2016] Track2 - OpenStack 기반 소프트웨어 정의 스토리지 기술
This document discusses software defined storage based on OpenStack. It provides background on the author's experience including medical image processing, Linux kernel development, and OpenStack components like Heat, SDN and OPNFV. It then discusses several OpenStack storage components - Cinder for block storage, Swift for object storage, and Manila for shared file systems. It explains how these components work, their APIs and plugins to interface with different backend storage systems. Finally, it compares Cinder, Swift and other technologies like Ceph.
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
This document summarizes key aspects of running Spark Streaming applications in production, including fault tolerance, performance, and monitoring. It discusses how Spark Streaming receives data streams in batches and processes them across executors. It describes how driver and executor failures can be handled through checkpointing saved DAG information and write ahead logs that replicate received data blocks. Restarting the driver from checkpoints allows recovering the application state.
Storage based on_openstack_mariocho
The document discusses software defined storage based on OpenStack. It provides background on the author's experience including medical image processing and OpenStack development. It then describes key OpenStack storage components including Cinder for block storage, Swift for object storage, and Manila for shared file systems. Cinder uses plugins to support different backend storage types and utilizes a scheduler to determine which host to provision volumes. Swift uses a ring hashing algorithm to partition and replicate data across multiple storage nodes for high scalability and availability.
NodeJSnodesforfreeinmyworldgipsnndnnd.pdf
This document contains lecture notes on Node.js. It discusses how Node.js uses an event-driven, non-blocking I/O model with callbacks instead of threads. It implements this using an event loop that processes callbacks from an event queue. Standard modules like the file system module wrap synchronous OS calls to utilize asynchronous callbacks. Node.js also includes a Buffer class for efficient binary data handling and a module system for reusable code packages.
Introduction to Chainer 11 may,2018
Published on 11 may, 2018
Chainer is a deep learning framework which is flexible, intuitive, and powerful.
This slide introduces some unique features of Chainer and its additional packages such as ChainerMN (distributed learning), ChainerCV (computer vision), ChainerRL (reinforcement learning), Chainer Chemistry (biology and chemistry), and ChainerUI (visualization).
Stackless Python In Eve
1) Stackless Python is used extensively in EVE Online to support over 130,000 active players and 24,000 concurrent users on a single server cluster.
2) Stackless Python enables lightweight tasklets that can be scheduled cooperatively to improve concurrency without the overhead of OS threads.
3) Channels provide a synchronization mechanism that allows tasklets to communicate and block waiting for data in a way that enables efficient tasklet switching.
Kubernetes for the PHP developer
Kubernetes is exploding in popularity right now and has all the buzz and cargo-culting that Docker enjoyed just a few years ago. But what even is Kubernetes? How do I run my PHP apps in it? Should I run my PHP apps in it ?
RESTful API In Node Js using Express
This document discusses creating REST APIs with Express, Node.js, and MySQL. It provides an overview of Express and its advantages. It then demonstrates how to set up a Node.js and MySQL environment, create an Express server, and implement API routes to GET, POST, DELETE, and PUT data from a MySQL database table. Code examples are provided to retrieve all todos, a single todo by ID, search todos by keyword, add a new todo, delete a todo, and update an existing todo.
An intro to Docker, Terraform, and Amazon ECS
This talk is a very quick intro to Docker, Terraform, and Amazon's EC2 Container Service (ECS). In just 15 minutes, you'll see how to take two apps (a Rails frontend and a Sinatra backend), package them as Docker containers, run them using Amazon ECS, and to define all of the infrastructure-as-code using Terraform.
Future Decoded - Node.js per sviluppatori .NET
"As an asynchronous event driven JavaScript runtime, Node is designed to build scalable network applications" così si presenta Node.js, piattaforma tecnologica che - grazie alla sua immediatezza e produttività - ha conquistato dapprima startup e piccole aziende, fino a ritagliarsi uno spazio importante in realtà come IBM, LinkedIn, Netflix e Yahoo. La stessa Microsoft ha riconosciuto le potenzialità della piattaforma, tanto da integrare Node.js in Visual Studio Code e nelle ultime release di Visual Studio, oltre a basarci alcuni dei propri servizi di Azure come "Mobile Services" e "Functions".
In questa sessione vedremo come implementare con Node.js alcuni scenari applicativi comuni nell’ambito dello sviluppo web, analizzando quando la sua adozione può portarci vantaggi nel nostro lavoro quotidiano. In conclusione, faremo una breve panoramica architetturale, descrivendo alcuni scenari di cooperazione tra .NET e Node.js nello stesso sistema.
Codice e demo: https://github.com/rucka/CommunityDays2016
Threaded Programming
The document discusses intra-machine parallelism and threaded programming. It introduces key concepts like threads, processes, synchronization constructs (locks and condition variables), and challenges like overhead and Amdahl's law. An example of domain decomposition for parallel rendering is presented to demonstrate how to divide a problem into independent tasks and assign them to threads.
Similar to Seastar @ SF/BA C++UG (20)
Programming Sideways: Asynchronous Techniques for Android
Programming Sideways: Asynchronous Techniques for Android
[OpenStack Days Korea 2016] Track2 - OpenStack 기반 소프트웨어 정의 스토리지 기술
[OpenStack Days Korea 2016] Track2 - OpenStack 기반 소프트웨어 정의 스토리지 기술
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Recipes for Running Spark Streaming Applications in Production-(Tathagata Das...
Recently uploaded
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Pitangent Analytics & Technology Solutions Pvt. Ltd
Discover top-tier mobile app development services, offering innovative solutions for iOS and Android. Enhance your business with custom, user-friendly mobile applications.Skybuffer SAM4U tool for SAP license adoption
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
How we managed to grow and scale a RAG application from zero to thousands of users in 7 months. Lessons from technical challenges around managing high load for LLMs, RAGs and Vector databases.
AppSec PNW: Android and iOS Application Security with MobSF
Mobile Security Framework - MobSF is a free and open source automated mobile application security testing environment designed to help security engineers, researchers, developers, and penetration testers to identify security vulnerabilities, malicious behaviours and privacy concerns in mobile applications using static and dynamic analysis. It supports all the popular mobile application binaries and source code formats built for Android and iOS devices. In addition to automated security assessment, it also offers an interactive testing environment to build and execute scenario based test/fuzz cases against the application.
This talk covers:
Using MobSF for static analysis of mobile applications.
Interactive dynamic security assessment of Android and iOS applications.
Solving Mobile app CTF challenges.
Reverse engineering and runtime analysis of Mobile malware.
How to shift left and integrate MobSF/mobsfscan SAST and DAST in your build pipeline.
Choosing The Best AWS Service For Your Website + API.pptx
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Session 1 - Intro to Robotic Process Automation.pdf
👉 Check out our full 'Africa Series - Automation Student Developers (EN)' page to register for the full program:
https://bit.ly/Automation_Student_Kickstart
In this session, we shall introduce you to the world of automation, the UiPath Platform, and guide you on how to install and setup UiPath Studio on your Windows PC.
📕 Detailed agenda:
What is RPA? Benefits of RPA?
RPA Applications
The UiPath End-to-End Automation Platform
UiPath Studio CE Installation and Setup
💻 Extra training through UiPath Academy:
Introduction to Automation
UiPath Business Automation Platform
Explore automation development with UiPath Studio
👉 Register here for our upcoming Session 2 on June 20: Introduction to UiPath Studio Fundamentals: https://community.uipath.com/events/details/uipath-lagos-presents-session-2-introduction-to-uipath-studio-fundamentals/
Astute Business Solutions | Oracle Cloud Partner |
Your goto partner for Oracle Cloud, PeopleSoft, E-Business Suite, and Ellucian Banner. We are a firm specialized in managed services and consulting.
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Direct losses from downtime in 1 minute = $5-$10 thousand dollars. Reputation is priceless.
As part of the talk, we will consider the architectural strategies necessary for the development of highly loaded fintech solutions. We will focus on using queues and streaming to efficiently work and manage large amounts of data in real-time and to minimize latency.
We will focus special attention on the architectural patterns used in the design of the fintech system, microservices and event-driven architecture, which ensure scalability, fault tolerance, and consistency of the entire system.
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
Dandelion Hashtable: beyond billion requests per second on a commodity server
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Must Know Postgres Extension for DBA and Developer during Migration
Mydbops Opensource Database Meetup 16
Topic: Must-Know PostgreSQL Extensions for Developers and DBAs During Migration
Speaker: Deepak Mahto, Founder of DataCloudGaze Consulting
Date & Time: 8th June | 10 AM - 1 PM IST
Venue: Bangalore International Centre, Bangalore
Abstract: Discover how PostgreSQL extensions can be your secret weapon! This talk explores how key extensions enhance database capabilities and streamline the migration process for users moving from other relational databases like Oracle.
Key Takeaways:
* Learn about crucial extensions like oracle_fdw, pgtt, and pg_audit that ease migration complexities.
* Gain valuable strategies for implementing these extensions in PostgreSQL to achieve license freedom.
* Discover how these key extensions can empower both developers and DBAs during the migration process.
* Don't miss this chance to gain practical knowledge from an industry expert and stay updated on the latest open-source database trends.
Mydbops Managed Services specializes in taking the pain out of database management while optimizing performance. Since 2015, we have been providing top-notch support and assistance for the top three open-source databases: MySQL, MongoDB, and PostgreSQL.
Our team offers a wide range of services, including assistance, support, consulting, 24/7 operations, and expertise in all relevant technologies. We help organizations improve their database's performance, scalability, efficiency, and availability.
Contact us: info@mydbops.com
Visit: https://www.mydbops.com/
Follow us on LinkedIn: https://in.linkedin.com/company/mydbops
For more details and updates, please follow up the below links.
Meetup Page : https://www.meetup.com/mydbops-databa...
Twitter: https://twitter.com/mydbopsofficial
Blogs: https://www.mydbops.com/blog/
Facebook(Meta): https://www.facebook.com/mydbops/
5th LF Energy Power Grid Model Meet-up Slides
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Apps Break Data
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Tomaz Bratanic
Graph ML and GenAI Expert - Neo4j
Recently uploaded (20)
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Dandelion Hashtable: beyond billion requests per second on a commodity server
Dandelion Hashtable: beyond billion requests per second on a commodity server
Must Know Postgres Extension for DBA and Developer during Migration
Must Know Postgres Extension for DBA and Developer during Migration
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
GraphRAG for LifeSciences Hands-On with the Clinical Knowledge Graph
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Seastar @ SF/BA C++UG
- 2. SEASTAR @ SF/BA C++UG Avi Kivity (@AviKivity) April 2016
- 3. Seastar: A C++ Asynchronous Programming Framework
- 6. Multi-domain async programming Async networking Async storage I/O Async communications for multi-core, NUMA
- 7. RESULTS
- 8. RESULTS (2)
- 9. THREADING MODELS Before: Thread model After: SeaStar shards
- 11. Seastar model Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace Application TCP/IP Task Scheduler queuequeuequeuequeuequeuesmp queue NIC Queue DPDK Kernel (isn’t involved) Userspace
- 12. Dual networking stacks Networking API Seastar (native) Stack POSIX (hosted) stack Linux kernel (sockets) User-space TCP/IP Interface layer DPDK Virtio Xen igb ixgb
- 13. Seastar model summary ● Each logical core runs a shared-nothing run-to-completion task scheduler ● Logical cores connected by point-to-point queues ● Explicit core-to-core communication ● Shard owns data ● Composable Multicore/Storage/Network APIs ● Optional userspace TCP/IP stack
- 14. CODING IT: Futures and promises
- 15. BASIC MODEL ● Futures ● Promises ● Continuations
- 16. F-P-C Defined: Future A future is a result of a computation that may not be available yet. ● Data buffer from the network ● Timer expiration ● Completion of a disk write ● Computation on another core ● Result computation that requires the values from one or more other futures.
- 17. F-P-C Defined: Promise A promise is an object or function that provides you with a future, with the expectation that it will fulfil the future.
- 18. F-P-C Defined: Continuation A continuation is a computation that is executed when a future becomes ready (yielding a new future).
- 19. Basic Future/Promise future<int> get(); // promises an int will be produced eventually future<> put(int) // promises to store an int furure<> f() { return get().then([] (int value) { return put(value + 1).then([] { std::cout << "value stored successfullyn"; }); }); }
- 20. Parallelism void f() { std::cout << "Sleeping... " << std::flush; using namespace std::chrono_literals; sleep(200ms).then([] { std::cout << "200ms " << std::flush; }); sleep(100ms).then([] { std::cout << "100ms " << std::flush; }); sleep(1s).then([] { std::cout << "Done.n"; engine_exit(); }); }
- 21. Zero-copy future<temporary_buffer<char>> connected_socket::read(size_t n); temporary_buffer points at driver-provided pages if possible discarded after use
- 22. Rich APIs ● HTTP Server ● HTTP Client ● RPC client/server ● map_reduce ● parallel_for_each ● iostreams ● iosched ● threads! ● sharded<> ● when_all() ● timers ● sleep ● semaphore ● gate ● pipe/queue ● Memory reclaimer
- 23. USE CASES
- 24. Applicability ● High I/O to compute ratio ● High concurrency ● Mix of disk and network I/O ● Complex loads ● Cluster (sharded) applications
- 25. Applicability ● Distributed databases ● Object stores, file systems ● Complex proxies/caches
- 27. Thank you.