Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
The biggest headine at the 2009 Oracle OpenWorld was when Larry Ellison announced that Oracle was entering the hardware business with a pre-built database machine, engineered by Oracle. Since then businesses around the world have started to use these engineered systems. This beginner/intermediate-level session will take you through my first 100 days of starting to administer an Exadata machine and all the roadblocks and all the success I had along this new path.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
The biggest headine at the 2009 Oracle OpenWorld was when Larry Ellison announced that Oracle was entering the hardware business with a pre-built database machine, engineered by Oracle. Since then businesses around the world have started to use these engineered systems. This beginner/intermediate-level session will take you through my first 100 days of starting to administer an Exadata machine and all the roadblocks and all the success I had along this new path.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...Severalnines
This presentation by Krzysztof Książek at Percona Live 2017 in Santa Clara, California gives detailed descriptions and comparisons of the leading open source database load balancing technologies
PerconaLive 2016 Santa Clara presentation on Hashicorp Vault with CTO Armon Dadger
https://www.percona.com/live/data-performance-conference-2016/sessions/using-vault-decouple-secrets-applications
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...confluent
Apache Kafka is changing the way we build scalable and highly available software systems. Providing a simplified path to eventual consistency and event sourcing Kafka gives us the platform to make these patterns a reality for a much broader segment of applications and customers than was possible in the past. Cloud Events is an interoperable specification for eventing that is part of the CNCF. This session will combine open source and open standards to show you how you can build highly reliable application that scale linearly, provide interoperability and are easily extensible leveraging both push and pull semantics. Concrete real world examples will be shown of how Kafka makes event sourcing more approachable and how streams and events complement each other including the difference between business events and technical events.
Understanding my database through SQL*Plus using the free tool eDB360Carlos Sierra
This session introduces eDB360 - a free tool that is executed from SQL*Plus and generates a set of reports providing a 360-degree view of an Oracle database; all without installing anything on the database.
If using Oracle Enterprise Manager (OEM) is off-limits for you or your team, and you can only access the database thorough a SQL*Plus connection with no direct access to the database server, then this tool is a perfect fit to provide you with a broad overview of the database configuration, performance, top SQL and much more. You only need a SQL*Plus account with read access to the data dictionary, and common Oracle licenses like the Diagnostics or the Tuning Pack.
Typical uses of this eDB360 tool include: databases health-checks, performance assessments, pre or post upgrade verifications, snapshots of the environment for later use, compare between two similar environments, documenting the state of a database when taking ownership of it, etc.
Once you learn how to use eDB360 and get to appreciate its value, you may want to execute this tool on all your databases on a regular basis, so you can keep track of things for long periods of time. This tool is becoming part of a large collection of goodies many DBAs use today.
During this session you will learn the basics about the free eDB360 tool, plus some cool tricks. The target audience is: DBAs, developers and consultants (some managers could also benefit).
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
CDC Stream Processing with Apache FlinkTimo Walther
An instant world requires instant decisions at scale. This includes the ability to digest and react to changes in real-time. Thus, event logs such as Apache Kafka can be found in almost every architecture, while databases and similar systems still provide the foundation. Change Data Capture (CDC) has become popular for propagating changes. Nevertheless, integrating all these systems, which often have slightly different semantics, can be a challenge.
In this talk, we highlight what it means for Apache Flink to be a general data processor that acts as a data integration hub. Looking under the hood, we demonstrate Flink's SQL engine as a changelog processor that ships with an ecosystem tailored to processing CDC data and maintaining materialized views. We will discuss the semantics of different data sources and how to perform joins or stream enrichment between them. This talk illustrates how Flink can be used with systems such as Kafka (for upsert logging), Debezium, JDBC, and others.
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
Data Orchestration Summit
www.alluxio.io/data-orchestration-summit-2019
November 7, 2019
Apache Iceberg - A Table Format for Hige Analytic Datasets
Speaker:
Ryan Blue, Netflix
For more Alluxio events: https://www.alluxio.io/events/
The Parquet Format and Performance Optimization OpportunitiesDatabricks
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Linux Kernel vs DPDK: HTTP Performance ShowdownScyllaDB
In this session I will use a simple HTTP benchmark to compare the performance of the Linux kernel networking stack with userspace networking powered by DPDK (kernel-bypass).
It is said that kernel-bypass technologies avoid the kernel because it is "slow", but in reality, a lot of the performance advantages that they bring just come from enforcing certain constraints.
As it turns out, many of these constraints can be enforced without bypassing the kernel. If the system is tuned just right, one can achieve performance that approaches kernel-bypass speeds, while still benefiting from the kernel's battle-tested compatibility, and rich ecosystem of tools.
SOSCON 2019.10.17
What are the methods for packet processing on Linux? And how fast are each packet processing methods? In this presentation, we will learn how to handle packets on Linux (User space, socket filter, netfilter, tc), and compare performance with analysis of where each packet processing is done in the network stack (hook point). Also, we will discuss packet processing using XDP, an in-kernel fast-path recently added to the Linux kernel. eXpress Data Path (XDP) is a high-performance programmable network data-path within the Linux kernel. The XDP is located at the lowest level of access through SW in the network stack, the point at which driver receives the packet. By using the eBPF infrastructure at this hook point, the network stack can be expanded without modifying the kernel.
Daniel T. Lee (Hoyeon Lee)
@danieltimlee
Daniel T. Lee currently works as Software Engineer at Kosslab and contributing to Linux kernel BPF project. He has interest in cloud, Linux networking, and tracing technologies, and likes to analyze the kernel's internal using BPF technology.
Marrying CDNs with Front-End Optimization Strangeloop
Slide deck from Strangeloop president Joshua Bixby's presentation at the 2012 Content Delivery Summit.
Many content owners are already using a content delivery network (CDN) to cache content closer to their visitors, but CDNs don't reduce the number of requests required to render each page, and they have no impact on browser efficiency. Front-end optimization (FEO) picks up where CDNs leave off, transforming the content itself so that it renders as quickly as possible in the browser.
In this presentation, attendees will see real-world examples of how leading e-commerce sites have combined CDN and FEO forces to reach new levels of performance for content-rich pages. Get real numbers on how quickly content-rich sites loaded pre-acceleration, then with just a CDN, then with a combined CDN/FEO solution.
Modern B2B Marketing in the Era of the Empowered BuyerScott Levine
In the past 20 years, B2B marketing has changed exponentially more than it has in the past 100 years. Today, buyers possess infinite choice and infinite power.
We live in an “instant” world. Whether it’s instant access, information, gratification, justification or rationalization, all of these “instants” have impacted the way that businesses who market to businesses, think, act and react.
Modern Business-to-Business Marketing in the Era of the Empowered Buyer is causing many organizations to rethink their strategies. We are hopeful that some of the information we have shared with you here will enable you to understand the current and possible future state of B-to-B marketing, and will help you to put your organization in a position to best deal with complexities caused by the Modern Empowered B-to-B Buyer.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
MySQL Load Balancers - Maxscale, ProxySQL, HAProxy, MySQL Router & nginx - A ...Severalnines
This presentation by Krzysztof Książek at Percona Live 2017 in Santa Clara, California gives detailed descriptions and comparisons of the leading open source database load balancing technologies
PerconaLive 2016 Santa Clara presentation on Hashicorp Vault with CTO Armon Dadger
https://www.percona.com/live/data-performance-conference-2016/sessions/using-vault-decouple-secrets-applications
Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova...confluent
Apache Kafka is changing the way we build scalable and highly available software systems. Providing a simplified path to eventual consistency and event sourcing Kafka gives us the platform to make these patterns a reality for a much broader segment of applications and customers than was possible in the past. Cloud Events is an interoperable specification for eventing that is part of the CNCF. This session will combine open source and open standards to show you how you can build highly reliable application that scale linearly, provide interoperability and are easily extensible leveraging both push and pull semantics. Concrete real world examples will be shown of how Kafka makes event sourcing more approachable and how streams and events complement each other including the difference between business events and technical events.
Understanding my database through SQL*Plus using the free tool eDB360Carlos Sierra
This session introduces eDB360 - a free tool that is executed from SQL*Plus and generates a set of reports providing a 360-degree view of an Oracle database; all without installing anything on the database.
If using Oracle Enterprise Manager (OEM) is off-limits for you or your team, and you can only access the database thorough a SQL*Plus connection with no direct access to the database server, then this tool is a perfect fit to provide you with a broad overview of the database configuration, performance, top SQL and much more. You only need a SQL*Plus account with read access to the data dictionary, and common Oracle licenses like the Diagnostics or the Tuning Pack.
Typical uses of this eDB360 tool include: databases health-checks, performance assessments, pre or post upgrade verifications, snapshots of the environment for later use, compare between two similar environments, documenting the state of a database when taking ownership of it, etc.
Once you learn how to use eDB360 and get to appreciate its value, you may want to execute this tool on all your databases on a regular basis, so you can keep track of things for long periods of time. This tool is becoming part of a large collection of goodies many DBAs use today.
During this session you will learn the basics about the free eDB360 tool, plus some cool tricks. The target audience is: DBAs, developers and consultants (some managers could also benefit).
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
CDC Stream Processing with Apache FlinkTimo Walther
An instant world requires instant decisions at scale. This includes the ability to digest and react to changes in real-time. Thus, event logs such as Apache Kafka can be found in almost every architecture, while databases and similar systems still provide the foundation. Change Data Capture (CDC) has become popular for propagating changes. Nevertheless, integrating all these systems, which often have slightly different semantics, can be a challenge.
In this talk, we highlight what it means for Apache Flink to be a general data processor that acts as a data integration hub. Looking under the hood, we demonstrate Flink's SQL engine as a changelog processor that ships with an ecosystem tailored to processing CDC data and maintaining materialized views. We will discuss the semantics of different data sources and how to perform joins or stream enrichment between them. This talk illustrates how Flink can be used with systems such as Kafka (for upsert logging), Debezium, JDBC, and others.
Apache Iceberg - A Table Format for Hige Analytic DatasetsAlluxio, Inc.
Data Orchestration Summit
www.alluxio.io/data-orchestration-summit-2019
November 7, 2019
Apache Iceberg - A Table Format for Hige Analytic Datasets
Speaker:
Ryan Blue, Netflix
For more Alluxio events: https://www.alluxio.io/events/
The Parquet Format and Performance Optimization OpportunitiesDatabricks
The Parquet format is one of the most widely used columnar storage formats in the Spark ecosystem. Given that I/O is expensive and that the storage layer is the entry point for any query execution, understanding the intricacies of your storage format is important for optimizing your workloads.
As an introduction, we will provide context around the format, covering the basics of structured data formats and the underlying physical data storage model alternatives (row-wise, columnar and hybrid). Given this context, we will dive deeper into specifics of the Parquet format: representation on disk, physical data organization (row-groups, column-chunks and pages) and encoding schemes. Now equipped with sufficient background knowledge, we will discuss several performance optimization opportunities with respect to the format: dictionary encoding, page compression, predicate pushdown (min/max skipping), dictionary filtering and partitioning schemes. We will learn how to combat the evil that is ‘many small files’, and will discuss the open-source Delta Lake format in relation to this and Parquet in general.
This talk serves both as an approachable refresher on columnar storage as well as a guide on how to leverage the Parquet format for speeding up analytical workloads in Spark using tangible tips and tricks.
Linux Kernel vs DPDK: HTTP Performance ShowdownScyllaDB
In this session I will use a simple HTTP benchmark to compare the performance of the Linux kernel networking stack with userspace networking powered by DPDK (kernel-bypass).
It is said that kernel-bypass technologies avoid the kernel because it is "slow", but in reality, a lot of the performance advantages that they bring just come from enforcing certain constraints.
As it turns out, many of these constraints can be enforced without bypassing the kernel. If the system is tuned just right, one can achieve performance that approaches kernel-bypass speeds, while still benefiting from the kernel's battle-tested compatibility, and rich ecosystem of tools.
SOSCON 2019.10.17
What are the methods for packet processing on Linux? And how fast are each packet processing methods? In this presentation, we will learn how to handle packets on Linux (User space, socket filter, netfilter, tc), and compare performance with analysis of where each packet processing is done in the network stack (hook point). Also, we will discuss packet processing using XDP, an in-kernel fast-path recently added to the Linux kernel. eXpress Data Path (XDP) is a high-performance programmable network data-path within the Linux kernel. The XDP is located at the lowest level of access through SW in the network stack, the point at which driver receives the packet. By using the eBPF infrastructure at this hook point, the network stack can be expanded without modifying the kernel.
Daniel T. Lee (Hoyeon Lee)
@danieltimlee
Daniel T. Lee currently works as Software Engineer at Kosslab and contributing to Linux kernel BPF project. He has interest in cloud, Linux networking, and tracing technologies, and likes to analyze the kernel's internal using BPF technology.
Marrying CDNs with Front-End Optimization Strangeloop
Slide deck from Strangeloop president Joshua Bixby's presentation at the 2012 Content Delivery Summit.
Many content owners are already using a content delivery network (CDN) to cache content closer to their visitors, but CDNs don't reduce the number of requests required to render each page, and they have no impact on browser efficiency. Front-end optimization (FEO) picks up where CDNs leave off, transforming the content itself so that it renders as quickly as possible in the browser.
In this presentation, attendees will see real-world examples of how leading e-commerce sites have combined CDN and FEO forces to reach new levels of performance for content-rich pages. Get real numbers on how quickly content-rich sites loaded pre-acceleration, then with just a CDN, then with a combined CDN/FEO solution.
Modern B2B Marketing in the Era of the Empowered BuyerScott Levine
In the past 20 years, B2B marketing has changed exponentially more than it has in the past 100 years. Today, buyers possess infinite choice and infinite power.
We live in an “instant” world. Whether it’s instant access, information, gratification, justification or rationalization, all of these “instants” have impacted the way that businesses who market to businesses, think, act and react.
Modern Business-to-Business Marketing in the Era of the Empowered Buyer is causing many organizations to rethink their strategies. We are hopeful that some of the information we have shared with you here will enable you to understand the current and possible future state of B-to-B marketing, and will help you to put your organization in a position to best deal with complexities caused by the Modern Empowered B-to-B Buyer.
A particle filter based scheme for indoor tracking on an Android SmartphoneDivye Kapoor
A particle filter based scheme for indoor tracking on an Android Smartphone.
These are slides accompanying the Masters thesis of the same name presented as part of the graduating requirements at IIT Roorkee. They detail a sensor fusion based approach taken to indoor tracking on smartphones.
These are the questions asked in the prelims round of Cybermania - a computer quiz held at Loyola School, Jamshedpur, India. The quiz was for students of standard 6 to 10.
If you have any doubts or further questions, contact the quizmaster at Twitter: @divyekapoor or on Google+ at http://gplus.to/divyekapoor
I'll be happy to share the slides with you if you ask me politely on any of these social networks. :)
Kernel Recipes 2015: The stable Linux Kernel Tree - 10 years of insanityAnne Nicolas
The Linux kernel gets a stable release about once every week.
This talk will go into the process of getting a patch accepted into the stable releases, how the release process works, and how Greg does a review and release cycle. It will consist of live examples of patches submitted to be added to the stable releases, as well as doing a release “live” on stage.
Greg KH, Linux Foundation
These are the questions asked in the main round of Cybermania - a computer quiz held at Loyola School, Jamshedpur, India. The quiz was for students of standard 6 to 10.
If you have any doubts or further questions, contact the quizmaster at Twitter: @divyekapoor or on Google+ at http://gplus.to/divyekapoor
I'll be happy to share the slides with you if you ask me politely on any of these social networks. :)
The Linux Kernel Implementation of Pipes and FIFOsDivye Kapoor
A walkthrough of the code structure used in the linux kernel to implement pipes and FIFOs.
This was presented to a Senior level class at the Indian Institute of Technology, Roorkee.
Part 04 Creating a System Call in LinuxTushar B Kute
Presentation on "System Call creation in Linux".
Presented at Army Institute of Technology, Pune for FDP on "Basics of Linux Kernel Programming". by Tushar B Kute (http://tusharkute.com).
(WEB401) Optimizing Your Web Server on AWS | AWS re:Invent 2014Amazon Web Services
Tuning your EC2 web server will help you to improve application server throughput and cost-efficiency as well as reduce request latency. In this session we will walk through tactics to identify bottlenecks using tools such as CloudWatch in order to drive the appropriate allocation of EC2 and EBS resources. In addition, we will also be reviewing some performance optimizations and best practices for popular web servers such as Nginx and Apache in order to take advantage of the latest EC2 capabilities.
Denial of Service Mitigation Tactics in FreeBSDSteven Kreuzer
Protecting your servers, workstations and networks can only go so far. Attacks which consume your available Internet-facing bandwidth, or overpower your CPU, can still take you offline. His presentation will discuss techniques for mitigating the effects of such attacks on servers designed to provide network intensive services such as HTTP or routing.
1 Million Writes per second on 60 nodes with Cassandra and EBSJim Plush
EBS has long been taboo in the Cassandra world for high performance workloads. That line of thinking has started to change with the introduction of EBS GP2 and the recent stability improvements made by the EBS team, which is why we have multiple PetaBytes of data relying on EBS every day. Running Cassandra on EBS will now let you run denser, cheaper Cassandra clusters with just as much availability as ephemeral storage instances. This talk will walk through a highly detailed use case and configuration guide for a multi PetaByte, million write per second cluster that needs to be highly performant and cost efficient. We will dive into the instance type choices, configuration and low level tuning that allowed us to hit 1.3 million writes per second with a replication factor of 3 on just 60 nodes. We will go into the details of why we chose to use the latest DateTieredCompactionStrategy and why that's a perfect fit for high volume time series workloads.
Many applications are network I/O bound, including common database-based applications and service-based architectures. But operating systems and applications are often untuned to deliver high performance. This session uncovers hidden issues that lead to low network performance, and shows you how to overcome them to obtain the best network performance possible.
HBaseCon2017 gohbase: Pure Go HBase ClientHBaseCon
gohbase is an implementation of an HBase client in pure Go: https://github.com/tsuna/gohbase. In this presentation we'll talk about its architecture and compare its performance against the native Java HBase client as well as AsyncHBase (http://opentsdb.github.io/asynchbase/) and some nice characteristics of golang that resulted in a simpler implementation.
Flink Forward Berlin 2017: Robert Metzger - Keep it going - How to reliably a...Flink Forward
Let’s be honest: Running a distributed stateful stream processor that is able to handle terabytes of state and tens of gigabytes of data per second while being highly available and correct (in an exactly-once sense) does not work without any planning, configuration and monitoring. While the Flink developer community tries to make everything as simple as possible, it is still important to be aware of all the requirements and implications In this talk, we will provide some insights into the greatest operations mysteries of Flink from a high-level perspective: - Capacity and resource planning: Understand the theoretical limits. - Memory and CPU configuration: Distribute resources according to your needs. - Setting up High Availability: Planning for failures. - Checkpointing and State Backends: Ensure correctness and fast recovery For each of the listed topics, we will introduce the concepts of Flink and provide some best practices we have learned over the past years supporting Flink users in production.
Many applications are network I/O bound, including common database-based applications and service-based architectures. But operating systems and applications are often untuned to deliver high performance. This session uncovers hidden issues that lead to low network performance, and shows you how to overcome them to obtain the best network performance possible.
In-memory processing has started to become the norm in large scale data handling. This is aclose to the metal analysis of highly important but often neglected aspects of memory accesstimes and how it impacts big data and NoSQL technologies.We cover aspects such as the TLB, the Transparent Huge Pages, the QPI Link, Hyperthreading and the impact of virtualization on high-memory footprint applications. We present benchmarks of various technologies ranging from Cloudera’s Impala to Couchbase and how they are impacted by the underlying hardware.The key takeaway is a better understanding of how to size a cluster, how to choose a cloud provider and an instance type for big data and NoSQL workloads and why not every core or GB of RAM is created equal.
We start by looking at distributed database features that impact latency. Then we take a deeper look at the HBase read and write paths with a focus on request latency. We examine the sources of latency and how to minimize them.
DPDK Summit 2015 - Aspera - Charles ShiflettJim St. Leger
DPDK Summit 2015 in San Francisco.
Presentation by Charles Shiflett, Aspera.
For additional details and the video recording please visit www.dpdksummit.com.
Scylla Summit 2016: Outbrain Case Study - Lowering Latency While Doing 20X IO...ScyllaDB
Outbrain is the world's largest content discovery program. Learn about their use case with Scylla where they lowered latency while doing 20X IOPS of Cassandra.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
Meet up Milano 14 _ Axpo Italia_ Migration from Mule3 (On-prem) to.pdfFlorence Consulting
Quattordicesimo Meetup di Milano, tenutosi a Milano il 23 Maggio 2024 dalle ore 17:00 alle ore 18:30 in presenza e da remoto.
Abbiamo parlato di come Axpo Italia S.p.A. ha ridotto il technical debt migrando le proprie APIs da Mule 3.9 a Mule 4.4 passando anche da on-premises a CloudHub 1.0.
Ready to Unlock the Power of Blockchain!Toptal Tech
Imagine a world where data flows freely, yet remains secure. A world where trust is built into the fabric of every transaction. This is the promise of blockchain, a revolutionary technology poised to reshape our digital landscape.
Toptal Tech is at the forefront of this innovation, connecting you with the brightest minds in blockchain development. Together, we can unlock the potential of this transformative technology, building a future of transparency, security, and endless possibilities.
3. Varnish Software
• Contributes a lot to the Varnish Cache project
• Not the Varnish Cache project
• Support and ad-on software for Varnish Cache
• Media, e-commerce, API and CDN workloads
4. What is this Varnish?
Client BackendVarnish
TTBF: 30 microseconds TTBF: 150 milliseconds
5. Varnish Cache: 30s primer
• High performance HTTP Caching reverse proxy
• 10 years old
• Policy-driven configuration language
• Massively threaded - event driven programming is a fad :-P
• Super easy to write modules (no event loop, see)
6. VCL Example
sub vcl_recv {
if (req.http.host == "www.example.com" &&
req.url ~ "^/fun/" &&
(req.http.referer && req.http.referer !~ "^http://www.example.com/")) {
return (synth(403, "No hotlinking please”));
}
}
7. So? What is Varnish?
Client BackendVarnish
Run high speed logic here.
10. What to tune
• Linux IP stack & Netfilter
• Linux ethernet - we’ll skip this for now. Most of you don’t have
ethernet interfaces anymore. :-)
• Varnish Cache
21. A suitable backend
• https://github.com/espebra/dummy-api
• Perfect for ad hoc testing
• Object size, latencies (ttfb, ttb) are all dynamic (from URL)
• Really fast (100K+ RPS)
• http://target:1337/?header-delay=50&body-delay=100&predictable-
content=10
27. Calculating BDP
• Max Bandwidth per flow x Delay
• 1000 Mbps x 0.1 seconds = 100megabits = 12megabytes
• Default: ~3.7 megabytes - 330 megabits @ 100ms latency
28. BDP Tuning
• Kernel autotunes the details - we just give it more room
• /proc/sys/net/core/(r|w)mem_max can be ignored
• /proc/sys/net/ipv4/tcp_(r|w)mem should be lifted -
• 10240 87380 16777216 is the usual recommendation
32. Playing with initcwnd
• Initial congestion window is now 10
• Increasing might break stuff
• Some CDNs increase initcwnd and show some improvement
33. accept()
• System call used by an application to accept a socket from the
kernel
• Multiple threads in Varnish issue accept() calls - one per thread pool
34. somaxconn
• Global limit on listen_depth
• Default is silly (128)
• Adds 3s/1s delay to incoming connections (initial syn gets
discarded)
• Increase it to 1 - 16K
35. tcp_max_syn_backlog
• Threshold for SYN Flood detection
• Limits number of TCP connection being established
• When exhausted - SYN Cookies are sent
• Do not rely on SYN Cookies
36. Local TCP ports
• Varnish will need local sockets in order to talk to backends
• Busy servers might run low on sockets
• Default: net.ipv4.ip_local_port_range = 32768 61000
• Can safely be increased to “2000 65500”
37. TIME_WAIT
• Socket is kept around after it is closed
• Linux used 2x FIN timeout
• Default is 60 seconds (no packet should be older than 60s)
• I’ve never seen a packet older than 10s
• net.ipv4.tcp_fin_timeout can be set to 10
38. More TIME_WAIT
• tcp_tw_recycle is dangerous (unbuckles seat belt)
• tcp_tw_reuse can cause problems with uses behind NAT - makes
sense on LAN w.o./NAT
• tcp_max_tw_buckets can mitigate TIME_WAIT attacks by destroying
sockets in TIME_WAIT state
• Increase tcp_max_tw_buckets to 256K or more
39. Connection tracking
• Linux firewall tracks connections
• Loaded implicitly when using certain iptables rules
• [11864.342438] nf_conntrack version 0.5.0 (3917 buckets, 15668 max)
• New connection are rejected when conntrack is full
• Set parameters when loading module (options nf_conntrack
hashsize=XXXXX) and
40. Linux tuning - summing up
• Leave most things as they are
• Increase somaxconn, tcp_max_backlog
• Increase local_port range
• Decrease tcp_fin_timeout to ~10
• Increase tcp_max_tw_buckets to ~256K
• Increase BDP buffer limit
42. Varnish Cache threads
• Number of pool: always 2
• thread_pool_max
• thread_pool_min
• You need ~ 1 thread per RPS
43. Workspace Tuning
• Varnish pre-allocates memory for the threads
• When it runs out of memory - it crashes
44. VSL Tuning
• /var/lib/varnish contains the VSL.
• Linux will try to sync the VSL to disk
• On busy servers: put VSL on RAMDISK
45. Keepalives
• 3 way handshake on long latency is expensive
• TLS handshake is worse
• idle_send_timeout (frontend) and backend_idle_timeout (backend)
47. Increasing cache hit rate
• Prolong TTLs - invalidate on change
• Normalize request headers when using Vary
48. Summing up: Varnish Cache
• Threads are in pools (you need two)
• Make sure there is enough threads
• Make sure there is enough memory
• Try to tune your cache hit ratio
49. Preemptive answers
• TLS is not in Varnish Cache due to OpenSSL QA issues
• H/2 support is experimental in Varnish Cache 5.0
• Full H/2 support in Varnish Cache 5.1 (with Hitch)