We're going to talk whether it's worth to migrate from ASP.NET Core 2.2 to 3.0 in terms of performance. In addition, we're raising topics about high-performance features like pipelines, span, and memory which are used in ASP.NET 3.0 by Microsoft to speed up the processing of requests.
go-git is a 100% Go libray used to interact with git repositories. Even if it already supports most of the functionality it still lags a bit in performance when compared with the git CLI or some other libraries. I'll explain some of the problems that we face when dealing with git repos and some examples of performance improvements done to the library.
CostFed: Cost-Based Query Optimization for SPARQL Endpoint FederationMuhammad Saleem
CostFed is a cost-based query optimization engine for federated SPARQL queries. It uses a novel index that buckets resources by cardinality and prefixes to enable join-aware source selection. CostFed estimates triple pattern and join cardinalities more accurately than existing systems by considering skewed distributions, multi-valued predicates, and average bucket selectivities. An evaluation shows CostFed outperforms state-of-the-art systems by selecting fewer sources and generating query plans that are up to 121 times faster on benchmark queries.
This document summarizes scaling Redis to 1 million operations per second. It discusses clustering and sharding options like partitioning the data across multiple Redis servers. It also covers patterns and anti-patterns when scaling, like using pipelines and avoiding running slow keys operations. Benchmarking tools like memtier-benchmark and redis-benchmark are presented. A demo is included and resources like documentation, modules, and training from Redis Labs are provided.
This document provides instructions for installing the PyGrib library within the CDAT path on Ubuntu. It describes downloading required packages like grib_api and pygrib, moving them to the CDAT path, and running the install scripts. It also gives examples of importing and using pygrib within Python to open GRIB files, extract data values and metadata, and write modified GRIB messages to a new file.
A compact bytecode format for JavaScriptCoreTadeu Zagallo
JavaScriptCore (JSC) is the multi-tiered JavaScript virtual machine in WebKit. The bytecode is a central piece in JSC: it’s executed by the interpreter and the source of truth for all of JSC’s compilers. In this talk we’ll look at the recent redesign of our bytecode format, which cut its size in half and enabled persisting the bytecode on disk without impacting the overall performance of the system.
There have been plenty of “explaining EXPLAIN” type talks over the years, which provide a great introduction to it. They often also cover how to identify a few of the more common issues through it. EXPLAIN is a deep topic though, and to do a good introduction talk, you have to skip over a lot of the tricky bits. As such, this talk will not be a good introduction to EXPLAIN, but instead a deeper dive into some of the things most don’t cover. The idea is to start with some of the more complex and unintuitive calculations needed to work out the relationships between operations, rows, threads, loops, timings, buffers, CTEs and subplans. Most popular tools handle at least several of these well, but there are cases where they don’t that are worth being conscious of and alert to. For example, we’ll have a look at whether certain numbers are averaged per-loop or per-thread, or both. We’ll also cover a resulting rounding issue or two to be on the lookout for. Finally, some per-operation timing quirks are worth looking out for where CTEs and subqueries are concerned, for example CTEs that are referenced more than once. As time allows, we can also look at a few rarer issues that can be spotted via EXPLAIN, as well as a few more gotchas that we’ve picked up along the way. This includes things like spotting when the query is JIT, planning, or trigger time dominated, spotting the signs of table and index bloat, issues like lossy bitmap scans or index-only scans fetching from the heap, as well as some things to be aware of when using auto_explain.
This document discusses M3, Uber's time series database. It provides an overview of M3 and compares it to Graphite, which Uber previously used. M3 was built to have better resiliency, efficiency, and scalability than Graphite. It provides both a Graphite-compatible query interface and its own query language called M3QL. The document describes M3's architecture, storage, indexing, and how it handles high write and read throughput. It also covers instrumentation, profiling, load testing, and optimizations used in M3's Go code.
This document provides a summary of ntpd configuration options for unicast and broadcast/multicast modes. It describes options for configuring time servers, peers, and polling intervals. It also outlines commands for ntpq, generating statistics, and the format of statistics files.
go-git is a 100% Go libray used to interact with git repositories. Even if it already supports most of the functionality it still lags a bit in performance when compared with the git CLI or some other libraries. I'll explain some of the problems that we face when dealing with git repos and some examples of performance improvements done to the library.
CostFed: Cost-Based Query Optimization for SPARQL Endpoint FederationMuhammad Saleem
CostFed is a cost-based query optimization engine for federated SPARQL queries. It uses a novel index that buckets resources by cardinality and prefixes to enable join-aware source selection. CostFed estimates triple pattern and join cardinalities more accurately than existing systems by considering skewed distributions, multi-valued predicates, and average bucket selectivities. An evaluation shows CostFed outperforms state-of-the-art systems by selecting fewer sources and generating query plans that are up to 121 times faster on benchmark queries.
This document summarizes scaling Redis to 1 million operations per second. It discusses clustering and sharding options like partitioning the data across multiple Redis servers. It also covers patterns and anti-patterns when scaling, like using pipelines and avoiding running slow keys operations. Benchmarking tools like memtier-benchmark and redis-benchmark are presented. A demo is included and resources like documentation, modules, and training from Redis Labs are provided.
This document provides instructions for installing the PyGrib library within the CDAT path on Ubuntu. It describes downloading required packages like grib_api and pygrib, moving them to the CDAT path, and running the install scripts. It also gives examples of importing and using pygrib within Python to open GRIB files, extract data values and metadata, and write modified GRIB messages to a new file.
A compact bytecode format for JavaScriptCoreTadeu Zagallo
JavaScriptCore (JSC) is the multi-tiered JavaScript virtual machine in WebKit. The bytecode is a central piece in JSC: it’s executed by the interpreter and the source of truth for all of JSC’s compilers. In this talk we’ll look at the recent redesign of our bytecode format, which cut its size in half and enabled persisting the bytecode on disk without impacting the overall performance of the system.
There have been plenty of “explaining EXPLAIN” type talks over the years, which provide a great introduction to it. They often also cover how to identify a few of the more common issues through it. EXPLAIN is a deep topic though, and to do a good introduction talk, you have to skip over a lot of the tricky bits. As such, this talk will not be a good introduction to EXPLAIN, but instead a deeper dive into some of the things most don’t cover. The idea is to start with some of the more complex and unintuitive calculations needed to work out the relationships between operations, rows, threads, loops, timings, buffers, CTEs and subplans. Most popular tools handle at least several of these well, but there are cases where they don’t that are worth being conscious of and alert to. For example, we’ll have a look at whether certain numbers are averaged per-loop or per-thread, or both. We’ll also cover a resulting rounding issue or two to be on the lookout for. Finally, some per-operation timing quirks are worth looking out for where CTEs and subqueries are concerned, for example CTEs that are referenced more than once. As time allows, we can also look at a few rarer issues that can be spotted via EXPLAIN, as well as a few more gotchas that we’ve picked up along the way. This includes things like spotting when the query is JIT, planning, or trigger time dominated, spotting the signs of table and index bloat, issues like lossy bitmap scans or index-only scans fetching from the heap, as well as some things to be aware of when using auto_explain.
This document discusses M3, Uber's time series database. It provides an overview of M3 and compares it to Graphite, which Uber previously used. M3 was built to have better resiliency, efficiency, and scalability than Graphite. It provides both a Graphite-compatible query interface and its own query language called M3QL. The document describes M3's architecture, storage, indexing, and how it handles high write and read throughput. It also covers instrumentation, profiling, load testing, and optimizations used in M3's Go code.
This document provides a summary of ntpd configuration options for unicast and broadcast/multicast modes. It describes options for configuring time servers, peers, and polling intervals. It also outlines commands for ntpq, generating statistics, and the format of statistics files.
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...Austin Benson
This document discusses matrix storage and data serialization techniques for scientific computing with Hadoop and Dumbo. It provides examples of storing matrices in HDFS using different approaches like storing each row separately, storing two rows per record, or flattening the matrix into a single list. It also discusses optimizing data serialization and switching programming languages. The document then presents an example of outputting many small matrices to disk and compares two MapReduce implementations for computing the Cholesky QR decomposition, identifying which approach is usually better and why.
Accelerate Reed-Solomon coding for Fault-Tolerance in RAID-like systemShuai Yuan
The document discusses accelerating Reed-Solomon erasure codes on GPUs. It aims to accelerate two main computation bottlenecks: arithmetic operations in Galois fields and matrix multiplication. For Galois field operations, it evaluates loop-based and table-based methods and chooses a log-exponential table approach. It also proposes tiling algorithms to optimize matrix multiplication on GPUs by reducing data transfers and improving memory access patterns. The goal is to make Reed-Solomon encoding and decoding faster for cloud storage systems using erasure codes.
Building a High-Performance Distributed Task Queue on MongoDBMongoDB
This document discusses building a distributed job queue in MongoDB. It begins by outlining the requirements of processing tasks across multiple servers. It then describes the design of storing messages and tasks in MongoDB collections, including different message and task states. It also covers techniques like composite tasks, reducing polling through tailable cursors, and optimizing performance through retry logic and oplog replay. The overall aim is to build an asynchronous and scalable way to process jobs in a distributed manner using MongoDB.
The document discusses several Java and Android internals topics:
1. How ArrayList and StringBuilder work internally using arrays and memory copying as the size increases. This can lead to inefficient memory usage.
2. How inner classes are implemented by compilers by generating additional accessor methods, increasing method count and affecting optimizations.
3. How the Android zygote process improves startup and memory usage by loading the framework once and sharing it across apps.
4. How the CPU cache works and how optimizing code to improve cache locality can significantly increase performance despite doing less work.
5. Issues like memory fragmentation that can occur if the Android garbage collector and compactor are unable to run due to the app being
The document summarizes a student project analyzing relative word frequencies using different algorithms implemented in Hadoop and Spark. It includes pseudocode and Java code for a pairs, stripe, and hybrid approach in Hadoop, as well as results. It also describes using Spark to analyze Apache log files by response code and errors.
Building Scalable, Distributed Job Queues with Redis and Redis::ClientMike Friedman
This document discusses using Redis and the Redis::Client Perl module to build scalable distributed job queues. It provides an overview of Redis, describing it as a key-value store that is simple, fast, and open-source. It then covers the various Redis data types like strings, lists, hashes, sets and sorted sets. Examples are given of how to work with these types using Redis::Client. The document discusses using Redis lists to implement job queues, with jobs added via RPUSH and popped via BLPOP. Benchmark results show the Redis-based job queue approach significantly outperforms using a MySQL jobs table with polling. Some caveats are provided about the benchmarks.
Optimizing with persistent data structures (LLVM Cauldron 2016)Igalia
By Andy Wingo.
Is there life beyond phi variables and basic blocks? Andy will report on his experience using a new intermediate representation for compiler middle-ends, "CPS soup". The CPS soup language represents programs using Clojure-inspired maps, allowing optimizations to be neatly expressed as functions from one graph to another. Using these persistent data structures also means that the source program doesn't change while the residual program is being created, eliminating one class of problems that the optimizer writer has to keep in mind. Together we will look at some example transformations from an expressiveness as well as a performance point of view, and we will also cover some advantages which a traditional SSA graph maintains over CPS soup.
Data correlation using PySpark and HDFSJohn Conley
This document discusses using PySpark and HDFS to correlate different types of data at scale. It describes some challenges with correlating out-of-order and high-volume data. It then summarizes three approaches tried: using Redis, RDD joins, and writing bindings to HDFS. The current recommended approach reads relevant binding buckets from HDFS to correlate records in small windows, supporting different temporal models. Configuration and custom logic can be plugged in at various points in the correlation process. While scalable, further improvements in latency and throughput are still needed.
This document provides examples of common SQL anti-patterns and related NoSQL alternatives. It discusses issues like using tables as trees, caches, queues, or logs. It also addresses dynamic schema/table creation, stored procedures, row padding, complex joins, and object-relational mismatches. The document recommends alternatives like document databases, key-value stores, message brokers, and denormalization. It includes examples of modeling book data in MongoDB and Redis.
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...InfluxData
The document discusses how an easy-to-use and fast database can have a complicated implementation for developers. It outlines four key areas: 1) Flexible writing schema requires schema merging at read time. 2) Fast reads prune non-covered data chunks through predicate push-down. 3) Loading duplicated data necessitates data deduplication and compaction operations. 4) Quick data deletion still needs data elimination at read time or in the background. The document provides examples to illustrate the tradeoffs between user and developer requirements.
Full text search in PostgreSQL is a flexible and powerful facility to search collection of documents using natural language queries. We will discuss several new improvements of FTS in PostgreSQL 9.6 release, such as phrase search, better dictionaries support and tsvector editing functions. Also, we will present new features currently in development - RUM index support, which enables acceleration of some important kinds of full text queries, new and better ranking function for relevance search, loading dictionaries into shared memory and support for search multilingual content.
The document discusses mining the top-K multidimensional gradients from a fact table. It proposes a gradient-based cubing approach that partitions gradient regions based on spreading factors to efficiently retrieve the top-K gradient cells. The approach builds inverted indexes and value-list indices from the base table before pruning non-valid regions and cells. It then calculates spreading factors to create a GR tree and further partition valid gradient regions.
The Ring programming language version 1.9 book - Part 17 of 210Mahmoud Samir Fayed
The document summarizes the new features in Ring 1.4. Key points include:
1. Basic extensions like ODBC, MySQL, SQLite, OpenSSL and Internet libraries are now separate loadable files rather than part of the RingVM.
2. A new Natural Language library allows defining languages and commands to write programs in a more natural way, like English.
3. A new style is added to the Ring Notepad text editor.
4. Functions are introduced to convert between numbers and bytes.
5. The standard library and web library are updated with new functions and better error handling.
The document discusses various techniques for optimizing web program performance, including improving data structures and algorithms, reducing network, disk, and memory I/O, using NoSQL databases, optimizing frontend resource loading and execution, and testing and refactoring code for performance. It also covers specific optimizations related to caching, content delivery networks, domain name resolution, module loading, code packaging, and handling slower users.
This document summarizes garbage collection in the .NET Common Language Runtime (CLR). It covers the basics of generations and segments, how allocation and collection work, and different flavors of GC like concurrent and server GC. It also discusses pinning, tools for analyzing GC, and challenges for 64-bit systems. The goal of generational GC is to improve efficiency by focusing collection on younger object generations where most objects die, in order to minimize pause times during full collections.
Fast Insight from Fast Data: Integrating ClickHouse and Apache KafkaAltinity Ltd
This document discusses using Kafka as a messaging system with ClickHouse for high throughput and low latency data ingestion. It provides an overview of Kafka and how it can be used with ClickHouse, including creating Kafka topics, Kafka and ClickHouse tables, materialized views to transfer data, and best practices. It also covers alternatives to the ClickHouse Kafka engine and the roadmap for further improving the Kafka integration.
Wayfair Use Case: The four R's of Metrics DeliveryInfluxData
Wayfair currently uses both Graphite and InfluxDB as a time series platform - The data is used by their developers, business stakeholders, by their internal alerting engine. Most importantly their 24x7 Ops Monitoring Center is using this data to constantly analyze the vital signs of Wayfair’s IT infrastructure and storefront operations.
This document discusses supporting HDF5 in GrADS, an interactive desktop tool for analyzing earth science data. It outlines how GrADS currently handles different data formats, including HDF4 and netCDF. The document proposes two options for supporting HDF5 in GrADS - linking with the NetCDF-4 library, which would be easy but limited, or linking directly with the HDF5 library, which would require a new interface but provide more general HDF5 support.
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
Workshop "Can my .NET application use less CPU / RAM?", Yevhen TatarynovFwdays
In most cases it’s very hard to predict the number of resources needed for your .NET application. But If you spot some abnormal CPU or RAM usage, how to answer the question “Can my application use less?”.
Let’s see samples from real projects, where optimal resource usage by the application became one of the values for the product owner and see how less resource consumption can be.
The workshop will be actual for .NET developers who are interested in optimization of .NET applications, QA engineers who involved performance testing of .NET applications. It also will be interesting to everyone who "suspected" their .NET applications of non-optimal use of resources, but for some reason did not start an investigation.
This document provides an introduction to software exploitation on Linux 32-bit systems. It covers common exploitation techniques like buffer overflows, format strings, and ret2libc attacks. It discusses the Linux memory layout and stack structure. It explains buffer overflows on the stack and heap, and how to leverage them to alter control flow and execute arbitrary code. It also covers the format string vulnerability and how to leak information or write to arbitrary memory locations. Tools mentioned include GDB, exploit-exercises, and Python. Overall it serves as a crash course on the basic techniques and concepts for Linux exploitation.
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...Austin Benson
This document discusses matrix storage and data serialization techniques for scientific computing with Hadoop and Dumbo. It provides examples of storing matrices in HDFS using different approaches like storing each row separately, storing two rows per record, or flattening the matrix into a single list. It also discusses optimizing data serialization and switching programming languages. The document then presents an example of outputting many small matrices to disk and compares two MapReduce implementations for computing the Cholesky QR decomposition, identifying which approach is usually better and why.
Accelerate Reed-Solomon coding for Fault-Tolerance in RAID-like systemShuai Yuan
The document discusses accelerating Reed-Solomon erasure codes on GPUs. It aims to accelerate two main computation bottlenecks: arithmetic operations in Galois fields and matrix multiplication. For Galois field operations, it evaluates loop-based and table-based methods and chooses a log-exponential table approach. It also proposes tiling algorithms to optimize matrix multiplication on GPUs by reducing data transfers and improving memory access patterns. The goal is to make Reed-Solomon encoding and decoding faster for cloud storage systems using erasure codes.
Building a High-Performance Distributed Task Queue on MongoDBMongoDB
This document discusses building a distributed job queue in MongoDB. It begins by outlining the requirements of processing tasks across multiple servers. It then describes the design of storing messages and tasks in MongoDB collections, including different message and task states. It also covers techniques like composite tasks, reducing polling through tailable cursors, and optimizing performance through retry logic and oplog replay. The overall aim is to build an asynchronous and scalable way to process jobs in a distributed manner using MongoDB.
The document discusses several Java and Android internals topics:
1. How ArrayList and StringBuilder work internally using arrays and memory copying as the size increases. This can lead to inefficient memory usage.
2. How inner classes are implemented by compilers by generating additional accessor methods, increasing method count and affecting optimizations.
3. How the Android zygote process improves startup and memory usage by loading the framework once and sharing it across apps.
4. How the CPU cache works and how optimizing code to improve cache locality can significantly increase performance despite doing less work.
5. Issues like memory fragmentation that can occur if the Android garbage collector and compactor are unable to run due to the app being
The document summarizes a student project analyzing relative word frequencies using different algorithms implemented in Hadoop and Spark. It includes pseudocode and Java code for a pairs, stripe, and hybrid approach in Hadoop, as well as results. It also describes using Spark to analyze Apache log files by response code and errors.
Building Scalable, Distributed Job Queues with Redis and Redis::ClientMike Friedman
This document discusses using Redis and the Redis::Client Perl module to build scalable distributed job queues. It provides an overview of Redis, describing it as a key-value store that is simple, fast, and open-source. It then covers the various Redis data types like strings, lists, hashes, sets and sorted sets. Examples are given of how to work with these types using Redis::Client. The document discusses using Redis lists to implement job queues, with jobs added via RPUSH and popped via BLPOP. Benchmark results show the Redis-based job queue approach significantly outperforms using a MySQL jobs table with polling. Some caveats are provided about the benchmarks.
Optimizing with persistent data structures (LLVM Cauldron 2016)Igalia
By Andy Wingo.
Is there life beyond phi variables and basic blocks? Andy will report on his experience using a new intermediate representation for compiler middle-ends, "CPS soup". The CPS soup language represents programs using Clojure-inspired maps, allowing optimizations to be neatly expressed as functions from one graph to another. Using these persistent data structures also means that the source program doesn't change while the residual program is being created, eliminating one class of problems that the optimizer writer has to keep in mind. Together we will look at some example transformations from an expressiveness as well as a performance point of view, and we will also cover some advantages which a traditional SSA graph maintains over CPS soup.
Data correlation using PySpark and HDFSJohn Conley
This document discusses using PySpark and HDFS to correlate different types of data at scale. It describes some challenges with correlating out-of-order and high-volume data. It then summarizes three approaches tried: using Redis, RDD joins, and writing bindings to HDFS. The current recommended approach reads relevant binding buckets from HDFS to correlate records in small windows, supporting different temporal models. Configuration and custom logic can be plugged in at various points in the correlation process. While scalable, further improvements in latency and throughput are still needed.
This document provides examples of common SQL anti-patterns and related NoSQL alternatives. It discusses issues like using tables as trees, caches, queues, or logs. It also addresses dynamic schema/table creation, stored procedures, row padding, complex joins, and object-relational mismatches. The document recommends alternatives like document databases, key-value stores, message brokers, and denormalization. It includes examples of modeling book data in MongoDB and Redis.
InfluxDB IOx Tech Talks: The Impossible Dream: Easy-to-Use, Super Fast Softw...InfluxData
The document discusses how an easy-to-use and fast database can have a complicated implementation for developers. It outlines four key areas: 1) Flexible writing schema requires schema merging at read time. 2) Fast reads prune non-covered data chunks through predicate push-down. 3) Loading duplicated data necessitates data deduplication and compaction operations. 4) Quick data deletion still needs data elimination at read time or in the background. The document provides examples to illustrate the tradeoffs between user and developer requirements.
Full text search in PostgreSQL is a flexible and powerful facility to search collection of documents using natural language queries. We will discuss several new improvements of FTS in PostgreSQL 9.6 release, such as phrase search, better dictionaries support and tsvector editing functions. Also, we will present new features currently in development - RUM index support, which enables acceleration of some important kinds of full text queries, new and better ranking function for relevance search, loading dictionaries into shared memory and support for search multilingual content.
The document discusses mining the top-K multidimensional gradients from a fact table. It proposes a gradient-based cubing approach that partitions gradient regions based on spreading factors to efficiently retrieve the top-K gradient cells. The approach builds inverted indexes and value-list indices from the base table before pruning non-valid regions and cells. It then calculates spreading factors to create a GR tree and further partition valid gradient regions.
The Ring programming language version 1.9 book - Part 17 of 210Mahmoud Samir Fayed
The document summarizes the new features in Ring 1.4. Key points include:
1. Basic extensions like ODBC, MySQL, SQLite, OpenSSL and Internet libraries are now separate loadable files rather than part of the RingVM.
2. A new Natural Language library allows defining languages and commands to write programs in a more natural way, like English.
3. A new style is added to the Ring Notepad text editor.
4. Functions are introduced to convert between numbers and bytes.
5. The standard library and web library are updated with new functions and better error handling.
The document discusses various techniques for optimizing web program performance, including improving data structures and algorithms, reducing network, disk, and memory I/O, using NoSQL databases, optimizing frontend resource loading and execution, and testing and refactoring code for performance. It also covers specific optimizations related to caching, content delivery networks, domain name resolution, module loading, code packaging, and handling slower users.
This document summarizes garbage collection in the .NET Common Language Runtime (CLR). It covers the basics of generations and segments, how allocation and collection work, and different flavors of GC like concurrent and server GC. It also discusses pinning, tools for analyzing GC, and challenges for 64-bit systems. The goal of generational GC is to improve efficiency by focusing collection on younger object generations where most objects die, in order to minimize pause times during full collections.
Fast Insight from Fast Data: Integrating ClickHouse and Apache KafkaAltinity Ltd
This document discusses using Kafka as a messaging system with ClickHouse for high throughput and low latency data ingestion. It provides an overview of Kafka and how it can be used with ClickHouse, including creating Kafka topics, Kafka and ClickHouse tables, materialized views to transfer data, and best practices. It also covers alternatives to the ClickHouse Kafka engine and the roadmap for further improving the Kafka integration.
Wayfair Use Case: The four R's of Metrics DeliveryInfluxData
Wayfair currently uses both Graphite and InfluxDB as a time series platform - The data is used by their developers, business stakeholders, by their internal alerting engine. Most importantly their 24x7 Ops Monitoring Center is using this data to constantly analyze the vital signs of Wayfair’s IT infrastructure and storefront operations.
This document discusses supporting HDF5 in GrADS, an interactive desktop tool for analyzing earth science data. It outlines how GrADS currently handles different data formats, including HDF4 and netCDF. The document proposes two options for supporting HDF5 in GrADS - linking with the NetCDF-4 library, which would be easy but limited, or linking directly with the HDF5 library, which would require a new interface but provide more general HDF5 support.
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
Workshop "Can my .NET application use less CPU / RAM?", Yevhen TatarynovFwdays
In most cases it’s very hard to predict the number of resources needed for your .NET application. But If you spot some abnormal CPU or RAM usage, how to answer the question “Can my application use less?”.
Let’s see samples from real projects, where optimal resource usage by the application became one of the values for the product owner and see how less resource consumption can be.
The workshop will be actual for .NET developers who are interested in optimization of .NET applications, QA engineers who involved performance testing of .NET applications. It also will be interesting to everyone who "suspected" their .NET applications of non-optimal use of resources, but for some reason did not start an investigation.
This document provides an introduction to software exploitation on Linux 32-bit systems. It covers common exploitation techniques like buffer overflows, format strings, and ret2libc attacks. It discusses the Linux memory layout and stack structure. It explains buffer overflows on the stack and heap, and how to leverage them to alter control flow and execute arbitrary code. It also covers the format string vulnerability and how to leak information or write to arbitrary memory locations. Tools mentioned include GDB, exploit-exercises, and Python. Overall it serves as a crash course on the basic techniques and concepts for Linux exploitation.
[ACNA2022] Hadoop Vectored IO_ your data just got faster!.pdfMukundThakur22
Since 2006 the world of big data has moved from terabytes to hundreds of petabytes, from local clusters to remote cloud storage, yet the original Apache Hadoop POSIX-based file APIs have barely changed.
It is wonderful that these APIs have worked so well, but we can do a lot better with remote object stores, by providing new operations which suit them better, targeted at columnar data libraries such as ORC and Spark. Only a few libraries need to migrate to these APIs for significant speedups of all big data applications.
This talk introduces a new Hadoop Filesystem API called "vectored read", coming in Hadoop 3.4. An extension of the classic FSDataInputStream it is automatically offered by all filesystem clients.
The S3A connector is the first object store to provide a custom implementation, reading different blocks of data in parallel. In Apache Hive benchmarks with a modified ORC library, we saw a 2x speedup compared to using the classic s3a connector through the Posix APIs.
We will introduce the API spec, the S3A implementation, and the benchmarks, and show how to use it in your own applications. We will also cover our ongoing work on providing similar speedups with other object stores, and the use of the API in other applications.
This document discusses ByteSurgeon, a library for runtime bytecode transformation in Smalltalk. ByteSurgeon provides a high-level API for modifying bytecode without changing the virtual machine. This allows runtime adaptation, debugging, and adding new language features. ByteSurgeon transformations are faster than recompilation. Examples show implementing tracing, logging, and a metaobject protocol (MOP) for intercepting instance variable access in just a few lines of code. Benchmarks show ByteSurgeon is over 5 times faster than standard recompilation approaches.
H2O World - What's New in H2O with Cliff ClickSri Ambati
This document provides an overview of updates to H2O, including a complete rewrite of its core internals and algorithms. Key changes include improvements to generalized linear modeling, gradient boosted machines, and the introduction of a new dimensionality reduction technique called generalized low rank modeling. It also outlines enhancements to H2O's Python and R clients for easier data munging, modeling, and interactive use on large datasets.
This document summarizes a novel concurrent reference counting garbage collection algorithm called sliding views. It reduces the overhead of reference counting by only logging the first modification of each pointer between collections. It also avoids synchronization costs by stopping threads one at a time and using dirty bits and buffers to track modifications. Evaluation shows it provides shorter and more consistent pause times than stop-the-world collection with good throughput. Later work extended it with cycle collection and other techniques.
Vector and ListBuffer have similar performance for random reads. Benchmarking showed no significant difference in throughput, average time, or sample times between reading randomly from a Vector versus a ListBuffer. Vectors are generally faster than Lists for random access due to Vectors being implemented as arrays under the hood.
The document describes a presentation about data processing with CDK (Cloud Development Kit). It includes an agenda that covers CDK and Projen, serverless ETL with Glue, Databrew with continuous integration/continuous delivery (CICD), and using Amazon Comprehend with S3 object lambdas. Constructs are demonstrated for building architectures with CDK across multiple programming languages. Examples are provided of using CDK to implement Glue workflows, Databrew CICD pipelines, and combining Comprehend with S3 object lambdas for PII detection and redaction.
Heading for a Record: Chromium, the 5th CheckPVS-Studio
We checked Chromium more than once before, and those who follow our blog could reasonably ask, "Why another check? Weren't there enough of them?" Sure, Chromium's source code is particularly clean, which was shown by each of the previous checks, but new errors inevitably continue to appear. Repeated checks prove that the more often you use static analysis, the better. A good practice is to use the analyzer every day. An even better practice is to analyze the new code right after you finish writing it (automatic analysis of recently modified code).
Fixed width data can be processed efficiently in Perl using forks and shared file handles. This talk describes the basic mechanism and alternatives for improving the performance in dealing with the records.
XDP in Practice: DDoS Mitigation @CloudflareC4Media
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2NtlaER.
Gilberto Bertin discusses the architecture of Cloudflare’s automatic DDoS mitigation pipeline, the initial packet filtering solution based on Iptables, and why Cloudflare had to introduce userspace offload. Bertin also describes how they switched from a proprietary offload technology to XDP for network stack bypass and how they are using XDP to load balance traffic. Filmed at qconlondon.com.
Gilberto Bertin works as a System Engineer at Cloudflare London. After working on variety of technologies like P2P VPNs and userspace TCP/IP stacks, he joined the Cloudflare DDoS team in London to help filter all the bad internet traffic.
Tapad's data pipeline is an elastic combination of technologies (Kafka, Hadoop, Avro, Scalding) that forms a reliable system for analytics, realtime and batch graph-building, and logging. In this talk, I will speak about the creation and evolution of the pipeline, and a concrete example – a day in the life of an event tracking pixel. We'll also talk about common challenges that we've overcome such as integrating different pieces of the system, schema evolution, queuing, and data retention policies.
Quantifying Container Runtime Performance: OSCON 2017 Open Container DayPhil Estes
A talk given at Open Container Day at O'Reilly's OSCON convention in Austin, Texas on May 9th, 2017. This talk describes an open source project, bucketbench, which can be used to compare performance, stability, and throughput of various container engines. Bucketbench currently supports docker, containerd, and runc, but can be extended to support any container runtime. This work was done in response to performance investigations by the Apache OpenWhisk team in using containers as the execution vehicle for functions in their "Functions-as-a-Service" runtime. Find out more about bucketbench here: https://github.com/estesp/bucketbench
This document discusses mass report generation using REST. It proposes an architecture where customer report data is stored as JSON strings in attachments, and metadata is stored as JSON on object fields. A process is initiated via scheduling, REST calls, or manually to generate PDF reports from the attachment contents for each object. The reports are re-attached and metadata manages the process. Limitations of callouts, CPU time, and the getContentAsPDF() function for bulk jobs are also covered.
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...Ontico
This document discusses improvements that can be made to full text search in PostgreSQL. It proposes changes to the GIN index to store additional positional information, calculate ranking scores directly in the index, and return results in sorted order. This would eliminate the need for a separate sorting step and heap scan, significantly speeding up full text queries. Testing on real datasets showed the approach increased query throughput by over 10 times compared to the existing implementation. The changes are available as a 150KB patch for PostgreSQL 9.3 and additional work is planned to further optimize index building and support partial matching.
In most of our articles about project checks, we mention that bugs are found by the PVS-Studio static code analyzer. In certain cases – when dealing with projects of a complex structure – it is this particular analyzer that is needed. However, many developers will also appreciate its lightweight version, the CppCat analyzer. In this connection, we decided to use CppCat this time, when checking the TortoiseGit project.
Node has revolutionized modern runtimes. Their async by default strategy boasts 3x the throughput of Java. And yet, the language runs 5x slower than C++ (when JS is interpreted).
This talk is an advanced intro into the world of Node where we take a closer look under the hood. What's the event loop? Why are there multiple compilers for JS in Node/V8? How many threads are actually used in Node and for what purpose? We'll answer these questions and more as we go over libuv, v8, the node core library, npm, and more.
If you're developing with Node, want to start, or are just curious about how it works, please check it out!
This document summarizes several proposed changes for Java 7 including better integer literals with underscores for clarity, improved type inference for constructors and argument positions, new features like string switches and automatic resource management, and new libraries such as NIO2 and the fork/join framework for parallel programming.
This document provides an overview of Apache Phoenix, including:
- What Phoenix is and how it provides a SQL interface for Apache HBase
- The current state of Phoenix including SQL support, secondary indexes, and optimizations
- New features in Phoenix 4.4 like functional indexes, user defined functions, and integration with Spark
The presentation covers the evolution and capabilities of Phoenix as a relational layer for HBase that transforms SQL queries into native HBase API calls.
Linux Kernel, tested by the Linux-version of PVS-StudioPVS-Studio
Since the release of the publicly available Linux-version of PVS-Studio, it was just a matter of time until we would recheck the Linux kernel. It is quite a challenge for any static code analyzer to check a project written by professionals from all around the world, used by people in various fields, which is regularly checked and tested by different tools. So, what errors did we manage to find in such conditions?
Similar to Performance .NET Core - M. Terech, P. Janowski (20)
Dandelion Hashtable: beyond billion requests per second on a commodity serverAntonios Katsarakis
This slide deck presents DLHT, a concurrent in-memory hashtable. Despite efforts to optimize hashtables, that go as far as sacrificing core functionality, state-of-the-art designs still incur multiple memory accesses per request and block request processing in three cases. First, most hashtables block while waiting for data to be retrieved from memory. Second, open-addressing designs, which represent the current state-of-the-art, either cannot free index slots on deletes or must block all requests to do so. Third, index resizes block every request until all objects are copied to the new index. Defying folklore wisdom, DLHT forgoes open-addressing and adopts a fully-featured and memory-aware closed-addressing design based on bounded cache-line-chaining. This design offers lock-free index operations and deletes that free slots instantly, (2) completes most requests with a single memory access, (3) utilizes software prefetching to hide memory latencies, and (4) employs a novel non-blocking and parallel resizing. In a commodity server and a memory-resident workload, DLHT surpasses 1.6B requests per second and provides 3.5x (12x) the throughput of the state-of-the-art closed-addressing (open-addressing) resizable hashtable on Gets (Deletes).
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
What is an RPA CoE? Session 1 – CoE VisionDianaGray10
In the first session, we will review the organization's vision and how this has an impact on the COE Structure.
Topics covered:
• The role of a steering committee
• How do the organization’s priorities determine CoE Structure?
Speaker:
Chris Bolin, Senior Intelligent Automation Architect Anika Systems
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Skybuffer SAM4U tool for SAP license adoptionTatiana Kojar
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
The Microsoft 365 Migration Tutorial For Beginner.pptxoperationspcvita
This presentation will help you understand the power of Microsoft 365. However, we have mentioned every productivity app included in Office 365. Additionally, we have suggested the migration situation related to Office 365 and how we can help you.
You can also read: https://www.systoolsgroup.com/updates/office-365-tenant-to-tenant-migration-step-by-step-complete-guide/
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
Essentials of Automations: Exploring Attributes & Automation ParametersSafe Software
Building automations in FME Flow can save time, money, and help businesses scale by eliminating data silos and providing data to stakeholders in real-time. One essential component to orchestrating complex automations is the use of attributes & automation parameters (both formerly known as “keys”). In fact, it’s unlikely you’ll ever build an Automation without using these components, but what exactly are they?
Attributes & automation parameters enable the automation author to pass data values from one automation component to the next. During this webinar, our FME Flow Specialists will cover leveraging the three types of these output attributes & parameters in FME Flow: Event, Custom, and Automation. As a bonus, they’ll also be making use of the Split-Merge Block functionality.
You’ll leave this webinar with a better understanding of how to maximize the potential of automations by making use of attributes & automation parameters, with the ultimate goal of setting your enterprise integration workflows up on autopilot.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/temporal-event-neural-networks-a-more-efficient-alternative-to-the-transformer-a-presentation-from-brainchip/
Chris Jones, Director of Product Management at BrainChip , presents the “Temporal Event Neural Networks: A More Efficient Alternative to the Transformer” tutorial at the May 2024 Embedded Vision Summit.
The expansion of AI services necessitates enhanced computational capabilities on edge devices. Temporal Event Neural Networks (TENNs), developed by BrainChip, represent a novel and highly efficient state-space network. TENNs demonstrate exceptional proficiency in handling multi-dimensional streaming data, facilitating advancements in object detection, action recognition, speech enhancement and language model/sequence generation. Through the utilization of polynomial-based continuous convolutions, TENNs streamline models, expedite training processes and significantly diminish memory requirements, achieving notable reductions of up to 50x in parameters and 5,000x in energy consumption compared to prevailing methodologies like transformers.
Integration with BrainChip’s Akida neuromorphic hardware IP further enhances TENNs’ capabilities, enabling the realization of highly capable, portable and passively cooled edge devices. This presentation delves into the technical innovations underlying TENNs, presents real-world benchmarks, and elucidates how this cutting-edge approach is positioned to revolutionize edge AI across diverse applications.
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor IvaniukFwdays
At this talk we will discuss DDoS protection tools and best practices, discuss network architectures and what AWS has to offer. Also, we will look into one of the largest DDoS attacks on Ukrainian infrastructure that happened in February 2022. We'll see, what techniques helped to keep the web resources available for Ukrainians and how AWS improved DDoS protection for all customers based on Ukraine experience
How information systems are built or acquired puts information, which is what they should be about, in a secondary place. Our language adapted accordingly, and we no longer talk about information systems but applications. Applications evolved in a way to break data into diverse fragments, tightly coupled with applications and expensive to integrate. The result is technical debt, which is re-paid by taking even bigger "loans", resulting in an ever-increasing technical debt. Software engineering and procurement practices work in sync with market forces to maintain this trend. This talk demonstrates how natural this situation is. The question is: can something be done to reverse the trend?
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframePrecisely
Inconsistent user experience and siloed data, high costs, and changing customer expectations – Citizens Bank was experiencing these challenges while it was attempting to deliver a superior digital banking experience for its clients. Its core banking applications run on the mainframe and Citizens was using legacy utilities to get the critical mainframe data to feed customer-facing channels, like call centers, web, and mobile. Ultimately, this led to higher operating costs (MIPS), delayed response times, and longer time to market.
Ever-changing customer expectations demand more modern digital experiences, and the bank needed to find a solution that could provide real-time data to its customer channels with low latency and operating costs. Join this session to learn how Citizens is leveraging Precisely to replicate mainframe data to its customer channels and deliver on their “modern digital bank” experiences.
11. 11
Hypothetical Use Case
Company needs to report all sale
transactions quaterly for statistics purposes.
Since the Company makes thousands of
transactions every day - report file is very
large.
12. 12
Hypothetical Use Case
Company needs to report all sale
transactions quaterly for statistics purposes.
Since the Company makes thousands of
transactions every day - report file is very
large.
Let’s assume the report is a CSV file.
13. 13
Hypothetical Use Case
Company needs to report all sale
transactions quaterly for statistics purposes.
Since the Company makes thousands of
transactions every day - report file is very
large.
Let’s assume the report is a CSV file.
15. 15
Naïve approach 2
// BETTER APPROACH
private async Task ProcessRequestAsync(Stream stream)
{
var buffer = new byte[1024];
while (true)
{
var bytesRead = await stream.ReadAsync(buffer, 0, buffer.Length);
if (bytesRead == 0)
{
return; // EOF
}
var newLinePos = -1;
var bytesChecked = 0;
do
{
newLinePos = Array.IndexOf(buffer, (byte)'n', bytesChecked, bytesRead - bytesChecked);
if (newLinePos >= 0)
{
var lineLength = newLinePos - bytesChecked;
ProcessRowData(buffer, bytesChecked, lineLength);
}
bytesChecked += newLinePos + 1;
}
while (newLinePos > 0);
}
}
16. 16
CSV might be tricky
Some product; 5.0; "Amazingn
Category"; 10n
Other product; 1.0; Boring category; 2n
17. 17
CSV might be tricky
Some product; 5.0; "Amazingn
Category"; 10n
Other product; 1.0; Boring category; 2n
18. 18
How we can improve our approach?
We can allocate bigger buffer when we face longer line or multiline record
19. 19
How we can improve our approach?
We can allocate bigger buffer when we face longer line or multiline record
We can create own pool of buffers, new buffer would be created when the previous gets
filled up (i.e. in 80% or more).
21. 21
System.IO.Pipelines to the rescue
The mentioned issues can easily be solved by using Pipe.
Pipe
PipeWriter PipeReader
22. 22
Pipes
private async Task ProcessRequestAsync(Stream stream)
{
var pipe = new Pipe();
var readFileTask = ReadFileAsync(stream, pipe.Writer);
var processFileTask = ProcessFileAsync(pipe.Reader);
await Task.WhenAll(readFileTask, processFileTask);
}
23. 23
PipeWriter
private async Task ReadFileAsync(Stream stream, PipeWriter pipeWriter)
{
while (true)
{
Memory<byte> memory = pipeWriter.GetMemory(BufferSize);
int bytesRead = await stream.ReadAsync(memory);
if (bytesRead == 0)
{
break;
}
pipeWriter.Advance(bytesRead);
// flush data to PipeReader
FlushResult flushResult = await pipeWriter.FlushAsync();
if (flushResult.IsCompleted)
{
break;
}
}
pipeWriter.Complete(); // we are done
}
24. 24
PipeReader
private async Task ProcessFileAsync(PipeReader pipeReader)
{
while(true)
{
ReadResult result = await pipeReader.ReadAsync();
ReadOnlySequence<byte> buffer = result.Buffer;
SequencePosition? position = null;
do
{
position = buffer.PositionOf((byte)'n'); // find position of newline character, read multiline row…
if (position != null)
{
ProcessRowData(buffer.Slice(0, position.Value));
buffer = buffer.Slice(buffer.GetPosition(1, position.Value)); // move to next line
}
}
while (position != null);
if (result.IsCompleted)
{
break;
}
pipeReader.AdvanceTo(buffer.Start, buffer.End); // let know the PipeReader how much bytes were consumed
}
pipeReader.Complete(); // we are done
}
25. 25
Partial read
Pipe.Reader.ReadAsync() Some product; 5
Some product; 5 .PositionOf((byte)'n') null
Pipe.Reader.ReadAsync() Some product; 5.0; Some Categoryn
Some product; 5.0; Some Categoryn .PositionOf((byte)'n') SequencePosition
Pipe.Reader.AdvanceTo( SequencePosition +1)
Pipe.Reader.ReadAsync() Other product; 3.0;…
39. 39
How is it relevant?
Pipes originally were introduced for internal use in Kestrel. Eventually, the’ve
evolved into a part of public API.
40. 40
How is it relevant?
Pipes originally were introduced for internal use in Kestrel. Eventually, the’ve
evolved into a part of public API.
The body of request in ASP.NET Core 3.0 is exposed via BodyReader property of
the HttpContext. BodyReader is in fact PipeReader.
41. 41
How is it relevant?
Pipes originally were introduced for internal use in Kestrel. Eventually, the’ve
evolved into a part of public API.
The body of request in ASP.NET Core 3.0 is exposed via BodyReader property of
the HttpContext. BodyReader is in fact PipeReader.
Response body can be written with BodyWriter (PipeWriter).
45. 45
Span<T> - Example
N A M E @ E M A I L
public ReadOnlySpan<char> GetName(ReadOnlySpan<char> email)
{
var @position = email.LastIndexOf(‘@’);
return @position == -1
? ReadOnlySpan<char>.Empty
: email.Slice(0, @position);
}
46. 46
Span<T> - Two versions
Background vector Created by brgfx - www.freepik.com
Tu myślałem wyjść od jakiegoś use-case'u. Np mamy jakiś duży plik CSV który chcemy sparsować. Prezentujemy stare podejście na streamach, płynnie przechodząc do System.IO.Pipelines i potem BodyReadera/Writera
Tu myślałem wyjść od jakiegoś use-case'u. Np mamy jakiś duży plik CSV który chcemy sparsować. Prezentujemy stare podejście na streamach, płynnie przechodząc do System.IO.Pipelines i potem BodyReadera/Writera
Tu myślałem wyjść od jakiegoś use-case'u. Np mamy jakiś duży plik CSV który chcemy sparsować. Prezentujemy stare podejście na streamach, płynnie przechodząc do System.IO.Pipelines i potem BodyReadera/Writera
KOD!
Możemy zaalokować większy bufor, jeżeli natrafimy na dłuższą linię lub rekord zawierający kilka linii. Przy okazji można stosować inne sztuczki, np. ArrayPool w celu minimalizowania liczby alokowanych buforów itp. Itd..
No ok. tylko wtedy przy zwiększaniu bufora kopiujemy pamięć. Dodatkowo warto by było potem zmniejszać ten bufor, żeby nie zajmować niepotrzebnie pamięci….pomysł nie do końca trafiony – generuje duże wykorzystanie pamięci.
Możemy poprzedni pomysł rozszerzyć o dodawanie dodatkowych buforów tylko wtedy, kiedy poprzedni zostanie wykorzystany. Potrzebujemy wtedy logiki do odczytywania fragmentów pojedynczego rekordu z wielu buforów, oznaczania zwolnionych już buforów (takich z których odczytaliśmy już dane rekordu) itp. Itd. Całość robi się bardzo skomplikowana…
A i tak nie jest do końca efektywnie, bo:
Odczyt danych ze strumienia jest uzależniony od szybkości parsowania.
Wykorzystanie zwykłej zarządzanej tablicy bajtów może mieć impact na GC – pinned memory, może prowadzić do fragmentacji pamięci.
Możemy zaalokować większy bufor, jeżeli natrafimy na dłuższą linię lub rekord zawierający kilka linii. Przy okazji można stosować inne sztuczki, np. ArrayPool w celu minimalizowania liczby alokowanych buforów itp. Itd..
No ok. tylko wtedy przy zwiększaniu bufora kopiujemy pamięć. Dodatkowo warto by było potem zmniejszać ten bufor, żeby nie zajmować niepotrzebnie pamięci….pomysł nie do końca trafiony – generuje duże wykorzystanie pamięci.
Możemy poprzedni pomysł rozszerzyć o dodawanie dodatkowych buforów tylko wtedy, kiedy poprzedni zostanie wykorzystany. Potrzebujemy wtedy logiki do odczytywania fragmentów pojedynczego rekordu z wielu buforów, oznaczania zwolnionych już buforów (takich z których odczytaliśmy już dane rekordu) itp. Itd. Całość robi się bardzo skomplikowana…
A i tak nie jest do końca efektywnie, bo:
Odczyt danych ze strumienia jest uzależniony od szybkości parsowania.
Wykorzystanie zwykłej zarządzanej tablicy bajtów może mieć impact na GC – pinned memory, może prowadzić do fragmentacji pamięci.
Pipe składa się z dwóch części:
PipeWriter – który zapisuje do naszej „rury” oraz PipeReader, który z tej „rury” czyta.
Mamy ładnie podzielony proces na 2 składowe – załadowanie danych ze strumienia oraz parsowanie tych danych.
Task.WhenAll wskazuje, że obie składowe będą wykonywały się asynchronicznie.
Pipe składa się z dwóch części:
PipeWriter – który zapisuje do naszej „rury” oraz PipeReader, który z tej „rury” czyta.
Mamy ładnie podzielony proces na 2 składowe – załadowanie danych ze strumienia oraz parsowanie tych danych.
Task.WhenAll wskazuje, że obie składowe będą wykonywały się asynchronicznie.
KOD
Jedną z największych zalet wykorzystanie Pipe’ów jest Partial Read – możemy w zasadzie pracować na danych bez ich „skonsumowania”.
Przykład ilustruje przykładowy parser HTTP.
Pod maską Pipe zarządza (linked) listą zaalokowanych buforów, które są wykorzystywane przez PipeWritera/PipeReadera. PipeReader.ReadAsync zwraca ReadOnlySequence<T>. Za jego pomocą możemy uzyskać dostęp do jednego lub więcej segmentów pamięci (ReadOnlyMemory<T>) – analogicznie do Span<T> i Memory<T> i stringów.
Pipe przechowuje informację o położeniu Writera i Readera w kontekście zaalokowanych danych z wykorzystaniem SequencePosition.
SequencePosition to nic innego jak wskaźnik na konkretne miejsce we wspomnianej linked liście.
ReadOnlySequence<T> i Sequence position to struktury.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
W idealnym przypadku to co przychodzi do serwera jest odczytywane przez jeden z wątków i jednocześnie parsowane i przetwarzane przez inny wątek. Samo parsowanie może jednak trwać znacznie dłużej niż ładowanie danych ze strumienia. W tej sytuacji wątek ładujący dane ze strumienia może albo alokować więcej pamięci, albo wstrzymać pracę na jakiś czas - należy znaleźć odpowiedni balans w celu zapewnienia optymalnej wydajności.
Do rozwiązania wspomnianego problemu Pipy zostały wyposażone w 2 właściwości – PauseWriterThreshold i ResumeWriterThreshold. Pierwsza definiuje ile danych powinno zostać zbuforowanych zanim wywołanie metody FlushAsync (PipeWriter) spowoduje wstrzymanie pracy Writera. Druga z kolei kontroluje ilość danych którą może zostać skonsumowanych przez PipeReadera zanim praca PipeWritera zostanie wznowiona.
Oczywiście w 99% przypadków wgl nie będziemy zajmowali się samodzielnym parsowaniem treści Body. Warto jednak wiedzieć jak to działa pod maską.
Oczywiście w 99% przypadków wgl nie będziemy zajmowali się samodzielnym parsowaniem treści Body. Warto jednak wiedzieć jak to działa pod maską.
Oczywiście w 99% przypadków wgl nie będziemy zajmowali się samodzielnym parsowaniem treści Body. Warto jednak wiedzieć jak to działa pod maską.
Oczywiście w 99% przypadków wgl nie będziemy zajmowali się samodzielnym parsowaniem treści Body. Warto jednak wiedzieć jak to działa pod maską.