This document discusses techniques for quickly parsing JSON documents. It begins by benchmarking different JSON parsing libraries and finds that simdjson, which uses SIMD instructions, can parse Twitter's JSON feed at 2.4 GB/s on a 3.4 GHz CPU, around 4 times faster than other libraries. It then describes four tricks used in simdjson to achieve this speed: 1) avoiding hard-to-predict branches, 2) using wide SIMD instructions, 3) avoiding memory and object allocation, and 4) measuring performance through benchmark-driven development. The document provides several examples showing how these tricks can dramatically speed up JSON parsing and related tasks.
TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices.
Interesting Observations (7 Sins of Programmers); The compiler is to blame; Archeological strata; The last line effect; Programmers are the smartest; Security, security! But do you test it?; You can’t know everything; Seeking a silver bullet.
Everything you wanted to know about Stack Traces and Heap DumpsAndrei Pangin
Stack traces and heap dumps are not just debugging tools; they open a doorway to the very heart of the Java Virtual Machine. This session is focused on peculiarities of JDK somehow related to heap walking and thread stacks.
• How to create dump in production without side effects?
• What is the performance impact of stack traces?
• How do jmap and jstack work under the hood, and what is special about 'forced' mode?
• Why do all profilers lie, and how to deal with it?
• How to scan heap using JVMTI?
The presentation supported by the live demos will address all these questions. It will also introduce new Java 9 Stack-Walking API (JEP 259) and reveal some useful but little known features of HotSpot JVM.
Down to Stack Traces, up from Heap DumpsAndrei Pangin
Глубже стек-трейсов, шире хип-дампов
Stack trace и heap dump - не просто инструменты отладки; это потайные дверцы к самым недрам виртуальной Java машины. Доклад будет посвящён малоизвестным особенностям JDK, так или иначе связанным с обоходом хипа и стеками потоков.
Мы разберём:
- как снимать дампы в продакшне без побочных эффектов;
- как работают утилиты jmap и jstack изнутри, и в чём хитрость forced режима;
- почему все профилировщики врут, и как с этим бороться;
- познакомимся с новым Stack-Walking API в Java 9;
- научимся сканировать Heap средствами JVMTI;
- узнаем о недокументированных функциях Хотспота и других интересных штуках.
The Source SDK is a software development kit compiled by Valve Corporation that is used to create games or mods for the Source engine. I downloaded and checked the project's source codes at the end of 2013 already and intended to write an article about it during the New Year holidays. But laziness prevailed over the craving for creativity, and I sat down to writing the article only on getting back to work. However, I doubt that the source codes have changed during this time. Now you are welcome to have a look at the suspicious code fragments found in the project code by the PVS-Studio code analyzer.
TensorFlow is an open source software library for high performance numerical computation. Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices.
Interesting Observations (7 Sins of Programmers); The compiler is to blame; Archeological strata; The last line effect; Programmers are the smartest; Security, security! But do you test it?; You can’t know everything; Seeking a silver bullet.
Everything you wanted to know about Stack Traces and Heap DumpsAndrei Pangin
Stack traces and heap dumps are not just debugging tools; they open a doorway to the very heart of the Java Virtual Machine. This session is focused on peculiarities of JDK somehow related to heap walking and thread stacks.
• How to create dump in production without side effects?
• What is the performance impact of stack traces?
• How do jmap and jstack work under the hood, and what is special about 'forced' mode?
• Why do all profilers lie, and how to deal with it?
• How to scan heap using JVMTI?
The presentation supported by the live demos will address all these questions. It will also introduce new Java 9 Stack-Walking API (JEP 259) and reveal some useful but little known features of HotSpot JVM.
Down to Stack Traces, up from Heap DumpsAndrei Pangin
Глубже стек-трейсов, шире хип-дампов
Stack trace и heap dump - не просто инструменты отладки; это потайные дверцы к самым недрам виртуальной Java машины. Доклад будет посвящён малоизвестным особенностям JDK, так или иначе связанным с обоходом хипа и стеками потоков.
Мы разберём:
- как снимать дампы в продакшне без побочных эффектов;
- как работают утилиты jmap и jstack изнутри, и в чём хитрость forced режима;
- почему все профилировщики врут, и как с этим бороться;
- познакомимся с новым Stack-Walking API в Java 9;
- научимся сканировать Heap средствами JVMTI;
- узнаем о недокументированных функциях Хотспота и других интересных штуках.
The Source SDK is a software development kit compiled by Valve Corporation that is used to create games or mods for the Source engine. I downloaded and checked the project's source codes at the end of 2013 already and intended to write an article about it during the New Year holidays. But laziness prevailed over the craving for creativity, and I sat down to writing the article only on getting back to work. However, I doubt that the source codes have changed during this time. Now you are welcome to have a look at the suspicious code fragments found in the project code by the PVS-Studio code analyzer.
A short and fast journey through some of the profiling options available in the Ruby 2.x world, including a look at flamegraphs and new ways of tracking memory usage in the MRI.
SFO15-500: VIXL
Speaker: Amaury Le Leyzour
Date: September 25, 2015
★ Session Description ★
VIXL is dynamic code generation toolkit for ARMv8 that we hope will enable JIT creators to rapidly target the ARM instruction set.
Over the past few years we (the ARM JIT team) have worked on the code generators of many of the leading JIT compilers for the JavaScript and Java languages. During that time we built up a strong knowledge base on some of the pitfalls and time-sinks involved in creating a good JIT compiler backend. This led us to develop some tools to help improve our productivity. With ARM announcing the new Cortex-A range of processors supporting the AArch64 execution state we decided that we would focus our efforts on A64 tooling to enable developers to rapidly port programming language virtual machines for this new processor range. Soon after we decided to support Aarch32 as well.
This presentation will introduce you to what VIXL is, what’s new in VIXL and how to use it and take advantage of all its components that cover all the aspects of software development on ARM CPUs.
★ Resources ★
Video: https://www.youtube.com/watch?v=XxMTSO4clQY
Etherpad: pad.linaro.org/p/sfo15-500
Pathable: https://sfo15.pathable.com/meetings/303091
★ Event Details ★
Linaro Connect San Francisco 2015 - #SFO15
September 21-25, 2015
Hyatt Regency Hotel
http://www.linaro.org
http://connect.linaro.org
We have checked the header files from the Embarcadero C++Builder XE3 project. In fact, it means that we have checked just a small number of inline-functions. Accordingly, quite few issues were found, but they are enough to write a small post.
Если нашлась одна ошибка — есть и другие. Один способ выявить «наследуемые» у...Positive Hack Days
Ведущий: Асука Накадзима (Asuka Nakajima)
Практика повторного использования исходного кода позволяет сократить расходы на разработку программного обеспечения. Тем не менее, если в оригинальном исходном коде кроется уязвимость, она будет перенесена и в новое приложение. Докладчик расскажет о необычном способе обнаружения «наследуемых» уязвимостей в бинарных файлах без необходимости обращаться к исходному коду или символьным файлам.
Работа с реляционными базами данных в C++corehard_by
Поговорим в первую очередь о библиотеках, которые унифицируют работу с различными БД: Oracle, MSSQL, Postgres, MySQL и др. Я поделюсь своим опытом работы с некоторыми из них. А также посмотрим, что может нас ожидать в будущем в плане работы с SQL базами данных.
Linux has this great tool called strace, on OSX there’s a tool called dtruss - based on dtrace. Dtruss is great in functionality, it gives pretty much everything you need. It is just not as nice to use as strace. However, on Linux there is also ltrace for library tracing. That is arguably more useful because you can see much more granular application activity. Unfortunately, there isn’t such a tool on OSX. So, I decided to make one - albeit a simpler version for now. I called it objc_trace.
A short and fast journey through some of the profiling options available in the Ruby 2.x world, including a look at flamegraphs and new ways of tracking memory usage in the MRI.
SFO15-500: VIXL
Speaker: Amaury Le Leyzour
Date: September 25, 2015
★ Session Description ★
VIXL is dynamic code generation toolkit for ARMv8 that we hope will enable JIT creators to rapidly target the ARM instruction set.
Over the past few years we (the ARM JIT team) have worked on the code generators of many of the leading JIT compilers for the JavaScript and Java languages. During that time we built up a strong knowledge base on some of the pitfalls and time-sinks involved in creating a good JIT compiler backend. This led us to develop some tools to help improve our productivity. With ARM announcing the new Cortex-A range of processors supporting the AArch64 execution state we decided that we would focus our efforts on A64 tooling to enable developers to rapidly port programming language virtual machines for this new processor range. Soon after we decided to support Aarch32 as well.
This presentation will introduce you to what VIXL is, what’s new in VIXL and how to use it and take advantage of all its components that cover all the aspects of software development on ARM CPUs.
★ Resources ★
Video: https://www.youtube.com/watch?v=XxMTSO4clQY
Etherpad: pad.linaro.org/p/sfo15-500
Pathable: https://sfo15.pathable.com/meetings/303091
★ Event Details ★
Linaro Connect San Francisco 2015 - #SFO15
September 21-25, 2015
Hyatt Regency Hotel
http://www.linaro.org
http://connect.linaro.org
We have checked the header files from the Embarcadero C++Builder XE3 project. In fact, it means that we have checked just a small number of inline-functions. Accordingly, quite few issues were found, but they are enough to write a small post.
Если нашлась одна ошибка — есть и другие. Один способ выявить «наследуемые» у...Positive Hack Days
Ведущий: Асука Накадзима (Asuka Nakajima)
Практика повторного использования исходного кода позволяет сократить расходы на разработку программного обеспечения. Тем не менее, если в оригинальном исходном коде кроется уязвимость, она будет перенесена и в новое приложение. Докладчик расскажет о необычном способе обнаружения «наследуемых» уязвимостей в бинарных файлах без необходимости обращаться к исходному коду или символьным файлам.
Работа с реляционными базами данных в C++corehard_by
Поговорим в первую очередь о библиотеках, которые унифицируют работу с различными БД: Oracle, MSSQL, Postgres, MySQL и др. Я поделюсь своим опытом работы с некоторыми из них. А также посмотрим, что может нас ожидать в будущем в плане работы с SQL базами данных.
Linux has this great tool called strace, on OSX there’s a tool called dtruss - based on dtrace. Dtruss is great in functionality, it gives pretty much everything you need. It is just not as nice to use as strace. However, on Linux there is also ltrace for library tracing. That is arguably more useful because you can see much more granular application activity. Unfortunately, there isn’t such a tool on OSX. So, I decided to make one - albeit a simpler version for now. I called it objc_trace.
No dark magic - Byte code engineering in the real worldtcurdt
ften byte code engineering is perceived as "black magic" and considered too low level. This session will not bore you with all the details of the JVM specification but provide you with a practical overview of how this byte code "swizzling" can be used in the real world instead. Several projects have sucessfully leveraged this technique to achieve some amazing things (e.g. AOP). This session will go through some of these examples and try to outline the differences between the two major libraries (BCEL and ASM). Maybe your next project can then benefit from some of this "magic".
An introduction to exploit development.
I gave this talk at Hack the North 2014, and most of this information is pulled out of classics like Smashing the Stack for Fun and Profit, so there shouldn't be anything novel in here.
MongoDB Europe 2016 - Debugging MongoDB PerformanceMongoDB
Asya is back, and so is Sherlock Holmes and his techniques to gather and analyze data from your poorly performing MongoDB clusters. In this advanced talk we take a deep look at all the diagnostic data that lives inside MongoDB - how to interrogate and interpret it to help you solve those frustrating performance bottlenecks that we all face occasionally.
100 bugs in Open Source C/C++ projects Andrey Karpov
This article demonstrates capabilities of the static code analysis methodology. The readers are offered to study the samples of one hundred errors found in open-source projects in C/C++.
Davide Berardi - Linux hardening and security measures against Memory corruptionlinuxlab_conf
The exploding popularity of Embedded/IoT computing facilitate this security problems using low or non-existent security policies and exploits countermeasures. So why not explore some security measures that are widely available in the Linux world? We will focus on memory corruption techniques.
The Linux kernel was always focused on security features and giving bad times to the exploiters. This talk will introduce some common exploits and techniques, showing the mitigations employed by the kernel. By focusing on the major threats that affects modern Linux boxes, we will see which are the main features that can give problems to the system administator and how a preliminary penetration test can be done, ensuring that the system is in a sane state. The talk will also focus on problematics of embedded/IoT Unix systems, showing how some recent attacks gained control over a big network of devices and how a simple embedded system can be analyzed, hunting for bugs. Talk outline: Penetration testing, Linux, netfilter/bpf, memory corruption, ASLR, Spectre/Meltdown.
Accurate and efficient software microbenchmarksDaniel Lemire
Software is often improved incrementally. Each software optimization should be assessed with microbenchmarks. In a microbenchmark, we record performance measures such as elapsed time or instruction counts during specific tasks, often in idealized conditions. In principle, the process is easy: if the new code is faster, we adopt it. Unfortunately, there are many pitfalls, such as unrealistic statistical assumptions and poorly designed benchmarks. Abstractions like cloud computing add further challenges. We illustrate effective benchmarking practices with examples.
Presentation on Roaring bitmaps for the Go Montreal meetup (Go 10th anniversary).
Roaring bitmaps are a standard indexing data structure. They are
widely used in search and database engines. For example, Lucene, the
search engine powering Wikipedia relies on Roaring. The Go library
roaring implements Roaring bitmaps in Go. It is used in several
popular systems such as InfluxDB, Pilosa and Bleve. This library is
used in production in several systems, it is part of the Awesome Go
collection. After presenting the library, we will cover some advanced
Go topics such as the use of assembly language, unsafe mappings, and
so forth.
Next Generation Indexes For Big Data Engineering (ODSC East 2018)Daniel Lemire
Maximizing performance in data engineering is a daunting challenge. We present some of our work on designing faster indexes, with a particular emphasis on compressed indexes. Some of our prior work includes (1) Roaring indexes which are part of multiple big-data systems such as Spark, Hive, Druid, Atlas, Pinot, Kylin, (2) EWAH indexes are part of Git (GitHub) and included in major Linux distributions.
We will present ongoing and future work on how we can process data faster while supporting the diverse systems found in the cloud (with upcoming ARM processors) and under multiple programming languages (e.g., Java, C++, Go, Python). We seek to minimize shared resources (e.g., RAM) while exploiting algorithms designed for the single-instruction-multiple-data (SIMD) instructions available on commodity processors. Our end goal is to process billions of records per second per core.
The talk will be aimed at programmers who want to better understand the performance characteristics of current big-data systems as well as their evolution. The following specific topics will be addressed:
1. The various types of indexes and their performance characteristics and trade-offs: hashing, sorted arrays, bitsets and so forth.
2. Index and table compression techniques: binary packing, patched coding, dictionary coding, frame-of-reference.
Ingénierie de la performance au sein des mégadonnéesDaniel Lemire
Les index logiciels accélèrent les applications en intelligence d'affaire, en apprentissage machine et en science des données. Ils déterminent souvent la performance des applications portant sur les mégadonnées. Les index efficaces améliorent non seulement la latence et le débit, mais aussi la consommation d'énergie. Plusieurs index font une utilisation parcimonieuse de la mémoire vive afin que les données critiques demeurent près du processeur. Il est aussi souhaitable de travailler directement sur les données compressées afin d'éviter une étape de décodage supplémentaire.
(1) Nous nous intéressons aux index bitmap. Nous les trouvons dans une vaste gamme de systèmes :
Oracle, Hive, Spark, Druid, Kylin, Lucene, Elastic, Git... Ils sont une composante de systèmes, tels que Wikipedia ou GitHub, dont dépendent des millions d'utilisateurs à tous les jours. Nous
présenterons certains progrès récents ayant trait à l'optimisation des index bitmap, tels qu'ils sont utilisés au sein des systèmes actuels. Nous montrons par des exemples comment multiplier la
performance de ces index dans certains cas sur les processeurs bénéficiant d'instructions SIMD (instruction unique, données multiples) avancées.
(2) Nous ciblons aussi les listes d'entiers que l'on trouve au sein des arbres B+, dans les indexes inversés et les index bitmap compressés. Nous donnons un exemple récent de technique de compression (Stream VByte) d’entiers qui permet de décoder des milliards d’entiers compressés par seconde.
SIMD Compression and the Intersection of Sorted IntegersDaniel Lemire
Sorted lists of integers are commonly used in inverted indexes and database systems. They are often compressed in memory. We can use the SIMD instructions available in common processors to boost the speed of integer compression schemes. Our S4-BP128-D4 scheme uses as little as 0.7 CPU cycles per decoded integer while still providing state-of-the-art compression. However, if the subsequent processing of the integers is slow, the effort spent on optimizing decoding speed can be wasted. To show that it does not have to be so, we (1) vectorize and optimize the intersection of posting lists; (2) introduce the SIMD Galloping algorithm. We exploit the fact that one SIMD instruction can compare 4 pairs of integers at once. We experiment with two TREC text collections, GOV2 and ClueWeb09 (Category B), using logs from the TREC million-query track. We show that using only the SIMD instructions ubiquitous in all modern CPUs, our techniques for conjunctive queries can double the speed of a state-of-the-art approach.
Decoding billions of integers per second through vectorizationDaniel Lemire
In many important applications -- such as search engines and relational database systems -- data is stored in the form of arrays of integers. Encoding and, most importantly, decoding of these arrays consumes considerable CPU time. Therefore, substantial effort has been made to reduce costs associated with compression and decompression. In particular, researchers have exploited the superscalar nature of modern processors and SIMD instructions. Nevertheless, we introduce a novel vectorized scheme called SIMD-BP128 that improves over previously proposed vectorized approaches. It is nearly twice as fast as the previously fastest schemes on desktop processors (varint-G8IU and PFOR). At the same time, SIMD-BP128 saves up to 2 bits per integer. For even better compression, we propose another new vectorized scheme (SIMD-FastPFOR) that has a compression ratio within 10% of a state-of-the-art scheme (Simple-8b) while being two times faster during decoding.
Logarithmic Discrete Wavelet Transform for High-Quality Medical Image Compres...Daniel Lemire
Nowadays, medical image compression is an essential process in eHealth systems. Compressing medical images in high quality is a vital demand to avoid misdiagnosing medical exams by radiologists. WAAVES is a promising medical images compression algorithm based on the discrete wavelet transform (DWT) that achieves a high compression performance compared to the state of the art. The main aims of this work are to enhance image quality when compressing using WAAVES and to provide a high-speed DWT architecture for image compression on embedded systems. Regarding the quality improvement, the logarithmic number systems (LNS) was explored to be used as an alternative to the linear arithmetic in DWT computations. A new LNS library was developed and validated to realize the logarithmic DWT. In addition, a new quantization method called (LNS-Q) based on logarithmic arithmetic was proposed. A novel compression scheme (LNS-WAAVES) based on integrating the Hybrid-DWT and the LNS-Q method with WAAVES was developed. Hybrid-DWT combines the advantages of both the logarithmic and the linear domains leading to enhancement of the image quality and the compression ratio. The results showed that LNS-WAAVES is able to achieve an improvement in the quality by a percentage of 8% and up to 34% compared to WAAVES depending on the compression configuration parameters and the image modalities.
Contemporary computing hardware offers massive new performance opportunities. Yet high-performance programming remains a daunting challenge.
We present some of the lessons learned while designing faster indexes, with a particular emphasis on compressed bitmap indexes. Compressed bitmap indexes accelerate queries in popular systems such as Apache Spark, Git, Elastic, Druid and Apache Kylin.
We consider the ubiquitous technique of VByte compression, which represents each integer as a variable length sequence of bytes. The low 7 bits of each byte encode a portion of the integer, and the high bit of each byte is reserved as a continuation flag. This flag is set to 1 for all bytes except the last, and the decoding of each integer is complete when a byte with a high bit of 0 is encountered. VByte decoding can be a performance bottleneck especially when the unpredictable lengths of the encoded integers cause frequent branch mispredictions. Previous attempts to accelerate VByte decoding using SIMD vector instructions have been disappointing, prodding search engines such as Google to use more complicated but faster-to-decode formats for performance-critical code. Our decoder (Masked VByte) is 2 to 4 times faster than a conventional scalar VByte decoder, making the format once again competitive with regard to speed.
Jeff Plaisance, Nathan Kurz, Daniel Lemire, Vectorized VByte Decoding, International Symposium on Web Algorithms 2015, 2015.
http://arxiv.org/pdf/1503.07387.pdf
Better bitmap performance with Roaring bitmaps
Bitmaps are used to implement fast set operations in software. They are frequently found in databases and search engines.
Without compression, bitmaps scale poorly, so they are often compressed. Many bitmap compression techniques have been proposed, almost all relying primarily on run-length encoding (RLE). For example, Oracle relies on BBC bitmap compression while the version control system Git and Apache Hive rely on EWAH compression.
We can get superior performance with a hybrid compression technique that uses both uncompressed bitmaps and packed arrays inside a two-level tree. An instance of this technique, Roaring, has been adopted by several production platforms (e.g., Apache Lucene/Solr/Elastic, Apache Spark, eBay's Apache Kylin and Metamarkets' Druid).
Overall, our implementation of Roaring can be several times faster (up to two orders of magnitude) than the implementations of traditional RLE-based alternatives (WAH, Concise, EWAH) while compressing better. We review the design choices and optimizations that make these good results possible.
La vectorisation des algorithmes de compression Daniel Lemire
Depuis la mise en marché du Pentium 4, nos processeurs bénéficient d'instructions vectorielles. En tenant compte explicitement de ces instructions dans la conception de nos algorithmes, nous pouvons grandement accélérer les calculs. À titre d'exemple, considérons la compression des listes d'entiers telle qu'elle s'effectue au sein de la plupart des moteurs de recherche ou des bases de données. En cette matière, nous utilisons souvent encore des algorithmes développés dans les années 70. Nous expliquerons comment on peut faire beaucoup mieux
en ce qui a trait à la vitesse en exploitant les instructions
vectorielles.
Decoding billions of integers per second through vectorization Daniel Lemire
In many important applications -- such as search engines and relational database systems -- data is stored in the form of arrays of integers. Encoding and, most importantly, decoding of these arrays consumes considerable CPU time. Therefore, substantial effort has been made to reduce costs associated with compression and decompression. In particular, researchers have exploited the superscalar nature of modern processors and SIMD instructions. Nevertheless, we introduce a novel vectorized scheme called SIMD-BP128 that improves over previously proposed vectorized approaches. It is nearly twice as fast as the previously fastest schemes on desktop processors (varint-G8IU and PFOR). At the same time, SIMD-BP128 saves up to 2 bits per integer. For even better compression, we propose another new vectorized scheme (SIMD-FastPFOR) that has a compression rate within 10% of a state-of-the-art scheme (Simple-8b) while being two times faster during decoding.
Extracting, Transforming and Archiving Scientific DataDaniel Lemire
It is becoming common to archive research datasets that are not only large but also numerous. In addition, their corresponding metadata and the software required to analyse or display them need to be archived. Yet the manual curation of research data can be difficult and expensive, particularly in very large digital repositories, hence the importance of models and tools for automating digital curation tasks. The automation of these tasks faces three major challenges: (1) research data and data sources are highly heterogeneous, (2) future research needs are difficult to anticipate, (3) data is hard to index. To address these problems, we propose the Extract, Transform and Archive (ETA) model for managing and mechanizing the curation of research data. Specifically, we propose a scalable strategy for addressing the research-data problem, ranging from the extraction of legacy data to its long-term storage. We review some existing solutions and propose novel avenues of research.
Innovation without permission: from Codd to NoSQLDaniel Lemire
Practitioners often fail to apply textbook database design principles. We observe both a perversion of the relational model and a growth of less formal alternatives. Overall, there is an opposition between the analytic thought that prevailed when many data modeling techniques were initiated, and the pragmatism which now dominates among practitioners. There are at least two recent trends supporting this rejection of traditional models:
(1) the rise of the sophisticated user,
most notably in social media is challenge to the rationalist view, as it blurs the distinction between design and operation,
(2) in the new technological landscape where there are billions of interconnected computers worldwide, simple concepts like
consistency sometimes become prohibitively expensive. Overall, for a wide range of information systems, design and operation are becoming integrated in the spirit of pragmatism. Thus, we are left with design methodologies which embrace fast and continual iterations and and exploratory testing. These methodologies allow innovation without permission in that the right to design new features is no longer so closely guarded.
Fo
Recent research results in optimizing column-oriented indexes for faster data warehousing. This talks aims to answer the following question: when is sorting the table a sufficiently good optimization?
Column-oriented databases have become fashionable following the work of Stonebraker et al. In the data warehousing industry, the terms "column oriented" and "column store" have become necessary marketing buzzwords. One of the benefits of column-oriented indexes is good compression through run-length encoding (RLE). This type of compression is particularly benefitial since it simultaneously reduce the volume of data and the necessary computations. However, the efficiency of the compression depends on the order of the rows in the table and this is even more important with larger tables. Finding the best row ordering is NP hard. We compare some heuristics for this problem including variations on the lexicographical order, Gray codes, and Hilbert space-filling curves.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.