Distem is a distributed systems emulator. When doing research on Cloud, P2P, High Performance Computing or Grid systems, it can be used to transform an homogenenous cluster (composed of identical nodes) into an experimental platform where nodes have different performance, and are linked together through a complex network topology, making it the ideal tool to benchmark applications targetting such environments.
In this presentation, you get an overview of what ZUGFeRD, a standard for e-invoices developed by the Forum elektronische Rechnung Deutschland is about. We look at XML standards for business, at PDF standards for archiving and we learn how ZUGFeRD combines both. We also look beyond ZUGFeRD at standards for digital signing.
All of Your Network Monitoring is (probably) Wrongice799
Monitorama 2016 talk about network monitoring covering topics like network device drivers, ethtool, and some interesting bugs/features.
For more information about monitoring and tuning the entire Linux network stack, see: blog.packagecloud.io/eng/2016/06/22/monitoring-tuning-linux-networking-stack-receiving-data/
All new features, expected features and speculations regarding the upcoming Java 9 release: the Jigsaw project, performance improvements and long awaited APIs:
www.takipiblog.com/java-9-the-ultimate-feature-list/
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
Broken benchmarks, misleading metrics, and terrible tools. This talk will help you navigate the treacherous waters of Linux performance tools, touring common problems with system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. You might discover that tools you have been using for years, are in fact, misleading, dangerous, or broken.
The speaker, Brendan Gregg, has given many talks on tools that work, including giving the Linux PerformanceTools talk originally at SCALE. This is an anti-version of that talk, to focus on broken tools and metrics instead of the working ones. Metrics can be misleading, and counters can be counter-intuitive! This talk will include advice for verifying new performance tools, understanding how they work, and using them successfully.
Video: https://www.youtube.com/watch?v=JRFNIKUROPE . Talk for linux.conf.au 2017 (LCA2017) by Brendan Gregg, about Linux enhanced BPF (eBPF). Abstract:
A world of new capabilities is emerging for the Linux 4.x series, thanks to enhancements that have been included in Linux for to Berkeley Packet Filter (BPF): an in-kernel virtual machine that can execute user space-defined programs. It is finding uses for security auditing and enforcement, enhancing networking (including eXpress Data Path), and performance observability and troubleshooting. Many new open source tools that have been written in the past 12 months for performance analysis that use BPF. Tracing superpowers have finally arrived for Linux!
For its use with tracing, BPF provides the programmable capabilities to the existing tracing frameworks: kprobes, uprobes, and tracepoints. In particular, BPF allows timestamps to be recorded and compared from custom events, allowing latency to be studied in many new places: kernel and application internals. It also allows data to be efficiently summarized in-kernel, including as histograms. This has allowed dozens of new observability tools to be developed so far, including measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more.
This talk will summarize BPF capabilities and use cases so far, and then focus on its use to enhance Linux tracing, especially with the open source bcc collection. bcc includes BPF versions of old classics, and many new tools, including execsnoop, opensnoop, funcccount, ext4slower, and more (many of which I developed). Perhaps you'd like to develop new tools, or use the existing tools to find performance wins large and small, especially when instrumenting areas that previously had zero visibility. I'll also summarize how we intend to use these new capabilities to enhance systems analysis at Netflix.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
Introduction to InfiniBand technology and its usage on Debian platform. Includes various examples of NAS/SAN over IB configuration (NFSoRDMA, SRP, and other standard IP and IPoIB based setups).
9. Bug ID: 4206909
• http://bugs.sun.com/view_bug.do?bug_id=4206909
• Synopsis: want java.util.zip to work for
interactive use (Z_SYNC_FLUSH)
• Submit Date: 28-JAN-1999
• Reported Against: 1.2, 1.3, 1.1.7, 1.1.8
• Release Fixed: 7(b77)
• State: 10-Fix Delivered, request for
enhancement
10. Bug ID: 4206909
• Description:
“Today it is not possible to reuse
java.util.zip for compression because the
Java VM calls deflate() in the libz with
Z_NO_FLUSH (and Z_FINISH for EOF)
only.”
11. Bug ID: 4206909
• Description:
“ This means that in most cases the
deflate() does not emit enough data for
the decompressor to reassemble the
complete data packet because it waits for
more input.”
19. JZlib
• 開発の動機
SSHのパケット圧縮[RFC4253]実現のため
“The 'zlib' compression is described in
[RFC1950] and in [RFC1951].The
compression context is initialized after each
key exchange, and is passed from one packet
to the next, with only a partial flush being
performed at the end of each packet. “
20. JZlib
• JZlibを使っているソフトウェア
– JSch
– Apache MINA
– Netty
– Zimbra
– as3zlib
– Zlib.cs
– IronPython.Zlib
– Nokia Data Gathering 等
27. SYNC_FLUSHの使用例
scala> val defl_infl = {
import java.util.zip._
val (defl, infl, buf) = (new Deflater, new Inflater, new Array[Byte](100))
def f(in: String): String = synchronized {
in.getBytes match {
case result =>
defl.setInput(result)
defl.deflate(buf, 0, buf.length, Deflater.SYNC_FLUSH) match{
case l => l は buf に書き出された圧縮データのサイズ
//
infl.setInput(buf, 0, l)
val len = infl.inflate(result, 0, result.length)
new String(result, 0, len)
}}} // ちゃんと実装するには、中間バッファ(buf)の溢れに注意
f_ // また、伸張後のバッファ溢れにも注意すべき
}