This document provides instructions for installing the PyGrib library within the CDAT path on Ubuntu. It describes downloading required packages like grib_api and pygrib, moving them to the CDAT path, and running the install scripts. It also gives examples of importing and using pygrib within Python to open GRIB files, extract data values and metadata, and write modified GRIB messages to a new file.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
I presented a whirlwind tour of the most common benchmark tools used to measure parallel file system performance and reviewed case studies of how these have been used in the procurement of NERSC's large file systems at the 2022 Lustre User Group.
Broken benchmarks, misleading metrics, and terrible tools. This talk will help you navigate the treacherous waters of Linux performance tools, touring common problems with system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. You might discover that tools you have been using for years, are in fact, misleading, dangerous, or broken.
The speaker, Brendan Gregg, has given many talks on tools that work, including giving the Linux PerformanceTools talk originally at SCALE. This is an anti-version of that talk, to focus on broken tools and metrics instead of the working ones. Metrics can be misleading, and counters can be counter-intuitive! This talk will include advice for verifying new performance tools, understanding how they work, and using them successfully.
Join us for a one-hour, introductory Postman learning session geared specifically for API testers. In this session, you’ll learn how to test the functionality and reliability of an API.
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Signals are interrupts sent to a process when an event occurs, allowing communication between processes. Processes can send signals using the kill() system call and receive default actions like termination or be handled by a signal handler. Message queues allow processes to exchange typed messages through first-in-first-out queues, created with message queue identifiers. Processes use msgsnd() and msgrcv() to place messages on and receive messages from queues.
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
This document discusses using the Linux perf profiling tool at Netflix. It begins with an overview of why Netflix needs Linux profiling to understand CPU usage quickly and completely. It then provides an introduction to the perf tool, covering its basic workflow and commands. The document discusses profiling CPU usage with perf, including potential issues like JIT runtimes and missing symbols. It provides several examples of perf commands for listing, counting, and recording events. The overall summary is that perf allows Netflix to quickly and accurately profile CPU usage across the entire software stack, from applications to libraries to the kernel, to optimize performance.
Video: https://www.youtube.com/watch?v=FJW8nGV4jxY and https://www.youtube.com/watch?v=zrr2nUln9Kk . Tutorial slides for O'Reilly Velocity SC 2015, by Brendan Gregg.
There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This tutorial explains methodologies for using these tools, and provides a tour of four tool types: observability, benchmarking, tuning, and static tuning. Many tools will be discussed, including top, iostat, tcpdump, sar, perf_events, ftrace, SystemTap, sysdig, and others, as well observability frameworks in the Linux kernel: PMCs, tracepoints, kprobes, and uprobes.
This tutorial is updated and extended on an earlier talk that summarizes the Linux performance tool landscape. The value of this tutorial is not just learning that these tools exist and what they do, but hearing when and how they are used by a performance engineer to solve real world problems — important context that is typically not included in the standard documentation.
I presented a whirlwind tour of the most common benchmark tools used to measure parallel file system performance and reviewed case studies of how these have been used in the procurement of NERSC's large file systems at the 2022 Lustre User Group.
Broken benchmarks, misleading metrics, and terrible tools. This talk will help you navigate the treacherous waters of Linux performance tools, touring common problems with system tools, metrics, statistics, visualizations, measurement overhead, and benchmarks. You might discover that tools you have been using for years, are in fact, misleading, dangerous, or broken.
The speaker, Brendan Gregg, has given many talks on tools that work, including giving the Linux PerformanceTools talk originally at SCALE. This is an anti-version of that talk, to focus on broken tools and metrics instead of the working ones. Metrics can be misleading, and counters can be counter-intuitive! This talk will include advice for verifying new performance tools, understanding how they work, and using them successfully.
Join us for a one-hour, introductory Postman learning session geared specifically for API testers. In this session, you’ll learn how to test the functionality and reliability of an API.
Talk by Brendan Gregg for USENIX LISA 2019: Linux Systems Performance. Abstract: "
Systems performance is an effective discipline for performance analysis and tuning, and can help you find performance wins for your applications and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes the topic for everyone, touring six important areas of Linux systems performance: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events) and tracing (Ftrace, bcc/BPF, and bpftrace/BPF), and much advice about what is and isn't important to learn. This talk is aimed at everyone: developers, operations, sysadmins, etc, and in any environment running Linux, bare metal or the cloud."
Signals are interrupts sent to a process when an event occurs, allowing communication between processes. Processes can send signals using the kill() system call and receive default actions like termination or be handled by a signal handler. Message queues allow processes to exchange typed messages through first-in-first-out queues, created with message queue identifiers. Processes use msgsnd() and msgrcv() to place messages on and receive messages from queues.
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
This document discusses using the Linux perf profiling tool at Netflix. It begins with an overview of why Netflix needs Linux profiling to understand CPU usage quickly and completely. It then provides an introduction to the perf tool, covering its basic workflow and commands. The document discusses profiling CPU usage with perf, including potential issues like JIT runtimes and missing symbols. It provides several examples of perf commands for listing, counting, and recording events. The overall summary is that perf allows Netflix to quickly and accurately profile CPU usage across the entire software stack, from applications to libraries to the kernel, to optimize performance.
XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu,...The Linux Foundation
This document discusses vSVM design on the Xen hypervisor. It proposes exposing SVM extensions in hardware like PASID, ATS and PRQ through virtual IOMMU capabilities in Xen. This would allow guest VMs to utilize shared virtual addressing between CPU and devices. The design would involve shadow extended context entries pointing to guest PASID tables and queuing invalidation of translation caches between host and guest. Currently Xen supports device assignment but not full IOMMU functionality or SVM extensions for shared virtual addressing across VMs.
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
BPF (Berkeley Packet Filter) has evolved from a limited virtual machine for efficient packet filtering to a new type of software called extended BPF. Extended BPF allows for custom, efficient, and production-safe performance analysis tools and observability programs to be run in the Linux kernel through BPF. It enables new event-based applications running as BPF programs attached to various kernel events like kprobes, uprobes, tracepoints, sockets, and more. Major companies like Facebook, Google, and Netflix are using BPF programs for tasks like intrusion detection, container security, firewalling, and observability with over 150,000 AWS instances running BPF programs. BPF provides a new program model and security features compared
This document provides an overview of Ingress in Kubernetes, including:
1) It describes the different types of Kubernetes services - ClusterIP, NodePort, LoadBalancer, ExternalName, and Headless - and examples of using each type.
2) It explains that Ingress resources define routing rules to services, and Ingress controllers watch for Ingress resources and update rules to satisfy conditions.
3) Ingress allows for name-based and path-based routing to services, and controllers provide a default backend for requests not handled by Ingress rules.
The document provides an overview of the key components of the Linux operating system, including:
1) The Linux kernel, which acts as a resource manager for processes, memory, and hardware devices.
2) Process and memory management systems that control how processes are allocated and memory is allocated and freed.
3) The file system which organizes how files are stored and accessed.
4) Device drivers that allow the operating system to interface with hardware.
5) The network stack which handles network protocols and connections.
6) Architecture-dependent code that provides hardware-specific implementations of core functions.
Cilium - Bringing the BPF Revolution to Kubernetes Networking and SecurityThomas Graf
BPF is one of the fastest emerging technologies of the Linux kernel. The talk provides an introduction to Cilium which brings the powers of BPF to Kubernetes and other orchestration systems to provide highly scalable and efficient networking, security and load balancing for containers and microservices. The talk will provide an introduction to the capabilities of Cilium today but also deep dives into the emerging roadmap involving networking at the socket layer and service mesh datapath capabilities to provide highly efficient connectivity between cloud native apps and sidecar proxies.
This talk discusses Linux profiling using perf_events (also called "perf") based on Netflix's use of it. It covers how to use perf to get CPU profiling working and overcome common issues. The speaker will give a tour of perf_events features and show how Netflix uses it to analyze performance across their massive Amazon EC2 Linux cloud. They rely on tools like perf for customer satisfaction, cost optimization, and developing open source tools like NetflixOSS. Key aspects covered include why profiling is needed, a crash course on perf, CPU profiling workflows, and common "gotchas" to address like missing stacks, symbols, or profiling certain languages and events.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
This document provides an overview of how APIs are defined and served in the Kubernetes API server (kube-apiserver). It describes:
- How API types are defined using Golang types and registered with the API server's scheme
- How the generic API server handles requests and calls back to installed API groups
- How custom resource definitions (CRDs) are supported via the CRD and aggregator APIs
- Key components like conversion, defaulting, storage, and filters that process each request
- Live debugging techniques for the API server using tools like mux routers and request tracing
In 3 sentences or less, it outlines the architecture of the Kubernetes API server for defining, registering, and serving custom
* What is different GitHub Flow and Git Flow?
* What is GitHub Actions?
* How to write the simple workflow?
* What's problem in GitHub Actions UI?
* What's problem with Secrets in GitHub Actions?
* How to write your first GitHub Actions and upload to the marketplace?
* What's a problem with environment variables in GitHub Actions?
O documento discute a fisiologia do trato gastrointestinal. Ele descreve que o trato gastrointestinal possui a função de realizar trocas com o meio externo, transportando nutrientes para o meio interno. Também discute os desafios do sistema digestório, como produzir enzimas potentes sem digerir as próprias células, equilibrar os balanços de massa e repelir invasores externos. Por fim, descreve os processos básicos da digestão, como digestão, absorção, motilidade e secreção.
This document introduces an upcoming book titled "Real-World Kafka: From Development to Operations". The book provides an in-depth look at Kafka's internal workings from development to operations in an enterprise environment. It discusses topics such as Kafka architecture, producer and consumer internals, operations and monitoring, security, schema registry, and enterprise Kafka patterns. The book is aimed at developers, architects, and operators interested in fully utilizing Kafka. It uses diagrams and examples to explain complex concepts in an accessible way.
The document provides an overview of OpenMAX-IL (OpenMAX Integrated Layer), which is a royalty-free, cross-platform API for accelerated multimedia components. It defines OpenMAX-IL entities like components, ports, and the core. Components have input/output ports and support non-tunneled or tunneled communication. The document describes component states, buffer handling, simple usage including tunnel setup, and state transitions for OpenMAX-IL.
Monitoring containerised apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this talk, Brice Fernandes from Weaveworks will introduce and demo the open source Prometheus monitoring toolkit and its integration with Kubernetes. After this talk, you'll be able to use Prometheus to monitor your microservices on a Kubernetes cluster. We'll cover:
- An introduction to Kubernetes to manage containers;
- The monitoring maturity model;
- An overview of whitebox and blackbox monitoring;
- Monitoring with Prometheus;
- Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system
Video: https://www.youtube.com/watch?v=JRFNIKUROPE . Talk for linux.conf.au 2017 (LCA2017) by Brendan Gregg, about Linux enhanced BPF (eBPF). Abstract:
A world of new capabilities is emerging for the Linux 4.x series, thanks to enhancements that have been included in Linux for to Berkeley Packet Filter (BPF): an in-kernel virtual machine that can execute user space-defined programs. It is finding uses for security auditing and enforcement, enhancing networking (including eXpress Data Path), and performance observability and troubleshooting. Many new open source tools that have been written in the past 12 months for performance analysis that use BPF. Tracing superpowers have finally arrived for Linux!
For its use with tracing, BPF provides the programmable capabilities to the existing tracing frameworks: kprobes, uprobes, and tracepoints. In particular, BPF allows timestamps to be recorded and compared from custom events, allowing latency to be studied in many new places: kernel and application internals. It also allows data to be efficiently summarized in-kernel, including as histograms. This has allowed dozens of new observability tools to be developed so far, including measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more.
This talk will summarize BPF capabilities and use cases so far, and then focus on its use to enhance Linux tracing, especially with the open source bcc collection. bcc includes BPF versions of old classics, and many new tools, including execsnoop, opensnoop, funcccount, ext4slower, and more (many of which I developed). Perhaps you'd like to develop new tools, or use the existing tools to find performance wins large and small, especially when instrumenting areas that previously had zero visibility. I'll also summarize how we intend to use these new capabilities to enhance systems analysis at Netflix.
The document discusses building software for embedded systems across different architectures using QEMU. It describes native building versus cross building, and compares building directly on a system emulator versus Scratchbox. It proposes that a simpler approach is to use just the user mode of QEMU, along with chroot and binfmt_misc. An example is given of setting up a basic ARM chroot environment and building software faster using user mode QEMU instead of a full system emulator. Issues like mounting proc and networking are also addressed.
This document discusses improving debugging of the Linux kernel using the open source debugger GDB. It begins with an overview of existing ways to debug the kernel using GDB, including via KGDB, Qemu, and JTAG probes like OpenOCD. It then discusses the concept of adding "Linux awareness" to GDB, which would allow it to better understand kernel concepts like threads and modules. Finally, it outlines three approaches to implementing this awareness: via a GDB scripting extension, in the GDB stub, or with a C extension to GDB. The overall goal is to make GDB a more full-featured and useful tool for kernel debugging.
This document discusses Linux kernel debugging. It provides an overview of debugging techniques including collecting system information, handling failures, and using printk(), KGDB, and debuggers. Key points covered are the components of a debugger, how KGDB can be used with gdb to debug interactively, analyzing crash data, and some debugging tricks and print functions.
XPDDS17: Shared Virtual Memory Virtualization Implementation on Xen - Yi Liu,...The Linux Foundation
This document discusses vSVM design on the Xen hypervisor. It proposes exposing SVM extensions in hardware like PASID, ATS and PRQ through virtual IOMMU capabilities in Xen. This would allow guest VMs to utilize shared virtual addressing between CPU and devices. The design would involve shadow extended context entries pointing to guest PASID tables and queuing invalidation of translation caches between host and guest. Currently Xen supports device assignment but not full IOMMU functionality or SVM extensions for shared virtual addressing across VMs.
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."
UM2019 Extended BPF: A New Type of SoftwareBrendan Gregg
BPF (Berkeley Packet Filter) has evolved from a limited virtual machine for efficient packet filtering to a new type of software called extended BPF. Extended BPF allows for custom, efficient, and production-safe performance analysis tools and observability programs to be run in the Linux kernel through BPF. It enables new event-based applications running as BPF programs attached to various kernel events like kprobes, uprobes, tracepoints, sockets, and more. Major companies like Facebook, Google, and Netflix are using BPF programs for tasks like intrusion detection, container security, firewalling, and observability with over 150,000 AWS instances running BPF programs. BPF provides a new program model and security features compared
This document provides an overview of Ingress in Kubernetes, including:
1) It describes the different types of Kubernetes services - ClusterIP, NodePort, LoadBalancer, ExternalName, and Headless - and examples of using each type.
2) It explains that Ingress resources define routing rules to services, and Ingress controllers watch for Ingress resources and update rules to satisfy conditions.
3) Ingress allows for name-based and path-based routing to services, and controllers provide a default backend for requests not handled by Ingress rules.
The document provides an overview of the key components of the Linux operating system, including:
1) The Linux kernel, which acts as a resource manager for processes, memory, and hardware devices.
2) Process and memory management systems that control how processes are allocated and memory is allocated and freed.
3) The file system which organizes how files are stored and accessed.
4) Device drivers that allow the operating system to interface with hardware.
5) The network stack which handles network protocols and connections.
6) Architecture-dependent code that provides hardware-specific implementations of core functions.
Cilium - Bringing the BPF Revolution to Kubernetes Networking and SecurityThomas Graf
BPF is one of the fastest emerging technologies of the Linux kernel. The talk provides an introduction to Cilium which brings the powers of BPF to Kubernetes and other orchestration systems to provide highly scalable and efficient networking, security and load balancing for containers and microservices. The talk will provide an introduction to the capabilities of Cilium today but also deep dives into the emerging roadmap involving networking at the socket layer and service mesh datapath capabilities to provide highly efficient connectivity between cloud native apps and sidecar proxies.
This talk discusses Linux profiling using perf_events (also called "perf") based on Netflix's use of it. It covers how to use perf to get CPU profiling working and overcome common issues. The speaker will give a tour of perf_events features and show how Netflix uses it to analyze performance across their massive Amazon EC2 Linux cloud. They rely on tools like perf for customer satisfaction, cost optimization, and developing open source tools like NetflixOSS. Key aspects covered include why profiling is needed, a crash course on perf, CPU profiling workflows, and common "gotchas" to address like missing stacks, symbols, or profiling certain languages and events.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
This document provides an overview of how APIs are defined and served in the Kubernetes API server (kube-apiserver). It describes:
- How API types are defined using Golang types and registered with the API server's scheme
- How the generic API server handles requests and calls back to installed API groups
- How custom resource definitions (CRDs) are supported via the CRD and aggregator APIs
- Key components like conversion, defaulting, storage, and filters that process each request
- Live debugging techniques for the API server using tools like mux routers and request tracing
In 3 sentences or less, it outlines the architecture of the Kubernetes API server for defining, registering, and serving custom
* What is different GitHub Flow and Git Flow?
* What is GitHub Actions?
* How to write the simple workflow?
* What's problem in GitHub Actions UI?
* What's problem with Secrets in GitHub Actions?
* How to write your first GitHub Actions and upload to the marketplace?
* What's a problem with environment variables in GitHub Actions?
O documento discute a fisiologia do trato gastrointestinal. Ele descreve que o trato gastrointestinal possui a função de realizar trocas com o meio externo, transportando nutrientes para o meio interno. Também discute os desafios do sistema digestório, como produzir enzimas potentes sem digerir as próprias células, equilibrar os balanços de massa e repelir invasores externos. Por fim, descreve os processos básicos da digestão, como digestão, absorção, motilidade e secreção.
This document introduces an upcoming book titled "Real-World Kafka: From Development to Operations". The book provides an in-depth look at Kafka's internal workings from development to operations in an enterprise environment. It discusses topics such as Kafka architecture, producer and consumer internals, operations and monitoring, security, schema registry, and enterprise Kafka patterns. The book is aimed at developers, architects, and operators interested in fully utilizing Kafka. It uses diagrams and examples to explain complex concepts in an accessible way.
The document provides an overview of OpenMAX-IL (OpenMAX Integrated Layer), which is a royalty-free, cross-platform API for accelerated multimedia components. It defines OpenMAX-IL entities like components, ports, and the core. Components have input/output ports and support non-tunneled or tunneled communication. The document describes component states, buffer handling, simple usage including tunnel setup, and state transitions for OpenMAX-IL.
Monitoring containerised apps creates a whole new set of challenges that traditional monitoring systems struggle with. In this talk, Brice Fernandes from Weaveworks will introduce and demo the open source Prometheus monitoring toolkit and its integration with Kubernetes. After this talk, you'll be able to use Prometheus to monitor your microservices on a Kubernetes cluster. We'll cover:
- An introduction to Kubernetes to manage containers;
- The monitoring maturity model;
- An overview of whitebox and blackbox monitoring;
- Monitoring with Prometheus;
- Using PromQL (the Prometheus Query Language) to monitor your app in a dynamic system
Video: https://www.youtube.com/watch?v=JRFNIKUROPE . Talk for linux.conf.au 2017 (LCA2017) by Brendan Gregg, about Linux enhanced BPF (eBPF). Abstract:
A world of new capabilities is emerging for the Linux 4.x series, thanks to enhancements that have been included in Linux for to Berkeley Packet Filter (BPF): an in-kernel virtual machine that can execute user space-defined programs. It is finding uses for security auditing and enforcement, enhancing networking (including eXpress Data Path), and performance observability and troubleshooting. Many new open source tools that have been written in the past 12 months for performance analysis that use BPF. Tracing superpowers have finally arrived for Linux!
For its use with tracing, BPF provides the programmable capabilities to the existing tracing frameworks: kprobes, uprobes, and tracepoints. In particular, BPF allows timestamps to be recorded and compared from custom events, allowing latency to be studied in many new places: kernel and application internals. It also allows data to be efficiently summarized in-kernel, including as histograms. This has allowed dozens of new observability tools to be developed so far, including measuring latency distributions for file system I/O and run queue latency, printing details of storage device I/O and TCP retransmits, investigating blocked stack traces and memory leaks, and a whole lot more.
This talk will summarize BPF capabilities and use cases so far, and then focus on its use to enhance Linux tracing, especially with the open source bcc collection. bcc includes BPF versions of old classics, and many new tools, including execsnoop, opensnoop, funcccount, ext4slower, and more (many of which I developed). Perhaps you'd like to develop new tools, or use the existing tools to find performance wins large and small, especially when instrumenting areas that previously had zero visibility. I'll also summarize how we intend to use these new capabilities to enhance systems analysis at Netflix.
The document discusses building software for embedded systems across different architectures using QEMU. It describes native building versus cross building, and compares building directly on a system emulator versus Scratchbox. It proposes that a simpler approach is to use just the user mode of QEMU, along with chroot and binfmt_misc. An example is given of setting up a basic ARM chroot environment and building software faster using user mode QEMU instead of a full system emulator. Issues like mounting proc and networking are also addressed.
This document discusses improving debugging of the Linux kernel using the open source debugger GDB. It begins with an overview of existing ways to debug the kernel using GDB, including via KGDB, Qemu, and JTAG probes like OpenOCD. It then discusses the concept of adding "Linux awareness" to GDB, which would allow it to better understand kernel concepts like threads and modules. Finally, it outlines three approaches to implementing this awareness: via a GDB scripting extension, in the GDB stub, or with a C extension to GDB. The overall goal is to make GDB a more full-featured and useful tool for kernel debugging.
This document discusses Linux kernel debugging. It provides an overview of debugging techniques including collecting system information, handling failures, and using printk(), KGDB, and debuggers. Key points covered are the components of a debugger, how KGDB can be used with gdb to debug interactively, analyzing crash data, and some debugging tricks and print functions.
The document discusses using Git and git-svn to manage version control for Bioconductor packages. It recommends cloning specific revisions of packages from the Bioconductor Subversion repository using git-svn to get the full history while avoiding downloading the entire repository. It provides examples of common Git and git-svn commands that can be used for making changes locally and committing them back to the Subversion repository. The workflow suggested makes changes on a local Git branch tracking the remote Subversion branch, then cherry-picks the commits to other branches as needed.
O'Reilly Velocity New York 2016 presentation on modern Linux tracing tools and technology. Highlights the available tracing data sources on Linux (ftrace, perf_events, BPF) and demonstrates some tools that can be used to obtain traces, including DebugFS, the perf front-end, and most importantly, the BCC/BPF tool collection.
This document provides an overview of DNS configuration and troubleshooting on BIG-IP systems. It includes:
1. A sample DNS zone file configuration with resource records for domains, name servers, mail servers, and hosts.
2. An introduction to BIG-IP DNS and how it can load balance DNS queries across datacenters using wide IPs, pools, and listeners.
3. Instructions for troubleshooting DNS resolution and iQuery communication between BIG-IP systems in the same sync group.
When the debugging of modern software is required, basic GDB techniques are insufficient, but new techniques can be created from the nearly 160 commands available in GDB. "Modern software" refers to multi-threading, using STL and other libraries, IPC, signals and exception
mechanisms. In this lecture, techniques for debugging large, modern software written in C++ will be explained.
Finding target for hacking on internet is now easierDavid Thomas
Finding target on internet for penetration testing involves searching internet using google or using Google Hacking/Dorking. There are google hacking queries available on internet, according to ethical hacking researcher of International Institute of Cyber Security it is the main source of passive attacks on internet. This whole process of finding target on internet using GHDB is automated using python based framework named as Katana framework.
OSSNA 2017 Performance Analysis Superpowers with Linux BPFBrendan Gregg
Talk by Brendan Gregg for OSSNA 2017. "Advanced performance observability and debugging have arrived built into the Linux 4.x series, thanks to enhancements to Berkeley Packet Filter (BPF, or eBPF) and the repurposing of its sandboxed virtual machine to provide programmatic capabilities to system tracing. Netflix has been investigating its use for new observability tools, monitoring, security uses, and more. This talk will be a dive deep on these new tracing, observability, and debugging capabilities, which sooner or later will be available to everyone who uses Linux. Whether you’re doing analysis over an ssh session, or via a monitoring GUI, BPF can be used to provide an efficient, custom, and deep level of detail into system and application performance.
This talk will also demonstrate the new open source tools that have been developed, which make use of kernel- and user-level dynamic tracing (kprobes and uprobes), and kernel- and user-level static tracing (tracepoints). These tools provide new insights for file system and storage performance, CPU scheduler performance, TCP performance, and a whole lot more. This is a major turning point for Linux systems engineering, as custom advanced performance instrumentation can be used safely in production environments, powering a new generation of tools and visualizations."
Multiplatform JIT Code Generator for NetBSD by Alexander Nasonoveurobsdcon
Abstract
The next release of NetBSD will have a support for Just-In-Time (JIT) compilation of bpf programs in the kernel; this change will greatly speed-up traffic sniffing on multiple platforms. Unlike similar interface in other operating systems, bpfjit uses a unified programming interface for code generation which is based on Stack Less JIT Compiler library (SLJIT) and which supports x86, mips, arm, sparc and some other platforms.
The speaker will give an overview of SLJIT API and discuss some implementation details of the bpfjit code with emphasis on supported optimizations of bpf programs by JIT engine. He will also touch on unit testing of dynamically generated code running inside the kernel and on other areas in the NetBSD project where bpfjit can help in boosting performance."
Speaker bio
Alex is a software developer working in the financial sector in the City of London. He often amuses fellow tube passengers with C or Lua coding in NetBSD console and sometimes even with the green kernel debugger prompt.
Profiling in Python provides concise summaries of key profiling tools in 3 sentences:
cProfile and line_profiler profile execution time and identify slow lines of code. memory_profiler profiles memory usage with line-by-line or time-based outputs. YEP extends profiling to compiled C/C++ extensions like Cython modules, which are not covered by the standard Python profilers.
Relational Database Access with Python ‘sans’ ORM Mark Rees
This document discusses various approaches for accessing relational databases from Python, including ORM libraries like Django and SQLAlchemy, raw SQL queries using the Python DB-API standard, and template libraries like SpringPython. It provides code examples for common database operations like SELECT, INSERT, and comparing database schemas. A number of Python database adapters are also described, supporting databases like PostgreSQL, MySQL, SQLite and more.
This document discusses using Git hooks to automate tasks during the Git workflow. It provides examples of using pre-commit hooks to check for debugging code and validate commit messages include ticket numbers. prepare-commit-msg hooks can template commit messages with default text like ticket references. post-commit hooks may open a browser to log work time against a referenced ticket after a commit. Server-side hooks on push can trigger tasks like running tests. Overall the document explores leveraging Git hooks for tasks like validation, templating, integration and to enforce best practices when committing code changes.
Explains how Docker and Nix work as deployment solutions, in what ways they are similar and different, and how they can be combined to achieve interesting results.
Developing with the Go client for Apache KafkaJoe Stein
This document summarizes Joe Stein's go_kafka_client GitHub repository, which provides a Kafka client library written in Go. It describes the motivation for creating a new Go Kafka client, how to use producers and consumers with the library, and distributed processing patterns like mirroring and reactive streams. The client aims to be lightweight with few dependencies while supporting real-world use cases for Kafka producers and high-level consumers.
The document discusses process IDs (PIDs) and PID namespaces in Linux containers. Some key points:
- In a new PID namespace, the first process created has PID 1. This process typically reaps orphaned child processes.
- Using a shell like /bin/sh as PID 1 can cause signals not to be properly propagated to child processes.
- Minimal init processes like Tini and Dumb-init act as PID 1 and properly handle reaping orphaned processes and propagating signals.
- Docker 1.13 introduced the --init flag to run an init process and improve process cleanup and signaling.
- While systemd can run inside containers, it provides more functionality than typically needed for
This document provides an overview of using the GNU Debugger (GDB) tool for debugging programs on an NVIDIA Jetson TX2 embedded system. It discusses setting breakpoints in code, running a program under GDB, examining variables and source code, and includes an example C code for demonstrating GDB commands. The document also briefly introduces concepts like stack memory and differences between arrays and stacks.
Custom CI/CD pipelines often don’t adapt well to existing platforms. OCluster is a cluster management system that’s currently deployed on thousands of cores powering various CI systems for the OCaml community. It composes recognized software components, from snapshotting filesystems to containers and virtualization management, in order to provide a lightweight and highly portable execution environment across operating systems and architectures. In this talk, we’ll present OCluster use cases, and how it fits into the OCurrent framework.
Goal of this talk is to get through performance analysis process of simple calculator application implemented as a set of microservices with following FOSS toolset: k6 and gatling for traffic generation, prometheus for monitoring, perf Linux profiler and FlameGraph project for flame graphs generation.
This document provides instructions for using wgrib2, an open source software tool for managing GRIB weather and climate data files. It answers various questions about how to extract, filter, and modify data from GRIB files using wgrib2 commands. Examples are given for converting files to binary, extracting specific records, filtering by date or forecast time ranges, combining files, masking data, and more. The document also provides tips for working with issues like reordering fields and aligning wind components correctly.
CDO (Climate Data Operators) is a free and open-source software that provides over 600 command line operators for performing statistical analysis and data manipulation on climate and weather data. It allows users to select, subsample, interpolate, perform arithmetic operations and simple statistical analysis on netcdf and grib files. Some basic CDO commands include using conda to install CDO, using cdo info to get file information, cdo merge to combine files, and cdo copy to change file formats.
Python is a simple yet powerful programming language that can be used for a wide range of tasks. It has an easy to read syntax and is used by developers for applications such as text processing, web development, science and math, and more. Python code is interpreted at runtime and uses indentation to define code blocks, making the code very readable. Some key features of Python include being free and open source, having a large standard library, and supporting object-oriented, imperative, and functional programming styles.
Python is a simple yet powerful programming language that can be used for a wide range of applications from web development to data science. It has an easy to read syntax and is easy to learn. Some key features of Python include being interpreted, having dynamic typing, automatic memory management, and being open source. Python can be used for tasks like text processing, system administration, GUI programming, web applications, databases, scientific computing, and more. It has a large standard library and can interface with other languages like C/C++ and Java.
Thermohaline Circulation & Climate ChangeArulalan T
Today I have presented "The Thermohaline Circulation and Climate Change" as Mini-Project for our Science of Climate Change Course ! We can expect THC shutdown around 2050s... OMG ! Yes, we can expect "The Day After Tomorrow" around 2100... All the images credited to the reference papers except one T-S-Sigmat created by me using CDAT5.2.
Python is a simple yet powerful programming language that can be used for a wide range of tasks. It is easy to learn, free and open source. Python code is easy to read due to its clear syntax and use of indentation. It supports object-oriented programming and functional programming. Python can be used for web development, scientific computing, data analysis and more. It has a large standard library and can interface with other languages like C.
This document provides an introduction to unit testing in Python and the Pytest framework. It discusses writing unit test code using the unittest framework and Pytest, how to test functions and handle failures. It also covers features of Pytest like skipping and expected failures, fixture setup/teardown, and distributed testing using the xdist plugin.
Python is a simple yet powerful programming language that can be used for a wide range of tasks. The document provides an introduction to Python, discussing why it is a good language to learn, its history and examples of common uses. It covers Python's syntax, data types, modules, object oriented programming and how to interface Python with databases and other languages. The goal is to illustrate Python's versatility and ease of use through examples.
The document provides an introduction to the Python programming language. It discusses that Python is a simple, easy to learn, open source programming language. It can be used for tasks like text handling, games, system administration, GUI programming, web applications, and scientific applications. The document then covers Python's history, what Python is, examples of Python code, data types like lists and dictionaries, functions, classes and object oriented programming, file handling, database integration, and how to learn Python. It also discusses how Python can interface with other languages like C, Java, and building GUI applications.
1. The document describes a project to visualize contour lines and plot dynamic isotherm lines over India in CDAT.
2. It involves web scraping current temperature data, generating contour levels, and drawing isotherm lines on the CDAT India map.
3. The project aims to make isolines in CDAT dynamic and allow users to create their own, as well as combine data from Arduino sensors to generate 3D contour visualizations of varying temperatures.
Closely spaced isobars or isohypses indicate a tight pressure gradient, which implies stronger winds. A tight pressure gradient means the pressure is changing rapidly over small distances. Since wind is caused by horizontal pressure differences, a tight gradient results in stronger winds as air moves from high to low pressure. Meteorologists analyze isobar/isohypse spacing to identify areas of stronger winds.
Q2. How can isotherms help identify areas of warm or cold air advection?
Answer: When isotherms are bunched closer together, it indicates warm air advection. When they are farther apart, it indicates cold air advection.
This document describes a student project to analyze contour lines and visualize them using the Climate Data Analysis Tool (CDAT). The project is conducted by Arulalan.T, a final year student at Saveetha Engg College, under the guidance of Mr. Thyagarajan Shanmugham. It thanks Mr. Charles Doutriaux of CDAT and Dr. Krishna AchutaRao of IIT Delhi for their support. The document provides instructions on installing CDAT, describes sample programs to create contour plots of cloud data over world and India maps, and explains how to edit the CDAT map and add custom contour lines by modifying netcdf files. It demonstrates creating isotherm lines over India at
Edit/correct India Map In Cdat Documentation - With Edited World Map Data Arulalan T
The document discusses how to edit the world map in CDAT to correctly depict the India political map. It describes downloading source data for the world map from CDAT, converting it to text format, and editing the data to correctly draw the borders of India and regions like Pakistan-occupied Kashmir. The edited map data is provided that draws the accurate borders of India when used in CDAT plotting.
This document provides an overview of nomography, which is the graphical representation of mathematical relationships using graphs called nomograms. Nomograms were commonly used before calculators to solve equations graphically. The document discusses different types of nomograms including parallel scale, Z, proportional, concurrent scale, and multi-variable nomograms. It explains how to construct these nomograms geometrically and set their scaling factors to represent different equations.
matplotlib-installatin-interactive-contour-example-guideArulalan T
This document provides instructions for installing Matplotlib and examples of interactive contour plotting in 3D using Matplotlib. It describes downloading and installing dependencies like NumPy, libpng, and freetype. It then explains downloading and installing Matplotlib. Two examples are given of interactive contour plotting where the contour levels can be changed: one takes input at the command line, the other reads levels from a file. The output demonstrates changing the contour levels in the 3D plot to see how it is updated.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfTechgropse Pvt.Ltd.
In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providersakankshawande
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Pygrib documentation
1. PyGrib Documentation
– by
Arulalan.T
arulalant@gmail.com
Project Associate,
Centre for Atmospheric Sciences ,
Indian Institute of Technology Delhi.
03.05.2011
-1-
2. Pygrib Installation in CDAT path procedure
Now we are going to see the procedure how to install PyGrib [1] inside the CDAT [2] installed path in
our system.
[1] http://code.google.com/p/pygrib/
[2] http://www2pcmdi.llnl.gov/cdat
Let us assume we have installed CDAT in our system and its working path is /usr/local/cdat5.2/bin/ .
Make sure that the path working fine as like below….
$ /usr/local/cdat5.2//bin/python
Executing /usr/local/cdat5.2//bin/python
Python 2.5.2 (r252:60911, Feb 2 2011, 15:29:23)
[GCC 4.4.5] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>>
To install PyGrib in your normal python path, kindly follow the link
http://code.google.com/p/pygrib/wiki/LinuxMacInstallation
First Install the following dependencies for pygrib in our ubuntu.
$ sudo aptget install libjasperdev libjasperruntime .
Note : Do Not install libjasperjava, jasper and openjpegtools.
If you installed these above 3 packages, then it should crash our ubuntu login and its X.
These are all stuff tested in ubuntu 10.10.
libjasperdev and libjasperruntime are working fine without making any problem.
Download grib_api1.9.5.tar.gz from http://www.ecmwf.int/products/data/software/download/grib_api.html
Download pyproj1.8.8.tar.gz from http://code.google.com/p/pyproj/downloads/list
Download pygrib1.8.0.tar.gz from http://code.google.com/p/pygrib/downloads/list
Now we need to move the three downloaded packages to the cdat path as like below …
$ mv grib_api1.9.5.tar.gz /usr/local/cdat5.2/bin/
$ mv pyproj1.8.8.tar.gz /usr/local/cdat5.2/bin/
$ mv pygrib1.8.0.tar.gz /usr/local/cdat5.2/bin/
Now change the current working directory into cdat .
$ cd /usr/local/cdat5.2/bin/
$ pwd
/usr/local/cdat5.2/bin/
$ su – root
-2-
3. Installing gribapi
Following these instructions should be safe on an average linux machine:
• mkdir grib_api_dir
• tar zxvf grib_api1.9.5.tar.gz
• cd grib_api1.9.5
• export CFLAGS=”O2 fPIC”
• ./configure –prefix= `pwd`/../grib_api_dir
• make
• make check
• make install
• cd ..
• pwd
/usr/local/cdat5.2/bin/
Installing pyproj
This one is very simple, just like the normal python install :
• tar zxvf pyproj1.8.8.tar.gz
• cd pyproj1.8.8
• /usr/local/cdat5.2/bin//python setup.py install
• cd ..
• pwd
/usr/local/cdat5.2/bin/
Installing pygrib
Installing pygrib inside cdat path.
• export GRIBAPI_DIR=`pwd`/grib_api_dir
• export JASPER_DIR=/usr
• tar zxvf pygrib1.8.0.tar.gz
• cd pygrib1.8.0
• /usr/local/cdat5.2/bin//python setup.py install
• /usr/local/cdat5.2/bin//python test.py
• pwd
/usr/local/cdat5.2/bin/pygrib
Checking pygrib in CDAT path
open a new terminal as a normal user
• $ /usr/local/cdat5.2//bin/pythonExecuting /usr/local/cdat5.2//bin/python
Python 2.5.2 (r252:60911, Feb 2 2011, 15:29:23)
[GCC 4.4.5] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
>>> import pygrib
>>>
Thats it. We have successfully installed pygrib in our cdat path and imported it.
-3-
4. Pygrib Usage
• from the python interpreter prompt, import the package:
>>> import pygrib
• open a GRIB file, create a grib message iterator:
>>> grbs = pygrib.open('sampledata/flux.grb')
• pygrib open instances behave like regular python file objects,
with seek, tell, read, readline and close methods, except that offsets are measured in grib messages
instead of bytes:
>>> grbs.seek(2)
>>> grbs.tell()
2
>>> grb = grbs.read(1)[0] # read returns a list with the next N (N=1 in this case) messages.
>>> grb # printing a grib message object displays summary info
3:Maximum temperature:K (instant):regular_gg:heightAboveGround:level 2 m:fcst time 108
120:from 200402291200
>>> grbs.tell()
3
• print an inventory of the file:
>>> grbs.seek(0)
>>> for grb in grbs:
>>> grb
1:Precipitation rate:kg m**2 s**1 (avg):regular_gg:surface:level 0:fcst time 108120:from
200402291200
2:Surface pressure:Pa (instant):regular_gg:surface:level 0:fcst time 120:from 200402291200
3:Maximum temperature:K (instant):regular_gg:heightAboveGround:level 2 m:fcst time 108
120:from 200402291200
4:Minimum temperature:K (instant):regular_gg:heightAboveGround:level 2 m:fcst time 108
120:from 200402291200
• find the first grib message with a matching name:
>>> grb = grbs.select(name='Maximum temperature')[0]
• extract the data values using the 'values' key (grb.keys() will return a list of the available keys):
# The data is returned as a numpy array, or if missing values or a bitmap
# are present, a numpy masked array. Reduced lat/lon or gaussian grid
# data is automatically expanded to a regular grid. Details of the internal
# representation of the grib data (such as the scanning mode) are handled
# automatically.
>>> data = grb.values # same as grb['values']
>>> data.shape, data.min(), data.max()
(94, 192) 223.7 319.9
• get the latitudes and longitudes of the grid:
>>> lats, lons = grb.latlons()
>>> lats.shape, lats.min(), lats.max(), lons.shape, lons.min(), lons.max()
(94, 192) 88.5419501373 88.5419501373 0.0 358.125
-4-
5. • get the second grib message:
>>> grb = grbs.message(2) # same as grbs.seek(1); grb=grbs.readline()
>>> grb
2:Surface pressure:Pa (instant):regular_gg:surface:level 0:fcst time 120:from 200402291200
• modify the values associated with existing keys (either via attribute or dictionary access):
>>> grb['forecastTime'] = 240
>>> grb.dataDate = 20100101
• get the binary string associated with the coded message:
>>> msg = grb.tostring()
>>> grbs.close() # close the grib file.
• write the modified message to a new GRIB file:
>>> grbout = open('test.grb','wb')
>>> grbout.write(msg)
>>> grbout.close()
>>> pygrib.open('test.grb').readline()
1:Surface pressure:Pa (instant):regular_gg:surface:level 0:fcst time 240:from 201001011200
Few More examples
• To open
>>> grbs = pygrib.open('gdas1.t00z.grbanl')
• To get the filename
>>> grbs.name
'gdas1.t00z.grbanl'
• To get how many messages/lines it has
>>> grbs.messages
361
• To get message of particular line no
>>> grbs[5]
5:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 900:fcst time 0:from 201005250000
>>> grbs.message(250)
250:V component of wind:m s**1 (instant):regular_ll:pressureFromGroundLayer:lev el None:fcst time 0:from 201005250000
Note : grb message numbers start at 1
• To get the next message from current position
>>> grbs.next()
251:V component of wind:m s**1 (instant):regular_ll:pressureFromGroundLayer:lev el None:fcst time 0:from 201005250000
• To move the file pointer to the first position
>>> grbs.seek(0)
>>> grbs.next()
1:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 1000:fcst tim e 0:from 201005250000
• To get the first message
>>> g=grbs[1]
>>> g.messagenumber
1
-5-
6. • To get the keys of the message
>>> g.keys()
['parametersVersion', 'definitionFilesVersion', 'two', 'three', 'truncateDegrees', 'offset', 'count', 'countTotal', 'unitsFactor', 'unitsBias', 'eps', 'globalDomain', 'libraryVersion',
'kindOfProdu ct', 'GRIBEditionNumber', 'offsetSection0', 'section0Length', 'totalLength', 'editionNumber', 'WMO', 'productionStatusOfProcessedData', 'section1Length', 'table2Version',
'centre', 'generatingProce ssIdentifier', 'gridDefinition', 'indicatorOfParameter', 'parameterName', 'parameterUnits', 'indicatorOfTypeOfLevel', 'pressureUnits', 'typeOfLevel', 'level',
'yearOfCentury', 'month', 'day', 'hour ', 'minute', 'second', 'unitOfTimeRange', 'P1', 'P2', 'timeRangeIndicator', 'numberIncludedInAverage','numberMissingFromAveragesOrAccumulations',
'centuryOfReferenceTimeOfData', 'subCentre', 'para mIdECMF', 'paramId', 'cfName', 'units', 'name', 'decimalScaleFactor', 'setLocalDefinition', 'dataDate', 'year', 'dataTime', 'julianDay',
'stepUnits', 'stepType', 'stepRange', 'startStep', 'endStep' , 'marsParam', 'validityDate', 'validityTime', 'wrongPadding', 'deleteLocalDefinition', 'localUsePresent', 'shortNameECMF',
'shortName', 'ifsParam', 'gridDescriptionSectionPresent', 'bitmapPresent' , 'section2Length', 'radius', 'shapeOfTheEarth', 'numberOfVerticalCoordinateValues', 'neitherPresent',
'pvlLocation', 'dataRepresentationType', 'gridDefinitionTemplateNumber', 'Ni', 'Nj', 'latitude OfFirstGridPoint', 'latitudeOfFirstGridPointInDegrees', 'longitudeOfFirstGridPoint',
'longitudeOfFirstGridPointInDegrees', 'resolutionAndComponentFlags', 'ijDirectionIncrementGiven', 'earthIsOblate ', 'resolutionAndComponentFlags3', 'resolutionAndComponentFlags4',
'uvRelativeToGrid', 'resolutionAndComponentFlags6', 'resolutionAndComponentFlags7', 'resolutionAndComponentFlags8', 'latitudeOfLas tGridPoint', 'latitudeOfLastGridPointInDegrees',
'longitudeOfLastGridPoint', 'longitudeOfLastGridPointInDegrees', 'iDirectionIncrement', 'jDirectionIncrement', 'scanningMode', 'iScansNegatively', ' jScansPositively',
'jPointsAreConsecutive', 'alternativeRowScanning', 'iScansPositively', 'scanningMode4', 'scanningMode5', 'scanningMode6', 'scanningMode7', 'scanningMode8',
'jDirectionIncrementInDegrees', 'iDirectionIncrementInDegrees', 'numberOfDataPoints', 'numberOfValues', 'latLonValues', 'latitudes', 'longitudes', 'distinctLatitudes', 'distinctLongitudes',
'PVPresent', 'PLPresent', 'le ngthOfHeaders', 'missingValue', 'tableReference', 'section4Length', 'halfByte', 'dataFlag', 'binaryScaleFactor', 'referenceValue', 'bitsPerValue',
'referenceValueError', 'sphericalHarmonics', 'comp lexPacking', 'integerPointValues', 'additionalFlagPresent', 'packingType', 'bitMapIndicator', 'values', 'numberOfCodedValues',
'packingError', 'unpackedError', 'maximum', 'minimum', 'average', 'num berOfMissing', 'standardDeviation', 'skewness', 'kurtosis', 'isConstant', 'dataLength', 'changeDecimalPrecision',
'decimalPrecision', 'bitsPerValueAndRepack', 'scaleValuesBy', 'offsetValuesBy', 'gr idType', 'getNumberOfValues', 'section5Length']
From the above keys we can choose any one and pass it to the g var as shown like below
• To get the data of the message
>>> g['values']
array([[ 289.1, 289.1, 289.1, ..., 289.1, 289.1, 289.1],
[ 300.8, 300.8, 300.8, ..., 301. , 300.9, 300.9],
[ 293.7, 293.6, 293.6, ..., 293.7, 293.7, 293.7],
...,
[ 231.2, 231.3, 231.4, ..., 231. , 231.1, 231.1],
[ 232.2, 232.3, 232.3, ..., 232.1, 232.1, 232.2],
[ 233. , 233. , 233. , ..., 233. , 233. , 233. ]])
• To get the missing Value of the message
>>> g['missingValue']
9999
• To get the level of the message
>>> g['level']
1000
• To get the latitudes as numpy array
>>> g['distinctLatitudes']
array([90. , 89.5, 89. , 88.5, 88. , 87.5, 87. , 86.5, 86. ,
…, …, …,
85.5, 86. , 86.5, 87. , 87.5, 88. , 88.5, 89. , 89.5, 90. ])
• To get the longitudes as numpy array
>>> g['distinctLongitudes']
array([ 0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5,
…, …, …,
356. , 356.5, 357. , 357.5, 358. , 358.5, 359. , 359.5])
• To get both latitudes and longitudes
>>> g.latlons()
(array([[90. , 90. , 90. , ..., 90. , 90. , 90. ],
...,
[ 90. , 90. , 90. , ..., 90. , 90. , 90. ]]), array([[ 0. , 0.5, 1. , ..., 358.5, 359. , 359.5],
...,
[ 0. , 0.5, 1. , ..., 358.5, 359. , 359.5]]))
-6-
7. • Using for loop
>>> for i in grbs:
print i
1:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 1000:fcst time 0:from 201005250000
2:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 975:fcst time 0:from 201005250000
3:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 950:fcst time 0:from 201005250000
…
…
360:V component of wind:m s**1 (instant):regular_ll:maxWind:level 0:fcst time 0:from 201005250000
361:V component of wind:m s**1 (instant):regular_ll:sigma:level 9950:fcst time 0:from 201005250000
• To get data and other properties belongs to one gribfile, variableName, typeOfLevel, and level
by fastest way using the file index
>>> file_name = 'gdas1.t00z.grbanl'
>>> fileidx = pygrib.index(file_name,'name','typeOfLevel','level')
>>> g = fileidx.select(name = "Geopotential Height", typeOfLevel = "isobaricInhPa", level = 1000)
>>> g
[1:Geopotential Height:gpm (instant):regular_ll:isobaricInhPa:level 1000:fcst time 0:from 201005250000]
>>> g[0]['values']
array([[ 289.1, 289.1, 289.1, ..., 289.1, 289.1, 289.1],
[ 300.8, 300.8, 300.8, ..., 301. , 300.9, 300.9],
...,
[ 233. , 233. , 233. , ..., 233. , 233. , 233. ]])
Notes:
1. Here this 'fileidx' belongs to the file_name = 'gdas1.t00z.grbanl'
2. fileindex is more faster than pygrib.open() method
3. In fileindex we can set the arguments maximum only the following three options ...
'name','typeOfLevel','level'
Links :
• Home Page : http://code.google.com/p/pygrib/
• Documentation : http://pygrib.googlecode.com/svn/trunk/docs/index.html
• Installation in System Python Path : http://code.google.com/p/pygrib/wiki/LinuxMacInstallation
• Installation in CDAT Path : http://tuxcoder.wordpress.com/2011/05/03/howtoinstallpygribinside
ourcdatpath/
-7-