Tech talk about performance tools provided with standard go distribution given at go meetup group in Seattle,
http://www.meetup.com/golang/events/231455969/
The document provides an introduction to Tsung, an open source load testing tool for HTTP and other protocols. It discusses Tsung's key features including being highly concurrent, supporting multiple protocols, easy setup, and distributed testing capabilities. It also summarizes the phases of a Tsung load test and provides examples of using Tsung to test HTTP services and monitor servers under load.
Tsung is an open source tool for load testing applications. It can perform load testing from a single machine or in a distributed manner across multiple machines. The document discusses how to install Tsung and Erlang, configure a basic load test using a test.xml file, run the test, and view reports. It also covers how to set up distributed load testing using multiple client nodes and load balancing across Erlang VMs. Load testing WebSocket applications with Tsung is also briefly mentioned.
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
New Ways to Find Latency in Linux Using TracingScyllaDB
Ftrace is the official tracer of the Linux kernel. It originated from the real-time patch (now known as PREEMPT_RT), as developing an operating system for real-time use requires deep insight and transparency of the happenings of the kernel. Not only was tracing useful for debugging, but it was critical for finding areas in the kernel that was causing unbounded latency. It's no wonder why the ftrace infrastructure has a lot of tooling for seeking out latency. Ftrace was introduced into mainline Linux in 2008, and several talks have been done on how to utilize its tracing features. But a lot has happened in the past few years that makes the tooling for finding latency much simpler. Other talks at P99 will discuss the new ftrace tracers "osnoise" and "timerlat", but this talk will focus more on the new flexible and dynamic aspects of ftrace that facilitates finding latency issues which are more specific to your needs. Some of this work may still be in a proof of concept stage, but this talk will give you the advantage of knowing what tools will be available to you in the coming year.
This document provides an overview of Docker commands and architecture including hub/image commands, container commands, and references. It discusses pulling and running the tutum/mongodb image exposing ports 27017 and 28017 and setting environment variables for username, password, and database name. Kitematic is also mentioned.
This presentation is for Go developers and operators of Go applications who are interested in reducing costs and latency, or debugging problems such as memory leaks, infinite loops, performance regressions, etc. of such applications. We'll start with a brief description of the unique aspects of the Go runtime, and then take a look at the builtin profilers as well as Go's execution tracer. Additionally we'll look at the interoperability with popular observability tools such as Linux perf and bpftrace. After this presentation you should have a good idea of the various tools you can use, and which ones might be the most useful to you in a production environment.
Using eBPF to Measure the k8s Cluster HealthScyllaDB
As a k8s cluster-admin your app teams have a certain expectation of your cluster to be available to deploy services at any time without problems. While there is no shortage on metrics in k8s its important to have the right metrics to alert on issues and giving you enough data to react to potential availability issues. Prometheus has become a standard and sheds light on the inner behaviour of Kubernetes clusters and workloads. Lots of KPIs (CPU, IO, network. Etc) in our On-Premise environment are less precise when we start to work in a Cloud environment. Ebpf is the perfect technology that fulfills that requirement as it gives us information down to the kernel level. In 2018 Cloudflare shared an opensource project to expose custom ebpf metrics in Prometheus. Join this session and learn about: • What is ebpf? • What type of metrics we can collect? • How to expose those metrics in a K8s environment. This session will try to deliver a step-by-step guide on how to take advantage of the ebpf exporter.
The document provides an introduction to Tsung, an open source load testing tool for HTTP and other protocols. It discusses Tsung's key features including being highly concurrent, supporting multiple protocols, easy setup, and distributed testing capabilities. It also summarizes the phases of a Tsung load test and provides examples of using Tsung to test HTTP services and monitor servers under load.
Tsung is an open source tool for load testing applications. It can perform load testing from a single machine or in a distributed manner across multiple machines. The document discusses how to install Tsung and Erlang, configure a basic load test using a test.xml file, run the test, and view reports. It also covers how to set up distributed load testing using multiple client nodes and load balancing across Erlang VMs. Load testing WebSocket applications with Tsung is also briefly mentioned.
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
New Ways to Find Latency in Linux Using TracingScyllaDB
Ftrace is the official tracer of the Linux kernel. It originated from the real-time patch (now known as PREEMPT_RT), as developing an operating system for real-time use requires deep insight and transparency of the happenings of the kernel. Not only was tracing useful for debugging, but it was critical for finding areas in the kernel that was causing unbounded latency. It's no wonder why the ftrace infrastructure has a lot of tooling for seeking out latency. Ftrace was introduced into mainline Linux in 2008, and several talks have been done on how to utilize its tracing features. But a lot has happened in the past few years that makes the tooling for finding latency much simpler. Other talks at P99 will discuss the new ftrace tracers "osnoise" and "timerlat", but this talk will focus more on the new flexible and dynamic aspects of ftrace that facilitates finding latency issues which are more specific to your needs. Some of this work may still be in a proof of concept stage, but this talk will give you the advantage of knowing what tools will be available to you in the coming year.
This document provides an overview of Docker commands and architecture including hub/image commands, container commands, and references. It discusses pulling and running the tutum/mongodb image exposing ports 27017 and 28017 and setting environment variables for username, password, and database name. Kitematic is also mentioned.
This presentation is for Go developers and operators of Go applications who are interested in reducing costs and latency, or debugging problems such as memory leaks, infinite loops, performance regressions, etc. of such applications. We'll start with a brief description of the unique aspects of the Go runtime, and then take a look at the builtin profilers as well as Go's execution tracer. Additionally we'll look at the interoperability with popular observability tools such as Linux perf and bpftrace. After this presentation you should have a good idea of the various tools you can use, and which ones might be the most useful to you in a production environment.
Using eBPF to Measure the k8s Cluster HealthScyllaDB
As a k8s cluster-admin your app teams have a certain expectation of your cluster to be available to deploy services at any time without problems. While there is no shortage on metrics in k8s its important to have the right metrics to alert on issues and giving you enough data to react to potential availability issues. Prometheus has become a standard and sheds light on the inner behaviour of Kubernetes clusters and workloads. Lots of KPIs (CPU, IO, network. Etc) in our On-Premise environment are less precise when we start to work in a Cloud environment. Ebpf is the perfect technology that fulfills that requirement as it gives us information down to the kernel level. In 2018 Cloudflare shared an opensource project to expose custom ebpf metrics in Prometheus. Join this session and learn about: • What is ebpf? • What type of metrics we can collect? • How to expose those metrics in a K8s environment. This session will try to deliver a step-by-step guide on how to take advantage of the ebpf exporter.
We describe a modification to the Linux Kernel which gives an SRE control over the combined bandwidth of logging on a node of a distributed system, while providing a way for the logging source owner (container or service) to control what happens when the bandwidth limit is hit.
Tips on High Performance Server ProgrammingJoshua Zhu
This document provides tips and best practices for high performance server programming. It discusses avoiding blocking, using efficient algorithms and data structures, separating I/O from business logic, and tuning for bottlenecks. It also covers various I/O models like blocking, non-blocking, and asynchronous I/O. Key aspects of designing high performance servers include using non-blocking I/O, event-driven architectures, and avoiding excessive threading.
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...Rajni Baliyan
This document provides an overview of logical replication in PostgreSQL 10. It discusses the history of PostgreSQL replication, the key concepts of logical replication including publications, subscriptions, and replication slots. It presents several use cases for logical replication such as cross-platform, cross-version, and write operations at the subscriber. The document also covers configuration settings, a quick setup demonstration, monitoring, resolving conflicts, and limitations of logical replication.
The document discusses using plProxy and pgBouncer to split a PostgreSQL database horizontally and vertically to improve scalability. It describes how plProxy allows functions to make remote calls to other databases and how pgBouncer can be used for connection pooling. The RUN ON clause of plProxy is also summarized, which allows queries to execute on all partitions or on a specific partition.
The document outlines the content and schedule for a JavaScript code academy that meets weekly. It introduces topics like React.js, unit testing, and managing application state that will be covered over 8 sessions starting at 6pm on Wednesdays for 20 minutes of Q&A followed by a presentation and coding. The introduction covers setting up Git, GitHub, and the discussion forum where code and materials will be shared. It also provides a brief history of JavaScript and an overview of JavaScript basics like functions, scope, higher order functions, and the 'this' identifier before concluding with instructions on setting up a development environment and creating a first project using NPM and package.json.
DB Latency Using DRAM + PMem in App Direct & Memory ModesScyllaDB
How does the latency of DDR4 DRAM compare to Intel Optane Persistent Memory when used in both App Direct and Memory Modes for In-Memory database access?
This talk is about the latency benchmarks that I performed by adding gettimeofday() calls around critical DB kernel operations.
This talk covers the technology, cache hit ratios, lots of histograms and lessons learned.
This slide will make the assumption that you have never used a task runner before and will walk through every step required to get up and running with gulp.
Haroon walked us through various tips and tricks on how we can enhance PostgreSQL performance while highlighting some typical pitfalls people encounter. If you are planning for capacity, doing scalability analysis, or simply facing degradation in performance of your apps or queries running against PostgreSQL, you should definitely attend this session.
This document discusses Salt Stack, an open source configuration management and remote execution system. Salt Stack allows users to manage configuration of systems through states written in YAML and execute commands remotely. It uses a master-minion architecture with asynchronous communication over ZeroMQ. Minions connect to the master and subscribe to events to receive configuration updates. Salt Stack provides modules to manage users, packages, files, services and more. It offers a simple and fast way to provision and manage systems at scale.
The document summarizes the results of benchmarking and comparing the performance of PostgreSQL databases hosted on Amazon EC2, RDS, and Heroku. It finds that EC2 provides the most configuration options but requires more management, RDS offers simplified deployment but less configuration options, and Heroku requires no management but has limited configuration and higher costs. Benchmark results show EC2 performing best for raw performance while RDS and Heroku trade off some performance for manageability. Heroku was the most expensive option.
Linux Talk: Deep Dive into Scheduled Tasks is a guide about cron, anacron, and incron. It discusses the importance of scheduled tasks, what cron is and how it works using crontab files. It covers special cron syntax, system-wide cron, crontab restrictions, and importing cron jobs. Anacron is described as an alternative to cron for non-24/7 systems. Incron is introduced as a way to trigger commands based on file events instead of time schedules. The document provides an overview of scheduling tools and commands commonly used on Linux systems.
OSNoise Tracer: Who Is Stealing My CPU Time?ScyllaDB
In the context of high-performance computing (HPC), the Operating System Noise (osnoise) refers to the interference experienced by an application due to activities inside the operating system. In the context of Linux, NMIs, IRQs, softirqs, and any other system thread can cause noise to the application. Moreover, hardware-related jobs can also cause noise, for example, via SMIs.
HPC users and developers that care about every microsecond stolen by the OS need not only a precise way to measure the osnoise but mainly to figure out who is stealing cpu time so that they can pursue the perfect tune of the system. These users and developers are the inspiration of Linux's osnoise tracer.
The osnoise tracer runs an in-kernel loop measuring how much time is available. It does it with preemption, softirq and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise tracer takes note of the entry and exit point of any source of interferences. When the noise happens without any interference from the operating system level, the tracer can safely point to a hardware-related noise. In this way, osnoise can account for any source of interference. The osnoise tracer also adds new kernel tracepoints that auxiliaries the user to point to the culprits of the noise in a precise and intuitive way.
At the end of a period, the osnoise tracer prints the sum of all noise, the max single noise, the percentage of CPU available for the thread, and the counters for the noise sources, serving as a benchmark tool.
Linux fundamental - Chap 15 Job SchedulingKenny (netman)
This document discusses different methods for automating system administration tasks by scheduling jobs, including cron, anacron, and at. Cron executes tasks on a recurring schedule, while anacron runs jobs that may have been missed when the system was offline. The at command executes a one-time job at a specified time in the future. Configuration files like cron.allow and cron.deny control which users can schedule jobs with these tools.
Scalability strategies for cloud based system architectureSangJin Kang
- Scalability & Availability for the Global Markets
- Global scaled Scalability, Availability and Security
- Architecture for 100, 1K, 100K, 500K, 1M and 10M global users
- Auto-Scaling
- Understand Cloud Services
- Cloud Demo(AWS, GCP, Azure and Cloudflare)
- Wrap-Up
A talk I gave at the Boston Web Performance Meetup in August 2014.
Performance is one of the most challenging issues in modern web app design, in large part because modeling, testing, and validating performance before deploying to production is so challenging. While many ops teams have nailed down the problem of re-creating pre-production environments that closely mimic production, those environments frequently rely on known-good components beyond the application code itself: AWS ELB, F5 load balancers, CDNs, Varnish, and more.
Testing plug-in components like that can be challenging, because their performance characteristics don't directly align with application metrics.
- How many simultaneous users can my load balancer support? - What sort of network load will I put on my CDN (i.e., how much will it cost?) - How do different user behavior patterns affect performance?
In this meetup, we'll introduce a novel tool in this toolbox: tcpreplay, an open-source tool for replaying packet capture files back at an application. By replaying user traffic to a staging environment, you can test the effects of
- Network saturation to the load balancer - High numbers of users / IPs - Lots of traffic to your other monitoring tools!
Gulp is a task runner that automates and enhances development workflows. It is commonly used to spin up web servers, automatically reload browsers when files are saved, use preprocessors like Sass and LESS, and optimize assets like CSS, JavaScript, and images using plugins like gulp-webserver, gulp-livereload, gulp-compass, and gulp-concat.
The document discusses techniques and tools for optimizing Rails applications. It covers topics like benchmarking tools, caching, session storage options, and common performance issues in Rails like slow helper methods and associations. The document provides recommendations on optimizing actions, views, and controllers in Rails.
Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 InstanceScyllaDB
In this talk I will walk you through the performance tuning steps that I took to serve 1.2M JSON requests per second from a 4 vCPU c5 instance, using a simple API server written in C.
At the start of the journey the server is capable of a very respectable 224k req/s with the default configuration. Along the way I made extensive use of tools like FlameGraph and bpftrace to measure, analyze, and optimize the entire stack, from the application framework, to the network driver, all the way down to the kernel.
I began this wild adventure without any prior low-level performance optimization experience; but once I started going down the performance tuning rabbit-hole, there was no turning back. Fueled by my curiosity, willingness to learn, and relentless persistence, I was able to boost performance by over 400% and reduce p99 latency by almost 80%.
This document discusses optimizing an Apache Pulsar cluster to handle 10 PB of data per day for a financial customer. Initial estimates showed the cluster would need over 1000 VMs using HDD storage. Various optimizations were implemented, including eliminating the journal, using direct I/O, compression, and C++ client optimizations. This reduced the estimated number of needed VMs to 200 using L-SSD storage per VM. The optimized cluster can now meet the customer's requirements of processing 10 PB of data per day with 3 hours of retention and zone failure protection.
The document describes the structure and workflow of the Bareon functional testing project. It discusses the Python APIs used to control services on the controller and slave nodes, as well as run functional tests in parallel across multiple slave nodes. Tests can validate ramdisk functionality alone or involve rebooting to a tenant image. Logs from the agent and tenant image are sent back to the controller. Tests are grouped and configured to share resources and test specific images, firmware, or nodes. Parallelism is achieved through running tests simultaneously across virtual or physical slave nodes managed by the controller.
EChr - CSR & HR - Presentation Dr/ Nicholas AndreouEChr
This document discusses the relationship between human resource management (HRM) and corporate social responsibility (CSR). It begins by providing background on the evolution of CSR, from focusing on corporate philanthropy to now being seen as a strategic responsibility. The document then examines how HRM practices contributed to the global financial crisis and the social impacts of downsizing. It explores how CSR can help address challenges in HRM perception and processes. Specifically, the document analyzes several CSR frameworks and identifies six key areas for HRM to integrate CSR: employment conditions, organizational development and culture, industrial relations, occupational health and safety, diversity and non-discrimination, and human rights.
We describe a modification to the Linux Kernel which gives an SRE control over the combined bandwidth of logging on a node of a distributed system, while providing a way for the logging source owner (container or service) to control what happens when the bandwidth limit is hit.
Tips on High Performance Server ProgrammingJoshua Zhu
This document provides tips and best practices for high performance server programming. It discusses avoiding blocking, using efficient algorithms and data structures, separating I/O from business logic, and tuning for bottlenecks. It also covers various I/O models like blocking, non-blocking, and asynchronous I/O. Key aspects of designing high performance servers include using non-blocking I/O, event-driven architectures, and avoiding excessive threading.
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...Rajni Baliyan
This document provides an overview of logical replication in PostgreSQL 10. It discusses the history of PostgreSQL replication, the key concepts of logical replication including publications, subscriptions, and replication slots. It presents several use cases for logical replication such as cross-platform, cross-version, and write operations at the subscriber. The document also covers configuration settings, a quick setup demonstration, monitoring, resolving conflicts, and limitations of logical replication.
The document discusses using plProxy and pgBouncer to split a PostgreSQL database horizontally and vertically to improve scalability. It describes how plProxy allows functions to make remote calls to other databases and how pgBouncer can be used for connection pooling. The RUN ON clause of plProxy is also summarized, which allows queries to execute on all partitions or on a specific partition.
The document outlines the content and schedule for a JavaScript code academy that meets weekly. It introduces topics like React.js, unit testing, and managing application state that will be covered over 8 sessions starting at 6pm on Wednesdays for 20 minutes of Q&A followed by a presentation and coding. The introduction covers setting up Git, GitHub, and the discussion forum where code and materials will be shared. It also provides a brief history of JavaScript and an overview of JavaScript basics like functions, scope, higher order functions, and the 'this' identifier before concluding with instructions on setting up a development environment and creating a first project using NPM and package.json.
DB Latency Using DRAM + PMem in App Direct & Memory ModesScyllaDB
How does the latency of DDR4 DRAM compare to Intel Optane Persistent Memory when used in both App Direct and Memory Modes for In-Memory database access?
This talk is about the latency benchmarks that I performed by adding gettimeofday() calls around critical DB kernel operations.
This talk covers the technology, cache hit ratios, lots of histograms and lessons learned.
This slide will make the assumption that you have never used a task runner before and will walk through every step required to get up and running with gulp.
Haroon walked us through various tips and tricks on how we can enhance PostgreSQL performance while highlighting some typical pitfalls people encounter. If you are planning for capacity, doing scalability analysis, or simply facing degradation in performance of your apps or queries running against PostgreSQL, you should definitely attend this session.
This document discusses Salt Stack, an open source configuration management and remote execution system. Salt Stack allows users to manage configuration of systems through states written in YAML and execute commands remotely. It uses a master-minion architecture with asynchronous communication over ZeroMQ. Minions connect to the master and subscribe to events to receive configuration updates. Salt Stack provides modules to manage users, packages, files, services and more. It offers a simple and fast way to provision and manage systems at scale.
The document summarizes the results of benchmarking and comparing the performance of PostgreSQL databases hosted on Amazon EC2, RDS, and Heroku. It finds that EC2 provides the most configuration options but requires more management, RDS offers simplified deployment but less configuration options, and Heroku requires no management but has limited configuration and higher costs. Benchmark results show EC2 performing best for raw performance while RDS and Heroku trade off some performance for manageability. Heroku was the most expensive option.
Linux Talk: Deep Dive into Scheduled Tasks is a guide about cron, anacron, and incron. It discusses the importance of scheduled tasks, what cron is and how it works using crontab files. It covers special cron syntax, system-wide cron, crontab restrictions, and importing cron jobs. Anacron is described as an alternative to cron for non-24/7 systems. Incron is introduced as a way to trigger commands based on file events instead of time schedules. The document provides an overview of scheduling tools and commands commonly used on Linux systems.
OSNoise Tracer: Who Is Stealing My CPU Time?ScyllaDB
In the context of high-performance computing (HPC), the Operating System Noise (osnoise) refers to the interference experienced by an application due to activities inside the operating system. In the context of Linux, NMIs, IRQs, softirqs, and any other system thread can cause noise to the application. Moreover, hardware-related jobs can also cause noise, for example, via SMIs.
HPC users and developers that care about every microsecond stolen by the OS need not only a precise way to measure the osnoise but mainly to figure out who is stealing cpu time so that they can pursue the perfect tune of the system. These users and developers are the inspiration of Linux's osnoise tracer.
The osnoise tracer runs an in-kernel loop measuring how much time is available. It does it with preemption, softirq and IRQs enabled, thus allowing all the sources of osnoise during its execution. The osnoise tracer takes note of the entry and exit point of any source of interferences. When the noise happens without any interference from the operating system level, the tracer can safely point to a hardware-related noise. In this way, osnoise can account for any source of interference. The osnoise tracer also adds new kernel tracepoints that auxiliaries the user to point to the culprits of the noise in a precise and intuitive way.
At the end of a period, the osnoise tracer prints the sum of all noise, the max single noise, the percentage of CPU available for the thread, and the counters for the noise sources, serving as a benchmark tool.
Linux fundamental - Chap 15 Job SchedulingKenny (netman)
This document discusses different methods for automating system administration tasks by scheduling jobs, including cron, anacron, and at. Cron executes tasks on a recurring schedule, while anacron runs jobs that may have been missed when the system was offline. The at command executes a one-time job at a specified time in the future. Configuration files like cron.allow and cron.deny control which users can schedule jobs with these tools.
Scalability strategies for cloud based system architectureSangJin Kang
- Scalability & Availability for the Global Markets
- Global scaled Scalability, Availability and Security
- Architecture for 100, 1K, 100K, 500K, 1M and 10M global users
- Auto-Scaling
- Understand Cloud Services
- Cloud Demo(AWS, GCP, Azure and Cloudflare)
- Wrap-Up
A talk I gave at the Boston Web Performance Meetup in August 2014.
Performance is one of the most challenging issues in modern web app design, in large part because modeling, testing, and validating performance before deploying to production is so challenging. While many ops teams have nailed down the problem of re-creating pre-production environments that closely mimic production, those environments frequently rely on known-good components beyond the application code itself: AWS ELB, F5 load balancers, CDNs, Varnish, and more.
Testing plug-in components like that can be challenging, because their performance characteristics don't directly align with application metrics.
- How many simultaneous users can my load balancer support? - What sort of network load will I put on my CDN (i.e., how much will it cost?) - How do different user behavior patterns affect performance?
In this meetup, we'll introduce a novel tool in this toolbox: tcpreplay, an open-source tool for replaying packet capture files back at an application. By replaying user traffic to a staging environment, you can test the effects of
- Network saturation to the load balancer - High numbers of users / IPs - Lots of traffic to your other monitoring tools!
Gulp is a task runner that automates and enhances development workflows. It is commonly used to spin up web servers, automatically reload browsers when files are saved, use preprocessors like Sass and LESS, and optimize assets like CSS, JavaScript, and images using plugins like gulp-webserver, gulp-livereload, gulp-compass, and gulp-concat.
The document discusses techniques and tools for optimizing Rails applications. It covers topics like benchmarking tools, caching, session storage options, and common performance issues in Rails like slow helper methods and associations. The document provides recommendations on optimizing actions, views, and controllers in Rails.
Extreme HTTP Performance Tuning: 1.2M API req/s on a 4 vCPU EC2 InstanceScyllaDB
In this talk I will walk you through the performance tuning steps that I took to serve 1.2M JSON requests per second from a 4 vCPU c5 instance, using a simple API server written in C.
At the start of the journey the server is capable of a very respectable 224k req/s with the default configuration. Along the way I made extensive use of tools like FlameGraph and bpftrace to measure, analyze, and optimize the entire stack, from the application framework, to the network driver, all the way down to the kernel.
I began this wild adventure without any prior low-level performance optimization experience; but once I started going down the performance tuning rabbit-hole, there was no turning back. Fueled by my curiosity, willingness to learn, and relentless persistence, I was able to boost performance by over 400% and reduce p99 latency by almost 80%.
This document discusses optimizing an Apache Pulsar cluster to handle 10 PB of data per day for a financial customer. Initial estimates showed the cluster would need over 1000 VMs using HDD storage. Various optimizations were implemented, including eliminating the journal, using direct I/O, compression, and C++ client optimizations. This reduced the estimated number of needed VMs to 200 using L-SSD storage per VM. The optimized cluster can now meet the customer's requirements of processing 10 PB of data per day with 3 hours of retention and zone failure protection.
The document describes the structure and workflow of the Bareon functional testing project. It discusses the Python APIs used to control services on the controller and slave nodes, as well as run functional tests in parallel across multiple slave nodes. Tests can validate ramdisk functionality alone or involve rebooting to a tenant image. Logs from the agent and tenant image are sent back to the controller. Tests are grouped and configured to share resources and test specific images, firmware, or nodes. Parallelism is achieved through running tests simultaneously across virtual or physical slave nodes managed by the controller.
EChr - CSR & HR - Presentation Dr/ Nicholas AndreouEChr
This document discusses the relationship between human resource management (HRM) and corporate social responsibility (CSR). It begins by providing background on the evolution of CSR, from focusing on corporate philanthropy to now being seen as a strategic responsibility. The document then examines how HRM practices contributed to the global financial crisis and the social impacts of downsizing. It explores how CSR can help address challenges in HRM perception and processes. Specifically, the document analyzes several CSR frameworks and identifies six key areas for HRM to integrate CSR: employment conditions, organizational development and culture, industrial relations, occupational health and safety, diversity and non-discrimination, and human rights.
Whatchamacallit: Controlled Vocabularies for Technical Writers (Write the Doc...Emilie Boillat
You can watch this talk here: https://www.youtube.com/watch?v=0kg-6YCDg64
An everyday dilemma in technical communication: What do you call that “thing”? Whether you ask developers, marketing folks or users, you often get very different views.
Along with user interfaces, documentation helps all stakeholders align their mental model of a product. Terminology plays a key role, and a controlled vocabulary is an ace up the technical writer's sleeve.
Controlled vocabularies are basically lists of concepts and the relationships between them. Beyond consistent naming, they are about structuring, categorizing and retrieving content. For example, you can use a vocabulary as a basis for documentation plans, tables of content, a help site's navigation, search filters, or even to organize test cases.
In this talk, I share ways to create and maintain various types of vocabularies, and explain what each type is useful for. You can start with a flat list of terms and expand it into a glossary, a thesaurus, a taxonomy, or even a full-fledge ontology.
The document provides information about the various museums and cultural centers run by the City of Austin's Museum & Cultural Programs Division. It introduces each site along with its manager and gives brief descriptions of their focuses and upcoming events for 2015-2016. The division aims to enrich the community through exhibits, events, classes and public art that showcase Austin's history and diverse cultural heritage.
Whatchamacallit: Controlled Vocabularies for Technical Writers (soap! 2015)Emilie Boillat
Note: Go to the last slide for a video of the same talk at Write the Docs Europe 2015 (https://www.youtube.com/watch?v=0kg-6YCDg64).
An everyday dilemma in technical communication: What do you call that “thing”? Whether you ask developers, marketing folks or users, you often get very different views.
Along with user interfaces, documentation helps all stakeholders align their mental model of a product. Terminology plays a key role, and a controlled vocabulary is an ace up the technical writer’s sleeve.
Controlled vocabularies are basically lists of concepts and the relationships between them. Beyond consistent naming, they are about structuring, categorizing and retrieving content. For example, you can use a vocabulary as a basis for documentation plans, tables of content, a help site’s navigation, search filters, or even to organize test cases.
In this talk, I will share ways to create and maintain various types of vocabularies, and explain what each type is useful for. You can start with a flat list of terms and expand it into a glossary, a thesaurus, a taxonomy, or even a full-fledged ontology.
Dante Ferretti is an Italian production designer born in 1943 who is renowned for his work on films such as Shutter Island, Gangs of New York, and Hugo. He has won 3 Oscars and received over 50 other awards and nominations for his contributions as a production designer on over 60 films and in art and set decoration departments on others.
This document outlines the science lesson plan for Prep G. It includes three rotation groups that will complete different activities - arranging egg development sequences, continuing a chicken diary, and finishing an art display. The full class will then review the chicken life cycle by ordering themselves, watching a video, and doing a sorting activity to explain each stage.
Steven ten Have. Integrated Coastal Zone (Change) ManagementSUSCOD
Integrated Coastal Zone (Change) Management by Steven ten Have, consultant and partner Ten Have Change Management, and professor of Change Management, VU University Amsterdam.
Dante Ferretti is an Italian production designer born in 1943 who is renowned for his work on films such as Shutter Island, Gangs of New York, and Hugo. He has won 3 Oscars and received over 50 other awards and nominations for his contributions as a production designer on over 60 films and in art and set decoration departments on others.
Как вы думаете - возможно ли управлять компанией только при помощи смартфона? Ведь это так удобно: нет нужны хранить огромное количество бумаг, постоянно присутствовать в офисе, лично контролировать работу каждого сотрудника.
Конечно же, возможно! И мы покажем вам, как.
Re-desenhando um novo Paradigma para a ECOnomia - Filipe Moreira AlvesPROUTugal
Jean-Jacques Rousseau argues that the concept of private property has caused many problems. He claims that the first person who fenced off land and claimed it as their own, convincing others this was legitimate, established the foundations of civil society. However, this also led to crimes, wars, murders and other misfortunes as people forgot that the fruits of the earth belong to all people equally. If this initial act establishing private property ownership had not occurred, humanity may have been spared significant harms.
This document discusses challenges related to corporate social responsibility (CSR) from a legal and corporate governance perspective. It notes that while corporate governance research has focused on linking good governance to financial performance, there are increasing expectations for businesses to address social concerns. However, most investors do not yet systematically factor environmental, social and governance issues into investment decisions. It argues that as social issues become subject to international standards and domestic legislation, corporate governance needs to bridge the divide between financial and social objectives. Moving forward, greater collaboration is needed between businesses, governments and academics to help companies both create value for shareholders and stakeholders and prevent adverse impacts.
The document provides system requirements for The Sims 3 game on Windows XP, Windows Vista, and Mac OS X. It lists recommended processors, graphics cards, and memory for each operating system. Recommended graphics cards include NVIDIA GeForce FX 500, 600, 700, 800, and 900 series cards. The game requires a Pentium 4 CPU, 1GB RAM for Windows and 2GB RAM for Mac.
The document proposes the creation of the North Carolina Sales Institute (NCSI) to address the growing demand for sales talent. NCSI would offer undergraduate and executive education programs in professional selling and sales leadership through partnerships with companies. Research shows sales education programs help students become productive salespeople 50% faster with 30% less turnover, saving companies $200k per new hire. NCSI aims to develop the next generation of effective sales professionals through classroom and experiential learning while providing value to partner companies through access to top students, sales research, and networking opportunities.
This document discusses profiling PHP applications to improve performance. It recommends profiling during development to identify inefficiencies. The document introduces Xdebug for profiling PHP code and Webgrind, a PHP frontend for visualizing Xdebug profiles. It provides an example of profiling a sample PHP application, identifying issues, making code changes, and verifying performance improvements through re-profiling.
This presentation demonstrates how to pinpoint performance bottlenecks in SAP BusinessObjects reports and dashboards. It explores tools that can collect and analyze data from the various components involved, such as the front-end applications, servers, databases, and network. Specific examples are provided on how to use traces, logs, and other tools to measure and break down the timing of different workflows involving Web Intelligence, Design Studio, and SAP HANA. The goal is to identify where time is spent and determine if improvements can be made to content design, system resources, configuration, or queries.
Performance Tuning Oracle Weblogic Server 12cAjith Narayanan
The document summarizes techniques for monitoring and tuning Oracle WebLogic server performance. It discusses monitoring operating system metrics like CPU, memory, network and I/O usage. It also covers monitoring and tuning the Java Virtual Machine, including garbage collection. Specific tools are outlined for monitoring servers like the WebLogic admin console, and command line JVM tools. The document provides tips for configuring domain and server parameters to optimize performance, including enabling just-in-time starting of internal applications, configuring stuck thread handling, and setting connection backlog buffers.
This document summarizes options for running the Odoo server from the command line, including:
- Specifying database, modules, and module paths for installation/updating
- Configuring workers for multiprocessing and limiting resources per worker
- Setting logging options like the log file, database, or specific log levels
- Advanced options like auto-reloading or enabling a proxy
PuppetDB: Sneaking Clojure into Operationsgrim_radical
The document provides an overview of PuppetDB, which is a system for storing and querying data about infrastructure as code and system configurations. Some key points:
- PuppetDB stores immutable data about systems and allows querying of this data to enable higher-level infrastructure operations.
- It uses techniques like command query responsibility separation (CQRS) to separate write and read pipelines for better performance and reliability.
- The data is stored in a relational database for efficient querying, and queries are expressed in an abstract syntax tree (AST)-based language.
- The system is designed for speed, reliability, and ease of deployment in operations. It leverages techniques from Clojure and the JVM.
Understanding Spark Tuning: Strata New YorkRachel Warren
How to design a Spark Auto Tuner.
The first section coves how to set basic Spark settings e.g. executor memory, driver memory, dynamic allocation, shuffle settings, number of partitions etc. The second section it covers how to collect historical data about a spark Job and the third section discusses designing an auto tuner application which will programmatically configure Spark jobs using that historical data.
Spark Autotuning Talk - Strata New YorkHolden Karau
This document discusses how to tune Apache Spark jobs for optimal performance. It begins with introductions of the presenters and an overview of what will be covered, including the most important Spark settings, using the auto tuner, examples of common errors that can be addressed by tuning, and collecting historical data. Examples are provided of how to address errors like out of memory issues by increasing resources or adjusting partitioning. While tuning can help with many issues, some problems like unnecessary shuffles or unbalanced data cannot be addressed without code changes.
Apache Spark is an amazing distributed system, but part of the bargain we’ve made with the infrastructure deamons involves providing the correct set of magic numbers (aka tuning) or our jobs may be eaten by Cthulhu. Tuning Apache Spark is somewhat of a dark art, although thankfully, when it goes wrong, all we tend to lose is several hours of our day and our employer’s money.
Holden Karau, Rachel Warren, and Anya Bida explore auto-tuning jobs using both historical and live job information, using systems like Apache BEAM, Mahout, and internal Spark ML jobs as workloads. Much of the data required to effectively tune jobs is already collected inside of Spark. You just need to understand it. Holden, Rachel, and Anya outline sample auto-tuners and discuss the options for improving them and applying similar techniques in your own work. They also discuss what kind of tuning can be done statically (e.g., without depending on historic information) and look at Spark’s own built-in components for auto-tuning (currently dynamically scaling cluster size) and how you can improve them.
Even if the idea of building an auto-tuner sounds as appealing as using a rusty spoon to debug the JVM on a haunted supercomputer, this talk will give you a better understanding of the knobs available to you to tune your Apache Spark jobs.
Also, to be clear, Holden, Rachel, and Anya don’t promise to stop your pager going off at 2:00am, but hopefully this helps.
Scaling python webapps from 0 to 50 million users - A top-down approachJinal Jhaveri
This document provides an overview of scaling a Python web application from 0 to 50 million users. It discusses key bottlenecks and solutions at different levels including the load balancer, web server, web application and browser. It emphasizes the importance of profiling, measuring and improving performance iteratively. Specific techniques mentioned include using Memcached to avoid database trips, asynchronous programming, compression, caching, and a performance strategy of measure, profile and improve.
Cómo se diseña una base de datos que pueda ingerir más de cuatro millones de ...javier ramirez
En esta sesión voy a contar las decisiones técnicas que tomamos al desarrollar QuestDB, una base de datos Open Source para series temporales compatible con Postgres, y cómo conseguimos escribir más de cuatro millones de filas por segundo sin bloquear o enlentecer las consultas.
Hablaré de cosas como (zero) Garbage Collection, vectorización de instrucciones usando SIMD, reescribir en lugar de reutilizar para arañar microsegundos, aprovecharse de los avances en procesadores, discos duros y sistemas operativos, como por ejemplo el soporte de io_uring, o del balance entre experiencia de usuario y rendimiento cuando se plantean nuevas funcionalidades.
The document provides guidance on learning about automotive embedded systems through a 10 part series. It recommends first studying parts on real-time operating system basics, OSEK/VDX, AUTOSAR basics, and automotive protocols. Then users should validate their understanding and solve practice questions. The document directs readers to online materials and emphasizes the importance of depth of learning to become professional in the field of embedded systems.
Java is one of the most popular languages and it's very important to understand the performance of Java servers. Modern JVMs compile the Java code in runtime using Just-In-Time (JIT) compiler and such JIT compiled code runs very close to optimized native code in terms of speed.
When understanding performance, it's important to know how Java works and we can also measure the performance using key metrics like Throughput and Latency. After measuring the performance, we can use profilers to understand the application behavior and find performance bottlenecks.
In this session, we will look at how Java manages the memory and how it optimizes the Java code using JIT compilation. We will also look at how we can use the Java Flight Recorder (JFR) to profile the JVM and find performance bottlenecks.
Finally, we can look at how "Flame Graphs" can be used to identify the most frequent code-paths quickly and accurately.
Performance Tuning Cheat Sheet for MongoDBSeveralnines
Bart Oles - Severalnines AB
Database performance affects organizational performance, and we tend to look for quick fixes when under stress. But how can we better understand our database workload and factors that may cause harm to it? What are the limitations in MongoDB that could potentially impact cluster performance?
In this talk, we will show you how to identify the factors that limit database performance. We will start with the free MongoDB Cloud monitoring tools. Then we will move on to log files and queries. To be able to achieve optimal use of hardware resources, we will take a look into kernel optimization and other crucial OS settings. Finally, we will look into how to examine performance of MongoDB replication.
The objective of this article is to describe what to monitor in and around Alfresco in order to have a good understanding of how the applications are performing and to be aware of potential issues.
Airflow is a workflow management system for authoring, scheduling and monitoring workflows or directed acyclic graphs (DAGs) of tasks. It supports authoring workflows as Python code with built-in operators to interact with external systems and databases. Airflow has a modular architecture allowing different executors like Celery and Kubernetes. It provides features like variables, connections, XComs and service level agreements. The user interface allows visualizing and monitoring DAGs and includes data profiling capabilities. Airflow also has a command line interface and supports security features like authentication and impersonation.
Netflix - Pig with Lipstick by Jeff Magnusson Hakka Labs
In this talk Manager of Data Platform Architecture Jeff Magnusson from Netflix discusses Lipstick, a tool that visualizes and monitors the progress and performance of Apache Pig scripts. This talk was recorded at Samsung R&D.
While Pig provides a great level of abstraction between MapReduce and dataflow logic, once scripts reach a sufficient level of complexity, it becomes very difficult to understand how data is being transformed and manipulated across MapReduce jobs. The recently open sourced Lipstick solves this problem. Jeff emphasizes the architecture, implementation, and future of Lipstick, as well as various use cases around using Lipstick at Netflix (e.g. examples of using Lipstick to improve speed of development and efficiency of new and existing scripts).
Jeff manages the Data Platform Architecture group at Netflix where he is helping to build a service oriented architecture that enables easy access to large scale cloud based analytical processing and analysis of data across the organization. Prior to Netflix, he received his PhD from the University of Florida focusing on database system implementation.
Slides from the Big Data Gurus meetup at Samsung R&D, August 14, 2013
This presentation covers the high level architecture of the Netflix Data Platform with a deep dive into the architecture, implementation, use cases, and future of Lipstick (https://github.com/Netflix/Lipstick) - our open source tool for graphically analyzing and monitoring the execution of Apache Pig scripts.
Netflix uses Apache Pig to express many complex data manipulation and analytics workflows. While Pig provides a great level of abstraction between MapReduce and data flow logic, once scripts reach a sufficient level of complexity, it becomes very difficult to understand how data is being transformed and manipulated across MapReduce jobs. To address this problem, we created (and open sourced) a tool named Lipstick that visualizes and monitors the progress and performance of Pig scripts.
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
Enchancing adoption of Open Source Libraries. A case study on Albumentations.AIVladimir Iglovikov, Ph.D.
Presented by Vladimir Iglovikov:
- https://www.linkedin.com/in/iglovikov/
- https://x.com/viglovikov
- https://www.instagram.com/ternaus/
This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation.
Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners.
This case study covers various aspects, including:
People: The contributors and community that have supported Albumentations.
Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions.
Challenges: The hurdles in monetizing open-source projects and measuring user engagement.
Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration.
Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community.
Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations.
Mental Health: Maintaining balance and not feeling pressured by user demands.
Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth.
Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects.
Explore more about Albumentations and join the community at:
GitHub: https://github.com/albumentations-team/albumentations
Website: https://albumentations.ai/
LinkedIn: https://www.linkedin.com/company/100504475
Twitter: https://x.com/albumentations
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Introducing Milvus Lite: Easy-to-Install, Easy-to-Use vector database for you...Zilliz
Join us to introduce Milvus Lite, a vector database that can run on notebooks and laptops, share the same API with Milvus, and integrate with every popular GenAI framework. This webinar is perfect for developers seeking easy-to-use, well-integrated vector databases for their GenAI apps.
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
2. - Performance bites hard when app hits scale
- Even though GO is garbage collected language, allocated objects have
implications on latency and gc time
- When measuring latency,look at percentiles not just averages
- Percentiles show you tail latencies - this also helps you to understand what perf is those
unlucky customers seeing (bottom 1% or 5%)
Importance of Performance Measurement
3. What is profiling
- Pause application and capture thread stack multiple times per-second
- Usually takes around 100 stack dumps per second
- For java developers it is similar to running jstack (or yourkit) couple of times in a second
- With profiling data we can do lot of things like,
- Shows what functions are used
- Can build call graphs
- Find out what functions are at the top of the stack (taking CPU time)
4. Viewing CPU/Memory profile snapshot
- Every node is a function call
- A vertex from X to Y indicates a call from X -> Y
- For example in the profile snapshot below
- FindLoops took 4.41s of CPU time
- Remaining 30.28s were spent on outgoing function call
- “web” command opens up web view of profiler
5. GO Profiling tools - pprof
- What we can profile?
- Standalone application (start with profiling enabled through commandline arg)
- A live process
- Using net.http.pprof, go can capture profile of a live process
- Many others features available thorugh web interface like viewing passed in
command line arguments, memory profile, cpu profile etc.
- What you can do with profiling data
- View methods that are taking more time, or allocation more objects
- Annotate code with cpu/memory profile data
- Slice and dice into different parts of the program for better understanding of cpu time
- GO allows you to do cpu and memory profile (and blocking)
- CPU allows you to look at what functions are taking cpu time
- Memory profile lets you see memory allocation per function
6. GO Profiling tools - benchmarking
- Run test X number of times and reports the average time
- Prints allocations per call
- Run with multiple CPU (GOMAXPROCs)
- Execute following command to get memory allocation along with runtime,
- $ go test -bench=. -benchmem
7. pprof commands (top)
- Top command lets you view functions that are taking up most CPU time
8. pprof commands (weblist)
- weblist annotates source code with profiler data (and assembly) in a
webview
- Drill down and expand each source line to see assembly instructions (pretty powerful)
10. - Type of garbage collectors
- STW - Stop the World
- Concurrent
- With 1.5, go started to have concurrent gc
- This means less time spent in STW phase (~ 10ms)
- Latencies improve overall
- More details here
- GO 1.6 does little better
- GC pauses is even lower
- See here for details
Garbage Collector
11. - Simplicity - core principle
- GOGC - the only parameter that you’d ever to tune
- Defaults to 100%, which really means that your heap size after garbage collection will be
kept at
- gctrace (GODEBUG=gctrace=1 commandline)
- Go program will start writing detailed information about GC on stdout
- Helpful in debuggin whether GC is the cause of latency or the not
- GOMAXPROC
- This sets the maximum number of processes GO process can use
- As of go1.5, there isnt any need to set this as runtime figures it out automatically
GO Performance Tuning (GC)
12. Summary
- Be judicious when allocating new objects
- See if you can use simpler data structures (e.g. slice instead of map)
- Reuse objects if possible (connection pooling, objects cache etc)
- Measure latencies in percentiles
- Enable web pprof in our application
- Doesnt cost much, and lets you take traces of live process
- For all users of 1.3, upgrade to 1.6
- Concurrent GC
- Less time spent in GC, more time for the app (mutator)
- Play with GOGC and gcdebug parameter
13. Links and further reading
- GO 1.5 gc release notes - https://blog.golang.org/go15gc
- Concurrent vs stop the world gc - https://talks.golang.org/2015/go-gc.pdf
- Testing and benchmarking - https://golang.org/pkg/testing/
- Running pprof from http - https://golang.org/pkg/net/http/pprof/
- Profiling GO Programs - https://blog.golang.org/profiling-go-programs
- http://www.stuartcheshire.org/rants/Latency.html