This document discusses integrating test automation and code coverage for web service applications. It introduces Postman for calling web services and testing responses, and Jenkins for build automation and tracking test results over time. It then demonstrates setting up a test automation workflow using these tools on a sample Laravel application, including starting and stopping coverage collection, running tests from Postman and PHPUnit, and merging the results. Some best practices and philosophies around test automation and code coverage are also discussed.
This document summarizes and compares Ruby HTTP client libraries. It discusses the sync and async APIs of 16 libraries including Net::HTTP, HTTPClient, and Faraday. It covers their compatibility, supported features like keep-alive connections, and performance based on benchmarks. The document recommends libraries based on priorities like speed, HTML handling, API clients, and SSL support. It encourages readers to check the detailed feature matrix and report any errors found.
This document provides steps for deploying a Node.js application without getting fired, from initial setup through scaling. It begins with installing Node.js on a VPS or virtual private server. Basic steps include initializing version control with Git, adding logging and error handling, and using Upstart to ensure the app runs at boot and respawns if it crashes. Advanced techniques include using multiple processes, a cache like Varnish, staging environments, and eventually multiple app servers with the database on separate boxes controlled by tools like node-control. The document emphasizes automating deployments and scaling gradually over time.
How Danga::Socket handles asynchronous processing and how to write asynchrono...Gosuke Miyashita
The document discusses asynchronous programming in Perl and how to write asynchronous plugins for Perlbal using Danga::Socket. Key points include:
1) Danga::Socket provides asynchronous I/O event handling using its main event loop and allows adding I/O watchers and timers.
2) To write asynchronous Perlbal plugins, the main process should be based on Danga::Socket's event loop and use callbacks. The plugin must return 1 and a callback must restart processing in ClientProxy.
3) Perlbal itself may also need patching to work with asynchronous plugins by checking an async completion flag in ClientProxy before running plugins.
Assurer - a pluggable server testing/monitoring frameworkGosuke Miyashita
Assurer is a server testing and monitoring framework that allows configuring tests, notifications, formatting, and publishing of results using plugins. It uses a plugin architecture and YAML configuration to make it easy to add new tests and change outputs. Tests are run in parallel and results can be filtered and published to various outputs like terminals, email, and files. It aims to be similar to Plagger but also cover monitoring use cases.
Caching and tuning fun for high scalability @ FrOSCon 2011Wim Godden
The document discusses using caching and tuning techniques to improve scalability for websites. It covers caching full pages, parts of pages, SQL queries, and complex processing results. Memcache is presented as a fast and distributed caching solution. The document also discusses installing and using Memcache, as well as replacing Apache with Nginx as a lighter-weight web server that can serve static files and forward dynamic requests.
Shared Object images in Docker: What you need is what you want.Workhorse Computing
Docker images require appropriate shared object files (".so") to run. Rather than assume Ubuntu has the correct lib's, use ldd to get a list and install the ones you know you need. This can reduce the underlying images from GB to a few MB.
Performance benchmarks are all too often inaccurate. This talk introduces some things to look for in setting up and running benchmarks to make them effective.
This document discusses integrating test automation and code coverage for web service applications. It introduces Postman for calling web services and testing responses, and Jenkins for build automation and tracking test results over time. It then demonstrates setting up a test automation workflow using these tools on a sample Laravel application, including starting and stopping coverage collection, running tests from Postman and PHPUnit, and merging the results. Some best practices and philosophies around test automation and code coverage are also discussed.
This document summarizes and compares Ruby HTTP client libraries. It discusses the sync and async APIs of 16 libraries including Net::HTTP, HTTPClient, and Faraday. It covers their compatibility, supported features like keep-alive connections, and performance based on benchmarks. The document recommends libraries based on priorities like speed, HTML handling, API clients, and SSL support. It encourages readers to check the detailed feature matrix and report any errors found.
This document provides steps for deploying a Node.js application without getting fired, from initial setup through scaling. It begins with installing Node.js on a VPS or virtual private server. Basic steps include initializing version control with Git, adding logging and error handling, and using Upstart to ensure the app runs at boot and respawns if it crashes. Advanced techniques include using multiple processes, a cache like Varnish, staging environments, and eventually multiple app servers with the database on separate boxes controlled by tools like node-control. The document emphasizes automating deployments and scaling gradually over time.
How Danga::Socket handles asynchronous processing and how to write asynchrono...Gosuke Miyashita
The document discusses asynchronous programming in Perl and how to write asynchronous plugins for Perlbal using Danga::Socket. Key points include:
1) Danga::Socket provides asynchronous I/O event handling using its main event loop and allows adding I/O watchers and timers.
2) To write asynchronous Perlbal plugins, the main process should be based on Danga::Socket's event loop and use callbacks. The plugin must return 1 and a callback must restart processing in ClientProxy.
3) Perlbal itself may also need patching to work with asynchronous plugins by checking an async completion flag in ClientProxy before running plugins.
Assurer - a pluggable server testing/monitoring frameworkGosuke Miyashita
Assurer is a server testing and monitoring framework that allows configuring tests, notifications, formatting, and publishing of results using plugins. It uses a plugin architecture and YAML configuration to make it easy to add new tests and change outputs. Tests are run in parallel and results can be filtered and published to various outputs like terminals, email, and files. It aims to be similar to Plagger but also cover monitoring use cases.
Caching and tuning fun for high scalability @ FrOSCon 2011Wim Godden
The document discusses using caching and tuning techniques to improve scalability for websites. It covers caching full pages, parts of pages, SQL queries, and complex processing results. Memcache is presented as a fast and distributed caching solution. The document also discusses installing and using Memcache, as well as replacing Apache with Nginx as a lighter-weight web server that can serve static files and forward dynamic requests.
Shared Object images in Docker: What you need is what you want.Workhorse Computing
Docker images require appropriate shared object files (".so") to run. Rather than assume Ubuntu has the correct lib's, use ldd to get a list and install the ones you know you need. This can reduce the underlying images from GB to a few MB.
Performance benchmarks are all too often inaccurate. This talk introduces some things to look for in setting up and running benchmarks to make them effective.
perl often doesn't get updated because people don't have a way to know if their current code works with the new one. The problem is that they lack unit tests. This talk describes how simple it is to generate unit tests with Perl and shell, use them to automate solving problems like missing modules, and test a complete code base.
The why and how of moving to PHP 5.5/5.6Wim Godden
With PHP 5.6 out and many production environments still running 5.2 or 5.3, it's time to paint a clear picture on why everyone should move to 5.5 and 5.6 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
The document discusses automating software deployment using Ansible. It provides an overview of Ansible's basic concepts like inventory files to define hosts, playbooks to execute tasks on hosts, and roles to bundle related tasks. It then discusses using Ansible roles to automate deployments, including the ansistrano roles which can deploy applications by copying files, managing releases, and supporting deployment hooks. Overall the document presents Ansible as a way to easily automate and standardize software deployment processes.
The why and how of moving to PHP 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
PSGI is a Perl port of Python's WSGI and Ruby's Rack that defines a common interface between web servers and frameworks. Plack provides reference implementations of PSGI servers as well as middleware and utilities. This allows frameworks to run on many servers like standalone, FastCGI, and Apache using a common PSGI application interface. Plack is fast, supports many frameworks through adapters, and provides tools like Plackup and middleware to help build and test PSGI applications.
"Da sempre fare il deploy di applicazioni multi-istanza rappresenta una sfida per lo sviluppatore e per il sistemista. Oggi grazie a strumenti gratuiti e un pò di esperienza in materia è possibile eseguire tale operazione lavorando su pipeline che prevedono testing automatico, validazione del code style e molto altro, in modo da ottenere una Continuous Integration consistente, efficace ed in tempi brevissimi. Scopriamo insieme come con un progetto Laravel e le pipeline di Bitbucket."
With PHP 5.4 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.3 and 5.4 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
Static typing in Vault refers to enforcing the structure and location of secrets stored in Vault. This can be achieved by using a script or tool to validate secrets against JSON schemas before they are written or accessed. The schemas define the required properties and structure for different secret types. Using a generic validation tool allows schemas to be centrally defined and ensures secrets match the expected format, reducing errors from incorrectly structured secrets.
Short introduction to the basics of Perl testing and some resources for further reading. Includes basics of Test::Simple, Test::More, using Inline for multi-language testing, testing coverage, testing tests.
What you need to remember when you upload to CPANcharsbar
Perlのモジュールを公開するときに気をつけておいた方がよい個のこと
This document provides tips for publishing Perl modules to CPAN. It discusses things to be aware of such as distributions being permanently archived on BackPAN even if deleted, CPAN accepting almost anything, and several testing services that can improve modules like PAUSE, CPAN Testers, CPANTS, and more. It also covers best practices for metrics evaluated by these services around availability, integrity, file layout, prerequisites and more. Developers are encouraged to use better tools from the Perl Toolchain Gang to avoid issues.
This document provides a taxonomy and survey of Ruby HTTP client libraries. It discusses several popular client libraries like HTTParty, Faraday, Net::HTTP, Rest-Client and Typhoeus. The survey found HTTParty and Faraday to be the most popular based on stars. It also provides code samples for making requests with some of the top libraries.
"Swoole: double troubles in c", Alexandr VronskiyFwdays
Practices in using Swoole ecosystem & migration real production marketplace app to async approach. Which benefits we got and what problems happens on stack with PHP8, Postgresql, Redis, RebbitMQ, Doctrine, coroutines/fibers, concurrency HTTP Server.
The document provides an overview and tutorial of Perlbal, an open source reverse proxy and load balancer written in Perl. It discusses how to install Perlbal using CPAN, describes the various roles it can perform like reverse proxy, web server, and load balancing. It also covers configuring and using Perlbal services, writing plugins, and its class architecture.
[Community Open Camp] 給 PHP 開發者的 VS Code 指南Shengyou Fan
This document summarizes Visual Studio Code and how it can be used for PHP and Laravel development.
Key points discussed include:
- Visual Studio Code's features for PHP like syntax highlighting, code snippets, and debugging with XDebug.
- Extensions that add additional PHP functionality like Composer, PHP Symbols, and Laravel snippets.
- Configuring VS Code and extensions for Laravel development using Wagon, Laravel Tracy for debugging, and the VS Code URL handler.
- Tips for PHP debugging with XDebug and VS Code like enabling XDebug in UwAmp and configuring the launch.json file.
The document discusses building a lightweight Docker container for Perl by starting with a minimal base image like BusyBox, copying just the Perl installation and necessary shared libraries into the container, and setting Perl as the default command to avoid including unnecessary dependencies and tools from a full Linux distribution. It provides examples of Dockerfiles to build optimized Perl containers from Gentoo and by directly importing a tarball for minimal size and easy distribution.
Beyond Breakpoints: A Tour of Dynamic AnalysisFastly
Despite advances in software design and static analysis techniques, software remains incredibly complicated and difficult to reason about. Understanding highly-concurrent, kernel-level, and intentionally-obfuscated programs are among the problem domains that spawned the field of dynamic program analysis. More than mere debuggers, the challenge of dynamic analysis tools is to be able record, analyze, and replay execution without sacrificing performance. This talk will provide an introduction to the dynamic analysis research space and hopefully inspire you to consider integrating these techniques into your own internal tools.
This document provides an introduction and overview of pl/php, which allows users to write functions and procedures in PHP and store them in a PostgreSQL database. It discusses how pl/php works, provides an example of creating a simple pl/php function, and covers installing and configuring pl/php, including compiling the pl/php library and installing the language into a database. It also presents some basic and more advanced uses of pl/php to integrate PHP logic and programming directly into PostgreSQL databases.
WP Weekend #2 - Corcel, aneb WordPress přes LaravelBrilo Team
The document discusses using Laravel and the Corcel library to integrate WordPress with other projects. It provides an overview of Laravel features like routing and Eloquent ORM. Corcel allows using WordPress as a backend for any PHP project using Eloquent to retrieve WordPress data. Examples show retrieving WordPress posts, users, and custom post types from Laravel or other projects using Corcel.
PuppetConf 2017: Use Puppet to Tame the Dockerfile Monster- Bryan Belanger, A...Puppet
You want to create an application? Great. Download a Docker image and install all your stuff. Sounds like a lot of work, huh? Wait, you also need to be able to patch your container too? That Dockerfile will become a Frankenfile! Well, guess what: Puppet has a an answer for you. Using Docker, Puppet and Jenkins we will show you how you can: 1. Put all your code in an easy to use project. 2. Give yourself a powerful toolkit for configuration 3. Automate your builds 4. Allow your project to automate security updates / patches
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
perl often doesn't get updated because people don't have a way to know if their current code works with the new one. The problem is that they lack unit tests. This talk describes how simple it is to generate unit tests with Perl and shell, use them to automate solving problems like missing modules, and test a complete code base.
The why and how of moving to PHP 5.5/5.6Wim Godden
With PHP 5.6 out and many production environments still running 5.2 or 5.3, it's time to paint a clear picture on why everyone should move to 5.5 and 5.6 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
The document discusses automating software deployment using Ansible. It provides an overview of Ansible's basic concepts like inventory files to define hosts, playbooks to execute tasks on hosts, and roles to bundle related tasks. It then discusses using Ansible roles to automate deployments, including the ansistrano roles which can deploy applications by copying files, managing releases, and supporting deployment hooks. Overall the document presents Ansible as a way to easily automate and standardize software deployment processes.
The why and how of moving to PHP 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
PSGI is a Perl port of Python's WSGI and Ruby's Rack that defines a common interface between web servers and frameworks. Plack provides reference implementations of PSGI servers as well as middleware and utilities. This allows frameworks to run on many servers like standalone, FastCGI, and Apache using a common PSGI application interface. Plack is fast, supports many frameworks through adapters, and provides tools like Plackup and middleware to help build and test PSGI applications.
"Da sempre fare il deploy di applicazioni multi-istanza rappresenta una sfida per lo sviluppatore e per il sistemista. Oggi grazie a strumenti gratuiti e un pò di esperienza in materia è possibile eseguire tale operazione lavorando su pipeline che prevedono testing automatico, validazione del code style e molto altro, in modo da ottenere una Continuous Integration consistente, efficace ed in tempi brevissimi. Scopriamo insieme come con un progetto Laravel e le pipeline di Bitbucket."
With PHP 5.4 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.3 and 5.4 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
Static typing in Vault refers to enforcing the structure and location of secrets stored in Vault. This can be achieved by using a script or tool to validate secrets against JSON schemas before they are written or accessed. The schemas define the required properties and structure for different secret types. Using a generic validation tool allows schemas to be centrally defined and ensures secrets match the expected format, reducing errors from incorrectly structured secrets.
Short introduction to the basics of Perl testing and some resources for further reading. Includes basics of Test::Simple, Test::More, using Inline for multi-language testing, testing coverage, testing tests.
What you need to remember when you upload to CPANcharsbar
Perlのモジュールを公開するときに気をつけておいた方がよい個のこと
This document provides tips for publishing Perl modules to CPAN. It discusses things to be aware of such as distributions being permanently archived on BackPAN even if deleted, CPAN accepting almost anything, and several testing services that can improve modules like PAUSE, CPAN Testers, CPANTS, and more. It also covers best practices for metrics evaluated by these services around availability, integrity, file layout, prerequisites and more. Developers are encouraged to use better tools from the Perl Toolchain Gang to avoid issues.
This document provides a taxonomy and survey of Ruby HTTP client libraries. It discusses several popular client libraries like HTTParty, Faraday, Net::HTTP, Rest-Client and Typhoeus. The survey found HTTParty and Faraday to be the most popular based on stars. It also provides code samples for making requests with some of the top libraries.
"Swoole: double troubles in c", Alexandr VronskiyFwdays
Practices in using Swoole ecosystem & migration real production marketplace app to async approach. Which benefits we got and what problems happens on stack with PHP8, Postgresql, Redis, RebbitMQ, Doctrine, coroutines/fibers, concurrency HTTP Server.
The document provides an overview and tutorial of Perlbal, an open source reverse proxy and load balancer written in Perl. It discusses how to install Perlbal using CPAN, describes the various roles it can perform like reverse proxy, web server, and load balancing. It also covers configuring and using Perlbal services, writing plugins, and its class architecture.
[Community Open Camp] 給 PHP 開發者的 VS Code 指南Shengyou Fan
This document summarizes Visual Studio Code and how it can be used for PHP and Laravel development.
Key points discussed include:
- Visual Studio Code's features for PHP like syntax highlighting, code snippets, and debugging with XDebug.
- Extensions that add additional PHP functionality like Composer, PHP Symbols, and Laravel snippets.
- Configuring VS Code and extensions for Laravel development using Wagon, Laravel Tracy for debugging, and the VS Code URL handler.
- Tips for PHP debugging with XDebug and VS Code like enabling XDebug in UwAmp and configuring the launch.json file.
The document discusses building a lightweight Docker container for Perl by starting with a minimal base image like BusyBox, copying just the Perl installation and necessary shared libraries into the container, and setting Perl as the default command to avoid including unnecessary dependencies and tools from a full Linux distribution. It provides examples of Dockerfiles to build optimized Perl containers from Gentoo and by directly importing a tarball for minimal size and easy distribution.
Beyond Breakpoints: A Tour of Dynamic AnalysisFastly
Despite advances in software design and static analysis techniques, software remains incredibly complicated and difficult to reason about. Understanding highly-concurrent, kernel-level, and intentionally-obfuscated programs are among the problem domains that spawned the field of dynamic program analysis. More than mere debuggers, the challenge of dynamic analysis tools is to be able record, analyze, and replay execution without sacrificing performance. This talk will provide an introduction to the dynamic analysis research space and hopefully inspire you to consider integrating these techniques into your own internal tools.
This document provides an introduction and overview of pl/php, which allows users to write functions and procedures in PHP and store them in a PostgreSQL database. It discusses how pl/php works, provides an example of creating a simple pl/php function, and covers installing and configuring pl/php, including compiling the pl/php library and installing the language into a database. It also presents some basic and more advanced uses of pl/php to integrate PHP logic and programming directly into PostgreSQL databases.
WP Weekend #2 - Corcel, aneb WordPress přes LaravelBrilo Team
The document discusses using Laravel and the Corcel library to integrate WordPress with other projects. It provides an overview of Laravel features like routing and Eloquent ORM. Corcel allows using WordPress as a backend for any PHP project using Eloquent to retrieve WordPress data. Examples show retrieving WordPress posts, users, and custom post types from Laravel or other projects using Corcel.
PuppetConf 2017: Use Puppet to Tame the Dockerfile Monster- Bryan Belanger, A...Puppet
You want to create an application? Great. Download a Docker image and install all your stuff. Sounds like a lot of work, huh? Wait, you also need to be able to patch your container too? That Dockerfile will become a Frankenfile! Well, guess what: Puppet has a an answer for you. Using Docker, Puppet and Jenkins we will show you how you can: 1. Put all your code in an easy to use project. 2. Give yourself a powerful toolkit for configuration 3. Automate your builds 4. Allow your project to automate security updates / patches
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
This document provides strategies for optimizing MySQL performance as databases grow in complexity and size. It discusses ways to optimize queries, schemas, hardware, software configuration, caching, and monitoring. The key lessons are to optimize queries, choose appropriate data types and storage engines, configure MySQL and cache settings properly, benchmark and monitor performance, and scale reads and writes separately.
Using MySQL without Maatkit is like taking a photo without removing the camera's lens cap. Professional MySQL experts use this toolkit to help keep complex MySQL installations running smoothly and efficiently. This session will show you practical ways to use Maatkit every day.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
This document discusses PostgreSQL and Solaris as a low-cost platform for medium to large scale critical scenarios. It provides an overview of PostgreSQL, highlighting features like MVCC, PITR, and ACID compliance. It describes how Solaris and PostgreSQL integrate well, with benefits like DTrace support, scalability on multicore/multiprocessor systems, and Solaris Cluster support. Examples are given for installing PostgreSQL on Solaris using different methods, configuring zones for isolation, using ZFS for storage, and monitoring performance with DTrace scripts.
This MariaDB workshop covers database concepts like tables, queries, indexes, and transactions. It discusses performance monitoring using the slow query log and tools like Percona Toolkit and Anemometer. Replication in MariaDB is explained, including traditional master-slave replication and Galera clusters. Hands-on exercises reinforce key concepts.
fog or: How I Learned to Stop Worrying and Love the CloudWesley Beary
Learn how to easily get started on cloud computing with fog. If you can control your infrastructure choices, you’ll make better choices in development and get what you need in production. You'll get an overview of fog and concrete examples to give you a head start on your provisioning workflow.
fog or: How I Learned to Stop Worrying and Love the Cloud (OpenStack Edition)Wesley Beary
The document discusses how to use the Fog library to interact with cloud services. Fog allows interacting with multiple cloud providers like AWS, Rackspace, etc in a portable way. It provides models, collections, and methods to manage resources like servers, storage, DNS etc. in an abstracted way across providers. The document demonstrates how to boot a server, install SSH keys, run commands via SSH, and ping a target using the Fog and Ruby APIs in just a few lines of code.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
How I Learned to Stop Worrying and Love the Cloud - Wesley Beary, Engine YardSV Ruby on Rails Meetup
Wesley Beary: Cloud computing scared the crap out of me - the quirks and nightmares
of provisioning computing and storage on AWS, Terremark, Rackspace,
etc - until I took the bull by the horns. Let me now show you how I
tamed that bull.
Learn how to easily get started cloud computing with fog. It gives you
the reins within any Ruby application or script. If you can control
your infrastructure choices, you can make better choices in
development and get what you need in production.
You'll get an overview of fog and concrete examples to give you a head
start on your own provisioning workflow.
There are many common workloads in R that are "embarrassingly parallel": group-by analyses, simulations, and cross-validation of models are just a few examples. In this talk I'll describe several techniques available in R to speed up workloads like these, by running multiple iterations simultaneously, in parallel.
Many of these techniques require the use of a cluster of machines running R, and I'll provide examples of using cloud-based services to provision clusters for parallel computations. In particular, I will describe how you can use the SparklyR package to distribute data manipulations using the dplyr syntax, on a cluster of servers provisioned in the Azure cloud.
Presented by David Smith at Data Day Texas in Austin, January 27 2018.
Beyond PHP - It's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
This document discusses the process of rebalancing in Voldemort. It begins by outlining the high-level steps taken, including getting the current and target cluster states, planning partition movements in batches, changing cluster metadata and rebalancing states, migrating data with redundancy checks, and rolling back changes if failures occur. Key aspects like maintaining consistency through proxying requests and handling failure scenarios are also summarized.
PostgreSQL Portland Performance Practice Project - Database Test 2 HowtoMark Wong
Fourth presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation is a set of examples to go along with the live presentation given on March 12, 2009.
Caching and tuning fun for high scalabilityWim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site.
If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
PostgreSQL as seen by Rubyists (Kaigi on Rails 2022)Андрей Новиков
PostgreSQL has become the most popular RDBMS in the Ruby ecosystem in the last decade. It has a great set of built-in features, including a variety of versatile data types, both common and very specific.
But when we load data from the database to our application code, we're working with Ruby data types: classes from the standard library, Rails, or other gems. So while they can seem to be the same as their PostgreSQL counterparts, they are not absolutely identical, and sometimes that could lead to surprising behavior.
In this talk, I would like to explore the power of data types in PostgreSQL and Ruby and how to work with them properly to use both Ruby and PostgreSQL on 100% of their power!
W świecie mikrousługowym architektura Lambda zadomowiła się już na dobre. Tak przetwarzania streamingowe, jak i batchowe buduje wiele firm. Na rynku (o ile o rynku można mówić w kontekście open source) istnieje wiele frameworków, każdy jednak ma pewne cechy, które — zwłaszcza przy dużych projektach — utrudniają pracę. Jedne służą do przetwarzania real-time, drugie lepiej spisują się w workloadach batchowych. Niektóre z nich zaś można uznać za „rock-solid” tylko jeśli uruchamiamy je na Hadoopie. Nie brak tych problemów jest jednak główną zaletą Beama. A co nią jest? Dowiecie się na prezentacji! Poruszymy takie kwestie jak model przetwarzania, use-case’y, w których Beam się sprawdza, a także środowiska uruchomieniowe. Zobaczycie też, jak uruchamiać joby Apache Beam na Google Cloud Platform.
ManageIQ currently runs on Ruby on Rails 3. Aaron "tenderlove" Patterson presents his effort to migrate to RoR 4, which entails some changes in the code to take advantage of the latest advances in RoR.
For more on ManageIQ, see http://manageiq.org/
PostgreSQL is a free and open-source relational database management system that provides high performance and reliability. It supports replication through various methods including log-based asynchronous master-slave replication, which the presenter recommends as a first option. The upcoming PostgreSQL 9.4 release includes improvements to replication such as logical decoding and replication slots. Future releases may add features like logical replication consumers and SQL MERGE statements. The presenter took questions at the end and provided additional resources on PostgreSQL replication.
Similar to SDPHP - Percona Toolkit (It's Basically Magic) (20)
Literature Review Basics and Understanding Reference Management.pptxDr Ramhari Poudyal
Three-day training on academic research focuses on analytical tools at United Technical College, supported by the University Grant Commission, Nepal. 24-26 May 2024
CHINA’S GEO-ECONOMIC OUTREACH IN CENTRAL ASIAN COUNTRIES AND FUTURE PROSPECTjpsjournal1
The rivalry between prominent international actors for dominance over Central Asia's hydrocarbon
reserves and the ancient silk trade route, along with China's diplomatic endeavours in the area, has been
referred to as the "New Great Game." This research centres on the power struggle, considering
geopolitical, geostrategic, and geoeconomic variables. Topics including trade, political hegemony, oil
politics, and conventional and nontraditional security are all explored and explained by the researcher.
Using Mackinder's Heartland, Spykman Rimland, and Hegemonic Stability theories, examines China's role
in Central Asia. This study adheres to the empirical epistemological method and has taken care of
objectivity. This study analyze primary and secondary research documents critically to elaborate role of
china’s geo economic outreach in central Asian countries and its future prospect. China is thriving in trade,
pipeline politics, and winning states, according to this study, thanks to important instruments like the
Shanghai Cooperation Organisation and the Belt and Road Economic Initiative. According to this study,
China is seeing significant success in commerce, pipeline politics, and gaining influence on other
governments. This success may be attributed to the effective utilisation of key tools such as the Shanghai
Cooperation Organisation and the Belt and Road Economic Initiative.
ACEP Magazine edition 4th launched on 05.06.2024Rahul
This document provides information about the third edition of the magazine "Sthapatya" published by the Association of Civil Engineers (Practicing) Aurangabad. It includes messages from current and past presidents of ACEP, memories and photos from past ACEP events, information on life time achievement awards given by ACEP, and a technical article on concrete maintenance, repairs and strengthening. The document highlights activities of ACEP and provides a technical educational article for members.
Understanding Inductive Bias in Machine LearningSUTEJAS
This presentation explores the concept of inductive bias in machine learning. It explains how algorithms come with built-in assumptions and preferences that guide the learning process. You'll learn about the different types of inductive bias and how they can impact the performance and generalizability of machine learning models.
The presentation also covers the positive and negative aspects of inductive bias, along with strategies for mitigating potential drawbacks. We'll explore examples of how bias manifests in algorithms like neural networks and decision trees.
By understanding inductive bias, you can gain valuable insights into how machine learning models work and make informed decisions when building and deploying them.
Hierarchical Digital Twin of a Naval Power SystemKerry Sado
A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.
Advanced control scheme of doubly fed induction generator for wind turbine us...IJECEIAES
This paper describes a speed control device for generating electrical energy on an electricity network based on the doubly fed induction generator (DFIG) used for wind power conversion systems. At first, a double-fed induction generator model was constructed. A control law is formulated to govern the flow of energy between the stator of a DFIG and the energy network using three types of controllers: proportional integral (PI), sliding mode controller (SMC) and second order sliding mode controller (SOSMC). Their different results in terms of power reference tracking, reaction to unexpected speed fluctuations, sensitivity to perturbations, and resilience against machine parameter alterations are compared. MATLAB/Simulink was used to conduct the simulations for the preceding study. Multiple simulations have shown very satisfying results, and the investigations demonstrate the efficacy and power-enhancing capabilities of the suggested control system.
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesChristina Lin
Traditionally, dealing with real-time data pipelines has involved significant overhead, even for straightforward tasks like data transformation or masking. However, in this talk, we’ll venture into the dynamic realm of WebAssembly (WASM) and discover how it can revolutionize the creation of stateless streaming pipelines within a Kafka (Redpanda) broker. These pipelines are adept at managing low-latency, high-data-volume scenarios.
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODELgerogepatton
As digital technology becomes more deeply embedded in power systems, protecting the communication
networks of Smart Grids (SG) has emerged as a critical concern. Distributed Network Protocol 3 (DNP3)
represents a multi-tiered application layer protocol extensively utilized in Supervisory Control and Data
Acquisition (SCADA)-based smart grids to facilitate real-time data gathering and control functionalities.
Robust Intrusion Detection Systems (IDS) are necessary for early threat detection and mitigation because
of the interconnection of these networks, which makes them vulnerable to a variety of cyberattacks. To
solve this issue, this paper develops a hybrid Deep Learning (DL) model specifically designed for intrusion
detection in smart grids. The proposed approach is a combination of the Convolutional Neural Network
(CNN) and the Long-Short-Term Memory algorithms (LSTM). We employed a recent intrusion detection
dataset (DNP3), which focuses on unauthorized commands and Denial of Service (DoS) cyberattacks, to
train and test our model. The results of our experiments show that our CNN-LSTM method is much better
at finding smart grid intrusions than other deep learning algorithms used for classification. In addition,
our proposed approach improves accuracy, precision, recall, and F1 score, achieving a high detection
accuracy rate of 99.50%.
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
SDPHP - Percona Toolkit (It's Basically Magic)
1. Percona Toolkit
(It's Basically Magic)
SDPHP | Business.com | 05-28-14
Notes:
Who Am I?
https://twitter.com/robertswisher
https://plus.google.com/+RobertSwisher
https://www.linkedin.com/in/robertswisher
robert@business.com
Notes:
2. Percona Toolkit
(It's Basically Magic)
SDPHP | Business.com | 05-28-14
Notes:
Who Am I?
https://twitter.com/robertswisher
https://plus.google.com/+RobertSwisher
https://www.linkedin.com/in/robertswisher
robert@business.com
Notes:
3. Percona?
Who the hell are they?!
Notes:
Formerly known as Maatkit & Aspersa
Baron Shwartz literally wrote the book on MySQL
Open-source collection of scripts to help common
tasks that every DBA and developer has to do.
- Development
- Profiling
- Configuration
- Monitoring
- Replication
- Same code, same developers, new branding
- Source now on LaunchPad (like Percona Server)
(https://launchpad.net/percona-toolkit)
What is Percona Toolkit?
(You should use Percona Server too!)
Notes:
4. Percona?
Who the hell are they?!
Notes:
Formerly known as Maatkit & Aspersa
Baron Shwartz literally wrote the book on MySQL
Open-source collection of scripts to help common
tasks that every DBA and developer has to do.
- Development
- Profiling
- Configuration
- Monitoring
- Replication
- Same code, same developers, new branding
- Source now on LaunchPad (like Percona Server)
(https://launchpad.net/percona-toolkit)
What is Percona Toolkit?
(You should use Percona Server too!)
Notes:
5. Basically anyone using running MySQL who has lots of data
Who Uses It?
(Or anyone smart and lazy like all of us)
Notes:
InstallationAs of writing current version is 2.2.7
Yum
Apt
or source
Notes:
6. Basically anyone using running MySQL who has lots of data
Who Uses It?
(Or anyone smart and lazy like all of us)
Notes:
InstallationAs of writing current version is 2.2.7
Yum
Apt
or source
Notes:
7. Tools
Notes:
What Do You Use It For?
- Schema changes
- Data archival
- Query optimization
- Data consistency
- Performance debugging
- General maintenance
Notes:
8. Tools
Notes:
What Do You Use It For?
- Schema changes
- Data archival
- Query optimization
- Data consistency
- Performance debugging
- General maintenance
Notes:
9. Schema Changes
- Always creates a copy of table before 5.6
(except fast index creation in 5.5 or 5.1 with innodb plugin)
- Table is locked during the change
- BIG tables = BIG TROUBLE (millions of rows take hours or more)
- Used to require trickery like ALTER on slave, promote to master,
ALTER on old master, promote to master again
(Gets really ugly with master-master or tiered replication)
Notes:
pt-online-schema-change
Triggers are trouble, but can be handled (dropped by default)
Foreign keys are trouble, but can be handled (dropped and rebuilt)
Takes longer than ALTER TABLE (up to 4x)
ALWAYS backup first
Notes:
10. Schema Changes
- Always creates a copy of table before 5.6
(except fast index creation in 5.5 or 5.1 with innodb plugin)
- Table is locked during the change
- BIG tables = BIG TROUBLE (millions of rows take hours or more)
- Used to require trickery like ALTER on slave, promote to master,
ALTER on old master, promote to master again
(Gets really ugly with master-master or tiered replication)
Notes:
pt-online-schema-change
Triggers are trouble, but can be handled (dropped by default)
Foreign keys are trouble, but can be handled (dropped and rebuilt)
Takes longer than ALTER TABLE (up to 4x)
ALWAYS backup first
Notes:
11. pt-online-schema-change
-- dry-run and --execute mutually exclusive
Use nohup with -- password `cat /tmp/pass`
Tune --max-lag and --max load for busy systems
Example:
nohup pt-online-schema-change --dry-run
--alter 'CHANGE `foo`
`foo` varchar(24) COLLATE 'latin1_bin' NULL AFTER `bar`'
--password `cat /tmp/pass` --print --nocheck-replication-filters
--max-load "Threads_connected:60,Threads_running:20"
D=your_db,t=really_big_table &
Notes:
Notes:
12. pt-online-schema-change
-- dry-run and --execute mutually exclusive
Use nohup with -- password `cat /tmp/pass`
Tune --max-lag and --max load for busy systems
Example:
nohup pt-online-schema-change --dry-run
--alter 'CHANGE `foo`
`foo` varchar(24) COLLATE 'latin1_bin' NULL AFTER `bar`'
--password `cat /tmp/pass` --print --nocheck-replication-filters
--max-load "Threads_connected:60,Threads_running:20"
D=your_db,t=really_big_table &
Notes:
Notes:
13. Notes:
Data Archival
- LOTS of writing to BIG tables = BAD
- Pruning BIG tables to only frequently accessed data = GOOD
- BIG tables more prone to corruption
- Deleting from BIG tables = SLOOOOOOW
- Long running transactions = REALLY SLOOOOOOOOOW
- DELETE locks MyISAM
Notes:
14. Notes:
Data Archival
- LOTS of writing to BIG tables = BAD
- Pruning BIG tables to only frequently accessed data = GOOD
- BIG tables more prone to corruption
- Deleting from BIG tables = SLOOOOOOW
- Long running transactions = REALLY SLOOOOOOOOOW
- DELETE locks MyISAM
Notes:
15. pt-archiver
Create destination table first
--dry-run exists, but --execute doesn't
If you use an auto-increment column, edit the schema
--limit is good for sequential data, but be careful if bouncing around
Use --progress to track
May want to archive from slave, then purge from master
ALWAYS backup first
Notes:
Notes:
16. pt-archiver
Create destination table first
--dry-run exists, but --execute doesn't
If you use an auto-increment column, edit the schema
--limit is good for sequential data, but be careful if bouncing around
Use --progress to track
May want to archive from slave, then purge from master
ALWAYS backup first
Notes:
Notes:
21. # Query 1: 0.00 QPS, 0.01x concurrency, ID 0x76F9EC92751F314A at byte 80096643
# This item is included in the report because it matches --limit.
# Scores: V/M = 188.68
# Time range: 2012-02-01 09:20:24 to 2013-10-04 10:47:56
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 1 490
# Exec time 14 384617s 9s 4869s 785s 1292s 385s 833s
# Lock time 2 11s 169us 6s 22ms 6ms 290ms 316us
# Rows sent 0 711.60k 0 4.82k 1.45k 4.27k 1.44k 685.39
# Rows examine 10 30.01G 0 123.80M 62.71M 117.57M 44.73M 75.78M
# Rows affecte 0 0 0 0 0 0 0 0
# Rows read 0 711.60k 0 4.82k 1.45k 4.27k 1.44k 685.39
# Bytes sent 0 21.90M 0 167.52k 45.77k 143.37k 52.07k 8.46k
# Tmp tables 8 1.91k 2 4 3.99 3.89 0.16 3.89
# Tmp disk tbl 0 0 0 0 0 0 0 0
# Tmp tbl size 3 3.36G 0 7.98M 7.02M 7.65M 1.26M 7.65M
# Query size 0 471.35k 982 986 985.03 964.41 0 964.41
# String:
# Databases bdc_ccm
# Hosts
# InnoDB trxID 13E9F1B2 (1/0%), 1402493D (1/0%)... 488 more
# Last errno 0
# Users semuser (488/99%), jackie.lam (1/0%)... 1 more
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms
# 100ms
# 1s #
# 10s+ ################################################################
# Tables
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_inbound'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_inbound`G
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_outbound_tp'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_outbound_tp`G
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_outbound'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_outbound`G
# EXPLAIN /*!50100 PARTITIONS*/
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc), 'AS' astag
from click_log_inbound a, click_log_outbound_tp b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and b.partner like'adsense'
and b.flag = 0
group by date(a.timestamp), a.referrer
union all
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc) , 'FL' astag
from click_log_inbound a, click_log_outbound b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and flag = 0
group by date(a.timestamp), a.referrer
union all
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc), 'TP' astag
from click_log_inbound a, click_log_outbound_tp b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and b.partner in ('capterra', 'bdc_network')
and b.flag = 0
group by date(a.timestamp), a.referrerG
Notes:
Data Consistency
- Replication isn't perfect
- Replication filters
- master-master replication
- 1062 “DUPLICATE KEY ERROR”
- Server crashes
- Non-deterministic (aka not idempotent) writes
Notes:
22. # Query 1: 0.00 QPS, 0.01x concurrency, ID 0x76F9EC92751F314A at byte 80096643
# This item is included in the report because it matches --limit.
# Scores: V/M = 188.68
# Time range: 2012-02-01 09:20:24 to 2013-10-04 10:47:56
# Attribute pct total min max avg 95% stddev median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count 1 490
# Exec time 14 384617s 9s 4869s 785s 1292s 385s 833s
# Lock time 2 11s 169us 6s 22ms 6ms 290ms 316us
# Rows sent 0 711.60k 0 4.82k 1.45k 4.27k 1.44k 685.39
# Rows examine 10 30.01G 0 123.80M 62.71M 117.57M 44.73M 75.78M
# Rows affecte 0 0 0 0 0 0 0 0
# Rows read 0 711.60k 0 4.82k 1.45k 4.27k 1.44k 685.39
# Bytes sent 0 21.90M 0 167.52k 45.77k 143.37k 52.07k 8.46k
# Tmp tables 8 1.91k 2 4 3.99 3.89 0.16 3.89
# Tmp disk tbl 0 0 0 0 0 0 0 0
# Tmp tbl size 3 3.36G 0 7.98M 7.02M 7.65M 1.26M 7.65M
# Query size 0 471.35k 982 986 985.03 964.41 0 964.41
# String:
# Databases bdc_ccm
# Hosts
# InnoDB trxID 13E9F1B2 (1/0%), 1402493D (1/0%)... 488 more
# Last errno 0
# Users semuser (488/99%), jackie.lam (1/0%)... 1 more
# Query_time distribution
# 1us
# 10us
# 100us
# 1ms
# 10ms
# 100ms
# 1s #
# 10s+ ################################################################
# Tables
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_inbound'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_inbound`G
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_outbound_tp'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_outbound_tp`G
# SHOW TABLE STATUS FROM `bdc_ccm` LIKE 'click_log_outbound'G
# SHOW CREATE TABLE `bdc_ccm`.`click_log_outbound`G
# EXPLAIN /*!50100 PARTITIONS*/
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc), 'AS' astag
from click_log_inbound a, click_log_outbound_tp b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and b.partner like'adsense'
and b.flag = 0
group by date(a.timestamp), a.referrer
union all
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc) , 'FL' astag
from click_log_inbound a, click_log_outbound b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and flag = 0
group by date(a.timestamp), a.referrer
union all
selectdate(a.timestamp), a.referrer, count(b.inbound_id), sum(b.cpc), 'TP' astag
from click_log_inbound a, click_log_outbound_tp b
wherea.id = b.inbound_id
and a.timestamp between '2012-02-08 00:00:00' and '2012-02-09 23:59:59'
and b.partner in ('capterra', 'bdc_network')
and b.flag = 0
group by date(a.timestamp), a.referrerG
Notes:
Data Consistency
- Replication isn't perfect
- Replication filters
- master-master replication
- 1062 “DUPLICATE KEY ERROR”
- Server crashes
- Non-deterministic (aka not idempotent) writes
Notes:
23. pt-table-checksum
Requires STATEMENT based replication for tiered replication
Replication filters are dangerous because a failed query can
break replication
May want to use nohup since it can be slow
Notes:
Notes:
24. pt-table-checksum
Requires STATEMENT based replication for tiered replication
Replication filters are dangerous because a failed query can
break replication
May want to use nohup since it can be slow
Notes:
Notes:
25. pt-table-sync
Notes:
pt-table-sync
--dry-run and --execute mutually exclusive
ALWAYS backup first
In a tiered replication setup or master-master
take extra care to think through what will be done
Run on master to sync all slaves
pt-table-sync --execute --replicate test.checksum master1
Run on master for slaves individually to sync to master
pt-table-sync --execute --sync-to-master slave1
Notes:
26. pt-table-sync
Notes:
pt-table-sync
--dry-run and --execute mutually exclusive
ALWAYS backup first
In a tiered replication setup or master-master
take extra care to think through what will be done
Run on master to sync all slaves
pt-table-sync --execute --replicate test.checksum master1
Run on master for slaves individually to sync to master
pt-table-sync --execute --sync-to-master slave1
Notes:
29. Performance Debugging
- Problems can be random
- Problems only last for a few seconds,
you can't connect and observe fast enough
- Problems like to happen at odd hours;
ETL, rollups, reporting, etc
- You can't ALWAYS log on
Notes:
pt-stalk
- Creates a lot of files
- Output inspected with pt-sift
Notes:
30. Performance Debugging
- Problems can be random
- Problems only last for a few seconds,
you can't connect and observe fast enough
- Problems like to happen at odd hours;
ETL, rollups, reporting, etc
- You can't ALWAYS log on
Notes:
pt-stalk
- Creates a lot of files
- Output inspected with pt-sift
Notes:
31. pt-stalk
Run as root
--daemonize fork and run in the background
-- sleep length to sleep between collects
-- cycles the number of cycles the var must be true to collect
--variable Threads_running and Execution_time are good ones
--disk-bytes-free don't collect if this threshold is hit
(best practice would be to set --log and --dest to a different disk
than your data lives on the same as other mysql logs)
Notes:
Notes:
32. pt-stalk
Run as root
--daemonize fork and run in the background
-- sleep length to sleep between collects
-- cycles the number of cycles the var must be true to collect
--variable Threads_running and Execution_time are good ones
--disk-bytes-free don't collect if this threshold is hit
(best practice would be to set --log and --dest to a different disk
than your data lives on the same as other mysql logs)
Notes:
Notes:
33. pt-sift
Pass it the path to the dir used with --data
(default /var/lib/pt-stalk)
Interactive program
Lots of data points collected from the time of the incident
Notes:
Notes:
34. pt-sift
Pass it the path to the dir used with --data
(default /var/lib/pt-stalk)
Interactive program
Lots of data points collected from the time of the incident
Notes:
Notes:
35. General Admin / Maintenance
pt-slave-restart - try to restart a slave skipping errors
if replication fails
pt-summary - gives a general summary of the MySQL instance
pt-upgrade - tests logged queries against a new MySQL version
pt-config-diff - show formatted diff of my.cnf files
pt-heartbeat - update table on master with heartbeat data from slaves
pt-kill - kill MySQL threads according to filters
pt-index-usage - report on index structure and usage
pt-variable-advisor - looks at runtime vars and makes suggestions
Notes:
http:s//cloud.percona.com to sign up for beta
Percona Cloud Tools
Notes:
36. General Admin / Maintenance
pt-slave-restart - try to restart a slave skipping errors
if replication fails
pt-summary - gives a general summary of the MySQL instance
pt-upgrade - tests logged queries against a new MySQL version
pt-config-diff - show formatted diff of my.cnf files
pt-heartbeat - update table on master with heartbeat data from slaves
pt-kill - kill MySQL threads according to filters
pt-index-usage - report on index structure and usage
pt-variable-advisor - looks at runtime vars and makes suggestions
Notes:
http:s//cloud.percona.com to sign up for beta
Percona Cloud Tools
Notes: