Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Caching and tuning fun for high scalabilityWim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - It's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
The why and how of moving to php 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Caching and tuning fun for high scalabilityWim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - It's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
The why and how of moving to php 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
Caching and tuning fun for high scalability @ LOAD2012Wim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
When dynamic becomes static: the next step in web caching techniquesWim Godden
Although tools like Varnish can improve performance and scalability for static sites, when user-specific content is needed, a hit to the PHP/Ruby/Python/.Net backend is still required, causing scalability issues. We'll look at a brand-new Nginx module which implements an ultra-fast and scalable solution to this problem, changing the way developers think about designing sites with user-specific content.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
The why and how of moving to PHP 5.5/5.6Wim Godden
With PHP 5.6 out and many production environments still running 5.2 or 5.3, it's time to paint a clear picture on why everyone should move to 5.5 and 5.6 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
Remove php calls and scale your site like crazy !Wim Godden
Although tools like Varnish can improve performance and scalability for static sites, when user-specific content is needed, a hit to the PHP/Ruby/Python/.Net backend is still required, causing scalability issues. We'll look at a brand-new Nginx module which implements an ultra-fast and scalable solution to this problem, changing the way developers think about designing sites with user-specific content.
The why and how of moving to PHP 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
We all have tasks from time to time for bulk-loading external data into MySQL. What's the best way of doing this? That's the task I faced recently when I was asked to help benchmark a multi-terrabyte database. We had to find the most efficient method to reload test data repeatedly without taking days to do it each time. In my presentation, I'll show you several alternative methods for bulk data loading, and describe the practical steps to use them efficiently. I'll cover SQL scripts, the mysqlimport tool, MySQL Workbench import, the CSV storage engine, and the Memcached API. I'll also give MySQL tuning tips for data loading, and how to use multi-threaded clients.
When high performance on a web application is a hard requirement Varnish can be of rescue. But does it’s name, the high-performance HTTP accelerator, really bring what you expect? What are the caveats, pitfalls and problems you introduce when developing your application when the released version is only able to run when there is a Varnish in front? This session will give you some answers, tips and tricks to aid in application design, development with PHP and solutions when there is no Varnish in front of your application.
Using MySQL without Maatkit is like taking a photo without removing the camera's lens cap. Professional MySQL experts use this toolkit to help keep complex MySQL installations running smoothly and efficiently. This session will show you practical ways to use Maatkit every day.
MySQL exposes a collection of tunable parameters and indicators that is frankly intimidating. But a poorly tuned MySQL server is a bottleneck for your PHP application scalability. This session shows how to do InnoDB tuning and read the InnoDB status report in MySQL 5.5.
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)Ontico
Postgres has always had strong support for relational storage. However, there are many cases where relational storage is either inefficient or overly restrictive. This talk shows the many ways that Postgres has expanded to support non-relational storage, specifically the ability to store and index multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage. The talk will cover many examples of multiple-value-per-field storage, including arrays, range types, geometry, full text search, xml, json, and records.
A high level introduction to Apache Cassandra followed by an introduction to pycassa, the Python client library for Cassandra.
Presented at PyTexas 2011 by Tyler Hobbs.
An overview of the main questions/design issues when starting to work with databases in Perl
- choosing a database
- matching DB datatypes to Perl datatypes
- DBI architecture (handles, drivers, etc.)
- steps of DBI interaction : prepare/execute/fetch
- ORM principles and difficulties, ORMs on CPAN
- a few examples with DBIx::DataModel
- performance issues
First given at YAPC::EU::2009 in Lisbon. Updated version given at FPW2011 in Paris and YAPC::EU::2011 in Riga
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Caching and tuning fun for high scalability @ LOAD2012Wim Godden
Caching has been a 'hot' topic for a few years. But caching takes more than merely taking data and putting it in a cache : the right caching techniques can improve performance and reduce load significantly. But we'll also look at some major pitfalls, showing that caching the wrong way can bring down your site. If you're looking for a clear explanation about various caching techniques and tools like Memcached, Nginx and Varnish, as well as ways to deploy them in an efficient way, this talk is for you.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
When dynamic becomes static: the next step in web caching techniquesWim Godden
Although tools like Varnish can improve performance and scalability for static sites, when user-specific content is needed, a hit to the PHP/Ruby/Python/.Net backend is still required, causing scalability issues. We'll look at a brand-new Nginx module which implements an ultra-fast and scalable solution to this problem, changing the way developers think about designing sites with user-specific content.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
The why and how of moving to PHP 5.5/5.6Wim Godden
With PHP 5.6 out and many production environments still running 5.2 or 5.3, it's time to paint a clear picture on why everyone should move to 5.5 and 5.6 and how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
Remove php calls and scale your site like crazy !Wim Godden
Although tools like Varnish can improve performance and scalability for static sites, when user-specific content is needed, a hit to the PHP/Ruby/Python/.Net backend is still required, causing scalability issues. We'll look at a brand-new Nginx module which implements an ultra-fast and scalable solution to this problem, changing the way developers think about designing sites with user-specific content.
The why and how of moving to PHP 5.4/5.5Wim Godden
With PHP 5.5 out and many production environments still running 5.2 (or older), it's time to paint a clear picture on why everyone should move to 5.4 and 5.5 and how to get code ready for the latest version of PHP. In this talk, we'll migrate an old piece of code using some standard and some very non-standard tools and techniques.
We all have tasks from time to time for bulk-loading external data into MySQL. What's the best way of doing this? That's the task I faced recently when I was asked to help benchmark a multi-terrabyte database. We had to find the most efficient method to reload test data repeatedly without taking days to do it each time. In my presentation, I'll show you several alternative methods for bulk data loading, and describe the practical steps to use them efficiently. I'll cover SQL scripts, the mysqlimport tool, MySQL Workbench import, the CSV storage engine, and the Memcached API. I'll also give MySQL tuning tips for data loading, and how to use multi-threaded clients.
When high performance on a web application is a hard requirement Varnish can be of rescue. But does it’s name, the high-performance HTTP accelerator, really bring what you expect? What are the caveats, pitfalls and problems you introduce when developing your application when the released version is only able to run when there is a Varnish in front? This session will give you some answers, tips and tricks to aid in application design, development with PHP and solutions when there is no Varnish in front of your application.
Using MySQL without Maatkit is like taking a photo without removing the camera's lens cap. Professional MySQL experts use this toolkit to help keep complex MySQL installations running smoothly and efficiently. This session will show you practical ways to use Maatkit every day.
MySQL exposes a collection of tunable parameters and indicators that is frankly intimidating. But a poorly tuned MySQL server is a bottleneck for your PHP application scalability. This session shows how to do InnoDB tuning and read the InnoDB status report in MySQL 5.5.
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)Ontico
Postgres has always had strong support for relational storage. However, there are many cases where relational storage is either inefficient or overly restrictive. This talk shows the many ways that Postgres has expanded to support non-relational storage, specifically the ability to store and index multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage. The talk will cover many examples of multiple-value-per-field storage, including arrays, range types, geometry, full text search, xml, json, and records.
A high level introduction to Apache Cassandra followed by an introduction to pycassa, the Python client library for Cassandra.
Presented at PyTexas 2011 by Tyler Hobbs.
An overview of the main questions/design issues when starting to work with databases in Perl
- choosing a database
- matching DB datatypes to Perl datatypes
- DBI architecture (handles, drivers, etc.)
- steps of DBI interaction : prepare/execute/fetch
- ORM principles and difficulties, ORMs on CPAN
- a few examples with DBIx::DataModel
- performance issues
First given at YAPC::EU::2009 in Lisbon. Updated version given at FPW2011 in Paris and YAPC::EU::2011 in Riga
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond PHP - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just writing PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Beyond php - it's not (just) about the codeWim Godden
Most PHP developers focus on writing code. But creating Web applications is about much more than just wrting PHP. Take a step outside the PHP cocoon and into the big PHP ecosphere to find out how small code changes can make a world of difference on servers and network. This talk is an eye-opener for developers who spend over 80% of their time coding, debugging and testing.
Wszyscy zostaliśmy oszukani! Automatyczne zarządzanie pamięci rozwiąże wszystkie Wasze problemy, mówili. W zarządzanych środowiskach takich jak CLR JVM nie będzie wycieków pamięci, mówili! Właściwie pamięć jest tania i nie musisz się już nią nigdy więcej martwić. Wszyscy kłamali. Automatyczne zarządzanie pamięcią jest wygodną abstrakcją i bardzo często działa dobrze. Ale jak każda abstrakcja, wcześniej czy później "wycieka" ona. I to najczęściej w najmniej spodziewanym i przyjemnym momencie. W tej sesji spróbuję otworzyć oczy na fakt, że błoga nieświadomość nt. tej abstrakcji może być kosztowna. Pokażę jak może się objawić frywolne traktowanie pamięci i co możemy zyskać pisząc kod zdając sobie sprawę, że pamięć jednak nie jest nieskończona, tania i zawsze jednakowo szybka.
Good and Bad Code
The Broken Window Theory
The Grand Redesign in the Sky
The Sushi Chef Rule
The Hotel Room Rule
The Boy Scout Rule
OOP Patterns and Principles
SOLID Principles
How to measure clean code?
Tools
Uma equipe de apenas 14 engenheiros (junho de 2017) cuida da Infraestrutura do principal banco de dados do Facebook. Toda ação no Instagram, Messenger, WhatsApp e claro, no FB, passa direta ou indiretamente pela infra de dezenas de milhares de servidores que rodam MySQL.
A linguagem usada pela equipe e por trás de toda a automação é Python. Nessa palestra, vamos mostrar como a linguagem possibilitou que chegássemos nessa escala, passando pela evolução, desafios e futuro:
Interfaces “tipadas” com Thrift;
Empacotamento através do Buck;
Type Checking com MyPy;
Asyncio para novos serviços;
Debugging com gdb 7 e pudb;
Além disso, durante a palestra, serão discutidas algumas decisões relacionadas a DevOps que podem inspirar soluções em outros ambientes: gerenciamento de dezenas de milhares de servidores e banco de dados, backups e restores contínuos, schema migrations, entre outros.
Como analisar planos de execução e estatísticas no PostgreSQL.
- Rastreamento de consultas lentas
- Uso do EXPLAIN
- Métodos de acesso
- Junções
- Parâmetros relevantes para o otimizador
Last year we decided to build an in-house solution for Funnel analysis which should be accessible to our business user through our BI tool. Backend part should run on Ap;ache Spark and since the BI tool can only run SQL queries that implies that the solution is a pure Spark SQL implementation of Funnel analysis. In this talk we will cover various Spark SQL features we have used to optimize query performance and implement various filters which enable end users to get actionable insights. KEY TAKEAWAYS: – single query approach to Funnel analysis (can be applied to any funnel-like problem) – using window functions to ensure ordering of the events in the funnel – examples of higher order functions to calculate funnel metrics
This talk shows how to extract (structured) value from the huge amount of (unstructured) information that logs contain using InfluxData technologies.
Particularly the task is achieved using two pieces of code I wrote: the influxdata/go-syslog library and the Telegraf Syslog Input Plugin.
The slides demonstrate how to parse logs and to store consequent time-series data into InfluxDB.
At this point it is possible to visualize them via the new Chronograf's Log Viewer, eliciting new meaningful metrics to plot (eg., number of process OOM killed) processing them via a Kapacitor UDF.
The stack used to achieve this is:
- Telegraf with the syslog input plugin, which uses this blazing fast go-syslog parser
- Chronograf with its new Log Viewer
- InfluxDB
- Kapacitor
Companion source code and repository is at http://bit.ly/logs-2-metrics-influx-code
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Data Provenance Support in...Data Con LA
Debugging data processing logic in Data-Intensive Scalable Computing (DISC) systems is a difficult and time consuming effort. To aid this effort, we built Titian, a library that enables data provenance tracking data through transformations in Apache Spark.
5 must have patterns for your microservice - techoramaAli Kheyrollahi
"Netflix is actually a log generating application that just happens to stream movies"
Building a service/Microservice is itself easy. Scaling it on the cloud is not that hard either but operating, maintaining and iterating a production large scale service is not just about linearisation. As Cockcroft points out, telemetry and monitoring is the most important aspect of building Microservices
We discuss 5 patterns that any serious Microservice should have:
- Canary (an endpoint reporting health of underlying dependencies)
- IO monitor (measuring all calls from Microservice to external dependencies)
- A circuit breaker
- An ActivityId-Propagator
- An exception and short timeout retry policy
Most today's software is highly static, even if it is written in a dynamic language like Smalltalk. Developers are not encouraged to extend the frameworks they are using; and end-users are unable to change the features of their software without initiating a new development effort. In contrast, extensible software is designed for change; and customizable software can be adapted to new needs without requiring an in-depth knowledge of the underlying implementation domain.
In this presentation I will investigate on how to write truly dynamic software and I will distill common patterns of software customizability. As running examples I present tools that I worked on during my path of discovering Smalltalk. One of these examples is Magritte, a dynamic meta-model that gives end-users the possibility to customize their applications without the need of an additional development effort. Another example is Helvetia, an infrastructure enabling on-the-fly customization of the programming language and development environment.
Data Wars: The Bloody Enterprise strikes backVictor_Cr
I would like to describe such cases when we create problems for "future us" just by an accident. I will show how different Java data types can ease or increase the pain in supporting the application later. Most common pitfals and tricky corner cases you probably have never thought about.
Who would have thought putting 140 charachter messages about one's life online or having a virtual farm game could ever be popular ? Then again, many of us have those weird (but sometimes brilliant) ideas.
But no matter how incredible your ideas might be, getting them launched successfully takes more than writing lots of php code, smacking a sleek design on it and dropping it on a server.
So what does it take ? Where do most ideas crashland and how can you avoid making the same mistakes and transform your ideas into reality ? We'll look at what steps are needed to make a service successful and sustainable.
With PHP 8.0 recently released and PHP 5.x still accounting for over 40% of all production environments, it's time to paint a clear picture on not just why everyone should move to 8.x, but on how to get code ready for the latest version of PHP. In this talk, we'll look at some handy tools and techniques to ease the migration.
With PHP 7.2 recently released and PHP 5.3 and 5.4 still accounting for over 40% of all production environments, it's time to paint a clear picture on not just why everyone should move to 7.0 (or preferably 7.1), but on how to get code ready for the latest version of PHP.
Using the version compatibility checker for PHP_CodeSniffer and a few simple step-by-step instructions, upgrading old code to make it compatible with the latest PHP versions becomes actually really easy. In this talk, we'll migrate an old piece of code and get rid of the demons of the past and ready for the present and future.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
The time of static or dynamically generated sites is long gone. Non-stop interaction with users is the new normal. However, polling with Ajax requests is processor intensive and cumbersome. Websockets allow you to interact with users in real-time without increasing system load. We'll go through the basics and see all the different options, illustrated with live examples of how and when to use it, as well as when not to use it.
Who would have thought putting 140 charachter messages about one's life online or having a virtual farm game could ever be popular ? Then again, many of us have those weird (but sometimes brilliant) ideas.
But no matter how incredible your ideas might be, getting them launched successfully takes more than writing lots of php code, smacking a sleek design on it and dropping it on a server.
So what does it take ? Where do most ideas crashland and how can you avoid making the same mistakes and transform your ideas into reality ? We'll look at what steps are needed to make a service successful and sustainable.
Your app lives on the network - networking for web developersWim Godden
Our job might be to build web applications, but we can't build apps that rely on networking if we don't know how these networks and the big network that connects them all (this thing called the Internet) actually work.
I'll walk through the basics of networking, then dive a lot deeper (from TCP/UDP to IPv4/6, source/destination ports, sockets, DNS and even BGP).
Prepare for an eye-opener when you realize how much a typical app relies on all of these (and many more) working flawlessly... and how you can prepare your app for failure in the chain.
With PHP 7.2 recently released and PHP 5.3 and 5.4 still accounting for over 40% of all production environments, it's time to paint a clear picture on not just why everyone should move to 7.0 (or preferably 7.1), but on how to get code ready for the latest version of PHP.
Using the version compatibility checker for PHP_CodeSniffer and a few simple step-by-step instructions, upgrading old code to make it compatible with the latest PHP versions becomes actually really easy. In this talk, we'll migrate an old piece of code and get rid of the demons of the past and ready for the present and future.
With PHP 7.2 recently released and PHP 5.3 and 5.4 still accounting for over 40% of all production environments, it's time to paint a clear picture on not just why everyone should move to 7.0 (or preferably 7.1), but on how to get code ready for the latest version of PHP.
Using the version compatibility checker for PHP_CodeSniffer and a few simple step-by-step instructions, upgrading old code to make it compatible with the latest PHP versions becomes actually really easy. In this talk, we'll migrate an old piece of code and get rid of the demons of the past and ready for the present and future.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
The time of static or dynamically generated sites is long gone. Non-stop interaction with users is the new normal. However, polling with Ajax requests is processor intensive and cumbersome. Websockets allow you to interact with users in real-time without increasing system load. We'll go through the basics and see all the different options, illustrated with live examples of how and when to use it.
Your app lives on the network - networking for web developersWim Godden
Our job might be to build web applications, but we can't build apps that rely on networking if we don't know how these networks and the big network that connects them all (this thing called the Internet) actually work.
I'll walk through the basics of networking, then dive a lot deeper (from TCP/UDP to IPv4/6, source/destination ports, sockets, DNS and even BGP).
Prepare for an eye-opener when you realize how much a typical app relies on all of these (and many more) working flawlessly... and how you can prepare your app for failure in the chain.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
From ReactPHP to Facebook Hack's Async implementation and many more, asynchronous programming has been a 'hot' topic lately. But how well does async programming support work in PHP and what can you actually use it for in your projects ? Let's look at some real-world use cases and how they leverage the power of async to do things you didn't know PHP could do.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
A practical step-by-step guide to Git, taking you through each phase of a project and explaining the use of Git at each step of the development process. Expect lots of how-to, but also some how-not-to, to avoid going down the wrong path.
With more and more sites falling victim to data theft, you've probably read the list of things (not) to do to write secure code. But what else should you do to make sure your code and the rest of your web stack is secure ? In this tutorial we'll go through the basic and more advanced techniques of securing your web and database servers, securing your backend PHP code and your frontend javascript code. We'll also look at how you can build code that detects and blocks intrusion attempts and a bunch of other tips and tricks to make sure your customer data stays secure.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
13. Who am I ?
Wim Godden (@wimgtr)
Founder of Cu.be Solutions (http://cu.be)
Open Source developer since 1997
Developer of OpenX, PHPCompatibility, PHPConsistent, Nginx
SLIC, ...
Speaker at PHP and Open Source conferences
14. Cu.be Solutions ?
Open source consultancy
PHP-centered
Training courses
High-speed redundant network (BGP, OSPF, VRRP)
High scalability development
Nginx + extensions
MySQL Cluster
Projects :
mostly IT & Telecom companies
lots of public-facing apps/sites
15. Who are you ?
Developers ?
Anyone setup a MySQL master-slave ?
Anyone setup a site/app on separate web and database server ?
→ How much traffic between them ?
16. The topic
Things we take for granted
Famous last words : "It should work just fine"
Works fine today
→ might fail tomorrow
Most common mistakes
PHP code ↔ PHP ecosystem
18. Database queries – complexity
SELECT DISTINCT n.nid, n.uid, n.title, n.type, e.event_start, e.event_start AS
event_start_orig, e.event_end, e.event_end AS event_end_orig, e.timezone,
e.has_time, e.has_end_date, tz.offset AS offset, tz.offset_dst AS offset_dst,
tz.dst_region, tz.is_dst, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst,
tz.offset) HOUR_SECOND AS event_start_utc, e.event_end - INTERVAL
IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND AS event_end_utc,
e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND +
INTERVAL 0 SECOND AS event_start_user, e.event_end - INTERVAL IF(tz.is_dst,
tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0 SECOND AS
event_end_user, e.event_start - INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset)
HOUR_SECOND + INTERVAL 0 SECOND AS event_start_site, e.event_end -
INTERVAL IF(tz.is_dst, tz.offset_dst, tz.offset) HOUR_SECOND + INTERVAL 0
SECOND AS event_end_site, tz.name as timezone_name FROM node n INNER
JOIN event e ON n.nid = e.nid INNER JOIN event_timezones tz ON tz.timezone =
e.timezone INNER JOIN node_access na ON na.nid = n.nid LEFT JOIN
domain_access da ON n.nid = da.nid LEFT JOIN node i18n ON n.tnid > 0 AND
n.tnid = i18n.tnid AND i18n.language = 'en' WHERE (na.grant_view >= 1 AND
((na.gid = 0 AND na.realm = 'all'))) AND ((da.realm = "domain_id" AND da.gid = 4)
OR (da.realm = "domain_site" AND da.gid = 0)) AND (n.language ='en' OR
n.language ='' OR n.language IS NULL OR n.language = 'is' AND i18n.nid IS NULL)
AND ( n.status = 1 AND ((e.event_start >= '2010-01-31 00:00:00' AND
e.event_start <= '2010-03-01 23:59:59') OR (e.event_end >= '2010-01-31 00:00:00'
AND e.event_end <= '2010-03-01 23:59:59') OR (e.event_start <= '2010-01-31
00:00:00' AND e.event_end >= '2010-03-01 23:59:59')) ) GROUP BY n.nid HAVING
(event_start >= '2010-02-01 00:00:00' AND event_start <= '2010-02-28 23:59:59')
OR (event_end >= '2010-02-01 00:00:00' AND event_end <= '2010-02-28 23:59:59')
OR (event_start <= '2010-02-01 00:00:00' AND event_end >= '2010-02-28
23:59:59') ORDER BY event_start ASC;
19. Database - indexing
'select id from stock where status = 2 order by qty'
→ aggregate index on (status, qty)
'select id from stock where status > 2 order by qty'
→ aggregate index on (status, qty) ?
→ Depends :
- Btree : yes
- Hash : range selection stops use of aggregate index
→ separate index on status and qty (since recent versions)
20. Database - indexing
Indexes make database faster
→ Let's index everything !
→ DON'T :
Insert/update/delete → Index modification
Each query → evaluation of all indexes
"Relational schema design is based on data
but index design is based on queries"
(Bill Karwin, Percona)
21. Databases – detecting problematic queries
Slow query log
→ SET GLOBAL slow_query_log = ON;
Queries not using indexes
→ In my.cnf/my.ini : 'log_queries_not_using_indexes'
General query log
→ SET GLOBAL general_log = ON;
→ Turn it off quickly !
Percona Toolkit (Maatkit)
pt-query-digest
26. Databases – next step : explain
Type of lookup
'system', 'const' and 'ref' = good
'ALL' = bad
Extra info
Using index = good
Using filesort = usually bad
27. For / foreach
$customers = CustomerQuery::create()
->filterByState('MN')
->find();
foreach ($customers as $customer) {
$contacts = ContactsQuery::create()
->filterByCustomerid($customer->getId())
->find();
foreach ($contacts as $contact) {
doSomestuffWith($contact);
}
}
29. Better...
10001 → 1 query
Sadly : people still produce code with query loops
Usually :
Growth not anticipated
Internal app → Public app
30. The origins of this talk
Customers :
Projects we built
Projects we didn't build, but got pulled into
Fixes
Changes
Infrastructure migration
15 years of 'how to cause mayhem with a few lines of code'
31. Client X
Jobs search site
Monitor job views :
Daily hits
Weekly hits
Monthly hits
Which user saw which job
32. Client X
Originally : when user viewed job details
Now : when job is in search result
Search for 'php' → 50 jobs = 50 jobs to be updated
→ 50 updates for shown_today
→ 50 updates for shown_week
→ 50 updates for shown_month
→ 50 inserts for shown_user
= 200 queries for 1 search !
33. Client X : the code
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}
38. Client X : possible cause ?
Code changes ?
→ According to developers : none
Action : turn on general log, analyze with pt-query-digest
→ 50+-fold increase in 4 queries
→ Developers : 'Oops we did make a change'
After 3 days : 2,5 days behind
Every hour : 50 min extra lag
39. Client X : But why is the slave lagging ?
Master Slave
File :
master-bin-xxxx.log
File :
master-bin-xxxx.logSlave I/O thread
Binlog dump
thread
Slave
SQL
thread
42. Client X : fix ?
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}
43. Client X : the code change
insert into shown_today values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_week values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_month values (5, 1), (8, 1), (12, 1), (18, 1), … on duplicate key … ;
insert into shown_user values (5, 23, "2013-11-12 12:01:00"), (8, 23, "2013-11-12
12:01:00"), … ;
44. Client X : the code change
$todayQuery = "
insert into shown_today(
jobId,
number
) values ";
foreach ($jobs as $job) {
$todayQuery .= "(" . $job['id'] . ", 1),";
}
$todayQuery = substr($todayQuery, 0, strlen($todayQuery) - 1);
$todayQuery .= "
)
on duplicate key
update
number = number + 1
";
$db->query($todayQuery);
45. Client X : the chosen solution
$db->autocommit(false);
foreach ($jobs as $job) {
$db->query("
insert into shown_today(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_week(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_month(
jobId,
number
) values(
" . $job['id'] . ",
1
)
on duplicate key
update
number = number + 1
");
$db->query("
insert into shown_user(
jobId,
userId,
when
) values (
" . $job['id'] . ",
" . $user['id'] . ",
now()
)
");
}
$db->commit();
46. Client X : conclusion
For loops are bad (we already knew that)
Add master/slave and it gets much worse
Use transactions : it will provide huge performance increase
Result : slave caught up 5 days later
47. Database → Network
Customer Y
Top 10 site in Belgium
Growing rapidly
At peak traffic :
Unexplicable latency on database
Load on webservers : minimal
Load on database servers : acceptable
50. Client Y : network overload
Cause : Drupal hooks → retrieving data that was not needed
Only load data you actually need
Don't know at the start ? → Use lazy loading
Caching :
Same story
Memcached/Redis are fast
But : data still needs to cross the network
51. Network trouble : more than just traffic
Customer Z
150.000 visits/day
News ticker :
XML feed from other site (owned by same customer)
Cached for 15 min
52. Customer Z – fetching the feed
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
unlink(APP_DIR . '/tmp/cacheFile.xml');
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
file_get_contents('http://www.scrambledsitename.be/xml/feed.xml')
);
}
$xmlfeed = ParseXmlFeed(APP_DIR . '/tmp/cacheFile.xml');
What's wrong with this code ?
53. Customer Z – no feed without the source
Feed source
54. Customer Z – no feed without the source
Feed source
55. Customer Z : timeout
default_socket_timeout : 60 sec by default
Each visitor : 60 sec wait time
People keep hitting refresh → more load
More active connections → more load
Apache hits maximum connections → entire site down
60. Customer Z : process early
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
$feed = file_get_contents('http://www.scrambledsitename.be/xml/feed.xml', false, $context);
if ($feed !== false) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
ParseXmlFeed($feed)
);
}
}
61. Customer Z : file_[get|put]_contents atomicity
$context = stream_context_create(
array(
'http' => array(
'timeout' => 5
)
)
);
if (filectime(APP_DIR . '/tmp/cacheFile.xml') < time() - 900) {
$feed = file_get_contents('http://www.scrambledsitename.be/xml/feed.xml', false, $context);
if ($feed !== false) {
file_put_contents(
APP_DIR . '/tmp/cacheFile.xml',
ParseXmlFeed($feed)
);
}
}
Relying on user → concurrent request → possible corrupt data
Better : run every 15min through cronjob
62. Network resources
Use timeouts for all :
fopen
curl
SOAP
…
Data source trusted ?
→ setup a webservice
→ let them push updates when their feed changes
→ less load on data source
→ no timeout issues
Add logging → early detection
63. Logging
Logging = good
Logging in PHP using fopen
→ bad idea : locking issues
→ Use monolog : file, syslog, mail, Pushover, HipChat, Graylog,
Rollbar, ElasticSearch (and 50 more)
For Firefox : FirePHP (add-on for Firebug)
Debug logging = bad on production
Watch your logs !
Don't log on slow disks → I/O bottlenecks
64. File system : I/O bottlenecks
Causes :
Excessive writes (database updates, logfiles, swapping, …)
Excessive reads (non-indexed database queries, swapping, small file
system cache, …)
How to detect ?
top
iostat
See iowait ? Stop worrying about php, fix the I/O problem !
Cpu(s): 0.2%us, 3.0%sy, 0.0%ni, 61.4%id, 35.5%wa, 0.0%hi, 0.0%si, 0.0%st
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.96 53.70 0.00 45.24
Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
sda 120.40 0.00 123289.60 0 616448
sdb 2.10 0.00 4378.10 0 18215
dm-0 4.20 0.00 36.80 0 184
dm-1 0.00 0.00 0.00 0 0
65. Much more than code
DB
server
Webserver
User
Network
XML feed
70. Step-by-step : most common issues
Using NFS ? Get rid of it ;-)
iowait on database server
I/O reads (use iostat) ? → missing/wrong indexes
I/O writes ?
→ no transactions ?
→ too many queries ?
→ bad DB engine settings
iowait on webserver (logs ? static files ?)
CPU on database server (missing/wrong/too many indexes)
CPU on webserver (PHP)
Editor's Notes
5kbit/sec or 100Mbit/sec ?
Let&apos;s talk about code
Without : we don&apos;t exist
What are most common mistakes in ecosystem
Let&apos;s start with the database
time spent per query pattern
how many queries of that query pattern
Get back to what I said
Lots of people use ORM
- easier
- don&apos;t need to write queries
- object-oriented
but people start doing this
Imagine 10000 customers → 10001 queries
Not best code
Uses deprecated mysql extension
no error handling
Master : 16 CPU cores
12 cores for SQL
1 core for binlog dump
rest for system
Slave : 16 CPU cores
1 core for slave I/O
1 core for slave SQL
Grouping
Works fine, but :
maximum size of string ?
PHP = no limit
MySQL = max_allowed_packet
Grouping
Works fine, but :
maximum size of string ?
PHP = no limit
MySQL = max_allowed_packet
All in a single commit
Note : transaction has max. size
Possible : combination with previous solution
took few moments to figure out
No network monitoring
→ iptraf
→ 100Mbit/sec limit
→ packets dropped
→ connections dropped
Customer : upgrade switch
Us : why 100Mbit/sec ?
Databases → network
What other network related issues ?
Server on which feed located : crashed
Fine for few minutes (cache)
15 minutes : file_get_contents uses default_socket_timeout
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
Better, not perfect.
What else is wrong ?
Multiple visitors hit expiring cache
→ file delete
→ xml feed hit a lot
How do you treat your data :
- where do you get it
- how long did you have to wait to get it
- how is it transported
- how is it processed
minimize the amount of data :
retrieved
transported
processed,
sent to db and users