Gearman is an open-source application framework that allows for the distribution of work across multiple machines. It uses a queueing system to dispatch jobs to workers to be processed asynchronously. Clients submit jobs to the Gearman server which handles distributing jobs to available worker processes to execute the requested tasks. This allows for scalable and fault-tolerant processing of jobs in parallel across a pool of machines.
Queues can provide parallel processing, cross language scripting and more! The talk was focused on Gearman but the principles apply to any alternative.
We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.
Faster PHP apps using Queues and WorkersRichard Baker
PHP apps typically perform tasks in a synchronous manner; Resizing an image or sending a push notification. For most applications this works well, but as apps grow or experience increased traffic, each task adds extra milliseconds to a request, leaving users waiting.
A common solution is to defer these tasks to the background using a cron task. However, there is a better way. Job queues not only help to decouple your application and improve resilience but will also cut request times.
In this talk I we’ll explore some common queue systems; the features and tradeoffs of each solution, what to queue, refactoring existing code into jobs, and running workers. By the end you’ll be ready to build your next app one job at a time.
Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.
Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.
Queues can provide parallel processing, cross language scripting and more! The talk was focused on Gearman but the principles apply to any alternative.
We use Gearman for managing queue system. This covers why we should use a queue in many situations on web-based interface as well as server-side application.
Faster PHP apps using Queues and WorkersRichard Baker
PHP apps typically perform tasks in a synchronous manner; Resizing an image or sending a push notification. For most applications this works well, but as apps grow or experience increased traffic, each task adds extra milliseconds to a request, leaving users waiting.
A common solution is to defer these tasks to the background using a cron task. However, there is a better way. Job queues not only help to decouple your application and improve resilience but will also cut request times.
In this talk I we’ll explore some common queue systems; the features and tradeoffs of each solution, what to queue, refactoring existing code into jobs, and running workers. By the end you’ll be ready to build your next app one job at a time.
Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.
Gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. In other words, it is the nervous system for how distributed processing communicates.
Mad scalability: Scaling when you are not GoogleAbel Muíño
A brief history of how Cabify grew during the last couple of years without using cloud platforms like AWS or Google Cloud
I gave this talk at the madScalability meet up on Madrid, March 2017.
Cachopo - Scalable Stateful Services - Madrid Elixir MeetupAbel Muíño
This is an introduction to building our services in a different way, where state is moved out of the database and into the services (as opposed to mainstream stateless servers).
It also describes one particular proof-of-concept tool that Cabify built during its annual offsite.
This talk was presented at OSCON 2009 and YAPC::NA 2009. The choices discussed here are still relevant, although Plack has opened up some new options. For a more up-to-date take on this material I would recommend Tatsuhiko Miyagawa's most recent Plack/PSGI talks.
Ruby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.comIlya Grigorik
A high-performance proxy server is less than a hundred lines of Ruby code and it is an indispensable tool for anyone who knows how to use it. In this session we will first walk through the basics of event-driven architectures and high-performance network programming in Ruby using the EventMachine framework.
Handling 10k requests per second with Symfony and Varnish - SymfonyCon Berlin...Alexander Lisachenko
It is believed that the Symfony framework is quite heavy and it can be difficult to develop a website that will be able to work under the high load. It is true, but does this mean that it is impossible to implement a dynamic caching and to update only a small part of entire page as data is updated? This talk will give your an answer to that. It is Varnish, ESI-blocks and load balancing. With hundreds of thousands of unique visitors and million hits a day, we continue to use Symfony, and do not see any problems.
Nagios Conference 2012 - Jason Cook - Nagios and Mod-GearmanNagios
Jason Cook's presentation on using Nagios with Mod-Gearman.
The presentation was given during the Nagios World Conference North America held Sept 25-28th, 2012 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna
Mad scalability: Scaling when you are not GoogleAbel Muíño
A brief history of how Cabify grew during the last couple of years without using cloud platforms like AWS or Google Cloud
I gave this talk at the madScalability meet up on Madrid, March 2017.
Cachopo - Scalable Stateful Services - Madrid Elixir MeetupAbel Muíño
This is an introduction to building our services in a different way, where state is moved out of the database and into the services (as opposed to mainstream stateless servers).
It also describes one particular proof-of-concept tool that Cabify built during its annual offsite.
This talk was presented at OSCON 2009 and YAPC::NA 2009. The choices discussed here are still relevant, although Plack has opened up some new options. For a more up-to-date take on this material I would recommend Tatsuhiko Miyagawa's most recent Plack/PSGI talks.
Ruby Proxies for Scale, Performance, and Monitoring - GoGaRuCo - igvita.comIlya Grigorik
A high-performance proxy server is less than a hundred lines of Ruby code and it is an indispensable tool for anyone who knows how to use it. In this session we will first walk through the basics of event-driven architectures and high-performance network programming in Ruby using the EventMachine framework.
Handling 10k requests per second with Symfony and Varnish - SymfonyCon Berlin...Alexander Lisachenko
It is believed that the Symfony framework is quite heavy and it can be difficult to develop a website that will be able to work under the high load. It is true, but does this mean that it is impossible to implement a dynamic caching and to update only a small part of entire page as data is updated? This talk will give your an answer to that. It is Varnish, ESI-blocks and load balancing. With hundreds of thousands of unique visitors and million hits a day, we continue to use Symfony, and do not see any problems.
Nagios Conference 2012 - Jason Cook - Nagios and Mod-GearmanNagios
Jason Cook's presentation on using Nagios with Mod-Gearman.
The presentation was given during the Nagios World Conference North America held Sept 25-28th, 2012 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/nwcna
Many services and applications are ill-equipped to handle a sudden rush of popularity, so their services either become unavailable or unbearably slow. By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and their ability to work together towards a common goal, you can achieve greater performance, more redundancy, higher availability and have the ability to scale services up and down as required easily. By leveraging systems such as gearman, memcache, daemons, message queues, load balancers and more, you too can enter the world of distributed systems and scalability.
Many services / applications now a day are ill equipped with handling a sudden rush of popularity, as is often the case on the internet now a days, to a point where the services either become unavailable or unbearably slow.
By taking a chapter from the ant colonies in the wild, where their strength lies in their numbers and the fact that everyone works together towards the same goal, we can apply the same principle to our service by using systems such as
- gearman
- memcache
- daemons
- message queues
- load balancers
and many more, you can achieve greater performance, more redundancy, higher availability and have the ability to scale your services up and down as required easily.
During this talk attendees will be lead through the world of distributed systems and scalability, and shown the how, where and what, of how to take the average application and splitting it into smaller more manageable pieces
WebCamp: Developer Day: N2O: The Most Powerful Erlang Web Framework - Максим ...GeeksLab Odessa
N2O: The Most Powerful Erlang Web Framework
Максим Сохацкий
N2O разрабатывался как многофункциональный WebSocket аппликейшин сервер широкого назначения. Будучи ответвлен изначально от классического сервер-рендер фреймворка Nitrogen, он развился в мощное средство (включающее в том числе и поддержку SPA приложений) обеспечения пула долгоживущих TCP соеднений и доставки кода/даных выступая в качестве высокоемкностного релея, благодаря надежной и высокоустойчивой платформе Erlang/OTP. Соединяя в себе все черты Full-Stack веб фреймворков: Роутинг, Темплейтинг, Pub/Sub, Сессии, Хранилище, REST и поддержка кастомных протоколов; N2O добавляет совершенно уникальные возможности для создания различных DSL: вы можете транслировать бизнес логику, написанную на Erlang языках (Elixir, Erlang, Joxa) в JavaScript, генерировать трансформаций Erlang рекордов в JSON структуры и много другое, специфицировать JavaScript протоколы на языке Erlang. Сейчас N2O развивается в сторону поддержки SVG клиентов и бинарных протоколов.
Gearman is a client/server infrastructure for generic tasks, usable on distributed servers, with little worry about the details. No matter what language you speak, Gearman can meet your needs in C, PHP, Perl, Ruby, shell scripting, and several more. Gearman can also work in conjunction with MySQL, either using UDFs, or simply through its basic architecture.
This talk will show examples of how to use Gearman for remote installation and how to call functions written in Perl from any other language or from inside MySQL server, with no knowledge of Perl at all.
Home automation with Perl (FHEM).
I've used a Raspberry Pi with 868MHz technology to gather data about my home, and view/analyse this with Logstash, Elasticsearch and Kibana.
Adopting GraalVM - Scale by the Bay 2018Petr Zapletal
After many years of development, Oracle finally published GraalVM and sparkled a lot of interest in the community. GraalVM is a high-performance polyglot VM with a number of potentially interesting traits we can take advantage of like increased performance and lowered cost. It can also tackle shortcomings of JVM/Scala we are struggling for years like slow-startup times or large jars. Lastly, thanks to its polyglot nature it can open interesting doors we may want to discover. On the other hand, GraalVM may still be bleeding edge technology and having a hard time to deliver the promised features. In this talk, I’d like to discuss advantages and disadvantages of adopting GraalVM, provide you guidance if you decide to do so and also share our story in this area including various samples, and recommendations. This talk is focused on JVM and Scala but should be beneficial for everyone with interested in this topic.
How Bitbucket Pipelines Loads Connect UI Assets Super-fastAtlassian
Connect add-ons deliver better user experience when they load fast. Between CDN, server-side rendering, service workers, and code splitting, there are loads of techniques you can use to achieve this. In this session, Atlassian Developer Peter Plewa will reveal Bitbucket Pipelines' secret for fast loads, and what they can do in the future to make Pipelines even faster.
Peter Plewa, Development Principal, Atlassian
Unit testing in JavaScript? There is no such thing.” This is something most of the Java developers would say. With AngularJS coming more and more to scene and Google standing behind it, testing is starting to be core part of all AngularJS project. I would like to show how you can do unit testing in pure JavaScript (AngularJS) application (together with backend mocking…).
My talk from DevOpsCon Berlin 2016.
Ansible is a radically simple and lightweight provisioning framework which makes your servers and applications easier to provision and deploy. By orchestrating your application deployments you gain benefits such as documentation as code, testability, continuous integration, version control, refactoring, automation and autonomy of your deployment routines, server and application configuration. Ansible uses a language that approaches plain English, uses SSH and has no agents to install on remote systems. It is the simplest way to automate and orchestrate application deployment, configuration management and continuous delivery.
In this tutorial you will be given an introduction to Ansible and learn how to provision Linux servers with a web-proxy, a database and some other packages. Furthermore we will automate zero downtime deployment of a Java application to a load balanced environment.
1. Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
2. Presentation done using info from... http://www.slideshare.net/pcdinh/gearman-and-asynchronous-processing-in-php-applications-6135047 http://assets.en.oreilly.com/1/event/45/The%20Gearman%20Cookbook%20Presentation.pdf http://www.gearman.org http://nz.php.net/manual/en/book.gearman.php others...
3.
4.
5. Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
6. Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension="gearman.so"
7.
8. “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
9.
10. Client Worker Job Server Job Server Client Client Client Worker Worker Worker
16. $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
17.
18. GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback