Redis & ZeroMQ: How to scale your applicationrjsmelo
Presented at #PHPLX 11 July 2013
When you need to do some heavy processing how do you scale you application?
You can use Redis and ZeroMQ to leverage the heavy work for you!
With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.
Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density.
Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed.
Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances.
www.serverdensity.com/puppetcamp/
Redis is being used as a message queue to asynchronously process image uploads on a website for gaming screenshots. When a user uploads images, the application server adds a message to the Redis queue containing metadata about the upload. A separate process polls the queue and processes each upload by resizing images, creating database entries, and more. This allows upload processing to happen in the background without blocking the user.
A talk I held at perl mongers Wellington about RabbitMQ and Net::AMQP. A brief introduction to RabbitMQ and some basic code samples on how to use it from perl.
A talk I gave at WordCamp Sofa 2016 on measuring and optimizing memory usage, dealing with memory related errors, as well as monitoring server memory health.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
Redis & ZeroMQ: How to scale your applicationrjsmelo
Presented at #PHPLX 11 July 2013
When you need to do some heavy processing how do you scale you application?
You can use Redis and ZeroMQ to leverage the heavy work for you!
With this presentation we will know more about this two technologies and how they can be used to help solve problems with the performance and scalability of your application.
Going from zero to Puppet by Pedro Pessoa, Operations Engineer at Server Density.
Abstract: Using out-of-the-box Puppet for non-sysadmin work - steps from going from no config management to managing 100 nodes and allowing non-sysadmin tasks to be performed.
Speaker Bio: Linux admin for 10+ years. Java/Python/C developer 12+ years. Ops engineer at http://www.serverdensity.com - a hosted server and website monitoring service. Currently processing 12TB+ per month into MongoDB running on dedicated and virtual instances.
www.serverdensity.com/puppetcamp/
Redis is being used as a message queue to asynchronously process image uploads on a website for gaming screenshots. When a user uploads images, the application server adds a message to the Redis queue containing metadata about the upload. A separate process polls the queue and processes each upload by resizing images, creating database entries, and more. This allows upload processing to happen in the background without blocking the user.
A talk I held at perl mongers Wellington about RabbitMQ and Net::AMQP. A brief introduction to RabbitMQ and some basic code samples on how to use it from perl.
A talk I gave at WordCamp Sofa 2016 on measuring and optimizing memory usage, dealing with memory related errors, as well as monitoring server memory health.
The document discusses Nouka, an open source inventory management tool for Linux. Nouka consists of three parts - Nouka data collector, Naya data store, and Yaoya data converter. Nouka data collector runs commands periodically on Linux machines and sends the results to Naya data store. Naya uses Fluentd and MongoDB to store the collected data. Yaoya then converts and outputs the data in various formats like JSON, CSV for analysis. Overall, Nouka provides an automatic and periodic way to collect and centralize inventory data from Linux machines.
agri inventory - nouka data collector / yaoya data convertorToshiaki Baba
This document provides instructions for setting up and using an agri inventory system called nouka and yaoya. Nouka collects data from servers using commands and sends it to the naya data store, which uses fluentd and MongoDB. It explains the components, data formats, and provides steps to get the required software and set up the system.
The document discusses extending Nginx functionalities with Lua. It provides an overview of Nginx architecture and how the lua-nginx-module allows running Lua scripts inside Nginx. This provides a powerful and performant programming environment while leveraging Nginx's event-driven architecture. Examples show how to access Nginx variables and APIs from Lua, issue subrequests, and handle requests non-blockingly using cosockets. Libraries like lua-resty-memcached reuse these extensions to build applications in a scalable manner.
The document contains instructions for creating various prank scripts that can be run on other people's computers without permission to annoy or confuse them. It includes scripts that continuously pop up messages, repeatedly open and close the CD tray, enable caps lock, type text in Notepad, and more. Most of the scripts are simple batches or VBScripts that can be easily created and run. The document encourages modifying the scripts for different purposes and pranking other users.
This document describes using Puppet to launch and maintain EC2 instances with pre-defined software and configurations. It discusses building AMIs with pre-installed but unconfigured software, and using Puppet classes to activate and configure specific packages, services, and monitoring tools on the instances. The classes split package, service, configuration, and monitoring roles, and can be included and parameterized to deploy NTP in different roles on instances.
Czy wiesz co potrafi zrobić twój serwer reverse-proxy? Wydaje Ci się, że żeby zrobić sprytny routing / uwierzytelnianie / autoryzację (niepotrzebne skreślić) między serwisami musisz go napisać w Javie lub jako moduł w C? A co jeżeli odpalanie JVM tylko po to, żeby do każdego żądania http dokleić jeden nagłówek to armata na wróbla? Zwłaszcza, że prawie na pewno gdzieś tam po drodze mijasz nginx... Zapraszam Cię do świata idealnej symbiozy nginx i Lua.
The document lists several dangerous commands that can be run on a Linux or Unix system and cause data loss or system instability. It also provides basic commands for file operations, package management, networking tasks and changing permissions and passwords. The dangerous commands include deleting all files in the root folder, formatting a partition, executing a fork bomb to overload the system, and overwriting the file system on a designated device. The basic commands cover file listing, copying, downloading, creating directories and viewing directory information. It also explains Linux file permissions in terms of read, write and execute attributes.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
Ondřej Šika: Docker, Traefik a CI - Mějte nasazené všeny větve na kterých pra...Develcz
The document describes setting up Docker, Traefik, and CI/CD pipelines. It includes a docker-compose.yml configuration file for Traefik that sets up port forwarding and SSL termination. It also includes a .gitlab-ci.yml file that defines a deploy job that builds a Docker image, pushes it to Docker Hub, and deploys it to a server using Traefik routing.
PSR-7 and PSR-15 are PHP standards for representing HTTP messages and server request handling. PSR-7 defines interfaces for HTTP messages like requests and responses, while PSR-15 defines interfaces for request handlers and middleware. These standards provide a common way for PHP libraries to interact with HTTP messages and requests, improving interoperability. They also allow applications to be built in a way that is compatible with evolving PHP frameworks and tools. Adopting these PSRs helps create a solid foundation for building HTTP applications and middleware in PHP.
Buffer overflow occurs when too much data is inserted into a buffer than it can handle, which may lead to arbitrary code execution if a memory pointer is overwritten. The document provides an example of a buffer overflow happening when user input longer than the buffer size is read using gets() into the buffer without limits. To prevent this, fgets() should be used instead, as it allows specifying the maximum number of characters to read from input.
OlinData Puppet Presentation for MOSC 2012Walter Heck
Walter Heck gave a presentation on Puppet at the 2012 Malaysia Open Source Conference. He introduced OlinData, which provides Puppet consulting and training, and discussed how Puppet allows for scalable, accountable, and versioned infrastructure configuration compared to manual methods. Heck also covered OlinData's Puppet training schedule and consulting services to help companies implement and improve their Puppet usage.
This document provides instructions for setting up a CoreOS cluster with services like MongoDB, SkyDNS, HAProxy, and Datadog monitoring. It describes using systemd unit files to run services like MongoDB and SkyDNS in Docker containers managed by Fleet. Etcd is used for service discovery and configuration management with ConfD. HAProxy provides load balancing of services, and Datadog monitors the cluster. Cloud-init scripts initialize new nodes joining the cluster.
This document provides an overview of basic Linux commands for tasks like file management, installation, networking, compression, searching, SSH, permissions and more. It explains commands for changing directories, listing files, copying/moving files, installing packages, pinging hosts, compressing files and directories, searching for patterns, remote login via SSH, and managing processes and system information.
An (abridged) Ruby Plumber's Guide to *nixEleanor McHugh
This document provides an overview of using Ruby to interface with Unix processes, files, and interprocess communication. It discusses using the Kernel module to spawn and manage processes, reading and writing files using IO methods and fcntl constants, communicating between processes via pipes and sockets, directly calling Unix syscalls via Kernel.syscall and the ruby/dl library, and examples of using POSIX semaphores and memory management with ruby/dl. Further reading links are provided for more in-depth information.
This document provides tips and tricks for using the command line more efficiently. It discusses using SSH configuration files and the screen command to manage terminal sessions. Other topics covered include compressing files with tar, navigating directories more easily, monitoring log files, managing background processes, and using shortcuts to save time typing commands.
This document provides an overview of configuration management with Cfengine. It discusses what Cfengine is and how it can be used to ensure files and processes are configured properly across multiple hosts. It then describes how to get started with Cfengine, including installing it, writing policies to configure a single host, and setting up a basic infrastructure to distribute policies to multiple hosts on a network. Key components like cfagent, cfservd, classes, and policies are defined. Examples of Cfengine configuration files are also provided.
This document provides instructions for installing the Nginx web server from source on Linux. It describes downloading the source code, verifying the signature, compiling it with make, and installing. It also covers configuring Nginx's directories and files, updating to a new version, and configuring a Yum repository to install via package manager.
The document provides an overview of common Linux commands and their functions, such as RMDIR to remove empty directories, CHMOD to change file permissions, GREP for pattern matching, FIND to locate files, ECHO to output text, and MORE to view files page by page. It also covers operators like pipes, logical operators, redirection, and command substitution. The document concludes with descriptions of conditional statements like IF/THEN, FOR, WHILE, UNTIL, and CASE that allow scripts to perform actions conditionally.
RestMQ is a message queue system based on Redis that allows storing and retrieving messages through HTTP requests. It uses Redis' data structures like lists, sets, and hashes to maintain queues and messages. Messages can be added to and received from queues using RESTful endpoints. Additional features include status monitoring, queue control, and support for protocols like JSON, Comet, and WebSockets. The core functionality is language-agnostic but implementations exist in Python and Ruby.
This document discusses the Puppet configuration management tool. It provides an overview of Puppet including its open source nature, supported platforms, file structure, and types of resources it can manage like files, packages, services. It also discusses Facter for collecting system facts. Several examples are shown of how to configure files, packages, services. Finally Amazon EC2 is mentioned as a way to deploy Puppet in a scalable environment.
Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.
The document contains instructions for creating various prank scripts that can be run on other people's computers without permission to annoy or confuse them. It includes scripts that continuously pop up messages, repeatedly open and close the CD tray, enable caps lock, type text in Notepad, and more. Most of the scripts are simple batches or VBScripts that can be easily created and run. The document encourages modifying the scripts for different purposes and pranking other users.
This document describes using Puppet to launch and maintain EC2 instances with pre-defined software and configurations. It discusses building AMIs with pre-installed but unconfigured software, and using Puppet classes to activate and configure specific packages, services, and monitoring tools on the instances. The classes split package, service, configuration, and monitoring roles, and can be included and parameterized to deploy NTP in different roles on instances.
Czy wiesz co potrafi zrobić twój serwer reverse-proxy? Wydaje Ci się, że żeby zrobić sprytny routing / uwierzytelnianie / autoryzację (niepotrzebne skreślić) między serwisami musisz go napisać w Javie lub jako moduł w C? A co jeżeli odpalanie JVM tylko po to, żeby do każdego żądania http dokleić jeden nagłówek to armata na wróbla? Zwłaszcza, że prawie na pewno gdzieś tam po drodze mijasz nginx... Zapraszam Cię do świata idealnej symbiozy nginx i Lua.
The document lists several dangerous commands that can be run on a Linux or Unix system and cause data loss or system instability. It also provides basic commands for file operations, package management, networking tasks and changing permissions and passwords. The dangerous commands include deleting all files in the root folder, formatting a partition, executing a fork bomb to overload the system, and overwriting the file system on a designated device. The basic commands cover file listing, copying, downloading, creating directories and viewing directory information. It also explains Linux file permissions in terms of read, write and execute attributes.
This document discusses Composer, an open source tool for dependency management in PHP. It describes what Composer is, how to install it, how to define dependencies in a composer.json file and composer.lock file, how Composer generates autoload files and installs vendor libraries, and some common Composer commands. It also provides information on joining the community and lists sources for more documentation on Composer.
Ondřej Šika: Docker, Traefik a CI - Mějte nasazené všeny větve na kterých pra...Develcz
The document describes setting up Docker, Traefik, and CI/CD pipelines. It includes a docker-compose.yml configuration file for Traefik that sets up port forwarding and SSL termination. It also includes a .gitlab-ci.yml file that defines a deploy job that builds a Docker image, pushes it to Docker Hub, and deploys it to a server using Traefik routing.
PSR-7 and PSR-15 are PHP standards for representing HTTP messages and server request handling. PSR-7 defines interfaces for HTTP messages like requests and responses, while PSR-15 defines interfaces for request handlers and middleware. These standards provide a common way for PHP libraries to interact with HTTP messages and requests, improving interoperability. They also allow applications to be built in a way that is compatible with evolving PHP frameworks and tools. Adopting these PSRs helps create a solid foundation for building HTTP applications and middleware in PHP.
Buffer overflow occurs when too much data is inserted into a buffer than it can handle, which may lead to arbitrary code execution if a memory pointer is overwritten. The document provides an example of a buffer overflow happening when user input longer than the buffer size is read using gets() into the buffer without limits. To prevent this, fgets() should be used instead, as it allows specifying the maximum number of characters to read from input.
OlinData Puppet Presentation for MOSC 2012Walter Heck
Walter Heck gave a presentation on Puppet at the 2012 Malaysia Open Source Conference. He introduced OlinData, which provides Puppet consulting and training, and discussed how Puppet allows for scalable, accountable, and versioned infrastructure configuration compared to manual methods. Heck also covered OlinData's Puppet training schedule and consulting services to help companies implement and improve their Puppet usage.
This document provides instructions for setting up a CoreOS cluster with services like MongoDB, SkyDNS, HAProxy, and Datadog monitoring. It describes using systemd unit files to run services like MongoDB and SkyDNS in Docker containers managed by Fleet. Etcd is used for service discovery and configuration management with ConfD. HAProxy provides load balancing of services, and Datadog monitors the cluster. Cloud-init scripts initialize new nodes joining the cluster.
This document provides an overview of basic Linux commands for tasks like file management, installation, networking, compression, searching, SSH, permissions and more. It explains commands for changing directories, listing files, copying/moving files, installing packages, pinging hosts, compressing files and directories, searching for patterns, remote login via SSH, and managing processes and system information.
An (abridged) Ruby Plumber's Guide to *nixEleanor McHugh
This document provides an overview of using Ruby to interface with Unix processes, files, and interprocess communication. It discusses using the Kernel module to spawn and manage processes, reading and writing files using IO methods and fcntl constants, communicating between processes via pipes and sockets, directly calling Unix syscalls via Kernel.syscall and the ruby/dl library, and examples of using POSIX semaphores and memory management with ruby/dl. Further reading links are provided for more in-depth information.
This document provides tips and tricks for using the command line more efficiently. It discusses using SSH configuration files and the screen command to manage terminal sessions. Other topics covered include compressing files with tar, navigating directories more easily, monitoring log files, managing background processes, and using shortcuts to save time typing commands.
This document provides an overview of configuration management with Cfengine. It discusses what Cfengine is and how it can be used to ensure files and processes are configured properly across multiple hosts. It then describes how to get started with Cfengine, including installing it, writing policies to configure a single host, and setting up a basic infrastructure to distribute policies to multiple hosts on a network. Key components like cfagent, cfservd, classes, and policies are defined. Examples of Cfengine configuration files are also provided.
This document provides instructions for installing the Nginx web server from source on Linux. It describes downloading the source code, verifying the signature, compiling it with make, and installing. It also covers configuring Nginx's directories and files, updating to a new version, and configuring a Yum repository to install via package manager.
The document provides an overview of common Linux commands and their functions, such as RMDIR to remove empty directories, CHMOD to change file permissions, GREP for pattern matching, FIND to locate files, ECHO to output text, and MORE to view files page by page. It also covers operators like pipes, logical operators, redirection, and command substitution. The document concludes with descriptions of conditional statements like IF/THEN, FOR, WHILE, UNTIL, and CASE that allow scripts to perform actions conditionally.
RestMQ is a message queue system based on Redis that allows storing and retrieving messages through HTTP requests. It uses Redis' data structures like lists, sets, and hashes to maintain queues and messages. Messages can be added to and received from queues using RESTful endpoints. Additional features include status monitoring, queue control, and support for protocols like JSON, Comet, and WebSockets. The core functionality is language-agnostic but implementations exist in Python and Ruby.
This document discusses the Puppet configuration management tool. It provides an overview of Puppet including its open source nature, supported platforms, file structure, and types of resources it can manage like files, packages, services. It also discusses Facter for collecting system facts. Several examples are shown of how to configure files, packages, services. Finally Amazon EC2 is mentioned as a way to deploy Puppet in a scalable environment.
Presentation on how Puppet has been introduced in Seat Pagine Gialle to automate system administration tasks and easy the cooperation between Ops and Others.
Puppet for Java developers - JavaZone NO 2012Carlos Sanchez
Example code at https://github.com/carlossg/puppet-for-java-devs
More info at http://blog.carlossanchez.eu/tag/devops
Video at http://vimeo.com/49483627
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,... in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.
Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,... and easily make changes that propagate across all nodes.
Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.
We’ll show how to install and manage Puppet nodes with JDK, multiple application server instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.
The document discusses how immutable infrastructure can be achieved through Puppet by treating systems configuration as code. Puppet allows defining systems in code and enforcing that state through automatic idempotent runs, compensating for inherent system mutability. This brings predictability to infrastructure and allows higher level operations by establishing a foundation of reliable, known states.
More info at http://blog.carlossanchez.eu/tag/devops
Video en español: http://youtu.be/E_OE4l3t5BA
The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.
Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.
In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.
Continuous Delivery with Maven, Puppet and Tomcat - ApacheCon NA 2013Carlos Sanchez
Continuous Integration, with Apache Continuum or Jenkins, can be extended to fully manage deployments and production environments, running in Tomcat for instance, in a full Continuous Delivery cycle using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations.
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,... in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Apache Continuum or Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.
Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,... and easily make changes that propagate across all nodes.
Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.
We will show how to install and manage Puppet nodes with JDK, multiple Tomcat instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.
Puppet is an open source tool for server configuration management. It allows systems to be configured and maintained in a consistent, automated way. Puppet uses a declarative language to describe system configuration and can manage a variety of operating systems. It provides benefits like reduced entropy, ability to quickly scale machines, change management tracking and repeatable states. Puppet uses a client-server architecture with SSL encryption and supports concepts like templates, defined types and ordering of resources.
Fail2ban is an open source intrusion prevention software developed using the Python programming language. It monitors system logs such as /var/log/pwdfail, /var/log/auth.log, and /var/log/secure for failed login attempts. When the maximum number of failed logins from an IP address within a certain time frame is reached, Fail2ban uses iptables to ban that IP address by adding a DROP rule. It can ban IP addresses for services beyond just SSH, such as SMTP, HTTP, and others. The document then provides instructions on installing and configuring Fail2ban on CentOS 6.3.
More info at http://blog.carlossanchez.eu/2011/11/15/from-dev-to-devops-slides-from-apachecon-na-vancouver-2011/
The DevOps movement aims to improve communication between developers and operations teams to solve critical issues such as fear of change and risky deployments. But the same way that Agile development would likely fail without continuous integration tools, the DevOps principles need tools to make them real, and provide the automation required to actually be implemented. Most of the so called DevOps tools focus on the operations side, and there should be more than that, the automation must cover the full process, Dev to QA to Ops and be as automated and agile as possible. Tools in each part of the workflow have evolved in their own silos, and with the support of their own target teams. But a true DevOps mentality requires a seamless process from the start of development to the end in production deployments and maintenance, and for a process to be successful there must be tools that take the burden out of humans.
Apache Maven has arguably been the most successful tool for development, project standardization and automation introduced in the last years. On the operations side we have open source tools like Puppet or Chef that are becoming increasingly popular to automate infrastructure maintenance and server provisioning.
In this presentation we will introduce an end-to-end development-to-production process that will take advantage of Maven and Puppet, each of them at their strong points, and open source tools to automate the handover between them, automating continuous build and deployment, continuous delivery, from source code to any number of application servers managed with Puppet, running either in physical hardware or the cloud, handling new continuous integration builds and releases automatically through several stages and environments such as development, QA, and production.
Capistrano is an open source tool for running scripts on multiple servers. It’s primary use is for easily deploying applications. While it was built specifically for deploying Rails apps, it’s pretty simple to customize it to deploy other types of applications.
capifony is a deployment recipes collection that works with both symfony and Symfony2 applications.
The document discusses various methods for hardening Linux security, including securing physical and remote access, addressing top vulnerabilities like weak passwords and open ports, implementing security policies, setting BIOS passwords, password protecting GRUB, choosing strong passwords, securing the root account, disabling console programs, using TCP wrappers, protecting against SYN floods, configuring SSH securely, hardening sysctl.conf settings, leveraging open source tools like Mod_Dosevasive, Fail2ban, Shorewall, and implementing security at the policy level with Shorewall.
Puppi is a Puppet modules that drives Puppet's knowledge of the Systems to a command line tool that you can use to check services availability, gather info on the system and deploy application with a single command.
1. The document discusses moving from a Dev to DevOps model by addressing issues like siloization between development and operations teams and embracing concepts like infrastructure as code.
2. It recommends several DevOps tools for infrastructure automation including Puppet, Vagrant, and VeeWee which allow developers to define infrastructure in code and provision environments.
3. The Puppet Domain Specific Language (DSL) is demonstrated for declaring resources like users, files, packages, and services with attributes and relationships between them in a declarative way.
The document discusses deploying a Rails application to Amazon EC2. It explains that the goals are to launch an EC2 instance, connect to it, set up the environment, deploy the application, and profit. It then outlines the plan to launch an instance, connect to it, install necessary packages like Ruby, Rails, and Nginx, configure Nginx and Unicorn, deploy the application using Capistrano, and start the Unicorn process.
Puppet is a tool that allows users to declaratively configure systems. It provides abstraction through defined resources like packages and files, ensures configurations are idempotent, and converges systems to the desired state declaratively rather than imperatively through scripts. Puppet code is organized into reusable modules and managed through version control. Modules should include tests, be validated with tools like puppet-lint, and tested in automated environments like Travis CI to ensure high quality.
PuppetCamp Ghent - What Not to Do with PuppetOlinData
The document discusses common mistakes to avoid when using Puppet, including design mistakes like putting multiple classes in a file, language mistakes like using default options without checking for failures, and resource definition mistakes like recursively declaring ownership on large directories. It provides examples of bad Puppet code patterns and suggests better alternatives to avoid issues like dependency loops. The document is intended to help Puppet users learn from ugly or problematic Puppet code examples.
PuppetCamp Ghent - What Not to Do with PuppetWalter Heck
The document discusses common mistakes to avoid when using Puppet, including design mistakes like poorly structured classes, language mistakes like misusing functionality, and dependency issues. It provides examples of problematic Puppet code and explanations of why they are problematic, such as putting multiple classes in one file, using default options without checking for failures, and creating dependency loops between resources. The goal is to help Puppet users identify and avoid ugly or erroneous Puppet code that could cause problems.
Nmap Scripting Engine and http-enumerationRobert Rowley
Nmap is a network scanning tool that scans hosts and networks for open ports. The Nmap Scripting Engine (NSE) allows Nmap to perform additional checks and functions beyond basic port scanning. NSE uses the Lua programming language to write scripts for tasks like service detection, vulnerability testing, and malware detection. Popular NSE scripts scan for vulnerabilities like SQL injection, fingerprint web servers and applications, perform service/version detection, and more. The NSE community develops and shares new scripts on the Nmap site to continually improve Nmap's scanning abilities.
Writing and Publishing Puppet Modules - PuppetConf 2014Puppet
The document discusses best practices for writing and publishing Puppet modules. It covers module structure, writing manifests and templates, testing modules, publishing to the Forge, and maintaining modules over time. The overall goal is to create reusable modules that are portable, configurable and well-tested.
Similar to Scaling antispam solutions with Puppet (20)
SpamAssassin 4.0 is going to be released soon.
In this version there will be a lot of new features useful to catch new types of spam messages and a lot of improvements.
ACME is a protocol for automating certificate lifecycle management communications between Certificate Authorities (CAs) and a company’s web servers.
The most known implementation is the one made by Let’s Encrypt non-profit CA.
There are many other implementation and one of the most attractive and easy to use is Apache httpd mod_md.
During the talk I will explain why ACME protocol is important to secure web sites and how mod_md could ease the transition to a more secure www.
In this talk I will describe what's new in SpamAssassin 3.4.2, the ideas for further developments and some tips to improve the efficiency of your antispam solution.
seccomp is a computer security facility in the Linux kernel, pledge is a similar security facility in the OpenBSD kernel. In this presentation Giovanni Bechis will review the development story and progress of both kernel interfaces and will analyze the main differences. There will be some examples of implementations of security patches made for some important open source projects.
pf(4) is the OpenBSD packet filter that provides stateful packet filtering and network address translation (NAT). It is used in OpenBSD, FreeBSD, NetBSD, DragonflyBSD, and other systems. Some key features of pf include its flexible rule syntax, atomic ruleset updates, integrated traffic shaping, and ability to divert packets to userspace processes like spamd for inspection. It provides logging in tcpdump format and can integrate with CARP and other services. The pf code was developed for OpenBSD after the previous IPFilter code was removed due to licensing issues.
This document summarizes an overview of the ELK stack presented at LinuxCon Europe 2016. It discusses the components of ELK including Beats, Logstash, Elasticsearch, and Kibana. It provides examples of using these components to collect, parse, store, search, and visualize log data. Specific topics covered include collecting log files using Filebeat and Logstash, parsing logs with Logstash filters, visualizing data in Kibana, programming Elasticsearch with REST APIs and client libraries, and alerting using the open source ESWatcher tool.
OpenSSH is a free SSH implementation used for secure communications and file transfers. It provides strong cryptography, authentication methods, and allows for features like port forwarding, X11 forwarding, and file transfers. OpenSSH uses a modular structure which separates the transport, authentication, and connection protocols. This makes it more secure than the original SSH version 1, especially against man-in-the-middle attacks. The OpenSSH codebase focuses on security through techniques like input validation, sandboxing, and process separation between the monitoring and slave processes.
OpenSMTPD is a FREE implementation of the server-side SMTP protocol as defined by RFC 5321, with some additional standard extensions. It allows ordinary machines to exchange e-mails with other systems speaking the SMTP protocol.
Started out of dissatisfaction with other implementations, OpenSMTPD nowadays is a fairly complete SMTP implementation.
This document discusses the Heartbleed bug in OpenSSL and the creation of LibreSSL as a more secure alternative. It notes that 17% of HTTPS servers were vulnerable to Heartbleed, which allowed attackers to steal passwords, credit cards, and other private data from server memory. LibreSSL was created to have fewer lines of code, modern coding practices, and fewer portability workarounds than OpenSSL to address bugs like Heartbleed. The document emphasizes fixing bugs quickly and not reinventing standard library functions.
LibreSSL is a version of the TLS/crypto stack forked from OpenSSL in 2014, with goals of modernizing the codebase, improving security, and applying best practice development processes.
Primary development occurs inside the OpenBSD source tree with the usual care the project is known for. On a regular basis the code is re-packaged for portable use by other operating systems.
SOGo: sostituire Microsoft Exchange con software Open SourceGiovanni Bechis
Microsoft Exchange è lo standard de-facto in ambito groupware, ora però lo si può sostituire con strumenti open source senza perdere funzionalità e guadagnandoci in efficenza, sicurezze e in possibilità di integrazione con altri software.
Questa è una breve introduzione alle possibilità offerte dalle piattaforme di cloud storage open source come owncloud e seafile; in particolare vengono analizzate le peculiarità del software Seafile.
Npppd is a VPN daemon in OpenBSD that supports PPTP, L2TP and PPPoE. It can authenticate using local files or RADIUS and tunnel IP packets. Npppd first appeared in OpenBSD 5.3 and its configuration file format has changed over time. Example configurations show how to set up basic and advanced npppd VPN tunnels using PPTP or L2TP with IPsec, and how to monitor active sessions. Future work may include better packet filtering and ARP cache integration.
Ssh è lo standard de-facto nelle comunicazioni sicure, questa è una breve ma esaustiva introduzione alle caratteristiche del programma e al suo utilizzo.
Il protocollo ipv6 è la nuova versione del protocollo ipv4, la base di internet così come lo conosciamo ora.
Sul protocollo ipv6 si baserà l'internet del futuro.
Relayd is a daemon to relay and dynamically redirect incoming connections to a target host.
Its main purposes are to run as a load-balancer, application layer gateway, or transparent proxy.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
How Can Hiring A Mobile App Development Company Help Your Business Grow?ToXSL Technologies
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
When it is all about ERP solutions, companies typically meet their needs with common ERP solutions like SAP, Oracle, and Microsoft Dynamics. These big players have demonstrated that ERP systems can be either simple or highly comprehensive. This remains true today, but there are new factors to consider, including a promising new contender in the market that’s Odoo. This blog compares Odoo ERP with traditional ERP systems and explains why many companies now see Odoo ERP as the best choice.
What are ERP Systems?
An ERP, or Enterprise Resource Planning, system provides your company with valuable information to help you make better decisions and boost your ROI. You should choose an ERP system based on your company’s specific needs. For instance, if you run a manufacturing or retail business, you will need an ERP system that efficiently manages inventory. A consulting firm, on the other hand, would benefit from an ERP system that enhances daily operations. Similarly, eCommerce stores would select an ERP system tailored to their needs.
Because different businesses have different requirements, ERP system functionalities can vary. Among the various ERP systems available, Odoo ERP is considered one of the best in the ERp market with more than 12 million global users today.
Odoo is an open-source ERP system initially designed for small to medium-sized businesses but now suitable for a wide range of companies. Odoo offers a scalable and configurable point-of-sale management solution and allows you to create customised modules for specific industries. Odoo is gaining more popularity because it is built in a way that allows easy customisation, has a user-friendly interface, and is affordable. Here, you will cover the main differences and get to know why Odoo is gaining attention despite the many other ERP systems available in the market.
Using Query Store in Azure PostgreSQL to Understand Query PerformanceGrant Fritchey
Microsoft has added an excellent new extension in PostgreSQL on their Azure Platform. This session, presented at Posette 2024, covers what Query Store is and the types of information you can get out of it.
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...XfilesPro
Wondering how X-Sign gained popularity in a quick time span? This eSign functionality of XfilesPro DocuPrime has many advancements to offer for Salesforce users. Explore them now!
Unveiling the Advantages of Agile Software Development.pdfbrainerhub1
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
UI5con 2024 - Bring Your Own Design SystemPeter Muessig
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
Microservice Teams - How the cloud changes the way we workSven Peters
A lot of technical challenges and complexity come with building a cloud-native and distributed architecture. The way we develop backend software has fundamentally changed in the last ten years. Managing a microservices architecture demands a lot of us to ensure observability and operational resiliency. But did you also change the way you run your development teams?
Sven will talk about Atlassian’s journey from a monolith to a multi-tenanted architecture and how it affected the way the engineering teams work. You will learn how we shifted to service ownership, moved to more autonomous teams (and its challenges), and established platform and enablement teams.
Artificia Intellicence and XPath Extension FunctionsOctavian Nadolu
The purpose of this presentation is to provide an overview of how you can use AI from XSLT, XQuery, Schematron, or XML Refactoring operations, the potential benefits of using AI, and some of the challenges we face.
WWDC 2024 Keynote Review: For CocoaCoders AustinPatrick Weigel
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
1. Scaling antispam mail servers with Puppet
Giovanni Bechis
<g.bechis@snb.it>
@g_bechis
PuppetCamp 2020, Netherlands
2. About Me
sysadmin and developer @SNB
OpenBSD hacker for ∼ 12 years
Apache SpamAssassin committer
Apache httpd committer
random patches in random open source software (amavisd-new,
courier-imap, cyrus-sasl, memcached, ...)
5. Puppet Facts
Some facts are interesting to scale in a right way all involved softwares
”os.name” to have different configuration for different operating systems
”processors.count” and ”memory.system.total bytes” to correctly
configure Clamav and Apache SpamAssassin
6. Puppet Facts and Erb templates
# Clamav template
# Maximum number of threads running at the same time.
# Default: 10
#MaxThreads 20
<% if @processors[’count’] != 1 -%>
MaxThreads 20
<% end -%>
# Path to a local socket file the daemon will listen on.
# Default: disabled (must be specified by a user)
#LocalSocket /tmp/clamd.socket
<% if @os[’name’] == ’OpenBSD’ -%>
LocalSocket /tmp/clamd
<% end -%>
7. Puppet Facts and Erb templates
# Mail::SpamAssassin::Plugin::ResourceLimits
<% if @memory[’system’][’total_bytes’] < 4142006272 -%>
resource_limit_mem 536870912
<% end -%>
8. Puppet Facts and Erb templates
redis::params { ’redis’:
bind => [’127.0.0.1’, ’172.16.99.134’],
masterauth => ’XXX’,
slaveof => ’172.16.99.133 6379’,
}
bind <%= @bind.join(’ ’) %>
<% if @masterauth -%>masterauth <%= @masterauth %><% end -%>
<% if @slaveof -%>slaveof <%= @slaveof %><% end -%>
9. Config files deploy
Config files deploy
Puppet server does a checkout of our
git repo and pulls updated
configuration files.
Most of the configuration files are the
same for all servers, lot of modules we
are using just copy the correct files in
the correct location.
12. Mimedefang configuration and deploy
Mimedefang deploy
Mimedefang has both code and
configuration files that need to be
deployed.
Mimedefang has several dependencies
like SpamAssassin and Clamav, the
same dependency tree is respected on
Puppet code.