The presentation describes CI environment for our product - Maxscale - database proxy server. To test such product we need a setup that consists of tens of machines: locally hosted virtual machines as well as machines from different clouds. All our Jenkins jobs are implemented in the form of Jenkins Job Builder code. Presentations also tells about our tool to manage virtual machines (wrapper over Vagrant)- MDBCI.
(Click 2nd slide for video) Deploy PHP apps faster in 2017. This talk focuses on how PHP developers can use simple Ansible scripts to rapidly configure new dev and production servers from scratch, and deploy their apps. No more "snowflake servers"!
This is a general introduction to DevOps essentials and Ansible, with a few extras for PHP developers, including some best practice tips and overview of two major Ansible-based PHP projects, Drupal-VM and Trellis (modern WordPress setup).
Ansible is a Configuration Management System that is very simple to use, because of its straightforward and robust model for managing automation and it’s low barrier to entry for ease of use in both development and production.
During OpenStack development, Ansible can be used in conjunction with Vagrant and Devstack to manage complex, multi-node development environments with relative ease.
In this presentation, Juergen Brendel and David Lapsley review Ansible and provide some sample playbooks to get developers up and running quickly. They also describes how to use Ansible, Vagrant, Devstack, and OpenStack to accelerate OpenStack development cycles.
Chef Provisioning a Chef Server Cluster - ChefConf 2015Chef
This talk is an exploration of how to build Chef Server 12 clusters with Chef Provisioning, including premium features like Chef Reporting and Chef Analytics. I will cover several new capabilities and tools in the Chef ecosystem that makes it easier than ever before to build and manage scalable Chef Server clusters.
https://youtu.be/HUQA1Ikm5Iw
AWS re:Invent 2016: Amazon ECR Deep Dive on Image Optimization (CON401)Amazon Web Services
“Are you struggling with bulky images or slow push and pull times? In this session we will walk through the anatomy of a Docker image and provide techniques you can use to optimize images for faster pushes and pulls and reduce your overall storage footprint. We will discuss Docker image building (build containers versus runtime containers to remove unnecessary software), Docker image composition (minimizing the number of layers), the Docker Remote API (optimizing how images are pushed and pulled), and CI/CD Integration (automate building, versioning, and deploying images to production). We’ll also examine the tools that ECR provides to make Docker image management easier so that you can focus on building your application. Finally, we'll hear from Pinterest about how they use ECR and Docker, some valuable experiences gained along the way, and best practices for using ECR with Apache Mesos.”
(Click 2nd slide for video) Deploy PHP apps faster in 2017. This talk focuses on how PHP developers can use simple Ansible scripts to rapidly configure new dev and production servers from scratch, and deploy their apps. No more "snowflake servers"!
This is a general introduction to DevOps essentials and Ansible, with a few extras for PHP developers, including some best practice tips and overview of two major Ansible-based PHP projects, Drupal-VM and Trellis (modern WordPress setup).
Ansible is a Configuration Management System that is very simple to use, because of its straightforward and robust model for managing automation and it’s low barrier to entry for ease of use in both development and production.
During OpenStack development, Ansible can be used in conjunction with Vagrant and Devstack to manage complex, multi-node development environments with relative ease.
In this presentation, Juergen Brendel and David Lapsley review Ansible and provide some sample playbooks to get developers up and running quickly. They also describes how to use Ansible, Vagrant, Devstack, and OpenStack to accelerate OpenStack development cycles.
Chef Provisioning a Chef Server Cluster - ChefConf 2015Chef
This talk is an exploration of how to build Chef Server 12 clusters with Chef Provisioning, including premium features like Chef Reporting and Chef Analytics. I will cover several new capabilities and tools in the Chef ecosystem that makes it easier than ever before to build and manage scalable Chef Server clusters.
https://youtu.be/HUQA1Ikm5Iw
AWS re:Invent 2016: Amazon ECR Deep Dive on Image Optimization (CON401)Amazon Web Services
“Are you struggling with bulky images or slow push and pull times? In this session we will walk through the anatomy of a Docker image and provide techniques you can use to optimize images for faster pushes and pulls and reduce your overall storage footprint. We will discuss Docker image building (build containers versus runtime containers to remove unnecessary software), Docker image composition (minimizing the number of layers), the Docker Remote API (optimizing how images are pushed and pulled), and CI/CD Integration (automate building, versioning, and deploying images to production). We’ll also examine the tools that ECR provides to make Docker image management easier so that you can focus on building your application. Finally, we'll hear from Pinterest about how they use ECR and Docker, some valuable experiences gained along the way, and best practices for using ECR with Apache Mesos.”
Covers a broad overview of how to use AWS for building a scalable web app. Covers some of the AWS services in depth, and also gives recommendations on many services.
This presentation explains how to deploy and use the Integrated Caching feature on Netscaler. I gave this presentation to Citrix staff, customers and partners in worldwide in 2011. The presentation covers best practices and gotchas :) Integrated Caching is an excellent feature that can greatly improve the performance of your website.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
DevOpsDays Austin 2016 talk. Compliance and security are the next steps after Infrastructure as Code and Test-Driven Infrastructure in expanding your DevOps workflow. Chef's open-source InSpec and audit cookbooks provide an accessible pattern for building compliance into your continuous delivery pipelines.
Best Practices for Running Kafka on Docker ContainersBlueData, Inc.
Docker containers provide an ideal foundation for running Kafka-as-a-Service on-premises or in the public cloud. However, using Docker containers in production environments for Big Data workloads using Kafka poses some challenges – including container management, scheduling, network configuration and security, and performance.
In this session at Kafka Summit in August 2017, Nanda Vijyaydev of BlueData shared lessons learned from implementing Kafka-as-a-Service with Docker containers.
https://kafka-summit.org/sessions/kafka-service-docker-containers
IT Infrastructure Through The Public Network Challenges And SolutionsMartin Jackson
Identifying the challenges that companies face when they wish to adopt Infrastructure as a Service like those from Amazon and Rackspace and possible solutions to those problems. This presentation seeks to provide insight and possible solutions, covering the areas of security, availability, cloud standards, interoperability, vendor lock in and performance management.
This presentation starts with an introduction to the rationale behind automated deployments in Continuous Delivery and DevOps. Then, I compare agent-based architectures, such as Chef and Puppet with the agentless architecture of the server orchestration engine Ansible. The presentation concludes with an automated deployment of Dynatrace into a simulated production environment.
Achieving Infrastructure Portability with ChefMatt Ray
Deploying to the cloud has made it easy to run large numbers of servers, but users may become dissatisfied with their particular cloud platform for reasons such as price, support and performance. There are a number of vendor lock-ins to avoid, this talk discusses how to do so with the open source configuration management and infrastructure automation platform Chef. Chef makes it easy to deploy to nearly every public and private cloud platform as well as virtualized and physical servers. Chef may also be used to deploy cloud infrastructures such as OpenStack, Eucalyptus or CloudStack. By abstracting away the platform, infrastructure becomes portable and you are free to deploy wherever necessary.
Managing big test environment and running tests with Jenkins, Jenkins Job bui...Timofey Turenko
A short presentation about our experience of using Jenkins and Jenkins Job Builder with Vagrant as a backend tool to manage complex environment (tens of virtual machines for every test run) for database proxy server testing.
Jenkins Job Builder is a tool to represent Jenkins jobs in the YAML format. This presentation is about using it to manage a large amount of build and test jobs for executing tests which require complex environment
Covers a broad overview of how to use AWS for building a scalable web app. Covers some of the AWS services in depth, and also gives recommendations on many services.
This presentation explains how to deploy and use the Integrated Caching feature on Netscaler. I gave this presentation to Citrix staff, customers and partners in worldwide in 2011. The presentation covers best practices and gotchas :) Integrated Caching is an excellent feature that can greatly improve the performance of your website.
Docker at Shopify: From This-Looks-Fun to Production by Simon Eskildsen (Shop...Docker, Inc.
Since July 2014 Shopify's been serving thousands of requests per second of production web traffic from Docker containers. This was an 8 month effort, with multiple pivots of direction from the team—and we're only getting started. This talk covers the lessons learned through the trial and error of an in-flight architecture redesign, spanning hundreds of hosts, as well as the technical vision of the future of our platform.
DevOpsDays Austin 2016 talk. Compliance and security are the next steps after Infrastructure as Code and Test-Driven Infrastructure in expanding your DevOps workflow. Chef's open-source InSpec and audit cookbooks provide an accessible pattern for building compliance into your continuous delivery pipelines.
Best Practices for Running Kafka on Docker ContainersBlueData, Inc.
Docker containers provide an ideal foundation for running Kafka-as-a-Service on-premises or in the public cloud. However, using Docker containers in production environments for Big Data workloads using Kafka poses some challenges – including container management, scheduling, network configuration and security, and performance.
In this session at Kafka Summit in August 2017, Nanda Vijyaydev of BlueData shared lessons learned from implementing Kafka-as-a-Service with Docker containers.
https://kafka-summit.org/sessions/kafka-service-docker-containers
IT Infrastructure Through The Public Network Challenges And SolutionsMartin Jackson
Identifying the challenges that companies face when they wish to adopt Infrastructure as a Service like those from Amazon and Rackspace and possible solutions to those problems. This presentation seeks to provide insight and possible solutions, covering the areas of security, availability, cloud standards, interoperability, vendor lock in and performance management.
This presentation starts with an introduction to the rationale behind automated deployments in Continuous Delivery and DevOps. Then, I compare agent-based architectures, such as Chef and Puppet with the agentless architecture of the server orchestration engine Ansible. The presentation concludes with an automated deployment of Dynatrace into a simulated production environment.
Achieving Infrastructure Portability with ChefMatt Ray
Deploying to the cloud has made it easy to run large numbers of servers, but users may become dissatisfied with their particular cloud platform for reasons such as price, support and performance. There are a number of vendor lock-ins to avoid, this talk discusses how to do so with the open source configuration management and infrastructure automation platform Chef. Chef makes it easy to deploy to nearly every public and private cloud platform as well as virtualized and physical servers. Chef may also be used to deploy cloud infrastructures such as OpenStack, Eucalyptus or CloudStack. By abstracting away the platform, infrastructure becomes portable and you are free to deploy wherever necessary.
Managing big test environment and running tests with Jenkins, Jenkins Job bui...Timofey Turenko
A short presentation about our experience of using Jenkins and Jenkins Job Builder with Vagrant as a backend tool to manage complex environment (tens of virtual machines for every test run) for database proxy server testing.
Jenkins Job Builder is a tool to represent Jenkins jobs in the YAML format. This presentation is about using it to manage a large amount of build and test jobs for executing tests which require complex environment
(ARC402) Deployment Automation: From Developers' Keyboards to End Users' Scre...Amazon Web Services
Some of the best businesses today are deploying their code dozens of times a day. How? By making heavy use of automation, smart tools, and repeatable patterns to get process out of the way and keep the workflow moving. Come to this session to learn how you can do this too, using services such as AWS OpsWorks, AWS CloudFormation, Amazon Simple Workflow Service, and other tools. We'll discuss a number of different deployment patterns, and what aspects you need to focus on when working toward deployment automation yourself.
Presentation at March 2019 Dutch Postgres User Group Meetup on lessons learnt while migrating from Oracle to Postgres, demo'ed via vagrant test environments and using generic pgbench datasets.
AWS Lambda with Serverless Framework and JavaManish Pandit
Serverless is a node.js based framework that makes creating, deploying, and managing serverless functions a breeze. We will use AWS Lambda as our FaaS (Function-as-a-Service) provider, although Serverless supports IBM OpenWhisk and Microsoft Azure as well.
In this session, we will talk about Serverless Applications, and Create and deploy a java-maven based AWS Lambda API. We will also explore the command line interface to manage lambda, which is provided out of the box by serverless framework.
Getting started with MariaDB? Whether it is on your laptop or server, containers are great ephemeral vessels for your applications. But what about the data that drives your business? It must survive containers coming and going, maintain its availability and reliability, and grow when you need it.
Continuous Delivery with Maven, Puppet and Tomcat - ApacheCon NA 2013Carlos Sanchez
Continuous Integration, with Apache Continuum or Jenkins, can be extended to fully manage deployments and production environments, running in Tomcat for instance, in a full Continuous Delivery cycle using infrastructure-as-code tools like Puppet, allowing to manage multiple servers and their configurations.
Puppet is an infrastructure-as-code tool that allows easy and automated provisioning of servers, defining the packages, configuration, services,... in code. Enabling DevOps culture, tools like Puppet help drive Agile development all the way to operations and systems administration, and along with continuous integration tools like Apache Continuum or Jenkins, it is a key piece to accomplish repeatability and continuous delivery, automating the operations side during development, QA or production, and enabling testing of systems configuration.
Traditionally a field for system administrators, Puppet can empower developers, allowing both to collaborate coding the infrastructure needed for their developments, whether it runs in hardware, virtual machines or cloud. Developers and sysadmins can define what JDK version must be installed, application server, version, configuration files, war and jar files,... and easily make changes that propagate across all nodes.
Using Vagrant, a command line automation layer for VirtualBox, they can also spin off virtual machines in their local box, easily from scratch with the same configuration as production servers, do development or testing and tear them down afterwards.
We will show how to install and manage Puppet nodes with JDK, multiple Tomcat instances with installed web applications, database, configuration files and all the supporting services. Including getting up and running with Vagrant and VirtualBox for quickstart and Puppet experiments, as well as setting up automated testing of the Puppet code.
DevOps, Continuous Integration and Deployment on AWS: Putting Money Back into...Amazon Web Services
Organizations around the globe are leveraging the cloud to accomplish world-changing missions. This session will address how AWS can help organizations put more money toward their mission and scale outreach and operations to achieve more with less. Hear some of AWS’s most advanced customers on how their organizations handle DevOps, continuous integration and deployment. Learn how these practices allow them to rapidly develop, iterate, test and deploy highly-scalable web applications and core operational systems on AWS. The discussion will focus on best practices, lessons learned, and the specific technologies and services they use.
In the presentation, we will learn on how to set up InnoDB cluster using the official MySQL Docker containers and run them with docker-compose. This presentation covers a demo, including how to connect to the cluster through MySQL Router using a simple application.
Code testing and Continuous Integration are just the first step in a source code to production process. Combined with infrastructure-as-code tools such as Puppet the whole process can be automated, and tested!
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Why React Native as a Strategic Advantage for Startup Innovation.pdfayushiqss
Do you know that React Native is being increasingly adopted by startups as well as big companies in the mobile app development industry? Big names like Facebook, Instagram, and Pinterest have already integrated this robust open-source framework.
In fact, according to a report by Statista, the number of React Native developers has been steadily increasing over the years, reaching an estimated 1.9 million by the end of 2024. This means that the demand for this framework in the job market has been growing making it a valuable skill.
But what makes React Native so popular for mobile application development? It offers excellent cross-platform capabilities among other benefits. This way, with React Native, developers can write code once and run it on both iOS and Android devices thus saving time and resources leading to shorter development cycles hence faster time-to-market for your app.
Let’s take the example of a startup, which wanted to release their app on both iOS and Android at once. Through the use of React Native they managed to create an app and bring it into the market within a very short period. This helped them gain an advantage over their competitors because they had access to a large user base who were able to generate revenue quickly for them.
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...informapgpstrackings
Keep tabs on your field staff effortlessly with Informap Technology Centre LLC. Real-time tracking, task assignment, and smart features for efficient management. Request a live demo today!
For more details, visit us : https://informapuae.com/field-staff-tracking/
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload.
Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.
Unleash Unlimited Potential with One-Time Purchase
BoxLang is more than just a language; it's a community. By choosing a Visionary License, you're not just investing in your success, you're actively contributing to the ongoing development and support of BoxLang.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
How Does XfilesPro Ensure Security While Sharing Documents in Salesforce?XfilesPro
Worried about document security while sharing them in Salesforce? Fret no more! Here are the top-notch security standards XfilesPro upholds to ensure strong security for your Salesforce documents while sharing with internal or external people.
To learn more, read the blog: https://www.xfilespro.com/how-does-xfilespro-make-document-sharing-secure-and-seamless-in-salesforce/
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Globus Compute wth IRI Workflows - GlobusWorld 2024
DB proxy server test: run tests on tens of virtual machines with Jenkins, Vagrant and our Vagrant wrapper MDBCI
1. DB proxy server test:
run tests on tens of
virtual machines with
Jenkins, Vagrant and
our Vagrant wrapper
MDBCI
Open Source
and Linux Lab
OSLL
2. • Test task
• How to prepare test environment
• Upper layer: CI tool
• Our additions
3. Our setup: product – DB proxy server Maxscale
• Product to build and test: Maxscale – database proxy
• lots of plugins (routers, filters, protocols, etc, etc, etc) = many configurations
• Integration tests, performance tests, “scientific experiments”
• Test environment:
– at least 9 VM (1 Maxscale server, 4 Master/Slave servers, 4-servers Galera cluster)
– libvirt/qemu
– some tests are heavy – powerful AWS virtual machines, but their life is short
– need to configure backend for every test run
– need to carefully clean up
4. Our setup: why is it so complex to run test?
• dependency on VM provider: it can easily FAIL
• 12 Linux distributions, 4 major MariaDB version, 3 major MySQL version, a lot of
Maxscale plugins – any combination should be testable
INFO: ==> galera_003: Starting domain.
ERROR: There was an error talking to Libvirt.
The error message is shown
ERROR: below:
ERROR:
ERROR: Call to virDomainCreateWithFlags
failed: internal error: process exited while
connecting to monitor: warning: host doesn't
support requested feature: CPUID.01H:EDX.ds
[bit 21]
ERROR: warning: host doesn't support
requested feature: CPUID.01H:EDX.acpi [bit 22]
ERROR: warning: host doesn't support
requested feature: CPUID.01H:EDX.ht [bit 28]
...
5. What does it mean for test management system?
• Several VMs are needed for every test round
• Lots of parameters for every test job
• Cascades of jobs: e.g. prepare backend, install Maxscale, run test, cleanup
• Error processing (e.g. backend VMs preparation failure) – how to do cleanup in this
case?
• Matrix jobs: test a lot of combinations of distros, backend MariaDB version, backend
parameters
• very dynamic setup
6. VM management: Vagrant
• libvirt, AWS, Docker, LXC, DigitalOcean, OpenStack, ...
• write Vagrantfile and 'vagrant up' ...
• but
– it can fail: restart logic is needed
– Vagrantfile is not the simplest thing to write
• Provisioning:
– lots of tools, let's just take simplest – Puppet
– many version of MariaDB/MySQL – need a tool to configure paths to binary repos
9. MDBCI vs Vagrantfile
• separate
– high-level test setup description
– VM box description
– Repositories description
– Chef recipes
– Credentials
• hide
– low-level provider-dependent stuff
### Import AWS Provider access config ###
require 'yaml'
aws_config = YAML.load_file('/home/vagrant/mdbci/aws-config.yml')['aws']
## of import AWS Provider access config
### Vagrant configuration block ###
####################################
Vagrant.configure(2) do |config|
config.omnibus.chef_version = '12.9.38'
### AWS Provider config block ###
###########################################################
config.vm.box = "dummy"
config.vm.provider :aws do |aws, override|
aws.keypair_name = "max-tst-02.mariadb.com_daily_test-382-2.1_1505122513"
aws.region = aws_config["region"]
aws.security_groups = aws_config["security_groups"]
aws.user_data = aws_config["user_data"]
override.ssh.private_key_path = "/home/vagrant/vms/daily_test-382-2.1/maxscale.pem"
override.nfs.functional = false
aws.aws_profile = "mdbci"
end ## of AWS Provider config block
# --> Begin definition for machine: node_000
config.vm.define :node_000 do |node_000|
node_000.vm.provider :aws do |aws,override|
aws.ami = "ami-1c2e8b6b"
aws.tags = { "hostname" => "max-tst-02.mariadb.com", "username" => "vagrant", "full_config_path
aws.instance_type = "t1.micro"
override.ssh.username = "ec2-user"
end
node_000.vm.synced_folder "~/build-scripts/test-setup-scripts/cnf", "/home/vagrant/cnf_templates", type: "r
node_000.vm.provision 'shell', inline: 'curl -L https://omnitruck.chef.io/install.sh | sudo bash -s -- -v 12.9.38'
node_000.vm.provision "chef_solo" do |chef|
chef.cookbooks_path = "/home/vagrant/mdbci/recipes/cookbooks/"
chef.roles_path = "."
chef.add_role "node_000"
chef.synced_folder_type = "rsync"
end #<-- end of chef binding
end # <-- End AWS definition for machine: node_000
10. Upper layer: BuildBot, Jenkins
• Buildbot
– has been in use from very beginning (since middle of 90th
)
– do we want different tools for every our product?
– there are many useful scripts for BuildBot
• Jenkins
– popular tool
– easy to start
– let's try for Maxscale
11. Jenkins: Job Builder
• A tool from OpenStack developers https://docs.openstack.org/infra/jenkins-job-builder/
• YAML
• Pipelines did not exist when we started
• our jobs: https://github.com/mariadb-corporation/maxscale-jenkins-jobs
12. What can we get from JJB?
• All our jobs (>60) are represented as a code (branches, forks, review, pull requests, …)
• Parameters values are in one place for all jobs (easy maintenance of parameters defaults
and lists)
• Automatic validation of jobs takes place at every push to jobs repo
• very easy jobs deployment
• at the beginning: app. 20 jobs, we spent 1 week to convert to JJB code (1.5 persons)
13. Our “run_test” job: thanks to Parametrized Trigger plugin
Build
Setup backend
Install Maxscale
Run tests
Extract logs, coredumps
cleanup VMs (or not)
Parse logs
Send report email
Put result to DB
Pipelines? may be...
15. Jenkins: results visualization problem
• Jenkins: all “run_test” jobs are the
same in the list
• any search engine in Jenkins?
• our solution:
– report email
– test script puts results into …
MariaDB, developers query DB
by writing SQL manually
16. Our additions
• simple wrapper to load jobs https://github.com/OSLL/jjg.git
– adds timestamps and git commit ID to job description
– easier way to manage Jenkins creds
– manage Views ('description' contains reference to 'View' to put the job)
• Matrix job killer groovy script
https://github.com/mariadb-corporation/maxscale-jenkins-jobs/tree/master/maxscale_jobs/inc
• Job Visualizer (not ready yet, but working)
https://github.com/OSLL/jenkins_job_builder_visualization
17. Job Visualizer: Call Graph
●
Do you call all the jobs
correctly?
●
Do you have cyclic
dependencies?
19. Jenkins job description pre-processing
description: 'View: test. This job perform integration testing of maxscale'
description: 'View: build. This job builds Maxscale'
20. Plans
• full automatic deployment
• move Jenkins to VM or container
• try JobDSL (via JJB?)
• BuildBot is still in the view
• results visualization: plugin? 3rd
party tool? something own?
21. Results
• easy to create any number of VMs (described as easy JSON)
• MDBCI does all dirty job
• Jenkins is good for scheduling and logs storing
• setup works!
• results visualization is bad (Buildbot?)
• Pipelines or JobDSL – worth to try
22. MariaDB Maxscale
Continuous Integration
Open Source
and Linux Lab
OSLL
in cooperation with
Open Source
and Linux Lab
Open Innovations Association FRUCT
and
http://fruct.org/
https://osllblog.wordpress.com/
https://mariadb.com/products/mariadb-maxscale
timofey.turenko@mariadb.com
Mark Zaslavskiy <mark.zaslavskiy@gmail.com>
Sergey Balandin <Sergey.Balandin@fruct.org>