Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?Carlos Sanchez
The Jenkins platform can be dynamically scaled by using several Docker cluster and orchestration platforms, using containers to run slaves and jobs and also isolating job execution. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? How do they compare? All of them can be used to dynamically run jobs inside containers. This talk will cover these main container clusters, outlining the pros and cons of each, the current state of the art of the technologies and Jenkins support.
All Things Containers - Docker, Kubernetes, Helm, Istio, GitOps and moreAll Things Open
Presented by: Brent Laster, SAS
Presented at All Things Open 2020
Abstract: In this workshop, students will get a quick overview of what containers are and why they form the basis for many of the key technologies that we use today in cloud environments.
We’ll explore what makes up a container and how they are managed and leveraged in key industry tooling including Docker, Kubernetes, Helm, and Istio. You’ll also learn the basics of these technologies, what they are used for, and see some simple examples of how to use them.
This workshop will include hands-on labs where you will get experience:
Building container images, running them as containers, and tagging and pushing them into a Docker repository.
Creating deployments, services, and pods for containers and instantiating and running those in Kubernetes.
Working with Helm to leverage templates for Kubernetes objects and managing releases in Kubernetes.
Working with Istio to do traffic shaping between multiple versions of your app, fault and delay injection for testing and validation in Kubernetes.
We’ll also briefly cover GitOps – the recommended Git-based way to manage infrastructure like your Kubernetes cluster.
Docker Birthday #3 - Intro to Docker SlidesDocker, Inc.
High level overview of Docker + Birthday #3 overview (app and challenge portion)!
Learn more about Docker Birthday #3 celebrations here: https://www.docker.com/community/docker-birthday-3
Scaling Jenkins with Docker: Swarm, Kubernetes or Mesos?Carlos Sanchez
The Jenkins platform can be dynamically scaled by using several Docker cluster and orchestration platforms, using containers to run slaves and jobs and also isolating job execution. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? How do they compare? All of them can be used to dynamically run jobs inside containers. This talk will cover these main container clusters, outlining the pros and cons of each, the current state of the art of the technologies and Jenkins support.
All Things Containers - Docker, Kubernetes, Helm, Istio, GitOps and moreAll Things Open
Presented by: Brent Laster, SAS
Presented at All Things Open 2020
Abstract: In this workshop, students will get a quick overview of what containers are and why they form the basis for many of the key technologies that we use today in cloud environments.
We’ll explore what makes up a container and how they are managed and leveraged in key industry tooling including Docker, Kubernetes, Helm, and Istio. You’ll also learn the basics of these technologies, what they are used for, and see some simple examples of how to use them.
This workshop will include hands-on labs where you will get experience:
Building container images, running them as containers, and tagging and pushing them into a Docker repository.
Creating deployments, services, and pods for containers and instantiating and running those in Kubernetes.
Working with Helm to leverage templates for Kubernetes objects and managing releases in Kubernetes.
Working with Istio to do traffic shaping between multiple versions of your app, fault and delay injection for testing and validation in Kubernetes.
We’ll also briefly cover GitOps – the recommended Git-based way to manage infrastructure like your Kubernetes cluster.
Docker Birthday #3 - Intro to Docker SlidesDocker, Inc.
High level overview of Docker + Birthday #3 overview (app and challenge portion)!
Learn more about Docker Birthday #3 celebrations here: https://www.docker.com/community/docker-birthday-3
Introduction to dockers and kubernetes. Learn how this helps you to build scalable and portable applications with cloud. It introduces the basic concepts of dockers, its differences with virtualization, then explain the need for orchestration and do some hands-on experiments with dockers
Compare Docker deployment options in the public cloudSreenivas Makam
Compare Docker public cloud deployment options using Docker machine, Docker Cloud, Docker datacenter, Docker for AWS, Azure and Google cloud, AWS ECS, Google Container engine, Azure Container service.
How we dockerized a startup? #meetup #docker Jonathan Martin
Docker in production, for real!
The Yuzu startup, helped by Vixns, chose to have a docker infrastructure with Mesos/Marathon/Consul. From the development environment to our prod monitoring, we share our mistakes, successes, workflows and tools.
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Carlos Sanchez
Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but running in just one machine is not enough and quickly needs to scale to a clustered setup. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? how do they compare? All of them can be used to dynamically run a cluster of containers.
Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but running in just one machine is not enough and quickly needs to scale to a clustered setup. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? how do they compare? All of them can be used to dynamically run a cluster of containers.
The Jenkins platform is an example of dynamically scaling by using several Docker cluster and orchestration platforms, using containers to run build agents and jobs, and also isolate job execution.
This talk will cover these main container clusters, outlining the pros and cons, the current state of the art of the technologies and Jenkins support.
The presentation will allow a better understanding of using Docker in the main Docker cluster/orchestration platforms out there (Docker Swarm, Apache Mesos, Kubernetes), sharing my experience and helping people decide which one to use, going through Jenkins examples and current support.
Docker provides a new, powerful way of prototyping, testing and deploying applications on cloud-based infrastructures. In this seminar we delve into the concept of Docker containers without requiring any previous knowledge from the audience.
Orchestrating Docker Containers with Google Kubernetes on OpenStackTrevor Roberts Jr.
Kubernetes, Docker, CoreOS, and OpenStack for container workload management.
No audio, but there are annotations to follow along with the workload.
A video accompanies a Microservices Meetup talk that I presented on February 18, 2015 at https://www.youtube.com/watch?v=RfyIYhOzyPY
Acknowledgements to Kelsey Hightower for the workflow that I used, and Google for the example application shown.
Orchestrating Docker containers at scaleMaciej Lasyk
Many of us already poked around Docker. Let's recap what we know and then think what do we know about scaling apps & whole environments which are Docker - based? Should we PaaS, IaaS or go with bare? Which tools to use on a given scale?
Docker for the new Era: Introducing Docker,its components and toolsRamit Surana
Containers have been evolved from generations behind today.So what's new with Docker ? What has changed during the current scenario ? Find out more on docker and how to implement it in your environments in the above slide show.
What is Docker and why should you care? A Docker container is like a
lightweight Virtual Machine. It gives you the benefits of a virtual machine,
isolation of your application, without the drawbacks, having to ship an entire
operating system with your application, slow startup time, and difficult
interaction with the host.
In this presentation you will learn why Docker and containerization is the
future of DevOps and how to use it efficiently. You will learn how to build,
run, and link containers, and what volumes are and what they are used for.
You will also learn about some of the many orchestration solutions that exists
for managing a cluster of containers, both locally and in the cloud.
Docker 101 - High level introduction to dockerDr Ganesh Iyer
This deck will help you understand the basics of Docker. It introduces dockers and containers, gives a comparison with virtualization and gives some getting started guides.
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudSamuel Chow
Interactive Workshop on discussing the target architecture and migration plan of migrating a legacy, on-premise IT infrastructure to Google Cloud Platform (GCP).
Presentation about Docker:
2016 Trends:
* Microservices: load balancing and orchestration
* Cloud
* Continuos integration
* Environment-less deployment
What are containers?
Why Docker?
Docker project
Docker. Inc
Docker VS VM
Docker basics
Some statistics about Docker and some Docker use case insights
Docker compose configuration file:
http://www.mediafire.com/download/lfmfzrkgn9wzegm/docker-compose.yml
Présentation link:
https://docs.google.com/presentation/d/1x11EgUqBVLAl70p53rZ-nJoLlL6FoZd2KbvTRxyVp1g/pub?start=false&loop=false&delayms=3000
Introduction to dockers and kubernetes. Learn how this helps you to build scalable and portable applications with cloud. It introduces the basic concepts of dockers, its differences with virtualization, then explain the need for orchestration and do some hands-on experiments with dockers
Compare Docker deployment options in the public cloudSreenivas Makam
Compare Docker public cloud deployment options using Docker machine, Docker Cloud, Docker datacenter, Docker for AWS, Azure and Google cloud, AWS ECS, Google Container engine, Azure Container service.
How we dockerized a startup? #meetup #docker Jonathan Martin
Docker in production, for real!
The Yuzu startup, helped by Vixns, chose to have a docker infrastructure with Mesos/Marathon/Consul. From the development environment to our prod monitoring, we share our mistakes, successes, workflows and tools.
Using Containers for Building and Testing: Docker, Kubernetes and Mesos. FOSD...Carlos Sanchez
Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but running in just one machine is not enough and quickly needs to scale to a clustered setup. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? how do they compare? All of them can be used to dynamically run a cluster of containers.
Building and testing is a great use case for containers, both due to the dynamic and isolation aspects, but running in just one machine is not enough and quickly needs to scale to a clustered setup. But which cluster technology should be used? Docker Swarm? Apache Mesos? Kubernetes? how do they compare? All of them can be used to dynamically run a cluster of containers.
The Jenkins platform is an example of dynamically scaling by using several Docker cluster and orchestration platforms, using containers to run build agents and jobs, and also isolate job execution.
This talk will cover these main container clusters, outlining the pros and cons, the current state of the art of the technologies and Jenkins support.
The presentation will allow a better understanding of using Docker in the main Docker cluster/orchestration platforms out there (Docker Swarm, Apache Mesos, Kubernetes), sharing my experience and helping people decide which one to use, going through Jenkins examples and current support.
Docker provides a new, powerful way of prototyping, testing and deploying applications on cloud-based infrastructures. In this seminar we delve into the concept of Docker containers without requiring any previous knowledge from the audience.
Orchestrating Docker Containers with Google Kubernetes on OpenStackTrevor Roberts Jr.
Kubernetes, Docker, CoreOS, and OpenStack for container workload management.
No audio, but there are annotations to follow along with the workload.
A video accompanies a Microservices Meetup talk that I presented on February 18, 2015 at https://www.youtube.com/watch?v=RfyIYhOzyPY
Acknowledgements to Kelsey Hightower for the workflow that I used, and Google for the example application shown.
Orchestrating Docker containers at scaleMaciej Lasyk
Many of us already poked around Docker. Let's recap what we know and then think what do we know about scaling apps & whole environments which are Docker - based? Should we PaaS, IaaS or go with bare? Which tools to use on a given scale?
Docker for the new Era: Introducing Docker,its components and toolsRamit Surana
Containers have been evolved from generations behind today.So what's new with Docker ? What has changed during the current scenario ? Find out more on docker and how to implement it in your environments in the above slide show.
What is Docker and why should you care? A Docker container is like a
lightweight Virtual Machine. It gives you the benefits of a virtual machine,
isolation of your application, without the drawbacks, having to ship an entire
operating system with your application, slow startup time, and difficult
interaction with the host.
In this presentation you will learn why Docker and containerization is the
future of DevOps and how to use it efficiently. You will learn how to build,
run, and link containers, and what volumes are and what they are used for.
You will also learn about some of the many orchestration solutions that exists
for managing a cluster of containers, both locally and in the cloud.
Docker 101 - High level introduction to dockerDr Ganesh Iyer
This deck will help you understand the basics of Docker. It introduces dockers and containers, gives a comparison with virtualization and gives some getting started guides.
GCPLA Meetup Workshop - Migration from a Legacy Infrastructure to the CloudSamuel Chow
Interactive Workshop on discussing the target architecture and migration plan of migrating a legacy, on-premise IT infrastructure to Google Cloud Platform (GCP).
Presentation about Docker:
2016 Trends:
* Microservices: load balancing and orchestration
* Cloud
* Continuos integration
* Environment-less deployment
What are containers?
Why Docker?
Docker project
Docker. Inc
Docker VS VM
Docker basics
Some statistics about Docker and some Docker use case insights
Docker compose configuration file:
http://www.mediafire.com/download/lfmfzrkgn9wzegm/docker-compose.yml
Présentation link:
https://docs.google.com/presentation/d/1x11EgUqBVLAl70p53rZ-nJoLlL6FoZd2KbvTRxyVp1g/pub?start=false&loop=false&delayms=3000
Market overview of Docker orchestrators. A detailed architecture's comparison of Kubernetes and Docker Swarm, including benefits and issues. Which orchestrator works better for microservice and highly available applications?
ContainerDays NYC 2015: "Container Orchestration Compared: Kubernetes and Doc...DynamicInfraDays
Slides from Darren Shepherd's talk "Container Orchestration Compared: Kubernetes and Docker Compose, Machine & Swarm" at ContainerDays NYC 2015: http://dynamicinfradays.org/events/2015-nyc/programme.html#orchestration
An Introduction to Container Organization with Docker Swarm, Kubernetes, Meso...Neo4j
Interest in Docker has increased significantly since its inception. According to a report compiled by a leading cloud-scale monitoring company, Datadog, two-thirds of the companies that try Docker adopt it, and the adopters have increased their container count by five times over a period of nine months. Neo4j has also embraced Docker by supporting official images and also offering specific images of its own.
While the interest in container technology is growing rapidly, so is the need to deploy containers over a cluster of machines to allow scalability and fault-tolerance. This highlights the need for orchestration which refers to the idea of automating the manual process of deploying, configuring and scaling the containers in an automated manner.
In this talk, we provide a hands-on introduction to the three most popular Docker orchestration tools: Kubernetes, Docker Swarm and Mesos. This talk offers a conceptual understanding of each of these technologies along with an insight into the concepts learned through a series of three demos. The demos will illustrate how to deploy and automatically scale a Neo4j container using each of the three orchestration platforms.
We realize that the scope of the topic in terms of the orchestration tools is too broad. The rationale behind choosing the three specific tools is based on the following two reasons: First is their potential use in our cluster at Cincinnati Children’s Hospital (CCHMC). Secondly, they also fall under the leading orchestration tools.
Docker Networking in Swarm, Mesos and Kubernetes [Docker Meetup Santa Clara |...PLUMgrid
The Docker container ecosystem is growing very fast and networking has taken an interesting direction with different networking models being introduced and it becomes even more interesting when container orchestration engines like Swarm, Mesos, Kubernetes have to implement networking for Docker containers. At this Meetup, we will talk about the networking capabilities for Docker, networking models like CNM (Container Network Model), how they fit into container orchestration frameworks, what's ready for production and what's in the design/discussion phase expected to be available in near future.
An introduction to Docker native clustering: Swarm.
Deployment and configuration, integration with Consul, for a product-like cluster to serve web-application with multiple containers on multiple hosts. #dockerops
DevoxxFR 2015 Talk http://cfp.devoxx.fr/2015/talk/WXY-1157/Scaling_Docker_with_Kubernetes
Kubernetes is an open source project to manage a cluster of Linux containers as a single system, managing and running Docker containers across multiple Docker hosts, offering co-location of containers, service discovery and replication control. It was started by Google and now it is supported by Microsoft, RedHat, IBM and Docker Inc amongst others.
Once you are using Docker containers the next question is how to scale and start containers across multiple Docker hosts, balancing the containers across them. Kubernetes also adds a higher level API to define how containers are logically grouped, allowing to define pools of containers, load balancing and affinity.
Evaluation of High Availability Performance of Kubernetes and Docker Swarm on...Ontico
The Internet of things (IoT) requires a transition of many paradigms that are commonly used today. One of them is the shift from *scaling up*, i.e. from a system design towards bigger machines with higher computational power to *scaling down*, i.e. a system design based on tiny devices with small computational power. Heavy virtual machines
and energy-hungry data center technology are not suited for sensor networks at the edge of the IoT.
With the latest release of Docker (August 2016), there is an orchestration tool available that is energy-efficient and lightweight enough to handle heavy load on small, but many IoT devices. This is because Docker has now been made available on the ARM CPU architecture, which is already used in a large share of small devices available today because ARM is optimized for small scale and energy-efficient computing.
Thus, with the new Docker ported to ARM, the power of container virtualization technology is now available for the IoT, thereby simplifying the development, deployment and operation of applications.
In the mean time, Kubernetes, Google's production-ready orchestration tool, has also been ported to the ARM architecture and therefore is also available to run large sensor networks at little computational power.
In this talk I present the results of a research project that evaluated the high availability performance of Kubernetes and Docker in an IoT-like environment: A multi-node Raspberry Pi cluster.
This talk includes a live demo if requested.
Video: https://youtu.be/C_u4_l84ED8
Karl Isenberg reviews the history of distributed computing, clarifies terminology for layers in the container stack, and does a head to head comparison of several tools in the space, including Kubernetes, Marathon, and Docker Swarm. Learn which features and qualities are critical for container orchestration and how you can apply this knowledge when evaluating platforms.
This webinar will walk you through the steps involved in migrating a multi-container application deployed in Docker Swarm to Kubernetes. It will map the concepts of Swarm to Kubernetes. Attend this webinar to learn how to apply your Docker skills to Kubernetes for running and managing containerized applications in production.
Docker Online Meetup #28: Production-Ready Docker SwarmDocker, Inc.
presented by Alexandre Beslic (@abronan)
Swarm v1.0 is now ready for running your apps in production!
Swarm is the easiest way to run Docker applications at large scale on a cluster. It turns a pool of Docker Engines into a single, virtual Engine. You don’t have to worry about where to put containers, or how they’re going to talk to each other - it just handles all that for you.
We’ve spent the last few months tirelessly hardening and tuning it, and in combination with multi-host networking and the new volume system in Docker Engine 1.9, we can confidently say that it’s ready for running your apps in production. In our tests, we’ve been running Swarm on EC2 with 1,000 nodes and 30,000 containers and it keeps on scheduling containers in less than half a second. Not even breaking a sweat! Keep an eye for a blog post soon with the full details.
Read more: http://blog.docker.com/2015/11/swarm-1-0/
The Nova driver for Docker has been maturing rapidly since its mainline removal in Icehouse. During the Juno cycle, substantial improvements have been made to the driver, and greater parity has been reached with other virtualization drivers. We will explore these improvements and what they mean to deployers. Eric will additionally showcase deployment scenarios for the deployment of OpenStack itself inside and underneath of Docker for powering traditional VM-based computing, storage, and other cloud services. Finally, users should expect a preview of the planned integration with the new OpenStack Containers Service effort to provide automation of advanced containers functionality and Docker-API semantics inside of an OpenStack cloud.
Note that the included Heat templates are NOT usable. See the linked Heat resources for viable templates and examples.
Docker for developers on mac and windowsDocker, Inc.
The whole Docker ecosystem exists today because of every single developer who found ways of using Docker to improve how they build software; whether streamlining production deployments, speeding up continuous integration systems or standing up an application on your laptop to hack on. In this talk we want to take a step back and look at where Docker sits today from the software developers point of view - and then jump ahead and talk about where it might go in the future. In this talk, we’ll discuss:
* Making Docker an everyday part of the developing software on the desktop, with Docker for Windows and Docker for Mac
* Docker Compose, and the future of describing applications as code
* How Docker provides the best tools for developing applications destined to run on any Kubernetes cluster
This session should be of interest to anyone who writes software; from people who want to hack on a few personal projects, to polyglot open source programmers and to professional developers working in tightly controlled environments. Everyone deserves a better developer experience.
JDO 2019: Tips and Tricks from Docker Captain - Łukasz LachPROIDEA
This session covers a bunch of tips and tricks for getting the most out of Docker. The tips were inspired by suggestions, blogs, and presentations and everyday challenges encountered by other Docker Captains but also the members of the Docker community. Come and see the unobvious and unexpected in terms of orchestration, image creation and management, also networking and volumes!
Similar to Docker, Kubernetes, and Google Cloud (20)
Terraforming your Infrastructure on GCPSamuel Chow
A talk I gave at the Google Cloud Platform LA Meetup event at Google Playa Vista on Nov 6, 2019. This is a 1+ hour-long, tutorial-oriented talk on Infrastructure as Code (IaC), Terraform (as a toolset for IaC and modern devops), and leverage the practice and tools in defining, deploying, and managing your infrastructure in GCP.
What is this Docker and Microservice thing that everyone is talking about? A primer to Docker and Microservice and how the two concepts complement each other.
Our prototype and notes from our development and field tests.
This document was produced as part of a project in course 15.980 - Product Design and Development - a core course in the System Design Management (http://sdm.mit.edu) program at MIT. We are sharing the information about the project under the Creative Commons (Non Commercial) license.
Team members: Samuel Chow, Christian LaFon, Matt LaMantia, Spiros Lekkakos, David Schiller, Yoav Shapira, Eugene Xiao
These are the initial design concepts that our team came up with for the design of an automated frisbee thrower.
This document was produced as part of a project in course 15.980 - Product Design and Development - a core course in the System Design Management (http://sdm.mit.edu) program at MIT. We are sharing the information about the project under the Creative Commons (Non Commercial) license.
Team members (SDM 2006): Samuel Chow , Christian LaFon, Matt LaMantia, Spiros Lekkakos, David Schiller, Yoav Shapira, Eugene Xiao
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Into the Box Keynote Day 2: Unveiling amazing updates and announcements for modern CFML developers! Get ready for exciting releases and updates on Ortus tools and products. Stay tuned for cutting-edge innovations designed to boost your productivity.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
7. DOCKER ON MAC
• Let’s focus on running Docker on the Mac
• Remember Docker only runs on Linux x64
• How do I run it on the Mac?
• Need Virtual Machine to emulate a Linux host
• Virtual machine (VM) running Linux x64
• Docker engine running on VM
• Mac client to communicate with Docker engine on Linux VM
8. DOCKER FOR MAC VS DOCKER TOOLBOX
Docker for Mac Docker Toolbox
# VMs 1 Multiple
Underlying VM
Hypervisor.framework
(xhyve)
VirtualBox
Base OS Alpine Boot2Docker
(VM) Management Tool Docker.app docker-machine
(VM) Management UI GUI CLI
9. MAC DOCKER ARCHITECTURE
Mac OS X
Virtual Machine (VirtualBox)Docker client
docker (CLI)
Kinematic (GUI)
Docker Machine
Linux (Boot2Docker)
Container
1
Container
2
Kernel
Docker Engine
Docker
Daemon
API
docker
(CLI)
11. KUBERNETES BASICS
• Tool to orchestrate containers at scale and managing the application/service stack
• Master
• API Server and kubectl (client) – communicate and define the desired state
• Scheduler – schedule workload on nodes
• Replication – correct number of pod replicas
• Config – distributed config store
• Node (Slave)
• Kubelet – communicate with master and start workloads
• Kube-proxy – load balancer and direct traffic
• Pod – group of 1..n containers tied together for admin and networking
• Cluster = masters + nodes
13. HELLO WORLD ON GOOGLE CLOUD
(KUBERNETES)
http://kubernetes.io/docs/hellonode/
14. PRE-REQUISITES – SERVER SIDE
1. Go to https://console.cloud.google.com/
2. Create a GCP Project
3. Copy the GCP Project ID
15. PRE-REQUISITES – CLIENT (MAC) SIDE
# Install node and nvm (node version manager)
$ brew update
$ brew install nvm
$ # Add the following to ~/.bash_profile
$ # export NVM_DIR=~/.nvm
$ # source $(brew --prefix nvm)/nvm.sh
$ nvm install 7.0.0
16. PRE-REQUISITES – CLIENT (MAC) SIDE II
$ # Install docker
$ brew install docker-compose # should also install docker and
docker-machine
$ # Install google cloud sdk
$ brew cask install google-cloud-sdk
$ gcloud components install kubectl
$ # You may want to add the following:
$ EXPORT PATH=$PATH:/opt/homebrew-cask/Caskroom/google-cloud-
sdk/latest/google-cloud-sdk/bin/
$ # Set up Google Cloud environment
$ export PROJECT_ID="my-google-cloud-project-id"
17. AUTHENTICATION
# Set up your account with google cloud sdk
$ gcloud auth login my-registered-email
$ gcloud config set project my-google-cloud-project-
id
$ gcloud auth list
# Optional: env var set for convenience
$ export PROJECT_ID="my-google-cloud-project-id"
$ # Note: your project-id != project name
18. NODE.JS CODE
// Filename: server.js
var http = require('http');
var handleRequest = function(request, response) {
console.log('Received request for URL: ' + request.url);
response.writeHead(200);
response.end('Hello World!');
};
var www = http.createServer(handleRequest);
www.listen(8080);
19. RUN DOCKER-MACHINE ON LOCAL VM
$ # Before running any docker commands, run docker-machine
to create a VirtualBox instance
$ docker-machine create --driver virtualbox default
$ docker-machine env
$ eval "$(docker-machine env default)"
$ docker-machine ls
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
default * virtualbox Running tcp://192.168.99.100:2376 v1.12.0
24. PUSH IMAGE TO PRIVATE GOOGLE REGISTRY
$ docker images
$ gcloud docker -- push gcr.io/$PROJECT_ID/helloworld:v1
$ # If gcloud docker -- push doesn’t work, you probably
didn’t set your project id properly.
$ # project name != project id. For example
$ # project name = helloworld-kubernetes
$ # project id = helloworld-kubernetes-148321
26. PUSH IMAGE TO PRIVATE GOOGLE REGISTRY
$ docker images
$ gcloud docker -- push gcr.io/$PROJECT_ID/helloworld:v1
$ # If gcloud docker -- push doesn’t work, you probably
didn’t set your project id properly.
$ # project name != project id. For example
$ # project name = helloworld-kubernetes
$ # project id = helloworld-kubernetes-148321
30. GET CREDENTIALS FOR KUBECTL
• API Manager > Create
Credentials > Service
Account Key
• JSON Key type
• Download the json file
31. AUTH FOR KUBECTL
$ # If you run kubectl, you see an error message
$ kubectl version
error: google: could not find default credentials. See https://
developers.google.com/accounts/docs/application-default-credentials
for more information.
$ You need to authenticate with the crentials
$ export GOOGLE_APPLICATION_CREDENTIALS=~/helloworld-
kubernetes-abcde00000.json
$ gcloud auth application-default login
$ kubectl version # Should work now
32. RUN KUBERNETES NODE
$ # Create and run a Kubernetes pod
$ kubectl run helloworld --image=gcr.io/$PROJECT_ID/helloworld:v1 --
port=8080
deployment "helloworld" created
$ # Print deployments
$ kubectl get deployments
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
helloworld 1 1 1 1 1m
$ # Print pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld-2696007752-golst 1/1 Running 0 5m
33. TEST WEBSITE
$ # Expose pod. By default a Kubernetes node is only
accessible by its internal IP address
$ kubectl expose deployment helloworld --
type="LoadBalancer"
$ kubectl get services helloworld
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helloworld 10.3.247.187 104.198.6.146 8080/TCP 2m
$ curl 104.198.6.146:8080
Hello World!
34. SCALE WEBSITE
$ # Scale the pod to 4 replicas
$ kubectl scale deployment helloworld --replicas=4
$ # Get status
$ kubectl get deployment
$ kubectl get pods
35. CHANGE CODE AND UPDATE GCP
$ # Edit server.js
$ vi server.js
$ # Build and push changes
$ docker build -t gcr.io/$PROJECT_ID/helloworld:v2 .
$ gcloud docker -- push gcr.io/$PROJECT_ID/helloworld:v2
$ # Deploy changes
$ kubectl set image deployment/helloworld helloworld=gcr.io/$PROJECT_ID/helloworld:v2
$ deployment "helloworld" image updated
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
helloworld-2696007752-bergs 1/1 Terminating 0 15m
helloworld-2696007752-c87rs 1/1 Terminating 0 15m
helloworld-2696007752-golst 1/1 Terminating 0 14h
helloworld-2696007752-zwpi4 1/1 Terminating 0 15m
helloworld-2777403465-e802v 1/1 Running 0 11s
helloworld-2777403465-ksyxe 0/1 ContainerCreating 0 5s
helloworld-2777403465-rgq7f 1/1 Running 0 11s
helloworld-2777403465-six3e 1/1 Running 0 4s
$ kubectl get services helloworld
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
helloworld 10.3.247.187 104.198.6.146 8080/TCP 14h
$ curl 104.198.6.146:8080
Hello World 2!