"Containers wrap up software with all its dependencies in packages that can be executed anywhere. This can be specially useful in HPC environments where, often, getting the right combination of software tools to build applications is a daunting task. However, typical container solutions such as Docker are not a perfect fit for HPC environments. Instead, Shifter is a better fit as it has been built from the ground up with HPC in mind. In this talk, we show you what Shifter is and how to leverage from the current Docker environment to run your applications with Shifter."
Watch the video presentation: http://wp.me/p3RLHQ-f81
See more talks in the Switzerland HPC Conference Video Gallery: http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Fast. Dangerous. Always in control.
Learn the dirty secrets of the Notes Client and how you can turn them into golden features that will make you shine. You will leave the workshop equipped with new knowledge for your next Notes Client deployment and/or optimization project. You will be able to get better Notes client performance and stability by using less of the system resources, like CPU, Memory and File I/O – just because of the right tailor-made configuration of the Notes client for your very own system requirements. Get geared up for your next Notes V11 deployment with the best-practice tips to get Notes Clients deployed, configured, maintained and ‘finally’ loved by your users.
Don’t forget, IBM Notes V11 is not far away from being released.
You might (or might not) have heard of Docker. But you have no idea what it is and why you should care. But if you are a database or APEX developer and still work with Virtual Machines, it is about time to broaden your horizon.
In this session you'll learn what Docker is and how you can benefit from it in your daily work.
In this presentation we will walk through the following subjects:
- What is Docker
- Where do I get my images
- Pull an image
- Start a Docker container / Stop / Restart
- Use a Docker container for APEX Development : Via the browser, SQL Developer, SQL Plus, etc
- Make host directories accessible within the container
- Use scripts to modify the image or create your own one
BP101 - Can Domino Be Hacked? Lessons We Can Learn From the Security Community from MWLUG-2017 with Howard Greenberg and Andrew Pollack
The Open Web Application Security Project (OWASP) is an open source community dedicated to improving software security. OWASP publishes a Top 10 list of common security issues in web applications with suggestions on how to alleviate them. This session will examine the OWASP Top Ten list of security suggestions and relate them to the Domino world and how you can better secure your Notes and Domino applications. Both administrators and developers will gain valuable insights into how to best protect sensitive information we maintain in our Domino environments!
Identifying privilege escalation paths within an Active Directory environment is crucial for a successful red team. Over the last few years, BloodHound has made it easier for red teamers to perform reconnaissance activities and identify these attacks paths. When evaluating BloodHound data, it is common to find ourselves having sufficient rights to modify a Group Policy Object (GPO). This level of access allows us to perform a number of attacks, targeting any computer or user object controlled by the vulnerable GPO.
In this talk we will present previous research related to GPO abuses and share a number of misconfigurations we have found in the wild. We will also present a tool that allows red teamers to target users and computers controlled by a vulnerable GPO in order to escalate privileges and move laterally within the environment.
PCF Platform Monitoring with Prometheus and GrafanaVMware Tanzu
SpringOne Platform 2017
Alan Strader, Northern Trust; Jamie Christian, Northern Trust
"This presentation will cover Northern Trust's platform monitoring solution which is Grafana, Prometheus and Alertmanager. Specifically:
Enterprise need for monitoring of the platform
Options considered
Rationale for using this particular solution
Architecture of the solution and how it monitors our 5 foundations
A demo or screen captures
What we find valuable and what we look at daily to better manage the platform
Issues encountered as we deployed the solution (bosh/yml/forwarders)
Stories on how it saved us"
Version Control System - for Agile Software Project Management.Bhavya Chawla
Version control, also known as source control, is the practice of tracking and managing changes to software code. Version control systems are software tools that help software teams manage changes to source code over time.
Fast. Dangerous. Always in control.
Learn the dirty secrets of the Notes Client and how you can turn them into golden features that will make you shine. You will leave the workshop equipped with new knowledge for your next Notes Client deployment and/or optimization project. You will be able to get better Notes client performance and stability by using less of the system resources, like CPU, Memory and File I/O – just because of the right tailor-made configuration of the Notes client for your very own system requirements. Get geared up for your next Notes V11 deployment with the best-practice tips to get Notes Clients deployed, configured, maintained and ‘finally’ loved by your users.
Don’t forget, IBM Notes V11 is not far away from being released.
You might (or might not) have heard of Docker. But you have no idea what it is and why you should care. But if you are a database or APEX developer and still work with Virtual Machines, it is about time to broaden your horizon.
In this session you'll learn what Docker is and how you can benefit from it in your daily work.
In this presentation we will walk through the following subjects:
- What is Docker
- Where do I get my images
- Pull an image
- Start a Docker container / Stop / Restart
- Use a Docker container for APEX Development : Via the browser, SQL Developer, SQL Plus, etc
- Make host directories accessible within the container
- Use scripts to modify the image or create your own one
BP101 - Can Domino Be Hacked? Lessons We Can Learn From the Security Community from MWLUG-2017 with Howard Greenberg and Andrew Pollack
The Open Web Application Security Project (OWASP) is an open source community dedicated to improving software security. OWASP publishes a Top 10 list of common security issues in web applications with suggestions on how to alleviate them. This session will examine the OWASP Top Ten list of security suggestions and relate them to the Domino world and how you can better secure your Notes and Domino applications. Both administrators and developers will gain valuable insights into how to best protect sensitive information we maintain in our Domino environments!
Identifying privilege escalation paths within an Active Directory environment is crucial for a successful red team. Over the last few years, BloodHound has made it easier for red teamers to perform reconnaissance activities and identify these attacks paths. When evaluating BloodHound data, it is common to find ourselves having sufficient rights to modify a Group Policy Object (GPO). This level of access allows us to perform a number of attacks, targeting any computer or user object controlled by the vulnerable GPO.
In this talk we will present previous research related to GPO abuses and share a number of misconfigurations we have found in the wild. We will also present a tool that allows red teamers to target users and computers controlled by a vulnerable GPO in order to escalate privileges and move laterally within the environment.
PCF Platform Monitoring with Prometheus and GrafanaVMware Tanzu
SpringOne Platform 2017
Alan Strader, Northern Trust; Jamie Christian, Northern Trust
"This presentation will cover Northern Trust's platform monitoring solution which is Grafana, Prometheus and Alertmanager. Specifically:
Enterprise need for monitoring of the platform
Options considered
Rationale for using this particular solution
Architecture of the solution and how it monitors our 5 foundations
A demo or screen captures
What we find valuable and what we look at daily to better manage the platform
Issues encountered as we deployed the solution (bosh/yml/forwarders)
Stories on how it saved us"
Version Control System - for Agile Software Project Management.Bhavya Chawla
Version control, also known as source control, is the practice of tracking and managing changes to software code. Version control systems are software tools that help software teams manage changes to source code over time.
A presentation delivered by Arctiq, onsite in Toronto, on Mar 1, 2017. The presentation discusses Ansible as an automation tool for Linux, Windows, and network devices. Reach out if you would like more information www.arctiq.ca
oVirt and OpenStack look kind of similar from a distance. But they cater to different use-cases. That said, they do have some common needs. How can they work together? And when is it better to use one over the other?
Docker Compose is the last piece of the orchestration puzzle. After provisioning Docker daemons on any host in any location with Docker Machine and clustering them with Docker Swarm, users can employ Docker Compose to assemble multi-container distributed apps that run on top of these clusters.
The first step to employing Docker Compose is to use a simple YAML file to declaratively define the desired state of the multi-container app:
containers:
web:
build: .
command: python app.py
ports:
- "5000:5000"
volumes:
- .:/code
links:
- redis
environment:
- PYTHONUNBUFFERED=1
redis:
image: redis:latest
command: redis-server --appendonly yes
This example shows how Docker Compose takes advantage of existing containers. Specifically, in this simple two-container app declaration, the first container is a Python app built each time from the Dockerfile in the current directory. The second container is built from the redis Official Repo on the Docker Hub Registry. The links directive declares that the Python app container is dependent on the redis container.
Not that it’s defined, starting your app is as easy as …
% docker up
With this single command, the Python container is automatically built from its Dockerfile and the redis container is pulled from the Docker Hub Registry. Then, thanks to the links directive expressing the dependency between the Python and redis containers, the redis container is started *first*, followed by the Python container.
Docker Compose is still a work-in-progress and we want your help to design it. In particular, we want to know whether or not you think this should be a part of the Docker binary or a separate tool. Head over to the proposal on GitHub to try out an alpha build and have your say.
Coda
All this is just the briefest introduction to Docker Machine, Docker Swarm, and Docker Compose. We hope you’ll take a moment to try them out and give us feedback – these projects are moving quickly and we welcome your input!
We also wish to thank the many community members who have contributed their experience, feedback, and pull requests during the pre-Alpha iterations of these projects. It’s thanks to you that we were able to make so much progress so quickly, and in the right direction.
Distributed apps offer many benefits to users – portability, scalability, dynamic development-to-deployment acceleration – and we’re excited by the role the Docker platform, community, and ecosystem are playing in making these apps easier to build, ship, and run. We’ve got a ways to go, but we’re psyched by this start – join us and help us get there faster!
Docker is popular open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers and provide an overview of Docker. Participants can learn important concepts in Docker step-by-step and learn by example by running commands with me. The main session involves using Docker CLI (Command Line Interface) covering all the key concepts such as creating images and managing containers. What is more, this workshop ends with a complete example of getting some amazing work done with ease using Docker. Presented in OSI Days '16: http://opensourceindia.in/osidays/workshops-osi-2016/
Cette formation présente les élements de base pour maîtriser git/gitLab. Elle contient aussi des astuces et de bonnes pratiques pour mieux utiliser git.
Elijah Charles from Intel presented this deck at the 2016 HPC Advisory Council Switzerland Conference.
"The Exascale computing challenge is the current Holy Grail for high performance computing. It envisages building HPC systems capable of 10^18 floating point operations under a power input in the range of 20-40 MW. To achieve this feat, several barriers need to be overcome. These barriers or “walls” are not completely independent of each other, but present a lens through which HPC system design can be viewed as a whole, and its composing sub-systems optimized to overcome the persistent bottlenecks."
Watch the video presentation: http://wp.me/p3RLHQ-f7X
See more talks in the Switzerland HPC Conference Video Gallery: http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
A short presentation at a CSC internal workshop of the prospects of using container technologies, especially Docker, in the context of High Performance Computing (HPC).
A presentation delivered by Arctiq, onsite in Toronto, on Mar 1, 2017. The presentation discusses Ansible as an automation tool for Linux, Windows, and network devices. Reach out if you would like more information www.arctiq.ca
oVirt and OpenStack look kind of similar from a distance. But they cater to different use-cases. That said, they do have some common needs. How can they work together? And when is it better to use one over the other?
Docker Compose is the last piece of the orchestration puzzle. After provisioning Docker daemons on any host in any location with Docker Machine and clustering them with Docker Swarm, users can employ Docker Compose to assemble multi-container distributed apps that run on top of these clusters.
The first step to employing Docker Compose is to use a simple YAML file to declaratively define the desired state of the multi-container app:
containers:
web:
build: .
command: python app.py
ports:
- "5000:5000"
volumes:
- .:/code
links:
- redis
environment:
- PYTHONUNBUFFERED=1
redis:
image: redis:latest
command: redis-server --appendonly yes
This example shows how Docker Compose takes advantage of existing containers. Specifically, in this simple two-container app declaration, the first container is a Python app built each time from the Dockerfile in the current directory. The second container is built from the redis Official Repo on the Docker Hub Registry. The links directive declares that the Python app container is dependent on the redis container.
Not that it’s defined, starting your app is as easy as …
% docker up
With this single command, the Python container is automatically built from its Dockerfile and the redis container is pulled from the Docker Hub Registry. Then, thanks to the links directive expressing the dependency between the Python and redis containers, the redis container is started *first*, followed by the Python container.
Docker Compose is still a work-in-progress and we want your help to design it. In particular, we want to know whether or not you think this should be a part of the Docker binary or a separate tool. Head over to the proposal on GitHub to try out an alpha build and have your say.
Coda
All this is just the briefest introduction to Docker Machine, Docker Swarm, and Docker Compose. We hope you’ll take a moment to try them out and give us feedback – these projects are moving quickly and we welcome your input!
We also wish to thank the many community members who have contributed their experience, feedback, and pull requests during the pre-Alpha iterations of these projects. It’s thanks to you that we were able to make so much progress so quickly, and in the right direction.
Distributed apps offer many benefits to users – portability, scalability, dynamic development-to-deployment acceleration – and we’re excited by the role the Docker platform, community, and ecosystem are playing in making these apps easier to build, ship, and run. We’ve got a ways to go, but we’re psyched by this start – join us and help us get there faster!
Docker is popular open-source software containerization platform. It provides an ability to package software into standardised units on Docker for software development. In this hands-on introductory session, I introduce the concept of containers and provide an overview of Docker. Participants can learn important concepts in Docker step-by-step and learn by example by running commands with me. The main session involves using Docker CLI (Command Line Interface) covering all the key concepts such as creating images and managing containers. What is more, this workshop ends with a complete example of getting some amazing work done with ease using Docker. Presented in OSI Days '16: http://opensourceindia.in/osidays/workshops-osi-2016/
Cette formation présente les élements de base pour maîtriser git/gitLab. Elle contient aussi des astuces et de bonnes pratiques pour mieux utiliser git.
Elijah Charles from Intel presented this deck at the 2016 HPC Advisory Council Switzerland Conference.
"The Exascale computing challenge is the current Holy Grail for high performance computing. It envisages building HPC systems capable of 10^18 floating point operations under a power input in the range of 20-40 MW. To achieve this feat, several barriers need to be overcome. These barriers or “walls” are not completely independent of each other, but present a lens through which HPC system design can be viewed as a whole, and its composing sub-systems optimized to overcome the persistent bottlenecks."
Watch the video presentation: http://wp.me/p3RLHQ-f7X
See more talks in the Switzerland HPC Conference Video Gallery: http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
A short presentation at a CSC internal workshop of the prospects of using container technologies, especially Docker, in the context of High Performance Computing (HPC).
In this video from the Docker Workshop at ISC 2015, Christian Kniep from QNIB Solutions shows how he uses Docker in his efforts to provide a HPC software stack in a box, encapsulating each layer in the HPC stack within a Linux Container.
Watch the video presentation: http://wp.me/p3RLHQ-eos
Learn more: http://qnib.org/about/
In this deck from the Docker Workshop at ISC 2015, Andreas Schmidt from Cassini Consulting describes Docker in a Nutshell
"As the newest flavor of Linux Containers, Docker gained a lot of momentum in the last 12 months. With a very convenient and open API-driven architecture Docker is able to help decrease the complexity of operations and increase the productivity of computation. During the last two years Andreas, Christian, and Wolfgang gained a lot of experience with Docker and were thrilled by its possible impact early on. Andreas started working with Docker in mid-2013 and is interested in developing tools for solving Enterprise IT requirements on networking and security. In 2014 he held talks and workshops about these topics. Christian started using Docker in 2013 to virtualize a complete HPC cluster stack and since then held multiple talks about how Docker might impact HPC. Wolfgang and his partner Burak Yenier introduced Docker as a corner-stone of the UberCloud Marketplace to drastically improve and simplify access to HPC cloud resources. UberCloud just announced their new containers for computational fluid dynamics software like Fluent, STAR-CCM+ and OpenFOAM."
Watch the video presentation: http://wp.me/p3RLHQ-enP
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
One might find it ironic that some of the world's fastest supercomputers -- vast clusters capable of trillions of floating point operations per second -- can take upwards of a half an hour to reboot in between jobs. While we often talk about the density advantages of containers, it's the opposite approach that we use in the High Performance Computing world! Here, we use exactly 1 system container per node, giving it unlimited access to all of the host's CPU, Memory, Disk, IO, and Network. And yet we can still leverage the management characteristics of containers -- security, snapshots, live migration, and instant deployment to recycle each node in between jobs. In this talk, we'll examine a reference architecture and some best practices around containers in HPC environments.
Using Docker for GPU Accelerated ApplicationsNVIDIA
Build and run Docker containers leveraging NVIDIA GPUs. Containerizing GPU applications provides several benefits, among them:
* Reproducible builds
* Ease of deployment
* Isolation of individual devices
* Run across heterogeneous driver/toolkit environments
* Requires only the NVIDIA driver to be installed
* Enables "fire and forget" GPU applications
* Facilitate collaboration
Présentation du système Docker animée par Sebastien Binet (CNRS/IN2P3/LPC) en Décembre 2015 au Proto204
http://reseau-loops.github.io/journee_2015_12.html
An overview of Docker and Linux containers. There are three parts:
An introduction to Docker and containers
A demo that the audience can try out
An overview of the various vendors and groups in this space
The demo is meant to be a simple, step-by-step recipe that introduces the basic commands and ends by spinning up a node.js app using two linked containers: node and redis.
The final section explores the companies and groups that are working on containers, either complementing Docker's contributions or in direct competition with them.
Rooting Out Root: User namespaces in DockerPhil Estes
This talk on the progress to bring user namespace support into Docker was presented by Phil Estes at LinuxCon/ContainerCon 2015 on Wednesday, Aug. 19th, 2015
It is a simple introduction to the containers world, starting from LXC to arrive to the Docker Platform.
The presentation is focused on the first steps in the docker environment and the scenarious from a developer point of view.
ExpoQA 2017 Using docker to build and test in your laptop and JenkinsElasTest Project
In this workshop the basics about container use in the development environment are presented. Then we go further by describing how to leverage containers in the CI server, using Jenkins and Pipelines.
Docker 1 0 1 0 1: a Docker introduction, actualized for the stable release of...Jérôme Petazzoni
If you're not familiar yet with Docker, here is your chance to catch up. This presentation includes a quick overview of the Open Source Docker Engine, and its associated services delivered through the Docker Hub. Recent features are listed, as well as a glimpse at what's next in the Docker world.
This presentation was given during OSCON, at a meet-up hosted by New Relic, with co-presentations from CoreOS and Rackspace OnMetal.
Introduction to Docker at the Azure Meet-up in New YorkJérôme Petazzoni
This is the presentation given at the Azure New York Meet-Up group, September 3rd.
It includes a quick overview of the Open Source Docker Engine and its associated services delivered through the Docker Hub. It also covers the new features of Docker 1.0, and briefly explains how to get started with Docker on Azure.
Docker is the next best thing in deployment and infrastructure management. This talk will go over a brief introduction of the Docker objects and how they interact.
JDD2014: Docker.io - versioned linux containers for JVM devops - Dominik DornPROIDEA
This presentation will introduce you to Docker - the new shiny star on the Devops horizon. It will teach you everything you need to know to get started with Docker, why you'd want to use it and which tools to use to get the most out of it. Additionally to showing the basics, it will introduce helpful libraries available for the JVM and how they can be used together with Docker to create secure, scalable and maintainable cloud setups for your applications.
Dockerizing Symfony2 application. Why Docker is so cool And what is Docker? And what are Containers? How they works? What are the ecosystem of Docker? And how to dockerize your web application (can be based on Symfony2 framework)?
Overview of Docker 1.11 features(Covers Docker release summary till 1.11, runc/containerd, dns load balancing ipv6 service discovery, labels, macvlan/ipvlan)
Real-World Docker: 10 Things We've Learned RightScale
Docker has taken the world of software by storm, offering the promise of a portable way to build and ship software - including software running in the cloud. The RightScale development team has been diving into Docker for several projects, and we'll share our lessons learned on using Docker for our cloud-based applications.
In this deck from the Stanford HPC Conference, Shahin Khan from OrionX describes major market Shifts in IT.
"We will discuss the digital infrastructure of the future enterprise and the state of these trends."
"We work with clients on the impact of Digital Transformation (DX) on them, their customers, and their messages. Generally, they want to track, in one place, trends like IoT, 5G, AI, Blockchain, and Quantum Computing. And they want to know what these trends mean, how they affect each other, and when they demand action, and how to formulate and execute an effective plan. If that describes you, we can help."
Watch the video: https://wp.me/p3RLHQ-lPP
Learn more: http://orionx.net
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Preparing to program Aurora at Exascale - Early experiences and future direct...inside-BigData.com
In this deck from IWOCL / SYCLcon 2020, Hal Finkel from Argonne National Laboratory presents: Preparing to program Aurora at Exascale - Early experiences and future directions.
"Argonne National Laboratory’s Leadership Computing Facility will be home to Aurora, our first exascale supercomputer. Aurora promises to take scientific computing to a whole new level, and scientists and engineers from many different fields will take advantage of Aurora’s unprecedented computational capabilities to push the boundaries of human knowledge. In addition, Aurora’s support for advanced machine-learning and big-data computations will enable scientific workflows incorporating these techniques along with traditional HPC algorithms. Programming the state-of-the-art hardware in Aurora will be accomplished using state-of-the-art programming models. Some of these models, such as OpenMP, are long-established in the HPC ecosystem. Other models, such as Intel’s oneAPI, based on SYCL, are relatively-new models constructed with the benefit of significant experience. Many applications will not use these models directly, but rather, will use C++ abstraction libraries such as Kokkos or RAJA. Python will also be a common entry point to high-performance capabilities. As we look toward the future, features in the C++ standard itself will become increasingly relevant for accessing the extreme parallelism of exascale platforms.
This presentation will summarize the experiences of our team as we prepare for Aurora, exploring how to port applications to Aurora’s architecture and programming models, and distilling the challenges and best practices we’ve developed to date. oneAPI/SYCL and OpenMP are both critical models in these efforts, and while the ecosystem for Aurora has yet to mature, we’ve already had a great deal of success. Importantly, we are not passive recipients of programming models developed by others. Our team works not only with vendor-provided compilers and tools, but also develops improved open-source LLVM-based technologies that feed both open-source and vendor-provided capabilities. In addition, we actively participate in the standardization of OpenMP, SYCL, and C++. To conclude, I’ll share our thoughts on how these models can best develop in the future to support exascale-class systems."
Watch the video: https://wp.me/p3RLHQ-lPT
Learn more: https://www.iwocl.org/iwocl-2020/conference-program/
and
https://www.anl.gov/topic/aurora
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Greg Wahl from Advantech presents: Transforming Private 5G Networks.
Advantech Networks & Communications Group is driving innovation in next-generation network solutions with their High Performance Servers. We provide business critical hardware to the world's leading telecom and networking equipment manufacturers with both standard and customized products. Our High Performance Servers are highly configurable platforms designed to balance the best in x86 server-class processing performance with maximum I/O and offload density. The systems are cost effective, highly available and optimized to meet next generation networking and media processing needs.
“Advantech’s Networks and Communication Group has been both an innovator and trusted enabling partner in the telecommunications and network security markets for over a decade, designing and manufacturing products for OEMs that accelerate their network platform evolution and time to market.” Said Advantech Vice President of Networks & Communications Group, Ween Niu. “In the new IP Infrastructure era, we will be expanding our expertise in Software Defined Networking (SDN) and Network Function Virtualization (NFV), two of the essential conduits to 5G infrastructure agility making networks easier to install, secure, automate and manage in a cloud-based infrastructure.”
In addition to innovation in air interface technologies and architecture extensions, 5G will also need a new generation of network computing platforms to run the emerging software defined infrastructure, one that provides greater topology flexibility, essential to deliver on the promises of high availability, high coverage, low latency and high bandwidth connections. This will open up new parallel industry opportunities through dedicated 5G network slices reserved for specific industries dedicated to video traffic, augmented reality, IoT, connected cars etc. 5G unlocks many new doors and one of the keys to its enablement lies in the elasticity and flexibility of the underlying infrastructure.
Advantech’s corporate vision is to enable an intelligent planet. The company is a global leader in the fields of IoT intelligent systems and embedded platforms. To embrace the trends of IoT, big data, and artificial intelligence, Advantech promotes IoT hardware and software solutions with the Edge Intelligence WISE-PaaS core to assist business partners and clients in connecting their industrial chains. Advantech is also working with business partners to co-create business ecosystems that accelerate the goal of industrial intelligence."
Watch the video: https://wp.me/p3RLHQ-lPQ
* Company website: https://www.advantech.com/
* Solution page: https://www2.advantech.com/nc/newsletter/NCG/SKY/benefits.html
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...inside-BigData.com
In this deck from the Stanford HPC Conference, Katie Lewis from Lawrence Livermore National Laboratory presents: The Incorporation of Machine Learning into Scientific Simulations at Lawrence Livermore National Laboratory.
"Scientific simulations have driven computing at Lawrence Livermore National Laboratory (LLNL) for decades. During that time, we have seen significant changes in hardware, tools, and algorithms. Today, data science, including machine learning, is one of the fastest growing areas of computing, and LLNL is investing in hardware, applications, and algorithms in this space. While the use of simulations to focus and understand experiments is well accepted in our community, machine learning brings new challenges that need to be addressed. I will explore applications for machine learning in scientific simulations that are showing promising results and further investigation that is needed to better understand its usefulness."
Watch the video: https://youtu.be/NVwmvCWpZ6Y
Learn more: https://computing.llnl.gov/research-area/machine-learning
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...inside-BigData.com
In this deck from the Stanford HPC Conference, DK Panda from Ohio State University presents: How to Achieve High-Performance, Scalable and Distributed DNN Training on Modern HPC Systems?
"This talk will start with an overview of challenges being faced by the AI community to achieve high-performance, scalable and distributed DNN training on Modern HPC systems with both scale-up and scale-out strategies. After that, the talk will focus on a range of solutions being carried out in my group to address these challenges. The solutions will include: 1) MPI-driven Deep Learning, 2) Co-designing Deep Learning Stacks with High-Performance MPI, 3) Out-of- core DNN training, and 4) Hybrid (Data and Model) parallelism. Case studies to accelerate DNN training with popular frameworks like TensorFlow, PyTorch, MXNet and Caffe on modern HPC systems will be presented."
Watch the video: https://youtu.be/LeUNoKZVuwQ
Learn more: http://web.cse.ohio-state.edu/~panda.2/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...inside-BigData.com
In this deck from the Stanford HPC Conference, Nick Nystrom and Paola Buitrago provide an update from the Pittsburgh Supercomputing Center.
Nick Nystrom is Chief Scientist at the Pittsburgh Supercomputing Center (PSC). Nick is architect and PI for Bridges, PSC's flagship system that successfully pioneered the convergence of HPC, AI, and Big Data. He is also PI for the NIH Human Biomolecular Atlas Program’s HIVE Infrastructure Component and co-PI for projects that bring emerging AI technologies to research (Open Compass), apply machine learning to biomedical data for breast and lung cancer (Big Data for Better Health), and identify causal relationships in biomedical big data (the Center for Causal Discovery, an NIH Big Data to Knowledge Center of Excellence). His current research interests include hardware and software architecture, applications of machine learning to multimodal data (particularly for the life sciences) and to enhance simulation, and graph analytics.
Watch the video: https://youtu.be/LWEU1L1o7yY
Learn more: https://www.psc.edu/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Ryan Quick from Providentia Worldwide describes how DNNs can be used to improve EDA simulation runs.
"Systems Intelligence relies on a variety of methods for providing insight into the core mechanisms for driving automated behavioral changes in self-healing command and control platforms. This talk reports on initial efforts with leveraging Semiconductor Electronic Design Automation (EDA) telemetry data from cross-domain sources including power, network, storage, nodes, and applications in neural networks as a driving method for insight into SI automation systems."
Watch the video: https://youtu.be/2WbR8tq-XbM
Learn more: http://www.providentiaworldwide.com/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoringinside-BigData.com
In this deck from the Stanford HPC Conference, Nicole Xu from Stanford University describes how she transformed a common jellyfish into a bionic creature that is part animal and part machine.
"Animal locomotion and bioinspiration have the potential to expand the performance capabilities of robots, but current implementations are limited. Mechanical soft robots leverage engineered materials and are highly controllable, but these biomimetic robots consume more power than corresponding animal counterparts. Biological soft robots from a bottom-up approach offer advantages such as speed and controllability but are limited to survival in cell media. Instead, biohybrid robots that comprise live animals and self- contained microelectronic systems leverage the animals’ own metabolism to reduce power constraints and body as an natural scaffold with damage tolerance. We demonstrate that by integrating onboard microelectronics into live jellyfish, we can enhance propulsion up to threefold, using only 10 mW of external power input to the microelectronics and at only a twofold increase in cost of transport to the animal. This robotic system uses 10 to 1000 times less external power per mass than existing swimming robots in literature and can be used in future applications for ocean monitoring to track environmental changes."
Watch the video: https://youtu.be/HrmJFyvInj8
Learn more: https://sanfrancisco.cbslocal.com/2020/02/05/stanford-research-project-common-jellyfish-bionic-sea-creatures/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Stanford HPC Conference, Peter Dueben from the European Centre for Medium-Range Weather Forecasts (ECMWF) presents: Machine Learning for Weather Forecasts.
"I will present recent studies that use deep learning to learn the equations of motion of the atmosphere, to emulate model components of weather forecast models and to enhance usability of weather forecasts. I will than talk about the main challenges for the application of deep learning in cutting-edge weather forecasts and suggest approaches to improve usability in the future."
Peter is contributing to the development and optimization of weather and climate models for modern supercomputers. He is focusing on a better understanding of model error and model uncertainty, on the use of reduced numerical precision that is optimised for a given level of model error, on global cloud- resolving simulations with ECMWF's forecast model, and the use of machine learning, and in particular deep learning, to improve the workflow and predictions. Peter has graduated in Physics and wrote his PhD thesis at the Max Planck Institute for Meteorology in Germany. He worked as Postdoc with Tim Palmer at the University of Oxford and has taken up a position as University Research Fellow of the Royal Society at the European Centre for Medium-Range Weather Forecasts (ECMWF) in 2017.
Watch the video: https://youtu.be/ks3fkRj8Iqc
Learn more: https://www.ecmwf.int/
and
http://www.hpcadvisorycouncil.com/events/2020/stanford-workshop/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Gilad Shainer from the HPC AI Advisory Council describes how this organization fosters innovation in the high performance computing community.
"The HPC-AI Advisory Council’s mission is to bridge the gap between high-performance computing (HPC) and Artificial Intelligence (AI) use and its potential, bring the beneficial capabilities of HPC and AI to new users for better research, education, innovation and product manufacturing, bring users the expertise needed to operate HPC and AI systems, provide application designers with the tools needed to enable parallel computing, and to strengthen the qualification and integration of HPC and AI system products."
Watch the video: https://wp.me/p3RLHQ-lNz
Learn more: http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Today RIKEN in Japan announced that the Fugaku supercomputer will be made available for research projects aimed to combat COVID-19.
"Fugaku is currently being installed and is scheduled to be available to the public in 2021. However, faced with the devastating disaster unfolding before our eyes, RIKEN and MEXT decided to make a portion of the computational resources of Fugaku available for COVID-19-related projects ahead of schedule while continuing the installation process.
Fugaku is being developed not only for the progress in science, but also to help build the society dubbed as the “Society 5.0” by the Japanese government, where all people will live safe and comfortable lives. The current initiative to fight against the novel coronavirus is driven by the philosophy behind the development of Fugaku."
Initial Projects
Exploring new drug candidates for COVID-19 by "Fugaku"
Yasushi Okuno, RIKEN / Kyoto University
Prediction of conformational dynamics of proteins on the surface of SARS-Cov-2 using Fugaku
Yuji Sugita, RIKEN
Simulation analysis of pandemic phenomena
Nobuyasu Ito, RIKEN
Fragment molecular orbital calculations for COVID-19 proteins
Yuji Mochizuki, Rikkyo University
In this deck from the Performance Optimisation and Productivity group, Lubomir Riha from IT4Innovations presents: Energy Efficient Computing using Dynamic Tuning.
"We now live in a world of power-constrained architectures and systems and power consumption represents a significant cost factor in the overall HPC system economy. For these reasons, in recent years researchers, supercomputing centers and major vendors have developed new tools and methodologies to measure and optimize the energy consumption of large-scale high performance system installations. Due to the link between energy consumption, power consumption and execution time of an application executed by the final user, it is important for these tools and the methodology used to consider all these aspects, empowering the final user and the system administrator with the capability of finding the best configuration given different high level objectives.
This webinar focused on tools designed to improve the energy-efficiency of HPC applications using a methodology of dynamic tuning of HPC applications, developed under the H2020 READEX project. The READEX methodology has been designed for exploiting the dynamic behaviour of software. At design time, different runtime situations (RTS) are detected and optimized system configurations are determined. RTSs with the same configuration are grouped into scenarios, forming the tuning model. At runtime, the tuning model is used to switch system configurations dynamically.
The MERIC tool, that implements the READEX methodology, is presented. It supports manual or binary instrumentation of the analysed applications to simplify the analysis. This instrumentation is used to identify and annotate the significant regions in the HPC application. Automatic binary instrumentation annotates regions with significant runtime. Manual instrumentation, which can be combined with automatic, allows code developer to annotate regions of particular interest."
Watch the video: https://wp.me/p3RLHQ-lJP
Learn more: https://pop-coe.eu/blog/14th-pop-webinar-energy-efficient-computing-using-dynamic-tuning
and
https://code.it4i.cz/vys0053/meric
Sign up for our insideHPC Newsletter: http://insidehpc.com/newslett
In this deck from GTC Digital, William Beaudin from DDN presents: HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD.
Enabling high performance computing through the use of GPUs requires an incredible amount of IO to sustain application performance. We'll cover architectures that enable extremely scalable applications through the use of NVIDIA’s SuperPOD and DDN’s A3I systems.
The NVIDIA DGX SuperPOD is a first-of-its-kind artificial intelligence (AI) supercomputing infrastructure. DDN A³I with the EXA5 parallel file system is a turnkey, AI data storage infrastructure for rapid deployment, featuring faster performance, effortless scale, and simplified operations through deeper integration. The combined solution delivers groundbreaking performance, deploys in weeks as a fully integrated system, and is designed to solve the world's most challenging AI problems.
Watch the video: https://wp.me/p3RLHQ-lIV
Learn more: https://www.ddn.com/download/nvidia-superpod-ddn-a3i-ai400-appliance-with-the-exa5-filesystem/
and
https://www.nvidia.com/en-us/gtc/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck, Paul Isaacs from Linaro presents: State of ARM-based HPC. This talk provides an overview of applications and infrastructure services successfully ported to Aarch64 and benefiting from scale.
"With its debut on the TOP500, the 125,000-core Astra supercomputer at New Mexico’s Sandia Labs uses Cavium ThunderX2 chips to mark Arm’s entry into the petascale world. In Japan, the Fujitsu A64FX Arm-based CPU in the pending Fugaku supercomputer has been optimized to achieve high-level, real-world application performance, anticipating up to one hundred times the application execution performance of the K computer. K was the first computer to top 10 petaflops in 2011."
Watch the video: https://wp.me/p3RLHQ-lIT
Learn more: https://www.linaro.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Versal Premium ACAP for Network and Cloud Accelerationinside-BigData.com
Today Xilinx announced Versal Premium, the third series in the Versal ACAP portfolio. The Versal Premium series features highly integrated, networked and power-optimized cores and the industry’s highest bandwidth and compute density on an adaptable platform. Versal Premium is designed for the highest bandwidth networks operating in thermally and spatially constrained environments, as well as for cloud providers who need scalable, adaptable application acceleration.
Versal is the industry’s first adaptive compute acceleration platform (ACAP), a revolutionary new category of heterogeneous compute devices with capabilities that far exceed those of conventional silicon architectures. Developed on TSMC’s 7-nanometer process technology, Versal Premium combines software programmability with dynamically configurable hardware acceleration and pre-engineered connectivity and security features to enable a faster time-to- market. The Versal Premium series delivers up to 3X higher throughput compared to current generation FPGAs, with built-in Ethernet, Interlaken, and cryptographic engines that enable fast and secure networks. The series doubles the compute density of currently deployed mainstream FPGAs and provides the adaptability to keep pace with increasingly diverse and evolving cloud and networking workloads.
Learn more: https://insidehpc.com/2020/03/xilinx-announces-versal-premium-acap-for-network-and-cloud-acceleration/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Zettar: Moving Massive Amounts of Data across Any Distance Efficientlyinside-BigData.com
In this video from the Rice Oil & Gas Conference, Chin Fang from Zettar presents: Moving Massive Amounts of Data across Any Distance Efficiently.
The objective of this talk is to present two on-going projects aiming at improving and ensuring highly efficient bulk transferring or streaming of massive amounts of data over digital connections across any distance. It examines the current state of the art, a few very common misconceptions, the differences among the three major type of data movement solutions, a current initiative attempting to improve the data movement efficiency from the ground up, and another multi-stage project that shows how to conduct long distance large scale data movement at speed and scale internationally. Both projects have real world motivations, e.g. the ambitious data transfer requirements of Linac Coherent Light Source II (LCLS-II) [1], a premier preparation project of the U.S. DOE Exascale Computing Initiative (ECI) [2]. Their immediate goals are described and explained, together with the solution used for each. Findings and early results are reported. Possible future works are outlined.
Watch the video: https://wp.me/p3RLHQ-lBX
Learn more: https://www.zettar.com/
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from the Rice Oil & Gas Conference, Bradley McCredie from AMD presents: Scaling TCO in a Post Moore's Law Era.
"While foundries bravely drive forward to overcome the technical and economic challenges posed by scaling to 5nm and beyond, Moore’s law alone can provide only a fraction of the performance / watt and performance / dollar gains needed to satisfy the demands of today’s high performance computing and artificial intelligence applications. To close the gap, multiple strategies are required. First, new levels of innovation and design efficiency will supplement technology gains to continue to deliver meaningful improvements in SoC performance. Second, heterogenous compute architectures will create x-factor increases of performance efficiency for the most critical applications. Finally, open software frameworks, APIs, and toolsets will enable broad ecosystems of application level innovation."
Watch the video:
Learn more: http://amd.com
and
https://rice2020oghpc.rice.edu/program-2/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
CUDA-Python and RAPIDS for blazing fast scientific computinginside-BigData.com
In this deck from the ECSS Symposium, Abe Stern from NVIDIA presents: CUDA-Python and RAPIDS for blazing fast scientific computing.
"We will introduce Numba and RAPIDS for GPU programming in Python. Numba allows us to write just-in-time compiled CUDA code in Python, giving us easy access to the power of GPUs from a powerful high-level language. RAPIDS is a suite of tools with a Python interface for machine learning and dataframe operations. Together, Numba and RAPIDS represent a potent set of tools for rapid prototyping, development, and analysis for scientific computing. We will cover the basics of each library and go over simple examples to get users started. Finally, we will briefly highlight several other relevant libraries for GPU programming."
Watch the video: https://wp.me/p3RLHQ-lvu
Learn more: https://developer.nvidia.com/rapids
and
https://www.xsede.org/for-users/ecss/ecss-symposium
Sign up for our insideHPC Newsletter: http://insidehp.com/newsletter
In this deck from FOSDEM 2020, Colin Sauze from Aberystwyth University describes the development of a RaspberryPi cluster for teaching an introduction to HPC.
"The motivation for this was to overcome four key problems faced by new HPC users:
* The availability of a real HPC system and the effect running training courses can have on the real system, conversely the availability of spare resources on the real system can cause problems for the training course.
* A fear of using a large and expensive HPC system for the first time and worries that doing something wrong might damage the system.
* That HPC systems are very abstract systems sitting in data centres that users never see, it is difficult for them to understand exactly what it is they are using.
* That new users fail to understand resource limitations, in part because of the vast resources in modern HPC systems a lot of mistakes can be made before running out of resources. A more resource constrained system makes it easier to understand this.
The talk will also discuss some of the technical challenges in deploying an HPC environment to a Raspberry Pi and attempts to keep that environment as close to a "real" HPC as possible. The issue to trying to automate the installation process will also be covered."
Learn more: https://github.com/colinsauze/pi_cluster
and
https://fosdem.org/2020/schedule/events/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects.
"The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future."
Watch the video: https://wp.me/p3RLHQ-luc
Learn more: https://extremecomputingtraining.anl.gov/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
3. What is Docker and how does it work?
§ “Docker containers wrap up a piece of software in a complete filesystem that
contains everything it needs to run: code, runtime, system tools, system libraries
– anything you can install on a server. This guarantees that it will always run the
same, regardless of the environment it is running in.” [*]
Shifter: Containers in HPC environments 3
Virtual Machines Containers
[*] https://www.docker.com/what-docker
[*]
[*]
4. What is Docker and how does it work?
§ A Docker image is a file that contains a filesystem within it. It can be a full
filesystem (i.e. CentOS) or partial (i.e. python3.4-slim). Images are stored either
on a public registry (Docker hub) or private
§ Docker leverages the namespaces feature of the kernel to isolate processes
§ On its simplest form, Docker basically
1. Pulls an image to the local system
2. Creates a chrooted environment with the image (=container)
3. Runs our application in the container (’isolated’ from the host thanks to kernel namespaces)
§ However, it can also do other things:
§ Isolate network by creating NAT or bridge devices
§ Can use a nice GUI
Shifter: Containers in HPC environments 4
5. Docker in HPC environments
§ Docker is a nice tool, but it’s not built for HPC environments, because:
§ Does not integrate well with workload managers
§ Does not isolate users on shared filesystems
§ Requires running a daemon on all nodes
§ Not designed to run on diskless clients
§ Network is, by default, ‘NATed’
§ Building Docker is done within a Docker container. It can be done outside, but is a complex
task (Go language, seriously??)
§ But after all, a sysadmin can make anything to work on a cluster, right?
§ We can create (and hopefully maintain) monstrous wrappers to run Docker containers…
Shifter: Containers in HPC environments 5
7. What is Shifter and how does it work?
§ Shifter is a container-based solution thought from the ground up for HPC
environments
§ It leverages the current Docker environment and can use Docker images to
create containers
§ Shifter basically
1. Pulls an image to a shared location (/scratch)
2. Creates a loop device with the image (=container)
3. Creates a chrooted environment on the loop device
4. Runs our application in chrooted environment
§ Designed to work on HPC clusters and, particularly, on Cray systems
§ It is possible to choose which filesystems to expose to the containers
Shifter: Containers in HPC environments 7
8. External nodeCompute node
Architecture of Shifter
§ Shifter consists on two parts
§ udiRoot is responsible for creating the loop devices, doing chroot and cleaning up after the
binary execution is done. Workload manager plugins are available. Written in C
§ imageGateway is responsible for fetching a Docker image, converting it to a squashfs file and
transfer it to a shared location on a filesystem. It also keeps track of the images and tells
udiRoot which are available. Written in Python
Shifter: Containers in HPC environments 8
scratch
udiRoot imageGateway
query
restful API
db
Docker
Hub
Or private
registry
9. Workflow
1. User creates an image on his/her computer and pushes it to Docker Hub
Shifter: Containers in HPC environments 9
Docker
image
1 2
$ docker build .
Uploading context 10240 bytes
Step 1 : FROM busybox
Pulling repository busybox
---> e9aa60c60128MB/2.284 MB (100%) endpoint: https://cdn-registry-
1.docker.io/v1/
Step 2 : RUN ls -lh /
---> Running in 9c9e81692ae9
total 24
drwxr-xr-x 2 root root 4.0K Mar 12 2013 bin
drwxr-xr-x 5 root root 4.0K Oct 19 00:19 dev
drwxr-xr-x 2 root root 4.0K Oct 19 00:19 etc
drwxr-xr-x 2 root root 4.0K Nov 15 23:34 lib
lrwxrwxrwx 1 root root 3 Mar 12 2013 lib64 -> lib
dr-xr-xr-x 116 root root 0 Nov 15 23:34 proc
lrwxrwxrwx 1 root root 3 Mar 12 2013 sbin -> bin
dr-xr-xr-x 13 root root 0 Nov 15 23:34 sys
drwxr-xr-x 2 root root 4.0K Mar 12 2013 tmp
drwxr-xr-x 2 root root 4.0K Nov 15 23:34 usr
---> b35f4035db3f
Step 3 : CMD echo Hello world
---> Running in 02071fceb21b
---> f52f38b7823e
Successfully built f52f38b7823e
Removing intermediate container 9c9e81692ae9
Removing intermediate container 02071fceb21b
docker build docker push
$ docker push miguelgila/wlcg_wn:20161212
Docker
Hub
Or private
registry
10. Workflow
1. User creates an image on his/her computer and pushes it to Docker Hub
2. User tells imageGateway to pull his image and make it available
Shifter: Containers in HPC environments 10
$ ssh santis01
$ module load shifter
$ shifterimg pull docker:ubuntu:14.04
$ sleep 300 # :-)
$ shifterimg images
santis docker READY 2934337e50 2015-12-10T09:03:38 python:2.7-slim
santis docker READY a4f04f71be 2015-12-10T09:47:17 python:3.5-slim
santis docker READY b0197931ac 2015-12-10T09:53:54 python:3.2-slim
santis docker READY 0bb6b50d84 2015-12-10T10:34:24 python:3.3-slim
santis docker READY cd86406b32 2015-12-10T10:36:55 python:3.5.1
santis docker READY 48bc52cc28 2015-12-10T10:47:35 python:3.4.3
santis docker READY 0b92f1735f 2015-12-10T11:17:26 python:3.4-slim
santis docker READY 3fba104814 2015-12-10T11:31:23 centos:6.7
santis docker READY 12c9d795d8 2015-12-10T11:37:35 centos:6.6
santis docker READY 3e0c71ada2 2015-12-10T15:19:24 ubuntu:15.10
santis docker READY e751d10964 2015-12-10T13:38:05 miguelgila/wlcg_wn:20151125v2
santis docker READY d55e68e6cc 2015-12-10T15:52:09 ubuntu:14.04
shifterimg pull
/scratch/
santis
imageGateway
1
2
3 4
Docker
image
Shifter
image
Shifter
image
Docker
Hub
Or private
registry
11. Compute node
Workflow
1. User creates an image on his/her computer and pushes it to Docker Hub
2. User tells imageGateway to pull his image and make it available
3. User runs SBATCH and prepends shifter to his/her executable
Shifter: Containers in HPC environments 11
#!/bin/bash –l
#SBATCH --job-name="shifter_osversion”
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=1
#SBATCH --time=00:30:00
#SBATCH --exclusive
#SBATCH --output=/users/miguelgi/jobs/out/shifter_hostname.stdout.log.%j
#SBATCH --error=/users/miguelgi/jobs/out/shifter_hostname.stdout.log.%j
#SBATCH --image=docker:miguelgila/wlcg_wn:20151218
#SBATCH --imagevolume=/users/miguelgi:/users/miguelgi
#SBATCH --imagevolume=/apps:/apps
#SBATCH --imagevolume=/scratch/santis:/scratch/santis
module load slurm
module load shifter/15.12.0
srun shifter --volume=/scratch/santis:/scratch/santis --volume=/users:/users cat
/etc/redhat-release
sbatch
/scratch/
santis
SRUN/SPANK
1
3
udiRoot
Loop device
/dev/loopY
Shifter
image
Create loop dev
shifter binary
Run bin in loop dev
/users
2
15. Multi-node containers
§ It is possible to run the same container across multiple nodes:
§ Working on getting MPI across nodes to function
§ Working on getting GPUs to be accessible to containers with good results so far (native
performance!)
Shifter: Containers in HPC environments 15
lucasbe@santis01 ~/shifter-gpu> sbatch ./nvidia-docker/samples/cuda-stream/benchmark.sbatch
Submitted batch job 496
lucasbe@santis01 /scratch/santis/lucasbe/jobs> cat shifter-gpu.out.log
Launching GPU stream benchmark on nid00012 ...
STREAM Benchmark implementation in CUDA
Array size (double precision) = 1073.74 MB
using 192 threads per block, 699051 blocks
Function Rate (GB/s) Avg time(s) Min time(s) Max time(s)
Copy: 184.3169 0.01167758 0.01165104 0.01170397
Scale: 183.1849 0.01175387 0.01172304 0.01178598
Add: 180.3075 0.01790012 0.01786518 0.01792288
Triad: 180.1056 0.01790700 0.01788521 0.01794291
GPU
#! /usr/bin/env python
import platform
import socket
print(platform.python_version())
print(socket.gethostname())
hostname.py
[miguelgi@santis01]-[10:39:43]-[~/examples]:-)$ salloc -t 00:15:00 -N2 -w nid00013,nid00014 --image=docker:python:3.2-slim
salloc: Granted job allocation 16285
salloc: Waiting for resource configuration
salloc: Nodes nid000[13-14] are ready for job
[miguelgi@santis01]-[10:41:21]-[~/examples]:-)$ srun shifter ./hostname.py
3.2.6
nid00013
3.2.6
nid00014
Python 3.2 – non-interactive session
16. Practical use case: WLCG Swiss Tier-2
§ CSCS operates the cluster Phoenix on behalf of CHIPP, the Swiss Institute of
Particle Physics
§ Phoenix runs Tier-2 jobs for ATLAS, CMS and LHCb, 3 experiments of the LHC
at CERN and part of WLCG (Worldwide LHC Computing Grid)
§ WLCG jobs need and expect RHEL-compatible OS. All software is precompiled
and exposed in a cvmfs[*] filesystem
§ But Cray XC compute nodes run CLE, a modified version of SLES 11 SP3
§ So, how do we get these jobs to run on a Cray?
Shifter: Containers in HPC environments 16 [*] https://cernvm.cern.ch/portal/filesystem
17. Practical use case: WLCG Swiss Tier-2
§ Using Shifter, we are able to run unmodified ATLAS,
CMS and LHCb production jobs on a Cray XC40 TDS
§ Jobs see standard CentOS 6 containers
§ Nodes are shared: multiple single-core and multi-core
jobs, from different experiments, can run on the same
compute node
§ Job efficiency is comparable in both systems
Shifter: Containers in HPC environments 17
JOBID USER ACCOUNT NAME NODELIST ST REASON START_TIME END_TIME TIME_LEFT NODES CPU
82471 atlasprd atlas a53eb5f8_34f0_ nid00043 R None 15:03:33 Thu 15:03 1-23:54:18 1 8
82476 cms04 cms gridjob nid00043 R None 15:08:39 Tomorr 03:08 11:59:24 1 2
82451 lhcbplt lhcb gridjob nid00043 R None 15:00:10 Tomorr 03:00 11:50:55 1 2
82447 lhcbplt lhcb gridjob nid00043 R None 14:59:31 Tomorr 02:59 11:50:16 1 2
82448 lhcbplt lhcb gridjob nid00043 R None 14:59:31 Tomorr 02:59 11:50:16 1 2
82449 lhcbplt lhcb gridjob nid00043 R None 14:59:31 Tomorr 02:59 11:50:16 1 2
82450 lhcbplt lhcb gridjob nid00043 R None 14:59:31 Tomorr 02:59 11:50:16 1 2
82446 lhcbplt lhcb gridjob nid00043 R None 14:49:01 Tomorr 02:49 11:39:46 1 2
82444 lhcbplt lhcb gridjob nid00043 R None 14:48:01 Tomorr 02:48 11:38:46 1 2
82445 lhcbplt lhcb gridjob nid00043 R None 14:48:01 Tomorr 02:48 11:38:46 1 2
19. § Can isolate network by creating NAT or
bridge devices. What about IB?
§ Users can write as root on exposed RW
filesystems
§ Needs a local daemon running
§ Isn’t SLURM-friendly
§ Can run on multiple nodes with own tool
(Swarm)
§ Can use GPUs?
§ MPI?
§ Can run images on private registry
§ It shows all /dev, /sys and /proc to the
container environment. Easy
§ Users can write as their $USER on any
exposed RW filesystem
§ Does not need a local daemon on CN
§ Is SLURM-friendly (SPANK plugin)
§ Can run on multiple nodes with WLM
integration
§ Can use GPUs. Working on it!
§ MPI on its way!
§ Can run images on private registry
Shifter: Containers in HPC environments 19
Docker vs. Shifter
Docker
20. Conclusion
§ Shifter works very well on our HPC environment
§ It’s being constantly developed and new features are appearing on a weekly
basis
§ It’s open source and developed by the HPC community
§ It needs some additional work to cover basic HPC use cases (MPI)
§ Interacting with some parts of Shifter is not very user-friendly:
§ The process of pulling images is easy, but has no visual feedback
§ At times, error messages are difficult to understand
§ No ACLs yet
Shifter: Containers in HPC environments 20
21. Reference links
§ NERSC info: http://www.nersc.gov/research-and-development/user-defined-images/
§ The code: https://github.com/NERSC/shifter
Shifter: Containers in HPC environments 21