Audience Level
Intermediate
Synopsis
High performance computing and cloud computing have traditionally been seen as separate solutions to separate problems, dealing with issues of performance and flexibility respectively. In a diverse research environment however, both sets of compute requirements can occur. In addition to the administrative benefits in combining both requirements into a single unified system, opportunities are provided for incremental expansion.
The deployment of the Spartan cloud-HPC hybrid system at the University of Melbourne last year is an example of such a design. Despite its small size, it has attracted international attention due to its design features. This presentation, in addition to providing a grounding on why one would wish to build an HPC-cloud hybrid system and the results of the deployment, provides a complete technical overview of the design from the ground up, as well as problems encountered and planned future developments.
Speaker Bio
Lev Lafayette is the HPC and Training Officer at the University of Melbourne. Prior to that he worked at the Victorian Partnership for Advanced Computing for several years in a similar role.
HPC Best Practices: Application Performance Optimizationinside-BigData.com
Pak Lui from the HPC Advisory Council presented this deck at the Switzerland HPC Conference.
"To achieve good scalability performance on the HPC scientific applications typically involves good understanding of the workload though performing profile analysis, and comparing behaviors of using different hardware which pinpoint bottlenecks in different areas of the HPC cluster. In this session, a selection of HPC applications will be shown to demonstrate various methods of profiling and analysis to determine the bottleneck, and the effectiveness of the tuning to improve on the application performance."
Watch the video presentation: http://wp.me/p3RLHQ-f8h
Learn more: http://www.hpcadvisorycouncil.com/best_practices.php
See more talks from the Switzerland HPC Conference:
http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Santa Fe, Yoonho Park from IBM presents: IBM Datacentric Servers & OpenPOWER.
"Big data analytics, machine learning and deep learning are among the most rapidly growing workloads in the data center. These workloads have the compute performance requirements of traditional technical computing or high performance computing, coupled with a much larger volume and velocity of data."
Watch the video: http://wp.me/p3RLHQ-gJv
Learn more: https://openpowerfoundation.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
This is the presentation on clusters computing which includes information from other sources too including my own research and edition. I hope this will help everyone who required to know on this topic.
Optimising nfv service chains on open stack using dockerAnanth Padmanabhan
Uploading slides presented in the OpenStack summit, at Austin in April, 2016. Here is the link to the video,
https://www.openstack.org/videos/video/optimising-nfv-service-chains-on-openstack-using-docker
HPC Best Practices: Application Performance Optimizationinside-BigData.com
Pak Lui from the HPC Advisory Council presented this deck at the Switzerland HPC Conference.
"To achieve good scalability performance on the HPC scientific applications typically involves good understanding of the workload though performing profile analysis, and comparing behaviors of using different hardware which pinpoint bottlenecks in different areas of the HPC cluster. In this session, a selection of HPC applications will be shown to demonstrate various methods of profiling and analysis to determine the bottleneck, and the effectiveness of the tuning to improve on the application performance."
Watch the video presentation: http://wp.me/p3RLHQ-f8h
Learn more: http://www.hpcadvisorycouncil.com/best_practices.php
See more talks from the Switzerland HPC Conference:
http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
In this video from the HPC User Forum in Santa Fe, Yoonho Park from IBM presents: IBM Datacentric Servers & OpenPOWER.
"Big data analytics, machine learning and deep learning are among the most rapidly growing workloads in the data center. These workloads have the compute performance requirements of traditional technical computing or high performance computing, coupled with a much larger volume and velocity of data."
Watch the video: http://wp.me/p3RLHQ-gJv
Learn more: https://openpowerfoundation.org/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
This is the presentation on clusters computing which includes information from other sources too including my own research and edition. I hope this will help everyone who required to know on this topic.
Optimising nfv service chains on open stack using dockerAnanth Padmanabhan
Uploading slides presented in the OpenStack summit, at Austin in April, 2016. Here is the link to the video,
https://www.openstack.org/videos/video/optimising-nfv-service-chains-on-openstack-using-docker
Christian Kniep presented this deck at the 2016 HPC Advisory Council Switzerland Conference.
"With Docker v1.9 a new networking system was introduced, which allows multi-host network- ing to work out-of-the-box in any Docker environment. This talk provides an introduction on what Docker networking provides, followed by a demo that spins up a full SLURM cluster across multiple machines. The demo is based on QNIBTerminal, a Consul backed set of Docker Images to spin up a broad set of software stacks."
Watch the video presentation:
http://wp.me/p3RLHQ-f7G
See more talks in the Swiss Conference Video Gallery:
http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter:
http://insidehpc.com/newsletter
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
April 2016 HUG: The latest of Apache Hadoop YARN and running your docker apps...Yahoo Developer Network
Apache Hadoop YARN is a modern resource-management platform that handles resource scheduling, isolation and multi-tenancy for a variety of data processing engines that can co-exist and share a single data-center in a cost-effective manner.
In the first half of the talk, we are going to give a brief look into some of the big efforts cooking in the Apache Hadoop YARN community.
We will then dig deeper into one of the efforts - supporting Docker runtime in YARN. Docker is an application container engine that enables developers and sysadmins to build, deploy and run containerized applications. In this half, we'll discuss container runtimes in YARN, with a focus on using the DockerContainerRuntime to run various docker applications under YARN. Support for container runtimes (including the docker container runtime) was recently added to the Linux Container Executor (YARN-3611 and its sub-tasks). We’ll walk through various aspects of running docker containers under YARN - resource isolation, some security aspects (for example container capabilities, privileged containers, user namespaces) and other work in progress features like image localization and support for different networking modes.
Speakers:
Vinod Kumar Vavilapalli is the Hadoop YARN and MapReduce guy at Hortonworks. He is a long term Hadoop contributor at Apache, Hadoop committer and a member of the Apache Hadoop PMC. He has a Bachelors degree from Indian Institute of Technology Roorkee in Computer Science and Engineering. He has been working on Hadoop for nearly 9 years and he still has fun doing it. Straight out of college, he joined the Hadoop team at Yahoo! Bangalore, before Hortonworks happened. He is passionate about using computers to change the world for better, bit by bit.
Sidharta Seethana is a software engineer at Hortonworks. He works on the YARN team, focussing on bringing new kinds of workloads to YARN. Prior to joining Hortonworks, Sidharta spent 10 years at Yahoo! Inc., working on a variety of large scale distributed systems for core platforms/web services, search and marketplace properties, developer network and personalization.
Teaching Apache Spark: Demonstrations on the Databricks Cloud PlatformYao Yao
Yao Yao Mooyoung Lee
https://github.com/yaowser/learn-spark/tree/master/Final%20project
https://www.youtube.com/watch?v=IVMbSDS4q3A
https://www.slideshare.net/YaoYao44/teaching-apache-spark-demonstrations-on-the-databricks-cloud-platform/
Apache Spark is a fast and general engine for big data analytics processing with libraries for SQL, streaming, and advanced analytics
Cloud Computing, Structured Streaming, Unified Analytics Integration, End-to-End Applications
In this deck, Gilad Shainer from Mellanox announces the world’s first HDR 200Gb/s data center interconnect solutions. "These 200Gb/s HDR InfiniBand solutions maintain Mellanox’s generation-ahead leadership while enabling customers and users to leverage an open, standards-based technology that maximizes application performance and scalability while minimizing overall data center total cost of ownership. Mellanox 200Gb/s HDR solutions will become generally available in 2017.
Watch the video presentation: http://insidehpc.com/2016/11/hdr-infiniband/
This is a presentation by Prof. Anne Elster at the International Workshop on Open Source Supercomputing held in conjunction with the 2017 ISC High Performance Computing Conference.
In this deck from the HPC User Forum in Austin, Yutaka Ishikawa from Riken AICS presents: Japan's post K Computer.
Watch the video presentation: http://wp.me/p3RLHQ-fJ6
Learn more: http://hpcuserforum.com
HPC and cloud distributed computing, as a journeyPeter Clapham
Introducing an internal cloud brings new paradigms, tools and infrastructure management. When placed alongside traditional HPC the new opportunities are significant But getting to the new world with micro-services, autoscaling and autodialing is a journey that cannot be achieved in a single step.
Christian Kniep presented this deck at the 2016 HPC Advisory Council Switzerland Conference.
"With Docker v1.9 a new networking system was introduced, which allows multi-host network- ing to work out-of-the-box in any Docker environment. This talk provides an introduction on what Docker networking provides, followed by a demo that spins up a full SLURM cluster across multiple machines. The demo is based on QNIBTerminal, a Consul backed set of Docker Images to spin up a broad set of software stacks."
Watch the video presentation:
http://wp.me/p3RLHQ-f7G
See more talks in the Swiss Conference Video Gallery:
http://insidehpc.com/2016-swiss-hpc-conference/
Sign up for our insideHPC Newsletter:
http://insidehpc.com/newsletter
Overview of the architecture, and benefits of Dell HPC Storage with Intel EE Lustre in High Performance Computing and Big Science workloads.
Presented by Andrew Underwood at the Melbourne Big Data User Group - January 2016.
Lustre is a trademark of Seagate Technology.
April 2016 HUG: The latest of Apache Hadoop YARN and running your docker apps...Yahoo Developer Network
Apache Hadoop YARN is a modern resource-management platform that handles resource scheduling, isolation and multi-tenancy for a variety of data processing engines that can co-exist and share a single data-center in a cost-effective manner.
In the first half of the talk, we are going to give a brief look into some of the big efforts cooking in the Apache Hadoop YARN community.
We will then dig deeper into one of the efforts - supporting Docker runtime in YARN. Docker is an application container engine that enables developers and sysadmins to build, deploy and run containerized applications. In this half, we'll discuss container runtimes in YARN, with a focus on using the DockerContainerRuntime to run various docker applications under YARN. Support for container runtimes (including the docker container runtime) was recently added to the Linux Container Executor (YARN-3611 and its sub-tasks). We’ll walk through various aspects of running docker containers under YARN - resource isolation, some security aspects (for example container capabilities, privileged containers, user namespaces) and other work in progress features like image localization and support for different networking modes.
Speakers:
Vinod Kumar Vavilapalli is the Hadoop YARN and MapReduce guy at Hortonworks. He is a long term Hadoop contributor at Apache, Hadoop committer and a member of the Apache Hadoop PMC. He has a Bachelors degree from Indian Institute of Technology Roorkee in Computer Science and Engineering. He has been working on Hadoop for nearly 9 years and he still has fun doing it. Straight out of college, he joined the Hadoop team at Yahoo! Bangalore, before Hortonworks happened. He is passionate about using computers to change the world for better, bit by bit.
Sidharta Seethana is a software engineer at Hortonworks. He works on the YARN team, focussing on bringing new kinds of workloads to YARN. Prior to joining Hortonworks, Sidharta spent 10 years at Yahoo! Inc., working on a variety of large scale distributed systems for core platforms/web services, search and marketplace properties, developer network and personalization.
Teaching Apache Spark: Demonstrations on the Databricks Cloud PlatformYao Yao
Yao Yao Mooyoung Lee
https://github.com/yaowser/learn-spark/tree/master/Final%20project
https://www.youtube.com/watch?v=IVMbSDS4q3A
https://www.slideshare.net/YaoYao44/teaching-apache-spark-demonstrations-on-the-databricks-cloud-platform/
Apache Spark is a fast and general engine for big data analytics processing with libraries for SQL, streaming, and advanced analytics
Cloud Computing, Structured Streaming, Unified Analytics Integration, End-to-End Applications
In this deck, Gilad Shainer from Mellanox announces the world’s first HDR 200Gb/s data center interconnect solutions. "These 200Gb/s HDR InfiniBand solutions maintain Mellanox’s generation-ahead leadership while enabling customers and users to leverage an open, standards-based technology that maximizes application performance and scalability while minimizing overall data center total cost of ownership. Mellanox 200Gb/s HDR solutions will become generally available in 2017.
Watch the video presentation: http://insidehpc.com/2016/11/hdr-infiniband/
This is a presentation by Prof. Anne Elster at the International Workshop on Open Source Supercomputing held in conjunction with the 2017 ISC High Performance Computing Conference.
In this deck from the HPC User Forum in Austin, Yutaka Ishikawa from Riken AICS presents: Japan's post K Computer.
Watch the video presentation: http://wp.me/p3RLHQ-fJ6
Learn more: http://hpcuserforum.com
HPC and cloud distributed computing, as a journeyPeter Clapham
Introducing an internal cloud brings new paradigms, tools and infrastructure management. When placed alongside traditional HPC the new opportunities are significant But getting to the new world with micro-services, autoscaling and autodialing is a journey that cannot be achieved in a single step.
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Accelerate Big Data Processing with High-Performance Computing TechnologiesIntel® Software
Learn about opportunities and challenges for accelerating big data middleware on modern high-performance computing (HPC) clusters by exploiting HPC technologies.
OS for AI: Elastic Microservices & the Next Gen of MLNordic APIs
AI has been a hot topic lately, with advances being made constantly in what is possible, there has not been as much discussion of the infrastructure and scaling challenges that come with it. How do you support dozens of different languages and frameworks, and make them interoperate invisibly? How do you scale to run abstract code from thousands of different developers, simultaneously and elastically, while maintaining less than 15ms of overhead?
At Algorithmia, we’ve built, deployed, and scaled thousands of algorithms and machine learning models, using every kind of framework (from scikit-learn to tensorflow). We’ve seen many of the challenges faced in this area, and in this talk I’ll share some insights into the problems you’re likely to face, and how to approach solving them.
In brief, we’ll examine the need for, and implementations of, a complete “Operating System for AI” – a common interface for different algorithms to be used and combined, and a general architecture for serverless machine learning which is discoverable, versioned, scalable and sharable.
Accelerating TensorFlow with RDMA for high-performance deep learningDataWorks Summit
Google’s TensorFlow is one of the most popular deep learning (DL) frameworks. In distributed TensorFlow, gradient updates are a critical step governing the total model training time. These updates incur a massive volume of data transfer over the network.
In this talk, we first present a thorough analysis of the communication patterns in distributed TensorFlow. Then we propose a unified way of achieving high performance through enhancing the gRPC runtime with Remote Direct Memory Access (RDMA) technology on InfiniBand and RoCE. Through our proposed RDMA-gRPC design, TensorFlow only needs to run over the gRPC channel and gets the optimal performance. Our design includes advanced features such as message pipelining, message coalescing, zero-copy transmission, etc. The performance evaluations show that our proposed design can significantly speed up gRPC throughput by up to 1.5x compared to the default gRPC design. By integrating our RDMA-gRPC with TensorFlow, we are able to achieve up to 35% performance improvement for TensorFlow training with CNN models.
Speakers
Dhabaleswar K (DK) Panda, Professor and University Distinguished Scholar, The Ohio State University
Xiaoyi Lu, Research Scientist, The Ohio State University
Red Hat multi-cluster management & what's new in OpenShiftKangaroot
More and more organisations are not only using container platforms but starting to run multiple clusters of containers. And with that comes new headaches of maintaining, securing, and updating those multiple clusters. In this session we'll look into how Red Hat has solved multi-cluster management, covering cluster lifecycle, app lifecycle, and governance/risk/compliance.
SDN 101: Software Defined Networking Course - Sameh Zaghloul/IBM - 2014SAMeh Zaghloul
Sameh Zaghloul
Technology Manager @ IBM
+2 0100 6066012
zaghloul@eg.ibm.com
SDN: Technology that enables data center team to use software to efficiently control network resources
SDN Overview
SDN Standards
NFV – Network Function Virtualization
SDN Scenarios and Use Cases
SDN Sample Research Projects
SDN Technology Survey
SDN Case Study
SDN Online Courses
SDN Lab SW Tools
- OpenStack Framework
- OpenDayLighyt – SDN Controller
- FloodLight – SDN Controller
- Open vSwitch – Virtual Switch
- MiniNet – Virtual Network: OpenFlow Switches, SDN Controllers, and Servers/Hosts
- OMNet++ Network Simulator
- Avior – Sample FloodLight Java Application
- netem - Network Emulation
- NOX/POX - C++/ Python OpenFlow API for building network control applications
- Pyretic = Python + Frenetic - Enables network programmers and operators to write modular network applications by providing powerful abstractions
- Resonance - Event-Driven Control for Software-Defined Networks (written in Pyretic)
SDN Project
Similar to The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, University of Melbourne (20)
Swinburne University of Technology - Shunde Zhang & Kieran Spear, AptiraOpenStack
We recently teamed up with our good friends at SUSE to build a very high-performing and scalable storage landscape at a fraction of the cost than with traditional storage systems for the Swinburne University of Technology.
The challenge that the IT team at Swinburne were facing is how to accommodate the ever-growing need for performance and capacity while sticking to a tight budget. With SUSE Enterprise Storage we have been able to deploy a compelling and affordable solution to support Swinburne’s storage needs. Their new storage platform is fast and flexible, meaning the IT team at Swinburne can support the needs of researchers more effectively.
https://aptira.com/swinburne-university-technology/
Related OSS Projects - Peter Rowe, Flexera SoftwareOpenStack
Audience Level
Intermediate
Synopsis
Today’s fast-paced development environment has changed the compliance landscape. Many software projects consist of more than 50% Open Source Software (OSS) components, but as much as 99% are undocumented, increasing the complexities of managing your company’s software compliance process.
Of particular concern is “Zombie software”, or software that is outdated and contains vulnerable versions of certain components. Zombies can live in your code forever if you’re not aware of them. The acceleration of modern development lifecycles and the breakdown of an undocumented software supply chain have opened up new pathways for zombies to enter your software – leaving you exposed to security threats.
This presentation discusses best practices for implementing an Open Source Software management strategy that covers common pitfalls and commercial licence issues as well as the optimal way to track and eliminate the risks associated with Zombies!
Speaker Bio:
Involved in and around IT development for over 20 years, starting as a web developer using NotePad in 1995 when the most exciting thing online was Sun’s animated Java coffee cup, through Numega Pre-Sales selling BoundsChecker and now into the brave, new World of Open Source and software composition analysis.
Supercomputing by API: Connecting Modern Web Apps to HPCOpenStack
Audience Level
Intermediate
Synopsis
The traditional user experience for High Performance Computing (HPC) centers around the command line, and the intricacies of the underlying hardware. At the same time, scientific software is moving towards the cloud, leveraging modern web-based frameworks, allowing rapid iteration, and a renewed focus on portability and reproducibility. This software still has need for the huge scale and specialist capabilities of HPC, but leveraging these resources is hampered by variation in implementation between facilities. Differences in software stack, scheduling systems and authentication all get in the way of developers who would rather focus on the research problem at hand. This presentation reviews efforts to overcome these barriers. We will cover container technologies, frameworks for programmatic HPC access, and RESTful APIs that can deliver this as a hosted solution.
Speaker Bio
Dr. David Perry is Compute Integration Specialist at The University of Melbourne, working to increase research productivity using cloud and HPC. David chairs Australia’s first community-owned wind farm, Hepburn Wind, and is co-founder/CTO of BoomPower, delivering simpler solar and battery purchasing decisions for consumers and NGOs.
Federation and Interoperability in the Nectar Research CloudOpenStack
Audience Level
Beginner
Synopsis
The Nectar Research Cloud provides an OpenStack cloud for Australia’s academic researchers. Since its inception in 2012 it has grown steadily to over 30,000 CPUs, with over 10,000 registered users from more than 50 research institutions. It is different to many clouds in being a federation across eight organisations, each of which runs cloud infrastructure in one or more data centres and contributes to a distributed help desk and user support. A Nectar core services team runs centralised cloud services. This presentation will give an overview of the experiences, challenges and benefits of running a federated OpenStack cloud and a short demonstration on using the Nectar cloud. We will also describe some current approaches that are looking to extend this federation to encompass other institutions including some in New Zealand, to extend the infrastructure using commercial cloud providers, and to move towards interoperability with the growing number of international science and research clouds through the new Open Research Cloud initiative.
Speaker Bio
Dr Paul Coddington is a Deputy Director of Nectar, responsible for the Nectar national Research Cloud, and also Deputy Director of eResearch SA. He has over 30 years experience in eResearch including computational science, high performance and distributed computing, cloud computing, software development, and research data management.
I am text block. Click edit button to change this text. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.
Audience Level
Intermediate
Synopsis
In this presentation, Shunde will show you how to simplify the migration process with a workload migration engine, making the move to OpenStack easy. This talk will address the various difficulties operators and administrators face when migrating workloads and resources between various cloud platforms, including removing time consuming, repetitive and complicated steps.
This tool can be applied to many cloud migrations, including between Virtual Machines and OpenStack, between Public and Private clouds, as well as between OpenStack and OpenStack. This tool integrates completely with other OpenStack projects minimising deployment and maintenance efforts. So whether you’re looking to upgrade from your existing traditional virtualisation platform, setup a new OpenStack instance, or upgrade to a newer version of OpenStack, we will show you how to simplify this process using GUTS.
Speaker Bio
Shunde is a senior software developer in Aptira with over 15 years experience in software development, automation and system administration. He has worked with OpenStack since the Diablo cycle and has been involved in projects from OpenStack infrastructure to distributed systems running on top of OpenStack.
Hyperconverged Cloud, Not just a toy anymore - Andrew Hatfield, Red HatOpenStack
Audience Level
Intermediate
Synopsis
Hypercoverged Compute, Network and Storage is ready for production workloads – where it makes sense.
Whether you’re a telecommunications carrier, service provider or enterprise; implementing Network Function Virtualisation (NFV), focusing on specific known workloads or simply a dev / test cloud – deploying a hypercoverged OpenStack cloud makes a lot of sense.
Come along and discover which workloads fit a hyperconverged architecture, see examples and look into the very near future and learn how OpenStack is truly ready to serve your every need.
Speaker Bio
Andrew has over 20 years experience in the IT industry across APAC, specialising in Databases, Directory Systems, Groupware, Virtualisation and Storage for Enterprise and Government organisations. When not helping customers slash costs and increase agility by moving to the software-defined future, he’s enjoying the subtle tones of Islay Whisky and shredding pow pow on the world’s best snowboard resorts.
Migrating your infrastructure to OpenStack - Avi Miller, OracleOpenStack
Audience Level
Beginner
Synopsis
Migrating is never simple, but migrating from a traditional infrastructure to a private cloud infrastructure adds a whole new layer of complexity and raises a number of questions for IT decision makers. Come learn first hand how to begin to migrate your traditional infrastructure management tools and processes to OpenStack.
This session will provide details on common questions and answers to help administrators avoid costly mistakes. Learn what to look out for, what to avoid, how to identify risks and how to mitigate them.
Speaker Bio:
Avi is an accomplished technical product manager with extensive experience across the operating system, virtualisation and application stacks.
A glimpse into an industry Cloud using Open Source Technologies - Adrian Koh,...OpenStack
Audience Level
All levels
Synopsis
Often times, prospects and existing OpenStack users wonder if there is indeed strong business and technical value proposition for cloud platforms serving a specific industry vertical.
In this session, EasyStack would like to share with the participants our experience engaging with an industry leader to build a credible solution platform catering to their current and future business and technology roadmap.
Speaker Bio:
Adrian is Director Global Business Development in EasyStack and has 20 years working experience in leading tech companies in the IT industry.
Prior to joining EasyStack, Adrian was with IBM Singapore and IBM China and served in roles such as Offering Manager, Engagement Manager, Solution Architect, Services Consultant, IT Specialist.
Adrian holds a Bachelor of Science degree with honors in Computer Sciences from University of Texas at Austin.
Enabling OpenStack for Enterprise - Tarso Dos Santos, VeritasOpenStack
Audience Level
All levels
Synopsis
OpenStack offers many advantages for organisations building out their cloud environments, including flexibility and community-driven innovation. However, enterprises looking to deploy OpenStack in production typically find its storage management capabilities wanting from the perspective of management complexity and business resiliency. Enterprises are also challenged when it comes to ensuring protection of their data and providing the necessary performance – especially for their tier one applications. Meeting these fundamental needs is critical for enterprises to proceed confidently with their OpenStack deployments.
Veritas HyperScale for OpenStack is a software-defined storage management solution uniquely developed for OpenStack based clouds. It leverages direct attached storage (DAS) and provides enterprise-strength capabilities that enable robust, production-scale deployment while meeting performance and data protection needs. Learn how this innovative solution, coupled with other relevant Veritas offerings, solve the remaining issues around implementing OpenStack within the enterprise.
Speaker Bio:
Tarso dos Santos works as a Technical Account Manager at Veritas, directly engaging with customers to develop strategies, architectures and solutions with focus on Cloud – Openstack, Containers, Data Protection, High Availability and Compliance.
He has over 21 years in the IT industry architecting, delivering and positioning solutions such as private clouds, distributed systems, hpc, storage, and high available platforms.
Tarso has a great interest in distributed systems performance, and scientific organizations that push the boundaries of existing technologies, but also need to link these into the Enterprise.
Tarso in his life has enjoyed working in some of the most amazing projects ranging from mission critical systems protecting Australian lives, to IT infrastructure projects that are looking at the sky and discovering new planets out in the space.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Understanding blue store, Ceph's new storage backend - Tim Serong, SUSEOpenStack
Audience Level
Intermediate
Synopsis
Ceph – the most popular storage solution for OpenStack – stores all data as a collection of objects. This object store was originally implemented on top of a POSIX filesystem, an approach that turned out to have a number of problems, notably with performance and complexity.
BlueStore, a new storage backend for Ceph, was created to solve these issues; the Ceph Jewel release included an early prototype. The code and on-disk format were declared stable (but experimental) for Ceph Kraken, and now in the upcoming Ceph Luminous release, BlueStore will be the recommended default storage backend.
With a 2-3x performance boost, you’ll want to look at migrating your Ceph clusters to BlueStore. This talk goes into detail about what BlueStore does, the problems it solves, and what you need to do to use it.
Speaker Bio:
Tim works for SUSE, hacking on Ceph and related technologies. He has spoken often about distributed storage and high availability at conferences such as linux.conf.au. In his spare time he wrangles pigs, chickens, sheep and ducks, and was declared by one colleague “teammate most likely to survive the zombie apocalypse”.
OpenStack Networks the Web-Scale Way - Scott Laffer, Cumulus NetworksOpenStack
Audience Level
Beginner
Synopsis
Layer 2 versus Layer 3, MLAG, Spanning-Tree, switch mechanism drivers, overlays and routing-on-the-host — What scales and what does not? The underlying plumbing of an OpenStack network is something you’d rather not have to think about. This presentation examines the network architectures of web-scale and large enterprise OpenStack users and how those same efficiencies can be used in deployments of all sizes.
Speaker Bio:
Scott is a Member of Technical Staff at Cumulus Networks where he designs, supports and deploys web-scale technologies and architectures in enterprise networks globally. Prior to becoming a founding member of the Cumulus office in Australia, Scott started his career as a network administrator before joining Cisco Systems to support their data centre products.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Diving in the desert: A quick overview into OpenStack Sahara capabilities - A...OpenStack
Audience Level
Intermediate
Synopsis
Data Analytics is a hot topic today for most organisations as they race to convert vast amounts of data into useful information that can be leveraged to make critical decisions or recommendations in a very limited time window.
Today, there is a widely accepted talent gap when it comes to creating and managing Hadoop Clusters. Even for the experts can take hours (or days) to get a fully functional Hadoop farm up and running.
On top of that it can be difficult to find java programmers that have enough experience to be productive with Map Reduce.
OpenStack Sahara is looking to address most of this challenges by facilitating the deployment of Hadoop clusters and provide a set of API to provide data processing tasks.
This session will provide an insight into OpenStack Sahara capabilities and how the end users can leverage on it.
Speaker Bio:
Alex has been working with Open Source enterprise technologies for the better part of his 15 years IT career in companies like Hewlett-Packard Enterprise, Red Hat, IBM and Sun Microsystems.
He started his OpenStack journey with Grizzly, delivering the first HPC cloud in APAC for a Singapore University making use of SRIOV technologies combined with big data. He has extensive deployment experience on configuration management and automation of private cloud based on OpenStack.
Alex is currently an APJ Cloud Consultant in the Helion Cloud team at Hewlett-Packard Enterprise, where he evangelizes the OpenSource side of the Helion portfolio (OpenStack / Docker / Ceph).
He enjoys running automation workshops and seminars in the APJ region for cloud adopters.
Building a GPU-enabled OpenStack Cloud for HPC - Blair Bethwaite, Monash Univ...OpenStack
Audience Level
Intermediate
Synopsis
M3 is the latest generation system of the MASSIVE project, an HPC facility specializing in characterization science (imaging and visualization). Using OpenStack as the compute provisioning layer, M3 is a hybrid HPC/cloud system, custom-integrated by Monash’s R@CMon Research Cloud team. Built to support Monash University’s next-gen high-throughput instrument processing requirements, M3 is half-half GPU-accelerated and CPU-only.
We’ll discuss the design and tech used to build this innovative platform as well as detailing approaches and challenges to building GPU-enabled and HPC clouds. We’ll also discuss some of the software and processing pipelines that this system supports and highlight the importance of tuning for these workloads.
Speaker Bio
Blair Bethwaite: Blair has worked in distributed computing at Monash University for 10 years, with OpenStack for half of that. Having served as team lead, architect, administrator, user, researcher, and occasional hacker, Blair’s unique perspective as a science power-user, developer, and system architect has helped guide the evolution of the research computing engine central to Monash’s 21st Century Microscope.
Lance Wilson: Lance is a mechanical engineer, who has been making tools to break things for the last 20 years. His career has moved through a number of engineering subdisciplines from manufacturing to bioengineering. Now he supports the national characterisation research community in Melbourne, Australia using OpenStack to create HPC systems solving problems too large for your laptop.
OpenStack and Red Hat: How we learned to adapt with our customers in a maturi...OpenStack
Audience Level
All levels
Synopsis
Peter has been involved in OpenStack community since its B-release, and he has been enabling and helping customers across various industries adopt OpenStack in strategic ways. In this session, you will learn from his experience what Red Hat’s perspective is on the current state of affairs in the OpenStack community and the path we see ahead that Red Hat is putting its efforts in. OpenStack is not a product that tries to solve any one business problem in particular, but a technology that aims to be usable for many – what are the required steps to make sure that your organisation is ready for the OpenStack-based cloudification and transformation.
Speaker Bio:
Peter Jung is a Senior Business Development Manager at Red Hat where he leads the practice in the areas of Cloud, SDN/NFV and IoT across Australia and New Zealand. He is passionate about open innovation and open source software development model as the foundation for next generation society and ICT systems. Prior to Red Hat, he had various roles at Cisco and Dell for 15 years. He holds a BSEE and an MBA.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Meshing OpenStack and Bare Metal Networks with EVPN - David Iles, Mellanox Te...OpenStack
Audience Level
Intermediate
Synopsis
The latest SDN revolution is centered on creating efficient virtualized data center networks using VXLAN & EVPN. We will talk about the scale, performance, and cost advantages of using a modern controller-free virtualized network solution built on 100 Gigabit Ethernet switches with hardware based VXLAN Routing. We will explore the ease of automating such a network in an OpenStack environment and take you through a real world use case of using OpenStack Network Node bridging between a bare metal cloud (EVPN) and a fully virtualized cloud environments (orchestrated by Neutron).
Speaker Bio:
David has held leadership roles at 3COM, Cisco Systems, Nortel Networks, and IBM where he promoted advanced network technologies including High Speed Ethernet, Layer 4-7 switching, Virtual Machine-aware networking, and Software Defined Networking.
David’s current focus is on the evolving landscape of data center networking, scale out storage, Open Networking, and cloud computing.
Audience Level
All levels
Synopsis
Our journey towards solving our Application and Infrastructure Problems using Immutability, Codification, Mesos, Docker and Ironic.
OpenStack Australia Day Melbourne 2017
https://events.aptira.com/openstack-australia-day-melbourne-2017/
Traditional Enterprise to OpenStack Cloud - An Unexpected JourneyOpenStack
Audience Level
Intermediate
Synopsis
Hostworks is a part of the Inabox group and an Australian based Managed Hosting and Services Provider with a strong background in management of infrastructure, servers, and applications following Traditional Enterprise management practices. 2 years ago, we began our transition to adopt and deploy cloud technologies and mindsets with an on On-Premises OpenStack platform as part of a larger hybrid cloud offering. This presentation details how we did it, what went wrong, and what went right, important lessons and recommendations for other businesses who wish to follow the same path.
Speaker Bio:
Daniel is Platform Engineer at Hostworks and specialises in their On-Premises OpenStack cloud offering.
Building a GPU-enabled OpenStack Cloud for HPC - Lance Wilson, Monash UniversityOpenStack
Audience Level
Intermediate
Synopsis
M3 is the latest generation system of the MASSIVE project, an HPC facility specializing in characterization science (imaging and visualization). Using OpenStack as the compute provisioning layer, M3 is a hybrid HPC/cloud system, custom-integrated by Monash’s R@CMon Research Cloud team. Built to support Monash University’s next-gen high-throughput instrument processing requirements, M3 is half-half GPU-accelerated and CPU-only.
We’ll discuss the design and tech used to build this innovative platform as well as detailing approaches and challenges to building GPU-enabled and HPC clouds. We’ll also discuss some of the software and processing pipelines that this system supports and highlight the importance of tuning for these workloads.
Speaker Bio
Blair Bethwaite: Blair has worked in distributed computing at Monash University for 10 years, with OpenStack for half of that. Having served as team lead, architect, administrator, user, researcher, and occasional hacker, Blair’s unique perspective as a science power-user, developer, and system architect has helped guide the evolution of the research computing engine central to Monash’s 21st Century Microscope.
Lance Wilson: Lance is a mechanical engineer, who has been making tools to break things for the last 20 years. His career has moved through a number of engineering subdisciplines from manufacturing to bioengineering. Now he supports the national characterisation research community in Melbourne, Australia using OpenStack to create HPC systems solving problems too large for your laptop.
Monitoring Uptime on the NeCTAR Research Cloud - Andy Botting, University of ...OpenStack
Audience Level
Intermediate
Synopsis
We will discuss how we do monitoring on the Nectar research cloud, utilising tools like OpenStack tempest, Nagios and translating this into a user facing dashboard.
Speaker Bio:
Andy is a DevOps engineer working at the University of Melbourne in the Core Services team for the Nectar Research Cloud.
Containers and OpenStack: Marc Van Hoof, Kumulus: Containers and OpenStackOpenStack
Containers and OpenStack
Audience: Intermediate
Topic: Infrastructure
Abstract: Containers are the new darling of the development world, and many are calling for an end of the IaaS world. But there are still key reasons that IaaS is important even as Container based development becomes the desired path for the development community. We will review containers in the context of their growth in popularity, and look at how OpenStack both continues to support and enable Container solutions, and the latest developments in OpenStack as a containerized solution directly.
Speaker Bio: Marc Van Hoof, Kumulus
Marc van Hoof has been in the technology industry for over 20 years, focused on developing, deploying, and scaling internet applications. He was part of a team that built the first internet data centre in Australia, has worked on some of the largest online real-time events, and advises companies on how to take advantage of the true benefits of migrating to the cloud.
OpenStack Australia Day Government - Canberra 2016
https://events.aptira.com/openstack-australia-day-canberra-2016/
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
JMeter webinar - integration with InfluxDB and Grafana
The Why and How of HPC-Cloud Hybrids with OpenStack - Lev Lafayette, University of Melbourne
1. The Why and How of HPC-Cloud Hybrids
with OpenStack
OpenStack Australia Day Melbourne
June, 2017
Lev Lafayette, HPC Support and Training Officer, University of Melbourne
lev.lafayette@unimelb.edu.au
2. 1.0 Management Layer
1.1 HPC for performance
High-performance computing (HPC) is any computer system whose architecture allows for
above average performance. Main use case refers to compute clusters with a teamster
separation between head node and workers nodes and a high-speed interconnect acts as a
single system.
1.2 Clouds for flexibility.
Precursor with virtualised hardware. Cloud VMs always have lower performance than HPC.
Question is whether the flexibility is worth the overhead.
1.3 Hybrid HPC/Clouds.
University of Melbourne model, "the chimera". Cloud VMs deployed as HPC nodes and the
Freiburg University Model, "the cyborg", HPC nodes deploying Cloud VMs.
1.4 Reviewing user preferences and usage.
Users always want more of 'x'; real issue identified was queue times. Usage indicated a high
proportion of single-node jobs.
1.5 Review and Architecture.
Review discussed whether UoM needed HPC; architecture was to use existing NeCTAR
Research cloud with an expansion of general cloud compute provisioning and use of a smaller
"true HPC" system on bare metal nodes.
3. 2.0 Physical Layer
2.1 Physical Partitions.
"Real" HPC is a mere c276 cores, 21 GB per core. 2 socket Intel E5-2643 v3 E5-2643,
3.4GHz CPU with 6-core per socket, 192GB memory, 2x 1.2TB SAS drives, 2x 40GbE
network. “Cloud” partitions is almost 400 virtual machines with over 3,000 2.3GHz Haswell
cores with 8GB per core and . There is also a GPU partition with Dual Nvidia Tesla K80s (big
expansion this year), and departmental partitions (water and ashley). Management and login
nodes are VMs as is I/O for transferring data.
2.2 Network.
System network includes: Cloud nodes Cisco Nexus 10Gbe TCP/IP 60 usec latency (mpi-
pingpong); Bare Metal Mellanox 2100 Cumulos Linux 40Gbe TCP/IP 6.85 usec latency and
then RDMA Ethernet 1.15 usec latency
2.3 Storage.
Mountpoints to home, projects (/project /home for user data & scripts, NetApp SAS aggregate
70TB usable) and applications directories across all nodes. Additional mountpoins to VicNode
Aspera Shares. Applications directory currently on management node, needs to be decoupled.
Bare metal nodes have /scratch shared storage for MPI jobs (Dell R730 with 14 x 800GB
mixed use SSDs providing 8TB of usable storage, NFS over RDMA)., /var/local/tmp for single
node jobs, pcie SSD 1.6TB.
4. 3.0 Operating System and
Scheduler Layer
3.1 Red Hat Linux.
Scalable FOSS operating system, high performance, very well suited for research
applications. In November 2016 of the Top 500 Supercomputers worldwide, every single
machine used a "UNIX-like" operating system; and 99.6% used Linux.
3.2 Slurm Workload Manager.
Job schedulers and resource managers allow for unattended background tasks expressed as
batch jobs among the available resources; allows multicore, multinode, arrays, dependencies,
and interactive submissions. The scheduler provides for paramterisation of computer
resources, an automatic submission of execution tasks, and a notification system for incidents.
Slurm (originally Simple Linux Utility for Resource Management), developed by Lawrence
Livermore et al., is FOSS, and used by majority of world's top systems. Scalable, offers many
optional plugins, power-saving features, accounting features, etc. Divided into logical partitions
which correlate with hardware partitions.
3.3 Git, Gerrit, and Puppet.
Version control, paired systems administration, configuration management.
3.4 OpenStack Node Deployment.
Significant use of Nova (compute) service for provisioning and decommissioning of virtual
machines on demand.
5. 4.0 Application Layer
4.1 Source Code and EasyBuild.
Source code provides better control over security updates, integration, development, and
much better performance. Absolutely essential for reproducibility in research environment.
EasyBuild makes source software installs easier with scripts containing specified compilation
blocks (e.g., configuremake, cmake etc) and specified toolchains (GCC, Intel etc) and
environment modules (LMod). Modulefiles allow for dynamic changes to a user's environment
and ease with multiple versions of software applications on a system.
4.2 Compilers, Scripting Languages, and Applications.
Usual range of suspects; Intel and GCC, for compilers (and a little bit of PGI), Python Ruby,
and Perl for scripting languages, OpenMPI wrappers. Major applications include: MATLAB,
Gaussian, NAMD, R, OpenFOAM, Octave etc.
Almost 1,000 applications/versions installed from source, plus packages.
4.3 Containers with Singularity.
A container in a cloud virtual machine on an HPC! Wait, what?
6. 5.0 User Layer
5.1 Karaage.
Spartan uses its own LDAP authentication that is tied to the university Security Assertion
Markup Language (SAML). Users on Spartan must belong to a project. Projects must be led
by a University of Melbourne researcher (the "Principal Investigator") and are subject to
approval by the Head of Research Compute Services. Participants in a project can be
researchers or research support staff from anywhere. Karaage is Django-based application for
user, project, and cluster reporting and management.
5.2 Freshdesk.
OMG Users!
5.3 Online Instructions and Training.
Many users (even post-doctoral researchers) require basic training in Linux command line, a
requisite skill for HPC use. Extensive training programme for researchers available using
andragogical methods, including day-long courses in “Introduction to Linux and HPC Using
Spartan”, “Linux Shell Scripting for High Performance Computing”, and “Parallel Programming
On Spartan”.
Documentation online (Github, Website, and man pages) and plenty of Slurm examples on
system.
7. 6.0 Future Developments
6.1 Cloudbursting with Azure.
Slurm allows cloudbursting via the powersave feature; successfully experiments (and bug
discovery) within the NeCTAR research cloud.
About to add Azure through same login node. Does not mount applications directory; wrap
necessary data for transfer in script.
6.2 GPU Expansion.
Plans for a significant increase in the GPU allocation.
6.3 Test cluster (Thespian).
Everyone has a test environment, some people also have a production and a test
environment.
Test nodes already exist for Cloud and Physical partitions. Replicate management and login
nodes.
6.3 New Architectures
New architectures can be added to the system with separate build node (another VM) and with
software built for that architecture. Don't need an entirely new system.