Leadership Without Management: Scaling Organizations by Scaling Engineersbcantrill
My talk at Surge 2013. Video is at http://www.youtube.com/watch?v=bGkVM1B5NuI Caution: Should not be consumed by stack-ranking six-sigma black belts with fragile constitutions.
Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole
Many have gone before you along this path. Many have failed. A few have succeeded. All have scars. Although the journey is different for everyone there are common aspects to them all. In this talk we will cover our experiences in moving applications into the Cloud. What you must do. What you must not. What matters, what doesn’t.
In moving to the cloud there is no try.
In this talk:
* We’ll cover the core aspects of how the cloud differs from local data centers in terms of application design, runtime characteristics and operational considerations.
* We’ll explain through various real life examples where things worked and where they didnt
* We end with a summary of the key elements to success and the major pitfalls to avoid.
Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over Brian Troutwine
Building computer systems that are reliable is hard. The functional programming community has invested a lot of time and energy into up-front-correctness guarantees: types and the like. Unfortunately, absolutely correct software is time-consuming to write and expensive as a result. Fault-tolerant systems achieve system-total reliability by accepting that sub-components will fail and planning for that failure as a first-class concern of the system. As companies embrace the wave of "as-a-service" architectures, failure of sub-systems become a more pressing concern. Using examples from heavy industry, aeronautics and telecom systems, this talk will explore how you can design for fault-tolerance and how functional programming techniques get us most of the way there.
Leadership Without Management: Scaling Organizations by Scaling Engineersbcantrill
My talk at Surge 2013. Video is at http://www.youtube.com/watch?v=bGkVM1B5NuI Caution: Should not be consumed by stack-ranking six-sigma black belts with fragile constitutions.
Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole
Many have gone before you along this path. Many have failed. A few have succeeded. All have scars. Although the journey is different for everyone there are common aspects to them all. In this talk we will cover our experiences in moving applications into the Cloud. What you must do. What you must not. What matters, what doesn’t.
In moving to the cloud there is no try.
In this talk:
* We’ll cover the core aspects of how the cloud differs from local data centers in terms of application design, runtime characteristics and operational considerations.
* We’ll explain through various real life examples where things worked and where they didnt
* We end with a summary of the key elements to success and the major pitfalls to avoid.
Fault-tolerance on the Cheap: Making Systems That (Probably) Won't Fall Over Brian Troutwine
Building computer systems that are reliable is hard. The functional programming community has invested a lot of time and energy into up-front-correctness guarantees: types and the like. Unfortunately, absolutely correct software is time-consuming to write and expensive as a result. Fault-tolerant systems achieve system-total reliability by accepting that sub-components will fail and planning for that failure as a first-class concern of the system. As companies embrace the wave of "as-a-service" architectures, failure of sub-systems become a more pressing concern. Using examples from heavy industry, aeronautics and telecom systems, this talk will explore how you can design for fault-tolerance and how functional programming techniques get us most of the way there.
Thierry de Pauw - Feature Branching considered Evil - Codemotion Milan 2018Codemotion
With DVCSs branch creation became very easy, but it comes at a certain cost. Long living branches break the flow of the software delivery process, impacting stability and throughput. The session explores why teams are using feature branches, what problems are introduced by using them and what techniques exist to avoid them altogether. It explores exactly what's evil about feature branches, which is not necessarily the problems they introduce - but rather, the real reasons why teams are using them. After the session, you'll understand a different branching strategy and how it relates to CI/CD.
Talk originally given at FISL 2012 in Porto Alegre, Brazil. Video was on YouTube but regrettably taken down. Fortunately, I gave a slightly updated (and frankly, tighter and better produced) version of this at the Liferay Symposium in the fall of 2012: https://www.youtube.com/watch?v=Pm8P4oCIY3g
DSC UTeM DevOps Session#1: Intro to DevOps Presentation SlidesDSC UTeM
DevOps has been such a buzzword in the IT field nowadays. If you look into job postings, you might be surprised to find terms like "work with DevOps team", "work in an agile team" etc.
What is DevOps? What is agile? And why all these? 樂
Join us on 24 May 2021, where we have a short session to explore on the events that led to the trend nowadays
We will be exploring on the current trends, tech stacks and the existence of DevOps itself! 朗
Mark this date on your calendar and we'll see you there!
* Note: This is an introductory "brief overview" session that gives you context on our upcoming events.
Slides by KwongTN.
Research grants from the European Research Council (ERC) are great to have, but hard to get, In this talk, I give twelve personal tips that have been helpful for preparing your grant proposal.
Slides of a talk at INRIA Nancy, 20 December 2017
Being Elastic -- Evolving Programming for the CloudRandy Shoup
eBay Chief Engineer Randy Shoup's keynote at QCon 2010 outlines several critical elements of the evolving cloud programming model – what developers need to do to develop successful systems in the cloud. It discusses state management and statelessness, distribution- and network-awareness, workload partitioning, cost and resource metering, automation readiness, and deployment strategies
Thinking About Prototyping: Sketching, Familiarity, Costs versus Ease of Prototyping, Prototypes and Production, Changing Embedded Platform, Physical Prototypes and Mass Personalisation, Climbing into the Cloud, Open Source versus Closed Source, Why Closed? Why Open? Mixing Open and Closed Source, Closed Source for Mass Market Projects, Tapping into the Community. Prototyping Embedded Devices: Electronics, Sensors, Actuators, Scaling Up the Electronics, Embedded Computing Basics, Microcontrollers, System-on-Chips, Choosing Your Platform, Arduino, Developing on the Arduino, Some Notes on the Hardware, Openness, Raspberry Pi, Cases and Extension Boards, Developing on the Raspberry Pi, Some Notes on the Hardware, Openness.
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard ...MeasureWorks
For the Velocity Conference Europe 2012 workshop day this presentation is about the essentials for creation and building a Web Performance dashboard. This with ultimate goal of providing the audience a framework for designing and building a web performance dashboard. The session will cover the following 3 items:
Design guidelines: What defines a web performance dashboard? How to make sure it’s actionable and for people to actually use it on day to day basis?
Data collection: Why performance data? The various ways there are to collect data (e.g. synthetic versus RUM data, Webpagetest, Mobile) and how to correlate the different types of data and tools
Building the dashboard: How to build the actual dashboard, providing an overview of the tools/techniques used
At the end of the workshop you will be able to design and build your own dashboard based on the framework provided, or to optimize the current dashboards within your organization.
#ATAGTR2021 Presentation : "Chaos engineering: Break it to make it" by Anupa...Agile Testing Alliance
Interactive Session on "Chaos engineering: Break it to make it" by Anupam Agarwal,Nagarro, Peeyush Girdhar, Cloud / DevOps Nagarro. at #ATAGTR2021.
#ATAGTR2021 was the 6th Edition of Global Testing Retreat.
The video recording of the session is now available on the following link: https://www.youtube.com/watch?v=4bM4f8xNp2A
To know more about #ATAGTR2021, please visit:https://gtr.agiletestingalliance.org/
Building a Successful Organization By Mastering Failurejgoulah
The Etsy organization has grown by a significant amount over the last five years. As a company grows, more thought must be put into the techniques that it uses to communicate and deal with failures. This talk will cover several techniques that have helped foster a Just Culture, one in which an effort is made to balance both safety and accountability
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...tboubez
This is my presentation from LISA 2014 in Seattle on November 14, 2014.
Most IT Ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this haystack of data and extracting signal from the noise is not easy and generates too many false positives.
In this talk I will show some of the types of anomalies commonly found in dynamic data center environments and discuss the top 5 things I learned while building algorithms to find them. You will see how various Gaussian based techniques work (and why they don’t!), and we will go into some non-parametric methods that you can use to great advantage.
Thierry de Pauw - Feature Branching considered Evil - Codemotion Milan 2018Codemotion
With DVCSs branch creation became very easy, but it comes at a certain cost. Long living branches break the flow of the software delivery process, impacting stability and throughput. The session explores why teams are using feature branches, what problems are introduced by using them and what techniques exist to avoid them altogether. It explores exactly what's evil about feature branches, which is not necessarily the problems they introduce - but rather, the real reasons why teams are using them. After the session, you'll understand a different branching strategy and how it relates to CI/CD.
Talk originally given at FISL 2012 in Porto Alegre, Brazil. Video was on YouTube but regrettably taken down. Fortunately, I gave a slightly updated (and frankly, tighter and better produced) version of this at the Liferay Symposium in the fall of 2012: https://www.youtube.com/watch?v=Pm8P4oCIY3g
DSC UTeM DevOps Session#1: Intro to DevOps Presentation SlidesDSC UTeM
DevOps has been such a buzzword in the IT field nowadays. If you look into job postings, you might be surprised to find terms like "work with DevOps team", "work in an agile team" etc.
What is DevOps? What is agile? And why all these? 樂
Join us on 24 May 2021, where we have a short session to explore on the events that led to the trend nowadays
We will be exploring on the current trends, tech stacks and the existence of DevOps itself! 朗
Mark this date on your calendar and we'll see you there!
* Note: This is an introductory "brief overview" session that gives you context on our upcoming events.
Slides by KwongTN.
Research grants from the European Research Council (ERC) are great to have, but hard to get, In this talk, I give twelve personal tips that have been helpful for preparing your grant proposal.
Slides of a talk at INRIA Nancy, 20 December 2017
Being Elastic -- Evolving Programming for the CloudRandy Shoup
eBay Chief Engineer Randy Shoup's keynote at QCon 2010 outlines several critical elements of the evolving cloud programming model – what developers need to do to develop successful systems in the cloud. It discusses state management and statelessness, distribution- and network-awareness, workload partitioning, cost and resource metering, automation readiness, and deployment strategies
Thinking About Prototyping: Sketching, Familiarity, Costs versus Ease of Prototyping, Prototypes and Production, Changing Embedded Platform, Physical Prototypes and Mass Personalisation, Climbing into the Cloud, Open Source versus Closed Source, Why Closed? Why Open? Mixing Open and Closed Source, Closed Source for Mass Market Projects, Tapping into the Community. Prototyping Embedded Devices: Electronics, Sensors, Actuators, Scaling Up the Electronics, Embedded Computing Basics, Microcontrollers, System-on-Chips, Choosing Your Platform, Arduino, Developing on the Arduino, Some Notes on the Hardware, Openness, Raspberry Pi, Cases and Extension Boards, Developing on the Raspberry Pi, Some Notes on the Hardware, Openness.
MeasureWorks - Velocity Conference Europe 2012 - a Web Performance dashboard ...MeasureWorks
For the Velocity Conference Europe 2012 workshop day this presentation is about the essentials for creation and building a Web Performance dashboard. This with ultimate goal of providing the audience a framework for designing and building a web performance dashboard. The session will cover the following 3 items:
Design guidelines: What defines a web performance dashboard? How to make sure it’s actionable and for people to actually use it on day to day basis?
Data collection: Why performance data? The various ways there are to collect data (e.g. synthetic versus RUM data, Webpagetest, Mobile) and how to correlate the different types of data and tools
Building the dashboard: How to build the actual dashboard, providing an overview of the tools/techniques used
At the end of the workshop you will be able to design and build your own dashboard based on the framework provided, or to optimize the current dashboards within your organization.
#ATAGTR2021 Presentation : "Chaos engineering: Break it to make it" by Anupa...Agile Testing Alliance
Interactive Session on "Chaos engineering: Break it to make it" by Anupam Agarwal,Nagarro, Peeyush Girdhar, Cloud / DevOps Nagarro. at #ATAGTR2021.
#ATAGTR2021 was the 6th Edition of Global Testing Retreat.
The video recording of the session is now available on the following link: https://www.youtube.com/watch?v=4bM4f8xNp2A
To know more about #ATAGTR2021, please visit:https://gtr.agiletestingalliance.org/
Building a Successful Organization By Mastering Failurejgoulah
The Etsy organization has grown by a significant amount over the last five years. As a company grows, more thought must be put into the techniques that it uses to communicate and deal with failures. This talk will cover several techniques that have helped foster a Just Culture, one in which an effort is made to balance both safety and accountability
Five Things I Learned While Building Anomaly Detection Tools - Toufic Boubez ...tboubez
This is my presentation from LISA 2014 in Seattle on November 14, 2014.
Most IT Ops teams only keep an eye on a small fraction of the metrics they collect because analyzing this haystack of data and extracting signal from the noise is not easy and generates too many false positives.
In this talk I will show some of the types of anomalies commonly found in dynamic data center environments and discuss the top 5 things I learned while building algorithms to find them. You will see how various Gaussian based techniques work (and why they don’t!), and we will go into some non-parametric methods that you can use to great advantage.
Speaker:
Alex Cruise (Dir. Architecture, Metafor Software)
Abstract:
The rise of the DevOps movement has brought into welcome focus something that is often learned only through painful experience and expense: the success of a software product critically depends not only on its implementation, maintenance and enhancement, but also on how it’s deployed and operated.
Distributed systems are hard, but you can’t escape them: you need to scale out, but wrapping proxy interfaces around remote resources so they look local is a recipe for a fragile system. Plus, as the complexity of components and services increases, local systems aren’t actually as reliable as we think! Concurrency is hard, but you can’t escape it: whether you’re using threads in a single process, or multiple processes on a single machine, you still need to synchronize state between them somehow. Fault tolerance is hard, but you can’t escape it: parts will fail, you need to cope without rebooting the whole application. Correctness is hard, but you can’t escape it: whether through laborious testing or a Sufficiently Advanced Compiler, you need to have some assurance that the software will work as intended.
Let’s talk about a set of architectural patterns (and, yes, frameworks) that can really help us achieve the goals of concurrency, fault tolerance and correctness, while affording us the flexibility we need to scale our deployments when we achieve terrifying success.
Data centre analytics toufic boubez-metafor-dev ops days vancouver-2013-10-25tboubez
Vancouver DevOps Days
25 October 2013
IT Ops collect a ton of data and produce reams of graphs to monitor systems and applications. Getting the right signal out of all that noise however is getting tougher and tougher. The traditional techniques to deal with such metrics, whether threshold-based or very simple statistical methods that were developed to deal with stable, static manufacturing processes, are failing in today’s dynamic environment. Interest in applying more advanced analytics and machine learning to detect anomalies is gaining steam but understanding which algorithms should be used to identify and predict anomalies without producing more false positives is not so easy.
This talk will begin with a brief definition of the types of anomalies commonly found in dynamic data center environments and then discuss some of the key elements to consider when thinking about anomaly detection such as:
Understanding your data’s characteristics
The two main approaches for analyzing operations data: parametric and non-parametric methods
Overview of some current simple statistical methods and their weaknesses
Simple data transformations that can give you powerful results
By the end of this talk, attendees will understand the pros and cons of the key statistical analysis techniques and walk away with examples as well as practical rules of thumb and usage patterns.
Velocity Europe 2013: Beyond Pretty Charts: Analytics for the cloud infrastru...tboubez
My presentation from Velocity Europe 2013 in London: Beyond Pretty Charts…. Analytics for the cloud infrastructure.
IT Ops collect tons of data on the status of their data center or cloud environment. Much of that data ends up as graphs on big screens so ops folks can keep an eye on the behavior of their systems. But unless a threshold is crossed, behavioral issues will often fall through the cracks. Thresholds are reactive, and humans are, well, human. Applying analytics and machine learning to detect anomalies in dynamic infrastructure environments can catch these behavioral changes before they become critical.
Current tools used to monitor web environments rely on fundamental assumptions that are no longer true such as assuming that the underlying system being monitored is relatively static or that the behavioral limits of these systems can be defined by static rules and thresholds. Thus interest in applying analytics and machine learning to predict and detect anomalies in these dynamic environments is gaining steam. However, understanding which algorithms should be used to identify and predict anomalies accurately within all that data we generate is not so easy.
This talk will begin with a brief definition of the types of anomalies commonly found in dynamic data center environments and then discuss some of the key elements to consider when thinking about anomaly detection such as:
Understanding your data’s characteristics
The two main approaches for analyzing operations data: parametric and non-parametric methods
Simple data transformations that can give you powerful results
By the end of this talk, attendees will understand the pros and cons of the key statistical analysis techniques and walk away with examples as well as practical rules of thumb and usage patterns.
Beyond pretty charts, Analytics for the rest of us. Toufic Boubez DevOps Days...tboubez
Current monitoring tools are clearly reaching the limit of their capabilities. That's because these tools are based on fundamental assumptions that are no longer true such as assuming that the underlying system being monitored is relatively static or that the behavioral limits of these systems can be defined by static rules and thresholds. Interest in applying analytics and machine learning to detect anomalies in dynamic web environments is gaining steam. However, understanding which algorithms should be used to identify and predict anomalies accurately within all that data we generate is not so easy.
This talk builds on an Open Space discussion that was started at DevOps Days Austin. We will begin with a brief definition of the types of anomalies commonly found in dynamic data center environments and then discuss some of the key elements to consider when thinking about anomaly detection such as:
Understanding your data and the two main approaches for analyzing operations data: parametric and non-parametric methods
The importance of context
Simple data transformations that can give you powerful results
Monitoring Complex Systems - Chicago Erlang, 2014Brian Troutwine
Imagine being responsible for monitoring 100 servers. Now imagine 1000. Each server has 100 different things to keep track of. What do you pay attention to and what do you ignore? What is important? In this talk Brian will show how Erlang can be used to capture more information without compromising clarity — i.e. to keep track of the forest without loosing site of the trees!
Similar to The Hurricane's Butterfly: Debugging pathologically performing systems (20)
Talk given at the OCP Open System Firmware engineering workshop on 5/17/22. Talk was recorded; video at https://www.youtube.com/watch?v=eNI0wFgBNmY#t=7044s
Hardware/software Co-design: The Coming Golden Agebcantrill
Talk I gave as a keynote at RailsConf 2021. There is no Rails in the talk, though; this is all about the revolutions in open source firmware and hardware that are changing the way we build systems. Video to come!
Tockilator: Deducing Tock execution flows from Ibex Verilator tracesbcantrill
Talk given on March 20, 2020 at Oxidize 1K, a virtual conference that went from first idea to 300+ person conference in a week during the COVID-19 pandemic.
Platform values, Rust, and the implications for system softwarebcantrill
Talk given at Scale By The Bay 2018. Video is at https://www.youtube.com/watch?v=2wZ1pCpJUIM. If you are interested in this talk, you might also be interested in my talk on Platform as a Reflection of Values from Node Summit 2017: https://www.slideshare.net/bcantrill/platform-as-reflection-of-values-joyent-nodejs-and-beyond
My Papers We Love talk in San Francisco on October 12, 2017 on "ARC: A self-tuning, low overhead replacement cache." Video at https://www.youtube.com/watch?v=F8sZRBdmqc0
How Recreation Management Software Can Streamline Your Operations.pptxwottaspaceseo
Recreation management software streamlines operations by automating key tasks such as scheduling, registration, and payment processing, reducing manual workload and errors. It provides centralized management of facilities, classes, and events, ensuring efficient resource allocation and facility usage. The software offers user-friendly online portals for easy access to bookings and program information, enhancing customer experience. Real-time reporting and data analytics deliver insights into attendance and preferences, aiding in strategic decision-making. Additionally, effective communication tools keep participants and staff informed with timely updates. Overall, recreation management software enhances efficiency, improves service delivery, and boosts customer satisfaction.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
How to Position Your Globus Data Portal for Success Ten Good PracticesGlobus
Science gateways allow science and engineering communities to access shared data, software, computing services, and instruments. Science gateways have gained a lot of traction in the last twenty years, as evidenced by projects such as the Science Gateways Community Institute (SGCI) and the Center of Excellence on Science Gateways (SGX3) in the US, The Australian Research Data Commons (ARDC) and its platforms in Australia, and the projects around Virtual Research Environments in Europe. A few mature frameworks have evolved with their different strengths and foci and have been taken up by a larger community such as the Globus Data Portal, Hubzero, Tapis, and Galaxy. However, even when gateways are built on successful frameworks, they continue to face the challenges of ongoing maintenance costs and how to meet the ever-expanding needs of the community they serve with enhanced features. It is not uncommon that gateways with compelling use cases are nonetheless unable to get past the prototype phase and become a full production service, or if they do, they don't survive more than a couple of years. While there is no guaranteed pathway to success, it seems likely that for any gateway there is a need for a strong community and/or solid funding streams to create and sustain its success. With over twenty years of examples to draw from, this presentation goes into detail for ten factors common to successful and enduring gateways that effectively serve as best practices for any new or developing gateway.
Launch Your Streaming Platforms in MinutesRoshan Dwivedi
The claim of launching a streaming platform in minutes might be a bit of an exaggeration, but there are services that can significantly streamline the process. Here's a breakdown:
Pros of Speedy Streaming Platform Launch Services:
No coding required: These services often use drag-and-drop interfaces or pre-built templates, eliminating the need for programming knowledge.
Faster setup: Compared to building from scratch, these platforms can get you up and running much quicker.
All-in-one solutions: Many services offer features like content management systems (CMS), video players, and monetization tools, reducing the need for multiple integrations.
Things to Consider:
Limited customization: These platforms may offer less flexibility in design and functionality compared to custom-built solutions.
Scalability: As your audience grows, you might need to upgrade to a more robust platform or encounter limitations with the "quick launch" option.
Features: Carefully evaluate which features are included and if they meet your specific needs (e.g., live streaming, subscription options).
Examples of Services for Launching Streaming Platforms:
Muvi [muvi com]
Uscreen [usencreen tv]
Alternatives to Consider:
Existing Streaming platforms: Platforms like YouTube or Twitch might be suitable for basic streaming needs, though monetization options might be limited.
Custom Development: While more time-consuming, custom development offers the most control and flexibility for your platform.
Overall, launching a streaming platform in minutes might not be entirely realistic, but these services can significantly speed up the process compared to building from scratch. Carefully consider your needs and budget when choosing the best option for you.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Custom Healthcare Software for Managing Chronic Conditions and Remote Patient...Mind IT Systems
Healthcare providers often struggle with the complexities of chronic conditions and remote patient monitoring, as each patient requires personalized care and ongoing monitoring. Off-the-shelf solutions may not meet these diverse needs, leading to inefficiencies and gaps in care. It’s here, custom healthcare software offers a tailored solution, ensuring improved care and effectiveness.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
2. Debugging system failure
• Failures are easiest to debug when they are explicit and fatal
• A system that fails fatally stops: it ceases to make forward
progress, leaving behind a snapshot of its state — a core dump
• Unfortunately, these are not all problems…
• A broad class of problems are non-fatal: the system continues
to operate despite having failed, often destroying evidence
• Worst of all are those non-fatal failures that are also implicit
3. Implicit, non-fatal failure
• The most difficult, time-consuming bugs to debug are those in
which the system failure is unbeknownst to the system itself
• The system does the wrong thing or returns the wrong result or
has pathological side effects (e.g., resource leaks)
• Of these, the gnarliest class are those failures that are not
strictly speaking failure at all: the system is operating correctly,
but is failing to operate in a timely or efficient fashion
• That is, it just… sucks
4. The stack of abstraction
• Our software systems are built as stacks of abstraction
• These stacks allow us to stand on the shoulders of history — to
reuse components without rebuilding them
• We can do this because of the software paradox: software is
both information and machine, exhibiting properties of both
• Our stacks are higher and run deeper than we can see or know:
software is silent and opaque; the nature of abstraction is to
seal us from what runs beneath!
• They run so deep as to challenge our definition of software…
5. The Butterflies
• When the stack of abstraction performs pathologically, its power
transmogrifies to peril: layering amplifies performance
pathologies but hinders insight
• Work amplifies as we go down the stack
• Latency amplifies as we go up the stack
• Seemingly minor issues in one layer can cascade into systemic
pathological performance
• These are the butterflies that cause hurricanes
8. Butterfly III: Kernel page-table isolation
Data courtesy Scaleway, running a PHP workload with KPTI patches for Linux. Thank you Edouard Bonlieu and team!
9. The Hurricane
• With pathologically performing systems, we are faced with
Leventhal’s Conundrum: given a hurricane, find the butterflies!
• This is excruciatingly difficult:
• Symptoms are often far removed from root cause
• There may not be a single root cause but several
• The system is dynamic and may change without warning
• Improvements to the system are hard to model and verify
• Emphatically, this is not “tuning” — it is debugging
10. Performance debugging
• When we think of it as debugging, we can stop pretending that
understanding (and rectifying) pathological system performance
is rote or mechanical — or easy
• We can resist the temptation to be guided by folklore: just
because someone heard about something causing a problem
once doesn’t mean it’s the problem now!
• We can resist the temptation to change the system before
understanding it: just as you wouldn’t (or shouldn’t!) debug by
just changing code, you shouldn’t debug a pathologically
performing system by randomly altering it!
11. How do we debug?
• To debug methodically, we must resist the temptation to quick
hypotheses, focusing rather on questions and observations
• Iterating between questions and observations gathers the facts
that will constrain future hypotheses
• These facts can be used to disconfirm hypotheses!
• How do we ask questions?
• How do we make observations?
12. Asking questions
• For performance debugging, the initial question formulation is
particularly challenging: where does one start?
• Resource-centric methodologies like the USE Method
(Utilization/Saturation/Errors) can be excellent starting points…
• But keep these methodologies in their context: they provide
initial questions to ask — they are not recipes for debugging
arbitrary performance pathologies!
13. Making observations
• Questions are answered through observation
• The observability of the system is paramount
• If the system cannot be observed, one is reduced to guessing,
making changes, and drawing inferences
• If it must be said, drawing inferences based only on change is
highly flawed: correlation does not imply causation!
• To be observable, systems must be instrumentable: they must
be able to be altered to emit a datum in the desired condition
14. Observability through instrumentation
• Static instrumentation modifies source to provide semantically
relevant information, e.g., via logging or counters
• Dynamic instrumentation allows for the system to be changed
while running to emit data, e.g. DTrace, OpenTracing
• Both mechanisms of instrumentation are essential!
• Static instrumentation provides the observations necessary for
early question formulation…
• Dynamic instrumentation answers deeper, ad hoc questions
15. Aside: Monitoring vs. observability
• Monitoring is an essential operational activity that can indicate a
pathologically performing system and provide initial questions
• But monitoring alone is often insufficient to completely debug a
pathologically performing system, because the questions that it
can answer are limited to that which is monitored
• As we increasingly deploy developed systems rather than
received ones, it is a welcome (and unsurprising!) development
to see the focus of monitoring expand to observability!
16. Aggregation
• When instrumenting the system, it can become overwhelmed
with the overhead of instrumentation
• Aggregation is essential for scalable, non-invasive
instrumentation — and is a first-class primitive in (e.g.) DTrace
• But aggregation also eliminates important dimensions of data,
especially with respect to time; some questions may only be
answered with disaggregated data!
• Use aggregation for performance debugging — but also
understand its limits!
17. Visualization
• The visual cortex is unparalleled at detecting patterns
• The value of visualizing data is not merely providing answers,
but also (and especially) provoking new questions
• Our systems are so large, complicated and abstract that there is
not one way to visualize them, but many
• The visualization of systems and their representations is an
essential skill for performance debugging!
18. Visualization: Gnuplot
• Graphs are terrific — so much so that we should not restrict
ourselves to the captive graphs found in bundled software!
• An ad hoc plotting tool is essential for performance debugging;
and Gnuplot is an excellent (if idiosyncratic) one
• Gnuplot is easily combined with workhorses like awk or perl
• That Gnuplot is an essential tool helps to set expectation
around performance debugging tools: they are not magicians!
21. Visualization: Statemaps
• Especially when trying to understand interplay between different
entities, it can be useful to visualize their state over time
• Time is the critical element here!
• We are experimenting with statemaps whereby state transitions
are instrumented (e.g., with DTrace) and then visualized
• This is not necessarily a new way of visualizing the system
(e.g., early thread debuggers often showed thread state over
time), but with a new focus on post hoc visualization
• Primordial implementation: https://github.com/joyent/statemap
27. The hurricane’s butterfly
• Finding the source(s) of pathologically performing systems must
be thought of as debugging — albeit the hardest kind
• Debugging isn’t about making guesses; it’s about asking
questions and answering them with observations
• We must enshrine observability to assure debuggability!
• Debugging rewards persistence, grit, and resilience more than
intuition or insight — it is more perspiration than inspiration!
• We must have the faith that our systems are — in the end —
purely synthetic; we can find the hurricane’s butterfly!