Leadership Without Management: Scaling Organizations by Scaling Engineersbcantrill
My talk at Surge 2013. Video is at http://www.youtube.com/watch?v=bGkVM1B5NuI Caution: Should not be consumed by stack-ranking six-sigma black belts with fragile constitutions.
Talk originally given at FISL 2012 in Porto Alegre, Brazil. Video was on YouTube but regrettably taken down. Fortunately, I gave a slightly updated (and frankly, tighter and better produced) version of this at the Liferay Symposium in the fall of 2012: https://www.youtube.com/watch?v=Pm8P4oCIY3g
Leadership Without Management: Scaling Organizations by Scaling Engineersbcantrill
My talk at Surge 2013. Video is at http://www.youtube.com/watch?v=bGkVM1B5NuI Caution: Should not be consumed by stack-ranking six-sigma black belts with fragile constitutions.
Talk originally given at FISL 2012 in Porto Alegre, Brazil. Video was on YouTube but regrettably taken down. Fortunately, I gave a slightly updated (and frankly, tighter and better produced) version of this at the Liferay Symposium in the fall of 2012: https://www.youtube.com/watch?v=Pm8P4oCIY3g
Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole
Many have gone before you along this path. Many have failed. A few have succeeded. All have scars. Although the journey is different for everyone there are common aspects to them all. In this talk we will cover our experiences in moving applications into the Cloud. What you must do. What you must not. What matters, what doesn’t.
In moving to the cloud there is no try.
In this talk:
* We’ll cover the core aspects of how the cloud differs from local data centers in terms of application design, runtime characteristics and operational considerations.
* We’ll explain through various real life examples where things worked and where they didnt
* We end with a summary of the key elements to success and the major pitfalls to avoid.
This is a slide deck from a presentation, that my colleague Shirin Glander (https://www.slideshare.net/ShirinGlander/) and I did together. As we created our respective parts of the presentation on our own, it is quite easy to figure out who did which part of the presentation as the two slide decks look quite different ... :)
For the sake of simplicity and completeness, I just copied the two slide decks together. As I did the "surrounding" part, I added Shirin's part at the place when she took over and then added my concluding slides at the end. Well, I'm sure, you will figure it out easily ... ;)
The presentation was intended to be an introduction to deep learning (DL) for people who are new to the topic. It starts with some DL success stories as motivation. Then a quick classification and a bit of history follows before the "how" part starts.
The first part of the "how" is some theory of DL, to demystify the topic and explain and connect some of the most important terms on the one hand, but also to give an idea of the broadness of the topic on the other hand.
After that the second part dives deeper into the question how to actually implement DL networks. This part starts with coding it all on your own and then moves on to less coding step by step, depending on where you want to start.
The presentation ends with some pitfalls and challenges that you should have in mind if you want to dive deeper into DL - plus the invitation to become part of it.
As always the voice track of the presentation is missing. I hope that the slides are of some use for you, though.
Based on my observations, in IT we suffer from continuous collective amnesia and we are even proud of it.
For at least 50 years meanwhile, we struggle how to build systems, that are easy to understand, to maintain, to change and to operate in a reliable way. Each time we hit the wall again, we start to look for a new silver bullet on the horizon, strongly believing that it will solve the problem for good.
The key word is "new": "New" is good in our community, while "old" is bad, worthless, crap. We suffer from youthism, not only in recruiting, but in all areas. This way we discard any "old" knowledge, no matter if it is valuable or not. We separate by age, not by value.
Additionally we continuously lose our collective memory with every new generation that leaves university as they are also taught not to value anything old and instead only look for the new, shiny stuff.
While not all old knowledge is worth being preserved, admittedly, there is still a lot of valuable old knowledge available, offering answers to the problems that we face today - creating maintainable and reliable systems, dealing with distribution and tackling complexity, just to name a few of the challenges.
This presentation is a journey through some (very) old computer science papers that contain a lot of very valuable knowledge regarding the problems we face today. For each of the papers, some of the key ideas are presented and how they address our current challenges.
Of course, the voice track is missing and there are a lot more papers that would be worth being mentioned in this presentation. Still, I hope that also the slides alone will be of some value for you - and convince you a bit that not everything "old" in IT is automatically worthless ... ;)
Thinking About Prototyping: Sketching, Familiarity, Costs versus Ease of Prototyping, Prototypes and Production, Changing Embedded Platform, Physical Prototypes and Mass Personalisation, Climbing into the Cloud, Open Source versus Closed Source, Why Closed? Why Open? Mixing Open and Closed Source, Closed Source for Mass Market Projects, Tapping into the Community. Prototyping Embedded Devices: Electronics, Sensors, Actuators, Scaling Up the Electronics, Embedded Computing Basics, Microcontrollers, System-on-Chips, Choosing Your Platform, Arduino, Developing on the Arduino, Some Notes on the Hardware, Openness, Raspberry Pi, Cases and Extension Boards, Developing on the Raspberry Pi, Some Notes on the Hardware, Openness.
Thierry de Pauw - Feature Branching considered Evil - Codemotion Milan 2018Codemotion
With DVCSs branch creation became very easy, but it comes at a certain cost. Long living branches break the flow of the software delivery process, impacting stability and throughput. The session explores why teams are using feature branches, what problems are introduced by using them and what techniques exist to avoid them altogether. It explores exactly what's evil about feature branches, which is not necessarily the problems they introduce - but rather, the real reasons why teams are using them. After the session, you'll understand a different branching strategy and how it relates to CI/CD.
This slide deck tries to give an overview about the consistency options we have today and what we can expect in the future. It's an updated version of slide deck I had uploaded to SlideShare before. Unfortunately, SlideShare removed the option to update a slide deck. Thus, I had to remove the old slide deck and create a new one, even if it actually was just an update.
It starts with examining RDBMS and ACID transactions - especially looking at the fact that "ACID" does not necessarily mean serializablity (what most developers think of if they reason about consistency).
Then it describes the current state of IT affairs concerning Cloud, microservices, NoSQL databases and BASE transactions. It then dives a bit deeper into the topic of polyglot persistence and its challenges by introducing a little "storage dimension model" that may support you in picking the right storage solution that fits your needs. Especially it is pointed out that in the storage world there is no one-size-fits-all solution available and especially that NoSQL databases - definitely having their sweet spots where they shine - are no drop-in replacement for a RDBMS.
Afterwards, BASE transactions are examined in more detail - especially the fact that their trade-off for providing us with better availability and scalability is a very hard programming model. To underline that point the typical types of inconsistencies that you often will see with BASE transactions are sketched on what you need to do on the application level in order to cope with them.
In the third section, a peek into the future is made by looking at some current research results that will eventually find their way into commercial application development. The core observation is that even in highly available systems there are a lot of options regarding consistency. Yet, most of those options require to reason about consistency explicitly on the application level and no longer solely on the read/write level of the underlying data stores - including building explicit consistency support into the application.
In the last section, some recommendation for our daily work are given that can be derived from the former sections.
As always the voice track is missing, but hopefully it will give you a few helpful pointers, though. Especially, at the end a lot of references to computer science papers are included that this slide deck is based on.
Digitization solutions - A new breed of softwareUwe Friedrichsen
This slide deck is about the challenges we have to face if we deal with digitization solutions. As this term currently is massively overused, I first introduce a very simple definition to define what I mean with "digitization solution" in the context of the presentation.
Afterwards, I list the challenges - at least the most relevant ones - that arise from moving into the digitzation solution domain. Based on that, I try to examine the trends, prerequisites and limitations that you are confronted with from an IT point of view and you better need to adapt to if you are confronted with digitization in your company. Last, but not least, I try to derive some practical hints for us as individuals, how we can prepare for such an environment.
As always, the voice track is missing, but I hope also the slides on their own bear some value for you.
The Xen Project has been producing open source virtualization technologies for the past 10 years. It currently enables some of the largest clouds in the industry. However, decisions made on both the community and business fronts a few years back nearly caused the Xen Project to collapse on itself. Through a series of corrective actions, Xen is once again moving forward with major advances and a growing community. We will discuss some of the lessons learned from this experience, including lessons from both community and business vantage points. We will also focus on maintaining a healthy relationship between a community and a business entity as a means of keeping a project vibrant.
OSCON 2013: "Case Study: What to do when your project outgrows your company"The Linux Foundation
In the last decade, the open source development model has been extraordinarily successful. However, this success came at a cost. Today, we are experiencing an explosion of open source projects, escalating competition amongst projects, and commercial interests driving projects at an ever increasing rate. To succeed in the long run, open source projects need to employ a set of management techniques to overcome these issues.
To illustrate how this is accomplished, we will draw on the 10 year history of the Xen project as a case study. Xen started in 2003 with strong university roots, quickly evolving into a thriving code-centric project with a development culture very similar to the Linux kernel. The project was instrumental in creating the cloud computing space as we know it today. Large companies such as Amazon, Citrix, and Rackspace, as well as many others, were able to build thriving businesses using open source Xen. Yet five years later, the project started struggling in multiple areas: it failed to engage its user base, becoming inward focused, and suffering from fragmentation caused by pressure from commercial interests. As a result, Xen lost the support of a number of key Linux distributions. Xen started to be perceived as old and outdated, despite a swelling user base, a growing and diversifying contributor community, an increasing number of Xen-based commercial solutions, and numerous Xen-powered open source projects. Some onlookers even predicted the imminent demise of the project. The project clearly had an image problem, which could have become a self-fulfilling prophecy. But, because of a program of cultural change initiated 2 years ago, the Xen project of 2013 is no longer crippled by the problems which nearly caused its downfall. We will explore the causes of the issues which arose, highlighting mistakes that were made, and revealing their effect on the community.
We will also show how almost any issue that threatens the success of an open source project can be addressed using a combination of good community management, excellent collaboration infrastructure, wise marketing, and good governance to align the interests of a project’s stake-holders. By employing each of these techniques in a complementary fashion, we can ensure the long-term success of a project.
Video as a prototyping tool for connected productsMartin Charlier
Martin Charlier @ O'Reilly Design Conference 2018
This is a talk about how video is a powerful tool for rapid low-fi prototyping of connected products & the Internet of Things. This talk argues how this method is useful, shows how it builds on existing prototyping methods and gives a practical example for how you can apply this method. Finally, the talk shows how the same ideas and principles of animated or filmed artefacts can be used at different levels of fidelity and focussed on different purposes.
Jax Devops 2017 Succeeding in the Cloud – the guidebook of FailSteve Poole
Many have gone before you along this path. Many have failed. A few have succeeded. All have scars. Although the journey is different for everyone there are common aspects to them all. In this talk we will cover our experiences in moving applications into the Cloud. What you must do. What you must not. What matters, what doesn’t.
In moving to the cloud there is no try.
In this talk:
* We’ll cover the core aspects of how the cloud differs from local data centers in terms of application design, runtime characteristics and operational considerations.
* We’ll explain through various real life examples where things worked and where they didnt
* We end with a summary of the key elements to success and the major pitfalls to avoid.
This is a slide deck from a presentation, that my colleague Shirin Glander (https://www.slideshare.net/ShirinGlander/) and I did together. As we created our respective parts of the presentation on our own, it is quite easy to figure out who did which part of the presentation as the two slide decks look quite different ... :)
For the sake of simplicity and completeness, I just copied the two slide decks together. As I did the "surrounding" part, I added Shirin's part at the place when she took over and then added my concluding slides at the end. Well, I'm sure, you will figure it out easily ... ;)
The presentation was intended to be an introduction to deep learning (DL) for people who are new to the topic. It starts with some DL success stories as motivation. Then a quick classification and a bit of history follows before the "how" part starts.
The first part of the "how" is some theory of DL, to demystify the topic and explain and connect some of the most important terms on the one hand, but also to give an idea of the broadness of the topic on the other hand.
After that the second part dives deeper into the question how to actually implement DL networks. This part starts with coding it all on your own and then moves on to less coding step by step, depending on where you want to start.
The presentation ends with some pitfalls and challenges that you should have in mind if you want to dive deeper into DL - plus the invitation to become part of it.
As always the voice track of the presentation is missing. I hope that the slides are of some use for you, though.
Based on my observations, in IT we suffer from continuous collective amnesia and we are even proud of it.
For at least 50 years meanwhile, we struggle how to build systems, that are easy to understand, to maintain, to change and to operate in a reliable way. Each time we hit the wall again, we start to look for a new silver bullet on the horizon, strongly believing that it will solve the problem for good.
The key word is "new": "New" is good in our community, while "old" is bad, worthless, crap. We suffer from youthism, not only in recruiting, but in all areas. This way we discard any "old" knowledge, no matter if it is valuable or not. We separate by age, not by value.
Additionally we continuously lose our collective memory with every new generation that leaves university as they are also taught not to value anything old and instead only look for the new, shiny stuff.
While not all old knowledge is worth being preserved, admittedly, there is still a lot of valuable old knowledge available, offering answers to the problems that we face today - creating maintainable and reliable systems, dealing with distribution and tackling complexity, just to name a few of the challenges.
This presentation is a journey through some (very) old computer science papers that contain a lot of very valuable knowledge regarding the problems we face today. For each of the papers, some of the key ideas are presented and how they address our current challenges.
Of course, the voice track is missing and there are a lot more papers that would be worth being mentioned in this presentation. Still, I hope that also the slides alone will be of some value for you - and convince you a bit that not everything "old" in IT is automatically worthless ... ;)
Thinking About Prototyping: Sketching, Familiarity, Costs versus Ease of Prototyping, Prototypes and Production, Changing Embedded Platform, Physical Prototypes and Mass Personalisation, Climbing into the Cloud, Open Source versus Closed Source, Why Closed? Why Open? Mixing Open and Closed Source, Closed Source for Mass Market Projects, Tapping into the Community. Prototyping Embedded Devices: Electronics, Sensors, Actuators, Scaling Up the Electronics, Embedded Computing Basics, Microcontrollers, System-on-Chips, Choosing Your Platform, Arduino, Developing on the Arduino, Some Notes on the Hardware, Openness, Raspberry Pi, Cases and Extension Boards, Developing on the Raspberry Pi, Some Notes on the Hardware, Openness.
Thierry de Pauw - Feature Branching considered Evil - Codemotion Milan 2018Codemotion
With DVCSs branch creation became very easy, but it comes at a certain cost. Long living branches break the flow of the software delivery process, impacting stability and throughput. The session explores why teams are using feature branches, what problems are introduced by using them and what techniques exist to avoid them altogether. It explores exactly what's evil about feature branches, which is not necessarily the problems they introduce - but rather, the real reasons why teams are using them. After the session, you'll understand a different branching strategy and how it relates to CI/CD.
This slide deck tries to give an overview about the consistency options we have today and what we can expect in the future. It's an updated version of slide deck I had uploaded to SlideShare before. Unfortunately, SlideShare removed the option to update a slide deck. Thus, I had to remove the old slide deck and create a new one, even if it actually was just an update.
It starts with examining RDBMS and ACID transactions - especially looking at the fact that "ACID" does not necessarily mean serializablity (what most developers think of if they reason about consistency).
Then it describes the current state of IT affairs concerning Cloud, microservices, NoSQL databases and BASE transactions. It then dives a bit deeper into the topic of polyglot persistence and its challenges by introducing a little "storage dimension model" that may support you in picking the right storage solution that fits your needs. Especially it is pointed out that in the storage world there is no one-size-fits-all solution available and especially that NoSQL databases - definitely having their sweet spots where they shine - are no drop-in replacement for a RDBMS.
Afterwards, BASE transactions are examined in more detail - especially the fact that their trade-off for providing us with better availability and scalability is a very hard programming model. To underline that point the typical types of inconsistencies that you often will see with BASE transactions are sketched on what you need to do on the application level in order to cope with them.
In the third section, a peek into the future is made by looking at some current research results that will eventually find their way into commercial application development. The core observation is that even in highly available systems there are a lot of options regarding consistency. Yet, most of those options require to reason about consistency explicitly on the application level and no longer solely on the read/write level of the underlying data stores - including building explicit consistency support into the application.
In the last section, some recommendation for our daily work are given that can be derived from the former sections.
As always the voice track is missing, but hopefully it will give you a few helpful pointers, though. Especially, at the end a lot of references to computer science papers are included that this slide deck is based on.
Digitization solutions - A new breed of softwareUwe Friedrichsen
This slide deck is about the challenges we have to face if we deal with digitization solutions. As this term currently is massively overused, I first introduce a very simple definition to define what I mean with "digitization solution" in the context of the presentation.
Afterwards, I list the challenges - at least the most relevant ones - that arise from moving into the digitzation solution domain. Based on that, I try to examine the trends, prerequisites and limitations that you are confronted with from an IT point of view and you better need to adapt to if you are confronted with digitization in your company. Last, but not least, I try to derive some practical hints for us as individuals, how we can prepare for such an environment.
As always, the voice track is missing, but I hope also the slides on their own bear some value for you.
The Xen Project has been producing open source virtualization technologies for the past 10 years. It currently enables some of the largest clouds in the industry. However, decisions made on both the community and business fronts a few years back nearly caused the Xen Project to collapse on itself. Through a series of corrective actions, Xen is once again moving forward with major advances and a growing community. We will discuss some of the lessons learned from this experience, including lessons from both community and business vantage points. We will also focus on maintaining a healthy relationship between a community and a business entity as a means of keeping a project vibrant.
OSCON 2013: "Case Study: What to do when your project outgrows your company"The Linux Foundation
In the last decade, the open source development model has been extraordinarily successful. However, this success came at a cost. Today, we are experiencing an explosion of open source projects, escalating competition amongst projects, and commercial interests driving projects at an ever increasing rate. To succeed in the long run, open source projects need to employ a set of management techniques to overcome these issues.
To illustrate how this is accomplished, we will draw on the 10 year history of the Xen project as a case study. Xen started in 2003 with strong university roots, quickly evolving into a thriving code-centric project with a development culture very similar to the Linux kernel. The project was instrumental in creating the cloud computing space as we know it today. Large companies such as Amazon, Citrix, and Rackspace, as well as many others, were able to build thriving businesses using open source Xen. Yet five years later, the project started struggling in multiple areas: it failed to engage its user base, becoming inward focused, and suffering from fragmentation caused by pressure from commercial interests. As a result, Xen lost the support of a number of key Linux distributions. Xen started to be perceived as old and outdated, despite a swelling user base, a growing and diversifying contributor community, an increasing number of Xen-based commercial solutions, and numerous Xen-powered open source projects. Some onlookers even predicted the imminent demise of the project. The project clearly had an image problem, which could have become a self-fulfilling prophecy. But, because of a program of cultural change initiated 2 years ago, the Xen project of 2013 is no longer crippled by the problems which nearly caused its downfall. We will explore the causes of the issues which arose, highlighting mistakes that were made, and revealing their effect on the community.
We will also show how almost any issue that threatens the success of an open source project can be addressed using a combination of good community management, excellent collaboration infrastructure, wise marketing, and good governance to align the interests of a project’s stake-holders. By employing each of these techniques in a complementary fashion, we can ensure the long-term success of a project.
Video as a prototyping tool for connected productsMartin Charlier
Martin Charlier @ O'Reilly Design Conference 2018
This is a talk about how video is a powerful tool for rapid low-fi prototyping of connected products & the Internet of Things. This talk argues how this method is useful, shows how it builds on existing prototyping methods and gives a practical example for how you can apply this method. Finally, the talk shows how the same ideas and principles of animated or filmed artefacts can be used at different levels of fidelity and focussed on different purposes.
My (very brief!) presentation at Interzone.io on March 11, 2015. A more in depth exploration of these ideas can be found at http://www.slideshare.net/bcantrill/docker-and-the-future-of-containers-in-production video: https://www.joyent.com/developers/videos/docker-and-the-future-of-containers-in-production
DPG 2014: "Context Sensitive and Time Dependent Relevance of Wikipedia Articles"Dr. Mirko Kämpf
Since the numbers of hypertext pages and hyperlinks in the WWW have been continuously growing for more than 20 years, the problem of finding relevant content has become increasingly important. We have developed and evaluated techniques for a time-dependent characterization of the global and local relevance of WWW pages based on document length, number of links, and cross-correlations in user-access time series. We focus on content and user activity in selected groups of Wikipedia articles as a first application mainly because of data availability. Our goal is the assignment of ranking values to a hypertext page
(node). The values shall cover static properties of the node and its neighbourhood (context) as well as dynamic properties derived from its page-view rates that depend on underlying communication processes. We show in several examples how this goal can be achieved.
This is a talk given by Jason Hoffman at a workshop given by Joyent called "Scale With Rails" in 2006. There's quite a bit of prescience in this presentation, including the first documented use of ZFS in production ("Fsck you if you think ZFS isn't production") and of OS-based virtualization (zones) in the cloud (which, it must be said, was not called "cloud" in 2006).
Empirical Methods in Software Engineering - an Overviewalessio_ferrari
A first introductory lecture on empirical methods in software engineering. It includes:
1) Motivation for empirical software engineering studies
2) How to define research questions
3) Measures and data collection methods
4) Formulating theories in software engineering
5) Software engineering research strategies
Find the videos at: https://www.youtube.com/playlist?list=PLSKM4VZcJjV-P3fFJYMu2OhlTjEr9Bjl0
Are you having trouble getting your bug reports fixed? It could be that you’ve yet to master the craft of bug reporting. It’s a common assumption that bug reports are easy to create, but a well-crafted bug report requires more than innate ability.
In this practical workshop, Neil will share his experiences (good and bad!) from ten years of bug reporting, and show how you can supercharge your bug reports:
First presented at the 2015 TestBash Workshop Day: http://www.ministryoftesting.com/training-events/testbash-workshop-day/
Lightening Talk I gave at Inaka in April 2014.
I was in charge of investigating test-driven development for our iOS mobile team. Since I realized it was such a big concept, after having gathered enough information and having played with it enough, I decided to introduce my fellows on the topic by presenting it in a formal talk with slides. The aim was teaching them a different way of developing, which, for us, at that moment, was completely new and controversial.
Command Query Responsibility Separation doesn’t only simplify your code—it simplifies your thought process in all kinds of areas. In this session we will look through the benefits of CQRS for data persistence, performance, supportability and testing. We will also look at the way that these concepts allow a team to organize around a design and the clarity that CQRS can help provide.
Gain a deeper understanding of what Exploratory Testing (ET) is, the essential elements of the practice with practical tips and techniques, and finally, ideas for integrating ET into the cadence of an agile process
We’re all doing Agile nowadays, aren’t we? We’ll all delivering software in an Agile way. But what does that mean? Does it mean sprints and stand-ups? Kanban even? But what about Extreme Programming? If as a development team we’re not using pair programming, test driven development, continuous integration, and other XP practices, then we’re not really doing Agile software development and we may be on a march to frustration, or even failure.
I’m going to look at why the current trend of companies and projects adopting Scrum, calling themselves Agile, but not transitioning their development to XP, is a recipe for disaster. I’d like to cover the main practices of XP as well as other good practices that can really help a team deliver quality software, whether they’re doing two-week sprints, Kanban, or even Waterfall.
https://www.youtube.com/watch?v=aZgnY9fAHOA
How To (Not) Open Source - Javazone, Oslo 2014gdusbabek
Releasing an open source project while maintaining a shipping product is hard! Different behaviors, attitudes and actions can help or hinder your cause; and they are not always obvious.
The Blueflood distributed metrics engine was released as open source software by Rackspace in August 2012. In the succeeding months the team had to strike a manageable balance between the challenges of growing a community, being good open source stewards, and maintaining a shipping product for Rackspace. Find out what worked, what did not work, and the lessons that can be applied as you endeavor to take your project out into the open.
In this presentation you will learn about strategies for releasing open source products, pitfalls to avoid, and the potential benefits of moving more of your development out in the open.
We have also made a few realizations about the community growing up around metrics. It is still young, and there are problems that come with that youth. I'll talk about some things we can do to make a better software ecosystem.
Feedback loops between tooling and cultureChris Winters
Discussion of how tools technologists create impact culture, and how culture impacts those tools. Not really a standalone presentation but hopefully useful.
The biggest DevOps problems you didn't know you had and what to do about themWayne Greene
Slide deck from www.ReleaseIQ.io webinar on August 25, 2021. Learn about the biggest problems DevOps teams face. You might be surprised what keeps DevOps teams stuck in mid-evolution.
Talk given at the OCP Open System Firmware engineering workshop on 5/17/22. Talk was recorded; video at https://www.youtube.com/watch?v=eNI0wFgBNmY#t=7044s
Hardware/software Co-design: The Coming Golden Agebcantrill
Talk I gave as a keynote at RailsConf 2021. There is no Rails in the talk, though; this is all about the revolutions in open source firmware and hardware that are changing the way we build systems. Video to come!
Tockilator: Deducing Tock execution flows from Ibex Verilator tracesbcantrill
Talk given on March 20, 2020 at Oxidize 1K, a virtual conference that went from first idea to 300+ person conference in a week during the COVID-19 pandemic.
Platform values, Rust, and the implications for system softwarebcantrill
Talk given at Scale By The Bay 2018. Video is at https://www.youtube.com/watch?v=2wZ1pCpJUIM. If you are interested in this talk, you might also be interested in my talk on Platform as a Reflection of Values from Node Summit 2017: https://www.slideshare.net/bcantrill/platform-as-reflection-of-values-joyent-nodejs-and-beyond
My Papers We Love talk in San Francisco on October 12, 2017 on "ARC: A self-tuning, low overhead replacement cache." Video at https://www.youtube.com/watch?v=F8sZRBdmqc0
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Innovating Inference - Remote Triggering of Large Language Models on HPC Clus...Globus
Large Language Models (LLMs) are currently the center of attention in the tech world, particularly for their potential to advance research. In this presentation, we'll explore a straightforward and effective method for quickly initiating inference runs on supercomputers using the vLLM tool with Globus Compute, specifically on the Polaris system at ALCF. We'll begin by briefly discussing the popularity and applications of LLMs in various fields. Following this, we will introduce the vLLM tool, and explain how it integrates with Globus Compute to efficiently manage LLM operations on Polaris. Attendees will learn the practical aspects of setting up and remotely triggering LLMs from local machines, focusing on ease of use and efficiency. This talk is ideal for researchers and practitioners looking to leverage the power of LLMs in their work, offering a clear guide to harnessing supercomputing resources for quick and effective LLM inference.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
May Marketo Masterclass, London MUG May 22 2024.pdfAdele Miller
Can't make Adobe Summit in Vegas? No sweat because the EMEA Marketo Engage Champions are coming to London to share their Summit sessions, insights and more!
This is a MUG with a twist you don't want to miss.
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteGoogle
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
👉👉 Click Here To Get More Info 👇👇
https://sumonreview.com/ai-pilot-review/
AI Pilot Review: Key Features
✅Deploy AI expert bots in Any Niche With Just A Click
✅With one keyword, generate complete funnels, websites, landing pages, and more.
✅More than 85 AI features are included in the AI pilot.
✅No setup or configuration; use your voice (like Siri) to do whatever you want.
✅You Can Use AI Pilot To Create your version of AI Pilot And Charge People For It…
✅ZERO Manual Work With AI Pilot. Never write, Design, Or Code Again.
✅ZERO Limits On Features Or Usages
✅Use Our AI-powered Traffic To Get Hundreds Of Customers
✅No Complicated Setup: Get Up And Running In 2 Minutes
✅99.99% Up-Time Guaranteed
✅30 Days Money-Back Guarantee
✅ZERO Upfront Cost
See My Other Reviews Article:
(1) TubeTrivia AI Review: https://sumonreview.com/tubetrivia-ai-review
(2) SocioWave Review: https://sumonreview.com/sociowave-review
(3) AI Partner & Profit Review: https://sumonreview.com/ai-partner-profit-review
(4) AI Ebook Suite Review: https://sumonreview.com/ai-ebook-suite-review
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
2. Containers at Joyent
• Joyent runs OS containers in the cloud via SmartOS — and we
have run containers in multi-tenant production since ~2006
• Adding support for hardware-based virtualization circa 2011
strengthened our resolve with respect to OS-based virtualization
• OS containers are lightweight and efficient — which is especially
important as services become smaller and more numerous:
overhead and latency become increasingly important!
• We emphasized their operational characteristics — performance,
elasticity, tenancy — and for many years, we were a lone voice...
3. Containers as PaaS foundation?
• Some saw the power of OS containers to facilitate up-stack
platform-as-a-service abstractions
• For example, dotCloud — a platform-as-a-service provider — built
their PaaS on OS containers
• Struggling as a PaaS, dotCloud pivoted — and open sourced
their container-based orchestration layer...
5. Docker and the container revolution
• Docker has brought the concept to a much broader audience
• Docker has used the rapid provisioning + shared underlying
filesystem of containers to allow developers to think operationally
• Developers can encode deployment procedures via an image
• Images can be reliably and reproducibly deployed as a container
• Images can be quickly deployed — and re-deployed
• Docker is doing to apt what apt did to tar
6. Docker at Joyent
• We wanted to create a best-of-all-worlds platform: the developer
ease of Docker on the production-grade substrate of SmartOS
• We developed a Linux system call interface for SmartOS,
allowing SmartOS to run Linux binaries at bare-metal speed
• In March 2015, we introduced Triton, our (open source!) stack
that deploys Docker containers directly on the metal
• Triton virtualizes the notion of a Docker host (i.e., “docker ps”
shows all of one’s containers datacenter-wide)
• Brings full debugging (DTrace, MDB) to Docker containers
7. Containers and the rise of microservices
• The container revolution has both accelerated and been
accelerated by the rise of microservices: small, well-defined
services that do one thing and do it well
• While the term provokes nerd rage in some, it is merely a new
embodiment of an old idea: the Unix Philosophy
• Write programs that do one thing and do it well.
• Write programs to work together.
• Write programs to handle text streams, because that is a
universal interface.
8. Containers in production
• Containers + microservices are great when they work — but what
happens when these systems fail?
• For continuous integration/continuous deployment use cases
(and/or other stateless services), failure is less of a concern…
• But as containers are increasingly used for workloads that matter,
one can no longer insist that failures don’t happen — or that
restarts will cure any that do
• The ability to understand failure is essential to leap the chasm
from development into meaningful production!
12. Container failure modes
• When deploying containers + microservices, there is an unstated
truth: you are developing a distributed system
• While more resilient to certain classes of force majeure failure,
distributed systems remain vulnerable to software defects
• We must be able to debug such systems; hope is not a strategy!
• Distributed systems are hard to debug — and are more likely to
exhibit behavior non-reproducible in development
• With container-based systems, we must debug not in terms of
sick pets but rather sick cattle
16. Debugging anti-patterns
• For too many, debugging is the process of making problems go
away rather than understanding the system!
• The view of bugs-as-nuisance has many knock-on effects:
• Fixes that don’t fix the problem (or introduce new ones!)
• Bug reports closed out as “will not fix” or “works for me”
• Users who are told to “restart” or “reboot” or “log out” or
anything else that amounts to wishful thinking
• And this is only when the process has obviously failed...
17. Darker debugging anti-patterns
• More insidious effects are felt when the problem appears to have
been resolved, but hasn’t actually been fully understood
• These are the fixes that amount to a fresh coat of paint over a
crack in the foundation — and they are worse than nothing
• Not only do these fixes not actually resolve the problem, they give
the engineer a false sense of confidence that spreads virally
• “Debugging” devolves into an oral tradition: folk tales of problems
that were made to go away
• Container-based systems are especially vulnerable to this!
18. Thinking methodically
• The way we think about debugging is fundamentally wrong; we
need to think methodically about debugging!
• When we think of debugging as the quest for understanding our
(misbehaving) systems, it allows us to consider it more abstractly
• Namely, how do we explain the phenomena that affect our world?
• We have found that the most powerful explanations reflect an
understanding of underlying structure — beyond what to why
• This deeper understanding allows us to not only to explain but
make predictions
19. Predictive power
• Valuing predictive power allows us to test our explanations: if our
predictions are wrong, our understanding is incomplete
• We can use the understanding from failed predictions to develop
new explanations and new predictions
• We can then test these new predictions to test our understanding
• If all of this is sounding familiar, it’s because it’s science — and
the methodical exploration of it is the scientific method
20. The scientific method
• The scientific method is to:
• Make observations
• Formulate a question
• Formulate a hypothesis that answers the question
• Formulate predictions that test the hypothesis
• Test the predictions by conducting an experiment
• Refine the hypothesis and repeat as needed
23. Science, seriously.
• Software debugging is a pure distillation of scientific thinking
• The limitless amount of data from software systems allows
experiments in seconds instead of weeks/months/years
• The systems we’re reasoning about are entirely synthetic,
discrete and mutable — we made it, we can understand it
• Software is mathematical machine; the conclusions of software
debugging are often mathematical in their unequivocal power!
• Software debugging is so pure, it requires us to refine the
scientific method slightly to reflect its capabilities...
24. The software debugging method
• Make observations
• Based on observations, formulate a question
• If the question can be answered through subsequent observation,
answer the question through observation and refine/iterate
• If the question cannot be answered through observation, make a
hypothesis as to the answer and formulate predictions
• If predictions can be tested through subsequent observation, test
the predictions through observation and refine/iterate
• Otherwise, test predictions through experiment and refine/iterate
25. Observation is the heart of debugging!
• The essence — and art! — of debugging software is making
observations and asking questions, not formulating hypotheses!
• Observations are facts — they constrain hypotheses in that any
hypothesis contradicted by facts can be summarily rejected
• As facts beget questions which beget observations and more
facts, hypotheses become more tightly constrained — like a
cordon being cinched around the truth
• Or, in the words of Sir Arthur Conan Doyle’s Sherlock Holmes,
“when you have eliminated all which is impossible, then whatever
remains, however improbable, must be the truth”
26. Making the hypothetical leap
• Once observation has sufficiently narrowed the gap between what
is known and what is wrong, a hypothetical leap should be made
• Debugging is inefficient when this leap is made too early — like
making a specific guess too early in Twenty Questions
• A hypothesis is only as good as its ability to form a prediction
• A prediction should be tested with either subsequent observation
or by conducting an experiment
• If the prediction proves to be incorrect, understanding is
incomplete; the hypothesis must be rejected — or refined
27. Experiments in software
• A beauty of software is that it is highly amenable to experiment
• Many experiments are programs — and the most satisfying
experiments test predictions about how failure can be induced
• Many “non-reproducible” problems are merely unusual!
• Debugging a putatively non-reproducible problem to the point of a
reproducible test case is a joy unique in software engineering
28. Software debugging in practice
• The specifics of observation depends on the nature of the failure
• Software has different kinds of failure modes:
• Fatal failure (segmentation violation, uncaught exception)
• Non-fatal failure (gives the wrong answer, performs terribly)
• Explicit failure (assertion failure, error message)
• Implicit failure (cheerfully does the wrong thing)
29. Taxonomizing software failure
Implicit
Explicit
Non-fatal Fatal
Gives the wrong answer
Returns the wrong result
Leaks resources
Stops doing work
Performs pathologically
Emits an error message
Returns an error code
Assertion failure
Process explicitly aborts
Exits with an error code
Segmentation violation
Bus Error
Panic
Type Error
Uncaught Exception
30. Cascading container failure modes
Implicit
Explicit
Non-fatal Fatal
Gives the wrong answer
Returns the wrong result
Leaks resources
Stops doing work
Performs pathologically
Emits an error message
Returns an error code
Assertion failure
Process explicitly aborts
Exits with an error code
Segmentation violation
Bus Error
Panic
Type Error
Uncaught Exception
31. Debugging fatal failure
• When software fails fatally, we know that the software itself is
broken — its state has become inconsistent
• By saving in-memory state to stable storage, the software can be
debugged postmortem
• To debug, one starts with the invalid state and reasons backwards
to discover a transition from a valid state to an invalid one
• This technique is so old, that the terms for this saved state dates
back to the dawn of the computing age: a core dump
• Not as low-level as the name implies! Modern high-level
languages (e.g., node.js and Go) allow postmortem debugging!
32. Debugging fatal failure: containers
• Postmortem analysis lends itself very well to containers:
• There is no run-time overhead; overhead (such as it is) is only
at the time of death
• The container can be safely (automatically!) restarted; the core
dump can be analyzed asynchronously
• Tooling need not be in container, can be made arbitrarily rich
• In Triton, all core dumps are automatically stored and then
uploaded into a system that allows for analysis, tagging, etc.
• This has been invaluable for debugging our own services!
33. Core dump management in Docker
• In Triton, all core dumps are automatically stored and then
uploaded into a system that allows for analysis, tagging, etc.
• This has been invaluable for debugging our own services!
• Outside of Triton, the lack of container awareness around
core_pattern in the Linux kernel is problematic for Docker: core
dumps from Docker are still a bit rocky (viz. docker#11740)
• Docker-based core dump management would be a welcome
addition, but the current proposal (docker#19289) seems to have
been lost in the noise
34. Debugging non-fatal failure
• There is a solace in fatal failure: it always represents a software
defect at some level — and the inconsistent state is static
• Non-fatal failure can be more challenging: the state is valid and
dynamic — it’s difficult to separate symptom from cause
• Non-fatal failure must still be understood empirically!
• Debugging in vivo requires that data be extracted from the system
— either of its own volition (e.g., via logs) or by coercion (e.g., via
instrumentation)
35. Debugging explicit, non-fatal failure
• When failure is explicit (e.g., an error or warning message), it
provides a very important data point
• If failure is non-reproducible or otherwise transient, analysis of
explicit software activity becomes essential
• Action in one container will often need to be associated with
failures in another
• Especially for distributed systems, this becomes log analysis, and
is an essential forensic tool for understanding explicit failure
• Essential observation: a time line of events!
36. Log management in Docker
• “docker logs” was designed for casual development use, not
production use cases
• Inadequate log solutions gave rise to anti-patterns that relied on
Docker host manipulation for production logging
• Docker 1.6 introduced the notion of log drivers, allowing for
Docker containers to log to Graylog, syslogd, fluentd, etc.
• Failure modes aren’t necessarily well considered: what happens
(or should happen) when logging becomes impaired?
• In Triton, failure to log constitutes container failure!
37. Aside: Docker host anti-patterns
• In the traditional Docker model, Docker hosts are virtual machines
to which containers are directly provisioned
• It may become tempting to manipulate Docker hosts directly, but
doing this entirely compromises the Docker security model
• Worse, compromising the security model creates a VM
dependency that makes bare-metal containers impossible
• And ironically, Docker hosts can resemble pets: the reasons for
backdooring through the Docker host can come to resemble the
arguments made by those who resist containerization entirely!
38. Debugging implicit, non-fatal failure
• Problems that are both implicit and non-fatal represent the most
time-consuming, most difficult problems to debug because the
system must be understood against its will
• Wherever possible make software explicit about failure!
• Where errors are programmatic (and not operational), they
should always induce fatal failure!
• Container-based systems break at the boundaries: each
thinks they are operating correctly, but together they’re broken
• Data must be coerced from the system via instrumentation
39. Instrumenting production systems
• Traditionally, software instrumentation was hard-coded and static
(necessitating software restart or — worse — recompile)
• Dynamic system instrumentation was historically limited to system
call table (strace/truss) or packet capture (tcpdump/snoop)
• Effective for some problems, but a poor fit for ad hoc analysis
• In 2003, Sun developed DTrace, a facility for arbitrary, dynamic
instrumentation of production systems that has since been ported
to Mac OS X, FreeBSD, NetBSD and (to a degree) Linux
• DTrace has inspired dynamic instrumentation in other systems
40. Instrumenting Docker containers
• In Docker, instrumentation is a challenge as containers may not
include the tooling necessary to understand the system
• Docker host-based techniques for instrumentation may be
tempting, but (again!) they should be considered an anti-pattern!
• DTrace has a privilege model that allows it to be safely (and
usefully) used from within a container
• In Triton, DTrace is available from within every container — one
can “docker exec -it bash” and then debug interactively
41. Instrumenting node.js-based containers
• We have invested heavily in node.js-based infrastructure to allow
us to meaningfully instrument containers in production:
• We developed Bunyan, a logging facility for node.js that
includes DTrace support
• Added DTrace support for node.js profiling
• An essential vector for iterative observation: turning up the
logging level on a running container-based microservice!
42. Debugging Docker containers in production
• Debugging methodically requires us to shift our thinking — and
learn how to carefully observe the systems we build
• Different types of failures necessitate different techniques:
• Fatal failure is best debugged via postmortem analysis —
which is particular appropriate in an all-container world
• Non-fatal failure necessitates log analysis and dynamic
instrumentation
• The ability to debug problems in production is essential to
successfully deploy and scale container-based systems!