SlideShare a Scribd company logo
1 of 137
Download to read offline
T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G
N OV E M B E R 1 2 , 2 0 2 0
Molly Struve
Calling Out a
Terrible On-Call
System
@molly_struve
Calling Out a Terrible
On-Call System
Hi! My name is Molly Struve and before I get started I want to point out that my Twitter handle is in the lower right hand corner of all of the slides. Molly underscore Struve. I have already tweeted out this slide deck so if you
would like to follow along…
@molly_struve
Calling Out a Terrible
On-Call System
head over there and click on the link. Welcome to Calling Out a Terrible On-Call System! I am the Lead Site Reliability Engineer for the community software provider Forem which is what the technical blogging
platform dev.to is built on. Being a…
@molly_struve
Site Reliability Engineer
Site Reliability Engineer means I am one of those weird people that thrives on being on-call. The adrenalin rush of having to figure out a bug as quick as possible really gets me going. But, I’m pretty positive the vast
majority of engineers are not like myself. Raise your hand if you…
@molly_struve
5
hate being on-call or in the past have had a horrible on-call experience? I can’t see any of you through my computer but I am sure, the majority of you have your hands up. On-call is a necessity to support the
applications we build but…
@molly_struve
6
it SHOULD NOT, I repeat, it should NOT make people miserable. If your engineers are miserable during on-call then you have a problem. I am here today to give you some suggestions and strategies you can use
to help you fix this common problem. All of these strategies….
@molly_struve
7
I am about to share unfortunately didn’t just hit me while I was sleeping one night. To figure all of this out I had to live through one of those terrible on-call systems and that experience showed me first hand the toll a
broken system can take on all of those involved. Here is the story of a terrible on-call system in the making!
@molly_struve
8
In the beginning…
In the beginning, I was working on a small engineering team and everyone participated in a…
@molly_struve
9
👩💻
👩💻
👨💻
👩💻
👨💻
single on-call rotation. Every dev on the team was in the rotation and each dev would go on-call for…
@molly_struve
10
👩💻
👩💻
👨💻
👩💻
👨💻
1 week
one week at a time. When we first started the rotation the team had 5 devs on it and it worked great! Everyone was very experienced with the application and with being on call bc everyone did it relatively often
However, as the years went by….
@molly_struve
11
👩💻
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👨💻
1 week
The team started to grow. Despite the team growth we still stuck with this single rotation. Eventually…
@molly_struve
12
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
1 week
the team got so big that people were going on-call once every 3-4 months. Being on-call once every few months may seem like a dream come true, but in reality, it’s far from it. This giant single rotation was
making…
@molly_struve
13
😱
☹
😭
😣
😡
😱
😡😞
😖
😣
😭
😬
😬
☹
1 week
All of the devs miserable for a variety of reasons. For starters this large, single rotation meant….
@molly_struve
14
😱
☹
😭
😣
😡
😱
😡😞
😖
😣
😭
😬
😬
☹
Infrequent
Shifts
Infrequent on-call shifts. As I mentioned, devs were going on-call once every few months. Bc on-call shifts were so infrequent, devs were not able to get the experience and practice they needed to know how to handle
on-call issues effectively. In addition, the code base…
@molly_struve
15
😱
☹
😭
😣
😡
😱
😡😞
😖
😣
😭
😬
😬
☹
Growing, complex
codebase
had grown tremendously and was vastly more complex than when we had started. There were so many things being developed at once that when a problem arose, there was a solid chance the on-call dev knew
nothing about it or the code that was causing it. And what happens when an alarm goes off and you have no idea what to do…
@molly_struve
16
You panic! And who can blame you? We have all been there. When you have to fix something you know nothing about its terrifying! When the devs would panic, they would turn to the people they knew could likely fix
the problem the fastest and that was…
@molly_struve
17
Site Reliability
Engineering Team
The Site Reliability Engineering team. Of course, the devs were right in their assumption, usually the Site Reliability team could fix the problem the fastest, but relying on a small set of people for everything doesn’t
scale. Constantly having to jump in and help….
@molly_struve
18
Site Reliability
Engineering Team
with on-call issues quickly began to drain a lot of the Site Reliability team's time and resources. Essentially, the team began to act as if they were on-call 24/7. The constant bombardment of questions and requests…
@molly_struve
19
Site Reliability
Engineering Team
😩 😩 😩
Began to burnout the Site Reliability team. Besides having a burned out and inefficiently used Site Reliability team, another problem…
@molly_struve
20
😱
☹
😭
😣
😡
😱
😡😞
😖
😣
😭
😬
😬
☹
with this single giant single on-call rotation was that developers felt like they had
@molly_struve
21
😱
☹
😭
😣
😡
😱
😡😞
😖
😣
😭
😬
😬
☹
No
Ownership
no ownership over the code they were responsible for while on-call. One person would write code and another person would be the one debugging it if it broke. The app was so big that there was no way anyone
could have a sense of ownership over the production code since there was way too much of it. This…
@molly_struve
22
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
1 week
Giant, seemingly innocuous On-Call rotation might seem harmless enough but what it leads to is…
@molly_struve
23
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
Infrequent Shifts
Infrequent On-Call Shifts which means less on-call experience and practice for developers. Lack of experience being on call leads to…
@molly_struve
24
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
Infrequent Shifts
Panicked Devs
Panicked Devs who have no clue how to handle issues when they arise. When those panicked and stressed out devs need constant help that leads to…
@molly_struve
25
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
Infrequent Shifts
Panicked Devs
Burned Out Site
Reliability Team
A burned out Site Reliability Team. No one in this entire on-call rotation situation was happy. To top it all off…
@molly_struve
26
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
No
Ownership
There was no feeling of ownership for anyone over the code they were supporting. As many of you know, lack of ownership leads to ambivalence at best. All of..
@molly_struve
27
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
Infrequent Shifts
Panicked Devs
Burned Out Site
Reliability Team
No
Ownership
These problems began adding up and eventually, it got …
@molly_struve
28
so bad that we knew something had to change. Now, before I tell you all about our solution I first want to briefly cover…
@molly_struve
29
Team
Organization
How the engineering team was organized at the time so you have some context about how we ended up with the solution we did. In the engineering department at the time there were…
@molly_struve
30
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
Team
Organization
3 separate dev teams. Each team had 5-7 devs on it…
@molly_struve
31
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
Team
Organization
plus a manager. Each team had its own set of projects but all of the teams worked across one…
@molly_struve
32
One Monolithic
Application
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
single, monolithic Rails application. Unlike other apps that might have very separate backend components owned by individual teams, there were no clear or obvious lines of ownership within…
@molly_struve
33
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
One Monolithic
Application
this single, monolithic rails application. This would prove to be the biggest hurdle when it came to fixing this broken on-call system. Now that you have a little background about the team organization, lets get to
the good stuff…
@molly_struve
34
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
The Solution
The solution to fixing this terrible and broken on-call system.
First and foremost, we knew we had to break up…
@molly_struve
35
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
This giant single rotation if we wanted to continue growing.
Despite all of these developers working….
@molly_struve
36
👩💻
👨💻
👩💻
👨💻
👨💻
👩💻
👩💻👩💻
👨💻
👨💻
👩💻
👩💻
👩💻
👩💻
One Monolithic
Application
across one monolithic application, we decided to break
the single rotation into 3, one rotation…
@molly_struve
37
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
for each of the 3 dev teams. Having 3 small rotations led..
@molly_struve
38
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
More Frequent
Shifts
to more frequent on-call shits which meant more practice and experience for those handling on-call. As backward as it may sound, being on-call on a regular cadence is a benefit because devs become…
@molly_struve
39
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💻 👨💻
👩💻
👩💻
👨💻
👨💼 👩💼
👨💼
More Frequent
Shifts
More Comfortable
a lot more comfortable with it and are able to really figure out a strategy that works best for them. So the first strategy we implemented…
@molly_struve
Overhauling On-Call
40
1
2
3
4
5
6
When Overhauling this on-call system was..
@molly_struve
Overhauling On-Call
41
1
2
3
4
3 Smaller On-Call Rotations
5
6
to split our giant rotation into 3 smaller on-call rotations. Those 3 On-Call rotations solved the problem of shift frequency but that still left the biggest problem of all…
@molly_struve
42
Application
Ownership
Application ownership. Who supports what? How do you define what team is going to cover what code? When it all boils down to it, no one..
@molly_struve
43
wants to support something they don't feel like they own.
To accomplish this we choose to split up the on-call…
@molly_struve
44
Application
Ownership
Team 1 Team 2
Team 3
application ownership amongst the 3 dev teams. Even though I am about to breeze through this split I want to be clear, this did not happen overnight. During this process there were a lot of meetings and planning
and collaborating …
@molly_struve
45
Team 1 Team 2
Team 3
Site
Reliability
Between the Site Reliability team and the dev teams to figure out the best and most logical way to split up the components of our monolithic application. I really want to highlight that this was not the Site Reliability
team calling the shots and handing over the “assignments” to the dev teams. We wanted this whole process of…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Splitting up the application ownership to be as collaborative as possible bc we knew that was going to give us the highest chance of succeeding. We first started by splitting up our…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Background workers. Team 1
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
got the Data processing workers. Team 2…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
Got the Overnight reporting workers and finally Team 3…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Got the Client communication workers. The next thing we needed to split up were our…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Service Alerts
Service Alerts. When I say service alerts here I am referring to alerts that were set up within the monitoring system to monitor things like databases, infrastructure, and code performance. Before it was a single person
staying on top of all of these different alerts. With this new system we decided to split them up as well. We gave…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Service Alerts
Redis and Worker
Queue Alerts
Team 1 the Redis and Worker queue alerts. We gave…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Service Alerts
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
Team 2 the Elasticsearch and API Alerts. And finally we gave…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Service Alerts
Elasticsearch and
API Alerts
Redis and Worker
Queue Alerts
MySQL and Page
Load Alerts
Team 3 the MySQL and Page load alerts. Now that existing service alerts and our background
workers were split up, the last thing to split up was the
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
Application components/Code. We were running a Rails application so this involved splitting up things like Models and Controllers within the codebase. We started by giving…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
Data Processing
Code
All the Data processing Code to Team 1. We figured this would pair well
with the background workers they were also assigned. We gave…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
Data Processing
Code
Reporting and
Emailing Code
Team 2 the emailing and reporting code which
paired well with their overnight workers. And finally we gave..
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
User and App
Alert Code
Data Processing
Code
Reporting and
Emailing Code
Team 3 the User and in App Alerting Code which paired well
with their user communication workers. Once..
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
User and App
Alert Code
Data Processing
Code
Reporting and
Emailing Code
the lines had been drawn, we stressed to each of the dev teams that despite doing our best to balance the code equally we might still have to move things around. This showed the devs that we were fully invested in
making sure this new on-call rotation was …
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
User and App
Alert Code
Data Processing
Code
Reporting and
Emailing Code
fair and better for everyone. I know I got in the weeds a bit here by breaking this all down but I wanted to get a little specific with how we split up our application so that hopefully it can give you…
@molly_struve
Splitting App Ownership
Team 1 Team 2 Team 3
Background
Workers
Service Alerts
Application
Code
Data Processing
Workers
Overnight
Reporting Workers
User Communication
Workers
Redis and Worker
Queue Alerts
Elasticsearch and
API Alerts
MySQL and Page
Load Alerts
User and App
Alert Code
Data Processing
Code
Reporting and
Emailing Code
some ideas about how you might go about splitting up ownership in a single application where lines might not be clearly drawn. And with that…
@molly_struve
Overhauling On-Call
62
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
6
Splitting up the application ownership slides into spot 2 in our overhauling on-call list. Now when it comes to instilling a feeling of ownership another big obstacle is constantly…
@molly_struve
63
Changing Code
Changing code. Having 15 devs meant we could turn out a lot of features, but then the question became, how did teams stay on top of the code they were responsible for when on-call and its changes. For this we……
@molly_struve
64
CODEOWNERS
took advantage of Gitlab's CODEOWNERS file. The CODEOWNERS file lives in…
@molly_struve
65
.github/CODEOWNERS
the .github directory of your application. This file allows you to specify who or what teams in your organization own a file. Here is an example of…
@molly_struve
66
/*.md @org/team-1
/app/controllers/reporting/ @org/team-2
/app/workers/data_processing/ @org/team-1
/config/database.yml @org/team-3
CODEOWNERS
a CODEOWNERS file. As you can see, you can assign…
@molly_struve
67
/*.md @org/team-1
/app/controllers/reporting/ @org/team-2
/app/workers/data_processing/ @org/team-1
/config/database.yml @org/team-3
CODEOWNERS
types of files to a team or person. You can assign…
@molly_struve
68
/*.md @org/team-1
/app/controllers/reporting/ @org/team-2
/app/workers/data_processing/ @org/team-1
/config/database.yml @org/team-3
CODEOWNERS
entire directories to a team or person. Or you can assign…
@molly_struve
69
/*.md @org/team-1
/app/controllers/reporting/ @org/team-2
/app/workers/data_processing/ @org/team-1
/config/database.yml @org/team-3
CODEOWNERS
just a single file to a team or person. Once you have..
@molly_struve
70
/*.md @org/team-1
/app/controllers/reporting/ @org/team-2
/app/workers/data_processing/ @org/team-1
/config/database.yml @org/team-3
CODEOWNERS
this file in place, when any file in your app directory is updated in a Pull Request, the owner of the file can automatically be tagged for review. This allowed the 3 teams to work across the entire codebase while also
staying on top of what was changing in the components they were responsible for during On-Call. Using…
@molly_struve
Overhauling On-Call
71
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
6
A CODEOWNERS file slips into the 3rd spot in our overhauling on-call strategy list. With the application components split up and a CODEOWNERS file to support and empower that ownership feeling, next…
@molly_struve
Overhauling On-Call
72
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
6
on our list was to make sure every team and every single person on each team was completely comfortable with the application components they had been given ownership over. To do this the SRE team…
@molly_struve
73
On-Call Training
Sessions
Hosted mini on-call training sessions. During these sessions we sat down…
@molly_struve
74
👨💻 👨💻
👩💻
👩💻
👨💻👩💼
👨💻 👨💻
👩💻
👩💻
👨💻👨💼
👨💻 👨💻
👩💻
👩💻
👨💼 👨💻
with each dev team to thoroughly review the…
@molly_struve
75
👨💻 👨💻
👩💻
👩💻
👨💻👩💼
👨💻 👨💻
👩💻
👩💻
👨💻👨💼
Code ✅
Workers ✅
Alerts ✅
👨💻 👨💻
👩💻
👩💻
👨💼 👨💻
code, workers, and alerts they were responsible for
covering during on-call. During these…
@molly_struve
On-Call Training Sessions
on-call training sessions we went over things like…
@molly_struve
On-Call Training Sessions
• Common issues
common issues that might pop up. For example, when this alert goes off usually it means xyz is broken with this piece of the code. We also took the time to…
@molly_struve
On-Call Training Sessions
• Common issues
• Code Functionality
Dive into all of the code functionality. We made sure every person on every team knew exactly what each piece of code they covered did. And last but not least, we made sure each team understood…
@molly_struve
On-Call Training Sessions
• Common issues
• Code Functionality
• Larger Application Impact
How their components impacted the rest of the application. For example, if say Redis went down, how did that affect the rest of the application as a whole. These On-Call training sessions gave devs…
@molly_struve
80
Confidence
a lot more confidence in their ability to handle on-call situations because they now had a clear picture of what they were responsible for and how to handle it. Even though they hadn’t built some of the code themselves,
they had an understanding of exactly how it all worked. Hosting…
@molly_struve
Overhauling On-Call
81
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
6
On-call training sessions takes the 4th spot in our overhauling on-call list. As I mentioned earlier, the purpose of these training sessions was to not only educate the devs about the code they were supporting, but also
to give them confidence. Another confidence booster for devs who are on-call is…
@molly_struve
82
On-Call Support
On-call support. What exactly do I mean by this? When a person is paged they aren’t always going to have all of the answers. Sometimes they need help and support from someone else to figure out the problem.
Originally…
@molly_struve
83
On-Call Support
Site Reliability
the Site Reliability team acted as support for the on-call dev. If the on-call dev had questions or needed help they would talk to the Site Reliability team member that was on-call that week. The problem with this
approach was that our Site Reliability team only…
@molly_struve
84
On-Call Support
🤓🤓 🤓
Had 3 people on it at the time and when you have 3 people trying to support 15+ its…
@molly_struve
85
On-Call Support
😥😕 😫
It’s not going to end well or be scalable! Our Site Reliability team got burned out pretty quick being the constant support system for the on-call devs. Not to mention, it cut into our time to do our own projects. With…
@molly_struve
86
👨💻 👨💻
👩💻
👩💻
👨💻👩💼
👨💻 👨💻
👩💻
👩💻
👨💻👨💼
👨💻 👨💻
👩💻
👩💻
👨💼 👨💻
the new system, each…
@molly_struve
87
👨💻 👨💻
👩💻
👩💻
👨💻👩💼
👨💻 👨💻
👩💻
👩💻
👨💻👨💼
👨💻 👨💻
👩💻
👩💻
👨💼 👨💻
dev that is on-call..
@molly_struve
88
👨💻 👨💻
👩💻
👩💻
👨💻👩💼
👨💻 👨💻
👩💻
👩💻
👨💻👨💼
👨💻 👨💻
👩💻
👩💻
👨💼 👨💻
acts as support for the others. If any one finds themselves overwhelmed or stuck on an issue they have two people they can reach out to for help. Having a support system like this is crucial for crafting an on-call
system that is comfortable for everyone. No one wants to feel alone when they are on-call so ensuring that…
@molly_struve
Overhauling On-Call
89
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6
On-Call has a solid support system in place is crucial. The last improvement we made to our system that was welcomed by everyone was that we…
@molly_struve
90
Focused On-Call
Responsibilities
Focused the responsibilities for the On-Call devs. With our…
@molly_struve
The old system…
91
😦
old system the lone on-call dev was responsible for….
@molly_struve
The old system…
92
😦 Technically debugging and
fixing the problem
Technically debugging and fixing the problem which we know is
a very big task in itself. In addition, they were responsible…
@molly_struve
The old system…
93
😦 Technically debugging and
fixing the problem
Setting a status page if
needed
For Setting a status page if needed. And last, it was their job to handle…
@molly_struve
The old system…
94
😦 Technically debugging and
fixing the problem
Communicating the problem
to the rest of the team
Setting a status page if
needed
communicating the problem to the rest of the team. Needless to say the duties of the
on-call dev were WAY overloaded. With the new system…
@molly_struve
The new system…
95
😦 Technically debugging and
fixing the problem
Communicating the problem
to the rest of the team
Setting a status page if
needed
the ONLY responsibility an on-call dev had was debugging and fixing the problem.
Narrowing the scope was crucial to…
@molly_struve
The new system…
96
😊 Technically debugging and
fixing the problem
Communicating the problem
to the rest of the team
Setting a status page if
needed
improving the on-call experience. It allowed the devs to focus on what they did best, fixing the technical problem at hand. The responsibility of setting the status page was moved…
@molly_struve
The new system…
97
😊 Technically debugging and
fixing the problem
Setting a status page if we
need it
Communicating the problem
to the rest of the team
to the support team. This made sense to us bc the support team is the closest to the customer, and therefore, are the best equipped to communicate any problems. When an incident occurred, the support team
was notified and was responsible for determining if a status page or any customer communication was needed. The responsibility of…
@molly_struve
The new system…
98
😊 Technically debugging and
fixing the problem
Setting a status page if we
need it
Communicating the problem
to the rest of the team
Communicating the problem internally was then moved to the manager of the on-call dev’s team. If updates need to be spread internally across the tech organization during an incident, the on-call dev’s manager
was responsible for doing it. Narrowing the scope of the on-call responsibilities was…
@molly_struve
Overhauling On-Call
99
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6 Narrowing On-Call Responsibility Scope
The last piece of the puzzle when it came to overhauling this terrible on-call system. I am sure many of you are thinking, “That sounds great, but what does an on-call system like that get me? How can my team.. .
@molly_struve
Overhauling On-Call
100
1
2
3
4
3 Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6 Narrowing On-Call Responsibility Scope
benefit from implementing some of these strategies?!” Earlier, I touched lightly on some of the benefits, but now I want to really drive them home and talk about…
@molly_struve
The Payoff
the Payoff of overhauling this terrible on-call system. The first big payoff was…
@molly_struve
102
Improved Alerting
Improved alerting. Originally the Site Reliability team had set up all the alerting and monitoring tools. However, once we split up the alerts and handed them over to each of the 3 dev teams, the dev teams took them and
ran. Because each team felt a renewed sense of ownership over their alerts they started to improve and build on them. Not only did they make more alerts, …
@molly_struve
103
Improved Alerting
but they improved the accuracy of the existing ones. The improved alerts in turn, led to happier on-call developers bc there were less false positives and alerts were tweaked to alert on problems sooner before
they became a bigger issue. Improved alerting…
@molly_struve
1
2
3
4
Improved Alerting
5
104
The Payoff
Wasn’t the only payoff we saw after overhauling the system. As I briefly mentioned, there was…
@molly_struve
105
Sense of
Ownership
A renewed sense of ownership among all of the devs. Even though one team would edit the code that another team supported, there was still a keen sense of ownership for the supporting team. The supporting team
acted as the domain experts over their section of code. The key strategy for ensuring this sense of ownership was…
@molly_struve
106
CODEOWNERS
using the CODEOWNERS file. The CODEOWNERS file ensured that the supporting team was always aware and could sign off on any changes made to the code they supported. In addition, splitting up the code
between the 3 teams meant each team had…
@molly_struve
107
Manageable
Code Chunks
A manageable chunk of code that they could actually learn and support, unlike before where every dev had to support the entire codebase which was way too much for any single person to handle. Shrinking…
@molly_struve
108
Manageable
Code Chunks
down the code that each dev was responsible for along with keeping them updated
on any changes to that code gave devs that…
@molly_struve
1
2
3
4
Improved Alerting
5
Sense Of Ownership
109
The Payoff
Sense of ownership again, and that sense of ownership made them excited to support their code.
Another benefit of the new system was…
@molly_struve
110
Faster Incident
Response
Faster incident response time. Incident response time improved for a couple of reasons. For one, with 3 devs on-call at once and each one of them focusing on a smaller piece of the application, they could…
@molly_struve
111
Identify Problems
Faster
Identify problems faster. As I also mentioned before, each team took time to improve their own alerts so that the alerts would notify them of problems before they turned into…
@molly_struve
112
Identify Problems
Faster
major issues. This decreased incident response times and even helped prevent some incidents altogether. In addition to identifying problems sooner, debugging and figuring out the root cause of problems…
@molly_struve
113
Debugging is
Quicker
became quicker was well because teams were intimately familiar with their alerts and the pieces of code they owned. When a problem arose, each team could debug it much more efficiently than before. Faster…
@molly_struve
1
2
3
4
Improved Alerting
5
Sense Of Ownership
Faster Incident Response
114
The Payoff
incident response is always the goal of any Site Reliability team and to be able to achieve it with a new on-call system was pretty awesome. Another payoff of this new system was that the person who was on-call
was…
@molly_struve
115
Never Alone
Never alone. Having 3 devs on-call at once meant that none of the devs were ever alone when they were on-call. If things started to fall apart in one section of the application, the dev that owned that section knew there
were two others available to…
@molly_struve
116
Never Alone
help if they needed it. Being On-call can be stressful, but knowing that there is always someone easily accessible to help can do wonders for a dev’s confidence. Ensuring that no one is ever alone may seem like
a small positive, but I want to add..
@molly_struve
117
Never Alone
this was the most requested attribute of an on-call system from the devs. Before starting this overhaul process, I spoke with a few devs to get a feel for what they wanted out of the new system and at…
@molly_struve
118
Never Alone
the very top of the list was having help and support when on-call. Don’t underestimate how much a multiple dev on-call system can improve the on-call experience for your devs. Devs…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
119
The Payoff
Never being alone while on call takes the 4th spot in our list of benefits from overhauling this broken on-call system. The last benefit we discovered with the new system was…
@molly_struve
120
Better Cross-Team
Communication
better cross-team communication. As I stated before, each of the 3 dev teams worked across an entire single application. This meant teams were often changing the code that another team was responsible for
during on-call. Having the CODEOWNERS file ensured that the on-call team was alerted to those changes. This..
@molly_struve
121
Better Cross-Team
Communication
not only allowed for a good technical review process but it kept each of the teams up to date on what the other teams were working on and doing. Team communication during incidents also improved. For
example, let's say Team 3 got…
@molly_struve
122
High MySQL Load
Alert
a high MySQL load alert. A ton of the code touched the MySQL database so it was not a guarantee that Team 3 also owned the application code causing the problem. However, because Team 3 was experienced with
the MySQL alert…
@molly_struve
123
High MySQL Load
Alert
they knew how to triage it quickly and efficiently. Once team 3 had triaged it and identified the code that might be the issue they would communicate and work with the on-call owner for that code to determine what
the problem was. This additional…
@molly_struve
124
Better Cross-Team
Communication
team communication helped teams stay current with each other's work while also ensuring that code domain experts were able to review and help with code before it got pushed to production. And with that…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
125
The Payoff
The list of payoffs of overhauling this terrible on-call system is complete. At the top…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
126
The PayoffThe Payoff
Improved alerting. Any small Site Reliability team knows that any outside help you can get with your alerting and monitoring systems is hugely appreciated and benefits everyone. Devs got a renewed…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
127
The Payoff
sense of ownership making them more enthusiastic about their on-call responsibilities…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
128
The Payoff
Incident response got faster thanks to the improved alerting and in depth knowledge of each team
over their on-call components. On-call devs were
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
129
The Payoff
never alone when they were on call giving them peace of mind and confidence. And finally…
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
130
The Payoff
cross-team communication improved which benefited the entire
technical organization. I think we can agree that these..
@molly_struve
1
2
3
4
Improved Alerting
Never Alone
5
Sense Of Ownership
Faster Incident Response
Better Cross-Team Communication
131
The Payoff
Benefits would benefit all of our teams and to get them from an on-call system is huge. If these are the benefits that your team is looking for and your devs are struggling within your on-call system then consider
overhauling it…
@molly_struve
Overhauling On-Call
132
1
2
3
4
Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6 Narrow On-Call Responsibility Scope
With the help of these 6 strategies. Smaller rotations to increase on-call frequency. Split up application ownership so devs can once again feel like they own what they are supporting. Use…
@molly_struve
Overhauling On-Call
133
1
2
3
4
Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6 Narrow On-Call Responsibility Scope
a CODEOWNERS file to further instill that sense of ownership for your devs. Host on-call training sessions for your teams and ensure that those who are on-call always have a support system. Finally…
@molly_struve
Overhauling On-Call
134
1
2
3
4
Smaller On-Call Rotations
Split Up Application Ownership
5
Use a CODEOWNERS file
On-Call Training Sessions
On-Call Support System
6 Narrow On-Call Responsibility Scope
keep the responsibilities for your on-call devs in check so they can focus on what
they do best, fixing technical problems. On-call…
@molly_struve
On-Call Shouldn’t Suck
is something that many people in this industry dread and it shouldn't be that way. If people are dreading on-call then something is broken with your system. Sure, everyone at some point will get that late night or
weekend page that is a pain, but that pain…
@molly_struve
On-Call Shouldn’t Suck
shouldn't be the norm. If on-call makes people want to pull their hair out ALL the time, then you have a problem that you need to fix. I hope this talk has given you some ideas to help you improve your own on-call
system so that it can benefit everyone. Thank you!
T R A C K : S I T E R E L I A B I LT Y E N G I N E E R I N G
THANK YOU TO OUR SPONSORS

More Related Content

Similar to All Day DevOps: Calling Out A Terrible On-Call System

Adprovising – sxsw2012
Adprovising – sxsw2012Adprovising – sxsw2012
Adprovising – sxsw2012Tim Leake
 
Top Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningTop Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningintroNetworks.com
 
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022Pedro Gustavo Torres
 
How To Run a 5 Whys (With Humans, Not Robots)
How To Run a 5 Whys (With Humans, Not Robots)How To Run a 5 Whys (With Humans, Not Robots)
How To Run a 5 Whys (With Humans, Not Robots)Dan Milstein
 
FLYSE Kick Off Event Presentation
FLYSE Kick Off Event PresentationFLYSE Kick Off Event Presentation
FLYSE Kick Off Event PresentationFilip Maertens
 
Joy of Coding Conference 2019 slides - Alan Richardson
Joy of Coding Conference 2019 slides - Alan RichardsonJoy of Coding Conference 2019 slides - Alan Richardson
Joy of Coding Conference 2019 slides - Alan RichardsonAlan Richardson
 
Philip Wilkinson @ FOWA Feb 07
Philip Wilkinson @ FOWA Feb 07Philip Wilkinson @ FOWA Feb 07
Philip Wilkinson @ FOWA Feb 07carsonsystems
 
'10 Great but now Overlooked Tools' by Graham Thomas
'10 Great but now Overlooked Tools' by Graham Thomas'10 Great but now Overlooked Tools' by Graham Thomas
'10 Great but now Overlooked Tools' by Graham ThomasTEST Huddle
 
Business Impossible Transformations
Business Impossible TransformationsBusiness Impossible Transformations
Business Impossible TransformationsDavid Patrishkoff
 
Off-Hours Critical Issue Escalation
Off-Hours Critical Issue EscalationOff-Hours Critical Issue Escalation
Off-Hours Critical Issue EscalationEvan Hamilton
 
Building a Modern Security Engineering Organization. Zane Lackey
 Building a Modern Security Engineering Organization. Zane Lackey Building a Modern Security Engineering Organization. Zane Lackey
Building a Modern Security Engineering Organization. Zane LackeyYandex
 
Elasticsearch 5 and Bust (RubyConf 2019)
Elasticsearch 5 and Bust (RubyConf 2019)Elasticsearch 5 and Bust (RubyConf 2019)
Elasticsearch 5 and Bust (RubyConf 2019)Molly Struve
 
Agile is a 4 letter word - dev nexus 2020
Agile is a 4 letter word - dev nexus 2020Agile is a 4 letter word - dev nexus 2020
Agile is a 4 letter word - dev nexus 2020Jen Krieger
 
Security Isn't Scary and It's Not Rocket Science either.
Security Isn't Scary and It's Not Rocket Science either.Security Isn't Scary and It's Not Rocket Science either.
Security Isn't Scary and It's Not Rocket Science either.Michele Butcher-Jones
 
If i wake evil 360
If i wake evil   360If i wake evil   360
If i wake evil 360John Strand
 
Building a solid foundation for a game startup
Building a solid foundation for a game startupBuilding a solid foundation for a game startup
Building a solid foundation for a game startupVlad Micu
 
Practical exploitation and social engineering
Practical exploitation and social engineeringPractical exploitation and social engineering
Practical exploitation and social engineeringTiago Henriques
 
Communicating in the virtual world
Communicating in the virtual worldCommunicating in the virtual world
Communicating in the virtual worldlarssudmann
 

Similar to All Day DevOps: Calling Out A Terrible On-Call System (20)

Adprovising – sxsw2012
Adprovising – sxsw2012Adprovising – sxsw2012
Adprovising – sxsw2012
 
Innotech2008
Innotech2008Innotech2008
Innotech2008
 
Innotech2008
Innotech2008Innotech2008
Innotech2008
 
Top Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningTop Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social Learning
 
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022
How to (screw up) learn safely in Production :: Talkdesk R&D Learning Fest 2022
 
How To Run a 5 Whys (With Humans, Not Robots)
How To Run a 5 Whys (With Humans, Not Robots)How To Run a 5 Whys (With Humans, Not Robots)
How To Run a 5 Whys (With Humans, Not Robots)
 
FLYSE Kick Off Event Presentation
FLYSE Kick Off Event PresentationFLYSE Kick Off Event Presentation
FLYSE Kick Off Event Presentation
 
Joy of Coding Conference 2019 slides - Alan Richardson
Joy of Coding Conference 2019 slides - Alan RichardsonJoy of Coding Conference 2019 slides - Alan Richardson
Joy of Coding Conference 2019 slides - Alan Richardson
 
Philip Wilkinson @ FOWA Feb 07
Philip Wilkinson @ FOWA Feb 07Philip Wilkinson @ FOWA Feb 07
Philip Wilkinson @ FOWA Feb 07
 
'10 Great but now Overlooked Tools' by Graham Thomas
'10 Great but now Overlooked Tools' by Graham Thomas'10 Great but now Overlooked Tools' by Graham Thomas
'10 Great but now Overlooked Tools' by Graham Thomas
 
Business Impossible Transformations
Business Impossible TransformationsBusiness Impossible Transformations
Business Impossible Transformations
 
Off-Hours Critical Issue Escalation
Off-Hours Critical Issue EscalationOff-Hours Critical Issue Escalation
Off-Hours Critical Issue Escalation
 
Building a Modern Security Engineering Organization. Zane Lackey
 Building a Modern Security Engineering Organization. Zane Lackey Building a Modern Security Engineering Organization. Zane Lackey
Building a Modern Security Engineering Organization. Zane Lackey
 
Elasticsearch 5 and Bust (RubyConf 2019)
Elasticsearch 5 and Bust (RubyConf 2019)Elasticsearch 5 and Bust (RubyConf 2019)
Elasticsearch 5 and Bust (RubyConf 2019)
 
Agile is a 4 letter word - dev nexus 2020
Agile is a 4 letter word - dev nexus 2020Agile is a 4 letter word - dev nexus 2020
Agile is a 4 letter word - dev nexus 2020
 
Security Isn't Scary and It's Not Rocket Science either.
Security Isn't Scary and It's Not Rocket Science either.Security Isn't Scary and It's Not Rocket Science either.
Security Isn't Scary and It's Not Rocket Science either.
 
If i wake evil 360
If i wake evil   360If i wake evil   360
If i wake evil 360
 
Building a solid foundation for a game startup
Building a solid foundation for a game startupBuilding a solid foundation for a game startup
Building a solid foundation for a game startup
 
Practical exploitation and social engineering
Practical exploitation and social engineeringPractical exploitation and social engineering
Practical exploitation and social engineering
 
Communicating in the virtual world
Communicating in the virtual worldCommunicating in the virtual world
Communicating in the virtual world
 

More from Molly Struve

Creating a Scalable Monitoring System That Everyone Will Love ADDO
Creating a Scalable Monitoring System That Everyone Will Love ADDOCreating a Scalable Monitoring System That Everyone Will Love ADDO
Creating a Scalable Monitoring System That Everyone Will Love ADDOMolly Struve
 
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)Molly Struve
 
Building a Scalable Monitoring System
Building a Scalable Monitoring SystemBuilding a Scalable Monitoring System
Building a Scalable Monitoring SystemMolly Struve
 
Cache is King: RubyConf Columbia
Cache is King: RubyConf ColumbiaCache is King: RubyConf Columbia
Cache is King: RubyConf ColumbiaMolly Struve
 
Cache is King - RailsConf 2019
Cache is King - RailsConf 2019Cache is King - RailsConf 2019
Cache is King - RailsConf 2019Molly Struve
 
Cache is King - RubyHACK 2019
Cache is King - RubyHACK 2019Cache is King - RubyHACK 2019
Cache is King - RubyHACK 2019Molly Struve
 
Cache is King: Get the Most Bang for Your Buck From Ruby
Cache is King: Get the Most Bang for Your Buck From RubyCache is King: Get the Most Bang for Your Buck From Ruby
Cache is King: Get the Most Bang for Your Buck From RubyMolly Struve
 
Taking Elasticsearch From 0 to 88mph
Taking Elasticsearch From 0 to 88mph Taking Elasticsearch From 0 to 88mph
Taking Elasticsearch From 0 to 88mph Molly Struve
 

More from Molly Struve (10)

Talk Horsey to Me
Talk Horsey to MeTalk Horsey to Me
Talk Horsey to Me
 
Talk Horsey To Me
Talk Horsey To MeTalk Horsey To Me
Talk Horsey To Me
 
Creating a Scalable Monitoring System That Everyone Will Love ADDO
Creating a Scalable Monitoring System That Everyone Will Love ADDOCreating a Scalable Monitoring System That Everyone Will Love ADDO
Creating a Scalable Monitoring System That Everyone Will Love ADDO
 
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
Creating a Scalable Monitoring System That Everyone Will Love (Velocity Conf)
 
Building a Scalable Monitoring System
Building a Scalable Monitoring SystemBuilding a Scalable Monitoring System
Building a Scalable Monitoring System
 
Cache is King: RubyConf Columbia
Cache is King: RubyConf ColumbiaCache is King: RubyConf Columbia
Cache is King: RubyConf Columbia
 
Cache is King - RailsConf 2019
Cache is King - RailsConf 2019Cache is King - RailsConf 2019
Cache is King - RailsConf 2019
 
Cache is King - RubyHACK 2019
Cache is King - RubyHACK 2019Cache is King - RubyHACK 2019
Cache is King - RubyHACK 2019
 
Cache is King: Get the Most Bang for Your Buck From Ruby
Cache is King: Get the Most Bang for Your Buck From RubyCache is King: Get the Most Bang for Your Buck From Ruby
Cache is King: Get the Most Bang for Your Buck From Ruby
 
Taking Elasticsearch From 0 to 88mph
Taking Elasticsearch From 0 to 88mph Taking Elasticsearch From 0 to 88mph
Taking Elasticsearch From 0 to 88mph
 

Recently uploaded

Class 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm SystemClass 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm Systemirfanmechengr
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Erbil Polytechnic University
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvLewisJB
 
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMchpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMNanaAgyeman13
 
home automation using Arduino by Aditya Prasad
home automation using Arduino by Aditya Prasadhome automation using Arduino by Aditya Prasad
home automation using Arduino by Aditya Prasadaditya806802
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxVelmuruganTECE
 
System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingBootNeck1
 
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptx
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptxGSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptx
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptxshuklamittt0077
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Romil Mishra
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...Erbil Polytechnic University
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxRomil Mishra
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substationstephanwindworld
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfalene1
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxsiddharthjain2303
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Sumanth A
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmDeepika Walanjkar
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleAlluxio, Inc.
 
Immutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfImmutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfDrew Moseley
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionMebane Rash
 

Recently uploaded (20)

Class 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm SystemClass 1 | NFPA 72 | Overview Fire Alarm System
Class 1 | NFPA 72 | Overview Fire Alarm System
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvv
 
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMchpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
 
POWER SYSTEMS-1 Complete notes examples
POWER SYSTEMS-1 Complete notes  examplesPOWER SYSTEMS-1 Complete notes  examples
POWER SYSTEMS-1 Complete notes examples
 
home automation using Arduino by Aditya Prasad
home automation using Arduino by Aditya Prasadhome automation using Arduino by Aditya Prasad
home automation using Arduino by Aditya Prasad
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptx
 
System Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event SchedulingSystem Simulation and Modelling with types and Event Scheduling
System Simulation and Modelling with types and Event Scheduling
 
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptx
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptxGSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptx
GSK & SEAMANSHIP-IV LIFE SAVING APPLIANCES .pptx
 
Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________Gravity concentration_MI20612MI_________
Gravity concentration_MI20612MI_________
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptx
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substation
 
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdfComprehensive energy systems.pdf Comprehensive energy systems.pdf
Comprehensive energy systems.pdf Comprehensive energy systems.pdf
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptx
 
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
Robotics-Asimov's Laws, Mechanical Subsystems, Robot Kinematics, Robot Dynami...
 
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithmComputer Graphics Introduction, Open GL, Line and Circle drawing algorithm
Computer Graphics Introduction, Open GL, Line and Circle drawing algorithm
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at Scale
 
Immutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdfImmutable Image-Based Operating Systems - EW2024.pdf
Immutable Image-Based Operating Systems - EW2024.pdf
 
US Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of ActionUS Department of Education FAFSA Week of Action
US Department of Education FAFSA Week of Action
 

All Day DevOps: Calling Out A Terrible On-Call System