Adding An Operator to Airflow: A Contributor Overflow Exception
•
0 likes•125 views
The story of adding the Singularity Operator to Apache Airflow, 2019-2010. Presented at the first Airflow Community Summit in 2020.
Recommended
More Related Content
Similar to Adding An Operator to Airflow: A Contributor Overflow Exception
Similar to Adding An Operator to Airflow: A Contributor Overflow Exception (18)
More from Vanessa S
More from Vanessa S (20)
Recently uploaded
Recently uploaded (20)
Adding An Operator to Airflow: A Contributor Overflow Exception
- 1. Adding an Operator to Airflow A contributor overflow exception Vanessa Sochat, PhD Research Software Engineer Stanford Research Computing Center
- 2. Why am I here?
- 3. Misaligned expectations: strained interaction
- 4. Once upon a time...
- 8. Once upon a time...
- 9. people
- 21. What happened?
- 22. “This looks neat!”Pusheenocat found Apache Airflow
- 23. - I love Python - I’m good with containers - I maintain singularity python Should I contribute?
- 24. “Yes! I should add a singularity operator”Should I contribute?
- 25. “I’ll mimic another operator.”How do I start?
- 26. What Happened? On March 5, 2019 Pusheenocat opened a pull request
- 27. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times
- 28. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months
- 29. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months - A community member said “Hey, this would be useful!”
- 30. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months - A community member said “Hey, this would be useful!” - Stalled for another 4 months
- 31. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months - A community member said “Hey, this would be useful!” - Stalled for another 4 months - Closed
- 32. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months - A community member said “Hey, this would be useful!” - Stalled for another 4 months - Closed - “They forgot about me.”
- 33. What Happened? On March 5, 2019 Pusheenocat opened a pull request - It was reviewed 6 times, 80 total comments, green multiple times - The review stalled for 1 month, another month, and then 4 months - A community member said “Hey, this would be useful!” - Stalled for another 4 months - Closed - “She wasn’t persistent enough. She didn’t follow instructions.”
- 36. contributors reviewers I care about the community expectations incentives She cares about the community
- 37. contributors reviewers I care about the community The review process is streamlined. I want an efficient review. She cares about the community
- 38. contributors reviewers I have to do this for my job. expectations incentives Why doesn’t she care about the community?
- 39. contributors reviewers expectations Why did they forget about me? The contributor needs to be persistent. incentives
- 40. Who is involved?
- 41. Reviewers
- 44. reviewers incentives expectations The long term contributor
- 45. Contributors
- 46. The job seeking deputy contributors incentives expectations
- 50. I’ll tell ya’ the story!
- 51. Adding an Executor to Airflow A contributor overflow exception Once upon a time, on March 15, 2019 Pusheenocat opened a pull request...
- 53. ✅ The pull request template is often the first interaction.
- 55. Contributor bandwidth Is limited ✅ Be selective about what you ask contributors to do
- 56. Adding an Executor to Airflow A contributor overflow exception One day later...
- 59. ✅ Reviewing promptly and answering questions
- 66. Expectation: copy this command, and done Against expectation: I want to learn how to do this
- 67. Expectation: copy this command, and done Against expectation: I want to learn how to do this
- 68. Expectation: copy this command, and done Against expectation: I want to learn how to do this
- 69. Expectation: copy this command, and done Against expectation: I want to learn how to do this
- 70. ✅ Have good communication for each step
- 71. ✅ Have good communication for each step
- 72. ✅ Be empathetic to your contributor
- 74. Adding an Executor to Airflow A contributor overflow exception One month later...
- 80. Adding an Executor to Airflow A contributor overflow exception Four months later...
- 82. “Hello I’m a robot here to tell you that you were again forgotten, and you are getting closed.”
- 83. Talking to the bot…
- 84. Talking to the bot… ✅ Have a community strategy for managing reviews
- 85. Hey, this would actually be useful, what’s up?
- 92. Why are you editing other files?
- 93. I didn’t?
- 94. Adding an Executor to Airflow A contributor overflow exception Meanwhile on Twitter...
- 99. We needed her to do something she didn’t know to do.
- 101. You are not humble or persistent enough. You are not empathetic. You don’t know how to rebase. You don’t know the right tools to use.
- 102. You must meet a quality bar you don’t understand
- 103. ✅ Be supportive, not patronizing
- 104. ✅ Be supportive, not patronizing
- 105. I understand this, but a new contributor needs more support from the community. A new contributor needs guidance. I didn’t know what to do, and was forgotten. This is not a community I would come back to. ✅ Communicate clearly, and respectfully
- 106. Adding an Executor to Airflow A contributor overflow exception A new contributor needs guidance...
- 117. Contributor bandwidth Is limited ✅ Be selective about what you ask contributors to do
- 118. Adding an Executor to Airflow A contributor overflow exception Meanwhile on Twitter...
- 119. We have best practices and it’s on you to know them!
- 120. We have best practices and it’s on you to know them! I can’t be expected to know the culture for a new community.
- 121. We have best practices and it’s on you to know them! I can’t be expected to know the culture for a new community. I am burnt out on this.
- 122. Devote more time to understand our community!
- 123. Devote more time to understand our community! Join another Slack channel!
- 124. Devote more time to understand our community! Join another Slack channel!
- 125. How to move forward?
- 129. Adding an Executor to Airflow A contributor overflow exception Why is this so hard for you?
- 131. ✅ Do not target individuals; do not be mean.
- 138. Adding an Executor to Airflow A contributor overflow exception Success!
- 140. pusheenocat reviewers Incentives Have fun Add Singularity operator Expectations Read all documentation Keep up to date Push for next steps
- 141. pusheenocat reviewers Expectations Supportive environment Incentives Contributor is advocate Informed contributor
- 142. What did we learn?
- 143. What did we learn? - A project cannot only assume one kind of contributor - Not all contributors want to please, even if they want to add a feature
- 144. What did we learn? - A project cannot only assume one kind of contributor - Not all contributors want to please, even if they want to add a feature - You should absolutely not bring personal issues up in a review - “Why is this so hard for you” does not belong there.
- 145. What did we learn? - A project cannot only assume one kind of contributor - Not all contributors want to please, even if they want to add a feature - You should absolutely not bring personal issues up in a review - “Why is this so hard for you” does not belong there. - Contributor and developer have shared responsibility for making progress - Clearly communicate expectations and ideas - The contributor is responsible as long as tests don’t pass, requests for changes present - The reviewer(s) are responsible in all other cases
- 146. What did we learn? - A project cannot only assume one kind of contributor - Not all contributors want to please, even if they want to add a feature - You should absolutely not bring personal issues up in a review - “Why is this so hard for you” does not belong there. - Contributor and developer have shared responsibility for making progress - Clearly communicate expectations and ideas - The contributor is responsible as long as tests don’t pass, requests for changes present - The reviewer(s) are responsible in all other cases - Expect the contributor to be informed within the scope of the contribution.
- 147. Adding an Executor to Airflow A contributor overflow exception Technical discussions require empathy.
- 148. How to be empathetic? - Why are they here? - The other party has goals that drive their incentives and actions. What are they?
- 149. How to be empathetic? - Why are they here? - The other party has goals that drive their incentives and actions. What are they? - What are our shared goals? - Focus on shared goals, and use them to re-center.
- 150. How to be empathetic? - Why are they here? - The other party has goals that drive their incentives and actions. What are they? - What are our shared goals? - Focus on shared goals, and use them to re-center. - What don’t I know? - What expectations does the other party have that I should ask about?
- 152. contribution
- 153. contribution
- 154. contribution
- 155. people Open Source!
- 156. people Open Source!
- 157. people Open Source!
- 158. people Open Source!
- 159. people Open Source!
- 160. Adding an Executor to Airflow A contributor overflow exception Each of us is living our own developer story...
- 161. Adding an Executor to Airflow A contributor overflow exception We’re part of a much bigger open source story.
- 162. Adding an Executor to Airflow A contributor overflow exception The community is responsible for the experience.
- 163. Adding an Executor to Airflow A contributor overflow exception You are responsible for creating positive experience.
- 164. Adding an Executor to Airflow A contributor overflow exception Who is the hero of the story?
- 166. Adding an Executor to Airflow A contributor overflow exception Be like Elmo!
- 167. Adding an Executor to Airflow A contributor overflow exception Twitter and GitHub @vsoch
Editor's Notes
- # Adding An Operator to Airflow: A Contributor Overflow Exception Engaging with a new community is a common experience in OSS development. There are usually expectations held by the project about the contributor's exposure to the community, and by the contributor about interactions with the community. When these expectations are misaligned, the process is strained. In this talk, I'll discuss a real life experience that required communication, persistence, and patience to ultimately lead to a positive outcome. ## Biography Vanessa Sochat is a research software engineer for the Stanford Research Computing Center. She received her PhD in Biomedical Informatics in 2016, and stayed at Stanford to focus on open source software development for scientific reproducibility. Her work includes development of container technologies, workflow software, and recipes for continuous integration. She is passionate about programming and system design, and continues to run the Singularity Hub container registry and maintain a large set of open source libraries. When not programming, Vanessa can be found eating avocados, recording podcast episodes or fun videos, making dinosaur noises, and running outside in the snow.
- So let’s start with a very meta question. Why am I here? Well probably the same reason that you are here - today we want to talk about community. Engaging with a new community is a common experience in OSS development. There are usually expectations held by the project about the contributor's exposure to the community, and by the contributor about interactions with the community.
- When these expectations are misaligned, the process is strained. In this talk, I'll discuss a real life experience that required communication, persistence, and patience to ultimately lead to a positive outcome.
- Once upon a time...
- There were contributors, and there were reviewers.
- The contributor might say “Hey! I’ve made these changes to the codebase. Could you please take a look?”
- And the maintainers might say “Ah, you’ve made changes to the codebase! Let me take a look.” But no, we’re already off a bit, because I can’t just shove people into such stark boxes. Let’s try again.
- Once upon a time...
- There were people.
- And they cared a lot about different kinds of open source software. Containers! Programming languages! Workflow managers!
- At the intersection of their communities is where collaboration happened. But people have their favorites so...
- Let’s scootch them a little closer.
- So let’s start with a special node - the open source developer, who can actually sit sometimes right in the middle of a web of projects. This is Pusheenacat, the open source developer.
- So what happened to Pusheenacat?
- Just to make this fun, I’m going to introduce to you three kinds of archetypes to help us think about this discussion.
- Both contributors and reviewers have expectations and incentives.
- Here is an easy example. A contributor might really care about the community, and the reviewers expect them to as well. This translates to simple things like being respectful, putting time and care into writing a response, or even just the emojis that you put on an issue comment.
- Both parties want the review to be efficient.
- The maintainer is someone that is uniquely responsible for the survival and longevity of the software. There can be more than one of course. I chose this particular picture because in a lot of cases, a maintainer plays a large role in interacting with people. In helping them, handling issues, or creating awareness for the library. This is a role that comes with a LOT of responsibility. Maintainers are generally responsible for the project. This means it’s sustainability and its quality. This is stressful. Should a new feature be added, is it really needed or is it going to make maintaining the project harder? And also, this person needs to learn how to act cordial even in the face of a disgruntled contributor. The maintainer needs to learn to say no. Are the pull requests being reviewed, and are the issues being addressed? There is a lot of weight on the maintainers shoulders.
- The long term contributor started contributing to the project at some point, and became familiar with it, or liked the community enough to stick around. He or she contributes by way of helping with issues, or being a reviewer on pull requests. This is a lot less stressful role than a core maintainer because they have less of a role in taking care of the long term sustainability or design of the project.
- The job seeking deputy: The job seeking deputy is a contributor that is trying to impress you, the project. They are going to be overly polite, read every single piece of documentation that you have, and generally kiss ass. They need to look good because they want to prove themselves to you.
- The worker bee: The worker bee is there because he has to be. Maybe his employer said “You need to do this thing for this software” OR maybe he’s getting paid by someone directly. It doesn’t matter. Because it’s a matter of business, this guy is incentivized to do everything right. He might not be aiming to please the project, but he’s possibly aiming to please his employer, and not get in trouble.
- The wandering baker (is really interested in his or her craft, baking, and literally wanders around the open source landscape looking for new recipes /things to try. This person will put in the time and energy to share her recipe, but she isn’t going to stick around to be a permanent fixture in the bake shop.