System Reliability That Plants Trees
- 1. System Reliability That PlantTrees Jason Gwartz, Ecosia.org
- 7. Jason Gwartz -Jazz musician - Ambulance driver - Tour guide - Apple Store Genius - Software Developer (Infrastructure Focus) @jasongwartz
- 11. More Kubernetes and machinelearning, please.
- 12. Ecosia DevOps Requirements Resilient -Think about what our users have come to expect... But lean! - Simple to operate and replicate - Developer-focused
- 14. from DevOps importCI, CD, Monitoring from SRE import Playbooks, Postmortems
- 19.
- 20. Ecosia’s Current Solution Structure Teams- cross-functional teams with product ownership SREs/Devops - “infrastructure-focused developers” embedded with application development teams SIG - ownership of platform and tooling
- 21. Ecosia’s Current Solution SharedResponsibility Developers - developers share daytime on-call, write config, deploy Infra Devs - maintain the platform, tools, SRE culture
- 22. Ecosia’s Current Solution Technology Kubernetes- developer-oriented application platform Jsonnet - extensible templating (“libraries for config”) Bazel - consistent build tools and pipeline
- 23. Ecosia’s Current Solution Environments LocalDev - with Bazel or Docker Test + QA - per-branch deployments CI/CD - trunk-based deploys
- 24.
- 25.
- 26.
- 29. Twitter: @ecosia @jasongwartz Github:@ecosia @jasongwartz ecosia.org
Editor's Notes
- #3 A tree is a nice thing. Trees remove carbon dioxide from the air, we know that. Trees bear fruit or nuts for us to eat. Trees provide shade, allowing other crops to grow around them. But trees also redistribute water and keep rivers running, enrich the soil and landscape around them, and help stabilise their environment. But we tear them down anyway. And it’s having an impact on the environment. Deforestation, largely driven by agriculture, is now the second leading cause of climate change after fossil fuels. Trees can help. But don’t take my word for it - the United Nations Food and Agriculture Organisation has said the world’s response to climate change must focus more on forests. (Source https://blog.ecosia.org/fao-report/) (Image from https://i.pinimg.com/originals/02/cd/d4/02cdd4bffc89dd48759fcd0b66bedd2c.jpg via Ecosia image search)
- #4 And Ecosia is doing something about it! Ecosia is the search engine that plants trees. And like all search engines, we give you web search results. Sure. But every search with Ecosia removes one kilogram of carbon dioxide from the air.
- #5 When our users make a web search, we show ads next to our regular search results. Every time a user clicks on one of these ads, we earn some money. Unlike any other search engine, we use this money to plant trees, by supporting reforestation projects around the world. Wait a minute - an internet startup that uses its profits for good? I mean, an internet startup that has profits? How does this make sense? It's honestly really simple - we earn money like any other search engine, then we use it to plant trees. And we’re getting pretty good at it - as of this morning, we’ve planted nearly forty-three million trees, and we currently plant nearly one tree per second. At this rate, in early 2019 we will plant our fifty millionth tree.
- #6 # - At least 1,500 species of vascular plants (>0.5% of the world’s total) are endemic - At least 70% of the original natural vegetation has been lost We plant these trees where they’re needed most - in “biodiversity hotspots” around the world. These hotspots are places with more than fifteen hundred species of vascular plants that don’t exist anywhere else in the world, and where 70% of the natural vegetation is lost. Our trees are currently being planted in twelve of these biodiversity hotspots, employing thousands of people. In some cases, the trees we plant are jointly owned with a local community, meaning they can create an economy for themselves with the produce gathered from the trees, like mangos or avocados. And in addition to helping communities, our trees have, so far, removed 2 million tonnes of carbon dioxide from the atmosphere. #In Morocco, for example, there are communities of women who previously weren’t able to finish school, needing to go work to help support their families. By paying them to plant and maintain trees, they can use their time outside of the planting or harvesting seasons to return to school.
- #7 Here’s a short video showing tree-planting work we’re doing in Tanzania, and how our trees have redefined the landscape.
- #8 Now, who am I? I’m Jason - I like trees, but mostly I like distributed systems and incident response. A bit about me: after short careers as a jazz saxophonist and an Apple Store Genius, I moved from Canada to Europe, eventually landing in Berlin, where I now work as a backend developer focusing on infrastructure.
- #9 How many people look at a graph that looks like this every day? Well, me too - we have lots of monitors for request rate, latency, resource utilisation, error rate, etc. But this isn’t that. What do you think this graph is showing? It’s a rough estimate of trees planted per second, in real time. We monitor this, because it’s what matters to our users! This laser focus on our mission leads to some a lot of requirements all around the company. As you can imagine, the more money we spend on running the company, the less we have for our tree-planting projects; I know all companies, especially startups, care a lot about how much money is being spent - but in our case, it can often be literally measured in trees. For example - I’m having a lot of fun being here in Galway, and I’d like to be able to come again, so it’d really help me out if you all grab the browser extensions and start using Ecosia!
- #10 And our users hold us accountable to that! By disclosing publicly our financial records, we’re letting our users make sure we behave in a responsible way, that is in line with our values. That means having enough developers to do the technical work we need to (and want to) accomplish, but staying lean. In addition to running the Ecosia search service, spreading the word about it, and planting lots of trees, we also put aside some money so we can occasionally do things like...
- #11 Try and buy a forest to save it from being torn down. We recently offered RWE, a German energy company, one million Euros to buy the last remaining piece of the ancient Hambacher Forst. This energy company was planning on tearing down the trees to do coal mining, as they’ve done with the rest of the forest. They declined, but we and our community are still fighting to keep the forest from getting torn down.
- #12 Okay, enough about trees. What does this have to do with DevOpsDays? How do I, an infrastructure engineer, fit in? Well, everything I’ve just said becomes part of the requirements for our development and operations strategy.
- #13 When users install the browser extension, or in some other way configure their browser to search with Ecosia, they have some expectations about how a search engine behaves. And they definitely have some expectations about how reliable that search engine is. How, you could say, that search engine’s site is reliably engineered… But our users also need us to respect their contributions, and put most of our income towards trees and our other projects. So a big developer team, or even a dedicated SRE group, isn’t really in the cards.
- #14 So we develop and maintain Ecosia search for our millions of users with a team of around ten developers, only two of which focus primarily on infrastructure. This provides some interesting and challenging constraints to how we can run devops. For example, those two developers who focus on infrastructure - well, one of whom is on this stage, and luckily for me, the other one is currently on-call. After a while of operating this way, the infrastructure-focused developers realized that doing operations like "throw it over the fence" was not going to work for us - there was too much to be done between maintaining the platform and being on call for us to also be doing production reviews and writing everyone's deployment configurations. The easiest way to demotivate infra devs is make them write YAML config files all day (even if it’s cool YAML like Kubernetes).
- #15 So what we definitely, definitely did was look at our favourite aspects of Devops and SRE, made a year-long transition plan, got all the stakeholders involved, and set clear transition goals. No, not really.
- #16 The beginning of a solution for us infrastructure-focused developers was in embedding ourselves in the product development teams.
- #17 This helped us find pain points in the development workflow, and gave us first-hand knowledge of what we should be automating. But it didn’t really reduce the operational overhead. We still needed to find a way to reduce our toil and get back to coding.
- #18 Now, the infrastructure-focused developers split our non-toil work between infra tasks and "normal" dev work. We also changed our responsibilities to more evenly share the load of toil between the infra devs and all other developers. But who owns the platform and tooling? #We've built our platform on top of Kubernetes and other technologies like Bazel and Jsonnet in a way that enables all developers to be involved in deployments, capacity planning, and postmortems.
- #19 Well, us as infrastructure-focused devs created a Special Interest Group - an opt-in set of developers from both product teams that are in charge of running the platform.
- #22 Lead to daytime on call - escalations are cheap - post-mortems are a checklist before escalating
- #23 Our goal as infrastructure developers is to accelerate what’s possible for all application developers in the company (which also includes the infra devs!). So we try and build a platform and tools that make it seamless to integrate into our development and production environments with the technologies we use most. Bazel is an open-source project by Google. Jsonnet is an open-source project by Google. Kubernetes is an open-source project that was kick-started by….Google. Search engines of a feather fly together.
- #24 We also streamlined the testing and QA process by introducing per-branch deployments. Previously, there were a few staging-ish test environments which developers would take turns on, testing their changes and getting QA on new features. As our developer team grew, from three to ten, this wasn’t practical when multiple new features were under testing at the same time. Our new per-branch deployments are created simply by opening a PR. Our CI pipeline creates a Kubernetes namespace, then, similar to Frank’s talk yesterday about Blast Radius, uses Bazel’s `query` functionality to check which applications in our monorepo are involved in any given PR. We then deploy into this namespace all necessary Kubernetes objects to test the application or applications involved in the PR.
- #25 Bazel lets us make our CI/CD pipeline 100% replicable locally, so rollbacks can be done lightning-fast. This is a grab from the best website on the internet, outage.party
- #26 And adopting a culture of calculated risk and no-blame postmortems means that developers can own their own deploys. All developers are also involved in the process of monitoring applications, which comes in handy when they take their shift of the daytime on-call. When our developers deploy new applications,
- #28 Remember that graph from the beginning? That’s user behaviour, tracked via Kubernetes and nginx-ingress, collected via Prometheus, and visualised using Grafana. So while it’s not a measure of trees actually going into the ground, it’s a projection of trees we will be able to plant - and it’s a good representation of whether or not our systems are healthy. Working on infrastructure can often be so abstract that I find it really valuable to tie our systems' health back to something more tactile. It also gives us developers a lot of pride to see the tangible impact on the environment that our software can have.
- #29 As an example of the impact our software engineering is having, I want to show this short video of Almaz, a woman in Ethiopia who will tell her story about tree planting and coffee. It's amazing to think that the work we do on silly things like platform stability and cost optimization contributes to this.
- #30 Thanks for listening to our story! Come find me or tweet at me if you want to get an Ecosia sticker, or just chat about your favourite tree. Thank you!