Don't Fear Failure
•Download as PPTX, PDF•
0 likes•453 views
The speaker discusses their many failures in technology careers over 12 years. Some key lessons are that failure is inevitable, it's okay not to be a "success story", no matter how hard you work failures can still happen, it's important to ask for help when you don't know what you're doing, own your mistakes, and have a plan for when failures occur so you can learn from them and try again. The speaker provides examples of failures with companies going under, building unsuccessful products, data migrations going wrong, but in each case emphasizes learning from mistakes and having contingencies to remedy problems.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (12)
Similar to Don't Fear Failure
Similar to Don't Fear Failure (20)
More from Sean Kelly
Recently uploaded
Recently uploaded (20)
Don't Fear Failure
- 1. DON’T FEAR FAILURE A TALK ABOUT SOME OF MY MANY FAILURES
- 2. SOME THINGS ABOUT ME • Most people call me Stabby (@StabbyCutyou) • I’ve never started my own company • I’ve never written any popular or even recognizable software • I am definitely not good at “Computer Science” • I almost dropped a live database once
- 3. KEY TAKEAWAYS FOR THIS TALK • I am a huge failure • Just have a plan • Own it • It’s OK to fail
- 5. Set out to do a thing Did it work?YES NO
- 7. OVER 12 YEARS OF MY LIFE • .NET, Javascript, Ruby, SQL, Go • Love what I do • Screw up tons of stuff in all the time • Manage to get enough right to get by
- 8. LESSONS TO LEARN FROM FAILING
- 9. LESSON ONE: ITS OK IF YOU’RE NOT A SUCCESS STORY
- 10. OUR CAREERS ARE DEFINED BY “SUCCESS” • Large valuations • Perks • Bonuses • Exits • Renown
- 11. Most of won’t have that kind of success
- 12. I THOUGHT MY FIRST JOB WAS GOING TO BE “IT” • Guess how that went • One day got an email my company sponsored gym membership was cancelled • Next day, company was closing down • Had to tell a recent hire who just started two weeks ago that no, this was not a joke.
- 13. I LEARNED A LOT ON THE WAY DOWN • Built a solid set of SQL skills • Learned a lot of ways you probably shouldn’t build a desktop app • Exposed to running a dreadful meeting every week • First encounter with bizarre interoffice politics / relationships
- 14. LESSON TWO: NO MATTER HOW HARD YOU WORK YOU CAN’T ALWAYS STOP FAILURE
- 15. FOR EXAMPLE… AWS • AWS is great • Network can be unreliable • Resiliency / Durability in your stack only goes so far • Partitions happen
- 16. ANOTHER JOB • Built the worlds worst Foodler clone • Amongst the worst times in my life • Barely slept, constant rewrites, no guidance or design • It had no chance in the market
- 17. There was no way to save it It was simply destined to fail I beat myself up over it for a long time
- 18. LESSON THREE: ASK FOR HELP
- 19. IT’S OK TO NOT KNOW WHAT YOU’RE DOING • No one really does at first • Many of us still don’t • Find someone who does
- 20. FOR EXAMPLE… RIAK • Built around the idea of replication • Writes get copied to 3 nodes (by default) • Highly Available - data always accessible • Conflict resolution: Siblings, Vector Clocks
- 21. WE WERE CONVERTING TO A NEW JOB SYSTEM • It kept hanging • I would wake up every few hours to check it • Worked with other engineers to solve the issue • Figured out how to predict it, put in mitigations • Realized no-one cared so long as it got fixed
- 22. LESSON FOUR: OWN YOUR MISTAKES
- 23. FOR EXAMPLE… GO • Writing a lot of Go • Lots of gripes • Fell in love with error handling in Go • Responsibility strictly delegated upward until you “care”
- 24. WE TRIED TO BUILD OUR OWN QUEUE • Sure was fun • But did not succeed like we hoped • Learned a lot of great information • But ultimately wasted a lot of time and effort
- 25. LESSON FIVE (a): IT’S OK TO FAIL…
- 26. LESSON FIVE (b): …SO LONG AS YOU HAVE A PLAN
- 27. FOR EXAMPLE… TCP • Built around the idea of missing data • User land VS Using your brain • Defends against problems in-depth
- 28. I MOVED A BUNCH OF FINANCIAL DATA • This was a big one • We moved some finance data into a more scalable system • During the migration some data was initially lost • We didn’t catch it • We paid out accounts based on incorrect data
- 30. BACK TO MY MAJOR SCREWUP… • Multiple copies of the missing data • We built scripts to compare datasets and find missing records • We repopulated the missing data • We worked with finance to fix the numbers • Did not get fired
- 31. CULMINATION OF ALL LESSONS LEARNED • We didn’t succeed like we hoped • We had multiple layers to verify, but not enough • We had a plan incase this happened • Asked for help from the right folks • Stepped up, personally accepted responsibility immediately
- 32. Set out to do a thing Did it work?YES NO Did you learn? YES NO Try Again
- 33. IN THREE DAYS I START A NEW JOB
- 34. LESSONS • Not everyone is a success story • Failure is inevitable • Ask for help • Own your failures • It’s ok to fail, just have a plan for it
Editor's Notes
- 1080i 60herz mirroring off Make sure no one can see your speaker notes. They contain mighty secrets that must not be revealed.
- I would say that, given the last 12 or so years of my life, I’d be considered something of an expert at failure. I’ve had a lot of it. I’ve had a lot of success too, but while success in my experience is more of the culmination of the lessons you’ve learned over time, failure is what contributes to learning those lessons. Anyone can make something work and come back for more, but can the same be said for someone who fails a few dozen times? Coming back for the next attempt is what it’s all about in this industry.
- I’ve spent a good deal of my life doing this, between .NET, JS, Ruby, SQL, Go, and a bunch of others, python, a tiny bit of erlang, and a very unhappy trip down Scala lane. But I really do love what I get to do. However, that doesn’t mean I don’t ruin things from time to time. Many times, actually. Somehow, I do just right enough to get by. But there are a lot of misses amongst my swings.
- We all see articles and blog posts about these amazing success stories, face books, twitters, twitchtv, uber, etc etc. Something that came along and somehow knocked the world on it’s feet overnight (although “overnight” typically means a few years and a lot of hard work, but I digress). And here you are, working on Uber for Zucchinis, or Blue Apron for Snakes, or startup that takes Soylent and turns it into Vape liquid PS If anyone goes and does that, you owe me 5% of the business.
- I’m not going to tell you, settle for less, or undervalue yourself, or desire to work at oddball go-nowhere companies. Quite the opposite. Don’t sell short the ability to learn and grow by trying new things, and not worrying about whether you succeed or not. You can learn a lot from a company that is, for lack of a better term, a little bit of a shitshow. Plus, and I do genuinely believe this is true - you’ll never appreciate what Success actually tastes like, unless you’re pretty familiar with the shovelfuls of dirt that Failure will feed you.
- And I really mean, we thought it was going to be our big out of college success story. My friend got me the job there, and we really believed the place was going to just somehow print money. Of course, we were wrong. He figured it out sooner than I did. He quit about a year and change after I started. I still thought it was going to work. That poor lady, she really thought it was some kind of odd joke. It was really depressing to have to explain to her that the company was shutting down, and I knew just as much about it as she did.
- Basically my job involved running SQL all day. I ran procedures, found bad data, fixed it. I built a terrible desktop app to handle some of the minutia, and learned a lot of ways not to build an app in the process I had to run a terrible, just nightmarish weekly meeting that I didn’t even realize was my job to run for the first few months And I had my first run in with really… unprofessional behavior, lets say, on a routine basis. So it taught me a lot about what to look out for, and what kind of things to distance myself from to make the most of my career.
- Look at something like AWS. Now, I love AWS, I think it makes a lot of sense to use. But if we’re being real, their network is unreliable. You can’t build a system in AWS, or really any cloud provider where you’re not on a dedicated backbone, where the network isn’t a problem laying in wait. You can work your ass off to add resiliency or durability to your communication channels, but you can’t prevent an issue where the network partitions.
- Now, a time when this lesson manifested in my own career, was at my next Job. Initially, I had to build a Foodler clone And I really mean the worst. I’m not gonna give away any names, I’m almost positive no one ever actually used it. But it was a .NET webforms 1.1 disaster of a mess. Building it was one of the most miserable periods in my life. I wasn’t sleeping or eating right, i worked endlessly on rewrite after rewrite when nothing seemed to work, and with no real strong direction from above. It I also had no experience building something of this scale, so that didn’t help either. The company that contracted with us also had no real plan, or hope, but that’s a different story
- I did everything I could, everything within my limited powers at the time, to try and make this thing work. But no matter what I could do, I couldn’t save a doomed project from itself. That might sound like a cop out, but I must have beat myself up over this for the next year, before I finally let it go. I worked for that company for another 3 years or so, and in that time we had as many CEOs and nearly as many rewrites of the platform. The final CEO decided to sell the company in pieces off to his friends. He offered to have me be part of the sale. I turned him down. Another engineer accepted his offer. And after a few more years of work, that company finally got bought out, and he finally turned the platform into a success. So even when success finally came, it took several changes of hands, several years, even more rewrites, etc.
- Asking for help has always been tough for me - I like to think of myself as someone who can figure things out, given enough time or resources. But there’s the rub - time (which is a resource itself) and resources are limited. The company or the project doesn’t always have time for me to spin my wheels on something. I routinely encounter issues that I have no prior experience with, or knowledge around. But these days, I’ve realized the best first step is to seek help. Talk to other engineers who are familiar with the space. They can help you avoid all the early, common mistakes, or give you things to think about that you might not have even been aware to ask questions on. Learn from the mistakes that others have made, reach new mistakes more quickly, reach knowledge more quickly
- Need to add “technical” slide talking about design of distributed datastores, built with failure in mind. At some point, assumptions break down and a promise is broken. Talk about how Riak defends against something like that with K=3, siblings, and vector clocks. “and if you can use vector clocks correctly,… then you are probably smarter than I am, let’s be honest”. When riak fails, it asks for help from the cluster. No one node is designed to share too strong of a burden.
- It was a pretty rough few weeks for me. Not a lot of sleep, in over my head a bit, struggling to find a solution. The best thing that happened was talking with other engineers who knew the space better. They had been writing ruby on linux for years. They taught me about monitoring for file leaks, using gdb, helped trace down where the problem was happening. I continued to use the tricks and knowledge I gained from figuring this problem out throughout the rest of my time at that job, and likely will keep using them throughout my career.
- So, the last 2 years or so, I’ve been writing a lot of Go. And I won’t lie, there’s plenty of things with Go that I’m not in love with. But one aspect that has grown on me over time is error handling. Now, a lot of people don’t like how it works, where errors are explicitly passed back from a lot of function calls. At first, it’s unnatural, error handling feels too verbose, people just ignore them. But once you get used to it, it actually greatly simplifies a lot of code. Yes, the boilerplate for dealing with them is a bit annoying. Yes, the current implementation falls a bit short of where it should be. But the way you build systems, where responsibility for problems is essentially strict delegation up the call stack until you care enough to take action on it, is something I’ve genuinely come to appreciate.
- This one was always really bitter sweet, because I loved working on this project. It just didn’t scale like we believed it would, and ultimately had to abandon it. But it did work! However, it was a bit of a black eye for my reputation, and the only choice was for me to own it head held high. It was my call, I made it, and it was not the right call to make. There were a lot of factors that I could point to to explain why it failed, but there was no excusing it. I’ll make plenty more bad calls in my life, and hopefully a few good ones. You need to be ready to own the outcome of the decision, come what may. If you’re not ready to be responsible for big choices, seek out advice and help from those around you. Kind of nice how these lessons all kind of dovetail into eachother, isn’t it?
- “And if we have any UDP fans in the house… I’m surprised any of you made it here” Lot of blog posts about the “benefits” of rolling your own TCP stack, but unless you’re one of the three companies in the world who would reap those benefits… good god just use the normal stack. I mean, come on. TCP is built around the idea that failure can and will happen, so we better have a plan to recover from it. Either missing packets, or things get out of order, TCP has a plan to defend against multiple problems
- I still remember sitting at home, talking to my wife, when suddenly my phone vibrates. I got a slack note from one of the guys working on this with me. I don’t remember exactly what it said, because quite frankly the next few seconds were a blur. But it essentially said that we lost some data in the migration, and only found out because partners complained about their payouts. This was a huge one. I braced for the worst, because you can screw a lot of things up and be forgiven, but money is one of those things that just doesn’t get swept under the rug. Luckily, we had an existing process in place to adjust any incorrect balances, and we had multiple copies of the data that we could reference to reconstruct the missing records.
- Haha, fooled you Theres also a picture of my sweet cat as well
- So, because of how dangerous and sensitive the data was, even though we did all this diligence around preventing problems, we knew they could still happen. So we kept multiple copies of the missing data, in multiple formats. Why we had them in different formats speaks more to certain business requirements than anything else, but we leveraged that to make sure that if there was any kind of failure at any level, we had another level to go back to, even as far down as what you might call the “cold storage” layer.
- We weren’t a success story. The project was definitely going to be tarnished by this, and for the next 2-3 months any billing issues (there always are some) would be sent our way to make sure we didn’t cause. We thought we had all of our bases covered and verified the data was correct, but clearly we had a blind spot in our testing and verification strategies However, we planned ahead and made sure we kept multiple copies of the data in a place where, if we had to, we could reconcile against the original and new storage systems, and replace anything that got lost. When the problem hit, I first reached out to a few people to find out everyone that needed to be part of the solution. I didn’t just ask one person, I asked multiple people then asked the people they thought should be involved. Since it was finance, and most people are too scared to touch it, the loop of folks we involved turned out to be fairly tight. We could have had a too-many-cooks problem here, but we knew to caste a wide net, then throw back anything we couldn’t use quickly. Finally, the first thing I did was give a multiple page write up of the issue, including timelines, how it happened, how we missed it, how we planned on fixing it, how we planned on preventing it from occurring again, and taking complete ownership of the problem. Now, this was finance, and I’m just an engineer. Theres only so much I could directly do. But I essentially became something of a foreman. I directed communication, fed the right people updates, got my hands dirty diagnosing and fixing the issue, and worked with another engineer to verify what we were doing was correct, and all the numbers lined up again. I could not do it all on my own, but I could make sure I was the nucleus of the problem solving efforts. And they didn’t fire me for it.
- And I just want to apologize to them in advance, because I am definitely going to screw up. So, thanks in advance for putting up with me, new job!
- So, 5 lessons on failure: Not everyone is going to be a success story, failure is inevitable, when you do fail (or even before you do) ask for help, own it - because everyones gonna know anyways, and at the end of the day, it’s perfectly fine to fail so long as you have a plan, and you’re able to learn from it
- Really, I mean thank you to anyone who came here and sat through me ramble on and embarrass myself with stories of how big of a screw up I am. If I’ve encouraged one person to just get over their fear of screwing up and try something new, if I’ve helped give one person the framework they need to think about how to turn failure into success, hell if I’ve kept one person entertained with those sweet dog pics, then I am deeply and profoundly satisfied to have been able to give this talk. I want to thank the folks at Abstractions for letting me come talk, again all the folks in this room for listening, all my former coworkers for dealing with a major failure like myself, and above all my wife for supporting me in all the things I do. Thanks