Dublin AWS Community day 2020 - Ten bad startup technical strategies for startups
•Download as PPTX, PDF•
0 likes•78 views
This document discusses 10 bad technical strategies for startups, including 5 good ones, and was written by Brian Scanlan from boringtechnology.club. It warns against building in security as an afterthought and biases towards high-level services over low-level components.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Dublin AWS Community day 2020 - Ten bad startup technical strategies for startups
- 1. Ten bad startup technical strategies for startups
- 2. (also 5 good ones)
- 3. Disclaimer
- 40. Bias towards high level services
Editor's Notes
- Hello, and thank you for joining me from my spare bedroom in Swords. It’s not quite the as cool as being on stage like at last year’s event at Dublin Castle, but the commute and coffee are better. My talk is going to be about a bunch of technical strategies for startups to avoid.
- As a bonus I’m also going to cover 5 good ones!
- These strategies are intended as reasonable defaults. They are my opinions. They are not rules, and they certainly will not fit every situation. Far from it. The strategies are opinions formed on the basis of my experiences working in technology, seeing what works in practice in many situations and talking to others about their strategies and successes. Strong opinions can make somebody come across like a jerk, which is not my intention at all!
- I work in Dublin for Intercom, a mid-stage startup. We do customer messaging for
- If you’ve paid any attention to many loud marketing efforts over the last few years, multi-cloud appears to be a thing, right? According to the Oxford English Dictionary, multicloud is deploying your application to a heterogenous cloud-based platform spread across multiple cloud providers. That doesn’t sound very bad.
- According to Corey Quinn, the world’s most notorious Cloud Economist, it is “the worst practice to be avoided by default”. Corey works with his customers on reducing their AWS bills, and he’s seen large numbers of cloud architectures in practice, so I think he’s a pretty good source on this.
- Your business probably has enough problems worth solving that are way more valuable to solve than any the mythical multi-cloud deployment will provide. Avoiding lock-in is mostly an illusion, and usually comes from vague future business needs. Abstracting away the use of any particular cloud provider is a lot of work that prevents you from getting the most leverage out of your use of the cloud. I think even thinking about multi-cloud architectures is prematurely optimising for practically all businesses, and especially startups. Ok, maybe your startup is running out of money and you have some credits for a different cloud provider to the one you’re on. Maybe you’re Netflix or Apple and are a large percentage of the entire traffic of the Internet. Fair enough, they are good reasons to be multi-cloud. But for the rest of us, go all in on one cloud provider and don’t even think about being able to move workloads between cloud providers. Going all in on one cloud provider is where the magic of cloud platforms is, that is ease of use, simplicity of the platform and efficiency.
- Don’t use the best tools for the job. In AWS, the best tool for a highly available key value data access is probably DynamoDB. The best tool for a bunch of time series data is the recently generally available Timestream, which just came out after being announced nearly 2 years ago. However if you already have fully production and operational MySQL Aurora installation in place, can you just put the data there instead? Even in the cloud, adding new technology to your stack can be a distraction. You should optimise globally, and that means using the tools you’re already using, unless you really really have to use something else, or you’re pretty that your use case will not be satisfied by using something you’re already using.
- At Intercom we call this “Run Less Software” and is part of our technical strategy of being technically conservative. We think it works for us, and we have avoided building and maintaining a lot of stuff that otherwise would have slowed us down over time. There’s also this great talk on this topic, which is at this amazing URL - boringtechnology.club and I strong recommend reading this, after this conference of course.
- If I had my own way, and a time machine, I’d probably rip out everything we use for logs and metrics and APM in our application and go all in on next generation observability tool like Honeycomb or Lightstep. Cloudwatch’s Embedded Metric format and Logs Insights are going in the right direction but still miles behind in providing a rich developer experience. The sooner you get real observability into your application, the sooner you’ll be able to understand what your users are actually experiencing.
- Day 1 of your startup is probably not the time to be learning k8s. Ok, if you have a multi-year runway and a significant infrastructure to build, maybe. Or if you’re in the infrastructure space selling to k8s users, then sure! Unless you are already quite proficient at k8s, chances are the quickest way to get a service up and running is to use the simplest, flexible and most common building blocks available, i.e. EC2 hosts.
- Lambda is an amazing technology, but I think it has its place. We have found success with it running as glue code in between AWS services. Like, upload an image to S3 and have It resized. Simple functions, kind of like stored procedures for the cloud. I would not like to run a complex application in it, system. Use it for hat it’s good at.or be building something from scratch. It’s not a general purpose computing platform, it’s limited but brilliantly joined up with the rest of the AWS eco
- A really good discussion of the use of Lambda is in this book. It’s written by a couple of ex-AWS engineers and is highly opinionated about which parts of AWS to use.
- Similar to Kubernetes, unless you’re highly skilled at these things and there are particular needs, chances are most startups shouldn’t go near micro services. There are more things that can go wrong, and it’s harder to see what’s going on. Adding network hops, . At Intercom about 6 years ago we thought that it was inevitable that significant new functionality should be developed in their own standalone service. We developed new features like our webhook processing, event processing etc. as small services that talked back to our monolith. Over time we noticed that teams hated working on these services. There was so much overhead and non-differentiating heavy lifting in maintaining these services, and adding new functionality seemed to take longer. Our teams wanted to build product, not maintain services. In the last few years, we’ve been folding these back into our Ruby on Rails monolith. I suspect similar applies to many service orientated architectures.
- Similar to Kubernetes, unless you’re highly skilled at these things and there are particular needs, chances are most startups shouldn’t go near micro services. There are more things that can go wrong, and it’s harder to see what’s going on. Adding network hops, . At Intercom about 6 years ago we thought that it was inevitable that significant new functionality should be developed in their own standalone service. We developed new features like our webhook processing, event processing etc. as small services that talked back to our monolith. Over time we noticed that teams hated working on these services. There was so much overhead and non-differentiating heavy lifting in maintaining these services, and adding new functionality seemed to take longer. Our teams wanted to build product, not maintain services. In the last few years, we’ve been folding these back into our Ruby on Rails monolith. I suspect similar applies to many service orientated architectures.
- I regret almost every time I configure something in the AWS Console. “Click ops” can be fast and effective, but the advantages of having a version controlled, peer review definition of your infrastructure are significant. It doesn’t matter much if you’re using Cloudformation, Terraform or higher level tools like CDK. Most of the time, infrastructure defined in code or configuration is easier to maintain. Having infrastructure defined in code doesn’t mean you need to make it complex though! Abstractions here using modules can be very powerful but can cause unexpected side effects, so I’d avoid DRYing things in favour of simple declarative instructions.
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Premature optimism is the root of all evil, wrote Donald Knuth over 50 years ago. You aren’t gonna need it, wrote somebody back in the day of Extreme Programming, I dunno maybe somebody like Martin Fowler. The cloud is a great place to build for scale, but that doesn’t mean you have to. Sure you get unfathomable scale at your fingertips by using the likes of S3, SQS and DynamoDB, but also computers these days are just really fast!
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Speaking of premature optimisation… nobody likes to waste money, and there sure are many ways to waste money in AWS. AWS billing and optimisation is hard, though getting easier thanks to vastly improved tooling from AWS and new ways of purchasing capacity like Savings Plans. I think it’s best to be reactive with costs. Ship whatever it is you’re building, then set a calendar reminder to check on the costs later on down the line. It can be hard to predict exactly what something costs - like if you’re building an entirely new service, how long would it take to figure out the bandwidth charges, Aurora Storage IO, and SQS costs?
- I also find it’s easy to “snack” on costs. Removing a few unused Elastic IPs or EBS can save a few dollars per month, and it’s so satisfying to clean things up. But will It really change the future of your business? I sometimes try to justify these cleanups as making our infrastructure more simple to understand, which is a problem when you have a 9 year old AWS account. Ironically I do spend a good bit of time optimising costs at Intercom. For a mature business with a significant AWS spend and clear business requirements here, this is work that is worth doing. But
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Reading the engineering blog of Netflix, Uber, AirBnB is a great way to get completely distracted and over-engineer solutions to problems you don’t really have. The information you really need typically doesn’t come out in a blog or conference talk. Those things are usually artefacts of some career plan that somebody has in. Peer relationships with similarly sized startups can be really effective though. Finding people to
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Whatever to be said about not looking at successful startups, you almost certainly do not want to be looking at humongous cloud providers like Amazon, Google and Microsoft. Some companies may benefit from monorepos or five-nines availability or microservices or SRE. But these are mostly the concerns of extremely large organisations. Instead of a startup worrying about their chaos engineering strategy, I’d much prefer that they built on a small set of well understood managed services with great redundancy built in.
- One sure fire terrible technical strategy for your startup is to do whatever you hear at a conference. Only you can understand your business context, technical challenges. The differences between core competencies and undifferentiating heavy lifting isn’t always as clear cut. There are myriad human factors to take into account when defining and implementing a technical strategy. So don’t blindly do any of these things. They’re my honest opinions, and they’re what work for me in my current role.
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Security is job number zero of, well, anything on the Internet these days. Not only are consumer expectations higher than ever, regulations like GDPR require a reasonable level of security built into your product. Building security in every product and feature you build from day zero is way easier than adding it afterwards. Burning customers through security problems is a sure way to lose customer confidence - I’ve been on both sides of this and it is real. As your startup moves upmarket, the conversations with larger purchasers will become more detailed and require more rigour in your product. Thankfully this is easier to do than ever with good secure options in the cloud, and constant improvements to things like S3 bucket availability that can help avoid problems.
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- At Intercom we say that “Shipping is your company’s heartbeat”. I don’t think it is a coincidence that companies who focus on shipping are successful. In fact, this has been shown with a great deal of rigour.
- I consider Accelerate is to be the bible of high performance technology organisations. The authors applied research methods to discovering best practice used in real companies to be successful. We liked what we read in this book so much at Intercom that we named the team task with developer productivity after it. If you care about the success of your organisation, no matter what the industry, the knowledge in this book will help you a lot.
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- Generally, you’ll need a lot of generalists who really want to grow. The growth mindset alone encourages growth, and boy are you going to need it in a fast growing environment. SMEs or specialists offer seductive productivity. Utilimately they might turn into silos that can slow you down, limiting advantages of collaboration and team owned problems. You won’t get the best solution if the entire team can’t work on your biggest problems. You want to people to grow towards owning and deeply understanding your businesses main problems - and not being owned by “the machine learning specialist” that works in their own silo. If you do hire people with deep expertise, you want them to give that expertise away and grow a team.
- Customer messaging startup. Dublin and SF. Joined 5 years ago, growing fast.
- I did mention earlier on about picking a small number of well understood services to use. Some can just be commodities though. Elasticache, SQS, RDS, etc. are far better defaults to use instead of rolling your own memcache, rabbitmq or clustered MySQL setup. Similarly I think some of the security and AI/ML services are looking really great, and I’d kick the tyres of them rather than building something. When you do need to solve some problems that go beyond your current tech stack, my preference is to avoid ever building anything and simply use something available in your existing cloud account.
- Last but not least…
- IS this a cliche? Technically, this means world class observability, monitoring, operational best practices, good enough uptime, performance and good security. I think that Jeff Bezos guy was along th right lines when he talked about always working backwards from the customer. I learned this at Amazon, and it’s truer than ever from my experience at Intercom. If you don’t know what your customers doing, experiencing and thinking, you are not focussing on the customer. Use great observability tools. Talk to your customers. Give them great experiences. Build appropriately and choose your technical strategy wisely.
- You can get me at my twitter handle or I guess type my name into Google. Also a reminder these strategies are opinions and related to my current environment that I work in! I am also going to be speaking at a panel discussion in a few minutes about “How to scale Startup on AWS”, I think it’s over on Track 2. See you over there and thanks again for listening!