Paul Teresi Think like a scientist - extended version
•Download as PPTX, PDF•
1 like•920 views
The extended version of How to experiment your way to sustainable app growth by Paul Teresi of Skyscanner. Visit https://medium.com/@skyscanner for more growth tips and https://www.skyscanner.net/mobile.html for a working example of our product in motion.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Paul Teresi Think like a scientist - extended version
- 1. Think Like a Scientist – Act like a Growth Hacker How to experiment your way to sustainable app growth Paul Teresi @skyscannergrwth
- 3. Our ultimate goal: to make travel booking as easy as it is today to buy a book online
- 6. Growth
- 7. Exponential and very healthy growth 0 10 20 30 40 50 60 70
- 8. 60,000,000
- 9. 70,000,000
- 10. Our highly-rated app is super easy to use.
- 12. Technology company first and foremost 50% of our staff are engineers
- 13. We’re different. Our unique self-built technology means we provide a unique content.
- 14. We go direct
- 15. • Free • Unbiased • Global • Local
- 17. Think Like a Scientist – Act like a Growth Hacker How to experiment your way to sustainable app growth Paul Teresi @skyscannergrwth
- 18. By the end of 2020 Apple’s App Store is expected to have over Sensor Tower, August 2016 million active apps
- 19. By the end of 2020 Compared to 2016 projected figures Sensor Tower, August 2016 73 %
- 20. How do you maneuver around the noise?
- 23. Growth Hacker - A hacker whose sole objective is to grow the number of users of a specific product, and make every strategic and tactical decision based on growth Douglas Cook, Abandoning conventional wisdom for hypergrowth 2016
- 24. How is this changing the way we work?
- 27. Define • Define actionable goals for experiments • Understand growth funnels • Prioritize experiments • Define tracking and analytics • How will you execute experiments and optimize based on learning Define Design Develop Test Learn • Lean and Agile Approaches • Test fast and Learn Faster • Limit your objectives
- 28. example • Logins • Registrations • Conversions • App Start
- 29. example Adopted our creative theming and strategy to relate to overall objectives
- 30. example Revenue Month 1 Month 5 145% ROI
- 31. Define Design Develop Test Learn Noise within the marketing industry
- 32. Define Define Design Develop Test Learn I C E mpact onfidence ase Growth Hackers, August 2016
- 33. Design • Design a robust hypothesis • Explain what you expect to happen (e.g. benchmarks/targets) • Metrics should be measurable • Determine the minimum detectable effect (MDE) • Ensure your test is driving the change and not just seasonal changes • Partake in peer review • Get collective buy in from your peers • Don’t be afraid to go outside of your immediate team Define Design Develop Test Learn
- 34. Learn • Is there significance in your results? • Causal Impact Analysis Define Design Develop Test Learn • How is what we do today changing what we did yesterday? • Changing App Store Landscape – are we seeing the full picture Kim Larsens, January 2016
- 35. Learn Define Design Develop Test Learn Causal impact analysis allows us to put science behind the metrics and decipher the impact of your metrics beyond the numbers attribution is able to report on
- 36. Learn • Use your data to know when to pivot, ramp up or persevere • Failure is not a bad thing Define Design Develop Test Learn
- 37. In closing • Define • Limit your objectives • Prioritize your tests to meet meet your needs today • Design • Ensure you are able to measure your goals • Learn • Look for the significance in your results • What did this tell you about your hypothesis? • Share your results Define Design Develop Test Learn
- 38. “Take a risk and keep testing, because what works today won’t work tomorrow, but what worked yesterday may work again.” — Amrita Sahasrabudhe, Director of Marketing at ARAMARK Parks and Destinations Design
- 39. Thank you for your time! Find my presentation and more info at: http://bit.ly/growthhackingapps Check out more stories of Skyscanner’s successes and failures here @SkyscannerGrwt
Editor's Notes
- Back in the early 2000s there were many budget airline sites in the UK. Gareth was regularly travelling to Europe to ski with his brother, and trawling all these sites proved an extremely time-consuming process. Gareth, Bonamy and Barry put their heads together to come up with a solution, and from a simple spreadsheet and a few lines of code, Skyscanner was born. It grew by word of mouth.
- We’ve celebrated double digit revenue growth for seven consecutive years, and continue to grow at an accelerated rate.
- Mobile is central to our business and to the future of travel planning and booking. We have long adopted a mobile-first approach but this is essentially a customer first strategy because of the quantity and growth of mobile users. In Oct 2013, Sequoia Capital Chairman Michael Moritz stated “Despite…[the] large number of existing online travel vendors, the market over the next decade will expand enormously, accelerated by a proliferation of mobile devices.”
- We launched our first app in 2011. Now, our apps have been downloaded over 70m times. In 2014 we acquired Distinction, bringing their 30-strong team on board to accelerate our already strong app development talent at Skyscanner. In 2015 we saw mobile visits +60% (59% of all visits), and mobile bookings grew 24%, comprising 42% of all conversions.
- Our tech-related spend is close to 50%, which is almost as much as companies like LinkedIn and Facebook. By contrast, the tech spend by online travel players is between 3% and 12%. We’ve a start-up culture driving rapid growth and we work in a squad and tribe model that ensures we’re innovative and ahead of the curve. We’re at the forefront of technology in our industry, and in 2015 celebrated industry first such as NDC and a skill with Amazon Alexa.
- We’re more than just a metasearch– our self-built technology is unique and our direct connections with partners mean we have the most comprehensive coverage out there. We can go direct, to get the most up-to-date prices out there.
- Thanks to our founders and engineering talent, we have built our travel search technology from the ground up, making it proprietary and best-in-class. This means we have direct connections with partners, so we can provide the best, most up-to-date range of options for travellers. This makes us a unique travel marketplace: unlike other travel search sites, we’re not reliant on obtaining our search results through a third party. This is something our consumers and recognise, and well as our partners and competitors: we power hundreds of well-known brands such as MSN and Lonely Planet.
- We’re the most global local travel search in the world. We tailor our products per market (example of India: buses), available in over 30 languages and 70 currencies, we couple comprehensive global coverage with far-reaching local coverage. We‘re not linked to any travel agents or hotels, and our results are completely unbiased; all you see are the lowest rates.
- By the year 2020, the Apple App Store is expected to have over 5 million active apps. That is 73% larger than the 2016 projected end figures.
- By the year 2020, the Apple App Store is expected to have over 5 million active apps. That is 73% larger than the 2016 projected end figures.
- With all of this action – it creates a lot of noise that is can be hard for a lot of brands to breakthrough. Take Skyscanner for instance – although we’re a travel brand – there are probably a lot of similarities between us. We only have 1 app in the market so for us it takes a lot work to make sure we are seen and heard. It takes an army to make sure we are up to date on Paid Aqusition, ASO, PR and more – so we need to start getting creative with our solutions.
- This is one of the reasons that helped contribute to our decisions to go from an ordinary marketing department to a growth organization – and more specifically into Growth hacking.
- This is one of the reasons that helped contribute to our decisions to go from an ordinary marketing department to a growth organization – and more specifically into Growth hacking.
- The term “Growth Hacker” has been passed around a lot in the digital space, but for today’s purposes we are going to use the definition above. Growth Hackers come in a disciplines as they encourage you to break down typical siloes and collaborate in a cross functional team to propel growth. The key here is strategic and tactical decision based on growth. Our organization is different than yours – and it’s important to recognize your growth levers need to meet your needs now in the moment.
- You’re probably wondering now – how does this affect me or how can I use these principals to adapt my team’s workflow?
- By adopting a T Shaped mentality – you are breaking down barriers that would ordinarily hold you back. Stepping outside your discipline and taking on new intiatives allow for agile and swift movements in order to be reactive in this high speed digital space. Being an app first company – at Skyscanner we have to ensure that both our online website and mobile app are getting attention and we can’t place focus on one of these entities alone – we must work collaboratively So how do I achieve this?
- We adopted this methodology to be lean and agile with our testing. Short, quick and fast. To that we need to know what we are testing, why we are testing and what does success look for. By utilizing the model above – it encourages proactive discussion and thought from the get-go. Today we will look into 3 fields and how this can applied to your work: Define Design Learn
- Before you can even begin you need to understand what you’re doing. When trying to test fast and lean – limit your objectives. We all want logged in, converting users at an all time low CPI without compromising ROI but when validating your tactics focus your results down to the actions you can have direct impact on.
- In a personal example at Skyscanner – we began by having a laundry list of objectives we were looking to hit. However – our flywheel wasn’t moving as fast as we wanted and We went from a laundry list of wants and narrowed it down to 3 funnel actions that correlate to revenue. By making this decision this also helped us articulate our goals and wants with our partners – they were clear what they needed to do to help yield these results. And by limiting our focus – we were actually able to pull back a curtain and see that some partners weren’t cutting it and which ones had the potential we were looking for. We took our overall focus and then translated what that meant for us on a creative front – and made sure we reflected what we want our users to experience from initial interaction with us. In the end we were actually able to increase our ROI by 145% which also helped us boost overall activity with our users.
- In a personal example at Skyscanner – we began by having a laundry list of objectives we were looking to hit. However – our flywheel wasn’t moving as fast as we wanted and We went from a laundry list of wants and narrowed it down to 3 funnel actions that correlate to revenue. By making this decision this also helped us articulate our goals and wants with our partners – they were clear what they needed to do to help yield these results. And by limiting our focus – we were actually able to pull back a curtain and see that some partners weren’t cutting it and which ones had the potential we were looking for. We took our overall focus and then translated what that meant for us on a creative front – and made sure we reflected what we want our users to experience from initial interaction with us. In the end we were actually able to increase our ROI by 145% which also helped us boost overall activity with our users.
- In a personal example at Skyscanner – we began by having a laundry list of objectives we were looking to hit. However – our flywheel wasn’t moving as fast as we wanted and We went from a laundry list of wants and narrowed it down to 3 funnel actions that correlate to revenue. By making this decision this also helped us articulate our goals and wants with our partners – they were clear what they needed to do to help yield these results. And by limiting our focus – we were actually able to pull back a curtain and see that some partners weren’t cutting it and which ones had the potential we were looking for. In the end we were actually able to increase our ROI by 145% which also helped us boost overall activity with our users.
- Direct objectives are a great start, but they don’t mean anything if you don’t have the right channels to implement them in. As the industry expands and new technology, partners and opportunities arise – it may seem like you need to jump on them all. However, sitting back and reflecting on what will lead to greatest growth for you is key. Just because your competitors are there doesn’t mean you need to be.
- Within in my team, we have limited resource so we rely on this ICE score calculator to make sure we are prioritizing the right thing. By using this format we are able to build a common buy in from the team on the impact it will have on growth, confidence it will work as expected and how easy it will be to implement. Is it better to work on high impact project that will take a month to complete or experiments that you’re confidant will succeed and make use of the resources that you got.
- Once you have nailed down the basics, it’s important that you design a test that is measurable. Ensure you can measure the changes you are looking for and don’t be afraid to get support from your peers outside your immediate area to ask questions and seek clarification
- When trying to test fast and lean – limit your objectives. We all want logged in, converting users at an all time low CPI without compromising ROI but when validating your tactics focus your results. http://multithreaded.stitchfix.com/blog/2016/01/13/market-watch/
- When trying to test fast and lean – limit your objectives. We all want logged in, converting users at an all time low CPI without compromising ROI but when validating your tactics focus your results. http://multithreaded.stitchfix.com/blog/2016/01/13/market-watch/
- At the end it’s important to know what your data is telling you. From there it’s key to know how to digest everything to know when to pivot, ramp up or preserver. Another key is to remember that failure is not bad – it’s an amazing learning opportunity. This is one of my favorite values to embrace at Skyscanner – embrace your failures and learn from them because they can teach you about your process.