Downloaded 34 times
Uploaded bySkyscanner
Applying science and data to drive sustainable Paid Growth by Ilana-Munckton, Skyscanner
The document summarizes Skyscanner's approach to paid growth through data-driven experimentation and testing. It discusses how Skyscanner: 1) Conducted an uncontrolled test of paid search ads on branded terms that found both cannibalization of organic traffic but also incremental gains. 2) Developed data science tools to measure the causal impact of changes and calculate adjusted ROI accounting for cannibalization. 3) Learned that while some traffic is cannibalized, paid search recovers this and provides incremental revenue, though returns vary between new and returning users. 4) Continues exploring user behavior and ad formats/extensions to further optimize paid acquisition as search landscapes evolve.
More Related Content
Similar to Applying science and data to drive sustainable Paid Growth by Ilana-Munckton, Skyscanner
![[EN].CleverGroup Vietnam Profile 20251202](https://cdn.slidesharecdn.com/ss_thumbnails/en-260120091417-fe6f88ec-thumbnail.jpg?width=640&height=640&fit=bounds)
Applying science and data to drive sustainable Paid Growth by Ilana-Munckton, Skyscanner
- 1. Applying science anddata to drive sustainable Paid Growth Ilana Munckton | Director, Growth @ilanamunckton | @SkyscannerGrwth
- 2.
- 3. Originally from New York State Workedin agencies in NY, London & Edinburgh Joined Skyscanner three years ago 11 Started in digital marketing in 2006 Now Director of Growth running Paid Acquisition
- 4. A little bitabout Skyscanner Our awesome travel products Our success story (so far)
- 5. Our awesome travelproducts
- 7. Our journey from marketingto growth How are we structured? 1 What do we believe? 2 How do we act? 3
- 8. A growth structure organisedinto central & regional tribes Central GrowthTribe EMEA Regional GrowthTribe AMER Regional GrowthTribe APAC Regional GrowthTribe
- 9. Data-driven experimentation as the routeto hyper growth Growth hacking ninjas!
- 10. Leveraging a clearframework to accelerate our velocity of testing
- 11. Applying our growthprinciples to paid acquisition The Paid Growth mission 1 But first, science 2 How we do it IRL 3
- 12. Optimising the paid acquisition oftravellers in our quest for global, sustainable growth
- 13. Growth Engineer Marketing automation Growth Analyst Attributionand measurement Growth Marketing Search, display and social advertising Growth Designer Creative design and implementation
- 14. Judson builds awesome dynamicHTML5 ads Craig automates a feed so each traveller sees personalised copy Kate delivers media targeting travellers based on intent Sri’s attribution model values each touchpoint in the media mix
- 15. But what aboutthat data- driven testing framework?
- 16. Experiments Testing Optimisation (scientificconditions) (uncontrolled tests) (machine learning) Or is the outcome the most important thing? Do we place a higher value on the insights that get us to a result?
- 17. Experiments Testing Optimisation Thedifference is important, but all are data-driven tools for growth (scientific conditions) (uncontrolled tests) (machine learning) Or is the outcome the most important thing? Do we place a higher value on the insights that get us to a result?
- 18. Actually a lotof what we do in Paid Acquisition is uncontrolled testing. Here’s an example.
- 19. We started withan insight: Travellers who search for ‘Skyscanner’ brand terms in search engines drive a high volume of qualified traffic. ‘But don’t we rank in position one organically?’
- 20.
- 21.
- 22. Our hypothesis: If webid on Google paid search ads on Branded terms, we will drive incremental traffic to Skyscanner.
- 23. We hoped tosee this: Without ads With ads Organic Paid Incremental Cannibalisation
- 24. The data sciencebit We built a causal impact tool that uses Bayesian structural time series model technique to validate if a change has an impact on a metric over time I.e. does turning on paid ads cause a change in organic traffic
- 25. The data sciencebit We developed a formula to calculate adjusted ROI that accounts for paid cannibalisation in the search engine I.e. if we pay for sessions we would acquire in organic, is the cumulative gain from Brand search profitable?
- 26. Test design: Identify a comparabletest & control market Phase ad delivery in the test market (i.e. turn it on/off) Change *nothing* in the control market Record what happens to traffic & revenue in both
- 27. It looked likethis: Market Phase 1 Phase 2 Phase 3 Phase 4 Test Control Paid ads running for each device type
- 28. So what didwe find?
- 29. Cannibalisation of organictraffic.1 But we recover all of this in paid.2 And there’s an incremental traffic gain. Sometimes. 3
- 30. There’s incremental revenuewith paid ads on, even when net traffic is flat. 4 But revenue varies between new and returning users! This is true for all device types.5 6 All users Returning New Mobile Desktop
- 31. We can beconfident we understand the right Growth strategy to maximise acquisition in the search landscape. Now what?
- 32. We continue tolearn! To further explore behaviour of new and returning users As we grow our product in various markets To understand the impact of new ad formats and ad extensions To monitor variation in incremental value as the SERPs evolve
- 33. Have a cleardata- driven framework to guide how you act Know when to experiment, to test, and to optimise Keep learning to accelerate (paid) acquisition Summing up 1 2 3
- 34. Prioritise experiments A handycalculator tool created by Skyscanner’s Principal Growth Manager Douglas Cook https://growthhackers.com/articles/ a-simple-ice-score-calculator Avoid these common pitfalls Tricks of the trade from Skyscanner’s Head of Experimentation Colin McFarland http://codevoyagers.com/2015/11/ 26/common-pitfalls-in- experimentation/ Tool kit Design like you’re right… …test like you’re wrong. An experimentation toolkit by Skyscanner’s Rik Higham http://www.experimentation hub.com/
- 35. Thank you forlistening. Find my slides (and much more!) at: http://bit.ly/ActivationPaidGrowth @SkyscannerGrwth | @ilanamunckton
Editor's Notes
- #3 Hello everyone! My name is Ilana Munckton. I’m a director of Growth at Skyscanner. I’m talking to you from our office in Edinburgh, Scotland. Today I’m going to tell you about how my Paid Growth squad apply data-driven experimentation to acquisition for Skyscanner.
- #4 A little bit about me. You might be able to detect from my accent that I’m not native to Scotland. I grew up in the US in New York. And have been working in the digital marketing industry for over ten years. I started my career working in media agencies in Manhattan, London, and Edinburgh. I joined Skyscanner three and a half years ago And now head up our global paid growth efforts for over 35 markets worldwide.
- #5 Let me give you some context about why Skyscanner is so great. With a little insight into our product and the growth we’ve experienced.
- #6 Of course most of you are already loyal Skyscanner fans, right? But for those who are new to our product: Skyscanner is a travel search engine that compares Flights, Hotels and Car Hire. Our awesome website and mobile apps help travellers to: search, plan, and book travel directly - from millions of options. We pride ourselves on our unbiased and comprehensive coverage, meaning you can always find the option that’s right for you. Whether that’s the cheapest hotel, the most convenient flight, or booking with your favourite car hire company.
- #7 And I’m really here today because we have an incredible growth story We have grown a business to over 50,000,000 monthly web visitors and 60,000,000 app installs And we now have a team of over 800 employees in 10 offices worldwide
- #8 We achieved this in part by partaking on our own journey From a marketing function to the Growth team we are today Let me give you a little intro to how we are structured, our principles for growth, and how we apply them.
- #9 We structured ourselves into cross-functional teams comprised of central and regional growth tribes. And each of these tribes contain squads combining marketing, engineering, and data science skillsets.
- #10 We embraced data-driven experimentation as a key growth principle As part of this, we launched a training programme that enables us to ‘growth hack like ninjas’ This is all about becoming T-shaped growth hackers, layering a wide understanding of growth disciplines with deeper domain specialisms. Eg I have domain expertise in digital media strategy, & leverage working knowledge of the relevant automation & data science tools.
- #11 But we are obsessed with translating this theory into how we *REALLY* operate So we developed a framework that guides how we act in practice
- #12 Going to talk about how we leverage these principles in my team. What my Paid Growth team are all about How we incorporate both digital media best practice and scientific experimentation in our work And I’ll share a case study.
- #13 I like to think my team make travel easier for more people. By surfacing our awesome Apps and website as travellers plan their journeys. We do this by using digital media advertising to reach people in market for travel with search, display, and social. And we of course do this by optimising for growth metrics In particular, ensuring travellers get value from using Skyscanner, and that our growth is sustainable (by maximising Revenue).
- #14 Our Paid Growth team is comprised of: Digital marketing skillsets with individuals who really know search, display, and social Working with engineers to automate how we work With Data scientists to advance the sophistication of our measurement And with some really talented designers who build our awesome creative
- #15 Let me show you some of our awesome display prospecting ads to demonstrate how my team works. 1 We designed some beautiful dynamic display ad templates 2 We automated a feed so each traveller sees tailored ad copy with live pricing for their destination 3 We target these ads to users based on their travel behaviours and interests 4 And measure the performance by valuing each touchpoint with a multi-touch attribution model
- #17 I want to talk a little bit about how we blend science and digital media best practice. Experiments use scientifically robust, controlled conditions. In an ideal world we’d run an A/B test every time. They allow us to be confident we’ll get the same result every time we do the same thing. In many cases this isn’t possible, so we have to design ‘uncontrolled tests’. We still try to validate statistically significant results when we have less than perfect test conditions. And sometimes we want to apply algorithmic optimisation, e.g. to let bidding engines improve results. One way to think about it is by answering: Is the outcome more important? Or do we place a higher value on the insights that get us to that outcome?
- #18 I want to talk a little bit about how we blend science and digital media best practice. Experiments use scientifically robust, controlled conditions. In an ideal world we’d run an A/B test every time. They allow us to be confident we’ll get the same result every time we do the same thing. In many cases this isn’t possible, so we have to design ‘uncontrolled tests’. We still try to validate statistically significant results when we have less than perfect test conditions. And sometimes we want to apply algorithmic optimisation, e.g. to let bidding engines improve results. One way to think about it is by answering: Is the outcome more important? Or do we place a higher value on the insights that get us to that outcome?
- #20 And you’ve probably heard this question before: Why should we bid on paid ads when we’re in the first organic search position for the same keywords?
- #22 And here’s just one reason why would bid on paid brand ads. But brand protection isn’t reason enough We need to be sure we use data to verify our acquisition of travellers drives sustainable growth.
- #24 We expected to see something like this. The yellow represents organic traffic, and on the left we see how much traffic we get when there are no paid ads turned on. The red represents the paid traffic gained by turning on ads. If the traffic on the right side is below the white line, there is no net gain. If it’s higher, then combining organic with paid ads drives incremental value.
- #25 This is the science part. I’m not going to go into detail on this. In layman’s terms - does turning on paid ads cause a change in organic traffic
- #26 And if we pay for sessions we would acquire in organic, is the cumulative gain from Brand search profitable?
- #28 As I mentioned we did nothing in the control market. In the test market we Had no Brand search ads live for the first 14 days We then had ads only on desktop Then ads only on mobile devices And finally ads on all device types
- #30 As we predicted, we had some cannibalisation of traffic. But happily, we recoup all of the traffic taken from organic in Paid. And in fact *overall*, adding paid drives incremental traffic as we’d hoped. So turning on ads and organic gives us more traffic than organic alone. BUT. Only sometimes. In some cases the total amount of traffic is unchanged. Not a loss, but not a gain.
- #31 We also found we generate more revenue when paid ads are turned on. Even when net traffic is flat, because paid traffic converts at a greater rate. This is the case across all device types too. And finally, we observed something we didn’t expect. This is exciting. We found that the gain in revenue varies between different user types.
- #32 We have used data-driven testing to identify the optimal Growth strategy to maximise acquisition in the search landscape.
- #33 We continue testing! To inform our bidding strategy for different user types To validate this relationship as the search landscape changes To assess things like, whether site links or app ad extensions are best on mobile ads And as we grow in various markets worldwide. So that’s where we are today!
- #34 To summarise all of that in three key takeaways:
- #35 Finally, this is just what worked for Skyscanner. This isn’t a silver bullet. But here are three things to get you started! 1 a prioritisation calculator 2 tricks of the trade from our head of experimentation 3 and an experimentation toolkit
- #36 Thank you for listening. Happy to answer any questions.