Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Draftkings: Launching w/ Confidence at Scale, FutureStack17 NYC


Published on

Learn how Draftkings' culture helps drive performance.

Be sure to subscribe and follow New Relic at:

Published in: Software
  • Be the first to comment

  • Be the first to like this

Draftkings: Launching w/ Confidence at Scale, FutureStack17 NYC

  1. 1. Launching with Confidence at Scale Sept 2017 A Guide To Successful Launches At DraftKings
  2. 2. 22 What to Expect from the Session • An Introduction to DraftKings and my role at DraftKings • DraftKings’ Infrastructure Overview • Deep Dive into the work behind product updates – Experiments, Feature Flags, and Insights • Getting specific – Reviewing several key DraftKings’ feature launches • Various Bits of Wisdom • Our Digital Moment of Truth
  3. 3. 3 Introductions
  4. 4. 44 My Story What long, strange trip its been • I have been at DraftKings ~3 of the 5 years. I joined to help lead the conversion from Monolith to Service Oriented Architecture, tasked originally with starting the various App teams – Web, iOS, Android • Spent ~5 years prior at WB Games, Turbine working on MMO’s and then as part of the Digital Platform • Lord of the Rings Online & Dungeons and Dragons • Batman Arkham City • Mortal Kombat • Injustice • Got a chance to live through 2015 when Daily Fantasy commercials dominated over the airwaves • More recently focused on Web Platform growth, growing out our React + Redux centric approach
  5. 5. DraftKings is an innovative sports-tech entertainment company forever changing the way consumers engage with their favorite sports, teams, leagues and athletes.
  6. 6. 66 A Brief History of DraftKings 5 years of Hyper Growth DraftKings is the leading daily fantasy platform (DFS), with over 8 million registered customers  DraftKings, launched in April 2012 and headquartered in Boston, is the leader in the daily fantasy sports industry  The company provides an online daily fantasy sports contest service that allow customers to compete against others  Offers both free and pay contests; games feature a salary cap or no salary cap format, and new contests are offered every day  Currently offers 10 sports: Football, Soccer, Basketball, Baseball, Golf, Hockey, MMA, NASCAR, CFL and eSports
  7. 7. 7 WHO WE WORK WITH DraftKings partners with the biggest brands in sports to create unparalleled access for its users.
  8. 8. 8 Highly Engaged Customers 16.7 8 6.3 5.5 3.1 2.8 2.8 2.2 1.7 0.6 Facebook DraftKings Snapchat Instagram ESPN Twitter Tumbler Yahoo Sports Pinterest LinkedIn Average Monthly Hours Spent on Mobile App + Desktop per Visitor Engagement Source: comScore Media Metrix Multi-Platform, U.S., July 2016. ESPN and Yahoo data based on ESPN and comScore analytics data from October 2016. 1 Data represents FY2016A. DraftKings’ customers are significantly more engaged than nearly all other major media partners. Daily Fantasy Sports
  9. 9. 99 New Relic helping us to drive our Performance Culture How we started using New Relic • Brought over from my experience using it at WB Games • First used cases for New Relic Browser were really around solving for Client Visibility and JS Error Alerting • Brought New Relic Mobile into Mobile for Android Application • Synthetics for Scripted Checks (difference between knowing the platform works and the platform is accessible) • Performance Profiling part of teams OKRs – NR is a key tool for us there • Insights Dashboards provides a way to get out of the aggregate and drill down into specifics
  10. 10. 13 DraftKings’ Infrastructure Overview
  11. 11. 1414 The Tech Stack at DraftKings What we’re made of • Started out as a C# dominated shop • Original was launched on C# MVC • REST Micro-Services are all written in C# • WEB Apps • Some of the existing MVC still alive and well • All new features in React + Redux • ANDROID is Java • iOS App • Started as Objective C • New features in Swift • Running multiple apps right now! • DRAFTKINGS • DK LIVE
  12. 12. 1515 DevOps @ DraftKings What we’re made of • Amazon Web Services • Teams Own Infrastructure as Code • Build • Deploy • Scaling • Teams Monitor and own Monitoring and Alerting • Insights Dashboards • New Relic Alerts
  13. 13. 16 Rolling out Features – Experiments, Feature Flags and Insights
  14. 14. 17 High Level Product Development Lifecycle Concept written • Spec is usually added into the roadmap with some WAG level estimates Design Phase • UX & Design works through Key items • Tech works through high level planning Consensus • All relevant stakeholders (product, design, tech) give okay to proceed in Green Light Active Development • Work begins with demos and check-ins ~2 weeks Rollout • Experiment System gradually brings new feature to market Follow-up • Iterate in small releases; clean up experiment code • Report on key KPIs
  15. 15. 18 Thank you, Martin Fowler (required reading btw)
  16. 16. 20 Release Toggles • Base Feature Toggle for longer development cycles • Unshipped Code as Inventory is a risk. • Allows us to keep quality and standards around Feature Branches & PRs • Continue to ”ship” without the risk of some merge conflicts or anyone ever seeing that. • Transitions cleanly to Internal Beta Group via Permission Toggle
  17. 17. 21 Experiment Toggles • Used for A/B Testing • Sends user down one code path vs the other based on Segment/Cohort assigned • Toggle can last hours, days, or even a week based on statistical significance • Need to maintain the code for each path being experimented • Could be simple – Button placement or Color • Could be complex – updated UI on User Generated Contest screen
  18. 18. 2222 Beta Group vs Canary Cohort Permissions Toggles, Point of 1st Contact Beta Group • “Champagne Brunch” – internal users drinking their own champagne • External VIPs get early access • Very targeted audience • Capable of qualitative feedback - Solicit reviews from individuals! • Manually intensive Canary Cohort • Randomly assigned Segment/Cohort • Some ~5% of traffic based on rate towards statistical significance • Monitor and report back on key KPI’s • Automated
  19. 19. 23 Ops Toggles • Client Circuit Breaker • The good ole fashioned kill switch • Make sure your client is resilient! What happens if you shut a feature off? • Take down intensive, less critical features while keeping things online
  20. 20. 24 Best Practices
  21. 21. 25 DraftKings Experiment System (Higgs API) Driving experiments out to the applications
  22. 22. 2626 Per-request Overrides Allow a toggle's On/Off state to be overridden on a per-request basis Priority Source Description 1 URL Param Check query string for override 2 Local Storage Checks here if Toggle Versions been set 3 Higgs DK Experiment System Lets Toggle know what Experiment to factor in 4 Release Flag System A structured config file Application specific Feature toggles 5 Default Value
  23. 23. 27 Carrying Cost & Clean Up • Keeping Inventory Low means that you need to actively manage it • Always add a story to the backlog to clean up toggles • Always add dates to toggles so you don’t need to hunt down how old it is • A soft warnings for old toggles • Fail builds when toggles exceed an agreed on age • Apply a Limit to the number of Toggles in place • Don’t allow team to exceed number with out exception to ensure cleanup • Adding a Toggle means removing an older one
  24. 24. 28 Updating Registration
  25. 25. 29 Product Launch - Example Note: In both flows we default all users to the registration screen and they need to switch We launched this new page and monitored KPIs: attempt rate, success rate, and error rate Old Page New Page
  26. 26. 30 Product Launch Strategy Use Experiment System  Experiment system designed to make testing easier  Can test things from small UX/copy changes to full product changes  Allows us to have multiple cohorts, control group, holdout group General Launch Strategy  Determine KPIs and goals of the product (increase metric, do no harm, etc.)  If controlled launch required, develop launch timeline for A/B test  Monitor KPIs and identify anomalies  If necessary, work with UX and Stakeholders to make product decision to fix issue  Re-release, re-test as needed
  27. 27. 31 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 14 16 18 20 22 0 2 4 6 8 10 12 14 16 18 20 22 2/8/2017 2/9/2017 Hour | Date Cumulative Login Attempt Rate New Page Old Page  A/B Testing showed us something was off  By mid-day we had found the source of the issue  Only affected Attempt Rate  Success rate, re-try rate, etc. in line  Must be something causing users to bounce from page  About 500 users over 2 day stretch 24.5% 26.8% Product Launch - Example Statistically Significant Difference after 6 Hours Hit the screen and attempt to login within 5 minutes
  28. 28. 32 Product Launch - Example Note: In both flows we default all users to the registration screen and they need to switch Old Page New Page Hypothesis: Removing the login method from header was causing users to become confused and bounce
  29. 29. 33 Product Launch - Example Note: In both flows we default all users to the registration screen and they need to switch Old Page New Page So we added that option
  30. 30. 34 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50% 15 16 17 18 19 20 21 22 23 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 2/15/2017 2/16/2017 Hour | Date Cumulative Login Attempt Rate New Page Old Page  Received a quick turn around from UX and Engineering  Resulting change lowered the attempt gap from 92% of old page to ~98% of old page  Same principles applied to all feature changes  Set and monitor KPIs  Identify differences in product from KPIs  Iterate until success 28.3% 27.7% Product Launch - Example Never Statistically Significant Difference Hit the screen and attempt to login within 5 minutes Re-Launch
  31. 31. 35 International Release Germany
  32. 32. 36 DraftKings Growing Internationally • Longer Term Beta process • Allows us to tune the experience prior to full launch • Insights Dashboards By Country Code • Copy and then Split up easily! • Synthetics checks setup • London • Frankfurt • Browser Filterable Geography!
  33. 33. 37 Our Digital Moment of Truth
  34. 34. 3838 Our Digital Moment of Truth Our Digital Moment of Truth, our Black Friday, is Week 1 of the NFL Season
  36. 36. NFL Contest Entries
  37. 37. NFL Related Sessions
  38. 38. 42 Summary • Ensure your life cycle empowers teams and reinforces autonomy • Do it safely – Spend time understanding how you intend to release • Layer Toggles in with Experiments • Drive experiments from a central system out to clients • Performance Culture demands a COMMON UNDERSTANDING of the metrics needed to: • Drive Consensus, • Execute, • Improve via iteration • Don’t Fly Blind! • Need for Metrics == Great Tools • New Relic provides these • If you didn’t try out DraftKings last week (like millions of people did) – try it!
  39. 39. Questions?