Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The World Is On Fire And So Is Your Website

12 views

Published on

How to architect and scale a complex ecosystem of custom tools that power political organizing work on a nonprofit budget

Published in: Technology
  • Be the first to comment

  • Be the first to like this

The World Is On Fire And So Is Your Website

  1. 1. The World Is On Fire And So Is Your Website Architecting systems for extremely bursty web traffic driven by the news cycle Ann Lewis, CTO @ @ann_lewis
  2. 2. What is MoveOn?
  3. 3. What is MoveOn? ● Grassroots campaigning ● Fighting for social justice, progressive policies, progressive candidates ● A community of millions of progressives in all 50 states
  4. 4. What is MoveOn? ● Small, scrappy, fully-distributed team ● Nationally impactful programs powered by tech tools and data ● A complex ecosystem of 30+ websites and tools that need to scale on a nonprofit budget
  5. 5. Who am I? ● MoveOn’s CTO since 2015 ● Software engineer and technical leader for 15+ years ● Alum of Carnegie Mellon, Amazon, Rosetta Stone, handful of startups, consulting companies ● Excited about building tech that powers collective action
  6. 6. Agenda ● The new attention economy ● Story: a protest goes viral ● The tech behind mass mobilization infrastructure ● How to scale a complex system architecture in the new attention economy, on a nonprofit budget
  7. 7. A walk down memory lane Show of hands: Who remembers Slashdot? Who remembers the internet before big social media?
  8. 8. The rest of you can get off my lawn
  9. 9. The “Slashdot” effect of the 90s A massive surge of web traffic that occurs when a popular website links to a smaller website.
  10. 10. The attention economy ● As the volume of information and news grows, attention becomes a scarce resource ● All content publishers compete for this aggregate attention ● Social media platforms attempt to control engagement around viral content Content Attention
  11. 11. The attention economy evolves ● Previous generation: ○ Social news sites like Slashdot aggregated attention ○ Virality happened via cumulative direct user actions, like upvoting ● Today: ○ Dominance of social media platforms ○ Virality is controlled by the platforms, who make the rules around who sees what, when and why
  12. 12. Feedback Loops The news cycle is a dumpster fire, and social media feedback loops are very effective at quickly amplifying the most inflammatory content to virality.
  13. 13. Oligarchy? ● America’s economic oligarchy: over the last generation, a small number of people have grown more rich while middle and working class wages have stagnated ● On most social media platforms, 0.1% of users have > 100K followers, and 2% have 10K-100K followers ● Most everyone else has 700 followers or less ● Social media is an oligarchy too!
  14. 14. Influencers ● Influencers: social media users with > 100K followers ● Micro-influencers: social media users with 10K-100K followers ● Influencers control the nature of virality in today’s attention economy ● Yes, your favorite gen Z-er was right about becoming an instagram influencer
  15. 15. Story: a protest goes viral
  16. 16. No One Is Above The Law ● Nov 6: US election day. Everyone working on elections is proud and exhausted. Highest turnout for a midterm since 1914! ● Nov 7 2:40pm: Trump crosses a Mueller investigation “red line”: fires Jeff Sessions and replaces with loyalist ● Nov 7 5:10pm: Trump Is Not Above the Law’ protest coordination network launches
  17. 17. Trump Is Not Above the Law ● Nov 7 5:10pm: Protest hub website lists 700 protest events nationwide, 400K people RSVPed ● Nov 7, 7pm: Protest call-to-action gets 10Ks of retweets, we observe moderate surges of traffic ● Nov 7, 9pm: Influencer Rachel Maddow mentions protest website on evening show, traffic surges to 3.5MM views, site falls over (but quickly comes back up)
  18. 18. Our viral Maddow moment
  19. 19. Trump Is Not Above The Law ● 11/8/2018 12pm ET: Protest hub website has accumulated ~1000 events nationwide, ~500K people RSVP. 300 new events and 100K more RSVPs in 24 hours! ● 11/8/2018 5pm local time: Nationwide protests!
  20. 20. Trump Is Not Above The Law
  21. 21. Key Technical Takeaways ● Today, the observed behavior of virality is tightly controlled by the social media platforms ● “Going viral” only means traffic surges if the platforms decide it does. ● With a major exception: influencers can still generate organic viral behavior
  22. 22. The Tech Behind Protest Networks
  23. 23. The Tech Behind Protest Networks ● Hub website: a database of protest events, protest prep material content hub, event map and search tools ● Crowdsourced event creation: anyone can host a protest ● Mobilization tools drive event creation and RSVPs: we email, text, and buy targeted social media ads to find people interested in nearby protest events
  24. 24. Stepping Up to Big Moments ● No one knows when the next big moment will happen ● We need to be able to react and launch quickly ● Massive scale is critical to impact ● ... all on a nonprofit budget!
  25. 25. Problems to Solve ● Can’t predict or control when content will go viral ● Can’t afford to maintain big company levels of tech infrastructure all the time ● Our infrastructure = a complex 30+ entity ecosystem of in-house and vendor tools. Scale testing complex architecture is very time-consuming.
  26. 26. Monitoring and Measurement ● Monitoring is key: monitor everything, through the architectural stack, including vendor tools ● SLAs are key: ○ Aggressive SLAs for in-house tools ○ Observe vendor uptime and availability ○ Plan around cascading failures EWarren has a plan. Do you?
  27. 27. Vendors ● Your system doesn’t scale if your vendors don’t scale. ● Get SLAs and incident response plans into your contracts ● Build a strong relationship with vendors before the next big scaling emergency. ● Do regular build vs buy and platform analysis and understand the cost of switching if you need to
  28. 28. Scaling Incident Response Plans ● What to do before, during and after a scaling incident ● Who to call, what to check, what decisions to make ● Hot backup failover plans for in-house systems ● Static or stopgap backups for vendor systems.
  29. 29. Granular Autoscaling ● Fast reaction time is key ● Breakout virality will have a 100x scaling impact within minutes, not hours ● User action curve will be order of magnitude minutes ● We can’t miss 15min waiting for autoscaling to kick in
  30. 30. Granular Autoscaling ● Consider microservices for scaling bottlenecks: spinning up additional containers is much faster than booting up additional virtual machines ● It’s often cheaper: the per-invocation cost of handling a traffic surge is 10% of the cost of dedicated hardware during the scaling period
  31. 31. Granular Autoscaling ● Scaling response plan should include all distributed systems scaling levers: ○ Quickly add servers (or containerized capacity) ○ Just-in-time upgrade hardware ○ Enable additional caching ○ Queue up bursts of writes to process later
  32. 32. Don’t Forget the CAP Theorem ● Consistency, Availability and Partition Tolerance: pick 2 ● Analyze your architecture ahead of the scaling incident and map out the choices to make in the event of loss of data consistency, component availability, and network partitioning ● Include this in your scaling incident response plan, and be prepared to make hard choices
  33. 33. Conclusion ● Big social media companies have changed the shape of the attention economy ● Social media is an oligarchy, and influencers win ● Traffic surges happen in O(minutes) instead of O(hours) ● Scale planning is harder ● Scale planning is also key: monitor everything, create scaling emergency response plans, get granular
  34. 34. Questions? @ann_lewis

×