Your SlideShare is downloading. ×
Inside Lanyrd's Architecture
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Inside Lanyrd's Architecture

1,217
views

Published on

Video and slides synchronized, mp3 and slide download available at http://bit.ly/13T5YoX. …

Video and slides synchronized, mp3 and slide download available at http://bit.ly/13T5YoX.

Andrew Godwin tells Lanyrd’s story, covering the technology stack, tricks used, and what they would do differently if they could start afresh. Filmed at qconlondon.com.

Andrew Godwin is a web developer, systems administrator and technical architect. He is a developer at Lanyrd as well as a core committer on the Django web framework and specializes in databases and visualization. He is also a private pilot and archer in his spare time. Twitter: @andrewgodwin

Published in: Technology

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,217
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Inside Architecture Lanyrd's Andrew Godwin Web Engineer, Lanyrd @andrewgodwin
  • 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /lanyrd-architecture
  • 3. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  • 4. WHO AM I? Andrew Godwin Web developer Systems administrator Technical architect Django core developer
  • 5. LANYRD: THE EARLY YEARS The Origin Story
  • 6. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 June 2010
  • 7. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 August 2010 Good music on, an orange juice and some CSS fun in front of me, we have an apartment in Casablanca! (for a week or two anyway :) ” ” @natbat 7:19 pm, 18 August 2010
  • 8. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 August 2010 We launched lanyrd.com/ ! Go easy on it, the log files are going a bit nuts, who knew Twitter was viral? ” ” @simonw 10:52 am, 31 August 2010
  • 9. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 August 2010 Right... this clearly isn't sustainable. Going to have to switch the site in to read only mode for a few hours, sorry everyone! ” ” @simonw 11:35 am, 31 August 2010
  • 10. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 January 2011 Natalie and Simon start three months of YCombinator, in California.
  • 11. LANYRD: THE EARLY YEARS 2010 2011 2012 2013 September 2011 Lanyrd closes a $1.4 million seed funding round, moves back to London.
  • 12. LANYRD TODAY 2010 2011 2012 2013 March 2013 ∙ Conferences ∙ Profile pages ∙ Emails ∙ Coverage ∙ Topics ∙ Guides ∙ Mobile app∙ Dashboard
  • 13. LANYRD TODAY 2010 2011 2012 2013 March 2013
  • 14. LANYRD TODAY 2010 2011 2012 2013 March 2013
  • 15. LANYRD TODAY 2010 2011 2012 2013 March 2013
  • 16. LANYRD TODAY 2010 2011 2012 2013 March 2013
  • 17. LANYRD TODAY 2010 2011 2012 2013 March 2013
  • 18. LANYRD TODAY 2010 2011 2012 2013 March 2013 Key dynamic parts: Users tracking/attending events Users tracking each other Users tracking topics and guides
  • 19. THE STACK TODAY What we run on
  • 20. THE STACK TODAY Browser Nginx HAProxy Varnish Gunicorn Main site runtime Amazon S3 Celery Task workers Redis PostgreSQL Solr SSL Termination Web Cache Load balancer Static files & uploads Tasks, Set calcs Search and facetingMain data store Memcached Fragment caching
  • 21. THE STACK TODAY Lanyrd is almost entirely Django (Python) Background tasks use Celery, a Django task queue Management tasks/cron jobs also run inside the framework The Django application is served by Gunicorn containers
  • 22. THE STACK TODAY Main data store for everything except uploads We run a master and a replicated slave Around 80GB of data in five databases Each server runs on a RAID 1 disk array PostgreSQL
  • 23. THE STACK TODAY Task queue transport for Celery and tweet listeners Contains user sets for every conference, user and topic Used for efficient narrowing of queries before Solr is hit Redis
  • 24. THE STACK TODAY Stores conferences, users, sessions and more Very rich metadata on each item Heavy use of sharding thoroughout the site Solr We run a master and a replicated slave
  • 25. THE STACK TODAY First point of call for all requests Caches most anonymous requests Enforces read-only mode if enabled Varnish One used and one hot spare at all times
  • 26. THE STACK TODAY Sits behind Varnish Distributes load amongst frontend servers Re-routes requests during deploys HAProxy Two in use at all times, identically configured
  • 27. THE STACK TODAY Stores all uploaded files from users Upload forms post directly to S3 Serves all static assets for the site (images, CSS, JS) S3 Static assets are versioned with hash to help cache break
  • 28. THE STACK TODAY Browser Nginx HAProxy Varnish Gunicorn Main site runtime Amazon S3 Celery Task workers Redis PostgreSQL Solr SSL Termination Web Cache Load balancer Static files & uploads Tasks, Set calcs Search and facetingMain data store Memcached Fragment caching
  • 29. THE STACK BEFORE What we've eliminated
  • 30. THE STACK BEFORE Stored analytics, logs and some other data Lack of schema meant some bad data persisted Poor complex query performance MongoDB Useful for quick prototyping
  • 31. THE STACK BEFORE Primary data store for things not in MongoDB Very poor complex query performance No advanced field types MySQL Full database locks during schema changes
  • 32. A TALE OF TWO DBS The Great Move of 2012
  • 33. A TALE OF TWO DBS Amazon EC2 MySQL Softlayer PostgreSQL
  • 34. A TALE OF TWO DBS Why? Predictable loading means EC2 unnecessary Better I/O throughput Both moves required database downtime
  • 35. A TALE OF TWO DBS How? Replicate Solr and Redis across to new servers Enter read-only mode Dump MySQL data Convert MySQL dump into PostgreSQL dump Load PostgreSQL dump Re-point DNS, proxy requests from old servers Exit read-only mode
  • 36. A TALE OF TWO DBS Time in read-only mode: 1 ½ hours Downtime: 0 hours
  • 37. CONTENT IS KING The Advantages of Content
  • 38. CONTENT IS KING Read-only mode is entirely viable An hour or two at most Everyone logged out Varnish blocks POSTs, caches everything aggressively
  • 39. CONTENT IS KING Indexing delay is acceptable Most site views are driven by Solr 1 or 2 minute indexing delay Some views add in recent changes directly
  • 40. FEATURE FLAGS Always be deploying
  • 41. FEATURE FLAGS Continuous Deployment We deploy at least 5 times a day, if not 20 Nearly all code goes into master or short-lived branches Anything unreleased is feature flagged
  • 42. FEATURE FLAGS Feature flags Simple named boolean toggles Settable by user, user tag, or conference Can change templates, view code, URLs, etc.
  • 43. FEATURE FLAGS Flag management User tag management
  • 44. WHO WROTE THAT? OH, ME Legacy code & decisions
  • 45. WHO WROTE THAT? OH, ME Technical Debt It's fine to have some - it can speed things up A good chunk of ours is gone, some remains Big schema changes get harder and harder
  • 46. SMALL AND NIMBLE The power of small teams
  • 47. SMALL AND NIMBLE Six people
  • 48. SMALL AND NIMBLE Six people 2.5 Back-end developers 1.75 Front-end developers 1.5 Designers 0.75 System administrators 0.75 Business operations 0.5 Mobile developers
  • 49. SMALL AND NIMBLE Awareness Everyone knows everything that's happening Daily stand-ups Weekly show-and-tell sessions
  • 50. SMALL AND NIMBLE Always deployable Master branch always shippable Large development behind feature flags Code review for nastier changes
  • 51. LESSONS LEARNED What's important here?
  • 52. LESSONS LEARNED Small and nimble Continuous deployment and development style allows easy project changing No long approval processes Less than ½ hour from report to shipped fix
  • 53. LESSONS LEARNED Content is great Read-only mode allows less painful downtimes Heavy caching smooths out our load Learnable load patterns
  • 54. LESSONS LEARNED Fix it while you can The bigger you get, the harder a fix We moved to PostgreSQL just in time Big schema changes now take days of coding
  • 55. LESSONS LEARNED Six amazing people You don't need a big team to write a complex product Communication is absolutely key Using Open Source well is also crucial
  • 56. Thank you. Andrew Godwin Sponsor or promote your company using events? Get in touch: @andrewgodwin http://aeracode.org info@lanyrd.com