Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling wix.com to 100 million users

767 views

Published on

In few years, Wix grew from a small startup with traditional system architecture (based on a monolithic server) to a company that serves 100 million users. Aviran Mordo explains how Wix evolved from a monolithic system to microservices, using some interesting patterns like CQRS to build a blazing-fast, highly scalable, and highly available system.

Published in: Software
  • Be the first to comment

Scaling wix.com to 100 million users

  1. 1. @aviranm Scaling Wix.com Aviran Mordo, Head of Wix Engineering To more than 100M Users linkedin/aviran aviransplace.com@aviranm
  2. 2. @aviranm With Wix You Can Create a Stunning Website for Free
  3. 3. @aviranm 20172014 2015201320122011200920072006 Wix is Founded 2008 2010 2016 1st Funding Open Beta 1M Users eCom Mobile 10M Users App Market HTML5 IPO Hive 50M Users Wix Worldwide 100M Users Hotels Rest Shoutout New Editor Bookings Music Events ADI
  4. 4. @aviranm Wix in Numbers Over 100M users (website builders) Static storage is >5PB of data ~1500 people work at Wix 3 data centers + clouds (Google, Amazon, Azure) 3B HTTP requests/day
  5. 5. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB ❤ Built for fast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability & testing Intended for short-term use Tomcat, Hibernate, Custom web framework Initial Architecture
  6. 6. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB One monolithic giant that handled everything – stability problem Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime Tomcat, Hibernate, Custom web framework Initial Architecture
  7. 7. @aviranm 💣 Large and entangled codebase Hard feature rollout While at the same time, the iPad was released We needed to enable Wix to move fast Poor Development Velocity
  8. 8. @aviranm
  9. 9. @aviranm We need new macs Not these macs...
  10. 10. New MACS 01
  11. 11. @aviranm Dev-Centric Culture – Involve the developers Developer ownership DevOps (to enable deployment and rollback, fully automated) Continuou s Delivery Product definition (with product) Testing (with QA engineers) Deployment / Rollback (with operations) Monitoring / BI (with BI team) Support Cycle Includes:
  12. 12. Breaking the monolith into Micro Services 02
  13. 13. @aviranm I am the server I do everything Resistance is futile! With a Monolithic Giant
  14. 14. @aviranm Understanding the System
  15. 15. @aviranm Concerns & SLAs ▪ Data Validation ▪ Security / Authentication ▪ Data consistency ▪ Lots of data Edit websites ▪ High availability ▪ High performance ▪ High traffic volume ▪ Long tail (mutable) View Sites (created by Wix Editor) ▪ High availability ▪ High performance ▪ Lots of static files ▪ Very high traffic volume ▪ Viewport optimization ▪ Long tail (immutable) Serve Media
  16. 16. @aviranm Networking Editor Segment Media Segment Public Segment Wix Segmentation
  17. 17. @aviranm Risk Scoping BA DC ❤ Independent build and deployment ❤ Independent scalability ❤ Independent database 💣 Remaining risk - only from interactions with other services
  18. 18. @aviranm HTML Editor Flash Editor MSM Private Media Public Media Premium Services List DB App Builder App Store App Market Dashboard Mailer Time Zone Public HTML API Public API (Flash) MSP Public Server HTML Renderer HTML SEO Renderer Flash Renderer Flash SEO Renderer Sitemap Renderer Robot.txt Renderer User Server Template Viewer ContactsHUB Site Members Provided Mailing Service Snapsho ter User Pref Feed Me Shout- out Hotels PETRI Site Pref Dist Logger Slicer eCom Renderer eCom Cart eCom Checkout eCom Catalog eCom Orders Payment Facade Account Info HTML API HTML Embeder BlogMobile Activity eCommerce Comments Editor Segment Public Segment Media Segment
  19. 19. @aviranm Micro Services Let you Scale and Move Fast Over 450 micro-services at Wix
  20. 20. Networking Editor Segment Media Segment Public Segment Segment Focus
  21. 21. @aviranm JSON Editor Server MySQL Active Sites MySQL Archive ▪ Immutable JSON pages (~20M / day) ▪ Site revisions ▪ Active – standby MySQL cross datacenters The Editor Server
  22. 22. @aviranm Find Your Critical Path ▪ DB outage with fast recovery = replication ▪ Data poisoning/corruption = revisions / backup ▪ Make the data available at all times = data distribution to multiple locations / providers Protect the Data
  23. 23. @aviranm Browser 200 OK Notify Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Save flow
  24. 24. @aviranm Browser 200 OK Notify Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Self-Healing Process
  25. 25. @aviranm Browser 200 OK Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive No DB Transactions 1. Save each page (JSON) as an atomic operation 2. Page ID is a content based hash (immutable/idempotent) 3. Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice 1 2 3
  26. 26. Networking Editor Segment Media Segment Public Segment Segment Focus
  27. 27. @aviranm Wix Media Platform (WixMP) ▪ Eventual-consistent distributed file system (5PB user media files) ▪ Dynamic media processing ▪ Multi datacenter aware ▪ Runs on commodity servers & cloud ▪ CDN cache (immutable files) ▪ Automatic fallback across DCs Critical Path - Serve Fast
  28. 28. @aviranm Wix Media Manager T WixMP Google Cloud get image.jpg fallback If not in CDN WixMP Amazon CDN
  29. 29. Networking Editor Segment Media Segment Public Segment Segment Focus
  30. 30. @aviranm Public Segment Roles ▪ Routing (resolve URLs) ▪ Dispatching (to a renderer) ▪ Rendering (HTML,XML,TXT) Public Server HTML Renderer HTML SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer Flash SEO Renderer www.example.com
  31. 31. @aviranm Find Your Critical Path Our goal: 99% response time <100ms at peak traffic Public SLA
  32. 32. @aviranm Editor Segment Public Segment Publish Site ▪ Copy site header (a map of pages for a site) ▪ Copy routing table (CQRS) On Publish:
  33. 33. @aviranm Built for Speed ▪ Minimize out-of-process hops (2 DB, 1 RPC) ▪ Lookup tables are cached in memory (using Koboshi github.com/wix/Koboshi) ▪ Denormalized data – optimize for read by primary key (MySQL) ▪ Minimize business logic
  34. 34. @aviranm How a Page Gets Rendered JavaScript imports JSON data (site header + dynamic data) Bootstrap HTML Template that Contains Only Data
  35. 35. @aviranm Offload the Rendering Work to the Browser The average Intel Core i750 can push up to 7 GFLOPS without overclocking Renderer
  36. 36. @aviranm ▪ Easy to parse in JavaScript and Java/Scala ▪ Fairly compact text format ▪ Highly compressible (5:1 even for small payloads) ▪ Easy to fix rendering bugs and cross browsers issues (just deploy a new code) Why JSON
  37. 37. @aviranm Minimum number of public servers needed to serve 100M sites 4
  38. 38. @aviranm Be Available 99.999% of the Time. Public SLA
  39. 39. @aviranm Browser Serving a Site A Sunny Day http://example.wix.com LB Public Renderer HTTP Request HTML CDN WixMP Resources / Media Failure Points!
  40. 40. @aviranm Browser Serving a Site DC Lost LB Public Renderer HTTP Request LB Public Renderer LB Public Renderer Change DNS CDN WixMP http://example.wix.com
  41. 41. @aviranm Serving a Site Public Lost CDN WixMP Browser LB Public Renderer HTTP Request Public Renderer LB Public Renderer Fallback to 2nd DC http://example.wix.com
  42. 42. @aviranm Serving a Site Living in the Browser CDN WixMP Browser LB Public Renderer HTTP Request HTML CDN WixMP JSON / Media CDN JSON / Media Fallback Editor Pages Fallback Fallback WixMP http://example.wix.com
  43. 43. Summary ▪ Identify concerns and SLA for different parts of the system ▪ Build redundancy in critical path (for availability) ▪ De-normalize data (for performance) ▪ Minimize out-of-process hops (for performance) ▪ Take advantage of client’s CPU power
  44. 44. @aviranm
  45. 45. @aviranm Methodologies Architecture Culture Stack 2014 20152013201220112010 2016 Wix Framework CI / CD / TDD DevOps Scala Micro Services TDD Redux Angular.js Companies & Guilds React.js Node.js
  46. 46. @aviranm Q&A http://engineering.wix.com@aviranmhttps://goo.gl/exJl0m @WixEng Aviran Mordo, Head of Wix Engineering

×