Scaling wix.com to 100 million users
•Download as PPTX, PDF•
5 likes•1,274 views
In few years, Wix grew from a small startup with traditional system architecture (based on a monolithic server) to a company that serves 100 million users. Aviran Mordo explains how Wix evolved from a monolithic system to microservices, using some interesting patterns like CQRS to build a blazing-fast, highly scalable, and highly available system.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Scaling wix.com to 100 million users
Similar to Scaling wix.com to 100 million users (20)
More from Aviran Mordo
More from Aviran Mordo (12)
Recently uploaded
Recently uploaded (20)
Scaling wix.com to 100 million users
- 1. @aviranm Scaling Wix.com Aviran Mordo, Head of Wix Engineering To more than 100M Users linkedin/aviran aviransplace.com@aviranm
- 2. @aviranm With Wix You Can Create a Stunning Website for Free
- 3. @aviranm 20172014 2015201320122011200920072006 Wix is Founded 2008 2010 2016 1st Funding Open Beta 1M Users eCom Mobile 10M Users App Market HTML5 IPO Hive 50M Users Wix Worldwide 100M Users Hotels Rest Shoutout New Editor Bookings Music Events ADI
- 4. @aviranm Wix in Numbers Over 100M users (website builders) Static storage is >5PB of data ~1500 people work at Wix 3 data centers + clouds (Google, Amazon, Azure) 3B HTTP requests/day
- 5. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB ❤ Built for fast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability & testing Intended for short-term use Tomcat, Hibernate, Custom web framework Initial Architecture
- 6. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB One monolithic giant that handled everything – stability problem Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime Tomcat, Hibernate, Custom web framework Initial Architecture
- 7. @aviranm 💣 Large and entangled codebase Hard feature rollout While at the same time, the iPad was released We needed to enable Wix to move fast Poor Development Velocity
- 8. @aviranm
- 9. @aviranm We need new macs Not these macs...
- 10. New MACS 01
- 11. @aviranm Dev-Centric Culture – Involve the developers Developer ownership DevOps (to enable deployment and rollback, fully automated) Continuou s Delivery Product definition (with product) Testing (with QA engineers) Deployment / Rollback (with operations) Monitoring / BI (with BI team) Support Cycle Includes:
- 12. Breaking the monolith into Micro Services 02
- 13. @aviranm I am the server I do everything Resistance is futile! With a Monolithic Giant
- 15. @aviranm Concerns & SLAs ▪ Data Validation ▪ Security / Authentication ▪ Data consistency ▪ Lots of data Edit websites ▪ High availability ▪ High performance ▪ High traffic volume ▪ Long tail (mutable) View Sites (created by Wix Editor) ▪ High availability ▪ High performance ▪ Lots of static files ▪ Very high traffic volume ▪ Viewport optimization ▪ Long tail (immutable) Serve Media
- 17. @aviranm Risk Scoping BA DC ❤ Independent build and deployment ❤ Independent scalability ❤ Independent database 💣 Remaining risk - only from interactions with other services
- 19. @aviranm Micro Services Let you Scale and Move Fast Over 450 micro-services at Wix
- 21. @aviranm JSON Editor Server MySQL Active Sites MySQL Archive ▪ Immutable JSON pages (~20M / day) ▪ Site revisions ▪ Active – standby MySQL cross datacenters The Editor Server
- 22. @aviranm Find Your Critical Path ▪ DB outage with fast recovery = replication ▪ Data poisoning/corruption = revisions / backup ▪ Make the data available at all times = data distribution to multiple locations / providers Protect the Data
- 23. @aviranm Browser 200 OK Notify Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Save flow
- 24. @aviranm Browser 200 OK Notify Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Self-Healing Process
- 25. @aviranm Browser 200 OK Save page(s) Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive No DB Transactions 1. Save each page (JSON) as an atomic operation 2. Page ID is a content based hash (immutable/idempotent) 3. Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice 1 2 3
- 27. @aviranm Wix Media Platform (WixMP) ▪ Eventual-consistent distributed file system (5PB user media files) ▪ Dynamic media processing ▪ Multi datacenter aware ▪ Runs on commodity servers & cloud ▪ CDN cache (immutable files) ▪ Automatic fallback across DCs Critical Path - Serve Fast
- 28. @aviranm Wix Media Manager T WixMP Google Cloud get image.jpg fallback If not in CDN WixMP Amazon CDN
- 30. @aviranm Public Segment Roles ▪ Routing (resolve URLs) ▪ Dispatching (to a renderer) ▪ Rendering (HTML,XML,TXT) Public Server HTML Renderer HTML SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer Flash SEO Renderer www.example.com
- 31. @aviranm Find Your Critical Path Our goal: 99% response time <100ms at peak traffic Public SLA
- 32. @aviranm Editor Segment Public Segment Publish Site ▪ Copy site header (a map of pages for a site) ▪ Copy routing table (CQRS) On Publish:
- 33. @aviranm Built for Speed ▪ Minimize out-of-process hops (2 DB, 1 RPC) ▪ Lookup tables are cached in memory (using Koboshi github.com/wix/Koboshi) ▪ Denormalized data – optimize for read by primary key (MySQL) ▪ Minimize business logic
- 34. @aviranm How a Page Gets Rendered JavaScript imports JSON data (site header + dynamic data) Bootstrap HTML Template that Contains Only Data
- 35. @aviranm Offload the Rendering Work to the Browser The average Intel Core i750 can push up to 7 GFLOPS without overclocking Renderer
- 36. @aviranm ▪ Easy to parse in JavaScript and Java/Scala ▪ Fairly compact text format ▪ Highly compressible (5:1 even for small payloads) ▪ Easy to fix rendering bugs and cross browsers issues (just deploy a new code) Why JSON
- 37. @aviranm Minimum number of public servers needed to serve 100M sites 4
- 38. @aviranm Be Available 99.999% of the Time. Public SLA
- 39. @aviranm Browser Serving a Site A Sunny Day http://example.wix.com LB Public Renderer HTTP Request HTML CDN WixMP Resources / Media Failure Points!
- 40. @aviranm Browser Serving a Site DC Lost LB Public Renderer HTTP Request LB Public Renderer LB Public Renderer Change DNS CDN WixMP http://example.wix.com
- 41. @aviranm Serving a Site Public Lost CDN WixMP Browser LB Public Renderer HTTP Request Public Renderer LB Public Renderer Fallback to 2nd DC http://example.wix.com
- 42. @aviranm Serving a Site Living in the Browser CDN WixMP Browser LB Public Renderer HTTP Request HTML CDN WixMP JSON / Media CDN JSON / Media Fallback Editor Pages Fallback Fallback WixMP http://example.wix.com
- 43. Summary ▪ Identify concerns and SLA for different parts of the system ▪ Build redundancy in critical path (for availability) ▪ De-normalize data (for performance) ▪ Minimize out-of-process hops (for performance) ▪ Take advantage of client’s CPU power
- 44. @aviranm
- 45. @aviranm Methodologies Architecture Culture Stack 2014 20152013201220112010 2016 Wix Framework CI / CD / TDD DevOps Scala Micro Services TDD Redux Angular.js Companies & Guilds React.js Node.js