Downloaded 18 times
Uploaded byAviran Mordo
Scaling wix.com to 100 million users
The document details the growth and architecture evolution of Wix, highlighting its journey to over 100 million users and the shift from a monolithic system to microservices for improved scalability and performance. It describes the challenges faced, including code entanglement and development velocity, and the implementation of a development-centric culture with continuous delivery practices. Key features include robust data handling, high availability, and leveraging client-side processing to enhance performance.
More Related Content
Similar to Scaling wix.com to 100 million users
Scaling wix.com to 100 million users
- 1. @aviranm Scaling Wix.com Aviran Mordo,Head of Wix Engineering To more than 100M Users linkedin/aviran aviransplace.com@aviranm
- 2. @aviranm With Wix YouCan Create a Stunning Website for Free
- 3. @aviranm 20172014 2015201320122011200920072006 Wix is Founded 20082010 2016 1st Funding Open Beta 1M Users eCom Mobile 10M Users App Market HTML5 IPO Hive 50M Users Wix Worldwide 100M Users Hotels Rest Shoutout New Editor Bookings Music Events ADI
- 4. @aviranm Wix in Numbers Over100M users (website builders) Static storage is >5PB of data ~1500 people work at Wix 3 data centers + clouds (Google, Amazon, Azure) 3B HTTP requests/day
- 5. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB ❤ Built forfast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability & testing Intended for short-term use Tomcat, Hibernate, Custom web framework Initial Architecture
- 6. @aviranm Lighttpd (file serving) Wix (Tomcat) MySQL DB One monolithicgiant that handled everything – stability problem Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime Tomcat, Hibernate, Custom web framework Initial Architecture
- 7. @aviranm 💣 Large and entangledcodebase Hard feature rollout While at the same time, the iPad was released We needed to enable Wix to move fast Poor Development Velocity
- 8.
- 9. @aviranm We need newmacs Not these macs...
- 10.
- 11. @aviranm Dev-Centric Culture –Involve the developers Developer ownership DevOps (to enable deployment and rollback, fully automated) Continuou s Delivery Product definition (with product) Testing (with QA engineers) Deployment / Rollback (with operations) Monitoring / BI (with BI team) Support Cycle Includes:
- 12. Breaking the monolithinto Micro Services 02
- 13. @aviranm I am theserver I do everything Resistance is futile! With a Monolithic Giant
- 14.
- 15. @aviranm Concerns & SLAs ▪Data Validation ▪ Security / Authentication ▪ Data consistency ▪ Lots of data Edit websites ▪ High availability ▪ High performance ▪ High traffic volume ▪ Long tail (mutable) View Sites (created by Wix Editor) ▪ High availability ▪ High performance ▪ Lots of static files ▪ Very high traffic volume ▪ Viewport optimization ▪ Long tail (immutable) Serve Media
- 16.
- 17. @aviranm Risk Scoping BA DC ❤ Independentbuild and deployment ❤ Independent scalability ❤ Independent database 💣 Remaining risk - only from interactions with other services
- 18.
- 19. @aviranm Micro Services Letyou Scale and Move Fast Over 450 micro-services at Wix
- 20.
- 21. @aviranm JSON Editor Server MySQL Active Sites MySQL Archive ▪ Immutable JSONpages (~20M / day) ▪ Site revisions ▪ Active – standby MySQL cross datacenters The Editor Server
- 22. @aviranm Find Your Critical Path▪ DB outage with fast recovery = replication ▪ Data poisoning/corruption = revisions / backup ▪ Make the data available at all times = data distribution to multiple locations / providers Protect the Data
- 23. @aviranm Browser 200 OK Notify Save page(s)Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Save flow
- 24. @aviranm Browser 200 OK Notify Save page(s)Editor Server Save page MySQL Active Sites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive WixMedia (Google) Upload GCS WixMedia (Amazon) S3 Self-Healing Process
- 25. @aviranm Browser 200 OK Save page(s) Editor Server Save page MySQL ActiveSites MySQL ArchiveDC Replication MySQL Active Sites MySQL Archive No DB Transactions 1. Save each page (JSON) as an atomic operation 2. Page ID is a content based hash (immutable/idempotent) 3. Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice 1 2 3
- 26.
- 27. @aviranm Wix Media Platform (WixMP) ▪Eventual-consistent distributed file system (5PB user media files) ▪ Dynamic media processing ▪ Multi datacenter aware ▪ Runs on commodity servers & cloud ▪ CDN cache (immutable files) ▪ Automatic fallback across DCs Critical Path - Serve Fast
- 28. @aviranm Wix Media Manager T WixMP Google Cloud getimage.jpg fallback If not in CDN WixMP Amazon CDN
- 29.
- 30. @aviranm Public Segment Roles ▪Routing (resolve URLs) ▪ Dispatching (to a renderer) ▪ Rendering (HTML,XML,TXT) Public Server HTML Renderer HTML SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer Flash SEO Renderer www.example.com
- 31. @aviranm Find Your Critical PathOur goal: 99% response time <100ms at peak traffic Public SLA
- 32. @aviranm Editor Segment Public Segment Publish Site ▪ Copysite header (a map of pages for a site) ▪ Copy routing table (CQRS) On Publish:
- 33. @aviranm Built for Speed ▪Minimize out-of-process hops (2 DB, 1 RPC) ▪ Lookup tables are cached in memory (using Koboshi github.com/wix/Koboshi) ▪ Denormalized data – optimize for read by primary key (MySQL) ▪ Minimize business logic
- 34. @aviranm How a PageGets Rendered JavaScript imports JSON data (site header + dynamic data) Bootstrap HTML Template that Contains Only Data
- 35. @aviranm Offload the RenderingWork to the Browser The average Intel Core i750 can push up to 7 GFLOPS without overclocking Renderer
- 36. @aviranm ▪ Easy toparse in JavaScript and Java/Scala ▪ Fairly compact text format ▪ Highly compressible (5:1 even for small payloads) ▪ Easy to fix rendering bugs and cross browsers issues (just deploy a new code) Why JSON
- 37. @aviranm Minimum number ofpublic servers needed to serve 100M sites 4
- 38. @aviranm Be Available 99.999% ofthe Time. Public SLA
- 39. @aviranm Browser Serving a Site ASunny Day http://example.wix.com LB Public Renderer HTTP Request HTML CDN WixMP Resources / Media Failure Points!
- 40. @aviranm Browser Serving a Site DCLost LB Public Renderer HTTP Request LB Public Renderer LB Public Renderer Change DNS CDN WixMP http://example.wix.com
- 41. @aviranm Serving a Site PublicLost CDN WixMP Browser LB Public Renderer HTTP Request Public Renderer LB Public Renderer Fallback to 2nd DC http://example.wix.com
- 42. @aviranm Serving a Site Livingin the Browser CDN WixMP Browser LB Public Renderer HTTP Request HTML CDN WixMP JSON / Media CDN JSON / Media Fallback Editor Pages Fallback Fallback WixMP http://example.wix.com
- 43. Summary ▪ Identify concernsand SLA for different parts of the system ▪ Build redundancy in critical path (for availability) ▪ De-normalize data (for performance) ▪ Minimize out-of-process hops (for performance) ▪ Take advantage of client’s CPU power
- 44.
- 45. @aviranm Methodologies Architecture Culture Stack 2014 20152013201220112010 2016 Wix Framework CI/ CD / TDD DevOps Scala Micro Services TDD Redux Angular.js Companies & Guilds React.js Node.js
- 46.