Wix Architecture at Scale 
Aviran Mordo 
Head of Back-End Engineering @ Wix 
@aviranm 
linkedin.com/in/aviran 
aviransplac...
Watch the video with slide 
synchronization on InfoQ.com! 
http://www.infoq.com/presentations 
/wix-architecture 
InfoQ.co...
Presented at QCon London 
www.qconlondon.com 
Purpose of QCon 
- to empower software development by facilitating the sprea...
Wix in Numbers 
Over 45,000,000 users 
1M new users/month 
Static storage is >800TB of data 
1.5TB new files/day 
3 data c...
Initial Architecture 
Tomcat, Hibernate, custom web framework 
Built for fast development 
Stateful login (Tomcat session)...
The Monolithic Giant 
One monolithic server that handled everything 
Dependency between features 
Changes in unrelated are...
Breaking the System Apart
Concerns and SLA 
Edit websites 
Data Validation 
Security / Authentication 
Data consistency 
Lots of data 
Serving Media...
Wix Segmentation 
Networking 
1. Editor Segment 2. Media Segment 3. Public Segment
Making SOA Guidelines 
Each service has its own database (if one is needed) 
Only one service can write to a specific DB 
...
1. Editor Segment
Editor Server 
Immutable JSON pages (~2.5M / day) 
Site revisions 
Active – standby MySQL cross datacenters 
Editor Server...
Protect The Data 
Protect against DB outage with fast recovery = replication 
Protect against data poisoning/corruption = ...
Saving Editor Data 
Browser 
Editor 
Server 
Static 
Grid 
Notify 
Google 
Cloud 
Storage 
MySQL 
Active 
Sites 
MySQL 
Ar...
Self Healing Process 
Browser 
Editor 
Server 
Static 
Grid 
Save Page(s) 
Save Page 
Upload 
Notify 
Download Page 
Googl...
No DB Transactions 
Save each page (JSON) as an atomic operation 
Page ID is a content based hash (immutable/idempotent) 
...
2. Media Segment
Prospero – Wix Media Storage 
800TB user media files 
3M files uploaded daily 
500M metadata records 
Dynamic media proces...
Prospero 
Eventual consistent distributed file system 
Multi datacenter aware 
Automatic fallback cross DC 
Run on commodi...
Prospero – Wix Media Manager 
Austin 
x36 
T 
x36 
T 
x32 
get image.jpg 
First 
fallback 
Second 
fallback 
If not in 
CD...
3. Public Segment
Public Segment Roles 
Routing (resolve URLs) 
Dispatching (to a renderer) 
Rendering (HTML,XML,TXT) 
Public 
Server 
HTML ...
Public SLA 
Response time <100ms at peak traffic
Publish A Site 
Publish site header (a map of pages for a site) 
Publish routing table 
Publish site header / routes 
Edit...
Built For Speed 
Minimize out-of-service hops (2 DB, 1 RPC) 
Lookup tables are cached in memory, updated every 5 minutes 
...
How a Page Gets Rendered 
Bootstrap HTML template that contains only data 
Only JavaScript imports 
JSON data (site-header...
Offload rendering work to the browser
The average Intel 
Core i750 can push 
up to 7 GFLOPS 
without overclocking
Why JSON? 
Easy to parse in JavaScript and Java/Scala 
Fairly compact text format 
Highly compressible (5:1 even for small...
Minimum Number of Public Servers 
Needed to Serve 45M Sites 
4
Public SLA 
Be Available 99.99999%
Serving a Site – Sunny Day 
CDN Statics 
Archive 
Browser 
http://example.wix.com 
HTTP 
Request 
Notify 
site view 
Store...
Serving a Site – DC Lost 
CDN Statics 
Archive 
Browser 
http://example.wix.com 
LB 
Public 
Renderer 
LB 
Public 
Rendere...
Serving a Site – Public Lost 
CDN Statics 
Archive 
Browser 
http://example.wix.com 
LB 
Public 
Renderer 
Get 
Cached HTM...
Living in the Browser 
CDN Statics 
Archive 
Browser 
http://example.wix.com 
LB 
Public 
Renderer 
Fallback 
Editor 
JSON...
Summary 
Identify your critical path and concerns 
Build redundancy in critical path (for availability) 
De-normalize data...
Q&A 
http://goo.gl/Oo3lGr 
Aviran Mordo 
Head of Back-End Engineering @ Wix 
@aviranm 
linkedin.com/in/aviran 
aviransplac...
Watch the video with slide synchronization on 
InfoQ.com! 
http://www.infoq.com/presentations/wix-architecture
Wix Architecture at Scale
Wix Architecture at Scale
Wix Architecture at Scale
Wix Architecture at Scale
Upcoming SlideShare
Loading in …5
×

Wix Architecture at Scale

1,112 views

Published on

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1trOqf6.

Aviran Mordo introduces Wix's architecture, a highly available eventually consistent system, along with patterns for rendering many websites with a relatively small number of servers. Filmed at qconlondon.com.

Aviran Mordo is the head of back-end engineering at Wix. He has over 20 years of experience in the software industry and has filled many engineering roles and leading positions, from designing and building the US National Electronic Records Archives prototype to building the infrastructure that powers indeed.com, the leading job search engine in the world.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,112
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Wix Architecture at Scale

  1. 1. Wix Architecture at Scale Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com
  2. 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /wix-architecture InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
  3. 3. Presented at QCon London www.qconlondon.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. Wix in Numbers Over 45,000,000 users 1M new users/month Static storage is >800TB of data 1.5TB new files/day 3 data centers + 2 clouds (Google, Amazon) 300 servers 700M HTTP requests/day 600 people work at Wix, of which ~ 200 in R&D
  5. 5. Initial Architecture Tomcat, Hibernate, custom web framework Built for fast development Stateful login (Tomcat session), Ehcache, file uploads No consideration for performance, scalability and testing Intended for short-term use Lighttpd (file serving) MySQL DB Wix (Tomcat)
  6. 6. The Monolithic Giant One monolithic server that handled everything Dependency between features Changes in unrelated areas of the system caused deployment of the whole system Failure in unrelated areas will cause system wide downtime
  7. 7. Breaking the System Apart
  8. 8. Concerns and SLA Edit websites Data Validation Security / Authentication Data consistency Lots of data Serving Media High availability High performance Lots of static files Very high traffic volume Viewport optimization Cacheable data View sites, created by Wix editor High availability High performance High traffic volume Long tail
  9. 9. Wix Segmentation Networking 1. Editor Segment 2. Media Segment 3. Public Segment
  10. 10. Making SOA Guidelines Each service has its own database (if one is needed) Only one service can write to a specific DB There may be additional read-only services that directly accesses the DB (for performance reasons) Services are stateless No DB transactions Cache is not a building block, but an optimization
  11. 11. 1. Editor Segment
  12. 12. Editor Server Immutable JSON pages (~2.5M / day) Site revisions Active – standby MySQL cross datacenters Editor Server MySQL Active Sites MySQL Archive
  13. 13. Protect The Data Protect against DB outage with fast recovery = replication Protect against data poisoning/corruption = revisions / backup Make the data available at all times = data distribution to multiple locations / providers
  14. 14. Saving Editor Data Browser Editor Server Static Grid Notify Google Cloud Storage MySQL Active Sites MySQL Archive Notify Archive (Amazon) Archive (Google) Save Page(s) 200 OK Upload Save Page DC replication Download Page MySQL Active Sites MySQL Archive
  15. 15. Self Healing Process Browser Editor Server Static Grid Save Page(s) Save Page Upload Notify Download Page Google Cloud Storage MySQL Archive MySQL Active Sites MySQL Archive DC replication Notify Archive (Amazon) Archive (Google) MySQL Active Sites 200 OK
  16. 16. No DB Transactions Save each page (JSON) as an atomic operation Page ID is a content based hash (immutable/idempotent) Finalize transaction by sending site header (list of pages) Can generate orphaned pages, not a problem in practice
  17. 17. 2. Media Segment
  18. 18. Prospero – Wix Media Storage 800TB user media files 3M files uploaded daily 500M metadata records Dynamic media processing • Picture resize, crop and sharpen “on the fly” • Watermark • Audio format conversion
  19. 19. Prospero Eventual consistent distributed file system Multi datacenter aware Automatic fallback cross DC Run on commodity servers & cloud
  20. 20. Prospero – Wix Media Manager Austin x36 T x36 T x32 get image.jpg First fallback Second fallback If not in CDN Google Cloud Tampa x36 T x36 T x32 CDN
  21. 21. 3. Public Segment
  22. 22. Public Segment Roles Routing (resolve URLs) Dispatching (to a renderer) Rendering (HTML,XML,TXT) Public Server HTML Renderer HTML SEO Renderer Flash Renderer Sitemap Renderer Robots.txt Renderer www.example.com Flash SEO Renderer
  23. 23. Public SLA Response time <100ms at peak traffic
  24. 24. Publish A Site Publish site header (a map of pages for a site) Publish routing table Publish site header / routes Editor Segment Public Segment
  25. 25. Built For Speed Minimize out-of-service hops (2 DB, 1 RPC) Lookup tables are cached in memory, updated every 5 minutes Denormalized data – optimize for read by primary key (MySQL) Minimize business logic
  26. 26. How a Page Gets Rendered Bootstrap HTML template that contains only data Only JavaScript imports JSON data (site-header + dynamic data) No “real” HTML view
  27. 27. Offload rendering work to the browser
  28. 28. The average Intel Core i750 can push up to 7 GFLOPS without overclocking
  29. 29. Why JSON? Easy to parse in JavaScript and Java/Scala Fairly compact text format Highly compressible (5:1 even for small payloads) Easy to fix rendering bugs (just deploy a new client code)
  30. 30. Minimum Number of Public Servers Needed to Serve 45M Sites 4
  31. 31. Public SLA Be Available 99.99999%
  32. 32. Serving a Site – Sunny Day CDN Statics Archive Browser http://example.wix.com HTTP Request Notify site view Store HTML to cache LB Public Renderer HTML Resources / Media HTTP Request
  33. 33. Serving a Site – DC Lost CDN Statics Archive Browser http://example.wix.com LB Public Renderer LB Public Renderer Change DNS HTTP Request
  34. 34. Serving a Site – Public Lost CDN Statics Archive Browser http://example.wix.com LB Public Renderer Get Cached HTML Version HTML HTTP Request
  35. 35. Living in the Browser CDN Statics Archive Browser http://example.wix.com LB Public Renderer Fallback Editor JSON / Media HTML HTTP Request Fallback
  36. 36. Summary Identify your critical path and concerns Build redundancy in critical path (for availability) De-normalize data (for performance) Minimize out-of-process hops (for performance) Take advantage of client’s CPU power
  37. 37. Q&A http://goo.gl/Oo3lGr Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com
  38. 38. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/wix-architecture

×