Scaling R&D
Organization While
Maintaining Quality
Aviran Mordo
Head Of Back-End Engineering @ Wix
@aviranm
http://www.linkedin.com/in/aviran
http://www.aviransplace.com
Wix In Numbers
• 44M users from 190 countries
• Static storage is >800TB of data

• 3 Data centers + 2 Clouds (Google, Amazon)
• 700M HTTP requests/day

• ~600 people work at Wix
• Of which ~ 200 in R&D
Jan 2014

Deployments (production changes) per month
Structures for Scalability
• There are 2 key common structures in the industry
• Functional unit model
• Business unit model
Functional Model
• Disadvantages
• Lack of product ownership
• Lack of product level expertise
• Hard to predict and plan product
roadmap
• Cross-function communication is
hard
• Less focus on delivery and TTM
Business Unit Model
• Disadvantages
•
•
•
•

Product integration is hard
Resource and work duplication
Architecture alignment is hard
Technology knowledge sharing is
hard
• Limited opportunity for
professional development
Our Assumptions
• There is no perfect model

• It depends on the company’s current challenges,
life cycle phase and culture
• Every model should be tuned constantly and evolve
with the company
Wix’s Gangs & Guilds Model –
How?
• Guild – a group of people
that share expertise,
knowledge, tools, code,
and practices
Guild

• Gang – a group of people
that work in a related
products
• Compose of all required
resources from the
different disciplines
(Guild)

Guild

Gang

Gang

Gang
Wix’s Gangs & Guilds Model –
Why? power of the Guild
• Technical
• Independence of the Gang
• Healthy balance between product and tech
• Product features and technical equal in priority

What

Guild Leader

How

Team member

How

Team member

Guild Leader
Dev-Centric Culture
Traditional Dev Pipeline

Product

21:31

Dev

QA

Operations
Product

21:31

Dev

QA

Operations
What Is The Common
Denominator?
• Product manager
• Project manager
• QA
• Operations
• DBA
Dev Centric Culture – Involve The Developer
– How?
• Product definition (with product)
• Development (with architect)
• Testing (with QA developers)
• Deployment / Rollback(with operations)
• Monitoring / BI (with BI team)
• DevOps – to enable deployment and
rollback, fully automated
Dev Centric Culture – Why?

esponsibility
wnership
uality
Project Rotation – How?
• Team usually has more than one project
• Team members rotate tasks between projects
• Developers can switch teams and Gangs
Project Rotation – Why?
• There is no single person with a knowledge on a
specific component
• Ongoing review
• Keep it interesting
• It is everyone's responsibility
• Everybody owns everything
Quality Thursday – How?
• A software engineer works 4 days with his Gang
• Thursday is a Guild day.
• Developers stop working on their designated
projects and conduct quality enhancing activities
with the Guild.
Quality Thursday – Why?
• Assimilates people into the Guild
• Builds cross-team relationships
• Shares knowledge
• Enhances the culture
• Promotes innovation
Bug Hunt – How?
• Service owner picks a service running on production
• Explains the service to all Guild members
• What the service does
• Every exception thrown in production
• Performance matrix

• Get a list of AI from group feedback (sometimes for
dependent services)
Bug Hunt - Why?
• Teach developers about services they don’t own
• Understand how your application behaves on
production
• Open discussion and ideas from group members
• Discover unexpected use patterns
• Find bugs on production
Retrospective – How ?
• Whenever developers encounter issues, they are
encouraged to post them on a board for public
discussion
• Anything can be discussed
• Topics posted on the board during the week
constitute the agenda of the retrospective
Retrospective – Why ?
• Solve problems
• Share lesson learned.
• Suggestions on how to improve:
•
•
•
•

Our team
Quality of products
Process
Effectiveness of the R&D organization.
Tech Talks – How?
• Every developer can give a talk about anything
• People from other departments in the company
• Guest speakers
• Open to anyone from Wix
• Using Meetup http://www.meetup.com/at-wix/ to
invite outsiders to our internal talks (if appropriate)
• Talks videos http://goo.gl/IDqXTi
Tech Talks – Why ?
• Training new and old employees
• Educating about other activities in the R&D
• Educating about other departments activities and
concerns in the company
Thursday Tasks – How ?
• Pay technical debt (preferably not your own)
• Work on a task / feature NOT in your project
• Read and learn something new
• Open source a project
• 1 on 1 mentoring
Thursday Tasks – Why?
• Learn other projects (easier to step in if necessary)
• Unbiased review of other projects
• Learn about problems and solutions other teams
faced and solved that you may also encounter.
• Find repeating patterns across projects and
generalize a solution
• Find interesting projects to switch to.
• Leave the code in a better shape than they got it
Test Driven Development – How?

• No new code is pushed to Git without being fully tested
• We currently have around 10,000 automated tests

• Before fixing a bug first write a test to reproduce the bug

• Cover legacy (untested) systems with Integration tests

21:31
Test Driven Development –
Why?
• Developers are responsible for their own code
• Less bugs
• Easier to enter someone else’s project
Do Not Compromise On Hiring
• Hire only good people
• Fit the culture
• Excellent technically

• Candidates can be dropped
• By anyone
• At any time

• If there is any doubt, then
there is no doubt
Quality

Trust

Independence

Growth
Dedicated Resources = Organization
Structure
Minimize Architectural
Dependencies
Lesson learned: System architecture that decouples concerns
• Service oriented architecture (SOA)
• Separation of client (UI) and server projects
• Dedicated data stores for each service
• Stateless services
Communication Channels
• To company wide activities
• To knowledge centers
• To key personnel
Quality

Trust

Independence

Growth
Lessons Learned: Growing
Teams
Seeds Of Ambassadors
• Train a group of ambassadors that practice devcentric culture from the Guild
• Build new teams with at least one dev-centric
ambassador to assimilate new teams into the
culture.
• Beware of hiring more people than you can train /
assimilate successfully
Q&A
Read some more about maintaining quality R&D: http://goo.gl/c3WLsz

Aviran Mordo
Head of Back-End Engineering @ Wix

@aviranm
linkedin.com/in/aviran
aviransplace.com

Scaling r&d org while maintaining quality

  • 1.
    Scaling R&D Organization While MaintainingQuality Aviran Mordo Head Of Back-End Engineering @ Wix @aviranm http://www.linkedin.com/in/aviran http://www.aviransplace.com
  • 2.
    Wix In Numbers •44M users from 190 countries • Static storage is >800TB of data • 3 Data centers + 2 Clouds (Google, Amazon) • 700M HTTP requests/day • ~600 people work at Wix • Of which ~ 200 in R&D
  • 4.
  • 5.
    Structures for Scalability •There are 2 key common structures in the industry • Functional unit model • Business unit model
  • 6.
    Functional Model • Disadvantages •Lack of product ownership • Lack of product level expertise • Hard to predict and plan product roadmap • Cross-function communication is hard • Less focus on delivery and TTM
  • 7.
    Business Unit Model •Disadvantages • • • • Product integration is hard Resource and work duplication Architecture alignment is hard Technology knowledge sharing is hard • Limited opportunity for professional development
  • 8.
    Our Assumptions • Thereis no perfect model • It depends on the company’s current challenges, life cycle phase and culture • Every model should be tuned constantly and evolve with the company
  • 10.
    Wix’s Gangs &Guilds Model – How? • Guild – a group of people that share expertise, knowledge, tools, code, and practices Guild • Gang – a group of people that work in a related products • Compose of all required resources from the different disciplines (Guild) Guild Gang Gang Gang
  • 11.
    Wix’s Gangs &Guilds Model – Why? power of the Guild • Technical • Independence of the Gang • Healthy balance between product and tech • Product features and technical equal in priority What Guild Leader How Team member How Team member Guild Leader
  • 13.
  • 14.
  • 16.
  • 17.
    What Is TheCommon Denominator? • Product manager • Project manager • QA • Operations • DBA
  • 18.
    Dev Centric Culture– Involve The Developer – How? • Product definition (with product) • Development (with architect) • Testing (with QA developers) • Deployment / Rollback(with operations) • Monitoring / BI (with BI team) • DevOps – to enable deployment and rollback, fully automated
  • 19.
    Dev Centric Culture– Why? esponsibility wnership uality
  • 20.
    Project Rotation –How? • Team usually has more than one project • Team members rotate tasks between projects • Developers can switch teams and Gangs
  • 21.
    Project Rotation –Why? • There is no single person with a knowledge on a specific component • Ongoing review • Keep it interesting • It is everyone's responsibility • Everybody owns everything
  • 23.
    Quality Thursday –How? • A software engineer works 4 days with his Gang • Thursday is a Guild day. • Developers stop working on their designated projects and conduct quality enhancing activities with the Guild.
  • 24.
    Quality Thursday –Why? • Assimilates people into the Guild • Builds cross-team relationships • Shares knowledge • Enhances the culture • Promotes innovation
  • 25.
    Bug Hunt –How? • Service owner picks a service running on production • Explains the service to all Guild members • What the service does • Every exception thrown in production • Performance matrix • Get a list of AI from group feedback (sometimes for dependent services)
  • 26.
    Bug Hunt -Why? • Teach developers about services they don’t own • Understand how your application behaves on production • Open discussion and ideas from group members • Discover unexpected use patterns • Find bugs on production
  • 27.
    Retrospective – How? • Whenever developers encounter issues, they are encouraged to post them on a board for public discussion • Anything can be discussed • Topics posted on the board during the week constitute the agenda of the retrospective
  • 28.
    Retrospective – Why? • Solve problems • Share lesson learned. • Suggestions on how to improve: • • • • Our team Quality of products Process Effectiveness of the R&D organization.
  • 29.
    Tech Talks –How? • Every developer can give a talk about anything • People from other departments in the company • Guest speakers • Open to anyone from Wix • Using Meetup http://www.meetup.com/at-wix/ to invite outsiders to our internal talks (if appropriate) • Talks videos http://goo.gl/IDqXTi
  • 30.
    Tech Talks –Why ? • Training new and old employees • Educating about other activities in the R&D • Educating about other departments activities and concerns in the company
  • 31.
    Thursday Tasks –How ? • Pay technical debt (preferably not your own) • Work on a task / feature NOT in your project • Read and learn something new • Open source a project • 1 on 1 mentoring
  • 32.
    Thursday Tasks –Why? • Learn other projects (easier to step in if necessary) • Unbiased review of other projects • Learn about problems and solutions other teams faced and solved that you may also encounter. • Find repeating patterns across projects and generalize a solution • Find interesting projects to switch to. • Leave the code in a better shape than they got it
  • 34.
    Test Driven Development– How? • No new code is pushed to Git without being fully tested • We currently have around 10,000 automated tests • Before fixing a bug first write a test to reproduce the bug • Cover legacy (untested) systems with Integration tests 21:31
  • 35.
    Test Driven Development– Why? • Developers are responsible for their own code • Less bugs • Easier to enter someone else’s project
  • 37.
    Do Not CompromiseOn Hiring • Hire only good people • Fit the culture • Excellent technically • Candidates can be dropped • By anyone • At any time • If there is any doubt, then there is no doubt
  • 38.
  • 40.
    Dedicated Resources =Organization Structure
  • 41.
    Minimize Architectural Dependencies Lesson learned:System architecture that decouples concerns • Service oriented architecture (SOA) • Separation of client (UI) and server projects • Dedicated data stores for each service • Stateless services
  • 42.
    Communication Channels • Tocompany wide activities • To knowledge centers • To key personnel
  • 43.
  • 44.
  • 45.
    Seeds Of Ambassadors •Train a group of ambassadors that practice devcentric culture from the Guild • Build new teams with at least one dev-centric ambassador to assimilate new teams into the culture. • Beware of hiring more people than you can train / assimilate successfully
  • 47.
    Q&A Read some moreabout maintaining quality R&D: http://goo.gl/c3WLsz Aviran Mordo Head of Back-End Engineering @ Wix @aviranm linkedin.com/in/aviran aviransplace.com

Editor's Notes

  • #6 Business unit – independent units with no collaborationFunctional – by skills (server, client) all mixed
  • #10 Spotify model (we were already doing it, it formalized it for us)
  • #20 זה לא מספיק להיות מעורב, התרבות והפעילות חשובות לא פחות
  • #37 People are the key
  • #41 This is the Gang