Makram Mansour, a product leader at LinkedIn, discusses managing an experimentation platform. He outlines LinkedIn's experimentation approach, including running 100 experiments per day across frontend, ranking algorithms, and backend infrastructure. He then discusses LinkedIn's prioritization framework, which quantifies business value across four pillars - value, leverage, urgency, and cost - to prioritize experiments. Finally, he discusses optimizing the product workflow by taking a design thinking approach and integrating experimentation at the core.

1. www.productschool.com
Managing an Experimentation Platform
by LinkedIn Product Leader

2. CERTIFICATES
Your Product Management Certificate Path
Product Leadership
Certificate™
Full Stack Product
Management Certificate™
Product Management
Certificate™

3. Corporate
Training
Level up your team’s Product
Management skills

4. Free Product Management Resources
BOOKS
EVENTS
JOB PORTAL
COMMUNITIES
bit.ly/product_resources
COURSES

5. Makram Mansour, PhD [Product Leader @ LinkedIn]
Managing an Experimentation Platform

6. Makram Mansour
Product Manager, LinkedIn
1 LinkedIn Profile
linkedin.com/in/ilmansour
2 Education
MS & PhD in EE from UIUC
Stanford LEAD from Stanford GSB
3 Product @ LinkedIn
LinkedIn Experimentation & Data Insights
4 Product @ Texas Instruments
Online Design Tools
5 Engineer @ Intel
Intel Server Chipsets
6 Values
Out-of-the-box thinker who is not afraid of taking
risks, gets more committed when people tell me “it
cannot be done”, and strongly believe in the
saying: “where there’s a will, there’s a way”

7. 1
2
3
4
Agenda
Experimentation @ LinkedIn
Backlog Prioritization
Workflow Optimization
Final note on Vision & Strategy

8. Experimentation @
LinkedIn

9. Create economic
opportunity for every member
of the global workforce.

10. Connect the world’s professionals to
make them more productive and
successful.

11. The Economic Graph
740M 57M 14M 38K 120K 280B
Members Companies Jobs Skills Schools Knowledge

12. LinkedIn’s data in motion
100B graph edges, 2B nodes
650K graph queries/second
1,500PB total data storage
11T messages/day on Kafka
25B model parameters in AI models
208M contributions/day
30B Feed Updates viewed/month
750PB average daily storage on Hadoop
12B page views/day
180M messages sent/day
96M profile actions/day

13. Complex growth engine full of “network” effects
One thing we learnt over the years is
that even small, localized changes
can have massive impact!
Maintaining and accelerating growth
requires a strong discipline around
experimentation and data.

14. Cost of not doing A/B testing can be high!
Profile top ads
5-pixel height
decrease
CTR drop on
11/11/2013

15. We test everything
Frontend, ranking algorithms, and backend infra
Experimentation activity
• 100/day new tests
• 400/day tests ramps
• 200/week AI/ML models
Experimentation adoption
• 5000+ experiment owners
• 2000+ WAU
• 41k tests running
simultaneously
Infrastructure activity
• 2M QPS
• 35T/day evaluations

16. 20k Metrics
8k A/B testable Metrics
Single source-of-truth
Self-served, custom
metrics
90k QPS 2.5PB/day
20T/day records
processed on Hadoop
Tracking Events
Kafka
Offline
Infrastructure Unified Metrics Platform
Visualization
Experimentation
Anomaly Detection &
Root Cause Analysis
Massive experimentation and metrics data

17. 1. Targeting
Helps run experiments on
different audience attributes:
- Location
- Job title
- Company
- Industry
- Education
- Language
- Device
- Connections
- etc.
2. Ramping
Helps you easily and safely
ramp/de-ramp a feature over
time.
Provides standard framework to
assist users to ramp with the right
balance of Speed, Quality, and
Risk
3. EXperimentation
Advanced experimentation
infrastructure built for large-scale
use.
Some of the features:
- Multivariate testing
- Advanced randomization
- Metrics Reporting & Alerting
- Insights for ease of interpreting
results
- Variance Reduction Methods
- Most Impactful Experiments
TREX is LinkedIn’s Experimentation Platform
It’s a unified platform for Targeting, Ramping, and EXperimentation

18. Backlog Prioritization

19. 1
2
3
4
Prioritization
Framework
Quantifies business value
Allow “apples-to-apples” stack-rank
and prioritization
Makes decision-making data-driven
and transparent
Sets clear guidelines on what data is
needed for every ask

20. Prioritization Framework
Four Pillars
Value
Measured by impact to Site Up,
Productivity, Revenue, Key Metrics,
or User Satisfaction
Leverage
Measured by # of users impacted in
12 months or less
Urgency
Measured by urgency: blocked,
impaired but have short-term
workaround, or inconvenient
Cost
Measured in engineer-quarters
(engineers needed * no. of quarters
to complete)

21. Computing Value (V) Rubric
Select up to two categories and multiply their corresponding numbers
E.g., V1 = G = 4, V2 = P = 2 ⇒ V = 4 * 2 = 8
GCNs
(G)
Productivity
(P)
Revenue
(R)
Metrics
(M)
User Satisfaction
(S)
GCN Savings:
1. 1 minor GCN/qrtr
2. 2-4 minor GCNs
3. 1-2 medium GCNs
4. 1+ major GCNs
Savings Criteria:
1. < 1hr /usr/mnth
2. 2 - 5 hrs saved
3. 6 - 10 hrs saved
4. > 10 hrs saved
Revenue Lift:
1. < $1M annually
2. $1M - $10M
3. $10M - $30M
4. > $30M
Key Metrics Lift:
1. < 0.5% lift
2. 0.5% - 1.0%
3. 1.0% - 2.0%
4. > 2.0%
NSAT Score Lift:
1. < +0.2 lift
2. +0.3 - +0.5
3. +0.6 - +0.9
4. > +1.0

22. Computing other Rubrics
T-REX Users Impacted
1. < 100 users
2. 100 - 499
3. 500 - 1999
4. > 2000
Leverage
(L)
User’s work impact Costing Formula
Urgency
(U)
Cost
(C)
1. Inconvenient
2. Impaired;
workaround in
place
3. Blocked
1. # of Engineers
2. # of Quarters
3. C = E * Q

23. Prioritization - Final Scoring Guidelines
Final Ranking Criteria
03
● Rank by largest Impact Score
● Override for large ROIs (Quick
Wins)
Calculate ROI
02 ROI = Impact Score / Cost
Calculate Impact Score (IS)
01 Impact Score = Leverage * Value * Urgency

24. Ideal Mix
40%
30%
20%
10%
Big Bets
Home Run
Quick Wins Fall Back
Level of Effort
Ranking

25. Workflow Optimization

26. Optimize your Workflow
Prioritized
50%
Power
Design
Sign-off
Plan Sign-
off
Feature
Request
Feature
Release
Code
Ready
Ideate
Flow diagram, Workflows,
Design, Use cases
Ramp
Verify solution, fix unforeseen
issues, capture metrics, Derive
success stories
Announce
Announce highlighting features
and impact. Include user
testimonials, training, and
documentation
Learn
Problem statements, User
stories, Impact assessment,
Requirements gathering,
Prioritization, PRD
Design
RFC, Hi-Fi Design,
Acceptance test
cases, WBS,
Timeline
Build
Modular code, Bug
bashes, Unit &
Integration testing,
Previews
Design Thinking Approach with Experimentation in the Core of your Purpose
Gain insights in what our customers really want and why.
Build clarity in terms of Before / After state.
Gradual release ramp to efficiently address unforeseen issues.
Measure success metrics and announce with impact!

27. Sample PRD
Areas of focus:
Problem Statement
Core Use Cases
Prioritization Details
Objectives and Key Results
Before and After
Milestones
Sign-offs

28. Note on Vision and
Strategy

29. Be clear on your Vision to Values
Vision
• The Dream. The Future.
What you inspire your product will impact and become.
Mission
• Briefly describes the goals and purpose of your product. Why the
product exists?
Target
Audience
• List down your primary user personas
• Prioritize them. Also be clear on non-users.
• Group them in Producers / Consumers if you have a platform product
Strategy • Series of strategic objectives and roadmap milestones
Priorities
• Stack ranked list of critical initiatives. “If we can only do one thing this
quarter, what would it be?”
Objectives
(Key Metrics)
• True North Metrics
• Sign-Post Metrics
• Guardrail Metrics
Values • Your guiding principles in making day-to-day decisions

30. Thank you

31. www.productschool.com
Part-time Product Management Training Courses
and
Corporate Training