This document discusses strategies for scaling a technology team to support high growth and large traffic volumes. Key points include operating distributed infrastructure across multiple data centers; relying on automation, configuration as code, and open source tools; emphasizing continuous deployment with short release cycles; monitoring all systems; and fostering a collaborative culture through open communication and learning opportunities. The team described supports over 25 billion page views per month with over 8 billion recommendations daily across 550 million users globally.

2. Scale Out your team
Building your technology team for scale
SVP Platform Engineering & Operations
Shai Peretz

3. Provide people with the
most interesting, relevant
and trusted content
Audience First.
Our Lighthouse

4. Widget Examples

8. Traffic: >25 Billion PVs per month
>8 Billion recs per day
Reach: >550M users globally
Data: multiple petabytes (dist.)
Servers: >4000 physical nodes
Monitoring: >4m metrics per minute
Team: ~130 Engineers (Dev + Ops)
High growth rate
Scaling Vectors

9. - We design and build our own data
centers (optimization, cost)
- Collocation (less clouds on the
horizon:)
- Active/Active approach
- Rely on external services when
needed (DNS, CDN)
Operational Decisions

10. - No SPOF (n+x)
- Vendor diversity
- Flexible architecture
- Commodity hardware
- Scale out – no central devices
- Open source
Design guidelines

11. - Automate using Chef
- Configuration as code
(Source control)
- Log changes automatically
Configuration Management

12. Architecture – tolerance
Service owner responsibility
War rooms Ops + Dev
Production Party
Open communication with
business
Disaster Recovery

13. Ops:
- Facilities
- Network and Infra
- Visibility
- Data systems
- Production Engineering
Platform Group Structure
Engineering:
- Data delivery & processing
- App Infrastructure
- Build/Dev tools
- Ops tools

14. Skilled Ops engineers
Sit with product development teams
Product/Business KPIs
What? – from product, How? - From Ops
PE team lead – sync, training, implementations
Two way communication
Production Engineering

15. -Very short release cycles
(>100 per day)
- Micro services
- Easy to find issue (fix or rollback)
- Automated deployment process
- Testing & monitoring
- Work procedures and culture
Continuous Deployment

16. Continuous Deployment
 Ownership & Trust
 Product Developers own
their Services
 Platform own
Infrastructure, Hardware &
Network services

17. - Ownership
- Trust
- Good communication
- Learning
Values

18. - Face to Face
- Sync and Share
- Hipchat – always on
- Open channels
Communication

19. - Prevention (Anomaly
detection, trends)
- MTTD, MTTR, MTTS
Technology will eventually fail,
we promise to fix it ASAP!
Stability Goals

20. - Graph everything
- Self serve
- Combination of internal and external
tools:
Collectd/Graphite/Nagios
Logstash/ElasticSearch/Kibana
New Relic/Boundary
Keynote/Pingdom/Catchpoint
- Dashboards – Graphitus/Grafana
- Escalation of critical alerts via
PagerDuty
Visibility

21. Prevention
Immune system
 Unit tests (10k every 10m)
 Integration and Regression
 Self tests
 Monitoring system
 Alerts

22. Keys to success:
- Self serve
- Eliminate false alarms
(Signal to noise ratio)
Automatic full coverage
Immune System

23. To NOC or not to NOC?
Mean Time To Detect

24. Ops on shift
Engineer on call
Escalation policy on PD
Manage on HipChat/Phone
Mean Time To Recover

25. Escalate only critical issues
Measure time to resolve
Blameless learn from events
(Take-Ins)
Respect your team’s sleep!
Mean Time To Sleep

26. After each event
Blameless
Action Items
Publish
Follow up
Take ins

27. Order 2-3 times a year
Load testing + Prediction
Elasticity for engineering
Automatic provisioning
Capacity planning

28. Weekly tech talks
IL TechTalks (+ techtalk week)
Reversim podcast and summit
Internal/External Lectures
Sunday School
Learning
Do you guys ever work??

29. Two weeks dedicated to the
needs of the technical teams
Quality Time

30. Thank You
shai@outbrain.com
Shai Peretz, SVP Platform Engineering & Operations
And Yes, we are hiring…