The BBC prepared extensively for hosting their online coverage of the 2012 London Olympics by conducting load testing, performance testing, and resilience testing on their systems and platforms. Through this testing, they found problems at various architecture layers that had not been discovered during testing on their staging environment. The testing exposed issues like poor application performance, timeouts not being properly configured, and a lack of caching. The extensive testing in the months leading up to the Olympics helped the BBC identify and fix issues before they impacted their online coverage during the live events.

1. The BBC’s
Experience of
preparing for the
2012 London
Olympics
For Velocity London
2012
Andy “Bob” Brockhurst
Principal Engineer BBC
Platforms/Frameworks

2. Introduction
• The Team
• LAMP (without the M)
• Tomcat Java Service Layer
• Custom apache modules
• Varnish with extensions
• ZendFramework..
– ...customised a.k.a PAL
• Barlesque

3. How the BBC works
• One domain[1]
• Two technology stacks[2]
• Cert’s and SSL
• ProxyPass’
• Apps are a TLD
• 360+ apps
• Everyone shares everything[3]
[1] Okay there are several but they are all really the same one.
[2] Okay, three if you are going to be picky.
[3] Yes really, everything!

5. Network Topology
• Dual DC[1]
• No DC affinity[2]
[1] One more soon(ish)
[2] Well a couple of apps do[3]
[3] We don't talk about them

6. Network Topology

7. Traffic Routing
• TM -> PAL
• TM -> Varnish -> TM -> PAL
• TM -> Service Layer
• TM -> Varnish -> TM -> Service Layer

8. Traffic Routing
Request
iplayer | everything else
sport |
v
TM .-> TM .--> TM .--> TM
| / | / | / |
| / | / | / |
v / v / api v / v
Varnish PAL Varnish Dynamite

9. Environments
• Integration
• Test
• Staging
• Live
• Journalism

10. Right let’s do some testing

12. Why?
• Too much change
– Network Architecture
– Server Configurations
– Load balancers
– Peering points
• High Profile
• Gain confidence

13. Gaining Confidence
• Load testing on Stage
– Tests individual applications
– Single endpoints only
– No concurrent load
• Real hardware
• Real data
– As much as possible
• Real Journalism

14. Other objectives
• Maintain BAU
• Handle failure gracefully
• Deliver Expectation

15. “What the Abdication did for Radio
and the Coronation did for
Television,
London 2012 will do for Online.”

16. Current Volumetrics
• Big numbers for sport
– 9M users/day
– 90M views/day
• Punishing peaks
– Saturday football final scores 4000 pv/s
– ~750k Concurrent users
• Wimbledon
– 1700 pv/s

17. Expected Volumetrics
• Expected peaks
– 1.5M concurrent users
– 60k different sports pages
• 2,500 per minute
– 30% video via iPlayer

19. Timeline
• March 2012 (T minus 5 months)
– Team members assigned
– Resilience testing
– Performance testing
• Testing with External Partner

20. Olympics Run-up
• Jubilee (2nd June)
• Euros (8th June)
• Wimbledon (25th June)
• Formula One

22. Cloud Testing
• International testing
• Detailed test results

23. Cloud Testing
• First performance test breaks live
• Exposed monitoring issues
• Couldn’t internally diagnose
• Lots of tail, grep, awk, sed.

24. Early Findings
• Stop tests
• Monitoring
• UK Data centre capacity
• UK Data centre network segments

25. (Not) Caching kills
• Conditional modules
• Non-Olympics related modules
– Commenting / Favourites
• Lowers cachability
• Testing an immature product
• Subsequent testing exposed more

27. What is a failure?
• Error 500?
• Blank pages?
• Stale content?
• Slow pages?
• Burning data centres?

28. Resilience Testing
• Kill backends
• Traffic Manager
– Screw with headers
– Screw with status (418 anyone)
– Truncate body
• Introduce waits
• Limit cache sizes
• Reduce network bandwidth

30. Early findings
• Failure mode testing
– Everything is a SPOF
– Performance sucks in a failure

31. Specific findings
• Monitoring Thresholds
• Verbose logging, everywhere
• Timeouts
• No data
• Volumetrics
• Unfair load balancing

32. Verbose Logging
• Wrong levels configured
• Diagnostic information
• Expected/Handled errors
• Too much detail
• Hurts health/forensic reporting

33. Not enough logging
• Fatals with no logging
• Unhandled conditions
• Monitoring holes
• Operations staff blind

34. Platform Configuration
• Unfair load-balancing
– Remove older commodity servers
• Competitive service applications
– Re-home critical applications

35. “Timeouts at lower levels in the architecture
MUST be set shorter than the timeouts
configured at higher levels of the
architecture.”

37. Timeouts
• Frontend/Backend timeouts
– Frontends with lower timeouts
– Caches never populated
• Alter backends to return early

38. More timeouts
• Unspecified timeouts
• Wrongly specified timeout units
– ms/sec

40. Poor Application Performance
• Multiple synchronous content requests
• International cachability
• Missing negative caching
– Bypassed shared caches

41. Testing frequency
• Every two weeks
• Every week
• Every other day

43. One week before…
The opening ceremony…
• 1st successful test on Live
– with no errors at all.

45. Performance Overview
• Did find problems
– Weren’t found on stage
• In all architecture layers
• Components believed to be “fine” were not
• Stage is not suitable for this level of testing
• Proposal for any future “high profile” event
• CDNs didn’t really get tested

46. Resilience Overview
• Teams never tested failure scenarios
• Assumed that services didn’t fail
• Inconsistent use of flagpoles
• Reliance on mod_cache stale-on-error

47. Other problems
• Running a “fake” Olympics
– That is invisible to the public
– Did consider publicising a test
• No A/B (bucket) testing capability
• Some tests affected BAU
• No real test of the HLS HDS streaming
• Platform monitoring cycle

48. Other problems
• RCA complicated by shared platform
• Testing stopped by BAU/TX
• High reliance on key staff
– Some tests suffered
• No CDN testing
– At their request
– Places unfair load on infrastructure
• Unable to simulate network congestion

50. Working with external tester
• Workflow testing differed
– User journeys
– Direct linking to hotspots
• Very responsive to altering tests
• Did add extra complexity

51. Did it work
• YES
– Found and fixed issues
– Before they bit us
– On production
– With little impact on BAU

52. Recommendations
• Increase stage capacity
• Intelligent load balancing
• Test NFRs in Development
• Caching, caching and more caching
• Kill load tests quickly
• Improve internal load testing
• Profile frontends under load
• Better post analysis tools

53. Some Statistics

54. Daily Reach (M)

55. Streaming Views (M) Wed 1st Aug

56. Unique Browsers (M)

58. Thanks for listening
• Thanks to flickr users:
– dgjones
• Office Dalek, London, 14-10-06
• http://www.flickr.com/photos/dgjones/284592369
– b3cft
• Bombe rebuild detail
• http://www.flickr.com/photos/b3cft/3797123899
– Karindalziel
• Clouds
• http://www.flickr.com/photos/nirak/644336486
– Enjoy Surveillance
• What are you looking at?
• http://www.flickr.com/photos/enjoy-surveillance/34795807/
– Solo
• 45th Annual Watsonville Fly-in and Air Show
• http://www.flickr.com/photos/donsolo/4959045491/in/photostream/
– SF Brit
• Sunset over Iguazu
• http://www.flickr.com/photos/cnbattson/4333692253/
• Olympics Photos: www.london2012.com
• Other Photos: EpicWin, FailBlog, Haha-Business

59. Special Thanks
to:
– David Holroyd
• Technical Architect BBC Sport (Olympics)
– Matt Clark
• Senior Technical Architect BBC Sport

60. Thanks for listening
• This presentation:
– TBC
• Me:
– Andy “Bob” Brockhurst
– Twitter: b3cft (and pretty much anywhere online)
– www.kingkludge.net