1. Performance Tuning on Go 2.0http://www.thoughtworks-studios.com/go/ Yogi Kulkarni

2. GoContinuous integration Release management

4. Pipeline Stage-2 Unit Test Stage-3 Acceptance Test Stage-1 Compile Run Unit Test Suite - 1 Accetance Test Suite - 1 Compile Job Run Unit Test Suite - 2 Acceptance Test Suite – 2 Run Unit Test Suite - 3

5. Pipeline Dashboard

7. Pipeline Dependencies

8. The Problem

9. Slow page loads

10. Jobs taking long to get assigned to agents

11. Jobs getting rescheduled

12. Artifact uploads timing out

13. Server VM freezing

14. Lots of concurrent activity

15. What was happening under the hood?Profiler revealed the culprits

16. Thread blocks

17. Threads blocked on the database connection pool

18. So we introduced asynchronicity

19. Big mistake!

20. Find the root cause

24. ObjectiveNo request should take > 100 msMost should be under 50 ms

25. Setup & Tools

26. Conservative hardware

27. “Performance Server”Dell 620 laptop2 Cores, 2GB RAM

28. Large Dataset5GB merged databaseLarge config file with 100 pipelinesReal-world data

29. Extreme load70 agentsEach Job generating lots of console outputEach Job uploading 10 MB artifactsApache Bench / Httperf

30. Automation for fast feedbackCapistrano scriptFeedback cycle down from hours to 10 min

31. Caching

32. Single application-wide cacheDomain objectsDTOsRails view fragments

34. EHCacheIn-memory onlyLeast-Recently-Used eviction policy

35. Transaction rollbacks == Incorrect cached dataIgnore cache puts in transactionsInvalidate caches on transaction commit only

36. Spring to the rescue

37. Fast deep-cloning of cached objectsJava Deep Cloning Library: http://robust-it.co.uk/clone/index.php

38. Locks

39. Be suspicious of synchronized methods

40. Lock on interned stringsFine granularityProfiler supportAlso used as cache keys

42. Use ReadWriteLocks for highly contended code

43. Use “volatile” fields for lockless reads

44. Slow

45. Not synchronized

46. Acquire locks before starting transactionsTo enforce aggregate invariants

47. Thread 1 Stage (Building) Job – 1 (Completed) Stage (Building) Stage (Building) Job – 2 (Building) Job – 1 (Completed) Job – 1 (Building) Thread 2 Job – 2 (Cmpleted) Job – 2 (Building) Stage (Building) Job – 1 (Building) Job – 2 (Completed)

49. Query Tuning

50. Use a large databaseUse the query analyzer to find missing indexes

51. Views are great… sometimes10x-20x faster for “TOP n” queriesDepends on database implementation

52. Query selectivity is important 200 ms Pipelines (20,000 rows) Stages (70,000 rows) BuildStateTransitions (4,000,000 rows)

53. Introduce an n+1 query 2 ms

54. JRuby on Rails Hacks

55. Single JRuby runtime + Rails multi-threaded mode Much lower memory footprintBut completely uncharted territory

56. Severe thread blocks in JRubyReported to JRuby communityFixed immediately in 1.5.0 & 1.5.1

57. Rails nested partials Mysterious lock contention10 concurrent requests take 90 secs to completeStill a mystery!

58. Desperate situations, desperate measuresStatically inline partials at build time!

59. Routes subsystem is slowCache url_for

60. LoggingReplace Rails BufferedLogger with Java Log4jLoggerTurn off Rails logging in production

61. Fragment cachingGenerate a cache-key that incorprates all fields that are needed to uniquely identify that object’s current stateLRU cache will evict it on state change

62. “pipelineDashboardFragment|dev|2[compile,1,passed][test,1,building]” Pipeline-name Stage detail Stage detail Key-identifier

63. “pipelineDashboardFragment|dev|2[compile,1,passed][test,1,passed]” Pipeline-name Stage detail Stage detail Key-identifier “pipelineDashboardFragment|dev|1[compile,1,passed][test,1,building]”

64. Lessons LearnedFind the root causeUse a profilerMeasure before and after each fix

65. How we got from that to thisINSERT IMAGE Threads after performance tuning

66. http://www.thoughtworks-studios.com/go/