Liran Tal presenting at the HP Office in Cluj Romania - review of how we optimized HP Live Network's web marketplace performance in various layers of the server-side stack to achieve 10x performance improvement.

1. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HPLiveNetwork
WebPerformanceOptimization
Liran Tal
Cluj, September 2014

2. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LiranTal
HPLiveNetwork,TeamLead
• Book Author of Drupal 7 Media
PACKT Publishing
• Author of daloRADIUS, and actively contributing and
supporting the open source community

3. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.3
Goal
• What is performance optimization
• How to approach performance optimization
• Learn from our experience at HPLN
source: http://en.wikipedia.org/wiki/Web_performance_optimization

4. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Whatiswebperformance
optimization?

5. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.5
What is web performance optimization?
“website optimization is the field of knowledge about increasing the speed in
which web pages are downloaded and displayed on the user's web browser”
source: http://en.wikipedia.org/wiki/Web_performance_optimization

6. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Motivation
Why should you care about performance optimization?

7. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.7
Motivation
“According to surveys done by Akamai and Gomez.com, nearly 50% of web users
expect a site to load in 2 seconds or less, and they tend to abandon a site that isn’t
loaded within 3 seconds.”
source: https://blog.kissmetrics.com/loading-time/?wide=1

8. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.8
Motivation
source: https://blog.kissmetrics.com/loading-time/?wide=1

9. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.9
Motivation
• Better User Experience
• Better SEO (2008, site speed was being incorporated into Adwords Quality Score)
• Customers satisfaction
• Happier customers leads to better conversion rates
• Increased visitor retention
• Increased visitor loyalty

10. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.10
Motivation
If not for your users, do it for the environment!
less data
traffic
less server
power
consumption
better
environmental
impact

11. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
DoingPerformanceOptimizationRight

12. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.12
Doing Performance Optimization Right

13. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.13
Doing Performance Optimization Right
Frustrating.
No magic rule.
No clear path.

14. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.14
Doing Performance Optimization Right
Web Performance Optimization is a process, not a one off fix.

15. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.15
Doing Performance Optimization Right
• Performance is a team effort
• Code review is important
• Allows finding performance leaks quickly before they reach production
• Focus on the quick wins which yield best results
• Constant monitoring and measuring

16. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.16
Doing Performance Optimization Right
Monitor
Performance
Improve
Test and
Measure

17. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HPLNPortal
ThePlatform

18. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.18
HPLN Portal
HP Live Network - Portal
https://hpln.hp.com

19. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.19
HPLN Portal

20. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.20
HPLN Portal - Infrastructure

21. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.21
HPLN Portal – Technology Stack
• Drupal
• Open source CMS, and a Framework
• LAMP-based stack
• IaaS-based on HP’s SaaS organization

22. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HPLNPortal
Challenges
Technical challenges

23. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.23
Monitoring HPLN Performance
2013
slower than 92% of tested
websites
16s page
load time

24. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.24
HPLN Portal – Drupal Challenges
• Drupal 6 vs Drupal 7 and upcoming Drupal 8
• Baseline is Drupal Commons
• Module overhead: 398 active modules
• Drupal is database, and memory intensive

25. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.25
HPLN Portal – Drupal Challenges
• Drupal 6 install-base
lines of code:
1.3 Million LOC
0.00
200000.00
400000.00
600000.00
800000.00
1000000.00
1200000.00
1400000.00
1600000.00
1/1/2011
2/1/2011
3/1/2011
4/1/2011
5/1/2011
6/1/2011
7/1/2011
8/1/2011
9/1/2011
10/1/2011
11/1/2011
12/1/2011
1/1/2012
2/1/2012
3/1/2012
4/1/2012
5/1/2012
6/1/2012
7/1/2012
LOC

26. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.26
HPLN Portal – Drupal Challenges
• Drupal 6 install-base
additional contributed
modules
~100k maintainable
LOC by the team

27. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.27
HPLN Portal – Infrastructure Challenges
• PHP 5.3 vs PHP 5.5
• RHEL 5 (5.8) vs RHEL 6
• MySQL 5.0 vs MySQL 5.6

28. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
PerformancePlan
Plan of attack

29. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.29
Performance Plan
• Improve Page Load time?
• Improve Server Response time?
• Handle scalability by improving Page Views?

30. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.30
Performance Plan

31. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
WebServer–PerformanceTuning

32. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.32
Web Server – Performance Tuning
1. Server processes
1. Total Ram = 16Gb (this is a dedicated web server with some extra juice like ApacheSolr)
2. MaxClients = (Total RAM – OS misc. RAM) / Memory by apache Process
3. StartServers
4. MinSpareServers
5. MaxSpareServers

33. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.33
Web Server – Performance Tuning
2. Apache’s loadable modules
1. Total modules enabled: 56
2. After revision, disabled 39

34. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.34
Web Server – Performance Tuning
3. Apache’s .htaccess file
1. When AllowOverride All is set, disk IO is wasted to lookup the .htaccess file
2. Disable the global directive for .htaccess
1. <Directory /> AllowOverride None </Directory>
3. Update expirations for more content types:
1. ExpiresByType image/x-icon
2. ExpiresByType image/jpg
3. ExpiresByType image/jpeg

35. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.35
Web Server – Performance Tuning
4. Apache mod_deflate to apply compression
1. AddOutputFilterByType DEFLATE text/plain
2. AddOutputFilterByType DEFLATE text/html
3. AddOutputFilterByType DEFLATE text/xml
4. AddOutputFilterByType DEFLATE application/xml
5. AddOutputFilterByType DEFLATE application/xhtml+xml
6. AddOutputFilterByType DEFLATE application/rss+xml
7. AddOutputFilterByType DEFLATE application/javascript

36. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.36
Web Server – Performance Tuning
Result
1. HPLN’s Products Page: 116kb size 1.07s time vs 18kb/116kb size 0.425 time
2. HPLN’s Community Page: 131kb size 0.972 time vs 21kb/127kb size 0.459 time
0
0.2
0.4
0.6
0.8
1
1.2
Community
Page
Products page
Prod1
Dev2
0
20
40
60
80
100
120
140
Community
Page
Products page
Prod1
Dev2

37. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Database(MySQL)–
PerformanceTuning

38. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.38
Database (MySQL) – Performance Tuning
1. Database table engines
1. Drupal 6 defaults to MyISAM vs InnoDB, convert them
ALTER TABLE sessions ENGINE=InnoDB;

39. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.39
Database (MySQL) – Performance Tuning
2. Missing table indexes
1. comment_upload module has no index on its fid field
mysql> explain SELECT f.*, cu.nid FROM files f INNER JOIN comment_upload cu ON f.fid = cu.fid
WHERE filepath = '/opt/files/group_logos/cm
'> s_0.jpg';
+----+-------------+-------+--------+---------------+---------+---------+----------------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+-------+--------+---------------+---------+---------+----------------+------+-------------+
| 1 | SIMPLE | cu | ALL | PRIMARY | NULL | NULL | NULL | 1129 | |
| 1 | SIMPLE | f | eq_ref | PRIMARY | PRIMARY | 4 | hpln_db.cu.fid | 1 | Using where |
+----+-------------+-------+--------+---------------+---------+---------+----------------+------+-------------+
2 rows in set (0.00 sec)

40. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.40
Database (MySQL) – Performance Tuning
3. Tune up MySQL
− max_allowed_packet=16M
− key_buffer_size=512M
− myisam_sort_buffer_size=64M
− table_cache=2048
− query_cache_size=512M
− thread_concurrency=2
− join_buffer_size=3M
− sort_buffer_size=3M
− read_buffer_size=1M
− read_rnd_buffer_size=10M
− skip-name-resolve
− innodb_file_per_table
− innodb_flush_method

41. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.41
Web Server – Performance Tuning
Result
1. HPLN’s Products page

42. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.42
Web Server – Performance Tuning
Result
1. HPLN’s Products page
259
238 236
385
235 236
1 2 3
Number of Queries
Series1 Series2
705.92 704.56
307.03
346.77
98.03 125.69
1 2 3
Time of Queries (ms)
Series1 Series2

43. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.43
Web Server – Performance Tuning
Result
1. HPLN’s Community
home page

44. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.44
Web Server – Performance Tuning
Result
1. HPLN’s Community Announcements page
308
622
293290 293 289
1 2 3
Number of Queries
Series1 Series2
660.13
1782.21
2120.54
364.26
186.25 168.89
1 2 3
Time of Queries (ms)
Series1 Series2

45. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.45
Web Server – Performance Tuning
Result
1. HPLN’s Community
Announcements page

46. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.46
Web Server – Performance Tuning
Result
1. HPLN’s Community Announcements page
308
622
293290 293 289
1 2 3
Number of Queries
Series1 Series2
660.13
1782.21
2120.54
364.26
186.25 168.89
1 2 3
Time of Queries (ms)
Series1 Series2

47. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
DBProfiling
Like code, but for queries!

48. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.48
DB Profiling
Percona Toolkit
• Query profiling
• Backups
• Index utilization

49. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CaseStudy#1
And so the journey begins…

50. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.50
Percona Toolkit
DB Infrastructure
• MySQL Slow Query Log

51. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.51
Percona Toolkit
DB Infrastructure
• pt-query-digest
• MySQL Slow query log ~1Gb – promising!

52. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.52
Percona Toolkit
DB Infrastructure
• pt-query-digest

53. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.53
Percona Toolkit
DB Infrastructure
• pt-query-digest
Average
query time

54. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.54
Percona Toolkit
DB Infrastructure
• pt-query-digest Variance to Mean Ratio
0 < VMR < 1 under-dispersed
VMR > 1 over-dispersed

55. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.55
Percona Toolkit
DB Infrastructure
• pt-query-digest

56. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.56
Percona Toolkit
DB Infrastructure
• MySQL explain

57. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.57
Percona Toolkit
DB Infrastructure
• Query Profile

58. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.58
Percona Toolkit
DB Infrastructure
• Query Profile

59. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.59
Percona Toolkit
DB Infrastructure
• Query Profile

60. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.60
Percona Toolkit
DB Infrastructure
• Query Profile

61. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.61
Percona Toolkit
DB Infrastructure
• Query Profile
540
560
580
600
620
640
660
680
700
720
Measure 1 Measure 2 Measure 3
Before

62. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.62
Percona Toolkit
DB Infrastructure
• Optimize query and profile it

63. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.63
Percona Toolkit
DB Infrastructure
• Query Profile
0
100
200
300
400
500
600
700
800
Measure 1 Measure 2 Measure 3
Before
After

64. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.64
DB Infrastructure

65. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CaseStudy#2

66. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.66
Percona Toolkit
DB Infrastructure
• pt-query-digest

67. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.67
Percona Toolkit
DB Infrastructure
• pt-query-digest

68. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.68
Percona Toolkit
DB Infrastructure
• pt-query-digest

69. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.69
Percona Toolkit
DB Infrastructure
• pt-query-digest

70. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CaseStudy#3
Unusued code – unnecessary queries and server load

71. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.71
Code Review
DB Infrastructure
• Code Review and Database Query Analysis

72. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.72
Code Review
DB Infrastructure

73. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.73
Code Review
DB Infrastructure
• The query is pretty heavy - takes about 200ms-300ms
• The query is useless due to refactoring in the UI – data is irrelevant and is of no use
0
100
200
300
400
500
600
Measure 1 Measure 2 Measure 3
Before
After

74. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.74
Code Review
DB Infrastructure

75. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.75
Code Review
DB Infrastructure
• Forums query returns forum posts and comments count
• Data isn’t important to show “real-time” information
0
100
200
300
400
500
600
700
800
Measure 1
Before (no cache)
After (cached)

76. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
CaseStudy#4
Know your DB Engines

77. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.77
Code Review
DB Infrastructure
• Access callback counts the communities which the user has access to
• Executed on every page load to determine menu items
• Joins un-necessary og_users_roles table
• Query uses SELECT COUNT() which for InnoDB needs to do a full table scan (compared to MyISAM which
saves this information as part of the index structure)

78. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.78
Code Review
DB Infrastructure

79. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.79
Code Review
DB Infrastructure
• Sometimes it also takes 9 seconds

80. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.80
Code Review
DB Infrastructure
• Re-structured query to not join irrelevant tables
• Re-structured query to search for an actual community entry in the DB, which utilizes indexes and return at
least one row

81. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.81
Code Review
DB Infrastructure
• Refactored 200ms-300ms into 0.02ms
0
50
100
150
200
250
300
Measure 1
Before
After

82. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Application(DrupalandPHP)–
PerformanceTuning

83. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.83
Application (Drupal and PHP) – Performance Tuning
1. Drupal’s installed module count
1. A plethora of modules installed due to Drupal Commons distribution profile
2. Removed un-used modules
3. Removed admin-only modules which aren’t
necessary in production
0
50
100
150
200
250
300
350
400
450
Drupal
Enabled
Modules
Before
After

84. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.84
Application (Drupal and PHP) – Performance Tuning
2. Apply and backport performance patches
1. http://groups.drupal.org/node/187209:
1. module_implements()
2. node_access()
3. Bootrstap’s bootstrap_invoke_all()
4. cache_url()
5. Forms’ element_children()
6. View’s menu_links_translate()
7. menu_get_item() (Drupal 7 caching backport: https://drupal.org/node/1167626)
8. Enable MySQL compression flag (https://drupal.org/node/11891)

85. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.85
Application (Drupal and PHP) – Performance Tuning
3. Log to syslog or a better logging mechanism
1. Offload the database
2. Offload the watchdog module

86. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Results

87. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.87
Monitoring HPLN Performance
2014
Faster than 34% tested
websites
4.6s page
load time

88. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.88
Monitoring HPLN Performance
0
500
1000
1500
2000
2500
Jan
Feb
Mar
Apr
May
Jun
Jul
Sep
Dec
Jan
Mar
Apr
May
Aug
HTTP Ping
1627ms @2013 562ms @2014

89. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.89
Monitoring HPLN Performance
2,479ms
594ms

90. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.90
Monitoring HPLN Performance
1,443ms
955ms