‘Tis the Season
A Holiday 2014 Ecommerce Checklist
Preparing for best case retail success
Holiday
2013 was
BIG
2014 will be
BIGGER
mobile
matters
your site
needs to
PERFORM
40 % abandon a
website that takes
more than 3
seconds to load
A 1 second delay
in page response
can result in a 7%
reducti...
If an ecommerce site is making
$10,000 a day, a 1 second
delay could cost you
$250,000 in lost sales every
year
Start Now
Prepare for the holiday rush
‘Tis the Season
A Holiday 2014 Ecommerce Checklist
Preparing for best case retail success
Elizabeth Scott, Tenzing
Director of Technical Services
@BethxScott
Brad Johnson, SOASTA
VP Product Marketing
@bradjohnson...
Infrastructure and Application
Monitoring
Load Testing
Marketing
Agenda
Poll Question
Infrastructure and
Application
Assess Your Infrastructure
Current infrastructure
performance and capacity
baseline:
 Server
Hardware
 Storage
 Network...
Assess Your Infrastructure
 Know how many users can be on
your site: Do a load test
 Understand your vendor’s capacity
p...
Ensure your
environment is up to date
on latest patch levels
Develop an emergency
patch and security plan
Secure Your En...
Ensure ports on your
servers are not open
unnecessarily
Limit access
Secure Your Environment
Develop and test your
Denial of Service
response strategy
Run a vulnerability scan
Secure Your Environment
Tune queries,
search results and
session limits
Leverage a CDN
Tune Your Application
Find free capacity:
 Adjust your caching
 Adjust ‘type ahead’ search
 Remove/adjust view all options
 Adjust other fea...
Monitoring
Application Performance
End User Experience
Traffic Levels
Measure, Analyze, Act
•Upstream Connections
Payment Gateway
Shipping and Fulfillment
Measure, Analyze, Act
Plan for the
Inevitable
Assess your risk …
WHAT WILL GO
DOWN AND WHY?
Plan for the Inevitable
•Create a plan
Website overload plan
Incident response plan
Security/DoS response plan
Plan for the Inevitable
COMMUNICATE
Plan for the Inevitable
Modern Load
Testing
True Story
• Division of $6B Company
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
True Story
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Mond...
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Mond...
• Division of $6B Company
• Cyber Monday & last Free Shipping Day
• “We wouldn’t disrupt the plan”
• Crashed on Cyber Mond...
Is there anything
missing from the
plan?
Prerequisite:
Know Your
Real Users
Modern Load Testing
Stop guessing
about what to test
 Most accurate models of real user
behavior
 Meaningful scale & appropriate
concurrency
 Control to start, stop & modif...
 Failover Scenarios
 Success Disasters
 End to End
Test what you don’t think needs to be
tested
Modern Load Testing
Measure,
Test and Act
in Real Time
Modern Load Testing
Don’t Wait until it’s too late
Compress processes for more test iterations
Traditional
Approach
TIME
Testdevelopment Infra...
Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Testdevelopment Inf...
Don’t Wait until it’s too late
Compress processes for more test iterations
/
Traditional
Approach
TIME
Testdevelopment Inf...
Use real user data to
iterate dev & test plans
Run iterative load
tests via Jenkins
Daily (or more frequent) updates
keep ...
s
Memory leaks
Test Scale
Stage/Team
Dev
& Test
Release
& Deploy
Network
& Ops
CDN file placement
Load Balancer configurat...
• Tests for flash sales with
unknown spikes
• mPulse analyzes real user
performance
• Continuous tests of mobile
and web t...
NO
ISSUES
“SOASTA was very easy to work with and could
accommodate our requirements in a four-week
timeframe.” Gavin Cliff...
Critical Factors for
Performance Testing Retail Sites
 Test for spikes
 Test what you think
doesn’t need to be
tested
 ...
Marketing
Communicate your campaign plan
internally and with stakeholders
Establish a campaign change
process
Approval
Communica...
Establish a change
freeze on application
and infrastructure
Limit publishing and
catalogue changes
Campaign Management
Campaign Best Practices
Segment your list into
smaller groups
Trickle your email
Manage your media content
Email Best Practices
Don’t get too fancy
Develop test cases
for all of your discount
codes
Promotions and Discounts
Avoid using discount
structures that require a
database search
Don’t use new discount
structures
Promotions and Discounts
TEST, TEST, TEST!
Promotions and Discounts
Are you interested
in a FREE Retail
Readiness
Performance Test
Contact SOASTA:
www.soasta.com
info@soasta.com
866.344.8766
Follow us:
Twitter.com/cloudtest
Facebook.com/cloudtest
Retail...
www.SOASTA.com
more than managed hosting. commerce anywhere.
www.tenzing.com
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness
Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness
Upcoming SlideShare
Loading in …5
×

Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness

1,488 views

Published on

‘Tis the Season – Holiday 2014 eCommerce Quality Checklist

Past Webinar
Archived (originally presented June 26th, 2014)


This year, your holiday traffic will increase 15% or more, and 50% of the users will be mobile. Recent research shows 71% of your revenue comes from multi-channel users, so if you haven’t started planning, you’re already behind. Leading retailers are preparing for Holiday “14 and testing their production sites for multi-channel access to 115% capacity, or beyond! If you’re not one of them, your plans are incomplete.

Cover your risks. Join Tenzing and SOASTA experts as they discuss the must-do checklist for peak performance.

In this webinar you’ll learn:

Align your Marketing and Quality plans
Cover the multichannel user experience
Test early in the lab and fully in production
Optimize end-to-end site speed and performance
When to freeze for the winter
Don’t miss this opportunity to “shop early” and see how the leading retailers are already beating the odds with cloud testing.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,488
On SlideShare
0
From Embeds
0
Number of Embeds
152
Actions
Shares
0
Downloads
15
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Tis The Season: Load Testing Tips and Checklist for Retail Seasonal Readiness

  1. 1. ‘Tis the Season A Holiday 2014 Ecommerce Checklist Preparing for best case retail success
  2. 2. Holiday 2013 was BIG
  3. 3. 2014 will be BIGGER
  4. 4. mobile matters
  5. 5. your site needs to PERFORM
  6. 6. 40 % abandon a website that takes more than 3 seconds to load A 1 second delay in page response can result in a 7% reduction in conversions 79% of shoppers who are dissatisfied with website performance are less likely to buy from the same site again.
  7. 7. If an ecommerce site is making $10,000 a day, a 1 second delay could cost you $250,000 in lost sales every year
  8. 8. Start Now Prepare for the holiday rush
  9. 9. ‘Tis the Season A Holiday 2014 Ecommerce Checklist Preparing for best case retail success
  10. 10. Elizabeth Scott, Tenzing Director of Technical Services @BethxScott Brad Johnson, SOASTA VP Product Marketing @bradjohnsonsv Your Hosts
  11. 11. Infrastructure and Application Monitoring Load Testing Marketing Agenda
  12. 12. Poll Question
  13. 13. Infrastructure and Application
  14. 14. Assess Your Infrastructure Current infrastructure performance and capacity baseline:  Server Hardware  Storage  Network  Database  Application
  15. 15. Assess Your Infrastructure  Know how many users can be on your site: Do a load test  Understand your vendor’s capacity planning process  Develop an emergency capacity plan
  16. 16. Ensure your environment is up to date on latest patch levels Develop an emergency patch and security plan Secure Your Environment
  17. 17. Ensure ports on your servers are not open unnecessarily Limit access Secure Your Environment
  18. 18. Develop and test your Denial of Service response strategy Run a vulnerability scan Secure Your Environment
  19. 19. Tune queries, search results and session limits Leverage a CDN Tune Your Application
  20. 20. Find free capacity:  Adjust your caching  Adjust ‘type ahead’ search  Remove/adjust view all options  Adjust other features which are resource hogs Tune Your Application
  21. 21. Monitoring
  22. 22. Application Performance End User Experience Traffic Levels Measure, Analyze, Act
  23. 23. •Upstream Connections Payment Gateway Shipping and Fulfillment Measure, Analyze, Act
  24. 24. Plan for the Inevitable
  25. 25. Assess your risk … WHAT WILL GO DOWN AND WHY? Plan for the Inevitable
  26. 26. •Create a plan Website overload plan Incident response plan Security/DoS response plan Plan for the Inevitable
  27. 27. COMMUNICATE Plan for the Inevitable
  28. 28. Modern Load Testing
  29. 29. True Story
  30. 30. • Division of $6B Company True Story
  31. 31. • Division of $6B Company • Cyber Monday & last Free Shipping Day True Story
  32. 32. • Division of $6B Company • Cyber Monday & last Free Shipping Day • “We wouldn’t disrupt the plan” True Story
  33. 33. • Division of $6B Company • Cyber Monday & last Free Shipping Day • “We wouldn’t disrupt the plan” • Crashed on Cyber Monday True Story
  34. 34. • Division of $6B Company • Cyber Monday & last Free Shipping Day • “We wouldn’t disrupt the plan” • Crashed on Cyber Monday • No answers on Thursday True Story
  35. 35. • Division of $6B Company • Cyber Monday & last Free Shipping Day • “We wouldn’t disrupt the plan” • Crashed on Cyber Monday • No answers on Thursday • Production testing on Friday reproduced the issue True Story
  36. 36. Is there anything missing from the plan?
  37. 37. Prerequisite: Know Your Real Users Modern Load Testing Stop guessing about what to test
  38. 38.  Most accurate models of real user behavior  Meaningful scale & appropriate concurrency  Control to start, stop & modify tests without delays  Test on your schedule Modern Load Testing Test for spikes – using the cloud
  39. 39.  Failover Scenarios  Success Disasters  End to End Test what you don’t think needs to be tested Modern Load Testing
  40. 40. Measure, Test and Act in Real Time Modern Load Testing
  41. 41. Don’t Wait until it’s too late Compress processes for more test iterations Traditional Approach TIME Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze Modern Load Testing
  42. 42. Don’t Wait until it’s too late Compress processes for more test iterations / Traditional Approach TIME Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze CloudTest Modern Load Testing
  43. 43. Don’t Wait until it’s too late Compress processes for more test iterations / Traditional Approach TIME Testdevelopment Infra.setup Fix Repeat?RetestTest Analyze / / / / Extra time!CloudTest Modern Load Testing
  44. 44. Use real user data to iterate dev & test plans Run iterative load tests via Jenkins Daily (or more frequent) updates keep teams quality focused Embedded failure details & bottleneck data isolate issues Manage to a performance baseline Mobile? Run functional automation on real devices for functional & performance validation Spin up suitable, consistent test environments in the cloud or the lab Jenkins runs any unit tests as usual Test Continuously – including performance! 5. Don’t Wait
  45. 45. s Memory leaks Test Scale Stage/Team Dev & Test Release & Deploy Network & Ops CDN file placement Load Balancer configuration Network bandwidth Network configuration DNS routing Inadequate server resources Default configuration settings Unbalanced web serversAuto-scaling failures Latency between systems Slow third-party plug-ins Garbage collection Database thread counts Inefficient database queriesSlow pages Conflict with other apps Test Lab Staging Production (100% +++) Search technology limits Method-level tuning Max sockets exceeded Firewall max capacity Global latency variance Security bottlenecks Continuous tuning for full-scale…and well beyond Start early, end in the real environment 6. Test in production and at Production scale
  46. 46. • Tests for flash sales with unknown spikes • mPulse analyzes real user performance • Continuous tests of mobile and web traffic “SOASTA was very easy to work with and could accommodate our requirements in a four-week timeframe.” Gavin Cliffe, OZSale CTO. OZSale
  47. 47. NO ISSUES “SOASTA was very easy to work with and could accommodate our requirements in a four-week timeframe.” Gavin Cliffe, OZSale CTO. OZSale 160% more visitors $5M AUD in one sale 50% mobile
  48. 48. Critical Factors for Performance Testing Retail Sites  Test for spikes  Test what you think doesn’t need to be tested  Test expecting failure  Analyze & act in real time  Don’t forget about the user experience  Monitor everything  Don’t wait until just before the holidays  Test in production
  49. 49. Marketing
  50. 50. Communicate your campaign plan internally and with stakeholders Establish a campaign change process Approval Communication Campaign Management
  51. 51. Establish a change freeze on application and infrastructure Limit publishing and catalogue changes Campaign Management
  52. 52. Campaign Best Practices
  53. 53. Segment your list into smaller groups Trickle your email Manage your media content Email Best Practices
  54. 54. Don’t get too fancy Develop test cases for all of your discount codes Promotions and Discounts
  55. 55. Avoid using discount structures that require a database search Don’t use new discount structures Promotions and Discounts
  56. 56. TEST, TEST, TEST! Promotions and Discounts
  57. 57. Are you interested in a FREE Retail Readiness Performance Test
  58. 58. Contact SOASTA: www.soasta.com info@soasta.com 866.344.8766 Follow us: Twitter.com/cloudtest Facebook.com/cloudtest Retail Readiness Page: soasta.com/retail-readiness/ Case studies: soasta.com/customers/ecommerce/ Webinars future and past: soasta.com/knowledge-center/webinars/ CloudTest Lite is FREE (forever)! soasta.com/FREE Resources from SOASTA Thank you
  59. 59. www.SOASTA.com more than managed hosting. commerce anywhere. www.tenzing.com

×