The document outlines troubleshooting steps for resolving eCommerce performance issues reported by customers. It emphasizes the importance of checking CDN configurations, analyzing the SEO page generator, and optimizing content to improve site performance. Various tools and methods are provided for debugging and enhancing website efficiency, including command-line tools and third-party services.

1. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.1
Mark Sweeting & Diego Cardozo
Troubleshooting Ecommerce Performance
April 24, 2018
COBRAND NAME

2. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.2
Safe Harbor Statement
The following is intended to outline our general product direction. It is
intended for information purposes only, and may not be incorporated into
any contract. It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making purchasing decisions.
The development, release, and timing of any features or functionality
described for Oracle’s products remains at the sole discretion of Oracle.

3. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.3
THE SCENARIO
A customer has contacted us stating that they have a
performance problem with their website.
They provide no details about the nature of the problem.

4. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.4
WHERE TO START – ANY METRICS?

5. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.5
STEP 1:
CONTENT
DELIVERY
NETWORK

6. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.6
WHAT IS A CONTENT DELIVERY NETWORK?

7. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.7
WHAT IS A CONTENT DELIVERY NETWORK?

8. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.8
TOOL:
DIG
> dig IN A www.example.com

9. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.9
IS THE CDN CONFIGURED AND WORKING?
Mac/Linux users, use dig command to check:
> dig IN A www.example.com
Windows users use nslookup:
> nslookup -class=IN -querytype=A www.example.com
• Note: use full hostname without protocol (http or https)

10. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.10
IS THE CDN CONFIGURED AND WORKING?
GOOD: domain resolves to Akamai A-records
www.example.com. 300 IN CNAME www.example.com.hosting.netsuite.com.
www.example…netsuite.com. 300 IN CNAME website-cdn.na1.netsuite.com.
website-cdn.na1.netsuite.com. 54 IN CNAME
website-cdn.na1.netsuite.com.mdc.edgesuite.net.
website-cdn.na1.netsuite.com.mdc.edgesuite.net. 415 IN CNAME a1280.q.akamai.net.
a1280.q.akamai.net. 14 IN A 213.123.255.6
a1280.q.akamai.net. 14 IN A 213.123.255.9

11. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.11
GOOD: domain resolves to Cloudfront A-records
www.example.com. 9 IN CNAME www.example.com.hosting.netsuite.com.
www.example......netsuite.com. 299 IN CNAME abcdefghijklmn.cloudfront.net.
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.242
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.3
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.96
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.102
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.166
abcdefghijklmn.cloudfront.net. 59 IN A 52.85.93.184
IS THE CDN CONFIGURED AND WORKING?

12. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.12
IS THE CDN CONFIGURED AND WORKING?
BAD: CNAME OK, but CDN tickbox not checked; resolves to Netsuite datacenter.
www.example.com. 600 IN CNAME www.example.com.hosting.netsuite.com.
www.example...ting.netsuite.com. 300 IN CNAME 9999999-origin.shop.netsuite.com.
9999999-origin.shop.netsuite.com. 300 IN CNAME shop.source.netsuite.com.
shop.source.netsuite.com. 66 IN CNAME shopping.netsuite.com.
shopping.netsuite.com. 286 IN A 167.216.129.13

13. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.13
IS THE CDN CONFIGURED AND WORKING?
BAD: CNAME not created; A record points straight to Netsuite datacenter.
www.example.com. 600 IN A 167.216.129.13

14. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.14
IS THE CDN CONFIGURED AND WORKING?
Use Akamai Debug Headers Chrome extension.

15. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.15
STEP 2:
SEO PAGE GENERATOR

16. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.16
WHAT IS THE SEO PAGE GENERATOR

17. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.17
TTFB: TIME TO FIRST BYTE
• Time it takes for first byte of response
content to reach the client browser
• Part of the following sequence:
−DNS LOOKUP
−TCP CONNECTION
−SSL NEGOTIATION
−REQUEST SEND
−TIME TO FIRST BYTE
−CONTENT DOWNLOAD (the HTML)
• Usually the time it takes for dynamic
server-side activities to happen

18. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.18
TIME TO FIRST BYTE

19. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.19
CHECK THE SEO
PAGE GENERATOR

20. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.20
TOOL:
PAGE GENERATOR
DEBUGGER
?seodebug=T

21. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.21
HOW TO DEBUG THE PAGE GENERATOR
1. Turn on the debugger:
− seodebug=T
2. Bust the cache:
− preview=<random string>
3. Force Pre-render (for additional useful output)
− seoprerender=T
Example:
http://www.example.com/some-page?seodebug=T&preview=12345
=> Then view the page source…

22. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.22
HOW TO DEBUG THE PAGE GENERATOR
• Check the main DIV isn’t empty:
<div id=“main”></div>
• Search for errors in the output
• Examine the timings of sub-requests (prerender only)
• Check the status codes of sub-requests (prerender only)
• Check sub-requests don’t include any unnecessary content (third parties!)

23. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.23
DEBUGGING: CHECK MAIN DIV IS POPULATED
😞
😎

24. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.24
DEBUGGING: SEARCH FOR ERRORS
[02:15:04.188] [ 50961 kb ] [ +34 ms ] [ -574 kb ] Envjs.XMLHttpRequest.Jav8 DEBUG: [Fri, 23 Feb
2018 10:15:04 GMT:187] contentEncoding utf-8
[02:15:04.190] [ 51035 kb ] [ +2 ms ] [ +74 kb ] Envjs.Jav8 ERROR: [Fri, 23 Feb 2018 10:15:04
GMT:190] printStackTrace error {anonymous}("java.io.IOException: Server returned HTTP response code: 400
for URL: https://www.example.com/api/items?country=US&currency=USD&custitem_ef_badges=Best
Seller&fieldset=details&language=en&limit=12&pricelevel=5"),printStackTrace(#object),{anonymous}("java.i
o.IOException: Server returned HTTP response code: 400 for URL:
https://www.example.com/api/items?country=US&currency=USD&custitem_ef_badges=Best
Seller&fieldset=details&language=en&limit=12&pricelevel=5"),{anonymou
[02:15:04.191] [ 51043 kb ] [ +1 ms ] [ +8 kb ] Envjs.XMLHttpRequest.Jav8 DEBUG: [Fri, 23 Feb
2018 10:15:04 GMT:191] Envjs.connection: finished

25. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.25
DEBUGGING: SLOW SUB-REQUESTS
[07:23:34.334] [ +0 ms ] Sub request total: 4.453000s
[07:23:34.335] [ +1 ms ] Details of 16 sub requests:
GET http://www.example.com/cms/2/assets/js/postframe.js [status 200]
Requested at 2018-02-19T15:23:29.414Z and responded by 2018-02-19T15:23:29.497Z (which took 83ms)
GET http://www.example.com/cms/2/cms.js [status 200]
Requested at 2018-02-19T15:23:29.414Z and responded by 2018-02-19T15:23:29.618Z (which took 204ms)
GET http://www.example.com/c.1234567/ShopFlow/sc.environment.ssp?v=1&lang=en_US&cur=AUD [status 302]
Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.614Z (which took 116ms)
GET http://www.example.com/c.1234567/ShopFlow/js/libs/Libraries-0145005a1354.js [status 200]
Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.617Z (which took 119ms)
GET http://www.example.com/c.1234567/ShopFlow/templates/Templates-01405a477c8c.js [status 200]
Requested at 2018-02-19T15:23:29.498Z and responded by 2018-02-19T15:23:29.615Z (which took 117ms)
GET http://www.example.com/c.1234567/ShopFlow/js/Application-01450087b360.js [status 200]
Requested at 2018-02-19T15:23:29.499Z and responded by 2018-02-19T15:23:29.616Z (which took 117ms)
GET http://www.example.com/c.1234567/CustomWS/common-shopping-customizations.js?v=80 [status 200]
Requested at 2018-02-19T15:23:29.499Z and responded by 2018-02-19T15:23:29.617Z (which took 118ms)
GET http://www.example.com/c.1234567/ShopFlow/languages/en_US.js?n=18 [status 200]
Requested at 2018-02-19T15:23:29.500Z and responded by 2018-02-19T15:23:29.796Z (which took 296ms)
GET http://www.example.com/c.1234567/ShopFlow/sc.environment.ssp?cur=AUD&lang=en_US&v=1&whence= [status 200]
Requested at 2018-02-19T15:23:29.614Z and responded by 2018-02-19T15:23:32.624Z (which took 3010ms)
GET http://www.example.com/api/cms/session/domain [status 200]
Requested at 2018-02-19T15:23:32.838Z and responded by 2018-02-19T15:23:32.947Z (which took 109ms)
GET http://www.example.com/c.1234567/ShopFlow/services/live-order.ss?internalid=cart&t=1519053812858 [status 200]
Requested at 2018-02-19T15:23:32.858Z and responded by 2018-02-19T15:23:33.147Z (which took 289ms)
GET http://www.example.com/c.1234567/cds/services/page.ss?internalid=25&ttl=500&cache=MEDIUM [status 200]
Requested at 2018-02-19T15:23:32.882Z and responded by 2018-02-19T15:23:33.367Z (which took 485ms)
GET http://www.example.com/api/cms/versions?site_id=16&c.1234567 [status 200]
Requested at 2018-02-19T15:23:32.948Z and responded by 2018-02-19T15:23:33.047Z (which took 99ms)
GET http://www.example.com/api/cms/pages/contents?version_id=2815&site_id=16&c.1234567&path=%2F&page_type= [status 200]
Requested at 2018-02-19T15:23:33.049Z and responded by 2018-02-19T15:23:33.247Z (which took 198ms)
GET http://www.example.com/c.1234567/CustomWS/BannerSlider/shopping/services/getSlides.ss [status 200]
Requested at 2018-02-19T15:23:33.394Z and responded by 2018-02-19T15:23:33.650Z (which took 256ms)
GET http://www.example.com/api/cms/pages/contents?version_id=2815&site_id=16&c.1234567&path=%2F&page_type=home-page [status 200]
Requested at 2018-02-19T15:23:33.399Z and responded by 2018-02-19T15:23:33.549Z (which took 150ms)

26. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.26
DEBUGGING: FAILED SUB-REQUESTS
[07:49:18.431] [ +0 ms ] Sub request total: 0.414000s
[07:49:18.431] [ +0 ms ] Details of 4 sub requests:
GET https://www.example.com/c.1234567/sca-dev-elbrus/shopping.environment.ssp?lang=en_US&cur=null [status 502]
Requested at 2018-02-19T15:49:17.415Z and responded by 2018-02-19T15:49:17.502Z (which took 87ms)
GET https://www.example.com/c.1234567/sca-dev-elbrus/languages/shopping_en_US.js?t=1516304321167 [status 502]
Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.613Z (which took 197ms)
GET https://www.example.com/c.1234567/sca-dev-elbrus/javascript/shopping.js?t=1516304321167 [status 502]
Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.615Z (which took 199ms)
GET https://www.example.com/cms/2/cms.js [status 502]
Requested at 2018-02-19T15:49:17.416Z and responded by 2018-02-19T15:49:17.615Z (which took 199ms)

27. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.27
STEP 3:
OPTIMIZING CONTENT

28. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.28
TOOL:
WEBPAGE TEST
http://www.webpagetest.org/

29. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.29
USING WEB PAGETEST

30. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.30
WEBPAGE TEST WATERFALL – 161 REQUESTS

31. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.31
WEBPAGE TEST – BLOCKING RESPONSES

32. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.32
WEBPAGE TEST – CONTENT BREAKDOWN

33. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.33
IMAGE SIZES
• Get the image sizes rights
• Use the image resizing service

34. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.34
IMAGE SIZES
• Get the image sizes rights
• Use the image resizing service

35. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.35
TOOL:
IMAGEOPTIM
https://imageoptim.com/

36. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.36
IMAGEOPTIM
https://imageoptim.com/mac

37. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.37
STEP 4:
PAGE RENDERING

38. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.38
RENDERING TIMELINE – WEB PAGETEST

39. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.39
PAGE RENDERING – THE USUAL SUSPECTS
• Slow blocking responses (i.e. blocking parallel downloads)
• Fonts – can seriously delay page rendering
• Third parties – insert blocking JavaScript and also delay page rendering
− WebPage Test can give you an overview of the percentage of
requests that correspond to third parties (domain breakdown)
− You can test the impact of removing third parties by right clicking
their requests on Chrome’s Dev Tools and selecting “Block
request domain”

40. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.40
SLOW SUB-REQUESTS
• By this stage we’ve ruled out CDN, page generator, and content
weight as causing problems, but there are many other possible
causes of slow page load.
• We may have identified some of these problems when looking
at the waterfall in Webpage Test, or when looking at the sub-
requests log in the SEO Debug output, for example:
−Item API
−Environment files
−Categories
−Third parties

41. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.41
STEP 5:
ITEM API

42. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.42
ITEM API
• We need to understand how search works in order to understand why it can be slow
• A SolR query is ran on the application server, which indicates which items are to be
returned
• A database query is ran on the database server, which returns data for those items
• We can see those timings the timings using the following parameter: &ssdebug=T
Browser SolR Database query

43. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.43
ITEM API
1. Open request on chrome dev tools
2. Add &ssdebug=T parameter
SQL = 2.5 s

44. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.44
ITEM API
Item
(24)
Item
Data
Related Items
Custom Fields
Item Data, Stock Info,
Images
• itemid
• internalid
• displayname
• ispurchasable
• showoutofstockmessage
• stockdescription
• outofstockmessage
• isbackorderable
• isinstock
Stock Info
Images
Pricing
Info
• onlinecustomerprice
• onlinecustomerprice_formatted
• priceschedule

45. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.45
ITEM API
Item
(24)
Item
Data
Related Items
Custom Fields
• ispurchasable
• showoutofstockmessage
• stockdescription
• outofstockmessage
• isbackorderable
• isinstock
Stock Info
Images
Pricing
Info
• onlinecustomerprice
• onlinecustomerprice_formatted
• priceschedule

46. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.46
STEP 6:
ENVIRONMENT FILES

47. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.47
ENVIRONMENT FILES – USER ENVIRONMENT
•shopping.user.environment is blocking but non-cacheable
•Only user-specific content should be found here
•Troubleshooting
− Check for any information that is not user-specific and remove it
− Developers sometimes place data here to make sure that it is always fresh
− The problem is that it is loaded in all pages

48. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.48
ENVIRONMENT FILES – REGULAR ENVIRONMENT
•shopping.environment.ssp is blocking but cacheable
•You can bootstrap information here, but only if it is needed on all pages
•If the information is not needed on all pages, you can implement a service
to retrieve it on demand
•shopping.environment.ssp usually includes:
− Categories
− CMS
− Configuration

49. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.49
STEP 7:
CATEGORIES

50. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.50
CATEGORIES
•Design your category structure for performance and usability
•3 golden rules for blazingly fast categories:
−Use commerce categories
−Your category tree shouldn’t be deeper than 3 levels
−Each product should be assigned to a single category

51. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.51
CATEGORIES

52. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.52
STEP 8:
SCRIPTABLE CART

53. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.53
SCRIPTABLE CART
•What is scriptable cart?
•Parallelism between the back end and the web store
Sales Order form  BOGO

54. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.54
SCRIPTABLE CART
•What is scriptable cart?
−Executes client scripts for the sales order
record on the web store
−Per-website setting
−Problem: it executes ALL client scripts
•If adding items to the cart is slow, this is
generally why

55. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.55
SCRIPTABLE CART - TROUBLESHOOTING
•Customization  Scripting  Scripted Records

56. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.56
SCRIPTABLE CART - TROUBLESHOOTING
•Solution: exclude client scripts from the web store
−If you set execution context to “All Roles”  it executes on the web store
−If you set it to “All Employees”  it doesn’t

57. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.57
STEP 9:
USER EVENTS

58. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.58
PLACE ORDER
•Issues on order placement times usually come from workflows and User
Events
•User Event script execution context
−Checkbox called “Execute in Commerce Context” within the script’s deployment
−When this option is not selected, the script will not trigger for events that take
place in SCA, SB or SCIS

59. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.59
PLACE ORDER – USER EVENT EXECUTION CONTEXT

60. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.60
PLACE ORDER
•Setup  Company  Enable Features  Web Presence
−Asynchronous AfterSubmit Sales Order Processing

61. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.61
TOOL:
NETSUITE APM

62. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.62
NETSUITE APM

63. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.63
NETSUITE APM

64. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.64
NETSUITE APM
•It only records back end activity
•These are the same scripts that run on the web store
•You need to know which scripts execute on the web
−Client scripts: will run for the shopper role
−User Events: if “execute in commerce” is enabled

65. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.65
IN A NUTSHELL

66. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.66
STEP-BY-STEP RECAP
1. Check the Content Delivery Network (CDN)
2. Check the SEO Page Generator
3. Optimize your content (use WebPage Test and imageoptim)
4. Optimize page rendering (fonts, third parties, blocking requests)
5. Optimize the Item API
6. Check environment files
7. Review your category structure for speed
8. Review scripts that impact cart timings (scriptable cart)
9. Review user events (use APM)
SCA
SC
SB

67. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.67
NEED SOME HELP?
WE ARE THE 10th STEP

68. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.68
PERFORMANCE SERVICES
If you still need some help with investigating or remedying performance issues, then we can help.
• Performance reports and benchmarking
• Monitoring and alerting
• Performance audits
• Built to Perform (for new projects)
diego.cardozo@oracle.com
mark.sweeting@oracle.com

69. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.69
Q&A

70. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.70

71. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.71
Get involved, visit on Level 3
SuiteWorld

72. Copyright © 2018, Oracle and/or its affiliates. All rights reserved.72
THANK YOU