The document discusses performance testing myths related to the migration from Sage 500 to Sage 1000, debunking eight common misconceptions about non-functional testing. It details the project's background, the testing process, and observations made during testing, emphasizing the importance of rigorous performance validation. The conclusion highlights the need for testing and the potential for cost-effective solutions while cautioning against relying solely on vendor testing.

1. From Sage 500 to 1000
Performance Testing myths exposed
Richard Bishop

2. From Sage 500 to 1000
…………..Performance Testing myths exposed
 Introduction
 Trust IV & Richard Bishop
 Project Background
 8 myths of (non-functional)
software testing
 What we did…breaking myths
 Conclusion

3. Introduction
Who are we?

Trust IV
• Founded in 2005
• Testing consultancy, specialising in automated non-functional testing

Richard Bishop, Trust IV Ltd
• IT consultant with 20 years experience
• Specialising in Microsoft platforms and performance engineering / testing
• HP specialist, UK leader of Vivit (HP software user group in UK)

Colleagues
• Mixture of consultants and contract resources
• Primarily HP LoadRunner specialists
• On customer sites and working remotely
• Skills in multiple test tools, platforms etc.

4. Non-Functional Testing
What on earth is NFT?
Non-functional, automated testing specialists
What on earth is NFT?
In a nutshell……….
Usability, reliability and scalability.
Compatibility testing
Compliance testing
Security Testing
Backup and Disaster Recovery Testing
Load Testing
Performance Testing
Scalability Testing
Stress Testing
y
usabilit
liability
re
ity
scalabil

5. Project Background
University Hospital Birmingham





Sage 500 to Sage 1000 migration
Concerns re: numbers of concurrent users supported
Required performance test to validate potential maximum user load
Test to include a single “user journey”, simulating a requisition process
Objective is to increase user load until system failure

6. 8 Myths
(of non-functional software testing)








The vendor/developer has already tested this so we don't need to
NFT not required if functional testing and UAT is OK
Can / Can’t test in live environment
Web applications are easy to test using low-cost / open source tools
Anyone can test……
If a test fails “it must be the test tool / tester's fault”
Testing is too expensive / time consuming
If it's slow we can “throw kit at it”

7. Myth 1
Vendor/developer already tested - we don’t need to…

8. Myth 2
NFT not required if functional testing and UAT is OK
Single user
DB Server
Load balancer
Forms Server
IIS servers

9. Myth 2
NFT not required if functional testing and UAT is OK
Single user
Multi user
DB Server
Load balancer
DB Server
Load balancer
Forms Server
IIS servers
Forms Server
IIS servers

10. Myth 2
NFT not required if functional testing and UAT is OK
Single user
Multi user
DB Server
Load balancer
DB Server
Load balancer
Forms Server
IIS servers
Forms Server
IIS servers

11. Myth 2
NFT not required if functional testing and UAT is OK
Single user
Multi user
DB Server
Load balancer
DB Server
Load balancer
Forms Server
IIS servers
Forms Server
IIS servers

12. Myth 2
NFT not required if functional testing and UAT is OK
Single user
Multi user
DB Server
Load balancer
DB Server
Load balancer
Forms Server
IIS servers
Forms Server
IIS servers

13. Myth 3
You can/can’t test in a live environment
..…happen
results can be unreliable…..

14. Myth 3
You can/can’t test in a live environment
You can…..
….. prior to launch
…..or with extremely careful planning
…..or by mistake

15. Myth 3
Live environments.…. a cautionary tale

16. Myth 4
Anyone can test an application.
Source: http://www.pixar.com/short_films/Theatrical-Shorts/Lifted

17. Myth 5
Web apps are easy to test using low-cost/no-cost tools

18. Myth 5
Web apps are easy to test using low-cost/no-cost tools



19. Myth 6
If a test fails "it must be the test tool / tester's fault“

20. Myth 7
Testing is too expensive / time consuming
“The money spent with Trust IV was the best
money spent on the whole project”

21. Myth 8
If it's slow we can "throw kit at it"

22. What we did….

23. Our standard test approach
POC
Analysis
Performance
tests
Scripting
Low vol.
tests

24. POC- January 2013
We had a “steroid ferret”

NOT “just” a web app

25. POC- January 2013
Not a “web only app”

“Not just a web app“

Can use low cost / open source tools to test.


26. POC- January 2013
Needed specialist skills

“Not just a web app“

Can use low cost / open source tools to test.

Anyone can test


27. Digging deeper…
SAGE 1000 uses two communications protocols

28. Digging deeper…
SAGE 1000 uses two communications protocols
Had to convert displayed “human readable” text to legacy formats
to allow SAGE 1000 to interpret our simulated user input…

29. Digging deeper…
Complex test date requirements

30. Digging deeper…
Correlation
135532676
=
0x08141084
=
0xA 0x08 0x14, 0x10, 0x84 0xD

31. What we did
Our plan…


Script simulating login and requisition
1000 user accounts, 30,000 products
BUT
 Application update in January, meant re-coding, added delay 
 Initial tests showed problems with scalability…before requisition step fully scripted 
•
•
•
Problems @ 20 user load
Blank screens / HTTP 500 errors
Spent time proving test tool
Manual tests with volunteers
& network traces to prove
simulation equivalent to real users

32. Iterative testing began…
…modified test approach as we found defects

33. Initial tests




V. high thread
count
> 2300 threads
associated with
the JCSP process
V. High context
switches rate
> 17,000
switches / sec

34. Initial Tests
Key observations
“System Idle”
“Manual test”
“Automated test”
<2%
<2%
<6%
Available MBytes
30.1 GB
28.73 GB
27.3 GB
JCSP thread count
82
290
371
Total thread count
797
1932
2446
Context switches/sec
275
2000
2500
Processor queue length
<1
<1
<1
CPU utilisation
High thread count and context switching are key issues

35. Initial Tests
User experience

Response times
> 20 seconds

36. Next steps
Reconfiguration & re-test

We made recommendations to improve performance
We asked Datel to check:
•Heap size
•Application pool size
•Timeout values

Datel reconfigured application server:
•Encrypted login credentials within the application
•Altered TCP/IP timeout values and keep alives
•Set lifetime session limit to 30 minutes
•Registry changes

37. Next Steps
Re-test

Re-tested, but….

38. Re-test
Observations
Next steps - retest
Despite load
balancer problem
Response times
consistent
No degradation
over time

39. Further tests
Test stats
1. 230 users
1289 logins / hr
2. 250 users
1383 logins / hr
3. 500 users
2715 logins / hr
4. 500 users
5412 logins / hr

40. Final report
….some caveats

We noticed large numbers of HTTP 404 errors

Still missed some “asynchronous” traffic

Hadn’t tested complex application flows, due primarily to time constraints

41. Conclusion
What have we learned ?

You probably need to test
• Reduced response times for SAGE 1000 login
from > 20 seconds (and timeouts) to ≈ 3 s
• Application worked, just not our particular configuration

Testing needn’t be expensive
• Thanks to UHB for the endorsement

Don’t trust vendors (or developers) to test

42. Conclusion
What have we learned ?



You probably need to test
• Reduced response times for SAGE 1000 login
from > 20 seconds (and timeouts) to ≈ 3 s
• Application worked, just not our particular configuration
N,
KE
O
BR
BE ……
TO OK
RE IS
Testing needn’t be expensive
HE HS
T
• Thanks to UHB for the T
endorsement T
N’ M Y
E G
R(or developers) to test
Don’t trust S A
vendors
E
KI N
UL REA
R
TB
BU

43. Q&A
richard.bishop@trustiv.co.uk
@richardbishop @TrustIV
0844 870 0301