A webinar by Joel Montevilsky on testing Practitest, a SaaS service.

2. About me - Joel Montvelisky
PractiTest – Co Founder & Prod. Architect
QA Instructor & Consultant
Mercury Interactive - QA Manager (retired )
TD, QC, WR, QTP, etc
ITCB (IL) Advisory Board
QABlog.practitest.com
© 2010 -

3. Today’s Agenda
 SaaS - Software as a Service
 Things we Can Stop Testing
 Non-trivial Stuff to Think About & TEST
 Leveraging Live Feedback into the Testing Lifecycle
 Open Season for Questions…
© 2010 -

4. What is SaaS?
© 2010 -

5. Software as a Service
 Hosted Platforms
– Total Control
– No Maintenance
 No Client Installations
 Customers buys each month – keep them happy 
Users don’t care it is SaaS,
it needs to be better than the alternative
© 2010 -

6. SaaS QA Management
 End-to-End QA Management (Reqs – Tests – Bugs)
 Enterprise Level SaaS Solution
- Methodology
- Customizations & Flexibility
 Global platform
- Communication
- Languages
- Available 24x7 Worldwide
 Intuitive & Simple
 Agile Dev, RoR, on Amazon EC2
© 2010 -

7. What does SaaS mean
to the Testing Process?
© 2010 -

8. Things we can STOP Testing
✘ Client or Server installations
✘ Multiplatform backend support
✘ Multiple version upgrades
“Savings” of approx. 1/3
✘ Backwards compatibility of the Traditional Enterprise
Application Testing Cycle
© 2010 -

9. The regular testing scenarios
 Functionality
- Manual scripts
- Exploratory sessions Agile Sprints & Updates
- Checklists of 3 – 4 weeks
- End-2-End Scenarios
- Selenium Each sprint’s cycle is
- Cucumber & RSpec based on its content
 Load & Stress
 Multi-platform & Multi-browser support
 I18N
English, Swedish, German, Chinese Simplified, Hebrew
© 2010 -

10. Non-trivial stuff to think about & TEST
1. Remote accessibility & usage
 Staging Servers in USA - Amazon EC2
Development & Testing teams in Israel (& WW collaborators)
 Globally distributed
monitoring services
(www.pingdom.com)
© 2010 -

11. Non-trivial stuff to think about & TEST
2. Security
Application-related: Infrastructure-based:
 Cross-site scripting  Secured communication
 SQL injections  Backups & storage policies
 HTTP header injections  Controlled Access to the
site & servers
etc…
© 2010 -

12. Non-trivial stuff to think about & TEST
3. Live updates & deployments (3 to 4 weeks)
 No concrete separation between Dev & IT
 Minimal service disruption
 Dressed rehearsals (including rollbacks!)
 Automated sanity & manual verification
 Patches & hot-fixes (extraordinary occasions )
 Prove of Concepts & Beta Releases
QA & Testing as a facilitator for Balance:
Stable & Professional, yet Flexible Environment
© 2010 -

13. Non-trivial stuff to think about & TEST
4. Disaster recovery procedures
Two Main scenarios:
(a) System down to be brought up quickly
 Configure machines
 Install & deploy software
 Restore data
Define a policy & schedule regular
drills to ensure you can achieve it!
(b) Rollback to last known stable data
(with or without data restore)
© 2010 -

14. Leveraging Real Data
into the Testing Process
© 2010 -

15. Levering live feedback into
the Testing Process
(1) Internally developed monitoring tools
- Check usage patterns
No need to
- Validate need & usability assumptions continue guessing!!!
- Analyze patterns and create realistic & prioritized testing scenarios
© 2010 -

16. Levering live feedback into
the Testing Process
(2) Site Monitoring systems
LiveAdmin (www.liveadmin.net)
 Real-time usage to schedule updates
 Allow users to contact the company
with questions or issues
Pingdom (www.pingdom.com)
 Up to date response times for real projects and data.
 Proactive alerts when things start slowing down
© 2010 -

17. Levering live feedback into
the Testing Process
(3) Proactive mail notifications on issues
(www.hoptoad.com)
 Real-time notifications on issues on the system.
 Information about the issue including system parameters that allow you to do an
initial trouble-shoot of the issue
© 2010 -

18. Levering live feedback into
the Testing Process
(4) Ask users for Real Projects
 No better way to test than using real data
 Most users will trust you not to abuse their information in exchange for
assurance of working software
 Since the data sits in your system it is easier to copy projects to your
testing environment
 Need to be careful with who has access to the data; making sure your
testing environment is as secured as your production environment
© 2010 -

19. Wrapping up…
© 2010 -

20. Wrapping up
 SaaS is only the delivery model, customers will not compromise
on Functionality, Stability or Response Time
 Since the system is hosted you can save all tests related to backend
installation and support
 You need to pay more attention to stuff like accessibility, security,
deployments & DRPs
 The lines between the R&D and IT organizations in SaaS are blurry
 SaaS application allows better access to real data to be leveraged to
improve the effectiveness of our testing operations.
© 2010 -

21. Open Season
for questions!
© 2010 -

22. Special thanks to Anne-Marie Charrett 
joel@practitest.com
http://qablog.practitest.com
© 2010 -