Slides (5 - The QA Focus Perspective) used in workshop session B4 on "Catching Mistakes: QA for your Web site" at the IWMW 2003 event held at the University of Kent on 11-13 June 2003. See http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-2003/sessions/index.html#workshops-b

1. QA Focus – Supporting JISC's
The QA Focus Perspective
Marieke Guy
UKOLN
M.Guy@ukoln.ac.uk
http://www.ukoln.ac.uk/qa-focus/http://www.ukoln.ac.uk/qa-focus/

2. QA Focus – Supporting JISC's
The QA Perspective
If you want to sort out the problems on your
Web site you can do so by developing a
Quality Assurance Framework
This involves:
• Testing
• Problem tracking and corrective action
• Policies
• Procedures
• Documentation

3. QA Focus – Supporting JISC's
What Can You Test?
• Functional testing
• Compatibility testing
• Load/performance testing
• Stress testing
• Usability testing
• Security testing
• Integration of unit testing
• Link testing
• HTML and CSS validation
• Reliability testing
• Regression testing
• …

4. QA Focus – Supporting JISC's
Automated Testing
• Should be scalable
• Only suitable approach for large sites
• Can push (email) information about
problems
• Many free testing tools available
but:
• Will not spot all usability/accessibility
problems
• Can provide too much information
• Comprehensive and configurable
testing tools can be expensive

5. QA Focus – Supporting JISC's
Manual Testing
• Can spot usability/accessibility problems
• Can make use of one’s community/user
base/peers
• Can use of key pages which will identify
problems which occur elsewhere
• Useful for testing new Web sites
• “5 users can spot most of your problems”
but:
• Not suitable for testing large numbers of
pages
• Quality of feedback may be variable
• Often not useful for testing
established Web sites

6. QA Focus – Supporting JISC's
Approaches to Automated Testing
Automated testing can make use of:
• Web-based tools
• Desktop tools
UKOLN has made use of Web-based testing
tools:
• Monitor accessibility, HTML, CSS,
compliance, page size, links quality, nos.
of links, etc.
• Benchmarking approach provides
comparison with one’s peers
http://www.ukoln.ac.uk/qa-focus/surveys/http://www.ukoln.ac.uk/qa-focus/surveys/

7. QA Focus – Supporting JISC's
Peer Review
• “Give me comments on my Web site and
I’ll also test yours”
• Can help in community-building
• Particularly useful for programmes in
which project Web sites being developed
at same time
But:
• Can be difficult in processing
unstructured responses
• Need to manage process

8. QA Focus – Supporting JISC's
Problem Tracking
There are three stages in a corrective action
system:
• The short term fix
• The cure
• Prevention of errors – identify potential
causes of error by analysing data to setect
trends which if allowed to continue would
result in errors

9. QA Focus – Supporting JISC's
Documentation
3 Levels of documentation:
• Policy e.g. Quality Manual
• System e.g. Processes and procedures
• Work practices e.g. instructions
More detailed, more specific, applies to
fewer people

10. QA Focus – Supporting JISC's
Policy
A plan or course of action intended to
influence and determine decisions
Procedure
A way of performing or effecting something
Process
A series of actions, changes, or functions
bringing about a result
Definitions

11. QA Focus – Supporting JISC's
Documentation
• Good documentation
• Policies, procedures and audits
• Define procedures for all processes
• Requirements
• Specifications
• Mission statements or statement of direction
• Define your audience
• Document the testing suite and tools
• Document authoring tools etc.
• Document problems tracking
• A Quality Management System
(QMS)??

12. QA Focus – Supporting JISC's
Issues I
• Goals for testing will be related to the kind
of site you have
• Testing time is limited – use automated
tools
• Documentation is critical – to make sure
practices are repeatable
• Implement your QA procedures slowly
over time to avoid stifling productivity
• QA planning at the start of development
means less time fixing things at the end

13. QA Focus – Supporting JISC's
Issues II
• Motivating/persuading people to
implement a QA Culture can be difficult –
run workshops to show people what you
are aiming for
• Make sure you have a feedback
mechanism for the procedures you are
implementing
• Having a great team/management is
crucial

14. QA Focus – Supporting JISC's
The QA Focus Perspective
QA Focus:
• Funded by JISC
• Currently provided by UKOLN and AHDS
• Currently supports JISC's 5/99 programme
• Staff:
• Brian Kelly, Project manager
• Marieke Guy, QA Focus officer, UKOLN
• Hamish James, QA Focus officer, AHDS
• Gareth Knight, QA Focus officer, AHDS

15. QA Focus – Supporting JISC's
Approach Taken
Two possible approaches to ensuring
compliance with standards and best practices:
Enforce
• Inspect all project’s work
• Strict auditing, with penalties for no-
compliance
Encourage
• Training for project staff
• Developmental, explaining reasons for
compliance, documenting examples of best
practices and providing advice on
implementation and monitoring
The latter approach is preferable,
especially in a HE context
The latter approach is preferable,
especially in a HE context

16. QA Focus – Supporting JISC's
QA Focus Work
• Documentation and Advice
• Briefing Papers
• Case Studies
• FAQs
• Descriptions of tools and architectures for
projects to implement best practices
• In-house QA
• Surveys
• Providing Motivation
• Workshops
• Liasing
• The QA Focus Toolkit

17. QA Focus – Supporting JISC's
Documentation: Briefing Papers
Advisory briefing
documents are being
produced
These are:
• Brief, focussed
documents
• Informed by
findings of the
surveys
Advisory briefing
documents are being
produced
These are:
• Brief, focussed
documents
• Informed by
findings of the
surveys

18. QA Focus – Supporting JISC's
Documentation: Case Studies
Case Studies are being
commissioned
These are:
• Written by projects
themselves
• Describe the solution
adopted to a
particular problem
• Include details of
lessons learnt – not
just a press release!
Case Studies are being
commissioned
These are:
• Written by projects
themselves
• Describe the solution
adopted to a
particular problem
• Include details of
lessons learnt – not
just a press release!

19. QA Focus – Supporting JISC's
Web Site Testing Tools
http://www.ukoln.ac.uk/qa-focus/qa-projects/http://www.ukoln.ac.uk/qa-focus/qa-projects/
• Full list of Web site
testing tools
available from
database
• Surveys run with a
number of the tools
• In the future there
will be reviews of
tools
• Full list of Web site
testing tools
available from
database
• Surveys run with a
number of the tools
• In the future there
will be reviews of
tools

20. QA Focus – Supporting JISC's
Describing Tools and Approaches
As an example of
approach which provides
easier access to testing
tools, see the ,tools
approach deployed on
UKOLN Web site.
As an example of
approach which provides
easier access to testing
tools, see the ,tools
approach deployed on
UKOLN Web site.
This approach:
• Applies to all resources on Web site
• Covers HTML and CSS validation and
various other tests
• Some recursive apps (,rvalidate)
• Easily implemented with single line redirect
This approach:
• Applies to all resources on Web site
• Covers HTML and CSS validation and
various other tests
• Some recursive apps (,rvalidate)
• Easily implemented with single line redirect

21. QA Focus – Supporting JISC's
In-house QA
• Policies
• Procedures
• Audits
To date:
• Web Standards - XHTML, CSS
• Linking - links to and from
• Accessibility
• Usage Statistic
• Metadata

22. QA Focus – Supporting JISC's
Documentation and Advice I
We have:
• Surveyed project Web sites and identified
areas of lack of compliance with standards
and best practices
• Demonstrated examples of the potential
importance of compliance for repurposing
resources

23. QA Focus – Supporting JISC's
Documentation and Advice II
In addition we need to provide:
• Brief focussed advice on the standards
• Information on how to monitor compliance
• More Case studies on solutions deployed
by projects themselves
• Guidance on dealing with implementation
difficulties and what to do when strict
compliance is difficult to achieve
• Reviews of testing tools

24. QA Focus – Supporting JISC's
Surveying The Community
Surveys of project Web sites have been carried
out in order to:
• Obtain a profile for the community
• Identify examples of best practices
• Identify areas in which further advice is
needed
Surveys included:
• Accessibility
• 404 error pages
• HTML & CSS compliance

25. QA Focus – Supporting JISC's
Survey Philosophy I
The surveys made use of freely-available Web-
based tools:
• Methodology is open
• No software needs to be installed locally
(apart from Web browser)
• Findings can be reproduced
• Latest results can be obtained by clicking
on link to testing service

26. QA Focus – Supporting JISC's
Survey Philosophy II
The surveys typically examined project entry
points and not entire Web site as:
• This page has the highest profile
• The aim is to validate a methodology
which can be deployed by projects
themselves, not to test every page on
behalf of the projects

27. QA Focus – Supporting JISC's
Survey Findings
Findings available from
http://www.ukoln.ac.uk/qa-focus/surveys/http://www.ukoln.ac.uk/qa-focus/surveys/

28. QA Focus – Supporting JISC's
Providing Motivation
There is a need to explain why compliance with
standards is important and to provide
motivation for projects to update their tools,
authoring procedures, etc.This may involve
• More workshops
• Evaluating project's QA plans
• Liaising with projects over the requirements of the
key QA procedures applicable to their work
• Liaising with the Advisory Services over their
provision of advice and address maintenance and
feedback mechanisms
• Overseeing QA work

29. QA Focus – Supporting JISC's
Self Assessment Toolkit
We will be developing a self-assessment toolkit
for projects to use, by individual projects or
across project clusters
The toolkit will consist of:
• Examples of QA procedures
• Documented examples of use of testing
tools
• Self-assessment questionnaires
• Advice on standards and best practices
• Case studies, FAQs etc.
http://www.ukoln.ac.uk/qa-focus/toolkit/http://www.ukoln.ac.uk/qa-focus/toolkit/

30. QA Focus – Supporting JISC's
QA for Digitisation
Do it once…..do it right:
• Project is fundamentally dependent upon
the quality of original product
• Quality is the pre-requisite to preservation
• Quality expectations will only grow
• Delivery problems can be fixed, but
capture problems normally can’t

31. QA Focus – Supporting JISC's
QA For Web Sites
• The Web is the main delivery mechanism for
projects and services
• An increasing awareness of the importance
of accessibility, use of new devices (PDAs,
WAP, e-books, …), repurposing of Web
content (e.g. archiving)
• But invalid HTML is still the norm
• Look to standards – XHTML, XML, CSS, WAI
• If proprietary formats need to be used, flag
them and use in most open way
• QA Focus is trying to explain why and how

32. QA Focus – Supporting JISC's
Next Steps
Extended Coverage
We will be moving on from Web and
digitisation to include other areas including:
• Metadata
• Multimedia
• Software development
• Deployment into service
• …
NextSteps

33. QA Focus – Supporting JISC's
Next Steps – Metadata
Thoughts on QA for metadata:
Syntax Check syntax on embedded DC
Cataloguing Rules Ensure projects have appropriate
cataloguing rules
Interoperability Ensure metadata can
interoperate with third parties
Fitness for purpose Ensure metadata is appropriate
for its purpose
Change control Ensure architecture for managing
metadata can cope with change
control (cf Exploit Case Study)

34. QA Focus – Supporting JISC's
Next Steps – Software
Thoughts on QA for software:
Specification Process for developing systems
spec, agreeing spec with
stakeholders, etc.
Methodology Documentation of systems
development methodology (cf UML
case study)
Testing Documentation of testing
methodology
Deployment Awareness of deployment
challenges

35. QA Focus – Supporting JISC's
Limitations
There are a number of limitations to the work we
have carried out so far:
• Project Web sites have different purposes
(information about the project;
communications with project partners;
project deliverables themselves; etc.)
• Projects have different levels of funding,
resources, expertise, etc.
• Projects are at different stages of
development (and some have finished)

36. QA Focus – Supporting JISC's
QA Focus And You!
What relevance has QA Focus for you?
• You can deploy QA Focus practices for
your projects, etc
• You can write case studies and advisory
documents:
• Seen to be working with JISC
• We can promote your approaches and work
• This will help instigate best practices within
your group
• You can develop QA procedures in
innovative areas (e.g. OAI, thesaurii,
ePrints, e-Books, etc.) which
QA Focus can use

37. QA Focus – Supporting JISC's
Questions
Any questions?