evaluating system effectiveness

1. INFORMATION SYSTEM
AUDIT-II
CHP 5. EVALUATING SYSTEM EFFECTIVENESS
MR. JAYANT. P. DALVI

2. INTRODUCTION
Need of Evaluation:
= to know if processes or systems of organization are suitable or
effective as input to their improvement process
Definition of Effectiveness and Suitablity:
1) Effectiveness: process or system is achieving planned goals or results
in capable manner with optimum use of resources
2) Suitability: chosen process or system and associated controls are
consistent with internal or external requirements or need

3. AUDIT QUALITY TEST
• 4 things to test during quality audit:
1) Is activity getting desired results?
2) Are resources(man, money, material) being used properly? Are they
cost-effective?
3) Are people be able to do it Right for first time?
4) Have right resources and controls been selected?

4. EFFECTIVENESS EVALUATION PROCESS
• Indicators of ineffectiveness:
- Excessive down time or idle time
- slow system response time
- excessive maintenance cost
- inable to interface with new software or hardware
- unreliable system outputs
- data loss
- frequent need for maintenance and modification

5. How to make system effective
• 2 approaches :
1) Goal centered view: to complete goals/objectives to achieve system
effectiveness, sacrificing of fundamental system qualities
2) System Resource view: identification of desirable qualities of system
and measurement of their levels.

6. TYPES OF SYSTEM EFFECTIVENESS EVALUATION
• Relative Evaluation:
- comparing of status of goal accomplished before implementation with
after implementation, improved task accomplishment and improved
quality of working life
Absolute Evaluation:
-assesment of size of goal accomplish after implementation of system

7. MEETING NEEDS FOR EFFECTIVENESS EVALUATION
• well managed organizations requires continous improvement with the
help of planning, implementing, checking and improving
• evaluations to be done to meet identified needs, execute plans, check
to see that needs were met and made improvements where required
• Benefits:
• Need for effectiveness evaluations continues to grow
• Departments have systematic processes to plan effectiveness
evaluation
• Departments complete most planned evaluations

8. QUALITY ASSURANCE PROCESSES
• It is designed to ensure that quality requirements are being met.
• helps to ensure that reports satisfy defined standards and provide
decision makers with reliable and useful evaluation findings.
• Example: Treasury Board of India orders 6 departments to perform
effectiveness valuation for their IS and give reports with
recommendations or response in it.

9. EVALUATING SYSTEM QUALITY
• Quality System Evaluation is:
• assessment of quality programs and processes continually to fulfill
internal and external expectations
• ensure to minimize risk and consistently produce high quality risk
• maximize profit and minimize losses
• verifies preparation of documentations for global system audits
• satisfy customer requirements and ensure their specification to be
meet consistently
• train employees to optimize quality of product within limits of
production process

10. • IS must be developed in accordance with all informational flows and
circuits and needs of decision systems
• Its quality evaluation can be made for systems that are in use or for
the systems that are in development stage.
• The results of audit process development must be analyzed and
conclusions must be translated into procedure and actions
iplemented in audited information systems.

11. EVALUATING INFORMATION QUALITY
• auditor must plan and perform audit procedures to obtain sufficient
appropriate audit evidence to provide reasonable basis for his opinion
• Sufficiency is measure of quantity of audit evidence
• Appropriateness is measures of quality of audit evidence i.e its
relevance and reliability.
• To be appropriate, audit evidence must be both relevant and reliable
to provide support for conclusions on which auditor's opinion is
based.

12. 1) Relevance: refers to objective of control being tested
it is depends on:
- design of audit procedure used to test the control
- timing of audit procedure used to test the control
2) Reliability: depends on nature and source of evidence and
circumstances under which it is obtained
- evidence obtained from external knowledge source is reliable than
internal company sources
- evidence obtained directly by auditor is more reliable.
- evidence obtained from original documents is more reliable than
photocopies, scanned copies, electronic copies

13. 3) Information Quality: describe quality of content of IS.
it is measure of value which information provides to user of that information
4 classes of IQ:
• Intrinsic IQ: accuracy, objectivity, believable, reputation
• Contexual IQ: relevancy, value added, timeliness, completeness, amt of info
• Representational IQ: Interpretability, Compatibility
• Accessibility IQ: accessibility, access security

14. EVALUATING PERCEIVED USEFULNESS
• Perceived usefulness is user's probability of using specific application will
increase his or her job performance within organizational context
• It is linked with whether user will get reward after using the system and
attitude they have toward the system.
• If they find it favourable they will use system most frequently otherwise not.
• Items to measure perceived usefulness:
users perceive that IS enables them to accomplish task associated with their job
more quickly, improve their job performance, increase their productivity and
effectiveness, useful.
• Auditors can include this items in questionaire or interviews if they want to
evaluate perceived usefulness

15. EVALUATING INFORMATION SYSTEM
SATISFACTION
• end user satifies if the characteristics of sytem are accurace and
reliable.
• end user satisfies when IS consists of web-based products and
services

16. EVALUATING INDIVIDUAL IMPACT
• impact of IT on individual enhances the performance of individual in
terms of productivity, efficiency and effectiveness.
• enhances task productivity, task innovation and individual sense of
accomplishment
• helps individual to create and try new ideas in their work
• enhances sense of togetherness and closeness

17. EVALUATING ORGANIZATIONAL IMPACT
• IT provides advantages to IS like ability of IT to link and enable
employees , communication increased.
• also various people from different department from same or different
organizations can communicate easily.
• quicker availability of information
• eliminates repetition of information because of boundary spanning.
• information can be stored, retrieved and transferred easily and
quickly.

18. EVALUATING COMPUTER SELF-EFFICACY
• it refers to individual's ability in his or her ability to apply computer
skills to wide range of tasks.
• it includes determinants and consequences.
• consequences are grouped into outcomes, beliefs and behaviours.
• here determinant is simple use of computer or frequency of using a
computer
• individual with high self-efficacy will perceive the system to be easy
and useful while individual with low self efficacy will avoid to use the
system
• it affects system usage directly and indirectly through perceived ease
of use and perceived usefullness

19. EVALUATING IS USE
• IS is combination of IT and people's activities that supports
operations, management and decision making.
• IS also referred as interaction between people, process, data and
technology
• it is a form of communication system in which data is processed and
presented to the user.
• IS is essential and beneficial if user resists to change.
• Ethics of IS- privacy, property, accuracy, access, data processing and
decision making, security, disaster recovery.
• usefulness of IS in following fields: finance and accounting, sales and
marketing, HRM, etc.ss