This document discusses the impact of using intelligent decision aids (IDAs) on auditors' procedural and declarative knowledge, addressing potential deskilling effects due to reliance on technology in auditing practices. It proposes research questions and hypotheses related to the relationship between IDA usage and auditors' knowledge retention, while also outlining a proposed methodology for testing these hypotheses through surveys and interviews. The findings aim to contribute to both theoretical understanding of technology dominance in decision-making and practical applications for designing effective audit support systems.

1. The effects of continued use of intelligent decision aids upon auditor procedural knowledge Micheal Axelsen Research Colloquium 2009

2. Overview Research problem Research questions Contributions Motivation Theory development Background Theoretical model Hypothesis development Methodology Potential limitations Appendix: References Agenda

3. Research problem The continued use of tools to support the audit process is needed to help ensure audit quality and consistency However, persistent use of this technology results in auditors being deskilled in terms of applying their judgment in the audit process So this research-in-progress is designed to examine the impact of the continued use of technology-based intelligent decision aids upon auditor skill levels

4. Motivation Practical: My practical experience in an audit firm for ten years is partly my motivation for this (including ‘junior-burger’ effect – substituting ‘juniors’ for highly-paid senior staff through use of an IDA) Dowling & Leech (2007) noted that practitioners did not raise deskilling as an issue - confirmed by my exploratory interviews Theoretical: The Theory of Technology Dominance (Arnold & Sutton 1998) outlines a deskilling proposition –tested only once (Dowling, Leech & Moroney 2006) for declarative knowledge Leech (2008) noted ‘alarm bells ringing’ - research into design of audit support systems and long-term consequences for knowledge development was needed

5. Research questions Does continued use of IDAs reduce auditor declarative knowledge? Does continued use of IDAs reduce auditor procedural knowledge? Does ability, experience, motivation, or environment have an impact upon this deskilling effect?

6. Contributions Contributions of this potential research to theory: Increase robustness through reconciliation with other established theories (anchoring & adjustment heuristic, cognitive load theory) to extend implications to other professional areas Extend theoretical testing to the field Test proposition 7 (deskilling) in the context of procedural knowledge (‘auditor know-how’) Contributions of this potential research to practice: Contributes towards establishing a basis for the design of effective audit support systems that reduce the deskilling effect

7. Background & theoretical model Reliance by auditors on information systems in the audit process is increasing (Dowling & Leech 2007; Leech 2008; Mascha 2001), and this is confirmed by experience and observation Many benefits of IDAs are recognised, including training , efficiency, consistency (Elliott & Kielich 1985; O’Leary 1987; Rose & Wolfe 1998; Sutton & Byington 1993) Managers also perceive that IDAs allow for the control of junior staff and improve risk management (Dowling & Leech 2007) There may be negative epistemelogical implications of IDA use in the long term that affect accounting expertise (Arnold & Sutton 1998; Dowling, Leech & Moroney 2006; Leech 2008;McCall, McCall, Arnold & Sutton 2008; Mitroff & Mason 1989; Todd & Benbasat 1992; Todd & Benbasat 1994)

8. Anchoring & adjustment heuristic After Tversky & Kahnemann (1974); Epley & Gilovich (1996)

9. Cognitive load theory After Sweller (1988); Libby & Tan (1994)

10. Theory of technology dominance Sutton (2006) – factors in reliance: At low to moderate level of experience, there is a negative relationship between task experience and reliance on a decision aid Positive relationship between task complexity and reliance on a decision aid Positive relationship between decision aid familiarity and reliance on the decision aid Positive relationship between cognitive fit and reliance on the decision aid

11. Theory of technology dominance Sutton (2006) – susceptibility to dominance by technology: When user expertise and an IDA are mismatched, there is a negative relationship between the user’s expertise level and the risk of poor decision making When user expertise level and an IDA are matched, there is a positive relationship between reliance on the aid and improved decision making Sutton (2006) – long-term effects (deskilling): Positive relationship between continued use of an IDA and the de-skilling of knowledge workers’ abilities for the domain in which the aid is used Negative relationship between the broad-based, long term use of an IDA in a given problem domain and the growth in knowledge and advancement of the domain

12. Theoretical model

13. Hypothesis development The following hypotheses are proposed: H1: The longer an auditor has continuously used an IDA the less declarative knowledge they possess H2: The longer an auditor has continuously used an IDA the less procedural knowledge they possess H3a: The higher an auditor’s experience the less effect the continuous use of an IDA has upon declarative knowledge H3b: The higher an auditor’s experience the less effect the continuous use of an IDA has upon procedural knowledge

14. Hypothesis development Proposed hypotheses (continued): H4a: The higher an auditor’s ability the less effect the continuous use of an IDA has upon declarative knowledge H4b: The higher an auditor’s ability the less effect the continuous use of an IDA has upon procedural knowledge H5a: The higher an auditor’s motivation the less effect the continuous use of an IDA has upon declarative knowledge H5b: The higher an auditor’s motivation the less effect the continuous use of an IDA has upon procedural knowledge

15. Approach A survey of public sector auditors to test the theoretical model is proposed - this will be supported by a series of semi-structured interviews Participants are potentially 1,096 Australian public sector audit staff – access may be possible to a further 1,700 international participants Semi-structured interviews with senior staff will be undertaken with seven public sector audit offices A survey will be undertaken with these participants; it is expected that approximately 219 usable responses across 7 offices (with differing implementations of IDAs) will be received (20% response) The survey instrument will draw upon existing instruments where possible, and new measures where necessary

16. Potential limitations External validity Conclusions limited to Australian public sector – may not generalise to private sector auditors Internal validity Potential bias in the auditor cohort as auditors leave public sector – deskilled auditors may leave and thus not be captured Construct validity Measuring procedural and declarative knowledge will be difficult and necessarily perceptual Concerned that amount of procedural knowledge in tools used is little or limited – may not see highly intelligent/restrictive IDAs instantiated in any instances we study, and declarative knowledge and structural restrictiveness have previously been dealt with comprehensively

17. References Appendices

18. References Anderson, J. R. (1993). Rules of the Mind. Hillsdale, New Jersey: Erlbaum. Anson, R., Bostrom, R., & Wynne, B. (1995). An experiment assessing group support system and facilitator effects on meeting outcomes. Management Science, 41(2), 189. Arnold, V., Clark, N., Collier, P., Leech, S., & Sutton, S. G. (2006). The Differential Use and Effect of Knowledge-Based System Explanations in Novice and Expert Judgment Decisions. MIS Quarterly, 30(1), 79-97. Arnold, V., Collier, P., Leech, S., & Sutton, S. G. (2004). Impact of intelligent decision aids on expert and novice decision-makers' judgments. Accounting and Finance, 44(1), 1-26. Arnold, V., & Sutton, S. G. (1998). The Theory of Technology Dominance: Understanding The Impact Of Intelligent Decision Aids On Decision Makers’ Judgments. Advances in Accounting Behavioral Research, 1(1), 175-194. Awasthi, V., & Pratt, J. (1990). The Effects of Monetary Incentives on Effort and Decision Performance: The Role of Cognitive Characteristics. The Accounting Review, 65(4), 797.

19. References Chu, P. C., & Elam, J. J. (1990). Induced system restrictiveness: An experimental demonstration. IEEE Transactions on Systems, Man and Cybernetics, 20, 195-201. DeSanctis, G., & Poole, M. S. (1994). Capturing the complexity in advanced technology use: Adaptive structuration theory. Organization Science, 5(2), 121. Dowling, C., Leech, S., & Moroney, R. (2006). The Deskilling of Auditors’ Abilities: An Empirical Test of the Theory of Technology Dominance Paper presented at the Second Asia/Pacific Research Symposium on Accounting Information Systems. Dowling, C., & Leech, S. (2007). Audit support systems and decision aids: Current practice and opportunities for future research. International Journal of Accounting Information Systems, 8(2), 92-116. Elliott, R. K., & Kielich, J. A. (1985). Expert systems for accountants. Journal of Accountancy, 160(3), 126-134. Hampton, C. (2005). Determinants of reliance: An empirical test of the theory of technology dominance. International Journal of Accounting Information Systems, 6(4), 217-240.

20. References Joyce, E. J., & Biddle, G. C. (1981). Anchoring and adjustment and probabilistic inference in auditing. Journal of Accounting Research, 19(1), 120-145. Kowalczyk, T., & Wolfe, C. (1998). Anchoring effects associated with recommendations from expert decision aids: an experimental analysis. Behavioral Research in Accounting, 10(Supplement), 147-169. Leech, S. (2008, 1 November 2008). The use and misuse of intelligent systems in accounting: the risk of technology dominance. Insights: Melbourne Economics and Commerce Retrieved 7th July 2009, 2009, from http://insights.unimelb.edu.au/vol4/08_Leech.html Libby, R., & Tan, H.-T. (1994). Modeling the determinants of audit expertise. Accounting, Organizations and Society, 19(8), 701-716. Mascha, M. F. (2001). The effect of task complexity and expert system type on the acquisition of procedural knowledge: some new evidence. International Journal of Accounting Information Systems, 2(2), 103-124.

21. References Mascha, M. F., & Smedley, G. (2007). Can computerized decision aids do “damage”? A case for tailoring feedback and task complexity based on task experience. International Journal of Accounting Information Systems, 8(2), 73-91. Masselli, J. J., Ricketts, R. C., Arnold, V., & Sutton, S. G. (2002). The Impact of Embedded Intelligent Agents on Tax Compliance Decisions. Journal of the American Taxation Association, 24(2), 60-78. McCall, H., Arnold, V., & Sutton, S. G. (2008). Use of Knowledge Management Systems and the Impact on the Acquisition of Explicit Knowledge. Journal of Information Systems, 22(2), 77-101. Mitroff, K., & Mason, R. (1989). Deep ethical and epistemological issues in the design of information systems. Expert System Review(Fall), 21-25. Noga, T., & Arnold, V. (2002). Do tax decision support systems affect the accuracy of tax compliance decisions? International Journal of Accounting Information Systems, 3(2), 125-144.

22. References O'Leary, D. (1987). Validation of expert systems with applications to auditing and accounting expert systems. Decision Science, 18(3), 468-486. Presutti, A. H. (1995). Anchor and adjustment heuristic effect on audit judgement. Managerial Auditing Journal, 10(9), 13-21. Rose, J. M., & Wolfe, C. J. (1998). Schema acquisition and the effects of cognitive load imposed by decision aid design alternatives: a tax setting. Paper presented at the Proceedings of the Fifth Accounting Information Systems Conference, Scottsdale, AZ. Silver, M. (1988). Descriptive analysis for computer-based decision support. Operations Research, 36(6), 904-916. Slovic, P., & Lichtenstein, S. (1971). Comparison of Bayesian and regression approaches to the study of information processing in judgment. Organizational behaviour and Human Performance, 6, 649-744. Speier, C., & Morris, M. G. (2003). The influence of query interface design on decision-making performance1. MIS Quarterly, 27(3), 397.

23. References Sutton, S. G., & Byington, J. (1993). An analysis of ethical and epistemological issues in the development and implementation of audit expert systems. Advanced Public Interest Accounting, 5, 234-243. Sutton, S. G. (2006). Theory of Technology Dominance: Reflections on the Research to Date and the Mysteries Unexplored. Paper presented at the Digital Accounting Research Conference, University of Huelva, Spain. Sweller, J. (1988). Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science, 12(2), 257-285. Todd, P., & Benbasat, I. (1992). The Use of Information in Decision Making: An Experimental Investigation of the Impact of Computer-Based Decision Aids. MIS Quarterly, 16(3), 373. Todd, P., & Benbasat, I. (1994). The influence of decision aids on choice strategies: An experimental analysis of the role of cognitive effort. Organizational Behavior and Human Decision Processes, 60(1), 36. Tversky, A., & Kahnemann, D. (1974). Judgment Under Uncertainty: Heuristics and Biases. Science, 185(4157), 1124-1131.

24. References Wheeler, B. C., & Valacich, J. S. (1996). Facilitation, GSS, and training as sources of process restrictiveness and guidance for structured group decision making: An empirical assessment. Information Systems Research, 7(4), 429.