Successfully reported this slideshow.

Shadow IT And The Failure Of IT Architecture

12

Share

1 of 69
1 of 69

Shadow IT And The Failure Of IT Architecture

12

Share

Download to read offline

The continued existence and growth of shadow IT gives IT architecture the opportunity show leadership. IT architecture can be the gateway for business IT solution requirements, from initial solution concept through to solution realisation.

Shadow IT is a set of reactions by business functions to an actual or perceived inability or unwillingness of the IT function to respond to business needs for IT solutions. There are many aspects of shadow IT:

• Shadow Projects
• Shadow Data
• Shadow Sourcing
• Shadow Development
• Shadow Solutions
• Shadow Support Arrangements

Shadow IT takes many forms and types

1. CUST – customised solution developed by a third-party
2. DEV – personal devices used to access business systems or authenticate access to hosted solutions used for business
3. DIY – end-user computing application developed by the business
4. HOME – organisation data sent to home devices to be worked on
5. MSG – public messaging and data exchange platforms
6. OPEN – open-source software used as a stand-alone solution or incorporated into other solutions
7. OUT – outsourced service solution
8. PROD – software product acquired by the business and implemented on organisation infrastructure
9. PUB – accessing organisation applications and data using public devices or networks
10. STOR – public data storage and exchange platforms
11. SVC – hosted software solution

Uncontrolled shadow IT represents a real risk to organisations. The experience from previous shadow IT examples is that they have resulted in real financial losses. IT architecture can and should take the lead in implementing structures and processes to mitigate risks while taking maximising the benefits of shadow IT.

The continued existence and growth of shadow IT gives IT architecture the opportunity show leadership. IT architecture can be the gateway for business IT solution requirements, from initial solution concept through to solution realisation.

Shadow IT is a set of reactions by business functions to an actual or perceived inability or unwillingness of the IT function to respond to business needs for IT solutions. There are many aspects of shadow IT:

• Shadow Projects
• Shadow Data
• Shadow Sourcing
• Shadow Development
• Shadow Solutions
• Shadow Support Arrangements

Shadow IT takes many forms and types

1. CUST – customised solution developed by a third-party
2. DEV – personal devices used to access business systems or authenticate access to hosted solutions used for business
3. DIY – end-user computing application developed by the business
4. HOME – organisation data sent to home devices to be worked on
5. MSG – public messaging and data exchange platforms
6. OPEN – open-source software used as a stand-alone solution or incorporated into other solutions
7. OUT – outsourced service solution
8. PROD – software product acquired by the business and implemented on organisation infrastructure
9. PUB – accessing organisation applications and data using public devices or networks
10. STOR – public data storage and exchange platforms
11. SVC – hosted software solution

Uncontrolled shadow IT represents a real risk to organisations. The experience from previous shadow IT examples is that they have resulted in real financial losses. IT architecture can and should take the lead in implementing structures and processes to mitigate risks while taking maximising the benefits of shadow IT.

More Related Content

More from Alan McSweeney

Related Books

Free with a 14 day trial from Scribd

See all

Shadow IT And The Failure Of IT Architecture

  1. 1. Shadow IT And The Failure Of IT Architecture Alan McSweeney http://ie.linkedin.com/in/alanmcsweeney https://www.amazon.com/dp/1797567616
  2. 2. Introduction •Failure to engage with the business to understand their information technology needs so the business frequently bypasses IT •Failure to address solution standards and solution definition and identification problems that cause delays in solution delivery to the business •Failure to define solutions and approaches to address the current widespread usage of shadow IT solutions May 20, 2019 2 • The continued existence of shadow IT represents multiple failures by the IT architecture capabilities of the IT function
  3. 3. IT Architecture Is Failing • It is failing the business − It is not delivering on business strategy and business objectives − It is not helping the business respond to external and internal pressures − It is not providing the consulting and advisory services to enable the business derive value from new technologies − It is not driving IT innovation − It is not making itself relevant or useful to the business • It is failing the IT organisation − It is not assisting with engagement with the business to architect solutions needed by the business − It does not work as an integrated function across all architectural areas − It is not defining IT architectures that enable a portfolio of solutions to be delivered and operated quickly − It is not innovating the IT portfolio and architecture to take advantage of and integrate new technologies 20 May 2019 3
  4. 4. Shadow IT Is The Symptom And Consequence Of IT Architecture Failures • Shadow IT – business diverting IT expenditures outside the IT function • The business bypasses what they view and experience as an unresponsive central IT organisation and goes directly to external service providers − Business shift to cloud service providers offering infrastructure- less solutions with no perceived IT involvement − Business need to respond to the interrelated developments of digital, mobile and social computing and perceived inability of the central IT function to respond − Outsourcing and the divestment of IT functions in response to business wishes to remove the overhead 20 May 2019 4
  5. 5. Consequences Of Failing IT Architecture Function • Inability to rapidly respond to challenges driven by business changes • Lack of commonality and consistency due to the absence of standards • Lack of focus on enterprise requirements • Lack of common direction and savings due to synergies • Incomplete visibility of the current and future target enterprise architecture vision • Inability to predict impacts of future changes • Increased gaps and architecture conflicts • Dilution and dissipation of critical information and knowledge of the deployed solutions • Rigidity, redundancy and lack of scalability and flexibility in the deployed solutions • Lack of integration, compatibility and interoperability between applications • Complex, fragile and costly interfaces between applications • Fragmented and ad hoc solution delivery driven by a tactical and reactive approach 20 May 2019 5
  6. 6. What Is Meant By IT Architecture? • IT Architecture roles and skills should concerned with: − The definition of solution implementation and operation frameworks and standards across the range of the IT landscape − The translation of business strategy and business objectives into the design and operation of required IT solutions − Planning, designing and assisting with the delivery of portfolio of IT systems and solutions to meet the needs of the organisation − The design and implementation of IT frameworks to enable IT solutions be acquired, implemented and moved to operation quickly − The design systems and processes to ensure the security of information and systems − The design and implementation of data frameworks to allow the comprehensive management of data across systems May 20, 2019 6 Business Objectives Business Operational Model Solution Portfolio Realisation And Delivery Solution Usage, Management, Support And Operations Business Strategy Business IT Strategy Solution Portfolio Design And Specification • The IT architecture functions should play a key role in ensuring this alignment and continuity from concept to achievement
  7. 7. IT Architecture Function And Disciplines • IT architecture should comprise the logical set of functional areas and sets of skills required within the IT function to achieve business and IT alignment and the successfully delivery of IT solutions all working together • It is not just about individual disciplines such as Enterprise Architecture • IT architecture is the sum of the individual disciplines 20 May 2019 7 IT Architecture Enterprise Architecture Application Architecture Business Architecture Solution Architecture Informationand DataArchitecture Security Architecture Technical Architecture Infrastructure Architecture Service Architecture
  8. 8. IT Architecture Disciplines – Need To Work Together To Create An Effective Business Solution Delivery Environment May 20, 2019 8 Enterprise Architecture – defines, develops, extends and manages the implementation and operation of the overall IT delivery and operation framework including standards and solution development and acquisition Application Architecture – defines application architectures including development, sourcing, deployment and integration Business Architecture – defines and manages the implementation of IT solutions and related organisation changes needed to implement business strategy and objectives Solution Architecture – designing and overseeing the implementation of a portfolio of IT solutions that translate business needs into operable and usable systems that comply with standards Service Architecture – designing and overseeing the implementation of service processes and supporting technologies and systems to ensure the successful operations of IT solutions including outsourced supplier management framework Security Architecture – designing data and system security processes and systems to ensure the security of information and systems across the entire IT landscape Information and Data Architecture – design, define and implement framework to manage information across the entire IT landscape and through its lifecycle Technical Architecture – translating solution designs into technical delivery, acting as a bridge between solution architecture and the delivery function and designing new delivery approaches Infrastructure Architecture – designing application, communication and data infrastructures to operate the portfolio of IT solutions
  9. 9. IT Architecture Operational Reality • Individual architecture disciplines all too frequently operate as inwardly focussed, disintegrated and siloed functions − Limited and poor communications − No overall management − Inconsistent approaches − Deficient or absent cooperation − Often adversarial relationships between disciplines, characterised by infighting − Overall lack of efficiency and effectiveness − Contributes to poor perception of IT by business • Individual architecture practices throw work over the wall at one another • Enterprise architecture function perceives itself as superior to other architectural areas 20 May 2019 9
  10. 10. IT Architecture’s Multiple Failings 20 May 2019 10 All to frequently inwardly focussed, staffed by IT personnel, focussed on IT rather than on the business Demonstrates aspects of groupthink and focalism Too remote from business concerns and not business oriented and focussed Concerned with documenting current IT technology state, standards and processes in detail rather than looking to the future Too dogmatic, rigid and inflexible Focused on compliance, control and government and adherence to rules Obsessed with architecture frameworks, reference models and patterns Overly controlling Reactive Work not linked to performance metrics Speaks the language of technology rather than business Communicates to the business badly, if at all Not concerned with delivery Does not measure its delivery in terms of business benefits realised Slows down rather than accelerates delivery through disproportionate governance
  11. 11. May 20, 2019 11 IT Too Often Fails to Support Business Needs And Changes Effectively • Technology integration is costly, risky and complicated • Information is everywhere but getting access to the right information at the right time is very difficult • The business wants IT to be fast, dynamic and flexible • The business gets IT that is sluggish and rigid • Modifying solutions takes too long and changes are difficult to communicate and implement effectively • Much of IT system and operations expenditure is bloated and fixed where operations run with excess redundant capacity • IT seen as a cost centre and not a source of business value
  12. 12. IT Architecture Failing Relationships 20 May 2019 12 IT Function Business IT Responds and Delivers Slowly Business Want Rapid Response to Need and Changes IT Does Not Understand or Invest in and Develop IT Architecture IT Architecture Does Not Provide Technology Leadership Business Does Not View IT Architecture As Provider of Technology Consulting Services IT Architecture Is Inwardly and Backwardly Focussed Rather Than Being Business Lead IT Architecture
  13. 13. Consequences Of Failing Relationships 20 May 2019 13 IT Function Business IT Responds and Delivers Slowly Business Want Rapid Response to Need and Changes IT Does Not Understand or Invest in and Develop IT ArchitectureIT Architecture Does Not Provide Technology Leadership Business Does Not View IT Architecture As Provider of Technology Consulting Services IT Architecture Is Inwardly and Backwardly Focussed Rather Than Being Business Lead IT Architecture External Service Provider External Service Provider External Service ProviderExternal Service Provider Outsourcing and Divestment of IT Functions Shadow IT Business Shift to External Service Providers Shadow IT Shadow IT Solutions
  14. 14. The Business Context Of Shadow IT • Shadow IT is the sum of all the business responses to unfulfilled requests for IT solutions or failure of IT to engage with business IT needs • It is an entire parallel IT solution universe May 20, 2019 14 End User (DIY) Computing Direct Business Sourcing of Solutions Outsourcing Of IT Services Abandonment Of Solution Need Unresponsive IT Function Business Requests for IT Solutions Them Us
  15. 15. The Wider Context Of Shadow IT • The wider context of Shadow IT is a set of reactions by business functions to an actual or perceived inability or unwillingness of the IT function to respond to business needs for IT solutions − End User Computing – the business develop the solution themselves − Direct Business Sourcing of IT Solutions – the business sources the IT solution from a service provider in an uncontrolled manner, either as a product installed within the organisation or as a service delivered through a hosted product − Outsourcing Of IT Services – the business takes a strategic decision to outsource elements of the internal IT service as a way of dispensing with the need for the internal IT function − Abandonment Of Solution Need – the business need remains latent, unfulfilled and in the shadows May 20, 2019 15
  16. 16. Core Solution Business Processing Stages And Shadow IT • Use of shadow IT solutions occurs routinely at multiple stages throughout the use of business systems, extending and enhancing their functionality or providing features not available or that area easier to use May 20, 2019 16 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12 Extract Data and Analyse Outside Solution Extract and Exchange Data With Other Party Reporting Using Separate Solution Use Separate Tool To Perform Work Extract and Send Data Outside Party Manually Enter Output from External Solution Perform Additional Steps Using Separate Solution Reporting and Analysis Shadow IT Occurs Pervasively Throughout the Use of Core IT Solutions
  17. 17. Core Solution Business Processing Stages And Shadow IT May 20, 2019 17 Step 1 Step 2 Step 3 Step 4 Step 5 Step 6 Step 7 Step 8 Step 9 Step 10 Step 11 Step 12 Extract Data and Analyse Outside Solution Extract and Exchange Data With Other Party Reporting Using Separate Solution Use Separate Tool To Perform Work Extract and Send Data Outside Party Manually Enter Output from External Solution Perform Additional Steps Using Separate Solution Reporting and Analysis • Shadow IT is frequently needed to make up for gaps in core business solutions, supplementing incomplete solutions and providing omitted functionality • Linking business solution together into an operational reality
  18. 18. The Long Long Shadow Of Shadow IT May 20, 2019 18 Shadow IT Shadow Projects Shadow Sourcing Shadow Development Shadow Solutions Shadow Support Arrangements May Give Rise To May Involve May Involve May Be Included In Projects that have never been subject to a formal evaluation and approval process, formally managed and tracked and who success or failure is not recorded Unapproved usage of third party services or product and service suppliers in the business not subject to format evaluation and approval process including costing and quality and that not is formally recorded and tracked Custom development of solutions performed by business personnel or contracted to third-parties not subject to formal design and delivery approaches including testing and quality Solution that comprises an information technology system that is developed or sourced and implemented by business users that is not approved by the IT function and is not part of the organisation's accepted, documented and supported information technology infrastructure portfolio Shadow Data Gives Rise To May Involve Which Require Informal, undocumented, unrecorded, uncosted and untracked arrangements to provide support for a shadow IT solution that typically involves effort by unapproved third parties or by business personnel for whom providing support is not their formal role Uncontrolled copies or extracts of data from formal IT solutions stored outside formal data structures or data generated by shadow IT solutions that may be held separately from formal data structures or that may be partially of completely entered into formal data structures Use And/Or Generate
  19. 19. Types Of Shadow IT Solution • Shadow IT takes many forms and types 1. CUST – customised solution developed by a third-party 2. DEV – personal devices used to access business systems or authenticate access to hosted solutions used for business 3. DIY – end-user computing application developed by the business 4. HOME – organisation data sent to home devices to be worked on 5. MSG – public messaging and data exchange platforms 6. OPEN – open-source software used as a stand-alone solution or incorporated into other solutions 7. OUT – outsourced service solution 8. PROD – software product acquired by the business and implemented on organisation infrastructure 9. PUB – accessing organisation applications and data using public devices or networks 10. STOR – public data storage and exchange platforms 11. SVC – hosted software solution May 20, 2019 19
  20. 20. Shadow IT Landscape May 20, 2019 20 Core IT Solutions On Premises EUC/DIY Solutions On Premises Product Solutions Hosted Product Solutions Outsourced Service Solutions Personal Devices DEV SVC OUT PROD DIY On Premises Third-Party Custom Solutions CUST Open Source Software OPEN Use of Public Networks or Devices PUB Send Data to Home Devices HOME Public Messaging Platforms MSG Within The Organisation Outside The Organisation Public Data Storage and Exchange Platforms STOR
  21. 21. Shadow IT Landscape • The organisational shadow IT landscape is a lot broader than you think or know • Within each type of shadow IT, there are many instances across different business units May 20, 2019 21
  22. 22. State Of Shadow IT – It’s Not Pretty May 20, 2019 22 Spending Decision Making Cloud and Data Knowledge Estimated Spending on Shadow IT: 2013 – 40% of Total1 2017 – 50% of Total2 76% of CIOs Do Not Know Spending on Cloud3 54% of CIOs Do Not Know The Number of Cloud Services Being Used4 The Business Uses 15 Times The Number of Cloud Applications IT Believe They Use12 90% of CIOs Are Bypassed Sometimes By Business in IT Spending7 31% of CIOs Are Routinely Bypassed By Business in IT Spending8 86% of Cloud Applications Represent Unsanctioned Shadow IT9 Only 8 % of Companies Know the Scope of Shadow IT10 58% of CIOs are Worried About the Spiralling Cost of Cloud Sprawl5 1 https://www.forbes.com/sites/tomgroenfeldt/2013/12/02/40-percent-of-it-spending-is-outside-cio-control/ 2 https://www.everestgrp.com/2017-04-eliminate-enterprise-shadow-sherpas-blue-shirts-39459.html/ 3,4,5 https://www.trustmarque.com/wp-content/uploads/2018/03/Cloud_Sprawl_and_Shadow_IT_Trustmarque.pdf 6,11 https://go.nttict.com/the-growth-of-shadow-IT-and-why-many-enterprises-are-now-dependent-on-it.html 7,8 https://www.logicalis.com/news/cios-line-up-to-transform-it-in-response-to-the-shadow-it-phenomenon/ 9 http://pages.ciphercloud.com/rs/ciphercloud/images/CipherCloud-Cloud-Adoption-and-Risk-Report.pdf 10 https://downloads.cloudsecurityalliance.org/initiatives/surveys/capp/Cloud_Adoption_Practices_Priorities_Survey_Final.pdf 12 https://blogs.cisco.com/cloud/shadow-it-rampant-pervasive-and-explosive 80% of Business Decision Makers Believe that Data Stored in Shadow IT is Critical to their Departments6 80% of Business Decision Makers Admit that Employees in their Department Were Using Cloud Services Without the IT Department’s Knowledge11
  23. 23. Breaking The Flow From Business Strategy To IT Solutions May 20, 2019 23 Business Objectives Business Operational Model Solution Portfolio Realisation And Delivery Solution Usage, Management , Support And Operations Business Strategy Business IT Strategy Solution Portfolio Design And Specification Business shadow IT expenditure External Suppliers and Service Providers External Suppliers and Service Providers Business-perceived or actual barriers to solution delivery by internal IT organisation Shadow IT solutions ultimately may be passed to the support function At least 40% of technology spending is diverted from IT Over 30% of CIOs routinely not consulted on IT solution acquisition and expenditure Them Us Them and Us Mentality
  24. 24. Shadow IT – Survey Results • In 2017, the Everest Group estimated that Shadow IT represented 50% of more of the total IT spending of large organisations − https://www.everestgrp.com/2017-04-eliminate-enterprise-shadow-sherpas-blue-shirts-39459.html/ • In 2013, CEB Global (now part of the Gartner Group) estimated that the proportion of IT spending outside the IT function was of the order of 40% − IT function estimated the proportion spent was just 20% − https://www.forbes.com/sites/tomgroenfeldt/2013/12/02/40-percent-of-it-spending-is-outside-cio- control/ • In 2015, Logicalis conducted a survey of over 400 global CIOs - 90% said there were sometimes bypassed the business − 31% of CIOs said they were routinely bypassed when the business was making IT buying decisions − https://www.logicalis.com/news/cios-line-up-to-transform-it-in-response-to-the-shadow-it- phenomenon/ − https://www.logicalis.com/globalassets/group/cio-survey/cio-survey-2015_final3.pdf • Cisco published in 2015 an analysis of cloud application usage that indicated that IT departments estimated their organisations were using 51 cloud services on average while in reality 730 cloud services were being used, a difference of 15 times − https://blogs.cisco.com/cloud/shadow-it-and-the-cio-dilemma − https://blogs.cisco.com/cloud/shadow-it-rampant-pervasive-and-explosive May 20, 2019 24
  25. 25. Shadow IT – Survey Results • Cloud Adoption & Risk Report in North America & Europe - 2014 Trends Published by CipherCloud in February 2015 − http://pages.ciphercloud.com/rs/ciphercloud/images/CipherCloud-Cloud-Adoption-and-Risk- Report.pdf 86% of cloud applications used by enterprises are unsanctioned “Shadow IT” Our study found that enterprises vastly underestimate the extent of Shadow IT cloud applications used by their organizations. Various media sources claim 10% to 50% of cloud applications are not visible to IT. Our statistics show that on average 86% of cloud applications are unsanctioned. For example, a major US enterprise estimated 10–15 file sharing applications were in use, but discovered almost 70. Enterprises Underestimate the Extent of Shadow IT We all know that the use of Shadow IT within businesses is exploding, but few enterprises have been able to accurately assess the extent of the problem. Self- reported surveys of the percent of enterprises using cloud services range from as low as 19% to 50%—clearly ignoring Shadow IT. Other surveys have shown as many as 80% of end-users admitting to using unsanctioned applications, but without any measurements of actual usage. May 20, 2019 25
  26. 26. Shadow IT – Survey Results • Cloud Adoption Practices & Priorities Survey Report Published by the Cloud Security Alliance − https://downloads.cloudsecurityalliance.org/initiatives/surveys/capp/Cloud_A doption_Practices_Priorities_Survey_Final.pdf The survey respondents’ primary concerns about Shadow IT are: Security of corporate data in the cloud (49 percent) Potential compliance violations (25 percent) The ability to enforce policies (19 percent) Redundant services creating inefficiency (8 percent) Only 8 percent of companies know the scope of shadow IT at their organizations, and an overwhelming majority (72 percent) of companies surveyed said they did not know the scope of shadow IT but wanted to know. May 20, 2019 26
  27. 27. Shadow IT – Survey Results • The CIO in 2017 Cloud Sprawl and Shadow IT: Why IT Leaders Need Visibility and Control − https://www.trustmarque.com/wp- content/uploads/2018/03/Cloud_Sprawl_and_Shadow_IT_Trustmarque.pdf 54% of CIOs don’t know how many cloud-based services and individual subscriptions their organisation has. 58% of CIOs are worried about the spiralling cost of cloud sprawl. 76% find it difficult to know how much their organisation is spending on cloud services. 45% don’t feel cloud providers give enough warning on costs incurred. May 20, 2019 27
  28. 28. Shadow IT – Survey Results • NTT Research Report June 2016 Growing Pains in the Cloud II − https://go.nttict.com/the-growth-of-shadow-IT-and-why-many-enterprises-are-now-dependent-on-it.html 78% of business decision makers admit that employees in their department were using cloud services without the IT department’s knowledge 57% of respondents believe that shadow IT is happening in at least half the departments in the enterprise 83% of respondents believe that the use of shadow IT will increase in the next two years 80% of respondents believe that data stored in shadow IT is critical to their departments 83% used free unregulated cloud storage applications for sharing company information 56% do not know where all or some of their data is stored when employees used shadow IT 73% believe their employees are knowingly breaking the rules and compliance when they provision their own cloud services May 20, 2019 28
  29. 29. Shadow IT Parallel Universe • Shadow IT represents an entire parallel IT solution universe whose extent is largely unknown May 20, 2019 29
  30. 30. History Of Shadow IT • Shadow IT has existed since there was a centralised IT function − The original PC was effectively a form of Shadow IT, reacting against the inflexibility, slowness and lack of access to information by providing end-user direct access to information processing facilities • Shadow IT in the form of end-user computing (EUC) – applications typically developed using tools such as Excel and Access – existed long before cloud applications became pervasively available and still continues to exist − These applications are typically developed without any formal analysis, design and testing − They evolve from the simple to the complex and become important to the daily operations of a business function or an organisation − They are contributed to by many people over time − They are not formally supported or documented − The well-proven risks that are associated with these EUC applications are now being transferred to cloud-based Shadow IT applications • There are many reports of substantial losses being attributed to EUC applications, especially Excel May 20, 2019 30
  31. 31. Some Excel Shadow IT Failures May 20, 2019 31 Publication Details Estimated Loss https://www.reuters.co m/article/us-solarcity- lazard-idUSKCN11635K Lazard Ltd (LAZ.N), the investment bank that advised SolarCity Corp SCTY.O on its $2.6 billion sale to Tesla Motors Inc (TSLA.O), made an error in its analysis that discounted the value of the U.S. solar energy company by $400 million, a regulatory filing by Tesla showed on Wednesday. $400 million http://ww2.cfo.com/spr eadsheets/2014/10/spr eadsheet-error-costs- tibco-shareholders- 100m/ Tibco Software shareholders will be getting $100 million less than originally anticipated from the company’s more than $4 billion sale to Vista Equity Partners as a result of a spreadsheet error that overstated Tibco’s equity value. According to a regulatory filing, Goldman Sachs, which is advising Tibco on the deal, used the spreadsheet in calculating that Tibco’s implied equity value was about $4.2 billion. The merger agreement, reflecting that number, was announced Sept. 29. $100 million http://calleam.com/WT PF/?p=5517 In an incident that drew worldwide attention, J.P. Morgan lost billions of dollars in the so called “London Whale” incident. The London Whale was a trader based in J.P. Morgan’s London Chief Investment Office (CIO). He had earned his nickname because of the magnitude of the trading bets he was making. It is said that his bets were so large his actions alone could move a market. Despite his undeniable power, things went seriously wrong between Apr and Jun 2012 and a poorly positioned trade resulted in losses that eventually totalled up into the billions of dollars. According to available reports, the part of the CIO office involved was responsible for managing the bank’s financial risk using complex financial hedging strategies in the derivatives markets. To support the operations J.P. Morgan had developed a “Synthetic Credit Value at Risk (VaR) Model” that helped them understand the level of risk they were exposed to and hence make decisions about what trades they should be making and when. The tool had been developed in-house in 2011 and was built using a series of Excel spreadsheets. According to J.P. Morgan’s own report to their shareholders that was published following the disaster, the spreadsheets “had to be completed manually, by a process of copying and pasting data from one spreadsheet to another”. Approximately $6B https://www.sec.gov/n ews/press/2011/2011- 37.htm Feb. 3, 2011 – The Securities and Exchange Commission today charged three AXA Rosenberg entities with securities fraud for concealing a significant error in the computer code of the quantitative investment model that they use to manage client assets. The error caused $217 million in investor losses. AXA Rosenberg Group LLC (ARG), AXA Rosenberg Investment Management LLC (ARIM), and Barr Rosenberg Research Center LLC (BRRC) have agreed to settle the SEC's charges by paying $217 million to harmed clients plus a $25 million penalty, and hiring an independent consultant with expertise in quantitative investment techniques who will review disclosures and enhance the role of compliance personnel. $232 million https://www.theglobea ndmail.com/report-on- business/human-error- costs-transalta-24- million-on-contract- bids/article18285651/ A slip of the hand in a computer spreadsheet for bidding on electricity transmission contracts in New York will cost TransAlta Corp. $24-million (U.S.), wiping out 10 per cent of the company's profit this year. $24 million
  32. 32. Excel Shadow IT • There are many other Excel-based Shadow IT example of major problems − Just search for “Excel Horror Stories” • Many companies have suffered and continue to suffer very substantial financial losses due to errors and misuse of computer applications, mainly Excel-based, developed by end users • Chartis Research produced in July 2016 an analysis of the risks of such EUC applications to financial services organisations − http://www.clusterseven.com/wp-content/uploads/2016/07/Quantification-of- EUC-Risk-Final.pdf Chartis estimates that the current End User Computing (EUC) Value at Risk (VaR) for the largest 50 FIs (Financial Institutions) is $12.1 billion (at a confidence interval of 97.5%, over a one-year period). The estimated annual average VaR for large FIs is $285 million per institution. The results of our methodology applied to publicly disclosed loss events gave an estimate of the VaR that large FIs are exposed to, though it does not take into account secondary effects such as regulatory fines, reputational damage, loss of customers etc. Chartis believes there is a strong qualitative argument that the potential secondary impact of EUC risk is significantly larger than the direct losses covered in this paper. May 20, 2019 32
  33. 33. Shadow IT – Learning From History • It may simply a matter of time before a similar set of stories regarding EUC applications such as Excel to emerge for cloud-based applications • The EUC Shadow IT problem has not been resolved • So the cloud application Shadow IT problem may not also be resolved easily. • The IT architecture functions seek to minimise both its use and the likelihood and impact of problems by engaging with the business earlier to identify the need for solutions • Today’s shadow IT will be the source of tomorrow’s problems May 20, 2019 33
  34. 34. Shadow IT Solutions Are Often Incomplete • Commonly they are tactical point solutions • Components omitted rendering the solution incomplete • Incompleteness will manifest itself over time May 20, 2019 34
  35. 35. Scope Of Complete Solution May 20, 2019 35 Changes to Existing Systems New Custom Developed Applications Information Storage Facilities Acquired and Customised Software Products System Integrations/Data Transfers/Exchanges New Business Processes Organisational Changes Reporting and Analysis Facilities Existing Data Conversions/Migrations Changes to Existing Business Processes New Data Loads Training and Documentation Central, Distributed and Communications Infrastructure Application Hosting and Management Services Cutover/Transfer to Production Parallel Runs Enhanced Support/Hypercare Sets of Maintenance, Service Management and Support Services Operational Functions and Processes Sets of Installation and Implementation Services Complete Solution Consists Of The Delivery Of A Set Of Components Scope of Complete Solution From Design To Operations
  36. 36. Gaps In Shadow IT Solutions May 20, 2019 36 Changes to Existing Systems New Custom Developed Applications Information Storage Facilities Acquired and Customised Software Products System Integrations/Data Transfers/Exchanges New Business Processes Organisational Changes Reporting and Analysis Facilities Existing Data Conversions/Migrations Changes to Existing Business Processes New Data Loads Training and Documentation Central, Distributed and Communications Infrastructure Application Hosting and Management Services Cutover/Transfer to Production Parallel Runs Enhanced Support/Hypercare Sets of Maintenance, Service Management and Support Services Operational Functions and Processes Sets of Installation and Implementation Services Shadow IT Solutions Rarely Encompass The Full Scope Of A Solution Scope of Complete Solution From Design To Operations
  37. 37. The Evolution And Trajectory Of Shadow IT Solutions May 20, 2019 37 It Makes Our Job So Much Easier Shadow IT Solution Is a Great Idea The People Who Developed It Are Rock Stars It Will Make Up For Functionality Not Available The Solution Is Difficult To Maintain, Support And Operate The People Who Developed It Move On Solution Support Becomes Patchy And Problematic The Solution Is Integrated Into IT Support The Solution Is Falls Into Disuse Users Become Dissatisfied With The Solution The Solution Is Redeveloped And Implemented In Production Basic Processes Are Implemented Around The Solution Information On The Use Of Solution Becomes Difficult To Obtain The Solution Persists Data Integration Is Complex The Solution Is Out Of Date And No Longer Fit For Purpose
  38. 38. Why Does Shadow IT Continue To Happen? • Missing or insufficient budget, resources or knowledge in the IT function • Local business implementation is (seen as) easier and faster • Cultural differences between business and IT • Business lacks information about the range of IT services and costs • Poor experience with IT projects or changes leading to lack of trust • Shadow IT starts as a small implementation of a prototype • Business adopts shadow IT to gain control or be autonomous • The business has gotten into the habit of implementing solutions locally • Business personnel are familiar with the technology • There are no controls or sanctions preventing shadow IT • The business can acquire shadow IT solutions easily without the need for IT involvement May 20, 2019 38
  39. 39. Why Shadow IT Arises – Business View And Experience Of IT May 20, 2019 39 Shadow IT Business and IT Misalignment Cost, Ease and Speed Power, Control and Ownership Behaviour Perceived or actual lack of alignment of IT and its direction and the IT solution requirements needs of the business or poor level of maturity in relationship between IT and the business Valid or invalid assumptions about the time, cost, resources required and complexity to create a formal IT solution when compared to an independent solution Desire by business function to be independent of IT or to (re)gain control and be the owners of the delivery of their IT solutions Staff are used to developing their own solutions, have the skills and experience or are familiar with the technologies being used or shadow IT evolves from locally-developed prototypes
  40. 40. Multiple Factors Contributing To Shadow IT •IT takes too long to respond to business requests •IT does not (or is perceived not to) listen to the needs of the business •IT function is difficult to engage with, is poor at relationship management or does not have an effective engagement model •IT does not implement the technologies required by the business •The business function has had previous poor experiences with the IT function •The IT function does not have the resources, skills and experience to address the business need •The business makes invalid assumptions about the difficulties of engaging with the IT function IT and Business Misalignment •IT function is too expensive at solution delivery and operation •The existing solutions do not provide the required facilities or they re too difficult to use •IT cannot develop prototypes sufficiently quickly •IT function is too slow and/or frequently late to deliver and does not react and deliver solutions quickly •IT function imposes too many controls on solution delivery •It is easier for the business function to source the solution outside the IT function •The business makes invalid assumptions about the time and cost of solution delivery by the IT function Cost, Ease and Speed •The business function wants to be independent of the IT function •The business function has the authority to source and implement local IT solutions •The business function is perceived as being difficult to work with and its uncontrolled sourcing of IT solutions is tolerated •The business function wants to be in control of the selection of its IT solutions •The business function has sufficient power to source solutions without the approval of the IT function Power, Control and Ownership •Personnel working in the business function have experience of developing or sourcing solutions outside IT control •Personnel working in the business function have skills and experience with the desired technologiesBehaviour May 20, 2019 40
  41. 41. Multiple Factors Contributing To Shadow IT • IT takes too long to respond to business requests • IT does not (or is perceived not to) listen to the needs of the business • IT function is difficult to engage with or is poor at relationship management or does not have an effective engagement model • IT does not implement the technolgies required by the business • The business function has had previous poor experiences with the IT function • The IT function does not have the resources, skills and experience to address the business need • The business makes invalid assumptions about the difficulties of engaging with the IT function IT and Business Misalignment • IT function is too expensive at solution delivery and operation • The existing solutions do not provide the required facilities or they re too difficult to use • IT function is too slow and/or frequently late to deliver and does not react and deliver solutions quickly • IT function imposes too many controls on solution delivery • It is easier for the business function to source the solution outside the IT function • The business makes invalid assumptions about the time and cost of solution delivery by the IT function Cost, Ease and Speed • The business function wants to be independent of the IT function • The business function has the authority to source and implement local IT solutions • The business function is perceived as being difficult to work with and its uncontrolled sourcing of IT solutions is tolerated • The business function wants to be in control of the selection of its IT solutions • The business function has sufficient power to source solutions without the approval of the IT function Power, Control and Ownership • Personnel working in the business function have experience of developing or sourcing solutions outside IT control • Personnel working in the business function have skills and experience with the desired technologies Behaviour May 20, 2019 41 Business Decision on Solution Fulfilment +-+ - +- +-+ - +-- - - +-+ - --- - + - Shadow IT Solution No Solution IT Provided Solution
  42. 42. Extent Of Shadow IT • Extent of shadow IT can vary from business acquiring point solutions to entire business-lead parallel autonomous IT solution acquisition and delivery process • Extent of the penetration and shadow IT not known, by its very nature • Technology-literate workforce increases the propensity of shadow IT to occur • Pervasive availability of cloud-based consumer and quasi- business applications lead to greater shadow IT May 20, 2019 42
  43. 43. Vendors And Shadow IT • Solution and service vendors love shadow IT, especially cloud- delivered solutions • They can sell services directly to business users without financial or functional due diligence or compliance with central IT standards • No requirements for formal integration to central IT solutions • Shorter sales cycle • No formal acquisition and due diligence process • No formal cost benefit analysis • No formal solution delivery process and associated controls • Opaque cost model frequently hides real long-term costs • Subscription-based pricing means predictable recurring revenue • Cloud-based enables offsite service delivery, reducing costs and increasing margin May 20, 2019 43
  44. 44. Multiple Factors Contributing To Shadow IT • There are many factors that contribute to the implementation of shadow IT solutions • Business will consciously or unconsciously evaluate these factors to make or justify a solution-sourcing decision • This has implications for the IT function − Better business engagement model especially for early engagement − Provide greater clarity on solution delivery approach to business − Most cost-effective, flexible and timely solution delivery including faster prototyping − Shared solution sourcing approach − Clearly articulate the risks of shadow IT to the business May 20, 2019 44
  45. 45. Wider Shadow Causal And Enabling Factors • Shadow IT happens when causal and enabling factors are greater than the barriers created by limitations and controls to shadow IT implementation • Barriers fail to hold back the latent demand from the business for solutions that meet their needs May 20, 2019 45 Business and IT Misalignment Cost, Ease and Speed Power, Control and Ownership Behaviour CausalandInfluencing Factors No Need to Involve IT Function Low Barriers to Use (Cost, Technical) Availability of Options User Skills and Experience EnablingFactors Policies, Standards, Education and Awareness User Understanding Financial Controls Preventative MeasuresLimitationsandControls Excess of Causing and Enabling Factors = Shadow IT Overspill
  46. 46. Wider Shadow IT Equation • Shadow IT has advantages and disadvantages − Advantages tend to the short-term − Disadvantages and increase accumulate over time • Not all factors have the same importance for all shadow IT solutions and business units and organisations • Factors are not constant over time − Disadvantages can grow and advantages can reduce over time May 20, 2019 46 Business and IT Misalignment Cost, Ease and Speed Power, Control and Ownership Behaviour CausalandInfluencing Factors No Need to Involve IT Function Low Barriers to Use (Cost, Technical) Availability of Options User Skills and Experience EnablingFactors Policies, Standards, Education and Awareness User Understanding Financial Controls Preventative Measures LimitationsandControls Employee Empowerment and Satisfaction Cost Savings of New Solution Delivery Greater Innovation Greater Productivity and Efficiency AdvantagesandBenefits New Solution Available More Quickly Application and Data Integration Problems Regulatory and Compliance Risks Security Risks Loss of Productivity and Efficiency DisadvantagesandLosses Data Redundancy, Proliferation and Risks Lack of Visibility and Ownership Ongoing Support and Maintenance + - = - Sum of Causal and Preventative Factors Advantages and Disadvantages
  47. 47. Wider Shadow IT Equation • The profile of the net causal, enabling and preventative factors leading to shadow IT and the balance of advantages over disadvantages will be different for each organisation May 20, 2019 47
  48. 48. Shadow IT And Solution Delivery Failure • Shadow IT solution delivery is regularly not subject to controls during implementation and operation − Financial management − Change management − Release management and transfer to production − Support model − Data quality − Knowledge management − Capacity planning and capacity management • Frequently implemented locally and in an ad hoc, disorganised and fragmented manner by individuals who subsequently move on − Solution knowledge is lost and solution operation becomes increasingly difficult May 20, 2019 48
  49. 49. Shadow IT Solution – Frequent Challenges Shadow IT Solution Issues Details Solution Architecture and Design • The underlying solution technology may not be sufficient • The solution may be implemented in obsolete technology • The underlying database and its data model may not be enforce data quality • The solution may not be scalable to handle required volumes of data, users or workload • The solution may not be extendable to provide additional functionality Implementation Standards • The solution may not be implemented and fully tested • The solution may not be reliable Documentation and Training • The solution may not be supplied with adequate documentation • There may not be adequate training in the use of the solution Data Standards and Quality • The data loaded into the solution is not accurate • The solution may not maintain data quality Solution Supplier • The supplier of the solution may go out of business or may no longer provide or support the solution Key Personnel • Key personnel involved in the design and implementation may move from the business function Operation and Use • The solution may be slow to use • The operation of the solution may be manually intensive Processing • The results generated by the solution may not be accurate Support • The support arrangements for the solution may not be sufficient • The underlying technology in which the solution was implemented may Technology Upgrades • The solution may not be supported due to technology upgrades Organisation Change • The solution may no longer be appropriate because of organisation changes Technology Initiatives • The solution may be rendered obsolete by new solutions or technology initiatives May 20, 2019 49
  50. 50. Technical Debt And Shadow IT • Technical debt is the sum of the differences between the current IT solution state and the desired target state • It represents the implied amount of work and its associated cost required to achieve the desired target state • Shadow IT increases the amount of the overall organisation’s technical debt • The size of this additional technical debt is not known May 20, 2019 50
  51. 51. Shadow IT Impact Assessment Approach May 20, 2019 51 Assessing Shadow IT Significance Strategic Importance Operational Security of IT Assets Internal Compliance External Compliance Business Processes Service Operations and Management Cost Quality Solution Quality Design Development and Implementation Solution Infrastructure Data Structures Integration Security Operations Data and Information Extent Effectiveness, Efficiency, Utility User Population Resources Consumed Replacement of Existing Core IT Solution(s) Potential to Incorporate into Core IT = Assessment Factor
  52. 52. Shadow IT Impact Assessment Approach • Assessment is difficult because the extent of shadow IT is unknown • Need to understand the impact of the problem as one input to defining a realistic and achievable resolution • The scoring of any assessment in inexact and informal • The individual factors are not independent − A poorly designed solution will have poor quality data and will require disproportionate resources to manage • The factors can be weighted to reflect their relative importance − For example, Strategic Importance of a shadow IT solution has a higher impact that Infrastructure • Different types of shadow IT solution will have different impact factor profiles − PROD and SVC type solutions will (presumably) have high Operational and Quality characteristics and thus low IT and organisational impacts May 20, 2019 52
  53. 53. Shadow IT Impact Assessment Factors Impact Assessment Factor Details Strategic Importance How does the use of shadow IT and the solutions implemented affect the organisation’s IT strategy? Does the use of shadow IT destabilise the overall IT strategy? Do the shadow IT solutions perform strategic business functions? What is the business value provided? Operational - Security of IT Assets Will problems due to unreliability of and errors in shadow IT solutions have the potential to affect the security of IT assets including data? Operational - Internal Compliance Will problems due to unreliability of and errors in shadow IT solutions have the potential to affect compliance with internal standards? Operational - External Compliance Will problems due to unreliability of and errors in shadow IT solutions have the potential to affect compliance with external regulations, directives and legislation? Operational - Business Processes Will problems due to unreliability of and errors in shadow IT solutions have the potential to affect the operation of business processes and the delivery of the associated services? Operational - Service Operations and Management Will problems due to unreliability of and errors in shadow IT solutions have the potential to affect how the solutions are supported, operated and managed? Operational – Cost How much do the solutions cost to operate, maintain and support? Quality - Solution Quality – Design What was the quality of the design of the solution and how will or could it impact on the solution? Quality - Solution Quality - Development and Implementation What was the quality of the development and implementation of the solution and how will or could it impact on the solution? Quality - Solution Quality - Solution What is the quality of the overall solution? Quality - Solution Quality - Infrastructure What is the quality of the infrastructure on which the solution operates and how will or could it impact on the solution? Quality - Solution Quality - Data Structures What is the quality of the data structures of the solution and how will or could they impact on the solution? Quality - Solution Quality - Integration What is the quality of the integration of the solution with other solution and how will or could it impact on the solution? How are the integrations achieved? Are they automated or manual? Are they secure? Quality - Solution Quality - Security What is the quality of the security controls and operation of the solution and how will or could they impact on the solution? Quality - Operations How effectively does the solution operate and implement the underlying business processes? Are there many manual or replicated steps and data redundancy? Can the solution be administered, managed and supported? Quality - Data and Information What is the quality of the data held in and generated by the solution? Extent - Effectiveness, Efficiency, Utility How many shadow IT solutions are being used? Do the shadow IT solutions duplicate one another or production solutions? How efficient are the solutions Extent - User Population How many users are using the shadow IT solutions? Extent - Resources Consumed What resources are needed to support, administer, manage and operate the shadow IT solutions? Replacement of Existing Core IT Solution(s) Can or should the shadow IT solution replace their comparable existing authorised solutions? Potential to Incorporate into Core IT Do the shadow IT solutions represent or incorporate innovative functions that should be adopted by the organisation and the IT function? May 20, 2019 53
  54. 54. Assess Shadow IT Across The Organisation • Assessment should cover the dimensions of the range of Shadow IT solutions across all business functions within the organisation • Assessment can be used to understand extent of Shadow IT solutions and make decisions on their future and the development of a long-term approach May 20, 2019 54 Range of Shadow IT Solutions Business Functions
  55. 55. Assess Shadow IT Across The Organisation • The assessment approach can be rolled-up from individual shadow IT solutions through business functions to create an organisation-wide view and assessment May 20, 2019 55 Rolled -up View
  56. 56. Addressing The Issue Of Shadow IT • Use assessment framework to decide on approach to shadow IT solutions 1. Renew – integrate into IT function, possibly enhance, redevelop or acquire 2. Productionise – transfer ownership and incorporate into IT operations and support 3. Accept and Monitor – know, categorise, accept and tolerate with controls 4. Stop – stop using and replace with alternative (existing) formal solution(s) or process(es) May 20, 2019 56
  57. 57. Making Decisions On The Future Of Shadow IT Solutions May 20, 2019 57 Strategic Significance/ Importance Operational Impact Replace Existing Business Solutions Solution Quality Characteristics Size, Extent, Effectiveness, Efficiency, Utility Potential To Incorporate Into Formal Business Solution Landscape RENEW STOP PRODUCTIONISE ACCEPT
  58. 58. Parallel Activity To Deciding On Current Shadow IT Solutions – Long-Term Approach To Shadow IT • In parallel to assessing the state of shadow IT and making decisions on the future of existing solutions, the IT function can take other actions on the long-term approach to shadow IT • Long-term approach needs to define when shadow IT is permissible • Define and implement security and risk control framework • Provide a controlled and secure (set of) platform(s) for shadow IT May 20, 2019 58
  59. 59. Long-Term Approach To Shadow IT Long-Term Approach To Shadow IT Definition Define Policies, Guidelines and Standards Define Education Approach And Collateral Identify And Resolve Gaps In Existing Central It Solutions That Give Rise To Shadow It Solutions Define Business Engagement Model To Understand And Seek To Address Business Needs At An Early Stage Define Control Framework Education Publish Policies Create Awareness Implementation and Operation Implement Security And Control Framework To Prevent Risks Allow The Use Of Some Types Of Shadow It Solutions Implement Business Engagement Approach Maintain and Update Policies Continuous Education May 20, 2019 59
  60. 60. Extended Shadow IT Model Within Organisations May 20, 2019 60 Causal and Influencing Factors Enabling Factors Limitations And Controls Advantages And Benefits Disadvantages And Losses Risk and Impact Assessment Framework Decisions on Existing Shadow IT Long-Term Approach To Shadow IT Give Rise to Shadow IT Stop Or Inhibit Give Rise To Shadow IT That Has That Has Balance Of Advantages and Disadvantages May Change Over Time Scope and Impact Can Be Understood By Allows Informed Decisions To Be Made Contributes To The Creation OfContributes To The Creation Of Gives Rise To Affects Shadow IT
  61. 61. Extended Shadow IT Model Within Organisations May 20, 2019 61 Disadvantages And Losses Shadow IT Risk and Impact Assessment Framework Causal and Influencing Factors Enabling Factors Limitations And Controls Advantages And Benefits Decisions on Existing Shadow IT Long-Term Approach To Shadow IT
  62. 62. Extended Shadow IT Model Within Organisations • The extended shadow IT model can be used as a framework to comprehensively evaluate, understand and create a long-term vision and solution May 20, 2019 62
  63. 63. Shadow IT And Productivity • Business caught between loss of productivity due to the absence of the desired solutions or the loss of productivity due to having to use transfer data between multiple separate solutions • Initial productivity gains from shadow IT can diminish over time • Shadow IT solutions supported within the business functions − Uncosted unplanned peer support • Accumulating backlog of solutions that have to be brought into formal support and/or need to migrate shadow IT solution and its data to a supported platform May 20, 2019 63
  64. 64. May 20, 2019 64 Shadow IT And Productivity • Short term productivity gains • Long-term productivity gap
  65. 65. Shadow IT And Innovation • Business-lead IT solutions can represent innovative ways to do business, work smarter, add value and achieve results − Improve employee experience and empowers employees • Shadow IT represents latent demand for solutions not being provided by the IT function − Represents an insight into what the IT solutions the business need • The IT function needs to engage with the business to encourage innovative solution ideas and bring them into formal IT support earlier − Early engagement approach - https://www.slideshare.net/alanmcsweeney/tthe-need-for-effective- early-engagement-in-solution-architecture-and-design − Rapid solution scoping offering - https://www.slideshare.net/alanmcsweeney/solution-architecture- approach-to-rapidly-scoping-the-initial-solution-options May 20, 2019 65
  66. 66. Shadow IT Risks • Organisation data is stored outside the central knowledge and control • Bypassing data backup and recovery/business continuity/archival/retention/deletion policies • Uncertain security, intrusion detection and access control − Security breaches may not be detected or may have happened for some time before being identified • Outside the scope of regulatory standards, compliance, audit and eDiscovery − Data breaches caused by shadow IT will occur and will cost companies money − There will be penalties, audits, lost revenue, brand damage, security remediation and costs • Uncontrolled shadow copies of data, not unsynchronised with main sources, used for reporting, analysis and decision-making • Supplier processes and solution architectures may not suit the data security requirements • Suppliers may go out of business May 20, 2019 66
  67. 67. IT Architecture Showing Leadership • Shadow IT gives IT architecture the opportunity show leadership • Develop model for IT as a solution and service broker − Service Oriented IT – SOIT • IT architecture can be the gateway for business IT solution requirements May 20, 2019 67
  68. 68. Summary • Uncontrolled shadow IT represents a real risk to organisations • The experience from previous shadow IT examples is that they have resulted in real financial losses • IT architecture can and should take the lead in implementing structures and processes to mitigate risks while taking maximising the benefits of shadow IT May 20, 2019 68
  69. 69. More Information Alan McSweeney http://ie.linkedin.com/in/alanmcsweeney https://www.amazon.com/dp/1797567616 20 May 2019 69

×