The Changing Data Quality & Data Governance Landscape
Upcoming SlideShare
Loading in...5
×
 

The Changing Data Quality & Data Governance Landscape

on

  • 1,349 views

 

Statistics

Views

Total Views
1,349
Views on SlideShare
1,306
Embed Views
43

Actions

Likes
0
Downloads
23
Comments
0

3 Embeds 43

https://twitter.com 38
http://www.scoop.it 3
http://tweetedtimes.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

The Changing Data Quality & Data Governance Landscape The Changing Data Quality & Data Governance Landscape Presentation Transcript

  • Be Certain, Be Trillium CertainThe Changing Data Quality &Data Governance Landscapea survival guide for data governance & data qualityprofessionalsTrillium Software webinar – Wednesday 12 DecemberNigel Turner, VP Information Management Strategy
  • The traditional DQ & Data GovernanceLandscape?2 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • The future DQ & Data GovernanceLandscape?© Copyright 2012, Trillium Software, Inc. All rights reserved.3
  • The changing landscape:potential disruptive eruptionsBIGDATACLOUDCOMPUTINGDATAVIRTUALIZATION4 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Disruptive eruption 1 –Big Data5 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – what is it?Set of new concepts, practices & technologies tomanage & exploit digital dataCan be defined as:“Data that exceeds the processing capability of conventionaldatabase systems. The data is too big, moves too fast, ordoesn’t fit the strictures of your database architecture”(Source: Ed Dumbill – O’Reilly Community)Its key premise is that all data has potential value if itcan be collected, analysed and used to generateactionable insight6 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • The characteristics of Big Data - the 3Vs• Reflects exponential growth of data – predicted 40-60% perannum• Today 2.5 quintillion bytes of data are created every day• 90% of all digital data was created in the last two years• Data generated more varied and complex than before:– Text, Audio, Images, Machine Generated etc.• Much of this data is semi-structured or unstructured• Traditional IT techniques ill equipped to process & analyse it• Data often generated in real time• Analysis and response needs to be rapid, often also real time• Traditional BI / DW environments becoming obsolescent –new approaches are needed7 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • What’s different about Big Data?New technologies which enable distributed & highlyscalable MPP (Massively Parallel Processing), e.g.Apache HadoopMapReduceNoSQL databasesStrong emphasis on analytical approachesEmergence of “data science”Predictive AnalyticsData MiningThe “democratisation” of dataData made available to all (cf Cloud Computing)Business and not IT led BI8 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Where does Big Data come from?SOCIALMEDIA &SOCIALNETWORKSMACHINEGENERATEDWIDELY KNOWNSOURCES9 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – Foundations of SuccessIdentifying the right data to solve the business problemor opportunityThe ability to integrate & match varied data from multipledata sourcesstructured, semi-structured, unstructuredBuilding the right IT infrastructure to support Big DataapplicationsHaving the right capabilities & skills to exploit the data10 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – Barriers & PitfallsThe sheer volume of data – what’s worth using?Data extraction challengesThe ability to match data from disparate sources /formats / mediaThe time taken to integrate new data sourcesThe risks of mismatching and incorrect identification ofindividualsLegal & regulatory pitfallsSecurity concerns – corporate & individualLack of skills & expertise11 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – the data integration challengeSOCIALMEDIASENSORSCSDATAEMAILMOBILESEXTERNALDATASOURCESINTERNALDATASOURCESCRMBILLINGOPSSALESPRODSANALYTICS PLATFORM 1ANALYTICS PLATFORM 2ANALYTICS PLATFORM 3ANALYTICS PLATFORM nACTIONABLE INSIGHT& KNOWLEDGE12 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – DQ as the key enablerSOCIALMEDIASENSORSCSDATAEMAILEXTERNALDATASOURCESINTERNALDATASOURCESCRMBILLINGOPSSALESPRODSANALYTICS PLATFORM 1ANALYTICS PLATFORM 2ANALYTICS PLATFORM 3ANALYTICS PLATFORM nACTIONABLE INSIGHT& KNOWLEDGEPROFILEPARSESTANDARDISEMATCHENRICHDATA QUALITY PLATFORMPROFILEPARSESTANDARDISEMATCHENRICHMOBILES13 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Big Data – the DG & DQ impact• Big Data will depend on dataquality to reap its claimedbenefits – the GIGO truism• The democratization of datawill expose poor DQ• The need for DataGovernance increases asdata becomes moreaccessible• Data skills will become morevalued for ‘data science’• Big Data will increase the3Vs of data• Control of data becomesmore difficult – scope andvariety of use increases• Data standards & businessrules become more complex• Potential legal & regulatoryminefield14 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Disruptive eruption 2 –Cloud Computing15 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing – Alternative Definitions“Cloud computing is the delivery of computing as aservice rather than a product, whereby sharedresources, software, and information are provided tocomputers and other devices as a metered service overa network (typically the Internet).” (Wikipedia)“Marketing term for the technologies that providecomputation, software, data access, and storageservices that do not require end-user knowledge of thephysical location or configuration of the system thatdelivers the services.” (Trillium Software)16 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing – the Wikipedia view17 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing – Key ElementsProvision of services via the Internet / networkVirtual not physical allocation of resourcesMulti-tenanted hostingPay as you use - not outright purchase (cf utilities)Cloud is a disruptive technology as it provides a clearalternative model to outright purchase of hardware,platforms & applications1818 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Types of clouds & servicesPublic/private/hybrid optionsPublic – via the internetPrivate – via an intranetHybrid – combinationCloud servicesInfrastructure as a service (IaaS)Platform as a service (PaaS)Software as a service (SaaS)et al (XaaS)19 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing: potential benefits (1)Speed to deploy new applications & servicesGreater standardisationScalability & elasticityLower initial implementation costs – CAPEX to OPEXBetter cost control and lower internal IT costs (e.g.help desks)20 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing: potential benefits (2)Benefits to SMEs who cannot afford to purchaseTry before you buy options – benefits bothcustomers & suppliersSelf-service and self-configuration of servicesBetter and faster user adoptionPotentially improved performanceAutomatic data back ups21 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing –barriers & risksDATADATASECURITYSECURITY& PRIVACY& PRIVACYCONCERNSCONCERNSCOMMERCIALCOMMERCIAL& OPERATIONAL& OPERATIONALFACTORSFACTORSAPPLICATIONAPPLICATION& DATA& DATAINTEGRATIONINTEGRATIONCHALLENGESCHALLENGESLEGAL &LEGAL &REGULATORYREGULATORYRESTRICTIONSRESTRICTIONS22 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Preparing data for migration• Scoping and scaling data to be migrated• Evaluating its suitability for integration with other data sources• Undertaking source data rationalization & cleanseMigrating to the cloud environment• Profiling data in advance of data migration• Enhancing data in preparation for migration• Maintaining DQ during ETL processesManaging data in the cloud• Enforcing business rules to be applied in the Cloud environment• Auditing data to ensure security, adherence and quality• Supporting data governance activitiesCloud – the role of DQ & DG23 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Cloud Computing – the DG / DQ impact• DQ / DG will be key toCloud migration success –before, during and aftermigration• Internal and external dataintegration will become key• Could improve DQ as fewerdevices will hold data• DQ host and applicationcompanies may offerDQaaS• Cloud will require anenhanced focus on datagovernance – within andoutside the enterprise• Organisations may losephysical control of data• DQ SLAs will be neededwith data hosts / suppliers• Legal & regulatorycompliance becomes amajor challenge24 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Disruptive eruption 3 –Data Virtualization25 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Data virtualization – a simple view26 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Data Virtualization – a less simple view27 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Data virtualization – the essentialsData is held in a variety of internal and external sources (e.g.DBMS, DW, Excel etc.)A middleware layer sits above the data sourcesCreates a virtual view at run time and creates temporarytables in a dedicated serverProcesses, assembles and presents the data to the applicationlayer / deviceBenefits claimed:Hides complexity from usersFlexibilitySpeed - as data can be cached in memory28 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Data virtualization – the DG / DQ impact• Will put the focus on DQ & datastandardisation as a keyenabler to DV interoperability• To work will require thedeployment of both real timeand batch DQ capability• Will require a Shared BusinessVocabulary (SBV) for shareddata model and data standardsacross an organisation• Need for better DQ in sourcesystems to enable run timeintegration• Data is physically held in awide variety of sources somakes coherent DataGovernance more difficult• Data at source will be used formultiple applications socommon business rules harderto agree• Run time integration requiresreal time DQ – manyorganisations do not have thiscapability29 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • The potential eruptions…DATAVIRTUALIZATIONBIGDATACLOUDCOMPUTING30 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • So what’s the impact of all this on DQ /DG practitioners?New DataQuality & DataGovernancechallengesWhat do weneed to do?Changing DQand DG roles& skills31 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • New DQ & Data Governance challengesPREDOMINANTLYBATCH DQCUSTOMERORGANISATIONFOCUSPROCEDURALFOCUS MAINLYWITHINTHE ENTERPRISETHE TRADITIONALLANDSCAPESUPPLIERORGANISATIONFOCUSPREDOMINANTLYREAL TIME DQGROWING FOCUSOUTSIDETHE ENTERPRISECOMMERCIALTHE CHANGINGLANDSCAPE32 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • Changing DQ and DG rolesDQ and Data Governance roles will become more ‘beyondorganisation’ facing – into hosting companies, data &application suppliers etc.Many data management and DQ specialists will work with orevolve into data scientistsDQ and DG people will need to enhance their understandingof global legal and regulatory environmentsCommercial and negotiation skills will become moreimportant33 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • What action should we take?Identify and get involved in any current or planned Big Data,Cloud or Data Virtualization initiatives within ourorganisationsEnsure that the DQ and DG implications & imperatives ofthese initiatives are understoodParticipate in any due diligence of potential third partyvendors & providersPlan for the new DQ and DG challenges that these trends willpose34 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • The changing landscapeBetter DQ needs to be achieved in an environment where data willcontinue to increase by 50% per annumThe claimed benefits of Big Data, Cloud & Data Virtualisation cannot beachieved without renewed emphasis on data quality management & datagovernanceData governance becomes increasingly challenging & extends within andoutside the enterpriseDQ services will increasingly be offered as DQaaS by vendors and datahosts, and more DQ / DG roles may be outsourcedAs DQ practitioners we need to understand, educate and get involvedwith those in our organisations who are creating the new landscape35 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • A final thought…“It’s not the will to winbut the will to prepare towin that makes thedifference”Bear Bryant –US Football Coach1913 – 198336 © Copyright 2012, Trillium Software, Inc. All rights reserved.
  • QuestionsContact: nigel.turner@trilliumsoftware.comwww.trilliumsoftware.com37 © Copyright 2012, Trillium Software, Inc. All rights reserved.