Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Leading from the Front
 Accelerating Data Integration through Metadata
  Scott Abbott
  Certified IT Architect, InfoSphere...
Context
                       C t t




IBM Insight Forum 09
IBM Insight Forum 09
   2                   Make change work...
Are you
                                                     e
                                                  constantl...
Often it’s
                                                  because we
                                                  ...
Typical Data Integration Project                                                            REPORTS


                    ...
The I f S h
       Th InfoSphere Software Evolution
                     S ft     E l ti




                             ...
InfoSphere Information Server




IBM Insight Forum 09    Make change work for you
                                       ...
Typical Data Integration Project                                                         REPORTS


                       ...
Pitfall
                            Pitf ll #1
                       “The C t
                       “Th Custom Model”
  ...
DI Pitfall #1
                                                            WAREHOUSE




                                  ...
DI Pitfall #1
  Accelerator
       80:20 rule (20% customization)
              Months not years


    Fully attributed da...
Typical Data Integration Project                                                         REPORTS


                       ...
Pitfall
                         Pitf ll #2
                         if we build it
                               b ild
 ...
14
                                                        DI Pitfall #2
                                                 ...
15
     Missing the Point
     Corporate Chi
     C       t Chinese Whi
                       Whispers

      Identify Hi...
16
     Bridging the Gap
     relating the new to the old
       l ti th        t th ld

                                 ...
IBM Insight Forum 09
  26                   Make change work for you
                                                  ®
IBM Insight Forum 09
  29                   Make change work for you
                                                  ®
Understanding Your D t
 U d t di Y         Data

                                                  InfoSphere
            ...
InfoSphere Business Glossary
        Web-based authoring, managing and
        sharing of business metadata
        Aligns...
Business Glossary Anywhere                                                         ANY
                                   ...
Typical Data Integration Project                                                               REPORTS


                 ...
Pitfall
                        Pitf ll #3
                       data
                       d t quality
                ...
DI Pitfall #3
                                                                      LEGACY
                               ...
IBM Insight Forum 09
  38                   Make change work for you
                                                  ®
IBM Insight Forum 09
  39                   Make change work for you
                                                  ®
IBM Insight Forum 09
  40                   Make change work for you
                                                  ®
IBM Insight Forum 09
  41                   Make change work for you
                                                  ®
IBM Insight Forum 09
  42                   Make change work for you
                                                  ®
IBM Insight Forum 09
  43                   Make change work for you
                                                  ®
IBM Insight Forum 09
  44                   Make change work for you
                                                  ®
IBM Insight Forum 09
  45                   Make change work for you
                                                  ®
IBM Insight Forum 09
  46                   Make change work for you
                                                  ®
IBM Insight Forum 09
  47                   Make change work for you
                                                  ®
IBM Insight Forum 09
  48                   Make change work for you
                                                  ®
IBM Insight Forum 09
  49                   Make change work for you
                                                  ®
IBM Insight Forum 09
  50                   Make change work for you
                                                  ®
IBM Insight Forum 09
  51                   Make change work for you
                                                  ®
IBM Insight Forum 09
  52                   Make change work for you
                                                  ®
IBM Insight Forum 09
  53                   Make change work for you
                                                  ®
IBM Insight Forum 09
  54                   Make change work for you
                                                  ®
IBM Insight Forum 09
  55                   Make change work for you
                                                  ®
IBM Insight Forum 09
  56                   Make change work for you
                                                  ®
IBM Insight Forum 09
  57                   Make change work for you
                                                  ®
IBM Insight Forum 09
  58                   Make change work for you
                                                  ®
IBM Insight Forum 09
  59                   Make change work for you
                                                  ®
InfoSphere Information Analyzer


      Data-centric analysis of application,
                                            ...
Typical Data Integration Project                                                           REPORTS


                     ...
Pitfall
                        Pitf ll #4
                         Iterative
                         It   ti
           ...
DI Pitfall #4


                                                                 3          DATA INTEGRATION




         ...
Where does the
       How d I Find Out …
       H   do Fi d O t                                          data for this
   ...
Pitfall
                          Pitf ll #4
                         Development
                         D   l       t
 ...
IBM Insight Forum 09
  80                   Make change work for you
                                                  ®
What is the InfoSphere Metadata Workbench?
 Web-based exploration of
 Information Assets generated and
                   ...
Typical Data Integration Project                                                           REPORTS


                     ...
Pitfall
                           Pitf ll #4
                         Development
                         D     l       ...
Typical Data Integration Project                                                             REPORTS


                   ...
InfoSphere FastTrack
To reduce costs of integration projects through automation

 Business analysts and IT
 collaborate in...
Typical Data Integration Project                                                             REPORTS


                   ...
93
     Information Server
     Optimizing A li ti D
     O ti i i Application Development
                              l...
94
     IBM InfoSphere Information Server
     Delivering information you can trust
                                      ...
95
     Bringing It All Together
         g g           g



           Business      Subject Matter   Architects         ...
Leading from the Front
     Greater Preparation will yield dramatically lower
     project costs/times

        Typical Wo...
97




                       Thank
                       Th k you


                       Questions?




IBM Insight Fo...
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata
Upcoming SlideShare
Loading in …5
×

InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata

1,782 views

Published on

InfoSphere - Leading from the Front - Accelerating Data Integration through Metadata. Presenter: Scott Abbott

Published in: Technology
  • Be the first to comment

InfoSphere: Leading from the Front - Accelerating Data Integration through Metadata

  1. 1. Leading from the Front Accelerating Data Integration through Metadata Scott Abbott Certified IT Architect, InfoSphere Software IBM Insight Forum 09 Make change work for you ®
  2. 2. Context C t t IBM Insight Forum 09 IBM Insight Forum 09 2 Make change work for you ® ®
  3. 3. Are you e constantly disappointed by your Data Integration I t ti projects? IBM Insight Forum 09 Make change work for you ®
  4. 4. Often it’s because we rush in without thinking what we are doing d i IBM Insight Forum 09 Make change work for you ®
  5. 5. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS REFERENCE DATA “if we build it they will come” MASTER DATA “The custom data model” “of course our “we’ll work it out data is good” in the testing” IBM Insight Forum 09 Make change work for you ®
  6. 6. The I f S h Th InfoSphere Software Evolution S ft E l ti DataMirror Change Data Ch D t Capture LAS Global Name Enrichment DWL Unicorn Operational Master Data Management Ascential Metadata Management SRD Transformation, Cleansing, Trigo Profiling and metadata integration Entity Resolution and Product Information Analysis Management IBM Insight Forum 09 Make change work for you ®
  7. 7. InfoSphere Information Server IBM Insight Forum 09 Make change work for you ®
  8. 8. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS REFERENCE DATA MASTER DATA METADATA IBM Insight Forum 09 Make change work for you ®
  9. 9. Pitfall Pitf ll #1 “The C t “Th Custom Model” M d l” IBM Insight Forum 09 IBM Insight Forum 09 9 Make change work for you ® ®
  10. 10. DI Pitfall #1 WAREHOUSE 1 “The custom data model model” NZ Customer Experience “who k “ h knows our industry i d • Project duration 24-36 mths better than us” • Model never fully deployed • Complex ETL feeds destabilized ti d t bili d entire BI system t “it will only take a couple of • Users bypass to get required months” information IBM Insight Forum 09 Make change work for you ®
  11. 11. DI Pitfall #1 Accelerator 80:20 rule (20% customization) Months not years Fully attributed data models across six industries Complete b i C l t business t templates f l t for industry KPIs Key Ke accelerators for migration & integration projects Act A t as acceleration t l ti templates within l t ithi Information Server & Cognos 8 BI IBM Insight Forum 09 Make change work for you ®
  12. 12. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 industry models LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS REFERENCE DATA MASTER DATA Target state METADATA IBM Insight Forum 09 Make change work for you ®
  13. 13. Pitfall Pitf ll #2 if we build it b ild they will come.. y IBM Insight Forum 09 IBM Insight Forum 09 13 Make change work for you ® ®
  14. 14. 14 DI Pitfall #2 REPORTS OLAP 4 “if we build it they will come” “it is what the business NZ Customer Experience asked for” • Multiple examples of BI solutions not meeting initial business drivers “the users will understand •UUsers perceive new BI i initiatives as burdens rather the new system” than assets IBM Insight Forum 09 Make change work for you ®
  15. 15. 15 Missing the Point Corporate Chi C t Chinese Whi Whispers Identify High Value Monthly Report on Customers to support Customers Revenue Call Centre & Web breakdown Personalization Business Subject Matter Architects Data Developers DBAs Users Experts Analysts IBM Insight Forum 09 Make change work for you ®
  16. 16. 16 Bridging the Gap relating the new to the old l ti th t th ld “item” “component” ? “part” ? IBM Insight Forum 09 Make change work for you ®
  17. 17. IBM Insight Forum 09 26 Make change work for you ®
  18. 18. IBM Insight Forum 09 29 Make change work for you ®
  19. 19. Understanding Your D t U d t di Y Data InfoSphere Business Glossary Captures Business Taxonomies Captures and defines shared searchable business glossary Assigns stewardship to key business terms Links business terms to technical assets IBM Insight Forum 09 Make change work for you ®
  20. 20. InfoSphere Business Glossary Web-based authoring, managing and sharing of business metadata Aligns the efforts of IT with the goals Subject Matter Business of the business Experts Users Provides business context to InfoSphere Business Gl I f S h B i Glossary information technology assets Establishes responsibility and Create and manage business vocabulary and relationships, while accountability y linking to physical sources Database = DB2 GL Account Number Schema = NAACCT The ten digit account number. Table = Sometimes DLYTRANS referred to as Technical Business Column = C l the th account ID. t ID ACCT_NO This value is of the form L- data type = FIIIIVVVV. Business View char(11) IBM Insight Forum 09 Make change work for you ®
  21. 21. Business Glossary Anywhere ANY User Real-time access to business glossary from any desktop application Features From Any From any desktop application, click on a term & Application.. view its business definition in a pop-up window . without any loss of context or focus Intelligent matching returns best candidates in a I t lli t t hi t b t did t i single search Search engine for terms and categories Access steward contact information directly Security enforced via the Information Server common security layer Benefits Increased trust and acceptance of information by delivering definitions in context Expanded adoption of enterprise glossary outside of Information Platform technologies Pop the Improved information availability with multiple access mechanisms for electronically stored information (ESI) Definition!
  22. 22. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 Correct 2 3 DATA INTEGRATION DATAMARTS Understood REFERENCE DATA Data Steward MASTER DATA Terms Target state METADATA IBM Insight Forum 09 Make change work for you ®
  23. 23. Pitfall Pitf ll #3 data d t quality lit IBM Insight Forum 09 IBM Insight Forum 09 36 Make change work for you ® ®
  24. 24. DI Pitfall #3 LEGACY SOURCES 2 “of course our data is good” NZ Customer Experience “the b i “ h business owner says the h • ETL Proof of Concept • Client assured data quality sufficient so information we need is in there” excluded data cleansing from scope • At end of 2wk pilot, project halted due to unsolvable data quality issues q y “the schema’s show they • Many 15-20 year old systems still in operation in NZ market have the same keys” IBM Insight Forum 09 Make change work for you ®
  25. 25. IBM Insight Forum 09 38 Make change work for you ®
  26. 26. IBM Insight Forum 09 39 Make change work for you ®
  27. 27. IBM Insight Forum 09 40 Make change work for you ®
  28. 28. IBM Insight Forum 09 41 Make change work for you ®
  29. 29. IBM Insight Forum 09 42 Make change work for you ®
  30. 30. IBM Insight Forum 09 43 Make change work for you ®
  31. 31. IBM Insight Forum 09 44 Make change work for you ®
  32. 32. IBM Insight Forum 09 45 Make change work for you ®
  33. 33. IBM Insight Forum 09 46 Make change work for you ®
  34. 34. IBM Insight Forum 09 47 Make change work for you ®
  35. 35. IBM Insight Forum 09 48 Make change work for you ®
  36. 36. IBM Insight Forum 09 49 Make change work for you ®
  37. 37. IBM Insight Forum 09 50 Make change work for you ®
  38. 38. IBM Insight Forum 09 51 Make change work for you ®
  39. 39. IBM Insight Forum 09 52 Make change work for you ®
  40. 40. IBM Insight Forum 09 53 Make change work for you ®
  41. 41. IBM Insight Forum 09 54 Make change work for you ®
  42. 42. IBM Insight Forum 09 55 Make change work for you ®
  43. 43. IBM Insight Forum 09 56 Make change work for you ®
  44. 44. IBM Insight Forum 09 57 Make change work for you ®
  45. 45. IBM Insight Forum 09 58 Make change work for you ®
  46. 46. IBM Insight Forum 09 59 Make change work for you ®
  47. 47. InfoSphere Information Analyzer Data-centric analysis of application, Subject Matter Data database and file-based sources Experts Analysts InfoSphere Information Analyzer Secure, detailed profiling of fields, across fields, and across sources Analyse source data structures, and monitor adherence to integration and quality rules lit l Creation of metadata from profiling results Results instantly promotable across IBM InfoSphere Information Server Physical View IBM Insight Forum 09 Make change work for you ®
  48. 48. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS Correct REFERENCE DATA Understood Data Steward MASTER DATA Terms Target ETL Source state Hints State METADATA IBM Insight Forum 09 Make change work for you ®
  49. 49. Pitfall Pitf ll #4 Iterative It ti Development p IBM Insight Forum 09 IBM Insight Forum 09 62 Make change work for you ® ®
  50. 50. DI Pitfall #4 3 DATA INTEGRATION “we’ll work it out in the testing” NZ Customer Experience • ETL development >75% total project $$ • Projects t ki P j t taking 2-3x l 2 3 longer th planned than l d • Some clients taking 70+% of dev.time doing impact analysis • Impact analysis methods very basic • Largely iterative development method • Unreliable forecast completion dates • Low levels of trust by business in IT ability to achieve BI outcomes • Substantial cost overruns • Expensive BI maintenance costs IBM Insight Forum 09 Make change work for you ®
  51. 51. Where does the How d I Find Out … H do Fi d O t data for this report come Data Analyst from? …where this data comes from? … when the job had been running last time? … the details for these assets? IBM Insight Forum 09 Make change work for you ®
  52. 52. Pitfall Pitf ll #4 Development D l t (Impact Analysis) ( p y ) IBM Insight Forum 09 IBM Insight Forum 09 65 Make change work for you ® ®
  53. 53. IBM Insight Forum 09 80 Make change work for you ®
  54. 54. What is the InfoSphere Metadata Workbench? Web-based exploration of Information Assets generated and g used by Information Server applications Out of the box reporting on data p g Data Developers Integration I t ti movement, data lineage, Managers business meaning, impact of InfoSphere Metadata Workbench® changes and dependencies Provides IT professionals with a tool for Tracing the data lineage of exploring and understanding the assets generated and used by the Information Business Intelligence Reports to Server suite. provide basis for compliance with legislation such as S Sarbanes- Oxley and Basel II
  55. 55. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS Correct REFERENCE DATA Understood Data Steward MASTER DATA Impact Terms Analysis Target ETL Source state Hints State METADATA IBM Insight Forum 09 Make change work for you ®
  56. 56. Pitfall Pitf ll #4 Development D l t (Iterative cycles) ( y ) IBM Insight Forum 09 IBM Insight Forum 09 89 Make change work for you ® ®
  57. 57. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS Correct Requirements REFERENCE DATA Understood ETL Code Data Generation Steward MASTER DATA Impact Terms Analysis Target ETL Source state Hints State METADATA IBM Insight Forum 09 Make change work for you ®
  58. 58. InfoSphere FastTrack To reduce costs of integration projects through automation Business analysts and IT collaborate in context to create project specification Leverages source analysis, analysis target models, and metadata to facilitate Specification mapping process Auto-generation of data transformation j jobs and reports p Auto-generates DataStage jobs Flexible Reporting
  59. 59. Typical Data Integration Project REPORTS OLAP WAREHOUSE 4 LEGACY SOURCES 1 2 3 DATA INTEGRATION DATAMARTS Correct Requirements REFERENCE DATA Understood ETL Code Data Generation Steward MASTER DATA Impact Terms Analysis Target ETL Source state Hints State METADATA IBM Insight Forum 09 Make change work for you ®
  60. 60. 93 Information Server Optimizing A li ti D O ti i i Application Development l t IBM Insight Forum 09 Make change work for you ®
  61. 61. 94 IBM InfoSphere Information Server Delivering information you can trust Information S I f ti Server InfoSphere Information Services Director InfoSphere Information Analyzer InfoSphere Business Glossary InfoSphere Federation Server InfoSphere QualityStage InfoSphere DataStage InfoSphere Data Architect InfoSphere Replication Server / EVP InfoSphere FastTrack InfoSphere Change Data Capture InfoSphere Metadata Server InfoSphere Metadata Workbench IBM Insight Forum 09 Make change work for you ®
  62. 62. 95 Bringing It All Together g g g Business Subject Matter Architects Data Developers DBAs Users Experts Analysts Information Server – Common Framework Simplify Integration Increase trust and confidence in information Facilitate h F ilit t change Increase compliance to I li t Design Operational management & reuse standards IBM Insight Forum 09 Make change work for you ®
  63. 63. Leading from the Front Greater Preparation will yield dramatically lower project costs/times Typical Work Effort for Migration Activities 15-30% of total project budget will be spent on Migration Activities 15-30% of total p j 15 30% g p g project budget will be spent on Migration Activities Discover Prepare Deliver 30% 40% 30% Understanding Cleaning, Standardising Conversion, Loading, Source Data Harmonizing, Management Interfaces, Connectivity This effort is the most unpredictable. The work can vary 50% Business greatly depending on condition of data, however it is 25% Business Coding transformations and loads. 75% Business Largely manual effort on small always the largest piece of work in the data initiative. Traditionally this effort is plagued with problems related to data quality and it Largely manual effort on 100% of data. This can mean percentage of data. Some manual can easily be pulled by necessity into the dozens of persons cleaning source systems manually t d f l i t ll to coding can review all data . 50% IT correct and augment data and manually aligning records 75% IT Cleaning, Standardising and Harmonising 25% IT to MRD. Some manual coding can reduce the manual area causing timing and budget problems. effort. IBM Insight Forum 09 Make change work for you ®
  64. 64. 97 Thank Th k you Questions? IBM Insight Forum 09 Make change work for you ®

×