Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Réseau des Professionnels de la                              Business Intelligence                               en Suisse...
AGENDA   14h00   Welcome   14h30   Le groupe LinkedIn   14h45   Les métadonnées de chargement   15h30   Coffee Break ...
AGENDALe groupe LinkedIn             Dario Mangano             Head Of Knowledge Management             Nestlé Nespresso S...
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
Le groupe LinkedIn
AGENDALa gestion des métadonnées de         chargement                   Anthony Brouard                   Business Intell...
Métadonnées de chargement
Métadonnées de chargement  ANTHONY’s Slides
Data Integration Framework                                    Overview                  Da t a M a n a g e me n t S e r v ic
Data Integration Best Practices orientedData integration is a family of techniques, most commonly including ETL (extract, ...
Metadata Management orientedMetadata is a key feature in Data Integration and Data Warehousing.This is the only way to get...
Data Integration Framework (DIF): Data Mngmt Framework                 D IF c o m p o n e n t s                           ...
Designs methods and tools to perform dataintegration services R e fe r e n c e                               M e tho      ...
DIF: Back end modular architecture                                                                                        ...
Potential for Global Monitoring Applicationenvironments                                             Shared Metadata Reposi...
Reporting services (Cognos/BO reports) – Ex1Using the reporting layer we can have access to the integrated metadata reposi...
Reporting services (Cognos/BO reports) - Ex2The value added of having integrated metadata, is to have report showing corre...
Reporting services (Cognos/BO reports) - Ex3Another example of the details we can get from the reports.Using publication m...
Appendix           26
Data Integration Application     Architecture                                                                             ...
Monitoring Implementation                                                                      1s t L e v e l             ...
AGENDALe chargement par fuseaux         horaires     Cedric Zbinden          Dario Mangano     BI architect            Hea...
Le charement par fuseaux            horairesQuestion: Comment gérer la cohérence  et l’intégrité des données du DWH lorsqu...
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires
Le charement par fuseaux        horaires Débat – propositions ?
Le charement par fuseaux                 horairesRésumé des discussions:-Le HQ et les marchés n’ont pas les mêmes besoins ...
AGENDAFutures tables rondes
Futures Tables Rondes- Sujets- Lieux- Communication
Futures Tables RondesPropositions:-Big Data-Démonstration d’un POC ClickView (groupe Mutuelle)-Exemple de governance perme...
MERCI Café !          
Linked in 4eme table ronde 20120601
Upcoming SlideShare
Loading in …5
×

Linked in 4eme table ronde 20120601

1,060 views

Published on

Linked in 4eme table ronde 20120601
- Metedatamanagement (ETL)
- Load by TimeZone

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Linked in 4eme table ronde 20120601

  1. 1. Réseau des Professionnels de la Business Intelligence en Suisse Romande 4ème Table Ronde Lausanne, le 1er juin 2012Dario ManganoHead Of Knowledge ManagementNestlé Nespresso S.A. HQ 
  2. 2. AGENDA 14h00 Welcome 14h30 Le groupe LinkedIn 14h45 Les métadonnées de chargement 15h30 Coffee Break 15h45 Le chargement par fuseaux horaires 16h30 Futures tables rondes 16h45 Coffee 17h30 Fin Lausanne, le 1er juin 2012
  3. 3. AGENDALe groupe LinkedIn Dario Mangano Head Of Knowledge Management Nestlé Nespresso S.A.
  4. 4. Le groupe LinkedIn
  5. 5. Le groupe LinkedIn
  6. 6. Le groupe LinkedIn
  7. 7. Le groupe LinkedIn
  8. 8. Le groupe LinkedIn
  9. 9. Le groupe LinkedIn
  10. 10. Le groupe LinkedIn
  11. 11. Le groupe LinkedIn
  12. 12. AGENDALa gestion des métadonnées de chargement Anthony Brouard Business Intelligence Expert Capital International
  13. 13. Métadonnées de chargement
  14. 14. Métadonnées de chargement ANTHONY’s Slides
  15. 15. Data Integration Framework Overview Da t a M a n a g e me n t S e r v ic
  16. 16. Data Integration Best Practices orientedData integration is a family of techniques, most commonly including ETL (extract, transform, and load),but also lots of related techniques that are inevitable when dealing with Data Integration: Metadata,Change Data Capture, File loading, Publication, Data quality,... moreover it is always involving differenttechnologies: DB server, DB scripting, Shell scripting, …All these techniques and technologies require development and support for a wide range of interfacesusing solution that can be hand-coded, based on vendor’s tool , or mix of both.With such complexity in Data Integration systems, to develop and support these solutions is becomingvery challenging.Having Best Practices and standards will ensure that all the systems are developed in a way that it ismuch easier to support and also much safer and scalable to afford future needs and data volume.The Data Integration Framework is a metadata driven development environment that is providing turnkey solutions for all these tasks around Data Integration: - Metadata - Change Data Capture - File loading - Data quality 17
  17. 17. Metadata Management orientedMetadata is a key feature in Data Integration and Data Warehousing.This is the only way to get answers to the following questions: - Which column did this data come from? - When was this data populated in the system? - How is calculated this result on my report? - Is my report up to date? - Is my system scalable?Having these answers will just increase the trust in the data, enable a pro active monitoring of DataIntegration processes, ensure that the data are loaded in a effective manner and at the end preventour system to lose value over the time by decreasing and absorbing the costs of understanding,maintenance and repair.The Data Integration Framework is providing a metadata management solution without anydevelopment effort required from the project team:  Collecting operational metadata in real time  Capturing business and technical metadata related to data integration processes 18  Integrating all these metadata in a metadata repository
  18. 18. Data Integration Framework (DIF): Data Mngmt Framework D IF c o m p o n e n t s R o g rvices ep rtin Metadata se Support Teams monitor Monitoring D ta p blication a pu lica n ub tio C an e D ta ile ad g ha ge a Archiving rch g F loa in File lo din Business a ture ap re A ivin C ptu h ng Source systems Rules Downstream systems Notification Graphical User Interface develop Use develop (see slide notes for comments) 19 Development Teams
  19. 19. Designs methods and tools to perform dataintegration services R e fe r e n c e M e tho D IF A r c h it e c t u r e ds C omponE v e n t -T r ig g e r e d E T L E T L D e v e lo p m e n t W r atp p e r en s M e t h o d o lo g y B a tc h E TL F ile L o a d e r P a r s in g , M a t c h in g & S t a n d a r d In t e g r a t io n C ha ng e D a ta C a p ture M e r g in g , M e t h o d s b y S u b je c t A r e a R e je c t M a n a g e m e n t C o n s o lid a t io n … Q u a lit y A u d it in g D a t a Q u a lit y C o n t r o l M e t h o d s D Q m o d u le P u b /S u b E v e n t B u l k P u b /S u b P a t t e r n P u b lis h e r M o d u le O p . & T e c h n ic a l M e ta d a ta M e ta d a ta d a ta mo d e l M e ta d a ta Ma na g e me nt C o lle c t io n (Operational & Technical) M e t a d a t a c o lle c t io n d a e m o n S ta nd a rd s 20
  20. 20. DIF: Back end modular architecture Metadata Sheduler Wrapper module Repository FileLoader module Publisher module Archiver module Purge module Data Quality module Notification module Project specific code to apply business rules and requirements (Powercenter, Shell script, Sql script, Store procedures,…) Re-usable inlcudes (logging routines, mail sending routines,…) DIF Minimal installation DIF available Modules/Services/Re- HP OV Metrics collection services usable components Powercenter Metrics collection services 21
  21. 21. Potential for Global Monitoring Applicationenvironments Shared Metadata Repository APP1 Project Team us e Data access layer Support Team Reusable components - Autosys wrapper Middleware Team e erver Reports - File loader APP2 use - CDC Metadata W bS - Rejects recycling Repository - Archiver - Publisher - ... Publication Extract External system e us Engines APP3 Retrieve key metadata from infrastructure and middleware components Application process Unix HP PowerCenter Scheduler Oracle Dbs Oracle Dbs servers Openview 22
  22. 22. Reporting services (Cognos/BO reports) – Ex1Using the reporting layer we can have access to the integrated metadata repository for all kind of reportor ad hoc query: - monitoring report - capacity planning - impact analysisExample of monitoring report with embedded navigation capabilities: Dril down button Open log file for more details 23
  23. 23. Reporting services (Cognos/BO reports) - Ex2The value added of having integrated metadata, is to have report showing correlated metadata on thesame view.For example this Gantt view execution report, will show if there is a correlation between a given interfaceexecution and server workload. Drill through thisThis is very useful to understandto interface step issue, but also for capacity planning purposes…. performance interface details Drill down report Gantt view for this interface 24
  24. 24. Reporting services (Cognos/BO reports) - Ex3Another example of the details we can get from the reports.Using publication module, the metadata will tell you what are the XML files that were produced, howmany rows were extracted from the database,..And also to which downstream applications the package was pushed to: 25
  25. 25. Appendix 26
  26. 26. Data Integration Application Architecture Reporting services Operational Metadata Repository Cognos configuration Exception Metadata Metrics Business Metadata Logs Objects Level 2 Supprt DEV Team (L3 support) Business Users Data Integration Framework Services (modules , monitoring services ) Data Change data Rejection Exception Data Quality Notification Archiving Auditing Workflow Publication Movement capture recycle MgmtInterface ETL ETL Task(s) Task(s) ETL ETL Task(s) Task(s) Flat File FileLoader Publisher Source Task(s) Script Script Task(s) system Task(s) Task(s) Data PublicationPublication Script Staging Integration Script Layer Layer Oracle Layer DWH DB Table(s) Layer Oracle store proc store proc Oracle Oracle store proc store proc Sql script Sql script Sql script Sql script Step (s) Step(s) Step(s) Step(s) Scheduler Data Flow 27
  27. 27. Monitoring Implementation 1s t L e v e l DB S e r v e r s S upport Application/service level P o we r C e n t e r A p p l ic a t io n & view enables Service We b S e r v e r s Desk to rapidly intercept fatal alerts & communicate serviceSc hedul DI F M o d u l e s a n d outages to affected er users S e r v ic e s 2 nd L e v e l S upport Identify root cause of Re p o r t s & Da s h b o a r d s issue & take effective Me t a d a t a action. Data is available R e p o s it o r y for analysis to anticipate Service-L evel V iew of the D ata I ntegration A pplication issues and bottlenecks. 28 rd
  28. 28. AGENDALe chargement par fuseaux horaires Cedric Zbinden Dario Mangano BI architect Head Of Knowledge Management Nestlé Nespresso S.A. Nestlé Nespresso S.A.
  29. 29. Le charement par fuseaux horairesQuestion: Comment gérer la cohérence et l’intégrité des données du DWH lorsque les données sont chargées par zones géographiques et par fuseaux horaires ?
  30. 30. Le charement par fuseaux horaires
  31. 31. Le charement par fuseaux horaires
  32. 32. Le charement par fuseaux horaires
  33. 33. Le charement par fuseaux horaires
  34. 34. Le charement par fuseaux horaires
  35. 35. Le charement par fuseaux horaires
  36. 36. Le charement par fuseaux horaires
  37. 37. Le charement par fuseaux horaires
  38. 38. Le charement par fuseaux horaires
  39. 39. Le charement par fuseaux horaires Débat – propositions ?
  40. 40. Le charement par fuseaux horairesRésumé des discussions:-Le HQ et les marchés n’ont pas les mêmes besoins en terme derafraicissmeent de données revoir si le HQ peut se satisfaire de J-2 ?-Utiliser le chargement de schémas différents afin de ne pas requêter leschémas qui est en train d’être lo9ader, puis faire un drop partition à lafin ?-Utiliser les master cubes Cognos
  41. 41. AGENDAFutures tables rondes
  42. 42. Futures Tables Rondes- Sujets- Lieux- Communication
  43. 43. Futures Tables RondesPropositions:-Big Data-Démonstration d’un POC ClickView (groupe Mutuelle)-Exemple de governance permettant de mieux cadrer les demandes desbusinesses (Business Case, Demand management committee, etc.)-In Memory appliances-BI Mobile-Column based DB / NoSQL-Data Virtualization-BI SaaS
  44. 44. MERCI Café ! 

×