I4 e subra hoysala_highmark_practical challenges in dw architecture cases


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

I4 e subra hoysala_highmark_practical challenges in dw architecture cases

  1. 1. Practical life challenges inimplementing data warehousePresented bySubramanya Hoysala17th June 2011
  2. 2. Agenda Overview  My Experience  Basics – in my own way! Challenges  Planning and implementing new data warehouse  Modifying existing data warehouse Merger and Integration challenges Questions
  3. 3. My Data warehouse experience Creation of new Oracle data warehouse for Variable Life Financial application (VL app) Federation of Oracle data on to a DB2 Data Mart. Recommendation and Implementation of Data Warehouse and ETL for Cash and Custody Application (C&CA)
  4. 4. Data Mart, Warehouse, ODS etc Data Mart: Consists of data that is used by single or few Business entities within the organization Data Warehouse: Consists of the interrelated data that may belong to multiple Business entities within the organization Operation Data Stores: Consists of the interrelated data but for a particular operation Operational Database: Existing databases tuned for faster insert/delete operations
  5. 5. How do you start? Functional Requirements  What data user needs Non Functional Requirements  How data will be used (Performance, Security, Maintenance, future use etc) Participants needed  Subject Matter experts, DBA, Data Architects, and PM
  6. 6. Creation of Oracle DW for VL App(New Data Warehouse Creation) Long process  Determine data elements used in existing application  Tabulate interrelation and enterprise meaning of data element – Metadata creation  Model database and apply non functional requirements  Finalize database  Plan for phased deployment Challenges  Focus – Large Scope, Broad Vision document  Keeping the Momentum – Lack of Management push, Other priorities, dwindling funds
  7. 7. Federation of Oracle data(Modifying Existing DW) Application dependent strategy  Analyze data needed for Claims system  Recommend data model for faster Claims processing  Implement real-time update from Oracle to DB2 Challenges  Consideration of data inconsistency between two databases  Lack of understanding of data usage from other applications
  8. 8. DW & ETL for Cash & Custody App(Merger and Integration – Creation of New DW & Modifying Existing DW) Recommendation  Analyze the existing databases, processes and applications  Recommend the solution for combined entities  Compare the future state with current  Work with Stakeholders to push for the solution  POC for the tool selected Challenges  Non functional requirements (Initial load, performance etc)  Different priorities (Lack of SME), changes in historic data requirements
  9. 9. Specific to Merger & Integration Understand the dynamics –need to analyze multiple options (not only the best options) Collaborate – Work with individual units separately and push them towards single solution Evaluate existing tools – Learn existing applications and tools Management Support – understand the management needs and recommend solutions
  10. 10. Appendix(DW & ETL for Cash and Custody)
  11. 11. Appendix - ETL Process facts(DW & ETL for Cash and Custody)• Loading New warehouse is architected using real-time data integration technology – Informatica Real Time (RT).• Informatica CDC solution and Informatica Data Integration (DI) solutions• Informatica Power Exchange implemented on multiple DB2 databases, coupling tightly with DB2 logs.• The Informatica CDC components track the changes operational data warehouse for cash, custody, FX rates and pricing, Corporate Action, Tax Reclaims.• Any change in data is passed to Informatica PowerCenter RT components.• The PowerCenter performs business v alidations on the data and loads to DW• Batch Informatica processes are scheduled using ESP scheduler
  12. 12. Appendix(DW & ETL for Cash and Custody) Non Functional Requirements  SLA of 30 seconds to report on 10 accounts  SLA of 12 Seconds of data consolidation, transformation and migration from source to target for real time reporting and presentation  Loading historical data volume (about 320 million records)
  13. 13. Contact Details Subramanya Hoysala  Email: shoysala@gmail.com  Phone: +1-412-512-7321