• Like
I4 e subra hoysala_highmark_practical challenges in dw architecture cases
Upcoming SlideShare
Loading in...5
×

I4 e subra hoysala_highmark_practical challenges in dw architecture cases

  • 302 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
302
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
5
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Practical life challenges inimplementing data warehousePresented bySubramanya Hoysala17th June 2011
  • 2. Agenda Overview  My Experience  Basics – in my own way! Challenges  Planning and implementing new data warehouse  Modifying existing data warehouse Merger and Integration challenges Questions
  • 3. My Data warehouse experience Creation of new Oracle data warehouse for Variable Life Financial application (VL app) Federation of Oracle data on to a DB2 Data Mart. Recommendation and Implementation of Data Warehouse and ETL for Cash and Custody Application (C&CA)
  • 4. Data Mart, Warehouse, ODS etc Data Mart: Consists of data that is used by single or few Business entities within the organization Data Warehouse: Consists of the interrelated data that may belong to multiple Business entities within the organization Operation Data Stores: Consists of the interrelated data but for a particular operation Operational Database: Existing databases tuned for faster insert/delete operations
  • 5. How do you start? Functional Requirements  What data user needs Non Functional Requirements  How data will be used (Performance, Security, Maintenance, future use etc) Participants needed  Subject Matter experts, DBA, Data Architects, and PM
  • 6. Creation of Oracle DW for VL App(New Data Warehouse Creation) Long process  Determine data elements used in existing application  Tabulate interrelation and enterprise meaning of data element – Metadata creation  Model database and apply non functional requirements  Finalize database  Plan for phased deployment Challenges  Focus – Large Scope, Broad Vision document  Keeping the Momentum – Lack of Management push, Other priorities, dwindling funds
  • 7. Federation of Oracle data(Modifying Existing DW) Application dependent strategy  Analyze data needed for Claims system  Recommend data model for faster Claims processing  Implement real-time update from Oracle to DB2 Challenges  Consideration of data inconsistency between two databases  Lack of understanding of data usage from other applications
  • 8. DW & ETL for Cash & Custody App(Merger and Integration – Creation of New DW & Modifying Existing DW) Recommendation  Analyze the existing databases, processes and applications  Recommend the solution for combined entities  Compare the future state with current  Work with Stakeholders to push for the solution  POC for the tool selected Challenges  Non functional requirements (Initial load, performance etc)  Different priorities (Lack of SME), changes in historic data requirements
  • 9. Specific to Merger & Integration Understand the dynamics –need to analyze multiple options (not only the best options) Collaborate – Work with individual units separately and push them towards single solution Evaluate existing tools – Learn existing applications and tools Management Support – understand the management needs and recommend solutions
  • 10. Appendix(DW & ETL for Cash and Custody)
  • 11. Appendix - ETL Process facts(DW & ETL for Cash and Custody)• Loading New warehouse is architected using real-time data integration technology – Informatica Real Time (RT).• Informatica CDC solution and Informatica Data Integration (DI) solutions• Informatica Power Exchange implemented on multiple DB2 databases, coupling tightly with DB2 logs.• The Informatica CDC components track the changes operational data warehouse for cash, custody, FX rates and pricing, Corporate Action, Tax Reclaims.• Any change in data is passed to Informatica PowerCenter RT components.• The PowerCenter performs business v alidations on the data and loads to DW• Batch Informatica processes are scheduled using ESP scheduler
  • 12. Appendix(DW & ETL for Cash and Custody) Non Functional Requirements  SLA of 30 seconds to report on 10 accounts  SLA of 12 Seconds of data consolidation, transformation and migration from source to target for real time reporting and presentation  Loading historical data volume (about 320 million records)
  • 13. Contact Details Subramanya Hoysala  Email: shoysala@gmail.com  Phone: +1-412-512-7321