2. Overview Context: Financial Services Industry Every firm different – you will see differing number of the characteristics covered here Highly dependent on the views of senior management
3. What is Outsourcing? “Getting someone else to do the work” Firms “profiled” in this presentation have: 75%+ of major systems are built by external firms Of these, 50%+ are ASP (hosted) Systems are already built – not built to a spec Systems unlikely to interface with each other out of the box but some do (i.e. BBG-Portia) Not referring to offshore development
4. Observations - Staff Small or non existent development team Correspondingly non existent or small QA team Knowledge in outdated technologies Stronger emphasis on vendor relationships Risk intolerance Medium to good experience with (vendor) release management Business Analysis is less structured that in-house development firms
5. Observations - Systems Systems are version outdated for long periods Upgrades are put off and painful if pursued Vendors exert great influence Any in house developed systems are small, “appendages” to vendor systems (feature gap) Systems likely to be hosted as well
6. Observations - Data Data Duplicated in multiple vendor systems Knowledge of data lineage weak and patchy Knowledge of data transformations from one system to another is weak Higher level of data inaccuracy tolerated (than firms with more developed systems) Data is the only intellectual property
7. Observations - Data Each vendor system is master for some type of data (i.e. securities, portfolios) Each vendor system is an operational database Mapping to a common representation doesn’t exist Data Quality measurements are non existent Business users typically have more leeway over data completeness than in other firms Quantity of historical data is somewhat limited
8. Strategy for Getting Control Create an architectural diagram listing all systems and connections among them Identify all the major event entities and reference entities in the firm’s business Identify which system(s) is the master of each entity Establish which systems should be the master! Start with understanding what’s there!
9. Getting Control Part 2 Bring all the data from all systems to a Staging database (big effort, but you could get the vendors to extract the data for you) Raw data only – take calculated data if that’s all you can get All data inbound to each vendor system should go through your firm first All data outbound from each vendor system should go through your firm first Make data entry attributes mandatory
10. Next Steps Data Quality program Data Normalization – for combining Data Standards Business Process Re-engineering Partner with data stewards for training and support DW? BI? Maybe, but take baby steps in this environment
11. Conclusion Where possible, keep firm-specific processing logic at your firm – don’t ask the vendor to build it – tougher upgrade Your data is the only common element throughout all the systems Place you firm between all connections – you are the data hub Are vendor systems actually cheaper to run than internal development?
Editor's Notes
Highly: how data is managed and what is important
Not really developing much – maybe scripting to send or receive files to the vendor systemOften let the vendor show their tests results or mentality: if it’s broken we’ll chase them to fix it – or Bas do the testing3) If they’ve had outsourced systems for a while, some/most are using older technologies like VB for the GUI or Sybase, or no use of XML. This is expected because the upgrading and switching costs are so high.4) Tend to have people who are experienced in vendor relationships, but still concentrated in a few individuals – good size legal department5) Afraid of changing anything – they know the consequences – vendor will charge arm and leg, will face resistance everywhere: ‘if isn’t broken, don’t touch it’
Mgmt Likes stabilityUpgrades painful because long apart, staff has moved on, requires extensive vendor involvementVendors advise best version to move to, not necessarily the firm choosingScripting, small enhancement that was many times cheaper than the vendor cost
Normal but more than internal systems because each vendor has their own requirements – not fed by a single source, often typedLineage often unknown, quite often manual – can tell from vendor’s audit trail but that could be a ‘special request’ (more $)Understanding how data gets from one system to another often handled by the vendor – black boxAccept things like 0.1 is OK because it means nothing in a billion dollar portfolio. Reality is that reconciliation is a massive job, which the vendors expect you to do and it becomes another system to outsource!
This system contains the most attributes for security so let’s use it – could be one with best GUI or has most experienced staff – not chosen as part of a strategyUsing “is” instead of “has”, logical not physical, whatever data that system produces, is considered the ‘operational data’Separate copies of the operational data exist, usually not exactly the same! 100 sec in system A and 103 sec in System BDQ measure not there because you need to have prerequisites in place first ( which I’ll describe shortly)Vendor software “flexible” hence allow customers to make many/most fields optional – bad for data qualityInstead of ‘keep it all’, I observed, what is the minimum the law says and then actually have a job to purge it – this mentality going away in post Enron era – chore to pull it together – heard: why would you need to look back?
Connections can be directional but coarse grained to start like, major business entities flowing – don’t go into to much detail because you want to gain support and understandingStandard DW approach – works well even when you aren’t building a DW Label current but also most suitable and these may be different. The systems with most controls over entry of data and stress completeness are good choices to start withStandard approach – you have to be able to accurately describe and communicate the “mess” before you can get the support you need
Big step – what you want is just a copy of the data so you can at least look at it – may need to run reports to pull it out – often you can coax the vendor into doing this for you – they’ve likely done this before for someone else, or even their own purposes Take all the raw data you can get, and get anything calculated but store separately Get it all the same time or as close as possible because timing differences in the data could be the source of mismatches when you analyzeIf you are sending data to vendor A, and it comes from data supplier B or Vendor B, have it stop or make a copy at your shop first – why? You can see what’s going through the vendor systems, you can trace lineage, you can see which vendors are sending good data and which are not May need to work with vendor to get all the data populated first (use specific values for ‘unknown’) like booleanY,N,X,null (again standard DW approach)
