1. Integrity in Very Large Information
Systems
Dealing with Information Risk Black Swans
Public, Presentation for CAiSE 2013
June 21, 2013
Beat Liver and Helmut Kaufmann

2. About Credit Suisse
One of the world’s leading financial
services providers
Offers to clients its expertise in
– Private Banking
– Investment Banking
– Asset Management
Operates in over 50 countries
– Around 550 locations
46,900 employees
June 21, 2013Beat Liver and Helmut Kaufmann 2/19

3. Agenda
Introduction
− Business-critical failures
− What is common? How to identify and prevent such failures?
− Why is this challenging?
Information risk
− Integrity risk vs. integrity criticality
− Levels of integrity criticality
Integrity controls
− Minimum bar integrity design standards
Integrity controls enhancements
− Minimum bar standards limitations
− Independent controls (Proof-of-Concept)
Experience
Conclusions
June 21, 2013Beat Liver and Helmut Kaufmann
Risk Controls
RatingDrivers
3/19

4. Business-critical Failures
A trading software bug
− generated wrong market orders
− resulting in a loss of 440 million USD within
30 minutes
After a software change
− a payment order processing batch failed.
− Around 7 million account holders were
impacted.
− Sorting out and restoring operations took
several weeks
A trader inadvertently entered
− an order to sell 610'000 shares at 16 Yen a
piece instead of 16 shares at 610'000 Yen.
− resulting in a loss of up to 100 million USD
Source: Risks Digest (see paper)
June 21, 2013Beat Liver and Helmut Kaufmann
Risk Controls
RatingDrivers
4/19
Source WikiMedia, Uwe Kils and Wiska Bodo
under Creative Commons license.

5. What is Common?
How to identify and prevent business-critical integrity failures?
Integrity failure – incorrect data processing
− Correct modifications
Business expectation
− Authorized modifications
Integrity understanding mostly used
Business-critical impact
− Huge financial loss
− Enterprise at risk (sometimes)
Black Swans characteristics
− Unexpected events
− Rationalized in hindsight
− Hard to foresee
June 21, 2013Beat Liver and Helmut Kaufmann
Risk Controls
RatingDrivers
5/19

6. Why is integrity challenging?
Very Large Information Systems in the Financial Services Industry
Size
− Such as, for instance, more than
6’000 Applications (e.g., red dot)
100’000’000 lines of code
10’000 employees
− globally distributed
Complexity
− Multiple business lines and entities
− Functional dependences (e.g., blue lines)
− Evolving requirements
24/7, low-latency and volume
Regulation (Basle III)
− Evolving technology
− Economic factors
Value-chain / vertical integration
Resource constraints
− Technical debts
Tailor-made IT systems
− Custom components
June 21, 2013Beat Liver and Helmut Kaufmann
Application landscape domain model with an Foreign Exchange
(FX) client order application with selected up- and downstream
dependencies (i.e., data flow).
Risk Controls
RatingDrivers
6/19

7. Integrity Risk vs. Integrity Criticality
Risk equation assumptions
− Statistical basis
− History
− Number of instances
Airplanes, cars, etc.
but, very large information systems
− Unique,
− Diverse
− Rapidly evolving
Risk assessment implications
− Can you assess the probability?
− Can you assess the impact?
June 21, 2013Beat Liver and Helmut Kaufmann
Which scenario is business-critical?
a) 10 erroneous payments over 100 Million CHF
each to banks
b) 10 million erroneous payments over 100 CHF
each to individuals on accounts with other banks
Risks in above examples
• In (a), bank’s return money but a counterparty
might default
• In (b), a recovery is possible but it costs too much
Risk Controls
RatingDrivers
Probability Impact [CHF] Risk [P x I] Criticality
0.01 1’000.00 10 low
0.001 1’000’000.00 1’000 medium
0.000001 1’000’000’000.00 1’000 high
7/19

8. Medium vs. High Integrity-Criticality
Probability (unsuitable parameter)
− Rare events
Impact defines integrity-criticality
− Negative black swans (concave losses)
− Possible Losses
Recoverable (I-2, normal critical)
– Cap on sum of residual possible loss
and recovery costs
Irrecoverable (I-1, business critical)
– Cap on absorbable possible loss
given
Business Controls
Assets at Risk
– Business objects
– Populations
June 21, 2013Beat Liver and Helmut Kaufmann
See also [Results from the 2008 Loss Data Collection Exercise, Bank for
International Settlements (BIS), July 2009,Table ILD6 - Distribution of Loss
Amount by Severity of Loss
Risk Controls
RatingDrivers
Rare Events
Possible Loss
0.00
0.10
0.20
0.30
0.40
0.50
0.60
I-2
I-1
Likelihood
-6000.00
-5000.00
-4000.00
-3000.00
-2000.00
-1000.00
0.00
8/19

9. Integrity Design Standards
Design to ensure that critical data is correctly modified
Audience
− Solution architects
− Application owners
Scope
− Individual applications
rated as normal and business critical
differentiation in development, testing
and operation
In comparison
− to industry standards
ISO/IEC 17799:2005 practice guide
− our standards are
Concrete and specific*
– Standards infrastructure
– Compliance criteria
Coherent and complete**
Technology agnostic
June 21, 2013Beat Liver and Helmut Kaufmann
Integrity Design Standard Summary
Data aspect, where critical data
− Must be identified and defined **
− Must be uniquely identifiable, golden-sourced via services**
− Sole identifiers in user interfaces must support validation *
Processing aspects, where critical processing must
− Log critical steps using standard infrastructure*
− Perform a timely reconciliation for exchanged critical data
− Use patterns of the standard consistency model*
− Use idempotent operations, services and
batches**
Validation aspects, where applications must on critical data
− Automatically validate the input/output plausibility
− Use second validation according to the four-eye principle
− Specify in service contracts authoritatively-validations**
− Resolve plausibility exceptions by sign-off,
degraded modes of operation or failure*
Recovery aspects, where application must implement
− Use backup procedure supporting a timely recovery
− Idempotent and restart-able recovery procedures ensuring timely
recovery including a integrity validation
Risk Controls
RatingDrivers
Similar to IT Auditing and Controls - A look at Application Controls, Kenneth Magee (InfoSec
Resources)
9/19

10. Integrity Controls Enhancements
Are the controls effective and efficient?
Integrity controls
− Controls limitations
Devil is in the details (post mortems)
– No safety critical-systems methods
Costs due to criticality propagation
− 2nd version independent controls
Abstraction
– Critical data attributes only
– Approximations are sufficient
– Process boundaries only
Independent
Application landscape
− Order business processes
External process boundaries
– Source (of external commitment)
– Interface (to outside)
Account booking
External payment
Internal process boundaries
– Aggregation (e.g., position keeping)
− Audit trail (design standard)
Integrity Controls 2nd Controls
I/O Validations
Application Landscape
Interface
Source
Order
Management
Settlement
Messaging
Gateway
Payment
Audit Trail
Logging
Infrastructure
ControlsRisk Controls
RatingDrivers
June 21, 2013Beat Liver and Helmut Kaufmann 10/19

11. Application Landscape
Independent Controls Proof-of-Concept
Modeling
− Application Finite State Machines (FSM)
Business objects life-cycle state
Business rules define transition conditions
− Communications among FSMs
Business rules define conditions
− What are the benefits?
Abstraction for a class of systems
Reusability and modularity
Automata facilitates criticality rating
Validations Engine
− Big Data analytics tool
− Modular correlation rule sets
FSM with its business rules
− Tracking life-cycle state in data base
Views based on deadlines
Reduce log retention duration
I/O ValidationInterface
Source
Order
Management
Settlemen
t
Messaging
Gateway
Payment
Audit
Trail
Logging
Infrastructure
Communicating FSM Business
Rules
ControlsRisk Controls
RatingDrivers
June 21, 2013Beat Liver and Helmut Kaufmann 11/19

12. Experience – Lessons Learned
Business/IT Alignment
+ Understanding (non-functional vs. functional)
+ Business controls and IT systems
Rating and minimum bar standards
+ Clearer directives and narrower discretion
- In-depth interdisciplinary understanding necessary
- Challenging institutionalization (comfort zone)
Independent controls Proof-of-Concept
+ Audit trails are suitable, but
- Heterogeneous format, representations, etc.
- Correlation identifiers segmented and directional
+ Standards infrastructure suitable with
moderate response time requirements (~ 10 s)
without automatic intervention (integrity gate)
o But, a reliance on independent controls is undesirable!
- Manual modeling costly and brittle – the killer criteria
Large number of business rules
Frequent modifications across the landscape
Deliverables
Information
Risk Assessment
Methodology
Minimal Bar
Design
Standards
Minimum Bar Standards
compliance assessment
of around 200 applications world-
wide
Independent Controls
Proof-of-Concept with standard
infrastructure and production audit
trails
ControlsRisk Controls
RatingDrivers
June 21, 2013Beat Liver and Helmut Kaufmann 12/19

13. Possible Loss
I-1
Conclusions
In ship fleet operations, watch icebergs. In banking IT, keep an eye on integrity.
Very-large banking information systems
− What is business-critical?
Do business and IT mean the same?
− What is integrity?
Authorized vs. correct modifications
− How to rate the integrity?
What is the impact?
What are the business controls?
How to mange dependencies?
− What integrity controls are necessary?
How to reduce the effort and increase the
assurance level?
Are independent controls an option?
Outlook
− Institutionalization and revision
− Research independent controls
Independent
Risk
Controls
RatingDrivers
June 21, 2013Beat Liver and Helmut Kaufmann 13/19
Source WikiMedia, U. Kils and W. Bodo
under Creative Commons license.

14. Appendix
June 21, 2013Beat Liver and Helmut Kaufmann 14/19

15. Integrity Criticality Rating
1) Information assets in scope
− Financial perspective
− Compliance perspective
− IT Risk Management perspective
2) Assets at Risk (financial values)
− Population of data objects
− Small and large attribute-value errors
3) Business controls (using other apps)
− Possible financial losses
Control check-points (time bounds)
− Recoverability
Capped residual loss + recovery costs
4) Application Business and IT criticality
rating (alignment of understanding)
5) Manage decencies using services
− Services offered/required integrity-criticality
− Consume services meeting integrity-criticality
− Application sub-systems differentiation
June 21, 2013Beat Liver and Helmut Kaufmann
Drivers
Financial
Compliance
Information Risk
Confidentiality
Availability
Integrity
Operation Scope
Modifications
"Correct"
Authorized
Data Scope
IT Risk Mgmt
Compliance
Financial
Application, Service, ...
Business vs. IT Rating
Criticality Rating
(Risk Assessment)
Integrity-criticality
Assets at Risk
Recoverabilty
Business Controls
Application, Service, ...
Business vs. IT Rating
Risk Controls
RatingDrivers
15/19

16. Risk-Adjusted Services
In a SOA, direct dependencies are sufficient
Service functionality
− Data service
EVENT publisher
Read-only ACTION
− Data Processing service
EVENT consuming demon
Write ACTION
Note: EVENT/ACTION refer to semantics and not the
transport!
Service integrity-criticality rating
− Determined by application sub-system
Adequate service consumption
− Service rating equal (or higher) to
consumer's
− Compensations
Service-based dependency management
June 21, 2013Beat Liver and Helmut Kaufmann
getMarketData
updatePosition
FX Order
Management
CRUD Order
createSettlement
Risk Controls
RatingDrivers
16/19

17. Order Example
Foreign Exchange Spot
Joe Smith buys 1'000'000.00 USD against CHF
at a spot exchange rate of 0.9401 USD/CHF
on 2013-04-05 07:45 UTC
Business object with critical attributes
− Order Type: FX Spot
− State {new, modified, canceled, matured}: new
− Counterparty: Joe Smith
− Traded Amount: 1'000'0000.00 USD
− Exchange rate: 0.9401 USD/CHF
− Trade date: 2013-04-05 07:45 UTC
FX Order Management Application
− Generates quotes given market data
− Order capture, modification and cancellation
Create, Read, Update, Delta SOA Service
Sends order life-cycle events down-stream
− Settlement application
− Position management application
June 21, 2013Beat Liver and Helmut Kaufmann
FX Order
Management
FX Position
Keeping
Market
Data
FX
Hedging
FX
Settlement
Payments
Messaging
Gateway
Risk Controls
RatingDrivers
17/19

18. Rating Example
Foreign Exchange Spot
Joe Smith buys 1'000'000.00 USD against CHF at a spot
exchange rate of 0.9401 USD/CHF on 2013-04-05 07:45
UTC
Business object with critical attributes
− Order Type: FX Spot
− State {new, modified, canceled, matured}: new
− Counterparty: Joe Smith
− Traded Amount: 1'000'0000.00 USD
− Exchange rate: 0.9401 USD/CHF
− Trade date: 2013-04-05 07:45 UTC
Asset at Risk: Spot Order population
− 1'000 new orders over 1'000'000 € per day
− 1 pip markup, i.e., 100'000 € markup per day
Data error scenarios - small vs. a few large
− Mispricing exchange rate
− Duplicate/ missing orders
− Duplicate cash settlement payments
Business Controls
− Are they detective and corrective?
− Are possible losses recoverable?
June 21, 2013Beat Liver and Helmut Kaufmann
Business
Control
FX Order
Management
population
FX Position
Keeping
Market
Data
FX
Hedging
CRUD Order
Volume
Profit/Loss
FX
Settlement
Payments
Messaging
Gateway
What is required from
consumed service?
Risk Controls
RatingDrivers
18/19

19. Disclaimer
This document was produced for information purposes and is for the exclusive
use of the recipient. No guarantee is made regarding reliability or completeness
of this document, nor will any liability be accepted for losses that may arise
from its use. This document may not be distributed in the United States or
given to any US persons (within the meaning of Regulation S under the US
Securities Act of 1933, as amended). The same applies in any other
jurisdiction except where compliant with the applicable laws. Copyright © 2013
Credit Suisse Group AG and/or its affiliated companies. All rights reserved.
June 21, 2013Beat Liver and Helmut Kaufmann 19/19