A Metadata-Driven Approach to Computing Financial Analytics in a Relational Database by David Rozenshtein, PhD Sandip K. Mehta Reuters October 2006

What are Financial Analytics? Algebraic formulas: Calculate financial results for companies, instruments, indices, industries, etc. Inputs: Financial metrics: items from financial statements, estimates, prices, interest rates, etc. Values computed for other analytics. Constants. Operations: Mathematical and logical operators and functions.

Algebraic formulas:

Calculate financial results for companies, instruments, indices, industries, etc.

Inputs:

Financial metrics: items from financial statements, estimates, prices, interest rates, etc.

Values computed for other analytics.

Constants.

Operations:

Mathematical and logical operators and functions.

Approaches to Financial Analysis Systems Typical approach: “Hardcode” analytic formulas into the system’s source code. Important disadvantage – need to change this source code as analytics are added, deleted or modified. Better approach: Represent analytic formulas as “metadata.” Build the analytic system as a formula interpreter . Analytics can now be added, deleted or modified without having to make any changes to the analytic system’s source code.

Typical approach:

“Hardcode” analytic formulas into the system’s source code.

Important disadvantage – need to change this source code as analytics are added, deleted or modified.

Better approach:

Represent analytic formulas as “metadata.”

Build the analytic system as a formula interpreter .

Analytics can now be added, deleted or modified without having to make any changes to the analytic system’s source code.

Where and How to Build the Interpreter? Possibility 1: Build the interpreter in standard 3GL – C++, C#, Java, etc. Advantage: Well known algorithms exist. Problem: Financial data is in the database. Analytic results should be placed in the database as well. Interpreter itself would be outside the database layer. Too much data movement between the database and the interpreter. Too slow!

Possibility 1:

Build the interpreter in standard 3GL – C++, C#, Java, etc.

Advantage:

Well known algorithms exist.

Problem:

Financial data is in the database.

Analytic results should be placed in the database as well.

Interpreter itself would be outside the database layer.

Too much data movement between the database and the interpreter.

Too slow!

Where and How to Build the Interpreter? Possibility 2: Build the interpreter in SQL. Advantage: Both data and computation are now completely within the database layer. Issues: How to represent formulas as data? How to actually code the interpreter? How to make all of this very efficient?

Possibility 2:

Build the interpreter in SQL.

Advantage:

Both data and computation are now completely within the database layer.

Issues:

How to represent formulas as data?

How to actually code the interpreter?

How to make all of this very efficient?

Building the Formula Interpreter in SQL Representing formulas: The set of analytic formulas is represented as an annotated directed acyclic graph (ADAG). The nodes of the ADAG represent inputs and outputs of the formulas. The edges represent operations and functions . The ADAG itself is stored as a table in the database. Implementing the interpreter: The interpreter is implemented as a special kind of graph traversal algorithm.

Representing formulas:

The set of analytic formulas is represented as an annotated directed acyclic graph (ADAG).

The nodes of the ADAG represent inputs and outputs of the formulas.

The edges represent operations and functions .

The ADAG itself is stored as a table in the database.

Implementing the interpreter:

The interpreter is implemented as a special kind of graph traversal algorithm.

Standard SQL Graph Traversal Algorithms Standard graph traversal algorithms are depth-first . Not suitable for SQL: Lead to “one/few rows at a time” computation. Too slow!

Standard graph traversal algorithms are depth-first .

Not suitable for SQL:

Lead to “one/few rows at a time” computation.

Too slow!

A Novel SQL Graph Traversal Algorithm We have developed a novel breadth-first graph traversal algorithm. Very good for SQL: Our system computes multiple analytic formulas for all business entities/fiscal period combinations at a time. The number of SQL statement executions is completely independent of the number of business entities or fiscal periods involved, and is proportional only to the depth of the analytic graph, which is usually the logarithm of the number of distinct financial metrics and analytic formulas. Very efficient!

We have developed a novel breadth-first graph traversal algorithm.

Very good for SQL:

Our system computes multiple analytic formulas for all business entities/fiscal period combinations at a time.

The number of SQL statement executions is completely independent of the number of business entities or fiscal periods involved, and is proportional only to the depth of the analytic graph, which is usually the logarithm of the number of distinct financial metrics and analytic formulas.

Very efficient!

The System and the Paper This paper describes an actual system built at Reuters. Computes at the rate of approx. 12 thousand analytic formulas per second. Runs on an approx. $10,000 computer. The system supports arithmetic and logical formulas over numerics. For arithmetic formulas, it supports arithmetic operators: +, -, * and /, and a limited number of function symbols. For logical formulas, it support binary comparators: =, !=, <, <=, > and >=, and the logical operators: NOT, AND and OR. Currently, the system is limited to formulas over numeric values only; however, it is trivial to extend it to other data types. Due to space limitations, the paper presents only the arithmetic component of the interpreter.

This paper describes an actual system built at Reuters.

Computes at the rate of approx. 12 thousand analytic formulas per second.

Runs on an approx. $10,000 computer.

The system supports arithmetic and logical formulas over numerics.

For arithmetic formulas, it supports arithmetic operators: +, -, * and /, and a limited number of function symbols.

For logical formulas, it support binary comparators: =, !=, <, <=, > and >=, and the logical operators: NOT, AND and OR.

Currently, the system is limited to formulas over numeric values only; however, it is trivial to extend it to other data types.

Due to space limitations, the paper presents only the arithmetic component of the interpreter.