Presented in this short document is a description of our technology we call “Sparse Observability”. Observability is the estimatability metric (Bagajewicz, 2010) to structurally determine that an unmeasured variable or regressed parameter is either uniquely solvable (observable) or otherwise unsolvable (unobservable) in data reconciliation and regression (DRR) applications. Ultimately, our purpose to use efficient sparse matrix techniques is to solve large industrial DRR flowsheets quickly and accurately. Most other implementations of observability calculation use dense linear algebra such as reduced row echelon form (RREF), Gauss-Jordan decomposition (Crowe et. al. 1983; Madron 1992), QR factorization which can now be considered as semi-sparse (Swartz, 1989; Sanchez and Romagnoli, 1996), Schur complements, Cholesky factorization (Kelly, 1998a) and singular value decomposition (SVD) (Kelly, 1999). A sparse LU decomposition with complete-pivoting from Albuquerque and Biegler (1996) for dynamic data reconciliation observability computation was used but it is uncertain if complete-pivoting causes extreme “fill-ins” of the lower and upper triangular matrices essentially making them near-dense. There is another sparse observability method using an LP sub-solver found in Kelly and Zyngier (2008) but this requires solving as many LP sub-problems as there are unmeasured variables which can be considered as somewhat inefficient. IMPL’s sparse observability technique uses the variable classification and nomenclature found in Kelly (1998b) given that if we partition or separate the unmeasured variables into independent (B12) and dependent (B34) sub-matrices then all dependent unmeasured variables by definition are unobservable. If any independent unmeasured variable is a (linear) function of any dependent variable then this independent variable is of course also unobservable because it is dependent on another non-observable variable.

1. Sparse Observability using LP Presolve and LT
DL Factorization in IMPL
(IMPL-SparseObservability)
i n d u s t r IAL g o r i t h m s LLC. (IAL)
www.industrialgorithms.com
August 2014
Introduction
Presented in this short document is a description of our technology we call “Sparse
Observability”. Observability is the estimatability metric (Bagajewicz, 2010) to structurally
determine that an unmeasured variable or regressed parameter is either uniquely solvable
(observable) or otherwise unsolvable (unobservable) in data reconciliation and regression
(DRR) applications. Ultimately, our purpose to use efficient sparse matrix techniques is to solve
large industrial DRR flowsheets quickly and accurately.
Most other implementations of observability calculation use dense linear algebra such as
reduced row echelon form (RREF), Gauss-Jordan decomposition (Crowe et. al. 1983; Madron
1992), QR factorization which can now be considered as semi-sparse (Swartz, 1989; Sanchez
and Romagnoli, 1996), Schur complements, Cholesky factorization (Kelly, 1998a) and singular
value decomposition (SVD) (Kelly, 1999). A sparse LU decomposition with complete-pivoting
from Albuquerque and Biegler (1996) for dynamic data reconciliation observability computation
was used but it is uncertain if complete-pivoting causes extreme “fill-ins” of the lower and upper
triangular matrices essentially making them near-dense. There is another sparse observability
method using an LP sub-solver found in Kelly and Zyngier (2008) but this requires solving as
many LP sub-problems as there are unmeasured variables which can be considered as
somewhat inefficient.
IMPL’s sparse observability technique uses the variable classification and nomenclature found
in Kelly (1998b) given that if we partition or separate the unmeasured variables into independent
(B12) and dependent (B34) sub-matrices then all dependent unmeasured variables by definition
are unobservable. If any independent unmeasured variable is a (linear) function of any
dependent variable then this independent variable is of course also unobservable because it is
dependent on another non-observable variable.
This can be easily seen using the following (linearized) DRR equation (Kelly, 1998b):
A * x + [B12 B34] * [y12T
y34T
]T
+ C * z = 0 or
A * x + B12 * y12 + B34 * y34 + C * z = 0
where x represents the measured variables in column vector form, y are the unmeasured
variables and z are the fixed variables or constants with corresponding sparse Jacobian
matrices A, B12, B34 and C respectively. By multiplying both sides by B12T and rearranging
we arrive at essentially the Normal equation found in typical ordinary least squares (OLS)
regression which is unique to the classification method of Kelly (1998b):
B12T
* B12 * y12 = -B12T
* (A * x + B34 * y34 + C * z) or
y12 = -(B12T
* B12)-1
* B12T
* (B34 * y34 + A * x + C * z)

2. Therefore, if (B12T
* B12)-1
* B12T
* B34 has any non-zero elements (relative to machine
precision) in any row then the corresponding independent unmeasured y12 variable is a linear
function of a non-observable unmeasured y34 variable.
Hence, the technology described below is an effective and reliable way to identify unobservable
unmeasured variables in large-scale and industrial-sized flowsheet DRR problems. It should be
mentioned that once the observability calculations have been performed, then the same
matrices can be used to identify redundant and non-redundant measured variables described in
Kelly (1998b). Redundant variables are observable if there measurement reading is missing,
deleted or removed from the data content of the DRR problem. Non-redundant variables are
unobservable if their measurement is no longer available given the remaining model and data of
the DRR problem. Obviously, it is important to have as much hardware (measurement
readings) and software (model relationships) redundancy in the system as possible given the
capital, operational and maintenance costs where this sensor network design optimization
problem is addressed in Kelly and Zyngier (2008).
Technology
IMPL’s sparse observability has two steps or parts: a presolve or preprocessing step to identify
essentially trivial observable (strongly independent) unmeasured variables and a LT
DL
factorization for sparse symmetric indefinite matrices on the remaining equations and
unmeasured variables to determine the dependent unmeasured variables.
Our DRR presolve is taken directly from primal LP presolve techniques (Andersen and
Andersen, 1995) and operates on the B sparse matrix (i.e., [B12 B34]) and to our knowledge,
we are the first to adopt primal LP presolving techniques to the application of identifying
observable unmeasured variables in DRR problems.
First, it involves removing singleton rows with a single column indicating that the column or
unmeasured variable is strongly observable because it is not a function of any other
unmeasured variable. And second, it removes doubleton rows of the form b * yi – b * yj = 0 and
one of the yi or yj unmeasured variables are removed because they are mirror images of one
another; thus reducing the number of rows and columns of B. These two operations are
repeated until there is no more improvement in reducing the rows and columns of B or there are
no rows or columns left in B.
These two straightforward primal LP presolve operations are very effective in practice and
usually shrink the B matrix considerably. As an example, Figure 1 shows the unmeasured
variable’s sparse matrix B taken from Crowe et. al. (1983)’s example #3.

3. Figure 1. Crowe et. al. (1983)’s Example #3.
All columns or unmeasured variables are observable and we determined these to be observable
using our LP presolving techniques. For instance, the first pass or iteration of our presolve finds
eight (8) singleton rows that can be removed i.e., “Node1 N2”, “Node1 Ar”, “Node2 N2”, “Node2
Ar”, “Node3 N2”, “Node3 H2”, “Node3 NH3”, “Splitter Constraints H2”. This means, that of the
total thirteen (13) unmeasured variables, 8 are determined to be strongly observable easily.
When these 8 rows and columns are removed a second pass further removes rows “Node2
NH3”, “Node2 H2”, “Node3 Ar” and “Node4 H2”. The third and final pass removes row “Node1
H2” and now completely identifies all columns as observable.
When not all unmeasured variables are declared to be observable using our presolve method,
we then need to proceed to identifying from the reduced or presolved set, which are
independent and dependent columns using either of the methods mentioned previously.
Instead, our approach in IMPL is to use the LT
DL (Cholesky-like) factorization for symmetric
indefinite sparse matrices from Intel’s Math Kernel Library (MKL) PARDISO (Schenk and
Gartner, 2006). LT
DL is similar to Cholesky factorization except that the former performs
symmetric and other types of pivoting required to precisely determine if a symmetric matrix is
singular (Hansen, 1987) where the later does not. More specifically, indefinite matrices have
both positive and negative eigenvalues and some can be zero (semi-definite) and these types of
indefinite matrices are found in non-convex and nonlinear interior-point optimization problems
such as those solved by IPOPT.
IMPL also supports locating independent and dependent unmeasured variables using a sparse
version of the modified Cholesky factorization from Kelly (1998a) and, even though it does not
employ any pivoting techniques, it is surprisingly accurate even for B matrices with large
condition numbers i.e., maximum singular value divided by its minimum singular value. And,
IMPL further supports a sparse LU decomposition technique with partial-pivoting to check for
near-zero diagonal elements of the upper triangular matrix (U). If Ui,i ~ 0 then this (usually)
indicates that column [B]i is dependent but is less accurate than the LT
DL and our modified
Cholesky.

4. In summary, when the LT
DL factorization in PARDISO is performed using parallel-processing
applied to the presolved BT
B matrix, this provides a very fast, accurate and robust technique to
identify observable and unobservable unmeasured variables in large-scale systems as well as
its complement of redundant and non-redundant measured variables. For nonlinear problems,
the A, B and C sparse matrices are linearized from the converged x and y solution values and
provide only “point or local estimates” of the observability and redundancy indicators.
References
Crowe, C.M., Garcia Campos, Y.A., Hrymak, A., “Reconciliation of process flowrates by matrix
projection, part 1: linear case”, American Institute of Chemical Engineering Journal, 29, 881-
888, (1983).
Hansen, P.C., “Detection of near-singularity in Cholesky and LDL factorizations”, Journal of
Computational and Applied Mathematics, 19, 293-299, (1987).
Swartz, C.L.E., “Data reconciliation for generalized flowsheet applications”, 197th
National Meeting, American Chemical Society (ACS), Dallas, Texas, (1989).
Madron, F., “Process plant performance: measurement and data processing for optimization
and retrofits”, Ellis Horwood, New York, NY, (1992).
Andersen, E.D., Andersen, K.D., “Presolving in linear programming”, Mathematical
Programming, 71, 221-245, (1995).
Albuquerque, J.S., Biegler, L.T., “Data reconciliation and gross-error detection for dynamic
systems”, American Institute of Chemical Engineering Journal, 42, 10, (1996).
Sanchez, M., Romagnoli, J., “Use of orthogonal transformations in data classification-
reconciliation”, Computers & Chemical Engineering, 20, 483-493, (1996).
Kelly, J.D., "On finding the matrix projection in the data reconciliation solution", Computers &
Chemical Engineering, 1553, (1998a).
Kelly, J.D., "A regularization approach to the reconciliation of constrained data sets", Computers
& Chemical Engineering, 1771, (1998b).
Kelly, J.D., "Reconciliation of process data using other projection matrices", Computers &
Chemical Engineering, 785, (1999).
Schenk, O., Gartner, K., “On fast factorization pivoting methods for sparse symmetric indefinite
systems”, Electronic Transactions on Numerical Analysis, 23, 158-179, (2006).
Kelly, J.D., Zyngier, D., "A new and improved MILP formulation to optimize observability,
redundancy and precision for sensor network problems", American Institute of Chemical
Engineering Journal, 54, 1282, (2008).
Bagajewicz, M., “Smart process plants: software and hardware for accurate data and profitable
operations”, McGraw-Hill (ISBN:978-0-07-160471), (2010).