Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

J2EE-Oracle Project Audit Trail Methodology


Published on

J2EE-Oracle Project Audit Trail Methodology Proposal

  • Be the first to comment

  • Be the first to like this

J2EE-Oracle Project Audit Trail Methodology

  1. 1. Asmira Audit Trail Methodology<br />presented by:<br />iAsia DigitalWorks, Inc.<br />prepared by: Benedict J. Martinka<br />January 11, 2004<br />
  2. 2. Asmira Audit Trail Methodology<br />Guiding Principles:<br />Full Auditing Capability and Accountability<br />Activity-Based Rollback Capability<br />Simplicity<br />Negligible Performance Degradation<br />Minimal Application Impact<br />Ease of Development and Reporting Queries<br />Database Cleanliness and Consistency<br />
  3. 3. Asmira Audit Trail Methodology<br />9 Aspects of the Audit Trail<br />Part 1. Parseable Activity Log<br />Part 2. Table & Activity ID Sequences<br />Part 3. Trigger-Generated Timestamps (moddtm & auddtm)<br />Part 4. Standby Database (asmira2)<br />Part 5. Historical Data Audit Tables (asmira2.tableA_aud)<br />Part 6. Fingerprinted DML Action Audit Table (asmira2.dml_aud)<br />Part 7. Database Event Auditing<br />Option A: Native Oracle 9i Database Audit Trail Table (sys.aud$) <br /> Option B: Custom Database Event Triggers & Events Table (evt_aud)<br />Part 8. Audit Table Archive DB (asmiraud for *_aud table archiving only)<br />Part 9. Fine-Grained Data Retrieval Audit (if client requests)<br />
  4. 4. Asmira Audit Trail Methodology<br />Part 1. Parseable Activity Log<br />Activity EJBs, Business Service & Data Service classes (3 layers) will log beginning & end of each such activity or service (and possibly intermediate subactivities or milestones) into a central log file in XML format (to be specified in a separate document) for ease of parsing, reporting, and reconstruction of nested services in a hierarchical manner.<br />Each entry is a composite XML element at the same level as any other entry, however, thus asyncronous chronological logging structure need not mirror logical service hierarchy. Entries for Business & Data Services shall be identified with the Activity that invoked them (directly or indirectly) by means of an Activity ID (actid) field, which must therefore be made part of each service interface signature. This ID shall be requested from a utility class whenever an Activity is begun. The utility will use an Oracle Sequence (see Part 2).<br />Each entry, regardless of type, will include at least:<br />Classname (and methodname, if applicable), actid, pertinent object IDs or data<br />Log entry type (e.g. actBeg/actEnd, svcBeg/svcEnd, daoBeg/daoEnd)<br />Username, role (if applicable), location, IP address, timestamp<br />Status or completion code (at least with “End” type entries, actEnd at a minimum)<br />ValidationReport in XML format for Data Services/Objects with this option enabled<br />(continued)<br />
  5. 5. Asmira Audit Trail Methodology<br />Part 1. Parseable Activity Log (continued)<br />Logging can be dynamically enabled/disabled for any class at runtime, ideally from a console if and when development of an administrative console is realized.An application-scope map of boolean flags keyed by classname is likely the best way to implement this to avoid requiring stateful EJBs and syncronized methods to modify static flag variables in each loggable class. Rather, a singleton class can manage this map as long as it is readable by each loggable class and writeable by the administrative console.<br />The log file shall be named actlog.xml, and shall be rotated by the application on a daily basis with closed logs renamed to actlogYYYYMMDD.xml using the spanned date in YYYYMMDD format in their filename.<br />A utility shall be provided to parse and report upon a sequence of log files using an XML parser, which can search either:<br />Based on a specified date range and filtered by criteria of interest, or<br />Based on a specified actid range, searching sequentially through log files beginning with a specified date until an actBeg type log entry is found with the beginning actid, and accumulating entries until an actEnd type log entry is found with the ending actid, in order to have sufficient data for reconstructing hierarchical activity recording (to which additional filtering may subsequently be applied).<br />
  6. 6. Asmira Audit Trail Methodology<br />Part 2: Table and Activity ID Sequences<br />Create an Oracle Sequence asmira.actid_seq and a corresponding utility class to generate the equivalent of an “auto-increment” ID to assign to each instance of an activity begun through an Activity EJB (see Part 1). This sequence will be shared by all activities and users unless scalability requires partitioning into multiple sequences at a later time.<br />For each table tableA in the asmira database, including enumeration and many-to-many junction tables, create an additional corresponding Oracle Sequence asmira.tableA_seqto generatethe equivalent of an “auto-increment” primary key ID for each insert to that table.<br />For each table tableA, define a single-valued PK (primary key) column named just “id” using tableA_seq. Junction tables are included in this requirement in order to allow auditing of spurious or unauthorized updates that might modify part of a composite key.<br />All FK (foreign key) column names should take the form tableR_id when constrained to related PK Consistent naming of PK & FK columns will facilitate code reuse.<br />In each table tableA, define an additional (indexed, non-unique) column “actid”, to record with any Insert or Update the identifier of the Activity instance responsible for the data modification, or for invoking or spawning the Business or Data Service that did so. While not part of any key, this field must become an attribute of every TopLink value object. The actid column & attribute must allow null to permit data modification outside of an Activity.<br />
  7. 7. Asmira Audit Trail Methodology<br />Part 3. Trigger-Generated Timestamps (moddtm & auddtm)<br />For each table tableA in the asmira database, including enumeration and many-to-many junction tables, define an (indexed, non-unique) “moddtm” (type timestamp) column, to contain the datetime at which each record was created or modified.<br />While not updated from Java, this field must be a read-only attribute of every TopLink value object, to serve as the version and basis for Optimistic Locking, and must therefore be read upon retrieving data for editing and double-checked by TopLink at update time.<br />In the asmira2 standby database (see Part 4), for each audit table tableA_aud (see Part 5), including dml_aud (see Part 6), define an additional “auddtm” (type timestamp) column, to contain the datetime at which each audit record was created or (unexpectedly) modified.<br />Both of these timestamps should be set using the SYSDATE feature of Oracle’s SYS_CONTEXT package, in a ‘Before Insert’ and ‘Before Update’ trigger on each table, rather than set as part of the application’s SQL (none for audit tables) – this will ensure it gets updated properly by any SQL statements executed outside of the application.<br />The “moddtm” in tableA_aud (see Part 5) should not be trigger-modified, but rather must be a mirror copy of the timestamp on the record being audited in the source table tableA, and must be combined with the column to form a composite Primary Key.A large disparity between moddtm and auddtm also indicates audit record manipulation. <br />
  8. 8. INVOICE (parent)<br />invoiceid (PK)<br />headerattributes…<br />LINEITEM (child)<br />lineitemid (PK)<br />invoiceid (FK)<br />itemattributes…<br />INVOICE <br />id (PK)<br />moddtm (indexed)<br />actid (indexed)<br />headerattributes…<br />LINEITEM<br />id (PK)<br />moddtm (indexed)<br />actid (indexed)<br />invoice_id (FK) itemattributes…<br />USERGRP<br />id (PK)<br />moddtm (indexed)<br />actid (indexed)<br />user_id (FK) <br />workgrp_id (FK)<br />usergrpattributes… (if any*) <br />USERGRP (junction table)<br />userid(FK to user table)<br />workgrpid (FK to workgrp)<br />Asmira Audit Trail Methodology<br />Part 3. Trigger-Generated Timestamps (moddtm & auddtm)<br />Typical Data Model (do not imitate)<br />Asmira Data Model (note differences in gold)* Note: It is possible, though rare, for a junction to have attributes of its own. The addition of a single-valued primary key makes this -- and any unauthorized changes to one of the foreign keys --more easily auditable.<br />
  9. 9. Asmira Audit Trail Methodology<br /> Part 4. Standby Database (asmira2)<br />Use Oracle 9i Standby Database and database logging feature to create a mirror database (asmira2) for purposes of backup, ETL functions, and auditing.<br />Except for triggers to generate moddtm timestamps (see Part 3), all other auditing actions/triggers described herein (see Parts 5 & 6) should be applied only to the standby database asmira2 rather than the operational database asmira, soas not to affect performance of the application. Database-level audit trail building will be thus be asyncronous with database transactions.<br />
  10. 10. Asmira Audit Trail Methodology<br />Part 5. Historical Data Audit Tables (tableA =&gt; tableA_aud)<br />In the standby DB (asmira2) only, for each operational data table tableA, including enumeration and junction tables, create a parallel and historical audit table with the suffix “_aud” (e.g. asmira2.tableA_aud).<br />In the standby DB (asmira2) only, for each operational data table tableA, create ‘After Insert’ & ‘After Update’ triggers that add a copy of the inserted or updated row to tableA_aud.(Not on Delete since that action creates no new version of data, though all three actions are audited in the DML Action Audit Table dml_aud – see Part 6.) <br />For each pair of tables, the structure of tableA_aud is identical to tableA except for the following differences: <br />Table tableA_aud has one additional timestamp column, auddtm (see Part 3).<br />Instead of setting/updating moddtm column, the ‘Before Insert’ & ‘Before Update’ triggers on table tableA_aud should set/update the auddtm column (moddtm, like the other columns, is merely copied from tableA by triggers on that table -- see above).<br />Columns & tableA_aud.moddtm must be made a composite Primary Key since id will no longer be unique, as this is a historical table containing multiple versions of the same record with timestamp being the only reliable differentiator (actid should also normally increase with each version, but may be null if a record was modified outside the application or outside of any predetermined Activity).<br />
  11. 11. INVOICE (parent)<br />id (PK)<br />moddtm<br />actid<br />headerattributes…<br />LINEITEM (child)<br />id (PK)<br />moddtm<br />actid<br />invoice_id (FK) itemattributes…<br />USERGRP (junction table)<br />id (PK)<br />moddtm<br />actid<br />user_id (FK) <br />workgrp_id (FK)<br />usergrpattributes… (if any) <br />INVOICE_AUD<br />id<br />moddtm<br />actid<br />headerattributes…<br />auddtm<br />LINEITEM_AUD<br />id<br />moddtm<br />actid<br />invoice_id (FK) itemattributes…<br />auddtm<br />USERGRP_AUD<br />id<br />moddtm<br />actid<br />user_id (FK) <br />workgrp_id (FK)<br />usergrpattributes… (if any) <br />auddtm<br />Asmira Audit Trail Methodology<br />Part 5. Historical Data Audit Tables (tableA =&gt; tableA_aud) <br />Historical Audit Trail Model (all tables & triggers shown would be in Standby DB) <br />tableA:‘After Insert’ & <br />‘After Update’ triggers insert audit row<br />} (PK)<br />} (PK)<br />} (PK)<br />tableA_aud:‘Before Insert’ <br />& ‘Before Update’ triggers set auddtm to SYSDATE<br />(examples)<br />
  12. 12. Asmira Audit Trail Methodology<br />Part 6. Fingerprinted DML Action Audit Table (dml_aud) <br />Create a DML (Data Manipulation Language) audit table in the standby database named asmira2.dml_aud to record all statement-level activity in the database, including any authorized or unauthorized modifications to historical audit trail tables tableA_aud. Table dml_aud shall have these columns:<br />auddtm (timestamp, pk, from SYSDATE)<br />modtbl (varchar, for name of table containing data changes)<br />modid (longint, for id column of affected row)<br />moddtm (timestamp, for moddtm column of affected row, part of PK in tableA_aud)<br />modtyp (char = ‘I’, ‘U’, or ‘D’ for Insert, Update, or Delete action taken)<br />actid (longint, for actid column of affected row, to correlate with activity logs)<br />dbuserid (varchar, for database user ID from SYS_CONTEXT environment)<br />osuserid (varchar, for operating system user ID from SYS_CONTEXT environment)<br />connip (varchar, for client IP address of connection from SYS_CONTEXT env.)<br />Typically, dbuserid, osuserid, and connip will remain the same as those used by the connection pool, however, DBA actions or unauthorized accesses will show some evidence in these identifiers. Specific application users and locations will generally be recorded in the activity log actlog.xml (see Part 1) which can be correlated with these records using actid and other columns in dml_aud.<br />(continued)<br />
  13. 13. Asmira Audit Trail Methodology<br />Part 6. Fingerprinted DML Action Audit Table (continued) <br />Create ‘After Insert’, ‘After Update’ & ‘Before Delete’ triggers on all tables in the standby database asmira2, including enumeration and junction tables as well as historical audit tables tableA_aud, each trigger adding a row to the dml_aud table.<br />Create ‘Before Insert’ & ‘Before Update’ triggers on dml_aud table itself, to set auddtm from SYSDATE and also set dbuserid, osuserid, and connip from SYS_CONTEXT. While there should normally be no Updates (or Deletes except for archiving) to dml_aud, only Inserts, the Before Update trigger adds a final margin of auditing safety.<br />This replaces statement-level recording in Oracle’s sys.aud$ table (see Part 7), with the following advantages:<br />Saves system tablespace and increases scalability.<br />Improves performance by moving this auditing to standby database.<br />Allows inclusion of actid field.<br />Allows easier correlation with specific data changes recorded in historical audit tables tableA_aud. <br />
  14. 14. INVOICE (tableA)<br />id (PK) = 1234567<br />moddtm =9/1/04 15:20:03<br />actid = 2345678<br />headerattributes…<br />INVOICE_AUD (tableA_aud)<br />id = 1234567<br />moddtm =9/1/04 15:20:03<br />actid = 2345678<br />headerattributes…<br />auddtm = 9/1/04 15:20:04<br />DML_AUD<br />auddtm =9/1/04 15:20:04<br />modtbl = “invoice”<br />modid = 1234567<br />moddtm = 9/1/04 15:20:03<br />modtyp = ‘I’<br />actid = 2345678<br />dbuserid = “asmira”<br />osuserid = “asmira”<br />connip = “”<br />DML_AUD<br />auddtm =9/1/04 15:20:05<br />modtbl = “invoice_aud”<br />modid = 1234567<br />moddtm = 9/1/04 15:20:03<br />modtyp = ‘I’<br />actid = 2345678<br />dbuserid = “asmira”<br />osuserid = “asmira”<br />connip = “”<br />Asmira Audit Trail Methodology<br />Part 6. Fingerprinted DML Action Audit Table (After Insert)<br />inserted 9/1/04 by johndeer<br />‘after insert’<br />trigger<br />} (PK)<br />insert<br />audit row<br />‘after insert’<br />trigger<br />insert<br />audit row<br />‘after insert’<br />trigger<br />insert<br />audit row<br />SYSDATE<br />} {<br />‘before<br />insert’<br />trigger<br />} {<br />SYS_<br />CONTEXT<br />
  15. 15. INVOICE (tableA)<br />id (PK) = 1234567<br />moddtm =9/2/04 11:08:55<br />actid = 2345922<br />headerattributes…<br />INVOICE_AUD (tableA_aud)<br />id = 1234567<br />moddtm =9/2/04 11:08:55<br />actid = 2345922<br />headerattributes…<br />auddtm = 9/2/04 11:08:56<br />DML_AUD<br />auddtm =9/2/04 11:08:56<br />modtbl = “invoice”<br />modid = 1234567<br />moddtm = 9/2/04 11:08:55<br />modtyp = ‘U’<br />actid = 2345922<br />dbuserid = “asmira”<br />osuserid = “asmira”<br />connip = “”<br />DML_AUD<br />auddtm =9/2/04 11:08:57<br />modtbl = “invoice_aud”<br />modid = 1234567<br />moddtm = 9/2/04 11:08:55<br />modtyp = ‘I’<br />actid = 2345922<br />dbuserid = “asmira”<br />osuserid = “asmira”<br />connip = “”<br />Asmira Audit Trail Methodology<br />Part 6. Fingerprinted DML Action Audit Table (After Update)<br />updated 9/2/04 by janedoe<br />‘after update’<br />trigger<br />} (PK)<br />insert<br />audit row<br />‘after update’<br />trigger<br />insert<br />audit row<br />‘after insert’<br />trigger<br />insert<br />audit row<br />(Note actid<br />change & <br />modtyp <br />differences)<br />
  16. 16. INVOICE (tableA)<br />id (PK) = 1234567<br />moddtm =9/2/04 11:08:55<br />actid = 2345922<br />headerattributes…<br />INVOICE_AUD (tableA_aud)<br />id = 1234567<br />moddtm =9/2/04 11:08:55<br />actid = 2345922<br />headerattributes…<br />auddtm =9/2/04 11:08:56<br />DML_AUD<br />auddtm =9/3/04 23:30:00<br />modtbl = “invoice”<br />modid = 1234567<br />moddtm = 9/2/04 11:08:55<br />modtyp = ‘D’<br />actid = null<br />dbuserid = “dba”<br />osuserid = “root”<br />connip = “”<br />DML_AUD<br />auddtm =9/3/04 23:59:59<br />modtbl = “invoice_aud”<br />modid = 1234567<br />moddtm = 9/2/04 11:08:55<br />modtyp = ‘D’<br />actid = null<br />dbuserid = “dba”<br />osuserid = “root”<br />connip = “”<br />Asmira Audit Trail Methodology<br />Part 6. Fingerprinted DML Action Audit Table (Before Delete)<br />deleted 9/3/04 by jihadude (unauthorized)<br />} (PK)<br />NOTE: <br />Deleting row<br />in tableA<br />does NOT<br />trigger delete<br />of row in<br />tableA_aud.<br />Two separate<br />explicit<br />delete actions<br />are shown<br />for dml_aud<br />effect only.<br />‘before delete’<br />trigger<br />insert<br />audit row<br />‘before delete’<br />trigger<br />insert<br />audit row<br />(Note userid <br />& IP change<br />and modtyp <br />differences)<br />
  17. 17. Asmira Audit Trail Methodology<br /> Part 7. Database Event Auditing: Option A – Native (sys.aud$) <br />Use Oracle sys.aud$ table to capture many types of DB actions<br />Enable audit at Privilege and Object levels<br />tracks changes to database privileges and structures<br />uses system table space -- growth must be monitored & maintained properly<br />disable Statement level to save space; redundant with dml_aud (see Part 6)<br />Set AUDIT_TRAIL=“DB” in init.ora to use database type audit trail so:<br />can use the predefined audit trail views of the system data dictionary<br />can combine in reports in queries against other audit trail components<br />optionally start DB instance from binary SPFILE rather than init.ora so this static parameter value can be altered using &apos;ALTER SYSTEM&apos; command<br />Captures:<br />Operating system login user name used by direct user or connection pool<br />Database connection user name used by direct user or connection pool<br />Session identifier & Terminal identifier of direct user or connection pool<br />Name of the schema object accessed, date and time stamp of access<br />Operation performed or attempted & completion code of the operation<br />
  18. 18. Asmira Audit Trail Methodology<br /> Part 7. Database Event Auditing: Option B – Custom (evt_aud) <br />Create table asmira.evt_aud with similar definition to sys.aud$<br />has the advantage of not using system table space – more scalable<br />Create Database Event Triggers on asmira DB to monitor:<br />privilege and object/structure changes as native sys.aud$ audit trail would<br />also database startup & shutdown events<br />also failed or successful user logins to database or schema<br />
  19. 19. Asmira Audit Trail Methodology<br />Part 8. Archival Data Recovery Tables<br />Create a separate archive DB asmiraud with an identical schema of historical audit tables only (e.g. asmiraud.tableA_aud) for each audit table described in Part 5 (e.g. asmira2.tableA_aud) <br />Create a nightly batch that flushes all entity & association audit tables to the asmiraud DB to improve transactional performance<br />As part of this batch process, tables asmira.evt_aud (or alternatively sys.aud$)and asmira2.dml_aud (see Part 7 and Part 6 slides respectively) should likewise be flushed and archived to asmiraud.evt_aud (or alternativelyasmiraud.sys_aud) and asmiraud.dml_aud, respectively <br />Audit reporting and data recovery operations will be performed against the asmiraud DB rather than the operational or standby DB or audit tables stored there, since history retained in the latter will be for the current day’s transactions only<br />
  20. 20. Asmira Audit Trail Methodology<br />Part 9. Fine-Grained Data Retrieval Audit (if client requires)<br />Allows auditing of SELECT actions that return columns or data values of specific concern (e.g. sensitive security info or financial data)<br />Can be used to trigger alerts when audited conditions are met<br />Requires COST based optimizer mode (our standard) and periodic analysis of audited objects by DBA<br />May impact application performance if used on many or commonly accessed objects, especially at high-volume times. Conditions are tested on audited tables regardless of whether they apply to the queried records or columns<br />Results stored in table dba_fga_audit_trail<br />Columns: timestamp, db_user, policy_name, sql_bind, sql_text<br />