Ssis event handler

SSIS : EVENT HANDLER, LOGGING AND
CONFIGURATION FILE
KIKI NOVIANDI – DATA PLATFORM MVP

Microsoft Data Platform MVP Since 2006
Founder SQL Server Indonesia User Group Community
ABOUT ME
My Name : Kiki Rizki Noviandi
Milis : sqlserver-indo@yahoogroups.com
https://www.facebook.com/groups/sqlserverindonesia
http://www.kwad5.com
https://mvp.microsoft.com/en-us/PublicProfile/33869?fullName=Kiki%20Rizki%20Noviandi

EVENT HANDLER, LOGGING & CONFIGURATION FILE

Basic aspects in
Datawarehouse
Integration
ETL
Package
Configurations
Error
Handling
Event
Handling /
tracking
Events
Logging /
Auditing
Package
Deployment
SQL SERVER INTEGRATION SERVICES (SSIS)

SQL Server Integration Services (SSIS)
ETL = Extract – Transform – Load
Get the data from
source system as
efficiently as possible
Source Destination
Perform
Calculations on the
data
Load the data in the
target storage

Reasons
 Failure of ancestors control
 Truncation or source/destination
connection issue
 Conversion failure
 Issues from migrating files and
data
 Package has an error due to
privileges of OS controls
 Other failure reasons
Package task failed !!!!
Error Handling in SSIS

Error Handling in SSIS
Control Flow:
 Add a failure constraint and redirects the workflow to an alternate
Data Flow:
 Send the row out to an error path by configuring the error output of
the Source/ Destination/Transformation as redirect to error row and
save it to review later.
 Use Event Handler: OnError event in a separate window. Write
custom script or just send an email to a team to notify the error.

Logging in SSIS: Log providers
Log Providers Description
Text File log provider writes log entries to ASCII text files in a
comma-separated value (CSV) format
SQL Server Profiler
log provider
writes traces that you can view using SQL
Server Profiler (.trc)
SQL Server log
provider
writes log entries to the sysssislog table in
a SQL Server database
Windows Event log
provider
writes entries to the Application log in the
Windows Event log on the local computer
XML File log writes log files to an XML file (.xml)

Event Handling in SSIS
Integration Services packages are event-driven. This means we can specify
routines to execute when a particular event occurs. An event can be the
completion of a task or an error that occurs during task execution.
Event Handlers Description
OnError Generated as the result of an error condition
OnPreValidate Fired before Validation process starts
OnQueryCancel Fired when user clicks on cancel or during an executable to determine whether it
should stop running
OnTaskFailed Signals the failure of a task and typically follows OnError Event
OnPreExecute Indicates that an executable component is about to be launched
OnPostExecute Takes place after an executable component finishes running

SSIS Auditing
 Add task(s) to the Event Handlers of the package.
 Select auditing for the entire package or for a specific task.
 Select events OnError, OnPostExecution, OnVariableValueChanged etc.
 Inside every dataflow task add row count components after source and
target to track extracted and loaded row count.
 Add Variables at package level scope to store rowcount for each dataflow.
 Add Variables in the OnPostExecute event handler scope to store certain
information about DataFlow source/target (e.g. Query, TableName ...)

SSIS Auditing:
Helps to answer the following questions
 Which package was run and for how long?
 Who owns the package? Or who modified the package?
 When was the package executed?
 What kind of data and how many records were written or changed by
ETL?
 What kind of errors and how many errors occurred?

Benefits that Custom Auditing and Logging can bring to your ETL
process
 Help you provide regulatory compliance
 Provide a deep understanding of database activity and additional
insight into data anomalies that may occur
 Can help answer important questions like, “When was that row last
updated?”
 help you identify specific data for targeted rollbacks

Logging vs Custom Auditing
Logging Custom Auditing
o Captures Metadata- Information
package execution itself
o Captures Information about data,
along with package metadata
o Errors encountered, Execution Time
package, Data bytes, data flow buffer
details, Machine name, Package
Task name
o Row counts of Extracts, Inserts,
Updates, Deletes and Errors, you can
default the status of the changed
and package execution
o Choose the Log providers and its
location
o Use Execute SQL task to define
variables , parameter binding, and
assigning values to the parameters
o Provides limited information on the
package
o Designed for DBAs/ Users who can
query and request for more
information about the data

Question:
If Change Data Capture (CDC)
reads and tracks every Historical
data and net changes from SQL
Server transaction logs, then
whey not use CDC for Auditing?

Answer: There’s a downside to this
1. The amount of history data can become huge fast
2. Does not return all information about the changes you
might need, for e.g. who made the change, when and
how? (when a record is deleted or updated)
3. Delay possible: The history data takes some time to catch
up, because it is based on the transaction logs and the
operation is asynchronous.
4. It depends on the SQL Server Agent. If the Agent is not
running or crashes, no history is being tracked.

Longer running package? How would you optimize the Package Execution?
 Parallel execution of SSIS tasks
 In case of Incremental load, Use Execute SQL task instead of OLEDB Command
transformation to process the updated/ new inserts
 Avoid processing the redundant columns in the data flow task
 Keep notice of buffers and execution tree
 Avoid using checkpoints while auditing SSIS package as they cannot store variables as
Type objects and cannot integrate with, or are most often ignored by event handlers
 Use Lookup, Conditional split to customize the SCD work flow
 Enable Error handling and logging on package failure

Key Components in SSIS Package Custom Auditing
ETL: Data warehouse tables (Staging,
Dimensions, Facts, Data marts)
Slowly Changing Dimensions Type 1/2
Extract Meta data and Row counts (DML
Operations)
Parent-Package Configuration
On-Error Event Handler to Capture Error message
Audit SSIS
Package

DEMO
An Audit table was structured by defining the components with the required information
regarding the Metadata and the transactional records.
Attributes Definition
Audit Key A global unique ID assigned to every execution of
ETL package or packages in the target table. Usually
it is an auto identity integer starting from ‘1’
ParentAuditkey Surrogate ID assigned to the execution of child
packages as metadata. ID inherited from the audit
key of master package. Batch/ Load ID mapped with
the process of every loads/ updates
PkgName assigns the Name of every Corresponding Package
executed including Master package and child
packages.
PkgID Internal GUID of every SSIS package
ExecStartDT Start time of package execution

Demo (contd.)
ExecStopDT End time of package execution
TableName Assigns the table name when package executes to
define or populate a table.
PkgName assigns the Name of every Corresponding Package
executed including Master package and child
packages.
ExecutionInstanceGUID Contains the Global Unique ID for every process,
generated by SSIS
ExtractRowCnt Contains the Count of records extracted from the
source file
InsertRowCnt Contains the Count of Inserted Records in Staging
and Dimension tables by the ETL process.
UpdateRowCnt: Contains the count of updated records, especially
in dimension Tables which uses the SCD Type II
functionality.

Demo (contd.)
ErrorRowCnt Contains the records which were erroneous or not
processed in ETL
TableInitialRowCnt Contains the number of records initially existing in
any staging and Dimension table
TableFinalRowCnt Contains the total number of records in any staging
and Dimension table after inserted , updated or
deleted in the execution process
DeleteRowCnt Contains the number records deleted in the ETL
process
SuccessfullyProcessingInd determines the status of execution of every ETL. If
successfully executed then sets ‘Y’. Default is set to
‘N’.

 Master package (Capturing Meta data)
Demo (contd.): Workflow
Execute SQL task that checks for the number of rows affected and inserts a Temporary
Dummy row in the beginning of the Audit table
Add Audit key and Parent Audit Key variables and add Execute SQL task that Stores
the highest value of Audit key in the parameter.
Add Execute SQL task which populates the Meta data in the Audit table. In the Task
Editor connect to the target database and write a T-SQL query to insert the Meta data
to the defined Parameters including Audit and Parent audit keys

 Master package (calling child packages in data warehouse)
Demo (contd.): Workflow
Add the Execute Package Task to call the Child package (e.g Audit package of a
Staging table) which processes the loading and auditing of the first Staging table
Add the Execute Package Task to call the Child package (e.g Audit package of a
Dimension table) which processes the loading and auditing of the firs Dicmension table
updates the Metadata specifically the end time of the execution and success status of
the execution process in the audit table

 SQL Server Integration Services is an exceptionally high performance integration
and transformation tool
 Customize the Auditing using Execute SQL tasks, Row counts, parameters, system
package variables to capture transactional information and metadata (DMLs)
 Recommended using an alternative to SCD transform component to preserve
historical record and counts in Dimension tables if implementing SCD Type 2
 Implement Error capturing strategies in Data flow and Control flow tasks
 Unless required, limit the use of Event handlers to OnError and/ or OnTaskFailed
events as it carries a large I/O overhead and can slow the application performance
dramatically
 Use Script component and Script task to customize the error information at the
package level and at individual task flow
TAKE AWAYS

Ssis event handler

Recommended

Recommended

More Related Content

Similar to Ssis event handler

Similar to Ssis event handler (20)

More from Kiki Noviandi

More from Kiki Noviandi (10)

Recently uploaded

Recently uploaded (20)

Ssis event handler

Editor's Notes