The document discusses SQL Server Integration Services (SSIS) monitoring, focusing on logging levels, execution context, and dashboard creation tools. It highlights the importance of transitioning to the project deployment model and details various logging options such as basic, performance, verbose, and custom logging since SSIS 2016. Additionally, it mentions community projects for dashboard development, including one based on HTML5 and a Power BI approach for creating SSIS dashboards.

1. SQL Server Integration
Services
Monitoring Deep Dive
DAVIDE MAURI
INFO@DAVIDEMAURI.IT - @MAURIDB
HTTP://SPEAKERSCORE.COM/D46B

2. About Me
Microsoft Data Platform MVP since 2006
Works with SQL Server from 6.5, on BI from 2003
Specialized in Data Solution Architecture, Database Design, Performance Tuning,
High-Performance Data Warehousing, BI, Big Data
Very strong developer background
Loves community!
◦ President of UGISS (Italian SQL Server UG) for 11 Years
◦ AppDev VG Leader since 2017
Regular Speaker @ SQL Server events
Director SW & Cloud @ Sensoria: http://www.sensoriafitness.com/
E-mail: info@davidemauri.it -Twitter: @mauridb
Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx

3. Agenda
Deep Dive into SSISDB Log-Related Catalog
Logging Levels
Building a Dashboard
◦ Native Solutions
◦ 3rd Community Party Solutions
◦ My Solutions:
◦ HTML 5
◦ Power BI

4. Notice to SSIS Men
Everything said here works for SQL Server 2012 and over and applies
to the Project Deployment Model *only*
If you’re sill using the Package Deployment model…
…change it! It’s OLD!
◦ And even O.L.D.: Omitted Log Data

5. SSISDB – SSIS Catalog
Contains *everything* needed to run, manage and monitor SSIS
Packages
◦ (only when using the “Project Deployment Model”)
“internal” schema := private API/Objects, for MS use only
“catalog” schema := public, supported, API/Objects

6. SSIS Server Package Execution
Each time a Package is executed:
◦ Create Package Execution “context”
◦ Set Execution Parameters Values
◦ Execute the Package

7. SSIS Server Package Execution Logging
Logs and Execution Statistics are written to several tables
◦ catalog.executable
◦ catalog.executable_statistic
◦ catalog.event_messages
◦ catalog.operations, catalog.operation_messages
The execution_id binds everything together
Named “operation_id” sometimes because event_messages
is an extension of operation_messages

8. SSIS Server Package Logging Levels
Logging is automatically performed by the server
◦ None: Logging is turned off. Only the package execution status is logged.
◦ Basic: (Default) All events are logged, except custom and diagnostic events.
◦ RuntimeLineage: Collects the data required to track lineage information in the
data flow. (2016 only)
◦ Performance: Only performance statistics, and OnError and OnWarning
events, are logged.
◦ Verbose: All events are logged, including custom and diagnostic events,
including the DiagnosticEx event.
Of course, logging level can be changed at execution-time, no special
design patterns needed

9. Reference Solution Used for Tests
Master, Child, GrandChild case
Master and Child have objects with same name
Master and Child have loops
Master and Child have parallel executions
Package and Projects connection managers have the same name
Package and Project Parameters
Scripts (Dts.Events.FireInformation) logging via Script Task
DataFlows

10. Demo
THE REFERENCE SOLUTION

11. Before Execution
Where data is stored in the SSISDB?
Especially interesting the parameter and connection managers
configured and default values
◦ They will be useful to have the complete execution context when doing log
analysis

12. Demo

13. Package Execution
Created (1)
Pending
(5)
Running
(2)
Stopping
(8)
Canceled (3)
Success
(7)
Completed
(9)
Failed
(4)
Unexpected
Termination /
Crash (6)
catalog.start_execution
catalog.create_execution catalog.stop_operation
ISServerExec
Unexpected Crash

14. Logging “None”
Really means “Minimal”
No logged events, but still some data is logged in some “basic” log
tables :
◦ catalog.executions
◦ catalog.execution_property_override_values
◦ catalog.execution_parameter_values
◦ catalog.executables
◦ catalog.executable_statistics
◦ Log-Like table

15. Logging “Basic”
Default Option
Data Available in “basic” log tables and
◦ catalog.event_messages
Logged Events:
◦ OnError, OnInformation, OnPostExecute, OnPostValidate, OnPreExecute,
OnPreValidate, OnWarning
No specific information on related to DataFlow

16. Logging “Performance”
Data Available in “basic” log tables and
◦ catalog.event_messages
Logged Events:
◦ OnError, OnWarning
Additional specific information on component “phases”
◦ catalog.execution_component_phases
No specific information on related to DataFlow

17. Logging “RuntimeLineage”
“Collects the data required to track lineage information in the data
flow. You can parse this lineage information to map the lineage
relationship between tasks. ISVs and developers can build custom
lineage mapping tools with this information.”
◦ No much more information…
◦ Available in SSIS 2016

18. Logging “Verbose”
Like Basic and Perfomance together PLUS DataFlow details
◦ DataFlow is logged in detail to catalog.execution_data_statistics
Logged Events
◦ All!
◦ Even Diagnostic and Custom log messages (Dts.Events.FireCustom)
Logging in event_messages is something like 100X more verbose
◦ Hence the name 

19. Logging “Custom”
Available only in SSIS 2016
You can decide your own logging level…like
◦ Performance +
◦ OnInformation +
◦ Data Flow

20. Demo

21. Something to keep in mind
Dataflow logs some interesting data as a text message only
◦ Lookup Duplicate Rows warning
◦ Memory Warning
◦ Memory Allocation
◦ Low Virtual Memory
◦ Rows sent to each destination
Verbose is just to verbose, but we still want some DataFlow logging,
so we can dig into the text into event_messages table

22. Demo

23. Building a Dashboard - Native
• Natively Available
• Basic User Experience
• “It just works”
• Not really a dashboard

24. 3Rd Party Community Project
SSIS Reporting Pack (by Jamie Thomson)
◦ No more maintained (AFAIK)
◦ Provides more than reporting
◦ usp_ssiscatalog
http://ssisreportingpack.codeplex.com/

25. HTML5
Based on
◦ HTML 5 / CSS 3
◦ Bootstrap
◦ jQuery, Morris.Js, SmartMenu
◦ Python + Flask
Code on GitHub
◦ https://github.com/yorek/ssis-queries
◦ https://github.com/yorek/ssis-dashboard
Working Demo:
◦ http://ssis-dashboard.azurewebsites.net/

26. HTML5

27. HTML5
An forked version has been created by @niphlod here
◦ https://github.com/niphlod/ssis_dash
Still based on Python
◦ But moved from Flask To Web2Py
◦ Supports security, authentication and authorization
◦ Can monitor more than one SSISDB instance
◦ …much more 
Very cool!

28. PowerBI
Using PowerBI Desktop it is really easy to create a nice SSIS
Dashboard
If you don’t have problems sending data to the Azure cloud, you can
also move the Dashboard to the PowerBI Service
◦ And use SSIS or Azure Data Factory or PowerBI Data Gateway to keep data in
Azure updated

29. One last note
Due to how security is applied in SSIS, if you’re not sysadmin or
ssis_admin queries can be *really* *really* slow
◦ Each row is under custom row security, using certificates
◦ *A lot of certificates* needs to be open and used: One for each Execution!!!!
◦ SQL Server 2012 SP2 fix this by allowing you to do it on project-basis instead
of Execition-basis
◦ https://support.microsoft.com/en-us/kb/2972285
A *unsupported* workaround is to fix the
[internal].[current_user_object_permissions] procedure, avoiding
the usage of the scalar function [get_principal_id_by_sid], use a JOIN
instead

30. Thanks!
QUESTIONS?

31. Demos and Tools
available on GitHub
HTTPS://GITHUB.COM/YOREK/SQLSAT613
HTTPS://GITHUB.COM/YOREK/SSIS-DASHBOARD
HTTPS://GITHUB.COM/YOREK/SSIS-QUERIES