In this session we'll see everything interesting is hidden in the SSISDB database, where you can gain a lot of insight on the outcome, the performance and the status of your SSIS Packages. I'll share everything I've learned building the SSIS Dashboard we're actually using in production and that you can test here http://ssis-dashboard.azurewebsites.net/. We’ll see the internals of SSISDB database, how we can add custom logging information and how we can use all these data in order to know exactly what happened on a specific point in time.
2. Davide Mauri
• Microsoft SQL Server MVP
• Works with SQL Server from 6.5, on BI from 2003
• Specialized in Data Solution Architecture, Database Design, Performance
Tuning, High-Performance Data Warehousing, BI, Big Data
• President of UGISS (Italian SQL Server UG)
• Regular Speaker @ SQL Server events
• Consulting & Training, Mentor @ SolidQ
• E-mail: dmauri@solidq.com
• Twitter: @mauridb
• Blog: http://sqlblog.com/blogs/davide_mauri/default.aspx
3. Agenda
• Deep Dive into SSISDB Log-Related Catalog
• Logging Levels
• Building a Dashboard
• Native Solutions
• 3rd Community Party Solutions
• My Solutions:
• HTML 5
• Power BI
4. Notice to SSIS Men
• Everything said here works for SQL Server 2012 and over and
applies to the Project Deployment Model *only*
• If you’re sill using the Package Deployment model…
• …change it! It’s OLD!
• And even O.L.D.: Omitted Log Data
5. SSISDB – SSIS Catalog
• Contains *everything* needed to run, manage and monitor SSIS
Packages
• (only when using the “Project Deployment Model”)
• “internal” schema := private API/Objects, for MS use only
• “catalog” schema := public, supported, API/Objects
6. SSIS Server Package Execution
• Each time a Package is executed:
• Create Package Execution “context”
• Set Execution Parameters Values
• Execute the Package
7. SSIS Server Package Execution Logging
• Logs and Execution Statistics are written to several tables
• catalog.executable
• catalog.executable_statistic
• catalog.event_messages
• catalog.operations, catalog.operation_messages
• The execution_id binds everything together
Named “operation_id” sometimes because event_messages
is an extension of operation_messages
8. SSIS Server Package Logging Levels
• Logging is automatically performed by the server
• None: Logging is turned off. Only the package execution status is
logged.
• Basic: (Default) All events are logged, except custom and diagnostic
events.
• Performance: Only performance statistics, and OnError and
OnWarning events, are logged.
• Verbose: All events are logged, including custom and diagnostic
events, including the DiagnosticEx event.
• Of course, logging level can be changed at execution-time, no
special design patterns needed
9. Reference Solution Used for Tests
• Master, Child, Child case
• Master and Child have objects with same name
• Master and Child have loops
• Master and Child have parallel executions
• Package and Projects connection managers have the same
name
• Package and Project Parameters
• Scripts (Dts.Events.FireInformation) logging via Script Task
• DataFlows
11. Before Execution
• Where data is stored in the SSISDB?
• Especially interesting the parameter and connection managers
configured and default values
• They will be useful to have the complete execution context when doing
log analysis
14. Logging “None”
• Really means “Minimal”
• No logged events, but still some data is logged in some “basic”
log tables :
• catalog.executions
• catalog.execution_property_override_values
• catalog.execution_parameter_values
• catalog.executables
• catalog.executable_statistics
• Log-Like table
15. Logging “Basic”
• Default Option
• Data Available in “basic” log tables and
• catalog.event_messages
• Logged Events:
• OnError, OnInformation, OnPostExecute, OnPostValidate, OnPreExecute,
OnPreValidate, OnWarning
• No specific information on related to DataFlow
16. Logging “Performance”
• Data Available in “basic” log tables and
• catalog.event_messages
• Logged Events:
• OnError, OnWarning
• Additional specific information on component “phases”
• catalog.execution_component_phases
• No specific information on related to DataFlow
17. Logging “Verbose”
• Like Basic and Perfomance together PLUS DataFlow details
• DataFlow is logged in detail to catalog.execution_data_statistics
• Logged Events
• All!
• Even Diagnostic and Custom log messages (Dts.Events.FireCustom)
• Logging in event_messages is something like 100X more
verbose
• Hence the name
18. Logging “Custom”
• Available only in SSIS 2016
• You can decide your own logging level…like
• Performance +
• OnInformation +
• Data Flow
20. Something to keep in mind
• Dataflow logs some interesting data as a text message only
• Lookup Duplicate Rows warning
• Memory Warning
• Memory Allocation
• Low Virtual Memory
• Rows sent to each destination
• Verbose is just to verbose, but we still want some DataFlow
logging
22. Building a Dashboard - Native
• Natively Available
• Basic User Experience
• “It just works”
• Not really a dashboard
23. 3Rd Party Community Project
• SSIS Reporting Pack (by Jamie Thomson)
• No more maintained (AFAIK)
• Provides more than reporting
• usp_ssiscatalog
http://ssisreportingpack.codeplex.com/
24. HTML5
• Based on
• HTML 5 / CSS 3
• Bootstrap
• jQuery, Morris.Js, MetisMenu
• Python + Flask
• Code on GitHub
• https://github.com/yorek/ssis-queries
• https://github.com/yorek/ssis-dashboard
26. HTML5
• An even more powerful version has been created by @niphlod here
• https://github.com/niphlod/ssis_dash
• Still based on Python
• But moved from Flask To Web2Py
• Supports security, authentication and authorization
• Can monitor more than one SSISDB instance
• …much more
• Very cool!
27. PowerBI
• Using PowerBI Desktop it is really easy to create a nice SSIS
Dashboard
• If you don’t have problems sending data to the Azure cloud, you
can also move the Dashboard to the PowerBI Service
• And use SSIS or Azure Data Factory or PowerBI Data Gateway to keep
data in Azure updated
28. One last note
• Due to how security is applied in SSIS, if you’re not sysadmin or
ssis_admin queries can be *really* *really* slow
• Each row is under custom row security, using certificates
• *A lot of certificates* needs to be open and used: One for each Execution!!!!
• SQL Server 2012 SP2 fix this by allowing you to do it on project-basis instead
of Execition-basis
• https://support.microsoft.com/en-us/kb/2972285
• A *unsupported* workaround is to fix the
[internal].[current_user_object_permissions] procedure, avoiding the
usage of the scalar function [get_principal_id_by_sid], use a JOIN
instead
Editor's Notes
Overview SSISDB
Diversi livelli di logging
Estrazione infomrazione log dalle tabelle del SSISDB
Master-Child
Estrazione Info errori
Estrazione ifo esecuzione (variabili, environment)
Data Tap
Describe the test solution, and deploy it
Deploy & Configure (with and without environments)
catalog.executable_statistics is the nearest thing to a log. Rows are added during package execution. The other tables are filled with rows *BEFORE* package starts the run phase
Logging levels demo
Extract DataFlow interesting data from event_messages