• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
SQL Server Integration Services – Enterprise Manageability
 

SQL Server Integration Services – Enterprise Manageability

on

  • 5,303 views

Talking about SQL Server Integration Services (SSIS) overview, deployment, security, and manageability

Talking about SQL Server Integration Services (SSIS) overview, deployment, security, and manageability

Statistics

Views

Total Views
5,303
Views on SlideShare
4,843
Embed Views
460

Actions

Likes
15
Downloads
0
Comments
0

2 Embeds 460

http://www.techgig.com 457
http://www.slideshare.net 3

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    SQL Server Integration Services – Enterprise Manageability SQL Server Integration Services – Enterprise Manageability Presentation Transcript

    • SQL Server Integration Services – Enterprise Manageability David Pendleton Dan English Senior Consultant Principal Consultant DavidPe@magenic.com DanE@magenic.com
    • Who are we? – Dan and David Dan English David Pendleton http://denglishbi.spaces.live.com/ • Developing with Microsoft technologies for over 10 years • 20+ years experience as a systems analyst, • database developer, designer and software Over 5 years experience with Data Warehousing and Business Intelligence engineer • • Experienced in ETL and Analysis Services Experienced technician with strengths in development, requirements gathering and data database design and development, systems modeling architecture and software development • • Microsoft Certified IT Professional (MCITP) and Specialized in ETL, database modeling and Microsoft Certified Technology Specialist (MCTS) design, SQL development, and change management
    • Who are we? – Magenic  Founded in 1995, Magenic is a technical consulting firm focused exclusively on Microsoft technologies and has designed and delivered more than 500 Microsoft-based applications  Headquartered in Minneapolis, with offices in Chicago, Boston, Atlanta and San Francisco  2005 Microsoft Partner of the Year, Custom Development Solutions – Technical Innovation  2007 Microsoft Partner of the Year Finalist, Data Management  Microsoft Gold Certified Partner and National Systems Integrator  40 Enterprise Data Services (EDS) consultants
    • Today‟s Agenda • SQL Server Integration Services Introduction and Architecture •8:30 to 9 (Dan English) • Team Development, Deployment and Configuration •9:15 to 10:30 (David Pendleton) • Security and Manageability •10:45 to 11:30 (Dan English) • Summary and Q&A
    • Quick Audience Poll •How many DBAs? SQL/BI Developers? Data Architects? Others? •How many are currently evaluating SSIS? •How many are currently using SSIS in production? •How many are currently using SSIS 2008? •Is SSIS being used for something besides DB Maintenance or ETL? •If so what? •Anyone still using DTS?
    • Today‟s Agenda • SQL Server Integration Services Introduction and Architecture •8:30 to 9 (Dan English) • Team Development, Deployment and Configuration •9:15 to 10:30 (David Pendleton) • Security and Manageability •10:45 to 11:30 (Dan English) • Summary and Q&A
    • SSIS Introduction SQL Server 6.5 SQL Server SQL Server • Transfer 2000 2008 Manager • DTS • SSIS • Bulk Copy Enhancements Enhancements Program SQL Server SQL Server 7.0 2005 • Data • Integration Transformation Services Services
    • Microsoft BI Tool Offerings DELIVERY SharePoint Server Analytic Excel Scorecards Plans Reports Dashboards Views Workbooks END USER TOOLS & PERFORMANCE MANAGEMENT APPS Excel PerformancePointServer BI PLATFORM SQL Server SQL Server Reporting Services Analysis Services SQL Server DBMS SQL Server Integration Services
    • SSIS Introduction – BIDS
    • SSIS Introduction What can you use SSIS for? 1. Extract, Transform, Load data from heterogeneous sources 2. Combine, sanitize, standardize, split, and summarize data 3. Convert data to different data types and put into different formats and sources 4. Database maintenance and transfer operations • Backup, check integrity, execute jobs, cleanup, rebuild indexes, update statistics, etc. 5. Transfer SQL Server database objects 6. Workflow engine supporting complex logic and precedence constraints 7. Perform Analysis Services processing and Data Mining querying 8. Use as a data source for Reporting Services report 9. Perform FTP operations 10. Extensible to create your own components (sources, transforms, control flow, etc.) 11. Utilize scripting tasks and access .NET code with VSTA (Visual Basic .NET or C#) 12. Send email notifications 13. Access Web Services 14. File archiving and maintenance tasks 15. And much, much more…
    • SSIS Architecture SSIS is made up of two core components: 1. Data Transformation Run-time engine • Handles the control flow of a package 2. Data Flow engine • Data Transformation Pipeline engine • Data Sources • Transformations • Destination Targets
    • SSIS Architecture Data Transformation Run-time engine • Provides services to the packages and the components • Handle the storage • Execution • Logging • Debugging • Event handling • Deployment • Variable management • Transactions • Connections
    • SSIS Architecture •The „package‟ is the core component of SSIS. SSIS packages are the basic unit of deployment and execution. •An SSIS package is XML. •Packages can be created using BIDS or programmatically.
    • SSIS Architecture What makes up a package? 1.Control Flow 2.Precedence constraints 3.Data Flow • Control Flow • Containers • Tasks
    • SSIS Architecture • Precedence constraints • Controls order of execution • Success, Failure, Completion • Expressions • AND/OR logic
    • SSIS Architecture • Data Flow • Source adapters • Transformations • Destination adapters
    • SSIS Architecture
    • SSIS Architecture Enhancements • What‟s new in SSIS 2008? – Improved Scripting with VSTA including support for C# – Data Flow Task performance improvements with smarter thread allocation and processing of execution trees – ADO NET source and destination adapters – Import/Export Wizard enhancements • ADO.Net support, data conversions (XML file), performance improvements – Cache Transform and Cache Connection Manager for Lookup Transform (Lookup Transform improved also – no match output) – Utilize CDC and Merge for incremental updates – Data Profiling Task (and Viewer)
    • SSIS Architecture Enhancements Data Profiling Demo
    • Today‟s Agenda • SQL Server Integration Services Introduction and Architecture •8:30 to 9 (Dan English) • Team Development, Deployment and Configuration •9:15 to 10:30 (David Pendleton) • Security and Manageability •10:45 to 11:30 (Dan English) • Summary and Q&A
    • SSIS Enterprise Manageability • SSIS Package Deployment – The SSIS runtime environment – Package deployment – Securing packages • SSIS Configurations – Understanding and using SSIS package configurations • Team Development using SSIS – SSIS package development in a team environment • Demo: Tying it all together
    • The SSIS runtime environment • Does not require SQL Server to run* – Although SQL Server is probably involved somewhere • Does not require the SSIS Service to run – The SSIS service enables package management and monitoring via SSMS • Any server can host SSIS packages – Provided the minimum requirements are met • .NET Framework, SQL Native Client, Setup support files • Installed using SQL Server Setup
    • Package Deployment • In general, there are two ways to deploy a package to a server – File System – SQL Server (msdb..sysdtspackages90) • Package Deployment Utility – Configured in BIDS, created by the build process – Uses the .SSISDeploymentManifest file extension • Deployment can also be done “manually”, e.g. Xcopy, SMSS import
    • Package Deployment • Deployment option advantages compared – File System • OS-based file encryption and ACLs • Not subject to SQL Server availability • Easier direct access for viewing or editing • Generally easier on the developer – SQL Server • Easier access by multiple individuals • Benefits from database security, roles and Agent interaction • Packages get backed up with normal database backup processes • Generally easier on the DBA
    • Package Deployment • Creating the Package Deployment Utility – Accessed via Project Properties in BIDS
    • Securing your packages • SSIS offers these options for securing packages – Setting the ProtectionLevel property of a package – Setting the ProtectionLevel and PackagePassword properties of a package – Controlling access by using SQL Server database-level roles – Securing the operational environment by controlling access to file locations – Guaranteeing the integrity of a package by signing with a certificate.
    • Securing your packages • The package ProtectionLevel property – Determines the way package content is secured • DontSaveSensitive • EncryptSensitiveWithUserKey • EncryptSensitiveWithPassword • EncryptAllWithPassword • EncryptAllWithUserKey • ServerStorage
    • Securing your packages • DontSaveSensitive – Removes sensitive information from the package during the save, e.g. passwords – When the package is opened, the removed items will have to be provided – Scheduled or unattended execution will likely fail – Actually your best option for most development
    • Securing your packages • EncryptAllWithUserKey – Encrypts the entire package by applying a user-specific key securely stored in the user‟s profile – The package can only be opened by the same user with access to the same user profile. – Guaranteed to make you unpopular with your team members
    • Securing your packages • EncryptSensitiveWithUserKey – Encrypts sensitive information in the package using a user- specific key securely stored in the user‟s profile – The package may be edited by other users, but the sensitive information will be automatically removed – Execution without access to the encryption key will fail – This is the default level
    • Securing your packages • EncryptAllWithPassword – Encrypts the entire package with a user-supplied password – Password is required to open or edit – Requires the /De[crypt] password option to run with DTEXEC
    • Securing your packages • EncryptSensitiveWithPassword – Encrypts sensitive in formation in the package using a user- supplied password – Package may be edited without the password, but sensitive information is automatically removed – Requires the /De[crypt] password option to run with DTEXEC
    • SSIS package Configurations • Configurations allow you to modify package property and variable values at load time • Available in several types – SQL – XML – Registry – Environment Variables – Parent Package • You can mix-and-match configurations in a package
    • SSIS Package Configurations • SQL configuration – Configurations are stored in a SQL Server table – You may call the table anything you like – May reside in any database, on any SQL Server – Contains one or more configurations • XML configuration – Plain-text XML file – Contains one or more configurations – Uses the file extension .dtsConfig by default – Easily deployed, easily shared
    • SSIS Package Configurations • Registry configuration – Configurations are stored in a registry key – Restricted to the HKEY_CURRENT_USER hive – Property path is stored in the package • Environment Variable configuration – Package property value is stored in an environment variable – Property path is stored in the package – Not the same as an Indirect Configuration* • Parent Package configuration – Used to fetch a value from a variable in a calling package
    • SSIS Package Configurations • Understanding how configurations are applied – SSIS 2005 1. The package is loaded 2. Configurations specified at design time are applied in the order they appear in the package 3. Command-line configuration values are applied 4. Parent Package variable configuration values are applied 5. The package is run
    • SSIS Package Configurations • Understanding how configurations are applied – SSIS 2008 1. The package is loaded 2. The configurations specified at design time are applied in the order they appear in the package 3. Command-line configuration values are applied 4. The configurations specified at design time are reloaded in the order specified in the package 5. Parent Package variable configuration values are applied 6. The package is run
    • Team Development and SSIS • Use a source control system – SSIS packages are stored as XML, but are effectively binary files – Merging packages is difficult, if not impossible – Only one developer can work on a package at a time • Require exclusive check-outs – Both Visual SourceSafe and Team Foundation Server will permit shared check-outs; this should be disabled • Perform regular source control operations – Ensure you are always working with the latest version of a package, check-in incrementally
    • Team Development and SSIS • Design your ETL process with team development in mind. – One package; one table – Package functionality should be as granular as is practical – Use Parent packages and the Execute Package Task to group packages together for end-to-end execution – Spread the work around • Your packages • Your team
    • Team Development and SSIS • Create and use package templates – A template is simply an SSIS package – Any package can be used as a template – Templates can be directly visible to BIDS, or not • A template in the DataTransformationItems template folder will appear in the Add|New Item dialog – Implement and reinforce standard practices in your templates • Logging • Checkpointing • Configurations • Annotations
    • Team Development and SSIS • Security considerations – Use Windows Authentication whenever possible, both in development and production – Manage the package ProtectionLevel property • ProtectionLevel = DontSaveSensitive – Maximum flexibility – Sensitive information can be managed in SQL configurations
    • Team Development and SSIS • Deploy your packages to the File System • Use a common folder structure – Provides a consistent, known file location regardless of the server or environment – All files of a certain type are stored in the same place – Allows you to make all locations dynamic within your package – Easier to manage
    • Team Development and SSIS • Common folder structure: the SSIS Root Folder ..SSIS ..SSISCheckpointFiles ..SSISConfigurations ..SSISErrorFiles ..SSISLogFiles ..SSISPackages ..SSISRawFiles • Use project-specific sub-folders to group related files below the root folders • Edit MsDtsSrvr.ini.xml to use Packages as the SSIS Package Store
    • Team Development and SSIS • Machine-specific Configuration file – XML configuration • Accessed via indirect configuration – Only one configuration file per machine – Contains two (and only two) configurations • RootFolder – Contains the path to the root level of the custom folder structure • ConfigurationServer – Contains the connection string to the SQL configuration server
    • Team Development and SSIS • SQL Server Configuration – Easily managed using DDL – Adding an identity column helps with DML tasks • ConfigurationID int IDENTITY(1,1) NOT NULL – Store multiple configurations in the same table – Allows multiple packages to access the same configurations – All the benefits of SQL Server security • Not only sensitive information, but all information can be protected – Offers the maximum flexibility
    • Demo: Tying it all together • Creating the common folder structure • Redirecting the SSIS Package Store • Implementing the machine-specific configuration file • Implementing the SQL configuration database and table(s) • Leverage the common folder structure in a template using variables and expressions • Making a template available to BIDS
    • Today‟s Agenda • SQL Server Integration Services Introduction and Architecture •8:30 to 9 (Dan English) • Team Development, Deployment and Configuration •9:15 to 10:30 (David Pendleton) • Security and Manageability •10:45 to 11:30 (Dan English) • Summary and Q&A
    • SSIS Security • Security in SQL Server Integration Services consists of several layers that provide a rich and flexible security environment. – These security layers include the use of: • digital signatures • package properties • SQL Server database roles • and operating system permissions • Security categories - Identity or Access control – Identity control goal: Ensure that you only open and run packages from trusted sources. (digital signatures) – Access control goal: Ensure that only authorized users open and run packages. (protection levels, encryption, msdb storage, db roles, file permissions)
    • SSIS Security – database roles • SQL Server Integration Services includes the three fixed database- level roles for controlling access to packages db_ssisadmin (sysadmin) – can do anything related to all packages 1. db_ssisltduser – can import packages; view, delete, execute, export, and 2. change own package roles db_ssisoperator – can view, execute, export all packages (including SQL 3. Server Agent) and they cannot import packages And Windows administrators – can view execution details and stop all 4. running packages • Roles can be implemented only on packages that are saved to the msdb database in SQL Server. You assign roles to a package using SQL Server Management Studio. The role assignments are saved to the msdb database.
    • SSIS Manageability - SSMS • The Integration Services server node presents two folders for working with SSIS packages: – the Running Packages folder • The Running Packages folder displays the SSIS packages that are currently executing on the local server – and the Stored Packages Folder.
    • SSIS Manageability - SSMS • The Stored Packages folder lists the saved SSIS packages that have been registered on the local server. By default this folder contains two subfolders: – the File System Folder – and the MSDB folder
    • SSIS Manageability - SSMS • Right-clicking a package displays a shortcut menu that enables you to perform a number of task including: • New Folder. Creates a new folder in Object Explorer for displaying packages saved in the file system or in the sysdtapackages90 table. • Import Package. Imports the package from the file system to the msdb database • Export Package. Exports the package from the msdb database to the file system. • Run Package. Executes the package using dtexecui. • Delete. Deletes the package. • Rename. Renames the package.
    • SSIS Manageability - dtutil • SSIS Package Management Utility (dtutil) SSIS contains the Package Management Utility (dtutil) that you can run from a command line. The dtutil command prompt allows you to manage SSIS packages that are stored in the database or the file system. You may specify to copy, move, delete, or verify the existence of a package using the appropriate command prompt options.
    • SSIS Manageability - dtutil • To copy a package that is stored in the msdb database on a local instance of SQL Server using Windows Authentication to the SSIS Package Store, use the following syntax: dtutil /SQL srcPackage /COPY DTS;destFolderdestPackage • To copy a package from a location on the File system to another location and give the copy a different name, use the following syntax: dtutil /FILE c:myPackagesmypackage.dtsx /COPY FILE;c:myTestPackagesmynewpackage.dtsx • To copy a package on the local file system to an instance of SQL Server hosted on another computer, use the following syntax: dtutil /FILE c:sourcepkg.dtsx /DestServer <servername> /COPY SQL;destpkgname • To create a new ID for a package after it is copied, use the following syntax: dtutil /I /FILE copiedpkg.dtsx • To create a new ID for all the packages in a specific folder, use the following syntax: for %%f in (C:testSSISPackages*.dtsx) do dtutil.exe /I /FILE %%f
    • SSIS Manageability - dtutil • To determine whether a package exists in the msdb database on a local instance of SQL Server that uses SQL Server Authentication, use the following syntax: dtutil SQL srcPackage /SOURCEUSER srcUserName /SOURCEPASSWORD *hY$d56b /EXISTS • To move a package that is stored in the msdb database on a local instance of SQL Server that uses SQL Server Authentication to the msdb database on another local instance of SQL Server that uses SQL Server Authentication, use the following syntax: dtutil /SQL srcPackage /SOURCEUSER srcUserName /SOURCEPASSWORD $Hj45jhd@X /MOVE SQL;destPackage /DESTUSER destUserName /DESTPASSWORD !38dsFH@v • To sign a package that is stored in a SQL Server database on a local instance of SQL Server that uses Windows Authentication, use the following syntax: dtutil /FILE srcPackage.dtsx /SIGN FILE;destpkg.dtsx;1767832648918a9d989fdac9819873a91f919 • The following sample encrypts the file-based PackageToEncrypt.dtsx to the file-based EncryptedPackage.dtsx using full package encryption, with a password. The password that is used for the encryption is EncPswd. • dtutil /FILE PackageToEncrypt.dtsx /ENCRYPT file;EncryptedPackage.dtsx;3;EncPswd
    • SSIS Manageability Stored Package Configuration Demo
    • SSIS Manageability - package • Checkpoint Restart – recover from a package failure; information about the packages execution and status is written to a checkpoint file. The Data Transformation Run-time engine can process the checkpoint file and the package can be restarted from the container that was being processed at the point of failure • Transaction Support – SSIS provides full support for database transactions. If enabled and the package fails that contains data flow tasks, all of the database update actions performed will be rolled back. • Transformation Error Handling – you can choose to ignore bad data or you can choose to redirect the row to a file or table for review and later processing; setup a central database for a central point of auditing and review. • Event Handlers – provide the ability to raise and handle events raised by containers and tasks. These could be used to send e-mail notifications, create logging information, or remove temporary files or database objects. • Execute Package Tasks – provide ability to run one package from another package; pass information from parent package to child package and setup a master package to control the flow of other packages.
    • SSIS Manageability - package Event Handlers Demo
    • SSIS Manageability - package Create smaller packages • Pros – less complex child packages – call them from multiple parent packages – simplifies development process – promotes reusability and manageability
    • SSIS Manageability – package execution Scheduling Package Execution – SQL Server Agent • Requires SQL Server installation • Setup credentials • Setup proxies • Create SQL job – Windows Task Scheduler • Setup user account for task – 3rd party scheduler
    • SSIS Manageability – package execution SQL Server Agent Job Demo
    • SSIS Manageability – package execution Remote Package Execution – SQL Server Agent job (sp_start_job) – Execute SQL Server Agent Job task (SSIS task) – .NET Code (SSIS Object Model) – Web Service (IIS – ASP.NET) – HTTP Endpoint – WCF application
    • SSIS Manageability – package execution Remote Package Execution Demo
    • SSIS Manageability – package logging • The SQL Profiler Log Provider - The SQL Profiler log provider enables you to create logs that can be opened using the SQL Profiler. • The SQL Server Log Provider - Using the SQL Server log provider enables you to capture SSIS package run-time data in the sysdtslog90 or sysssislog table of the database. • The Windows Event Log Provider - writes entries to the Application log in the Windows Event log on the local computer. • The XML File Log Provider - writes log files to an XML file and the default file extension is .xml. • The Text File Log Provider - writes log entries to ASCII text files in a comma-separated value (CSV) format. The default file name extension for this provider is .log. • Custom Log Provider - You can create your own custom log provider to obtain information to accommodate your business needs. Using a custom log provider allows you to integrate SSIS logs within your applications.
    • SSIS Manageability – package logging Events Description OnError Writes a log entry when an error occurs. OnExecStatusChanged Writes a log entry when the execution status of the executable changes. OnInformation Writes a log entry during the validation and execution of an executable to report information. OnPostExecute Writes a log entry immediately after the executable has finished running. OnPostValidate Writes a log entry when the validation of the executable finishes. OnPreExecute Writes a log entry immediately before the executable runs. OnPreValidate Writes a log entry when the validation of the executable starts. OnProgress Writes a log entry when measurable progress is made by the executable. OnQueryCancel Writes a log entry at any juncture in the task processing where it is feasible to cancel execution. OnTaskFailed Writes a log entry when a task fails. OnVariableValueChanged Writes a log entry when the value of a variable changes. OnWarning Writes a log entry when a warning occurs. For each data flow component, writes a log entry for each phase of validation and execution. The log entry specifies PipelineComponentTime the processing time for each phase. Writes a log entry that provides diagnostic information. Diagnostic For example, you can log a message before and after every call to an external data provider.
    • SSIS Manageability – package logging
    • SSIS Monitoring •SQL Server Management Studio •Logging •Notifications •Performance Monitor •SSIS Service •SSIS Pipeline •PowerShell •WMI
    • Today‟s Agenda • SQL Server Integration Services Introduction and Architecture • Team Development, Deployment and Configuration • Security and Manageability • Summary and Q&A
    • Q&A Any Questions?
    • Resources SQL Server 2008 – Integration Services Portal Page http://msdn.microsoft.com/en-us/sqlserver/cc511477.aspx SQLIS.com - SQL Server Integration Services - SSIS http://sqlis.com Jamie Thomson (UK) – SQL Server MVP – SSIS and forum moderator http://blogs.conchango.com/jamiethomson/ SQL Server 2008 Upgrade Technical Reference Guide http://www.microsoft.com/downloads/details.aspx?FamilyID=66d3e6f5-6902-4fdd-af75-9975aea5bea7&displaylang=en Channel9 MSDN BI Screencasts http://channel9.msdn.com/Showforum.aspx?forumid=38&tagid=277 SQL Server Best Practices http://msdn.microsoft.com/en-us/sqlserver/bb671432.aspx Microsoft Virtual Labs (TechNet and MSDN) http://www.microsoft.com/events/vlabs/default.mspx Microsoft BI Site http://www.microsoft.com/bi/ Magenic Blogs http://blog.magenic.com/blogs
    • Contact Information – Thank You! Contact us to find out how your business can benefit from a complimentary strategy session with one of our consultants and look into one of our BI quickstart engagements. Dan - http://denglishbi.spaces.live.com David - Magenic - info@magenic.com