SQL Server Integration Services


Published on

This session provides an introduction to using SSIS. This is an update to my older presentation on the topic: http://www.slideshare.net/rmaclean/sql-server-integration-services-2631027

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Create a new packageAdd a data flow componentAdd a flat file connection – set it to the exercise.csv data. So suggested data times.Add a flat file source – bind to connectionAdd conditional splitLink to flat fileSplit on distance > 0Add sortLink to main split outputSort on dateAdd ADO.NET data destination connection manager – to spacedata.exerciseAdd ADO.NET DestinationLink to sort and connection managerDo mappingAdd Variable-Make sure scope is packageAdd Row countLink to conditional split elseLink to variableOn Control flowAdd SMTP taskSMTP connection to webmail.bbd.co.za and windows auth-Message body expression from file and change variableSave -> Run -> CrashChange distance to float on flat file connection, trickle changesSave -> Run -> Email
  • SSIS adds layers of support, logging etc... These all add a performance hit and you should not waste time using it for things where a quick BCP or even c# code would be better.SSIS is perfect for batch solutions, it is BAD for near real time solutions. There are tools built into SQL server (linked servers) and other tools (BizTalk) which handle real time well.SSIS is not a SOA/ESB/B2B tool it is a ETL tool.
  • Use management studio script to clean data outAdd Excel SourceDelete flat file sourceConnect Excel source to split inputOpen excel source, create new OLE DB connectionTrickle changesSave -> Run -> CrashProject -> Properties -> Debugging -> Run64bitRuntime -> FalseSave -> Run
  • Record set info: http://blogs.conchango.com/jamiethomson/archive/2006/06/28/SSIS_3A00_-Comparing-performance-of-a-raw-file-against-a-recordset-destination.aspxMemory – Make sure you have enoughSELECT * - all that meta data and unused columns need processing!Small packages – You can call one package from another. Allows work to be broken up, which means a team can easily work on the solution, makes fault finding easier, lowers over headsComments – DUH!Understand the components is key to super usage – Many can be used to do the same thing. Lookup and Merge Join for instance can both be used to lookup data. Lookup has three modes which imposed performance vs. Memory trade offs where merge join does not. Merge join requires sorted input while lookups don’t. Execute SQL allows any SQL dialect while execute T-SQL allows only T-SQLBecause things can be put into parallel that doesn’t mean they execute in parallel. Some are async and some aren’t!
  • SQL Server Integration Services

    1. 1. SSIS<br />
    2. 2. Microsoft’s ETL solution bundled with SQL Server<br />E – Extract<br />T – Transform<br />L – Load<br />What is it?<br />Destination<br />Source<br />Source<br />Write<br />Read<br />SSIS<br />
    3. 3. A simple ETL<br />
    4. 4. When to use Script<br />YES<br />NO<br />Can one or more components<br />do the same thing?<br />NO<br />YES<br />Use those<br />Unique situation<br />Build custom component<br />Use script<br />
    5. 5. Confusion<br />
    6. 6. Performance<br /><ul><li>Consider SQL or BCP for simple imports
    7. 7. File system performance</li></ul>Data Latency<br /><ul><li>SSIS is not a near real time solution</li></ul>SOA, ESB, B2B Integration<br /><ul><li>No business rules support
    8. 8. Very basic queue support
    9. 9. XML support limited</li></ul>Reasons not to consider SSIS<br />
    10. 10. Merging Data from Heterogeneous Data Stores<br />Populating Data Warehouses<br />Cleaning and Standardizing Data<br />Building Business Intelligence into a Data Transformation Process<br />Automating Administrative Functions and Data Loading<br />Areas SSIS is strong<br />
    11. 11. Command line tools (dtexec, dtutil) cannot co-exist with 32bit versions.<br />No DTS support.<br />Limitations on data providers – No Access, Excel or SQL Compact<br />IA64 has more limitations including no designer support<br />X64 Limitations<br />
    12. 12. Running 32bit on 64bit<br />
    13. 13. RecordSet Destination<br /><ul><li>SLOW (5 times than a raw file!)</li></ul>Memory<br /><ul><li>SSIS is an in memory process.</li></ul>SELECT *<br /><ul><li>Exceptionally bad in SSIS</li></ul>Use many small packages<br />Comments!!!<br />Understand the components<br /><ul><li>Many do the same things in different ways with different trade offs
    14. 14. Lookup vs. Merge Join or Execute SQL vs. Execute T-SQL
    15. 15. Understand which components run asynchronously and which run synchronously</li></ul>What to watch out for<br />
    16. 16. Parallelism Example I<br />
    17. 17. Parallelism Example II<br />
    18. 18. Parallelism Example III<br />
    19. 19. Source Control<br />Highest Encryption<br />Easy loading into designer<br />Access by multiple users<br />DB Roles, DTS Roles<br />SQL Backups<br />Able to filter packages<br />Files<br />Server<br />Storage Options<br />Details http://www.sqljunkies.com/WebLog/knight_reign/archive/2005/05/05/13523.aspx<br />
    20. 20. Scheduling a package<br />
    21. 21. No more often than 3x avg. execution time.<br />Settings in configuration files.<br />Enable logging (step) and notifications (job).<br />Execute signed packages only.<br />Do not make packages which execute themselves.<br />Scheduling Guidelines<br />
    22. 22. DTS Upgrade Options<br />Run DTS in 2005 or 2008<br />Missing the package logs<br />Runs under 32 bit <br />Upgrade using MS Wizard<br />Not compatible with most package<br />Upgrade using DTS xChange<br />Minutes per package<br />Starting from scratch<br />About 3-6 hrs per package conservatively<br />
    23. 23. Package Upgrade Wizard<br />Built into SQL Server 2008<br />Pros:<br />Free<br />Works on simple packages<br />Cons:<br />Does not handle ODBC<br />Only handles a few types of text file use cases<br />No Dynamic Properties Task<br />No UDL or legacy database support in data pump<br />Packages only have about a 20% chance of working<br />
    24. 24. Profiles DTS packages to help with a conversion project plan<br />Rapidly converts DTS Packages to SSIS (2005 or 2008) and applies SSIS best practices<br />Converts tasks that are not handled by the existing SQL Server conversion wizard<br />Includes a SSIS logging repository and reports for trending and alerting<br />Includes BI xPress for new SSIS packages<br />DTS xChangeUpgrade<br />
    25. 25. ActiveX Script Upgrade<br />Both tools mentioned migrate DTS ActiveX to ActiveX in SSIS<br />ActiveX migrates to SSIS but you would not want to keep it there and it may not run<br />Need for ActiveX Script Task has been replaced with built-in, easy to maintain SSIS tasks<br />File System Object = File System Task<br />Mail objects = Send Mail Task (now has SMTP)<br />ADO objects = Execute SQL Task<br />
    26. 26.
    27. 27. http://www.microsoft.com/technet/prodtechnol/sql/2005/technologies/ssisperfstrat.mspx<br />http://ssisblog.replicationanswers.com/2008/01/04/when-to-not-use-ssis--directory--file-iterations.aspx<br />http://marcusrosen.blogspot.com/2008/04/sql-server-2005-integration-service.html<br />http://www.eggheadcafe.com/software/aspnet/32465151/linked-servers-versus-ssi.aspx<br />http://blogs.conchango.com/jamiethomson/archive/2006/06/28/SSIS_3A00_-Comparing-performance-of-a-raw-file-against-a-recordset-destination.aspx<br />http://www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx<br />References<br />