SQL Server Integration Services

8,320 views

Published on

The presentation from my talk on what is SQL Server Integration Services (SSIS). It looks at it in Microsoft SQL Server 2008.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
8,320
On SlideShare
0
From Embeds
0
Number of Embeds
99
Actions
Shares
0
Downloads
445
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • PREP:SQL Server BIDS – Demo Solution openSQL Server Management Studio with cleanup scriptZoomITMake sure the solution is in 64bit mode!
  • Create a new packageAdd a data flow componentAdd a flat file connection – set it to the exercise.csv data. So suggested data times.Add a flat file source – bind to connectionAdd conditional splitLink to flat fileSplit on distance > 0Add sortLink to main split outputSort on dateAdd ADO.NET data destination connection manager – to spacedata.exerciseAdd ADO.NET DestinationLink to sort and connection managerDo mappingAdd Variable-Make sure scope is packageAdd Row countLink to conditional split elseLink to variableOn Control flowAdd SMTP taskSMTP connection to webmail.bbd.co.za and windows auth-Message body expression from file and change variableSave -> Run -> CrashChange distance to float on flat file connection, trickle changesSave -> Run -> Email
  • Show completed solution data flow
  • At a simple level moving data around sounds very easy, but when you break it down, it gets more and more complex and different needs requires different tools.SSIS is not a swiss army knife. It is a specialist tool.
  • SSIS adds layers of support, logging etc... These all add a performance hit and you should not waste time using it for things where a quick BCP or even c# code would be better.SSIS is perfect for batch solutions, it is BAD for near real time solutions. There are tools built into SQL server (linked servers) and other tools (BizTalk) which handle real time well.SSIS is not a SOA/ESB/B2B tool it is a ETL tool.
  • Lookup PerformanceThe ability to take rows that do not have matching entries in the reference dataset and load those rows into the cache.The ability to use separate data flows to load the reference dataset into the cache and to perform lookups on the reference dataset.Data profiling is used to profile the data in SQL server to see the quality/type of data. Useful for identifing improvements and fixing bugsThere is a tool to convert to the new format.
  • Use management studio script to clean data outAdd Excel SourceDelete flat file sourceConnect Excel source to split inputOpen excel source, create new OLE DB connectionTrickle changesSave -> Run -> CrashProject -> Properties -> Debugging -> Run64bitRuntime -> FalseSave -> Run
  • Record set info: http://blogs.conchango.com/jamiethomson/archive/2006/06/28/SSIS_3A00_-Comparing-performance-of-a-raw-file-against-a-recordset-destination.aspxMemory – Make sure you have enoughSELECT * - all that meta data and unused columns need processing!Small packages – You can call one package from another. Allows work to be broken up, which means a team can easily work on the solution, makes fault finding easier, lowers over headsComments – DUH!Understand the components is key to super usage – Many can be used to do the same thing. Lookup and Merge Join for instance can both be used to lookup data. Lookup has three modes which imposed performance vs. Memory trade offs where merge join does not. Merge join requires sorted input while lookups don’t. Execute SQL allows any SQL dialect while execute T-SQL allows only T-SQLBecause things can be put into parallel that doesn’t mean they execute in parallel. Some are async and some aren’t!
  • SQL Server Integration Services

    1. 1. SQL Server integration services<br />
    2. 2. What is it?<br />Microsoft’s ETL solution bundled with SQL Server<br />E – Extract<br />T – Transform<br />L – Load<br />Destination<br />Source<br />Write<br />Read<br />SSIS<br />
    3. 3. Demo<br />A simple ETL Demo<br />
    4. 4. Demo<br />You can get a little crazy...<br />
    5. 5. Why?<br />
    6. 6. Confusion reigns!<br />
    7. 7. Reasons Not Consider SSIS<br />Performance<br /><ul><li>Consider SQL or BCP for simple imports
    8. 8. File system performance</li></ul>Data Latency<br /><ul><li>SSIS is not a near real time solution</li></ul>SOA, ESB, B2B Integration<br /><ul><li>No business rules support
    9. 9. Very basic queue support
    10. 10. XML support limited</li></li></ul><li>Reasons To Consider SSIS<br />Merging Data from Heterogeneous Data Stores<br />Populating Data Warehouses<br />Cleaning and Standardizing Data<br />Building Business Intelligence into a Data Transformation Process<br />Automating Administrative Functions and Data Loading<br />
    11. 11. What’s New in 2008<br />Lookup transformation performance improvements and new caching options<br />ADO.NET Source and destination components<br />Data profiling task and viewer<br />Wizard interface for defining source and destination<br />Scripts (for the Script Transform) are now done in Visual Studio and thus in .NET languages.<br />New package format<br />Three new data formats for working with times<br />
    12. 12. x64 Limitations<br />Command line tools (dtexec, dtutil) cannot co-exist with 32bit versions.<br />No DTS support.<br />Limitations on data providers – No Access, Excel or SQL Compact<br />IA64 has more limitations including no designer support<br />
    13. 13. Demo<br />Running in 32bit on 64bit!<br />
    14. 14. What to Watch Out For?<br />RecordSet Destination<br /><ul><li>SLOW (5 times than a raw file!)</li></ul>Memory<br /><ul><li>SSIS is an in memory process.</li></ul>SELECT *<br /><ul><li>Exceptionally bad in SSIS</li></ul>Use many small packages<br />Comments!!!<br />Understand the components<br /><ul><li>Many do the same things in different ways with different trade offs
    15. 15. Lookup vs. Merge Join or Execute SQL vs. Execute T-SQL
    16. 16. Understand which components run asynchronously and which run synchronously</li></li></ul><li>Parallelism Example<br />
    17. 17. Storage: Files or Servers<br />Details http://www.sqljunkies.com/WebLog/knight_reign/archive/2005/05/05/13523.aspx<br />
    18. 18. References<br />http://www.microsoft.com/technet/prodtechnol/sql/2005/technologies/ssisperfstrat.mspx<br />http://ssisblog.replicationanswers.com/2008/01/04/when-to-not-use-ssis--directory--file-iterations.aspx<br />http://marcusrosen.blogspot.com/2008/04/sql-server-2005-integration-service.html<br />http://www.eggheadcafe.com/software/aspnet/32465151/linked-servers-versus-ssi.aspx<br />http://blogs.conchango.com/jamiethomson/archive/2006/06/28/SSIS_3A00_-Comparing-performance-of-a-raw-file-against-a-recordset-destination.aspx<br />http://www.microsoft.com/technet/prodtechnol/sql/2005/ssisperf.mspx<br />

    ×