SQL Server Days 2013 - Create ETL solutions faster with metadata driven development

4,523 views
4,225 views

Published on

Working on your 20th staging package? Do you need to import 30 different flat files? Don’t have the time to create all those packages?

By using Biml – Business Intelligence Markup Language – you can describe your BI solution in a declarative fashion. You can turn tedious, repetitive work into reusable scripts which generate SSIS packages for you, cutting back drastically on development time.

This session will introduce you to Biml and get you started on building reusable patterns for common BI tasks.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,523
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
164
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • Blue - Use for Cloud on Your Terms specific content
  • Green - Use for Mission Critical Confidence specific content
  • Orange - Use for Breakthrough Insight specific content
  • Blue - Use for Cloud on Your Terms specific content
  • SQL Server Days 2013 - Create ETL solutions faster with metadata driven development

    1. 1. CREATE ETL SOLUTIONS FASTER WITH METADATA DRIVEN DEVELOPMENT KOEN VERBEECK SQL SERVER DAYS 2013
    2. 2. WHO AM I?
    3. 3. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    4. 4. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    5. 5. INTRODUCTION
    6. 6. INTRODUCTION • large percentage of BI projects fail • Gartner - http://www.gartner.com/newsroom/id/492112 • one of the reasons is underestimating development effort ETL • Kimball: 70% time of building a DWH goes into ETL http://www.informationweek.com/the-38-subsystems-of-etl/55300422
    7. 7. INTRODUCTION “I choose a lazy person to do a hard job. Because a lazy person will find an easy way to do it” Bill Gates
    8. 8. INTRODUCTION • a lot of SSIS packages are very similar • • • • • packages importing flat files packages writing change data to staging tables packages exporting data to excel (for some reason) packages updating dimensions … • … but they take a lot of time to create
    9. 9. INTRODUCTION • solution? • code reuse o SSIS basically only supports copy-paste o copy-paste has improved in SSIS 2012 • design patterns o for example: incremental load package o SQL Server 2012 Integration Services Design Patterns • enable through templates o build a template package o save it to C:Program Files (x86)Microsoft Visual Studio 11.0Common7IDEPrivateAssembliesProjectItemsDataTransformationProjectDataTransformationIt ems • but still requires you to edit each package! • (and what if you forget to edit a crucial piece?)
    10. 10. INTRODUCTION • metadata driven development to the rescue! • (aka code generating code) • automate generation of common logic in SSIS packages • first option is the “dynamic SSIS package” 1. 2. 3. 4. reads metadata from tables generates code o usually outputs T-SQL or bcp commands o uses T-SQL or C# o for example: SELECT … INTO statements loops over the generated code executes each statement • disadvantages • • • • complex project no parallelism difficult row based error handling difficult to incorporate “business logic”
    11. 11. INTRODUCTION • second option: BIML • started as a project at MS: http://vulcan.codeplex.com/ • developer left to found company Varigence http://www.varigence.com/ o took the idea (not the code) and developed BIML • BIML is a markup language and compiler o translates metadata into business intelligence solutions for SQL Server o supports SSIS and SSAS o Varigence made part of BIML available as open source
    12. 12. INTRODUCTION • BIDS Helper has open source implementation of BIML • it’s free! • it’s already in the add-on you love! • it is available for SSIS 2005, 2008, 2008R2, 2012 (and 2014?) • BIML offers • powerful code generation o only some parts of the project deployment model are not supported • reuse BI patterns and components o create your pattern in BIML and generate all your packages with the same structure o BIML files can reference each other • .NET based script language o C# code can be incorporated into BIML to generate objects based on metadata o Intellisense (sometimes) available • don’t like BIML? o generated packages are just SSIS packages, you can edit them using BIDS/SSDT/SSDTBI o no vendor lock-in
    13. 13. INTRODUCTION • scenario for our demos • import different flat files o exports from ERP systems, other database vendors, 3rd party providers, … • each type of flat file has a different structure o no single SSIS package for all flat files • the name of the flat files can change o for example the name includes a timestamp • this would normally require 1 SSIS package per flat file type • couple of hours/days work? • let’s solve it with BIML!
    14. 14. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    15. 15. HELLO WORLD • basic BML script structure Tasks BIML Dataflow Connections FileFormats Packages Tasks Containers Precedence constraints Transformations You can also specify • events • log handlers • variables • parameters • custom tasks • script tasks/components • …
    16. 16. HELLO WORLD • let’s take a look at a simple BIML script
    17. 17. HELLO WORLD • BIML root node • add connections • add packages
    18. 18. HELLO WORLD • specify Tasks • specify specific properties
    19. 19. HELLO WORLD • check for errors & generate package • result
    20. 20. DEMO show Hello World BIML
    21. 21. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    22. 22. READ FLAT FILE • specify FlatFileFormat • columns: name, data type, size, delimiter (, code page) • what you’d normally specify in the flat file connection manager • specify connection
    23. 23. READ FLAT FILE • specify data flow with transformations • if no input/output connectors are specified, transformations are connected in the order specified in the BIML file • result
    24. 24. DEMO import flat file with BIML
    25. 25. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    26. 26. READ FLAT FILE IN LOOP • now let’s loop over a bunch of flat files • specify variables to hold path to current file and source folder • add an expression on the flat file connection manager
    27. 27. READ FLAT FILE IN LOOP • add a for each loop • which has its own tasks child element
    28. 28. READ FLAT FILE IN LOOP • result
    29. 29. DEMO import flat file using for each loop with BIML
    30. 30. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    31. 31. METADATA DRIVEN DEVELOPMENT • BIML is nice • … but isn’t the GUI much faster to developer packages? • time to enhance BIML with some C# goodness! called BIMLScript use C# to read metadata loop over metadata and create multiple objects entire website dedicated with tutorials and code snippets http://bimlscript.com/ • also has an online editor • • • •
    32. 32. METADATA DRIVEN DEVELOPMENT • Add namespaces • Declare variables
    33. 33. METADATA DRIVEN DEVELOPMENT • Retrieve metadata (stored in a SQL Server table) • Loop over metadata and create corresponding objects
    34. 34. METADATA DRIVEN DEVELOPMENT • result
    35. 35. METADATA DRIVEN DEVELOPMENT • remarks • make sure the code or the metadata doesn’t contain invalid XML characters o <>“& • using C# can mess with the Intellisense o Visual Studio thinks it’s not valid XML anymore o color coding can disappear > right click file and choose Open With… o Intellisense can stop working in Visual Studio > use online editor • beware of the protection levels • some elements can only appear once o do not put those in a loop o e.g. Connections, Packages
    36. 36. DEMO generate multiple packages using BIMLScript
    37. 37. OUTLINE Introduction Hello World Read Flat File Read Flat Files While Looping Metadata driven development Conclusion
    38. 38. CONCLUSION • BIML can radically reduce SSIS development time • for frequently used package patterns • when combined with BIMLScript • BIML supports all versions of SSIS • but some project deployment functionality is missing • bit of a learning curve • good understanding of SSIS is necessary • basic C# skills needed • return of investment is in next projects
    39. 39. RESOURCES • Official BIML • Varigence BIML product page http://www.varigence.com/Products/Biml/Capabilities • BIMLScript resource hub http://bimlscript.com/ • BIDS Helper on Codeplex http://bidshelper.codeplex.com/ • Blogs • Stairway to BIML by Andy Leonard http://www.sqlservercentral.com/stairway/100550/ • BIML articles by Joost van Rossum http://microsoft-ssis.blogspot.be/search/label/BIML • BIML articles by Marco Schreuder http://blog.in2bi.eu/tags/biml/ • BIML articles by John Welch http://agilebi.com/jwelch/tag/biml/ • Introduction to BIML part I by Koen Verbeeck http://www.mssqltips.com/sqlservertip/3094/introduction-to-business-intelligence-markup-languagebiml-for-ssis/
    40. 40. Q&A SQL SERVER DAYS 2013
    41. 41. THANKS FOR LISTENING! koen.verbeeck@element61.be @Ko_Ver http://www.linkedin.com/in/kverbeeck SQL SERVER DAYS 2013
    42. 42. © 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

    ×