Intelligent ETL with SQL Server<br />Jyoti Gupta, Senior Presales Engineer, expressor software<br />
About me<br />10 years of ETL experience<br />Most of my career as an ETL Presales Engineer<br />Currently at expressor so...
Agenda<br />~40 minutes<br />Modularity<br />Abstraction <br />Modularity and abstraction in ETL<br />~10 minutes<br />Q/A...
Topic 1: Modular Design<br />
When you think of “modular” design, what do you think of?<br />Snap ins<br />Components<br />Assembly<br />Who remembers “...
Let’s Explore the Fast Food Example<br />Basic Ingredients<br />Basic toppings<br />Menu Choice<br />Hamburger<br />Cheese...
Benefits of Modular Design<br />Combinability 					= 	meet the demands of more consumers<br />Adaptability					= 	earliest...
Drawbacks of Modular Design<br />Change one/change all is a double edged sword<br />Module management<br />Storing<br />Or...
Topic 2: Abstraction<br />
What is abstraction?<br />The thought process wherein ideas…<br />		are distanced….<br />		from objects….<br />
Abstractions simplify and organize<br />Orange<br />Orange<br />Pumpkin<br />Basketball<br />
The process by which data and programs are defined with a representation similar to its meaning (semantics)<br />Applies t...
Data Abstraction<br />
Action abstraction<br />
Modularity and Abstraction in ETL<br />
Overview of ETL<br />What is ETL?<br />Where does SQL Server fit in?<br />
Modularity in ETL<br />Which components of ETL apps can and should be modularized?<br />
Data Abstraction in ETL<br />Data type wrangling… ever felt like a cowboy/cowgirl trying to herd the unherdable? <br />Git...
Function Abstraction in ETL<br />Think about using abstract functions. Does your transformation language support abstracti...
Put it all together<br />Modular, abstracted ETL programming environments:<br />Modular connectivity objects<br />Modular ...
What else?<br />Think about how you are designing your ETL applications<br />How much of your code is dealing with data qu...
Conclusions and Deep Thoughts<br />You’re doing something wonderful when you’re doing ETL!<br />ETL is applied fundamental...
Professional Association for SQL Server<br />Thank you to our sponsor<br />
Save 25%: Register by April 12th<br />www.sqlpass.org/sqlrally<br />May 11-13, Orlando, FL<br />Register by March31st: sav...
Upcoming SlideShare
Loading in …5
×

24 hoursofpassppt jyotigupta_spring2011_fixed

505 views

Published on

expressor software presentation at the 24 Hour of PASS event on Mar 15 - 16, 2011.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

24 hoursofpassppt jyotigupta_spring2011_fixed

  1. 1. Intelligent ETL with SQL Server<br />Jyoti Gupta, Senior Presales Engineer, expressor software<br />
  2. 2. About me<br />10 years of ETL experience<br />Most of my career as an ETL Presales Engineer<br />Currently at expressor software<br />jgupta@expressor-software.com<br />
  3. 3. Agenda<br />~40 minutes<br />Modularity<br />Abstraction <br />Modularity and abstraction in ETL<br />~10 minutes<br />Q/A<br />Wrap up<br />
  4. 4. Topic 1: Modular Design<br />
  5. 5. When you think of “modular” design, what do you think of?<br />Snap ins<br />Components<br />Assembly<br />Who remembers “units” knitwear?<br />
  6. 6. Let’s Explore the Fast Food Example<br />Basic Ingredients<br />Basic toppings<br />Menu Choice<br />Hamburger<br />Cheeseburger<br />Cheese<br />Hamburger Patty<br />Double Cheeseburger<br />Chicken Sandwich<br />Hamburger Bun<br />Grilled Chicken Breast<br />Ranch Chicken Sandwich<br />Chicken Sandwich w/Cheese<br />Wrap<br />Ranch Chicken Sandwich w/Cheese<br />Fish Filet<br />Fish Sandwich<br />Ranch Dressing<br />Fish Wrap<br />
  7. 7. Benefits of Modular Design<br />Combinability = meet the demands of more consumers<br />Adaptability = earliest responder to changes in market conditions<br />Efficiency = lower cost of product<br />Consistency in (good) quality = lowers perceived risk and lowers “barriers to entry”<br />
  8. 8. Drawbacks of Modular Design<br />Change one/change all is a double edged sword<br />Module management<br />Storing<br />Organizing<br />Versioning<br />
  9. 9. Topic 2: Abstraction<br />
  10. 10. What is abstraction?<br />The thought process wherein ideas…<br /> are distanced….<br /> from objects….<br />
  11. 11. Abstractions simplify and organize<br />Orange<br />Orange<br />Pumpkin<br />Basketball<br />
  12. 12. The process by which data and programs are defined with a representation similar to its meaning (semantics)<br />Applies to<br />Data<br />Actions<br />Abstraction in computer science<br />
  13. 13. Data Abstraction<br />
  14. 14. Action abstraction<br />
  15. 15. Modularity and Abstraction in ETL<br />
  16. 16. Overview of ETL<br />What is ETL?<br />Where does SQL Server fit in?<br />
  17. 17. Modularity in ETL<br />Which components of ETL apps can and should be modularized?<br />
  18. 18. Data Abstraction in ETL<br />Data type wrangling… ever felt like a cowboy/cowgirl trying to herd the unherdable? <br />Git along li’lnvarchar<br />SQL Server: nvarchar<br />ODBC: SQL_VARCHAR<br />MySQL: <br />varchar<br />PostgreSQL: character varying<br />
  19. 19. Function Abstraction in ETL<br />Think about using abstract functions. Does your transformation language support abstraction?<br />Examples<br />Lua anonymous functions and closures<br />function makeaddfunc(x)<br />return <br />function(y) <br /> return x + y<br /> end <br />end <br />plustwo= makeaddfunc(2) <br />C# anonymous function<br />Func<int,int> foo = x => x*x; <br />
  20. 20. Put it all together<br />Modular, abstracted ETL programming environments:<br />Modular connectivity objects<br />Modular schema objects<br />Modular data types that can be abstracted<br />Modular functions that can be abstracted<br />Build your applications like a fast food hamburger<br />Mix n’ match applications are built efficiently<br />
  21. 21. What else?<br />Think about how you are designing your ETL applications<br />How much of your code is dealing with data quality issues that have to be fixed manually?<br />Null values<br />Invalid values<br />Invalid formats<br />Missing values<br />Data out of range or not in the proscribed range<br />Find those errors first (and make fixing them someone else’s problem)<br />It’s hard to predict the future, so adaptable and modular designs have long-terms benefits.<br />
  22. 22. Conclusions and Deep Thoughts<br />You’re doing something wonderful when you’re doing ETL!<br />ETL is applied fundamental computer science.<br />
  23. 23. Professional Association for SQL Server<br />Thank you to our sponsor<br />
  24. 24. Save 25%: Register by April 12th<br />www.sqlpass.org/sqlrally<br />May 11-13, Orlando, FL<br />Register by March31st: save 40% and have the chance to win a cruise to Alaska! <br />“24HR11” code gets you $100 off<br />www.sqlpass.org/summit<br />Oct 11-14, Seattle, WA<br />

×