Successfully reported this slideshow.

Blazing Fast Queries - When Indexes Are Not Enough


Published on

If you want optimum performance in SQL Server, you have to use indexes. But what if you already implemented indexes and your solution is still slow or it doesn’t scale like you want? You may have to rethink the way you write your queries. How you write your queries is directly related to how you approach and solve your business problems, and you might think that the easiest way to solve them is with a procedural, row-by-row approach. But although cursors, loops, and user-defined functions (UDFs) might seem the easiest method, this "easier" way leads to slower performance, poor scalability, and more complex code that’s bug-prone and difficult to maintain. In this session, you’ll see how to use a set-based approach to solving problems, even those that most developers think are solvable only by using a procedural approach. We’ll then compare the two approaches to see the differences in terms of scalability, performance, and complexity. You'll be amazed at the boost in performance and maintainability you can give your applications!

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Blazing Fast Queries - When Indexes Are Not Enough

  1. 1. April 21 – 23. Neuss, Germanyw ww.sqlpass .org/eu 2 010Blazing Fast QueriesWhen Indexes Are Not EnoughDavide Mauri, Solid Quality Mentors,
  2. 2. Blazing Fast QueriesAbout MeMCDBA, MCAD, MCTMicrosoft SQL Server MVPWorks with SQL Server from 6.5Specialized in BI, Database TuningPresident of UGISS (Italian SQL Server UG)Mentor @ Solid Quality MentorsItalian Subsidiary
  3. 3. Blazing Fast QueriesA developer storyI’ve started as a developer, I hated SQL….…at some point my applications started to showperformance and scalability problems...(more successful they was more problems I had)…I started to optimized the application finding clever ideasto gain performances…the application became more and more complex tomaintain(and thus much less cost effective and profitable)
  4. 4. Blazing Fast Queries…and the story endsAt some point I decided that maybe I should check ifsomething can be done also on the database….…I started learning database and SQL….…and I discovered that using correct indexes and writingcorrect queries I could gain more performance andscalability than what I can ever dream of!So, now everyone told you about indexes….I’ll tell youabout queries and the fabulous secret technique of….
  5. 5. Blazing Fast QueriesSet Based ThinkingThinking in Sets allows you to leverage all the power ofdatabase engineOptimizer is clever then what you can imagine! Thinking in Sets means that you have to look at theproperties that a set has and that you can use to solveyour problemThinking in Sets means that we can find the solution froma purely logical perspective and then translate it into SQLcommandsAfter all SQL is Logic (cit. Itzik Ben-Gan)
  6. 6. Blazing Fast QueriesSet Based ThinkingThinking in Sets means that you can use a declarative approachinstead of a procedural oneAsk to the database what you want, not how to get itLet the optimizer do his work! We have a problem:Row-by-Rows solutions are usually what comes to our mind as a firstsolutionSince we’re used to think at how we can solve the problemwith the programming language we will use to implement it
  7. 7. Blazing Fast QueriesSet Based ThinkingSo, we start to implement this first-on-mind solution immeditaly......and then fix all the problems that arises from itIt does not scaleIt does not offer performanceSolution:Stop doing this brute-force-approach and try to find the best algorithm(independent from programming language) that can solve theproblemUnlearn to think row-by-row! 
  8. 8. Blazing Fast QueriesSet Based ThinkingAt the first time it’s difficult……but everything is difficult at the first time!This not:This is easy: But it should bequite clear who willwin on a competitionAll we need isExercising!
  9. 9. Blazing Fast QueriesLet’s start exercisingI’ll propose some typical real world scenarioAll problems derives from situation I faced duringconsultancyThen we’ll see how we can solve them using a purely set-based approachAnd then we’ll compare some of them to the “easier” row-by-row approach
  10. 10. Blazing Fast QueriesWhat is NOT a set-based approach?• Using cursors to iterate over all rows, of course• Calling a scalar UDF that reads rows from a table, foreach row in a SELECT statement• Calling a rowset UDF using CROSS/OUTER APPLY(except for INLINE UDFs)• Using a SELECT to define the value of a column for eachrow in a SELECT statement• WHILE Loops
  11. 11. Blazing Fast QueriesLet’s start! 1st problem: LoansKnowing the value of the requested loan, interests andnumber for rates, generate a row for each rate, from thebeginning up until the end of the loanLoan Value # Rates Payment Frequency % Interest€ 10.000,00 12 1 5,00%€ 20.000,00 12 6 6,00%€ 30.000,00 12 6 5,50%Rate Date€ 875,00 January-10€ 875,00 February-10€ 875,00 March-10€ 875,00 April-10€ 875,00 May-10€ 875,00 June-10€ 875,00 July-10€ 875,00 August-10€ 875,00 September-10€ 875,00 October-10€ 875,00 November-10€ 875,00 December-10
  12. 12. Blazing Fast QueriesLoansDemo
  13. 13. Blazing Fast QueriesGroups and Correlated ValuesGiven a list of machines statuses, tell the latest reportedstatus of each machineControllerABCController Status TimeStampA StandBy 1001B StandBy 1001C StandBy 1003D Working 1004A Working 1004B StandBy 1006D StandBy 1009D Working 1010D Working 1011
  14. 14. Blazing Fast QueriesGroups and Correlated ValuesDemo
  15. 15. Blazing Fast QueriesStocksGiven a table with stock transactions get theHigh, Low, Open, Close & Volume values for eachstock, each hour, each daySymbolID TransactionDateTime Price Volume1 2008-01-01 09:15:21.000 75.800 25891 2008-01-01 09:25:44.000 68.200 43861 2008-01-01 09:29:31.000 74.300 28371 2008-01-01 09:34:42.000 68.900 29371 2008-01-01 09:39:13.000 72.300 45131 2008-01-01 09:43:35.000 67.300 8381 2008-01-01 09:51:57.000 73.800 13801 2008-01-01 09:56:42.000 68.700 4190tran_hour from_datetime to_datetime symbol_id high low volume open close92008-01-0109:15:21.0002008-01-0109:56:42.000 1 75.800 67.300 23670 75.800 68.700
  16. 16. Blazing Fast QueriesStocksDemo
  17. 17. Blazing Fast QueriesPerformance Comparison47 426 75474079051044259805230100000020000003000000400000050000006000000700000010000 100000 1000000I/OReadsReads Set-BasedReads Row-By-RowI/OsRows
  18. 18. Blazing Fast QueriesPerformance Comparison134 97012534404303433355050001000015000200002500030000350004000010000 100000 1000000Time (msec)Time (msec) Set-BasedTime (msec) Row-By-RowTimeRows
  19. 19. Blazing Fast QueriesFree Seats / Free Warehouse SpacesGive a group of person of defined size, find in which row ina cinema (or theater, or stadium, or whatever ), there isenough free space to make the seat near each other
  20. 20. Blazing Fast QueriesFree Seats / Free Warehouse Spaces
  21. 21. Blazing Fast QueriesFree Seats – Row by Row3 4 5 6 7 9A. Set the “free seat counter” (FSC) to 1B. Start from the first free seat (n), store this number asFirst Free Seat (FFS)C. If next free seat is n+1 then increase the FSCD. If next free seat is not n+1, store it somehere along with FFSE. restart from point A1 2 5 1
  22. 22. Blazing Fast QueriesFree SeatsDemoRow By Row Solution
  23. 23. Blazing Fast QueriesFree Seats – Set BasedA. If SQL Server could see the picture above would be able to see thatthere are groups of free spacesThen we could simply use “GROUP BY”B. We need turn this “for-eyes-only” this property into somethingSQL Server can handle
  24. 24. Blazing Fast QueriesFree Seats – Set BasedA. Let’s enumerate all the seats (free and occupied)B. And then enumerate all the free seatsC. Do a simple difference….D. And now we can use a GROUP BY 1 2 3 4 5 6 7 8 9 10…1 2 3 4 5 6 7 …2 2 2 2 2 3 3
  25. 25. Blazing Fast QueriesFree SeatsDemoSet Based Solution
  26. 26. Blazing Fast QueriesPerformance Comparison20 36 176381363036098050001000015000200002500030000350004000030 300 3000I/OSet BasedCursor
  27. 27. Blazing Fast QueriesPerformance Comparison9 22 1138746935160500100015002000250030003500400030 300 3000Time (ms)Set BasedCursor
  28. 28. Blazing Fast QueriesValue PackingNow, from the same machines statuses table, we need to“pack” all the rangesController Status TimeStampA StandBy 1001B StandBy 1002B StandBy 1003B StandBy 1004A Working 1005A Working 1006A Working 1007B Working 1008B Working 1009A StandBy 1010A StandBy 1011B StandBy 1012Controller Status From ToA StandBy 1001 1001B StandBy 1002 1004A Working 1005 1007B Working 1008 1009A StandBy 1010 1011B StandBy 1012 1012
  29. 29. Blazing Fast QueriesValue PackingDemo
  30. 30. Blazing Fast QueriesTemporal ProblemsSet Based solution comes to the rescue also in temporalproblemsPacking ProblemsGet the minimum and the maximum date of validity ofsomethingAuditing or Temporal Denormalization ProblemsGiven tables that hold data with different validintervals, create a result that holds all the validcombination in time
  31. 31. Blazing Fast QueriesTemporal – Auditing/DenormalizationGive tables that contains validity period for attributes thatchanges of over time, generate a result that shows all thecombinations that existed at some point in timeValue From ToPhone 1 2007-12-19 2008-02-02Phone 2 2008-02-03 nowValue From ToAddresses 1 2007-12-01 2007-12-31Addresses 2 2008-01-01 nowPhone Number Changes Address Changes2007-12-01 2008-01-012007-12-19 2008-02-03Timeline
  32. 32. Blazing Fast QueriesTemporal Auditing / DenormalizationDemo
  33. 33. Blazing Fast QueriesConclusionsSet Based Thinking is hard….…but deserves our attention!And after all row-by-row solution is not that easy!We can spend more time in thinking instead of writingugly code We’ve been able to optimize our query making them up to30 times faster!Stop crawling, and start to run!Big customers will be very satisfied with us! 
  34. 34. Blazing Fast QueriesIs all Gold what shines?YES!!!…in Theory In 99.9% a set based solution is preferableBut in some very rare (one?) case is still not the bestThe problem is that the ACTUAL implementation of T-SQL in not complete and doesn’t allow to easily get“previous” or “next” valuesVote on Connect!!!
  35. 35. Blazing Fast QueriesAdditional ResourcesItzik Ben-Gan & Friends books:• SQL Server 2008 Fundamentals• SQL Server 2008 Querying• SQL Server 2008 ProgrammingThinking in Sets• Joe CelkoSQL and Relational Theory• Chris Date
  36. 36. Questions
  37. 37. Thank you!
  38. 38. Don’t forget to fill out yoursession evaluation forms