DDD 2016: How to speed up .NET and SQL Server web apps (Reading, UK)
•Download as PPTX, PDF•
1 like•631 views
Talk on performance tuning .NET and SQL Server web apps, with a focus on middle tier, service, and database issues. This version of the talk does include a short discussion of client side issues, since these are becoming more important, particularly when targeting mobile devices and tablets.
Recommended
More Related Content
Similar to DDD 2016: How to speed up .NET and SQL Server web apps (Reading, UK)
Similar to DDD 2016: How to speed up .NET and SQL Server web apps (Reading, UK) (20)
Recently uploaded
Recently uploaded (20)
DDD 2016: How to speed up .NET and SQL Server web apps (Reading, UK)
- 1. How to speed up .NET and SQL Server web apps Bart Read
- 2. Who am I? • Bart Read, Web, Database, and Mobile Performance Consultant • Previously worked for Red Gate • Contacts • E: bart@bartread.com • W: www.bartread.com • B: www.bartread.com/blog • T: @bart_read • GH: https://github.com/bartread
- 4. Always measure! (But it’s almost never the code!)
- 7. Other Handy Ancillary Tools
- 8. A word of warning about SSMS Read Erland Sommarskog’s post, “Slow in the application, fast in SSMS?”: http://www.sommarskog.se/query-plan-mysteries.html
- 9. Scenario 1 The Slow Support Centre
- 10. Customer Support Centre (MS Dynamics)
- 28. Grant Fritchey’s FREE eBook on execution plans: http://www.red- gate.com/community/books/s ql-server-execution-plans-ed-2 Tool: http://www.supratimas.com/
- 32. Scenario 2 The Memory Leak in the eCommerce Site
- 33. eCommerce Site (MS Dynamics) Other technologies: - NServiceBus - Endeca
- 40. Scenario 3 Locking, Blocking and Deadlocking
- 41. Locking, Blocking & Deadlocking
- 43. Concurrency • Object level concurrency • Row level concurrency
- 45. Transaction Isolation Levels • Read Uncommitted • Read Committed <- SQL Server default • Repeatable Read • Snapshot • Serializable ACID
- 48. Locking pattern • Reads • A -> B -> C -> D -> A -> B -> C -> D -> E -> A … • Etc. • Writes • A –> B -> C -> D -> E -> A -> B -> A -> B -> C -> D … • Etc.
- 49. Solutions • DEADLOCK_PRIORITY • http://msdn.microsoft.com/en-us/library/ms186736.aspx • SET DEADLOCK_PRIORITY LOW | NORMAL | HIGH • Read Committed Snapshot Isolation • http://www.brentozar.com/archive/2013/01/implementing-snapshot-or- read-committed-snapshot-isolation-in-sql-server-a-guide/ • Disciplined ordering of object access • Unit of Work Pattern • Martin Fowler • http://martinfowler.com/eaaCatalog/unitOfWork.html
- 50. Scenario 4 Networking Stuart Cheshire, “It’s the latency, stupid” (1996): http://www.stuartcheshire.org/rants/latency.html
- 51. Fast/Slow Query
- 55. Query Frame
- 56. Result Set Frame
- 57. Virtual NIC Setting To Blame
- 58. Client-side Scenarios - General Observations • Payload • Chattiness/latency • CSS transitions and animations • Code execution time again becomes an issue • Garbage • High memory usage
- 60. Other Scenarios • Indexes • Missing indexes • Maintenance • Statistics • NTEXT (etc.) preventing online rebuilds • http://www.bartread.com/2014/09/17/convert-all-ntext-columns-to-nvarcharmax-sql- server/ • Stored procedures and cached execution plans • WITH RECOMPILE
- 61. Questions? E: bart@bartread.com W: www.bartread.com B: www.bartread.com/blog T: @bart_read GH: https://github.com/bartread Red Gate coding challenge: http://www.red-gate.com/our-company/entrypage/coding-challenge
Editor's Notes
- With this talk I’m going to talk you through the kind of diagnostic techniques I use for isolating performance problems in .NET and SQL Server web apps. I’m going to start off with an overview of the tools I use, and then go through some scenarios illustrating how to use them. I’m going to focus mostly on the server side but, obviously, nowadays with the rise of the single page app, and mobile devices, the client has become a lot more important performance-wise for a good user experience. How many of you are writing client-side code using a framework such as Angular, React, Knockout, Durandal, Ember or whatever? How many of you are building apps that you expect people to use from mobile devices and tablets? Obviously, particularly for these, performance and resource usage is critical. You often find you can get away with murder in desktop Chrome whereas on a phone your app runs like an absolute dog. I’ll talk about this a little at the end if there’s time.
- Ideally you want to be monitoring production performance of your app using appropriate tools. System wide: New Relic App Dynamics DynaTrace Stackify SQL Server SQL Monitor Solar Winds Spotlight
- Obviously I’m going to tell you that ANTS Performance Profiler and ANTS Memory Profiler are the best for this and, on the client side, you’re obviously going to use Chrome Dev Tools, which includes some extremely good performance and memory profiling functionality. I’m not going to talk much about the client but, if we have time at the end, I’ll mention a couple of things. The problem with client side performance is the common issues are so heavily dependent on which framework you’re using (if any). If you identify a problem with SQL you’re probably going to want to pull that out into SQL Server Management Studio and investigate in more detail, although you can now see execution plans in ANTS Performance Profiler, which arguably provides a better visualisation. The venerable SQL Profiler
- I’ve included a couple of load testing tools. Neustar BlazeMeter These tools allow you to script out tests where you’ll concurrency submit large numbers of requests to your site or API to investigate its scaling behaviour. The view they give you tends to be quite coarse-grained so, if you do see problems you’ll want to These are things you really want to use pre-emptively to shake out problems on staging or UAT instances. If you find your site doesn’t scale well under load then you need to re-run your load test whilst running a detailed diagnostic tool like ANTS Performance Profiler.
- Sometimes a query is just slow, in which case no worries. Sometimes, however, it runs slow in the application but fast in SQL Server Management Studio, even with the exact same parameters used in the application. When SQL Server first executes a query, it will create an execution plan for that query and place it in the plan cache. For stored procedures and parameterised SQL it will sniff the parameters, and use the statistics in the tables queried, to build what it thinks is the best execution plan. It obviously has no idea whether the parameters supplied are representative of normal operation. The next time you execute the query, perhaps with different parameter values, it will reuse the cached execution plan. Perhaps it will perform well, perhaps not. That interface for that cache is not dissimilar to an associative array. There are keys and values. The values are obviously the execution plan. Each value is accessed using a number of cache keys. These cache keys not only include the query text or stored procedure name, but also the user, and a bunch of SET options that were in force at the time the query was executed. If there is ANY mismatch SQL Server will generate a new execution plan and put that in the cache using the keys for that execution. This explains why you see queries that run slowly in your application and yet THE EXACT SAME QUERY WITH THE EXACT SAME PARAMETERS will run fast in SQL Server Management Studio. A detailed discussion here is beyond the scope of this talk, but do yourself two favours: Uncheck SET ARITHABORT in SSMS Read Erland Sommarskog’s article: http://www.sommarskog.se/query-plan-mysteries.html Also, if you’re using parameterised queries, you must use EXEC sp_executesql N’query’, N’parameter declarations’, @param = value, … Importantly you MUST NOT change the formatting of the SQL, not even by a single space, or you may get a different execution plan.
- Remember what I said earlier: this can affect the execution plan. If something is fast in SSMS and slow in the app it means different execution plans are being used and you need to be more careful to make sure execution conditions are exactly as they are in the app.
- You DON’T need to do this any more, in general, but it can still be handy if you’re just investigating slow queries from the SQL Server perspective using SQL Server profiler. I did this for a client a while ago: first by filtering queries that were taking more than 5000ms, optimising them, then dropping the time, optimising again, etc.
- It’s often worth checking Show All Columns because then you can apply more selective column filters. Generally I tend to uncheck everything except RPC:Completed, SQL:BatchCompleted, and SQL:BatchStarting. As I say, it’s also a good idea to apply column filters, especially if you’re running this against a production instance, so you’re not overwhelmed with data.
- I tend to filter on ApplicationName, and DatabaseName, at the very least. I often filter on duration.
- As you can see here, I’ve isolated an instance of my long running query, with the relevant parameter value.
- Worth noting that you can save results into a trace table in SQL Server. This is literally just a database table that the profiler will write the data that it collects into. The advantage with this is that you can then use SQL to query the table anyway you like without being limited or frustrated by SQL Profiler’s slightly clunky interface. This means you can potentially be much less selective about data you collect, and therefore avoid accidentally filtering out something important. The flip side is you can end up taking up a lot of space with data in this table.
- Here’s a shot showing the table that SQL Profiler has created.
- So now, having found a parameter value, we need to include the execution plan.
- This is a really simple example of an execution plan. As you can see, the diagram in SSMS is a bit primitive. You can also use a tool like Supratimas online to visualise your plan. These are clearly really simple plans. The reality can often be a lot more intimidating, with dozens of nodes spewed out in a tree spanning multiple screenfuls of data. One of the things you can do at this point is try to break down your complex query into a set of smaller queries and look at their execution plans individually. Bear in mind that you can be misled by doing this though, because you obviously won’t be seeing the original execution plan for the whole query. Worth pointing out that one nice thing the SSMS view does is that it will often tell you where missing indexes are the problem. Do take its advice with a pinch of salt though. You may just need to add a column to an existing index rather than create a new one. If you just blindly follow its advice it’s easy to end up with a table that has 25 indexes against it, which increases storage requirements and can degrade performance on its own.
- I’ve anonymised the name, but equally, I’ve given it a name that gives you a clue as to what’s going on.
- I mentioned that nowadays you don’t need to go through this process in SSMS the way you used to. Here you can see a shot of ANTS Performance Profiler displaying an execution plan for a SQL query. The profiler includes a view of SQL queries, along with how long they took, and their parameters, so you can immediately re-execute a slow query with the correct parameters and see the execution plan. Note that to view the execution plan APP needs to re-run the query (it doesn’t collect plans automatically as this would be very expensive), so it will ask you for the database connection details, including credentials. Make sure these match those used by your application, if possible, to ensure that the same execution plan is used when the query is re-run.
- I also had batch started and batch completed selected further down. It’s more information than I needed to collect but, remember, like I said earlier you can export this to a database table to make it easier to work with. Generally I’d always err on the side of too much information rather than too little just because it’s a pain if you have to go back and run another profiling session later, especially if you’re in a situation like I was where access to production infrastructure is very tightly controlled.
- With this we had a query that was executing fast some of the time (see the bottom few rows) and then slow some of the time. This was weird. Bit of background: SQL Server running on bare metal – powerful box: 24cores, 128GB RAM. Web hosts running on VMs hosted in VMWare VSphere. 10GB connection between them; same rack in data centre.
- Obvious place to start is to have a look at what’s going on on SQL Server. Here’s the view from SQL profiler for that query. At this point I’ve created a test case isolating that query. Note that CPU, reads, writes, and duration are all consistent. Note, importantly, that duration is 0. At this point one starts to suspect a network issue but this had already been raised with the infrastructure provider and they’d said there was very low load on the network. Well we sort of know that but, IT’S THE LATENCY, STUPID. The problem is, of course, I have to prove this to them because they’re not going to lift a finger until I’ve done so.
- Well, here’s the same query, isolated in a test case, run on the web host over and over again from Nunit with a couple of different parameters. (Note that this problem does not manifest itself on dev boxes, only in test environments and on production.) One always returns 1 row, the other 0 rows. There’s no correlation between execution time and number of rows returned. You can see in this case that the query is mostly slow, but it was sometimes fast most of the time. Very inconsistent. At this point I wanted to run Wireshark both on the web host and on the SQL Server instance. Trouble is, every time I fired up Wireshark on the web host, the problem goes away and doesn’t come back until a reboot (apparently). Still, I did manage to get a capture from the SQL Server host.
- SQL Server has transmitted this back less than 0.25ms after the query frame is received. Thus, the problem is nothing to do with SQL Server and something to do with the connection in between the VM host and the SQL Server host.
- This also explains why running Wireshark fixes the problem. It disables RSC: https://ask.wireshark.org/questions/9074/does-wireshark-capturesupport-rsc-packets And we weren’t the first to run into this issue: https://ask.wireshark.org/questions/50123/win-2012-r2-vm-network-much-faster-after-wireshark-was-started
- Payload matters, particularly on mobile, particularly not on WiFi. - Do you really need that library or framework? Do you really need those massive images, or can you use smaller ones? Keep your APIs coarse grained to reduce chattiness. Bandwidth is often OK, even on mobile, even over 3G, but latency is more of an issue. Chatty, fine-grained protocols optimise for latency, not bandwidth, which is a bad strategy where latency is significant! It’s the latency, stupid! CSS transitions and animations do not always work well on mobile devices; consider simplifying them for these scenarios. Feature-based development is obviously the best way to go (e.g., modernizr.js), but you can run into problems where different platforms support the same feature but there are small implementation differences or, sometimes, large performance differences. Actual time spent running code becomes an issue, particularly with heavyweight front-end frameworks. Angular 1.x and its two-way binding/digest cycle spring immediately to mind here, mostly because I’ve had a lot of experience with it, but all Fx have their issues. These issues become particularly acute on mobile devices, which don’t have the processing power. Problems here can be more noticeable on Android devices than on iOS – it pains me to admit it, but the JIT is simply better; plus single core performance tends to also be better on iOS. JavaScript is a managed memory environment with garbage collection, just like .NET. In fact the implementations are remarkably similar. You probably won’t notice too many problems with garbage collection on desktop and laptops – you can get away with murder. If you have a very interactive, or just complex page, and you’re creating lots of shortlived objects, you certainly can see problems on mobile though. Again, in my experience these tend to be more obvious on Android. Object pooling can be a good strategy here. High memory usage may never show up as a performance issue on desktop, although your users may not thank you if you have a page that swallows 100MB of memory, but on mobile this really can cause issues, particularly when coupled with creating lots of garbage, because it means the garbage collector has more memory to check – it has to work harder – when it does a full collection.
- The JS Heap shows a sawtooth pattern typical of garbage collected environments. This in itself isn’t cause for concern. What IS cause for concern is the size of the teeth, which represent 5-6MB of memory consumed and freed. The way this manifests itself on Android, from the user’s perspective, is with really choppy framerate after playing the game for 30 seconds or so (usually enough to get a few levels in).