Performance tuning

Enterprise Software Development

Eric Phan | Chief Architect @ SSW
Performance Tuning
from the Trenches
Join the Conversation #NetUG @EricPhan

Case Study
Optimizing .NET
Optimizing SQL
Optimizing Web Front End
Summary – Lesson’s Learnt
False Start
Agenda

B.E. - Software Engineering, Certified Scrum Master
Eric Phan
@EricPhan
 Chief Architect @ SSW
 MCTS TFS, MCP
 Loves playing with the latest
technology

Case Study – A large retailer
• Problems with their existing online shopping site
• Old, outdated, running on classic ASP
• Website crashed every time they had sales events
• Losing $$$$
• Volume starts increasing in the lead up to Xmas (from about late august)
• Physical shop + website closes after Xmas
• Massive reopening sale mid January (this is usually when the website crashes)

Case Study – A large retailer
• SSW was engaged to:
• Upgrade their site to a later version of their eCommerce platform
• Develop a few new features (gift registry, gift cards, integration with
their inventory tracking system)
• Ensure the site stays up during sales

Dev dev dev
• As Jan approaches, get ready for go live
• Sprint just focussed on performance
• Run MS web + load tests then optimize

Tweaking performance
• Run .NET profilers (ANTS, JetBrains)
• Run SQL Profiler
• Add caching

Send it live!

Crash and burn
• Roll back to old site (still crashes)
• Restart IIS constantly
• Now that we’ve missed this date, there’s more time to
dig deeper

Post Mortem
We concluded that:
• The web tests didn’t accurately reflect how users would use the
site
• There were a lot more users hitting the site than the initial
target (due to it being the big sale)
• The tests also didn’t accurately represent the production
environment

Then we embarked on a journey…
• Fix tests to more accurately represent the load
• Identify bottle necks
• Optimize!

Fixing the test
• Start at 200
• Ramp up to 5000
• New goal was 4000
concurrent users
• See the fail!

Identify bottlenecks
• Code profiling
• SQL profiling
• Hardware

No matter what we did

So we suspected something to do with
the Hardware and Hosting
• Simplified test to pull 1 image from the web server
(no DB, no .NET)
• Same problem!

Checking the IIS server
• Log onto the IIS server and monitor CPU
• Start Performance Monitor and add stats for:
• Users
• Sessions
• Throughput
• When we hit the 5 minute fail mark…
• No users, no session, no throughput logged in IIS…

Cause?
• DDOS protection

Rerun Load Tests
• This time, instead of failing within 5 mins,
• We are getting to 40mins before failing
• Log another ticket with the host

Progress!

What happened here?

Running the proper tests (not the single
image)
Now that the infrastructure issues have been sorted we
can run the actually test scripts
• Search for products
• Add 10 to shopping cart
• Go through checkout process

So this time..
• CPU + RAM are good on DB + IIS
• But page still aren’t being served?

Network saturation
• Hitting the max capacity of the network cards
between the web and SQL server
• Switching to gigabit fixed the issue
• But it’s still a lot of chatter!

Identifying Bottlenecks
• Now that we know our
infrastructure is good….
• How do we find where our
app is slow?

Identifying Bottlenecks
• User feedback
• Glimpse
• IntelliTrace
• APM tools
• Profiling

Glimpse
• Install-Package Glimpse
• F5
• ~/glimpse.axd
• g

Intelli Trace
• Good to quickly eyeball what’s going on under the
covers

Application Insights
https://rules.ssw.com.au/rules-to-better-application-
insights

Optimize SQL
1. Run SQL Profiler
2. Use the Duration template
3. Identify slow running queries

Analyse profile results
1. Look for anything that takes over 100ms to execute
2. Drill into each query and manually run it in SSMS
1. Use the Execution Plans and Client statistics to help
3. See if you can speed it up by
1. Adding an index
2. Rewriting the query

Remember this when tuning SQL
CHECKPOINT;
GO
DBCC DROPCLEANBUFFERS;
GO

SQL Execution Plans
Tells you where the bottlenecks are
• Fat pipes
• Scan vs Seek
• Estimated Rows vs Actual Rows
• Warnings

Execution Plan Resources
• https://www.simple-
talk.com/sql/performance/execution-plan-basics/
• https://www.simple-talk.com/sql/performance/why-
developers-need-to-understand-execution-plans/

Other things to look out for…
1. Excessive calls
• SQL profiler showed that to load the search page, it would make
300+ SQL calls
2. Stuff that doesn’t make sense
• What’s worse is that the search triggered 1 insert and 1 update
• This was one of the causes of the high network traffic

Parameter Sniffing
• exec procGetComplicatedStuff 1
• exec procGetComplicatedStuff 15
• Execution plan is cached
• Problem when there is uneven distribution of results based on the parameters
• https://www.brentozar.com/blitzcache/parameter-sniffing/
• https://www.brentozar.com/archive/2013/11/why-parameter-sniffing-can-slow-down-queries-video/
• https://www.brentozar.com/archive/2013/06/the-elephant-and-the-mouse-or-parameter-sniffing-in-sql-server/

Indexes and Index Usage
https://www.simple-talk.com/sql/performance/tune-
your-indexing-strategy-with-sql-server-dmvs/

Simulating live traffic
• Use profiler to capture a trace for a period of time
that represents a typical usage
• Run the trace through the Database Tuning advisor
• Recommended indexes
• Recommended Statistics

DB Optimization Summary
Find bottlenecks:
• SQL Profiler
Fix bottlenecks:
• Execution Plans
• Database Tuning Advisor

Optimize Queries
• SELECT only the columns you need
• Use JOINs effectively
• AVOID cursors like the plague
• Watch for Parameter Sniffing
• Keep database statistics up to date

Other tips
• Denomalize heavily accessed tables
• SSDs
• More RAM
• Separate physical drives for data and logs

Case Study
Optimizing .NET
Optimizing SQL
False Start

Finding bottlenecks
• Use profiling tools
• RedGate ANTZ
• Telerik JustTrace
• Visual Studio Performance Wizard
• Use APM tools on production
• NewRelic
• App Insights

Find the hotspots with Just Trace
Drill into the code identified by the profiling tool

Bad Example
foreach (var o in Obligations) {
foreach (var m in o.ObligationGroup.ObligationGroupMembers) {
Console.WriteLine(o.ObligationGroup.GroupName + " - " +
m.OrganisationEntityId);
}
}

What’s wrong with this code?

Watch out for LINQ Queries
• Easy to write
• Generates some nasty SQL
• Watch out for loops
• Does a SELECT * if you’re not projecting

Fixed
var groupMembers = Obligations.Select(x => new {
GroupName = x.ObligationGroup.GroupName,
EntityId = x.ObligationGroup.ObligationGroupMembers
.Select(m => m.OrganisationEntityId)
});
foreach (var m in groupMembers)
{
Console.WriteLine(m.GroupName + " - " + m.EntityId);
}

Debug = false
https://www.ssw.com.au/ssw/HealthCheck

Caching
• Cache data that doesn’t change often
• E.g. Product descriptions, product categories, menus
• Expire the cache when the data changes
• Why cache?
• Reduce calls to SQL
• Reduce CPU used for complicated calculations that don’t change often
• (It’s like giving your application a cheat sheet with the answers)

Output Caching
• You can also cache entire pages if they are static by
using the [OutputCache] attribute
• http://www.asp.net/mvc/overview/older-versions-
1/controllers-and-routing/improving-performance-
with-output-caching-cs

Smarter data structures
IEnumerable<Products> products = GetProducts();
Product match = null;
foreach (var product in products) {
if (product.Code == “ABC”) {
match = product;
break;
}
}
Or
match = products.FirstOrDefault(x => x.Code == “ABC”)
O(n)

Smarter Data Structures
IDictionary<string, Product> products = GetProducts()
.ToDictionary(x => x.Code, x => x);
Product match = products[“ABC”];
O(1)

Improving the search
• Searching for keyword “red”
• Check product name, description, attributes like
category, brand and colour
• Can you imagine the SQL?

SELECT *
FROM Product p
LEFT OUTER JOIN ProductColour pc ON pc.ProductId = p.ProductId
LEFT OUTER JOIN ProductCategory c ON c.ProductCategoryId = p.ProductCategoryId
LEFT OUTER JOIN ProductBrand b ON b.ProductBrandId= p.ProductBrandId
WHERE
p.Name LIKE ‘%red%’ OR p.Description LIKE ‘%red%’
OR c.Name LIKE ‘%red%’
OR b.Name LIKE ‘%red%’

Improving the search
• You could use SQL Full Text Indexes

SELECT p.*
FROM CONTAINSTABLE(Products, *, ‘red') pi
INNER JOIN Products p on pi.[KEY] = p.ProductId
LEFT OUTER JOIN ProductColour pc ON pc.ProductId = p.ProductId
INNER JOIN CONTAINSTABLE(ProductColour, *, ‘red') pci ON pci.[KEY] = pc.ProductColourId
LEFT OUTER JOIN ProductCategory c ON c.ProductCategoryId = p.ProductCategoryId
INNER JOIN CONTAINSTABLE(ProductCategory, *, ‘red') ci ON ci.[KEY] = c.ProductCategoryId
LEFT OUTER JOIN ProductBrand b ON b.ProductBrandId= p.ProductBrandId
INNER JOIN CONTAINSTABLE(ProductBrand, *, ‘red') bi ON bi.[KEY] = b. ProductBrandId

Improving the Search
• We opted to use Lucene.NET
• http://stackoverflow.com/questions/37059/lucene-net-and-sql-
server
• http://tilt.carr.no/Post/1/full-text-searching-with-lucene-net
• http://people.apache.org/~mikemccand/lucenebench/
• As a bonus, you can add Solr to Lucene.NET and get a free
REST API for querying your indexes

Other coding tips
• Consider using parallel task library for large data sets
• https://msdn.microsoft.com/en-us/library/dd997392(v=vs.110).aspx
• http://www.codeproject.com/Articles/362996/Multi-core-
programming-using-Task-Parallel-Library
• Start using the Async Await pattern
• http://www.tugberkugurlu.com/archive/how-and-where-concurrent-
asynchronous-io-with-asp-net-web-api

Keep up to date
• Move From .NET 4.0 to .NET Framework 4.5 (30%
faster startup, 30% lower memory usage)
• Move to ASP.NET Core
• http://web.ageofascent.com/asp-net-core-exeeds-1-15-million-
requests-12-6-gbps/

Optimizing .NET
For the shopping site, we added caching and improved
the code behind heavily accessed pages like the
shopping cart.
• Searching for products went from:
• 300-500ms and 300 database reads, 1 insert, 1 update to,
• 28ms and 0 database calls

Optimizing the front end
• All about reducing the time from a user action to
when they see something happen
• First load
• Navigation

Key things to optimize
• Reduce # of requests
• Move static images, javascript and CSS to a CDN (more
concurrent requests)
• Reduce size of files
• Cache everything you can
• Use AJAX calls to fetch data and dynamically load them on a
page instead of having to refresh

Find bottlenecks
• Google Page Speed (built into chrome)
• Yslow
• Pingdom

More info
Mads Kristensen
https://channel9.msdn.com/Events/TechEd/NorthAmeri
ca/2014/DEV-B418

Case Study
Optimizing .NET
Optimizing SQL
False Start
Summary

Where did we end up?

Performance
• Now SQL server was humming along during the load
tests with about 10% CPU utilization
• Easily hitting 4000 concurrent users
• Web server was now able to server way more
requests, so the CPUs were starting run hot ~80%...

Hosting Environment
SQL DB
Web Server
IIS
Business Logic
Front End
Internet
End User

Lesson’s learnt
• Know what you’re target is (100 users? 1000 users?)
• Have Load tests setup and ready to go early in the
development cycle
• Ensure the infrastructure is good before starting
• Sometimes you don’t get what you expect (100MB network vs 1000MB)
• Sometimes you can’t see what’s happening (resource contention from
other VMs)

Lesson’s learnt
• Sometimes throwing more hardware at the problem is a quick and
easy fix
• CPUs running @ 100% update CPU architecture and now 20%
• CPU architecture changes every 2 years
• Caching – cache all the things!
• Session state and View State are bad
• Chatty SQL is bad, sometimes a stored procedure will do the trick

Lesson’s Learnt
• Don’t launch a new site when you know it’s going to be
smashed
• Give it some breathing room to make adjustments
• Constantly have performance in mind when coding
• Constantly run load tests to know when performance
changes

It’s important to constantly run load tests
After ShadowProtect was installed

Optimizing Database
• Indexes
• Denormalise
• Watch out for parameter sniffing/snooping
• Move complex logic from business layer into stored
procedures
• Good DB practices

Optimizing Software
• Data Caching
• Caution when using loops
• Parallel
• Move long running tasks to background/other
threads or use async

Optimizing Web UIs
• Reduce Size
• Compression
• Minification
• Reduce Requests
• Bundling
• Caching

Optimizing Hardware
• Load balancers
• Murphy’s Law
• More RAM
• More CPUs
• Update processor architecture
• Network connections between front end and db

Preventing this sort of thing
• Monitor & Be Proative!
• Run load tests regularly (daily, weekly, at the end of the
sprint, as part of your CI)
• Load Storm
• Loader.io
• Azure Load Tests (20,000 free virtual user minutes per month = users x
duration)

Azure Load Tests

Preventing this sort of thing
• Use a APM tool see performance in real time!
• New Relic
• Stackify – prefix.io
• App Insights
• In Visual Studio use the Intelli Trace Diagnostic Window while coding
• Make performance part of the DOD
• Look at APM tools before scrum or during the sprint planning to see if there’s an issue
that needs to be added to the backlog
• Gated check-ins that run load tests

Remember
• Performance tuning is a fine art that you can spend days on
just to improve a few hundred milliseconds of performance.
• Always measure performance before and after
• Weigh up the $$$ sometimes it’s cheaper to throw more
hardware at the problem
• http://www.slideshare.net/EricPhan6/performance-tuning-
61030451

Stuff for next time
Deep Dive – Optimizing the Front End
Deep Dive – Optimizing SQL Queries
Scaling in Azure

@EricPhan #NetUG
https://www.slideshare.net/EricPhan6/performance-
tuning-61030451
Tweet your favourite tip
Join the Conversation #DevOps @AdamCogan @DanijelMalik @EricPhan

Tweet your favourite video @SSWTV
Check out tv.ssw.com
Join the Conversation #DevOps @AdamCogan @DanijelMalik @EricPhan

Thank you!
info@ssw.com.au
www.ssw.com.au
Sydney | Melbourne | Brisbane | Adelaide

Performance tuning

More Related Content

What's hot

Similar to Performance tuning

Recently uploaded

Performance tuning

Editor's Notes