15. Why do leaks matter?
• Low end devices
• Degrades performance of application over time
• May impact your heaviest users the most
• Reputation can be affected
24. Measuring memory usage
• Visit application URL in Chrome
--expose-gc
• Perform garbage collection
• Take initial memory snapshot
window.performance.memory.usedJSHeapSize
• Perform action a set number of times
• Perform garbage collection
• Take another snapshot and subtract initial snapshot
31. Takeaways
• Understanding memory leaks
• What a memory leak is?
• Debugging
• Measuring retained memory to get visibility
• Automate executing a UI action and measuring the memory retained.
• Storing over time to monitor any memory leaks.
• Preventing memory leaks by gating
• Writing tests to assert retained memory is below a given threshold.
32. • Frame rate – ticking prices
• Rendering times – components
Other monitoring
Hi everyone,
I am James, I work for a fintech trading company called IG on the trading platforms architecture team; focusing on building performant web applications.
I have worked commercially with web technologies for more than 15 years and 7 of those years in my current role at IG.
Today I am going to talk about our experiences in ensuring quality in our web applications.
But first who are IG?
We are the world’s No.1 spread betting and CFD provider, giving retail investors leveraged access to over 15,000 financial markets
We were established 45 years ago in 1974 and currently have 3 developer hubs here in Bangalore(India), Krakow (Poland) and London (UK); working on both collaborative and autonomous projects.
IG supports a wide variety of devices including Android, Android Tablet, iPhone, Ipad and Web.
We have up to 1.5 million trades a day
Over 11 million executions a month
And most orders are filled within 10 milliseconds
Today I am going to focus on our web trading platform.
So now you are familiar with IG and what challenges we face. What will we be discussing today?
I want to share with you the tips and lessons we have learned from developing our web trading platforms.
Firstly I will give an overview of our web platform and what kind of functionality it has.
Then I will do a deep dive into the challenges we faced around memory leaks that will cover
What memory leaks are
Debugging techniques
How to measure retained memory
Then how to prevent memory leaks getting into production
And finally a demo
I will then cover how these techniques could be used in other areas
So during my 7 years at IG we began building our newest web trading offering. Which is a single page application that shows real time financial dataand that has users with high performance expectations; who can have the application open for long periods of time.
If the application crashes or runs slowly, it could cost our users money. This is why performance is paramount at IG.
Our application offers an desktop operating system like experience with users opening “panels” of content.
These panels of content could be used by a client to browse markets for an opportunity to trade while viewing the live prices in our watchlist panel.
Placing a trade by picking a direction; “buy” if they think the market will rise or “sell” if they think the price will fall.
Or managing their current portfolio of trades via the positions panel.
We have had a around 40 developers work on the codebase, who work in an agile environment, releasing the product intraday through a continuous delivery pipeline with post release checks on critical path health. Such as placing a trade and logging in.
Developing this application brought about some challenges and when we released the application to a Beta audience back in 2017; we got some reports from users of the application slowing down over time and even crashing. As many of you may know this can be a warning sign your application has some memory leaks.
This was a blow to the developers and after fixing the existing memory leaks we wanted to come up with a way of preventing them in the future.
OK lets have a quick recap of memory works in JavaScript.
As many other programming languages JavaScript has the following memory lifecycle…
Allocation – where memory allocated by the Operating System. JavaScript is a high level language, so unlike lower level languages this operation isn’t explicit.
Usage – Where the program uses the memory to and performs read/write operations, such as assigning a variable.
Release – where the allocated memory is no longer needed and release. Again this is not explicit
Memory is collected via the mark and sweep method.
Root nodes are identified by the garbage collector, such as Window in the browser environment or Global in Node environment.
Then all child nodes are inspected and marked as active or inactive
The inactive ones are then cleaned up and that memory is then released.
You can see this in the diagram the active nodes are being marked as green and the inactive ones left as blue.
The inactive nodes and then collected when the garbage collection takes place.
Memory leaks can occur if retained paths are accidently left pointing to objects.
We can see in this simple object graph array has a retaining path to the DOM node.
The code below shows the cache array being initialized and the DOM node being pushed into it. Then later the DOM node being removed from the document.
However as long as the cache array still has the retaining path it will not be garbage collected.
This is often how leaks are caused; it is worth considering that leaks are a human error, so can be nearly always be prevented.
But why do leaks matter though right?
It could be your users are on low end devices which maybe hit particularly hard and will not get the best from your application
Performance can degrade over time which in turn may end up impacting your heaviest, and possibly most valuable users. This is especially true if you are building a single page application like us and cannot rely on the user navigating away; we have users who potentially can be using the application for hours while monitoring their current portfolio or waiting for the right time to trade.
Lastly your reputation with your users; it can degrade trust that you are offering reliable software if it is seen to be crashing or running at a sluggish pace.
While our application may not be the most common; a typical routed single page application can still suffer the same kind of issues. If event handlers are not cleaned up at the correct hooks.
So lets talk through the steps for debugging a memory leak. I am going to use a simple example that is a small application that adds 100 items to a list and then allows you to then clear that list.
We would expect an application, such as the one just shown, that creates short lived objects after an action to have a sawtooth type pattern in the performance timeline that represents the objects being created and holding memory and then when removed that memory being freed.
If an application has a leak it would be steadily increasing and not dropping back down to the baseline.
To debug a leak you could begin by opening up the task manager in Chrome.
The Task Manager is allows you monitor memory is being used by a page in realtime.
You would then toggle the memory and Javascript memory columns.
The Memory column represents native memory. DOM nodes are stored here.
And the Javascript memory column which is the JSHeap.
As you can see when we use the application, by adding and removing the items, the memory usage is increasing, so there maybe a memory leak and it is worth investigating further.
Once you have identified an action you suspect of leaking. You would then typically switch to the performance tab to investigate further.
Then enable the Memory checkbox.
Force a garbage collection to free up any available memory by clicking the trashcan icon.
Then start a recording
Repeat the action a number of times, clearing the garbage after each cycle.
By repeating the action you can see a potential leak appearing on the timeline.
You can see here that the memory usage is going up and being retained after a GC.
Lastly you might head to the memory tab and record a heap snapshot before the action and one after the action for a comparison of the objects created. This allows for much more of a detailed view of the problem once you have established where it lies.
You can see here some detached HTML elements that should have been cleared up.
After discussions with the team the performance timeline steps seemed reproduceable and we looked to Codify them.
Just like a bug you have reproduced that you would then go on to cover with a test case to ensure it does not resurface.
So first we needed the memory usage of the application. This is available via the memory object in the performance API.
This is a global object to give access to memory usage information. Which contains…
size limit – which is the memory the heap is limited to.
total Size - The memory the heap has allocated including free space for the page.
used Size - The memory currently being used by the page.
This is one we use as we are interested in as we need the currently used memory at points in our applications.
All these are returned in bytes
Next we needed to find a way to clear the garbage programmatically.
So luckily we found Chrome has a way of doing this. It expose a js flag you can launch Chrome with to enable a global “gc” function that forces a garbage collection when invoked.
Once we had these tools available to us we came up with a recipe to try and get visibility on memory leaks by measuring them…
Visit the application in Chrome with the expose GC to enable garbage collection programmatically.
Perform a GC to release any available memory
Take an initial snapshot, using the memory API
Perform an action number of times to exaggerate the problem, for us this might be launching a ”panel
Perform another GC to free up any memory
Then another snapshot of memory and subtract the initial one
This gives us the retained memory, if any.
Here you can see how we went about fitting this into our release process.
Our current continuous delivery flow runs on after every commit to master.
Once the testing, coverage and linting gates have all passed it will be deployed to production
Our application is static, so this only involves deploying an html entry point and associated assets.
Once a successful build has taken place we then kick off our performance build step.
This launches Chrome via selenium
Runs through the performance scenarios, like launching one of our panels a number of times
Measures the retained memory
And then stores in influxDB
It is worth mentioning these scenarios are run on a box inside our infrastructure, but this could be achieved with a cloud service
Those results stored in InfluxDB are then charted using Grafana to monitor usage over time and get visibility of memory leaks in the wild.
You can see here that on the Y axis we have retained memory in MB and on the X axis time.
We then have a threshold line of 0.25MB.
We allow this threshold due to browser libraries or frameworks often holding onto memory due to caches.
The memory leak on the chart of 0.6MB was introduced in May last year and present for 4 months before being fixed in September and dropping well below the threshold.
The memory leak of 0.6MB that was present for 4 months before being fixed and dropping below the red threshold.
The length of time the leak was open shows us how long it can take to fix an issue like this once it is in production and has to be prioritized against other work.
While monitoring memory retention was a big leap from where we were, but was a very reactive way of dealing with the issue and didn’t actually prevent leaks.
So we decided try and be more proactive and build them into our existing test process therefore allowing us to gate them.
So here you can see the updated Pipeline.
Kicking off a build after a commit to master remains the same
However this time the scenarios have been converted into tests that run on headless Chrome.
You can see that if the test passes a deployment takes place and the build passes
Or if the build failed feedback is given back to the developer and the deployment is prevented
This allowed us to completely integrate memory leak regression testing into our CD pipeline.
So what does a performance test look like?
We can see it is very much the same as the performance scenarios we ran.
We do a garbage collection
We take an initial snapshot
We run a scenario 10 times, rendering a component, then making sure to destroy it afterwards
We then collect the garbage again, take a snapshot and compute the retained memory.
But this time instead of reporting; we assert the memory retained is below a threshold
As shown in the previous slide we can now use this to gate whether a build passes or fails.
Test framework
So to wrap up we have talked about…
Understanding what a memory leak is and how to debug them.
Executing a UI action in a controlled way and compare usage before and after.
Monitoring over time to get visibility of any memory leaks
Writing tests to assert retained memory is below a given threshold which allows you to gate them
For our application we still use our controlled monitoring method, of launching a browser with selenium and running a performance scenario, for some of our other metrics such as…
Monitoring the frame rate of intensive actions – live our ticking prices
And rendering times of our components
Again these run after every successful build of master.
Here we have our price ticking dashboard which shows the frame rate of our ticking prices and the threshold we impose of 30 FPS.
We have Frames per second on the Y axis and time on the X axis.
Displaying for Chrome and Firefox. Chrome in green and Firefox in yellow here.
And here the positions component rendering time in Milliseconds and how it performs across Chrome, Firefox and Internet Explorer. Chrome in green, Firefox in orange and internet explorer blue.
We have milliseconds on the Y axis and datetime on the X axis.
These dashboards could both be moved to tests in the same way to prevent regressions in the future; once an acceptable threshold had been agreed with either the business or the developers depending on the area.
So hopefully this helps you understand how at IG we have gained visibility of our performance and prevented performance regressions from being released to production.
For us this approach has been very successful and prevented leaks reaching users on a quite a few occasions and given the developers confidence in the code they are deploying.
So thanks for your time and for those interested the code for the demo is available on my Github profile in the link shown.
