We just found a memory leak (again) in our latest code base that we’re going to release to production next month, seems that we’re having this issue for every release :p i think it’s unavoidable considering the amount of changes that we’re making on the app for each release ;)
Now, when I tried to do the regular steps to investigate the root cause, the CLR Profiler was not responding when I clicked the Show Heap now button and eventually display the pop-up below.
Then realised that since we have just upgraded the application to use .Net 4.0, so there’s a chance that there’s a new CLR Profiler on the block for .Net 4.0, and turned out there is actually a new one available in MSDN. And it’s not surprising, considering that .Net 4.0 comes with a new CLR as well (click on the image below to go to the msdn website for more info)
After downloaded the new CLR Profiler 4, it looks like below, notice the new Attach Process and Detach Process buttons, those will be very useful as we always need to start the application in the previous version, sweet! :)
Now, it’s time to duck into the the heap :)
I found that memory leak is pretty much of a recurring issue and you normally won’t realize this until you’re getting/processing a significant amount of objects in the memory which can’t be disposed and hence the application crashes with out of memory exception as the memory could not be reclaimed by the garbage collector and keep piling up.
I have been doing this for couple of times in our client-based application (WPF) and thought of putting the steps here for my own references and hopefully can help someone out there as well :)
So, normally when we notice that the memory keeps growing in the task manager even after you close a particular window/user control in your application, it’s a good indication that some memory can’t be released/reclaimed by the garbage collector. However it doesn’t mean that every time that you close a window/user control, the memory will always be released/reclaimed at that time, because the GC will only run on its own logic when it determines that there’s a memory pressure on the application.
To find out more or to confirm the leak, you can use these free tools below:
- Performance Monitor -> Available under Control Panel -> Administrative Tools
- CLR Profiler -> Available for download from here, make sure that you get the right version (2.0 works for .Net 3.5 Apps), user guides are available in the download package and can also be found in the msdn.
1. Create a new window/user control with a Force GC button which will do GC.Collect
2. Run the application using CLR Profiler.
Notice that I unchecked the profiling active option as it’s making the process a bit faster and it’s not really required for this example scenario. For more details of the option usage, please refer to the word doc included in the CLRProfiler download package.
3. Open the Performance Monitor and add several counters below for the particular application process
- .Net CLR Memory – #Bytes in all heaps, # Gen 0 Collections, # Gen 1 Collections, # Gen 2 Collections, Gen 0 heap size, Gen 1 heap size, Gen 2 heap size, Large Object Heap size
- Process – Private Bytes, Working Set
Take note of the Gen 2 heap size which is about 30MB in size.
4. Perform activities which are suspected for causing memory leak e.g. add control or open window, etc. Best if it can generate lots of data so the leak is more obvious.
5. Perform activity which suppose to release the references of the memory being used e.g. remove control or close window. Note that even you shouldn’t expect the memory to drop as soon as you do this, this will only happen when the GC.Collect has been executed. Notice that the Gen 2 heap size has grown to 480Mb
6. Remember the Force GC button that we created earlier, click that button to force the GC collection (you can do this as many times as you want)
Okay so you see that the Gen 2 heap size has dropped into 340Mb but remember that our initial state was only 30Mb, so shouldn’t it drop to the original state? Yes is the quick answer and next step is how to find out where the 300Mb is being used (or not being released even after force the GC)
7. On the CLR Profiler, click on the “Show Heap now” button, this will give you the graphical view of the managed heap.
As we have executed couple of times GC collections, it should be pretty easy to locate the leaked objects (this is why I asked you to generate some reasonable amount of data earlier). If you see the image below, I can see that I have 67K of objects which were created in my user control / window which has been closed but they are still in the memory for some reason.
How to find the root cause? In this graph, the callers are to the left of the callees, and the allocated objects are to the right of the methods that allocated them. This is a very important rule and was taken from the word doc. And this is exactly how you can find what are the parent object holding onto your leaked objects, just follow the chain to the left and see which one might be the culpit :)
As I have mentioned above, there are just too many root causes for the leak and the solution may differ as well for each case. But one thing that I keep following is the steps to identify the leak above. Till date, I have probably identified 4-5 UIs in our app which I managed to confirm by the steps above :)
Oh and btw, if you ever stumble into this kind of chart movement below when capturing the perfmon logs, this most likely means that you have the memory leak in the unmanaged heap. Notice that the increase rate of private bytes is higher than the #Bytes in all heaps or Gen 2 Heap size. You can think of Private Bytes = Unmanaged heap + Managed heap (#Bytes in all heaps).
To confirm this, you will need to use the windbg tool and SOS extension as CLR Profiler will only show you the managed heap side.
Hope this helps :)
There are several benefits that I can always think of with DebugView:
- The time information is always useful, it’s always there for each line, very useful especially when looking into performance
- The outputs are not being cluttered by the other output messages such as in Visual Studio output messages
- It’s running externally, which also means that we can run this separately with the real app and see the debug/trace messages in there without any configuration required.
- You can even connect to other machine to capture the output, this is really useful if you have an application server, you can just fire up the tool and say that you want to connect to that particular server and walahhh you can see the debug/trace outputs (of course with administrator access to that machine in mind)
Now, I actually just realized today at home when trying to create and debug a simple app with VS 2010, the output is not being captured anymore in the DebugView and sadly this was said by Design.
I can see that a lot of people have the same frustrations (here and here) as well because of this, you can find couple of suggested workarounds for this, however it seems that you still can’t get the debug/trace being output into the DebugView when debugging the code (you can see the debug/trace in DebugView by just running the app). I haven’t tried whether this is working or not when running unit tests in VS2010, but I would suspect the same thing will happen.
I was looking an alternative where we can actually find a way to “inject” the messages into somewhere that DebugView can capture by probably having a derived class of TraceListener, but i didn’t manage to find one unfortunately, yet.
This is just my rant because of something that we can get easily is now not anymore :|
Love to hear if you have any findings on this, but I’ll probably live by using the Ctrl+F5 for now as I don’t feel like doing -> Debug.WriteLine(DateTime.Now.ToString() + ” – ” + someDebugMessage);