Visual C++ 2010: Detecting Memory Leaks For Global Variables

This article is a hint for those who feel desperate with finding the origin of a memory leak in their programs. I suppose you've already read the Microsoft's documentation on this topic - Finding Memory Leaks Using the CRT Library - and set up your project to enable leak detection.

The documentation states that if you did everything properly, the Output window should display all the leaked memory blocks along with source file names and line numbers. However sometimes it's not the case and there's nothing shown except a bare memory address and allocation size. And you end up staring at these numbers, puzzling over what you could do wrong with leak detection setup, placing exit()s all over the code and trying to understand logically where those damned leaks could originate from.

Actually this might be not your fault that you don't see line numbers. To be precise, it's not your fault but it's a problem of your project's design. Probably your project uses many global variables which get initialized long before source line number tracking is in effect and thus in the end debug CRT detects leaks but cannot report line numbers. Global variable usage is unavoidable in large and complex projects so we need some method to fix such leaks.

The good news is that such method exists. The bad news is that it involves much manual work. But at least it works.

• First you'll need to make a whole program run to get the entire memory leak report in the Output window. Copy the memory leak report somewhere, e.g. to a Notepad window, as later you'll need these numbers in curly braces called memory allocation numbers.
• Open the crt0dat.c file in Visual Studio. I assume that during the Visual C++ installation you had chosen the default folder, so that file should be located in "C:\Program Files\Microsoft Visual Studio 10.0\VC\crt\src".
• Within the crt0dat.c file, search for the following string (no quotes): "__cdecl _initterm_e". Place a breakpoint at the first statement of the _initterm_e() function.
• Run your program again. The execution stops at your breakpoint. Now go to the Watch window and type "_crtBreakAlloc" (no quotes) in the Name column. In the Value column most probably you'll see -1.
• Disable the breakpoint in crt0dat.c. You won't need it to be hit again during this program run.
• Now get back to your Notepad window and copy to the clipboard the first of the memory allocation numbers in curly braces. Go to the Watch window in Visual Studio and in the Value column replace the value shown with the value in the clipboard. Press Enter.
• Resume the execution by hitting F5 or choosing Continue from the menu. After a while Visual Studio should display a message that reads "<YourProgram> has triggered a breakpoint" and stops at some location within the CRT debug code, most likely in the dbgheap.c file.
• Now go to the Call Stack window and scan it from the top to the bottom until you find a function that is known to be written by you. Now you can conclude on what can be the reason of the memory leak. It might turn so that at the top or in the middle of the call stack there are gray lines containing only addresses. This means that symbol information is absent for some libraries used in your project. Hit Shift-F11 until you get rid of gray lines before the known functions. Ignore "No source code available" messages if they appear and keep hitting Shift-F11.
• Once you finished your investigation with one of the leaks, you may continue with another without restarting the program. Just get the next memory allocation number from the previous memory leak report, paste it into the Value column for _crtBreakAlloc in the Watch window and hit F5. Investigate the cause of the leak. Repeat these steps until you examine all leaks you have. This works thanks to that memory leaks reported in the same order as the corresponding memory allocations happen.

Why do we need a breakpoint inside crt0dat.c? Because need to capture a memory allocation event before our main() function starts. Once main() is entered, all global variable initializations already happened and we've lost the chance to track allocation events either by hardcoding allocation numbers (using statements like "_crtBreakAlloc = 1234;") or by editing the _crtBreakAlloc watch value during the debug suspend mode. The _initterm_e() function just seems to be a good choice to place a breakpoint.

It is crucial to run your program every time with the same input conditions so as to memory allocation numbers stay unchanged between runs. Once you fixed a leak, you'll need to repeat the whole process from the start as allocation numbers likely have changed.

Hope this info will help fixing your very own leaks.

more >>

Visual C++ 2010: How To Fix The "Up-to-date Project Always Gets Rebuilt" Problem

Sometimes when you hit F5 or F7, Visual Studio acts as if something has changed in your project and rebuilds it, even immediately after a fresh rebuild. This might be very annoying and time-consuming, especially when debugging big projects. I'll try to summarize what can be done to eliminate this problem.

It's all about the new MSBuild build system. Something is fooling it and instead of seeing in the build output window a message like this:

========== Build: 0 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

you always see this:

========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========

The following can be done then:

• Check your project settings regarding the intermediate output directory. If you have more than one project in your solution, no two projects can share the same intermediate output directory. No manually placed files should reside in this directory and no manual corrections to automatically generated files should be made. This directory should be exclusively under Visual Studio's control. Try to "Clean" the project or the entire solution using Visual Studio's command and then test the build behavior. If no luck then try to clean the directory manually and test again.

• Your project might reference a non-existent file. MSBuild's up-to-date check mechanism will assume that a new build is required. Try to locate and remove non-existent files from the project.

• Your project is converted from a project of a previous Visual Studio's version. And your project somehow became "broken". It also can "break" in some other mysterious ways during "normal use", even if it's not a conversion project. The cure is to re-create the project from scratch, despite how dull and tedious it may sound.

I personally encountered all the three above situations and managed to solve the problem. If you have something else to say regarding this topic please let me know.

more >>

Visual C++ 2010: IDE Memory & Performance Problems. Consider Precompiled Headers

If all you need is just the PCH solution you can scroll down to “The PCH Solution” section immediately. Otherwise you may want to read about how I came to this solution and hopefully find some clue for solving your own problem.

Well, when I first installed Visual C++ 2010 I was full of anticipation and excitement about new IDE’s features. Certainly, with its IntelliSense technology Microsoft had gone far ahead of all its rivals. Visual Studio’s brand-new on-the-fly compilation and real-time error reporting capabilities drastically speed up coding and make the whole development process much easier.

Quite a bit of time has passed since then. I enjoyed the new extremely convenient IDE and kept on praising Microsoft for this incredible invention. But one day things began to get worse.

I noticed some strange IDE’s behavior. After a few minutes of editing the code the IDE was getting almost irresponsible, the window was failing to redraw fast and intensive HDD I/O was occurring. Something was happening and until it ends it was practically impossible to work. This effect started to appear more and more often. In some editing sessions I was getting it every 20 to 40 new lines of code.

I can’t point out exactly when it began. Maybe when I added a couple of headers containing some tricky macros. Or maybe when the line count in my entire solution exceeded some critical value.

All I knew it was about IntelliSense. Simply turning it off was not an option as without it the whole IDE would have lost all its appeal. No-o-o... I desperately needed IntelliSense, but a working IntelliSense!

My first step was using Google (sounds weird, huh? )) Soon I found that world falls into the following two parts. The first one is the people who are completely satisfied with new Visual Studio 2010 and have not even a single performance problem with it or a problem that can be solved relatively easily. The other world’s part is a bit unluckier and suffers from performance problems which seem mystical as Microsoft can’t do anything with them. Except referring people to absolutely useless solution checklists.

I read many blog posts, advices and other info and nothing helped. However I think some of those are worth looking at:

http://support.microsoft.com/kb/981741/en-us

http://weblogs.asp.net/scottgu/archive/2007/11/01/tip-trick-hard-drive-speed-and-visual-studio-performance.aspx

Then I started monitoring CPU and memory usage via Process Explorer. It showed that the moments of performance drop relate to high physical memory usage followed by the massive flush to the page file. When I was typing the lines of code memory usage was growing gradually until the memory was totally flooded and Windows had no other way but to flush it to the page file. This explains intensive hard disk I/O which hindered the whole system’s performance. While all this was happening no CPU was being consumed at all. At the same time no particular process in Process Explorer was looking as a memory hog (although multitudes of vcpkgsrv.exe and msbuild.exe processes were spawning and dying all them looked pretty decent). Probably memory was being allocated internally by some system processes, I don’t know.

I must say I have Windows XP SP3 on my development machine. I searched for any information which could regard specifically to XP problems but found none. The next idea was to increase my RAM size. I had only 2GB and since I already did consider a memory upgrade just before these VS2010 problems it took me not long to decide. With 4GB of RAM things became better but not much. It still was hogging all the 4 gigs to the end and then flushing. Only these “hog-flush” loops got longer allowing me to type a bit more code.

Then I tried things such as:

• Turning off/on and tweaking various options in the Tools\Options\Text Editor\C/C++\Advanced section. Didn’t help...
• Turning on Diagnostic Logging (located also in the above mentioned section) with various verbosity levels and trying to look for any errors. Grrrrr, errors were found but too cryptic to give me a clue...
• Rearranging code in files and recreating project/solution files. No luck...
• Refactoring my most tricky macros. Nothing...
• Installed Visual Studio 2010 Service Pack 1 Beta. Didn’t help

The PCH Solution

I kept trying and finally found the solution! This is it:

http://blogs.msdn.com/b/vcblog/archive/2010/01/26/precompiled-header-files-in-visual-studio-2010.aspx

Although it doesn’t contain a direct indication to my memory problem, these two phrases made me to start exploring:

“The intellisense compiler can load these iPCH files to save not only parse time, but memory as well: all translation units that share a common PCH will share the memory for the loaded PCH, further reducing the working set.”

“iPCH and build compiler PCH share the same configuration settings (configurable on a per-project or per-file basis through  “Configuration Properties->C/C++->Precompiled Headers”).”

The fact is when I started my project I took decision not to use precompiled headers at all. First, because at that time it was of course a very small project. Second, I thought I’d better have more flexibility as to header inclusion and therefore every translation unit would only have a minimal set of headers required for the compilation.

But time has passed and now the project had grown to a moderate size of ~40K lines of code and uses STL and 7 other 3rd party libraries some of which are quite big.

It took me about an hour to study the details and set up the whole PCH thing which involved rearranging includes in every file and changing the project settings. The following info helped me much and I suggest reading it thoroughly:

http://www.cygnus-software.com/papers/precompiledheaders.html

http://msdn.microsoft.com/en-us/library/z0atkd6c.aspx

And this is what I’ve got now:

• No memory flooding problems
• IntelliSense is working smo-o-o-thly
• Compilation became blazingly fast
• Piece of mind

Wish your VS2010 problems got solved!

more >>

Getting Boost.Regex + Unicode to work with Microsoft VC++ 2010

[ This article pertains to Boost version 1.44. It may or may not remain useful in the future, when newer Boost versions appear, depending on how Boost.Regex’s developers would arrange the build process. ]

Although the subject above may seem to be overly specific, I think many people might want to use regular expressions with international text support in their programs being developed in Windows/Microsoft environment. I’m in no way a Boost.Regex expert or even seasoned Boost user. I just would like to share my “newbie” experience with programmers who are trying to achieve the same as I did. And for the current Boost’s version - 1.44 - it is not so easy.

And one thing should be mentioned before we start. I adhere to the principle of least resistance. For the process of software library linking that means the following: use vendor-supplied compiled binaries as much as you can and use your hands as sparingly as you can. Probably the moment when you need to build a library from sources (which can be quite troublesome) or do some manual tweaking will come. Maybe, but it’s a future, so leave all hassles for that future. For now we are going to get our libraries up and running, and to do it fast. At least on Windows platform, this approach seems to be sound.

OK, let’s start. As it’s noted here, to support Unicode you have two approaches: the simple one and the good one. For some cases the simple approach can suffice however actually it is not portable and has several disadvantages. To help you build a true Unicode-aware application, Boost.Regex - one of the best regular expression libraries - requires ICU – one of the best Unicode libraries. The following describes how to get Boost.Regex and ICU work together.

First we need to obtain the latest release of ICU from here. In the download page choose ICU4C as we need to interface the ICU library with C/C++ programs. Download the Windows binaries .zip file and unpack it to some folder on your hard drive, say C:\Distr\icu.

What we need to do in order to build Boost.Regex with ICU is described in detail here. However building from scratch is still not a preferred option, and when I was looking for an alternative I came across the BoostPro Installer which can provide us with binaries for every possible Boost library, every compiler, threading model, etc. But… BoostPro does not cover per-library specifics such as ICU-enabled Regex.

Therefore we have no other choice but to compile Boost.Regex by hand. However BoostPro Installer still can be of use for us because it can serve as a convenient download tool that will provide us with Boost sources and (what is more important) some compiled Boost tools (you will be able to see below why we need them). For now run BoostPro and proceed until the wizard page appears where compilers and variants can be selected. There you should select no checkboxes at all. On the next page leave only four checkboxes checked at the top of the list. Then advance to the end of the wizard and wait for files to be downloaded and installed.

Assume you chose to install Boost to the default installation folder (C:\Program Files\boost\boost_1_44). In the C:\Program Files\boost\boost_1_44\libs\regex\build folder you will find several vc*.mak files which – what a pleasant surprise! - outnumber those mentioned in the documentation on the Boost's web site. But what is not so good surprise is that they lack one for MSVC10. Running vc9.mak by VC10’s nmake.exe gives many errors and unknown option warnings which look a bit scary. Editing makefiles manually is not an option for us as we need the result as soon as possible.

Somehow guys at BoostPro had managed to built Regex library with MSVC10 and are unlikely to conceal their secret knowledge from the public, so it has to be somewhere around. In Boost.Regex’s build folder there are also some .sh files which are Unix shell scripts, a .cpp file and a cryptic Jamfile.v2. Relating this name and the information obtained from Regex’s building guide lets us conclude that we need a program called bjam (which in fact is a part of Boost’s own build system) to run Jamfile.v2. Fortunately, as a result of installation by BoostPro, a compiled bjam.exe can be found in the C:\Program Files\boost\boost_1_44\bin folder.

Now let’s launch a command line window and change the working directory to Regex’s build folder. The straight-forward (partially borrowed from the Regex installation guide and Google search results) command

"C:\Program Files\boost\boost_1_44\bin\bjam" -sICU_PATH="C:\Distr\icu" toolset=msvc-10.0 release threading=multi link=shared

among other info outputs an undesired message

has_icu builds : no,

which indicates that the test program has_icu_test.exe could not be compiled or linked correctly and the build script falls back to compilation/linking of Regex without ICU. After struggling for a while with this strange problem I found that by some reason the build tool requests for debug versions of ICU libraries which of course I have none of and didn’t intend to use. I must confess, I’m not fond of studying new software tools that I’m sure I wouldn’t use much on but here I had no other choice. Eventually I found that ‘lib’ rule sets select wrong alternatives, even when configuration features are explicitly provided via command-line arguments, just like in the command above. E.g. this block in the Jamfile.v2 file

   lib icuuc : : <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

   lib icuuc : : <toolset>msvc <variant>debug <name>icuucd <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

   lib icuuc : : <name>this_is_an_invalid_library_name ;

chooses the second line, i.e. the icuucd library (debug version of one of ICU’s libraries) but supposed to choose the first line, i.e. icuuc library (release version of that ICU’s library). Other rule sets behave in a similar way. I had no time to investigate why this may happen and I don’t see any reasons not to follow the path of least resistance and use a no-brainer: I commented out in the Jamfile.v2 file all alternatives which may lead to selection of incorrect libraries, like this:

   lib icuuc : : <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

#   lib icuuc : : <toolset>msvc <variant>debug <name>icuucd <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

#   lib icuuc : : <name>this_is_an_invalid_library_name ;

#   lib icudt : : <search>$(ICU_PATH)/lib <name>icudata <link>shared <runtime-link>shared ;

   lib icudt : : <search>$(ICU_PATH)/lib <name>icudt <toolset>msvc <link>shared <runtime-link>shared ;

#   lib icudt : : <name>this_is_an_invalid_library_name ;

#   lib icuin : : <search>$(ICU_PATH)/lib <name>icui18n <link>shared <runtime-link>shared ;

#   lib icuin : : <toolset>msvc <variant>debug <name>icuind <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

   lib icuin : : <toolset>msvc <variant>release <name>icuin <search>$(ICU_PATH)/lib <link>shared <runtime-link>shared ;

#   lib icuin : : <name>this_is_an_invalid_library_name ;

Mostly this is it. Save corrections to the Jamfile.v2 file and run the above command again. This should work the right way. My configuration for bjam was release threading=multi link=shared which means: I need release (non-debug) version, I need multi-threaded, I need DLL. You might probably want another configuration, then you may change release to debug, multi to single or shared to static.

When everything is going right, at the beginning of bjam’s console output you should see a message has_icu builds : yes.

As a result of bjam’s work, all generated files will be placed into the C:\Program Files\boost\boost_1_44\bin.v2\libs\regex\build\msvc-10.0\release\threading-multi folder (the last two subfolders can vary depending on your options).

To double check, you can examine a couple of .rsp files which also would reside in the output folder. E.g. the file c_regex_traits.obj.rsp should contain the line

   -DBOOST_HAS_ICU=1

And the file boost_regex-vc100-mt-1_44.dll.rsp should contain the lines

   "icuuc.lib"                   

   "icudt.lib"

   "icuin.lib"

That’s it. For how to use ICU-powered Boost.Regex please refer here

more >>