A mini-FAQ for valgrind, version 1.9.6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Last revised 5 May 2003 ~~~~~~~~~~~~~~~~~~~~~~~ ----------------------------------------------------------------- Q1. Programs run OK on valgrind, but at exit produce a bunch of errors a bit like this ==20755== Invalid read of size 4 ==20755== at 0x40281C8A: _nl_unload_locale (loadlocale.c:238) ==20755== by 0x4028179D: free_mem (findlocale.c:257) ==20755== by 0x402E0962: __libc_freeres (set-freeres.c:34) ==20755== by 0x40048DCC: vgPlain___libc_freeres_wrapper (vg_clientfuncs.c:585) ==20755== Address 0x40CC304C is 8 bytes inside a block of size 380 free'd ==20755== at 0x400484C9: free (vg_clientfuncs.c:180) ==20755== by 0x40281CBA: _nl_unload_locale (loadlocale.c:246) ==20755== by 0x40281218: free_mem (setlocale.c:461) ==20755== by 0x402E0962: __libc_freeres (set-freeres.c:34) and then die with a segmentation fault. A1. When the program exits, valgrind runs the procedure __libc_freeres() in glibc. This is a hook for memory debuggers, so they can ask glibc to free up any memory it has used. Doing that is needed to ensure that valgrind doesn't incorrectly report space leaks in glibc. Problem is that running __libc_freeres() in older glibc versions causes this crash. WORKAROUND FOR 1.1.X and later versions of valgrind: use the --run-libc-freeres=no flag. You may then get space leak reports for glibc-allocations (please _don't_ report these to the glibc people, since they are not real leaks), but at least the program runs. ----------------------------------------------------------------- Q2. [Question erased, as it is no longer relevant] ----------------------------------------------------------------- Q3. My (buggy) program dies like this: valgrind: vg_malloc2.c:442 (bszW_to_pszW): Assertion `pszW >= 0' failed. And/or my (buggy) program runs OK on valgrind, but dies like this on cachegrind. A3. If valgrind shows any invalid reads, invalid writes and invalid frees in your program, the above may happen. Reason is that your program may trash valgrind's low-level memory manager, which then dies with the above assertion, or something like this. The cure is to fix your program so that it doesn't do any illegal memory accesses. The above failure will hopefully go away after that. ----------------------------------------------------------------- Q4. I'm running Red Hat Advanced Server. Valgrind always segfaults at startup. A4. Known issue with RHAS 2.1, due to funny stack permissions at startup. However, valgrind-1.9.4 and later automatically handle this correctly, and should not segfault. ----------------------------------------------------------------- Q5. I try running "valgrind my_program", but my_program runs normally, and Valgrind doesn't emit any output at all. A5. Valgrind doesn't work out-of-the-box with programs that are entirely statically linked. It does a quick test at startup, and if it detects that the program is statically linked, it aborts with an explanation. This test may fail in some obscure cases, eg. if you run a script under Valgrind and the script interpreter is statically linked. If you still want static linking, you can ask gcc to link certain libraries statically. Try the following options: -Wl,-Bstatic -lmyLibrary1 -lotherLibrary -Wl,-Bdynamic Just make sure you end with -Wl,-Bdynamic so that libc is dynamically linked. If you absolutely cannot use dynamic libraries, you can try statically linking together all the .o files in coregrind/, all the .o files of the skin of your choice (eg. those in memcheck/), and the .o files of your program. You'll end up with a statically linked binary that runs permanently under Valgrind's control. Note that we haven't tested this procedure thoroughly. ----------------------------------------------------------------- Q6. I try running "valgrind my_program" and get Valgrind's startup message, but I don't get any errors and I know my program has errors. A6. By default, Valgrind only traces the top-level process. So if your program spawns children, they won't be traced by Valgrind by default. Also, if your program is started by a shell script, Perl script, or something similar, Valgrind will trace the shell, or the Perl interpreter, or equivalent. To trace child processes, use the --trace-children=yes option. If you are tracing large trees of processes, it can be less disruptive to have the output sent over the network. Give valgrind the flag --logsocket=127.0.0.1:12345 (if you want logging output sent to port 12345 on localhost). You can use the valgrind-listener program to listen on that port: valgrind-listener 12345 Obviously you have to start the listener process first. See the documentation for more details. ----------------------------------------------------------------- Q7. My threaded server process runs unbelievably slowly on valgrind. So slowly, in fact, that at first I thought it had completely locked up. A7. We are not completely sure about this, but one possibility is that laptops with power management fool valgrind's timekeeping mechanism, which is (somewhat in error) based on the x86 RDTSC instruction. A "fix" which is claimed to work is to run some other cpu-intensive process at the same time, so that the laptop's power-management clock-slowing does not kick in. We would be interested in hearing more feedback on this. Another possible cause is that versions prior to 1.9.6 did not support threading on glibc 2.3.X systems well. Hopefully the situation is much improved with 1.9.6. ----------------------------------------------------------------- Q8. My program dies, printing a message like this along the way: disInstr: unhandled instruction bytes: 0x66 0xF 0x2E 0x5 A8. Valgrind doesn't support the full x86 instruction set, although it now supports many SSE and SSE2 instructions. If you know the failing instruction is an SSE/SSE2 instruction, you might be able to recompile your progrma without it by using the flag -march to gcc. Either way, let us know and we'll try to fix it. ----------------------------------------------------------------- Q9. My program dies complaining that __libc_current_sigrtmin is unimplemented. A9. Should be fixed in 1.9.6. I would appreciate confirmation of that. ----------------------------------------------------------------- Q10. I upgraded to Red Hat 9 and threaded programs now act strange / deadlock when they didn't before. A10. Thread support on glibc 2.3.2+ with NPTL is not as good as on older LinuxThreads-based systems. We have this under consideration. Avoid Red Hat >= 8.1 for the time being, if you can. 5 May 03: 1.9.6 should be significantly improved on Red Hat 9, SuSE 8.2 and other glibc-2.3.2 systems. ----------------------------------------------------------------- Q11. I really need to use the NVidia libGL.so in my app. Help! A11. NVidia also noticed this it seems, and the "latest" drivers (version 4349, apparently) come with this text DISABLING CPU SPECIFIC FEATURES Setting the environment variable __GL_FORCE_GENERIC_CPU to a non-zero value will inhibit the use of CPU specific features such as MMX, SSE, or 3DNOW!. Use of this option may result in performance loss. This option may be useful in conjunction with software such as the Valgrind memory debugger. Set __GL_FORCE_GENERIC_CPU=1 and Valgrind should work. This has been confirmed by various people. Thanks NVidia! ----------------------------------------------------------------- Q12. My program dies like this (often at exit): VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error: (loads of text) A12. One possible cause is that your program modifies its environment variables, possibly including zeroing them all. Valgrind relies on the LD_PRELOAD, LD_LIBRARY_PATH and VG_ARGS variables. Zeroing them will break things. As of 1.9.6, Valgrind only uses these variables with --trace-children=no, when executing execve() or using the --stop-after=yes flag. This should reduce the potential for problems. ----------------------------------------------------------------- Q13. My program dies like this: error: /lib/librt.so.1: symbol __pthread_clock_settime, version GLIBC_PRIVATE not defined in file libpthread.so.0 with link time reference A13. This is a total swamp. Nevertheless there is a way out. It's a problem which is not easy to fix. Really the problem is that /lib/librt.so.1 refers to some symbols __pthread_clock_settime and __pthread_clock_gettime in /lib/libpthread.so which are not intended to be exported, ie they are private. Best solution is to ensure your program does not use /lib/librt.so.1. However .. since you're probably not using it directly, or even knowingly, that's hard to do. You might instead be able to fix it by playing around with coregrind/vg_libpthread.vs. Things to try: Remove this GLIBC_PRIVATE { __pthread_clock_gettime; __pthread_clock_settime; }; or maybe remove this GLIBC_2.2.3 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; or maybe add this GLIBC_2.2.4 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; GLIBC_2.2.5 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; or some combination of the above. After each change you need to delete coregrind/libpthread.so and do make && make install. I just don't know if any of the above will work. If you can find a solution which works, I would be interested to hear it. To which someone replied: I deleted this: GLIBC_2.2.3 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; and it worked. ----------------------------------------------------------------- Q14. My program uses the C++ STL and string classes. Valgrind reports 'still reachable' memory leaks involving these classes at the exit of the program, but there should be none. A14. First of all: relax, it's probably not a bug, but a feature. Many implementations of the C++ standard libraries use their own memory pool allocators. Memory for quite a number of destructed objects is not immediately freed and given back to the OS, but kept in the pool(s) for later re-use. The fact that the pools are not freed at the exit() of the program cause valgrind to report this memory as still reachable. The behaviour not to free pools at the exit() could be called a bug of the library though. Using gcc, you can force the STL to use malloc and to free memory as soon as possible by globally disabling memory caching. Beware! Doing so will probably slow down your program, sometimes drastically. - With gcc 2.91, 2.95, 3.0 and 3.1, compile all source using the STL with -D__USE_MALLOC. Beware! This is removed from gcc starting with version 3.3. - With 3.2.2 and later, you should export the environment variable GLIBCPP_FORCE_NEW before running your program. There are other ways to disable memory pooling: using the malloc_alloc template with your objects (not portable, but should work for gcc) or even writing your own memory allocators. But all this goes beyond the scope of this FAQ. Start by reading http://gcc.gnu.org/onlinedocs/libstdc++/ext/howto.html#3 if you absolutely want to do that. But beware: 1) there are currently changes underway for gcc which are not totally reflected in the docs right now ("now" == 26 Apr 03) 2) allocators belong to the more messy parts of the STL and people went at great lengths to make it portable across platforms. Chances are good that your solution will work on your platform, but not on others. ----------------------------------------------------------------- Q15. My program dies with a segmentation fault, but Valgrind doesn't give any error messages before it, or none that look related. A15. The one kind of segmentation fault that Valgrind won't give any warnings about is writes to read-only memory. Maybe your program is writing to a static string like this: char* s = "hello"; s[0] = 'j'; or something similar. Writing to read-only memory can also apparently make LinuxThreads behave strangely. ----------------------------------------------------------------- Q16. When I trying building Valgrind, 'make' dies partway with an assertion failure, something like this: make: expand.c:489: allocated_variable_append: Assertion `current_variable_set_list->next != 0' failed. A16. It's probably a bug in 'make'. Some, but not all, instances of version 3.79.1 have this bug, see www.mail-archive.com/bug-make@gnu.org/msg01658.html. Try upgrading to a more recent version of 'make'. ----------------------------------------------------------------- Q17. I tried writing a suppression but it didn't work. Can you write my suppression for me? A17. Yes! Use the --gen-suppressions=yes feature to spit out suppressions automatically for you. You can then edit them if you like, eg. combining similar automatically generated suppressions using wildcards like '*'. If you really want to write suppressions by hand, read the manual carefully. Note particularly that C++ function names must be _mangled_. ----------------------------------------------------------------- (this is the end of the FAQ.)