A mini-FAQ for valgrind, version 1.9.6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Last revised 22 Apr 2003 ~~~~~~~~~~~~~~~~~~~~~~~~ ----------------------------------------------------------------- Q1. Programs run OK on valgrind, but at exit produce a bunch of errors a bit like this ==20755== Invalid read of size 4 ==20755== at 0x40281C8A: _nl_unload_locale (loadlocale.c:238) ==20755== by 0x4028179D: free_mem (findlocale.c:257) ==20755== by 0x402E0962: __libc_freeres (set-freeres.c:34) ==20755== by 0x40048DCC: vgPlain___libc_freeres_wrapper (vg_clientfuncs.c:585) ==20755== Address 0x40CC304C is 8 bytes inside a block of size 380 free'd ==20755== at 0x400484C9: free (vg_clientfuncs.c:180) ==20755== by 0x40281CBA: _nl_unload_locale (loadlocale.c:246) ==20755== by 0x40281218: free_mem (setlocale.c:461) ==20755== by 0x402E0962: __libc_freeres (set-freeres.c:34) and then die with a segmentation fault. A1. When the program exits, valgrind runs the procedure __libc_freeres() in glibc. This is a hook for memory debuggers, so they can ask glibc to free up any memory it has used. Doing that is needed to ensure that valgrind doesn't incorrectly report space leaks in glibc. Problem is that running __libc_freeres() in older glibc versions causes this crash. WORKAROUND FOR 1.1.X and later versions of valgrind: use the --run-libc-freeres=no flag. You may then get space leak reports for glibc-allocations (please _don't_ report these to the glibc people, since they are not real leaks), but at least the program runs. ----------------------------------------------------------------- Q2. My program dies complaining that syscall 197 is unimplemented. A2. 197, which is fstat64, is supported by valgrind. The problem is that the /usr/include/asm/unistd.h on the machine on which your valgrind was built, doesn't match your kernel -- or, to be more specific, glibc is asking your kernel to do a syscall which is not listed in /usr/include/asm/unistd.h. The fix is simple. Somewhere near the top of coregrind/vg_syscalls.c, add the following line: #define __NR_fstat64 197 Rebuild and try again. The above line should appear before any uses of the __NR_fstat64 symbol in that file. If you look at the place where __NR_fstat64 is used in vg_syscalls.c, it will be obvious why this fix works. ----------------------------------------------------------------- Q3. My (buggy) program dies like this: valgrind: vg_malloc2.c:442 (bszW_to_pszW): Assertion `pszW >= 0' failed. And/or my (buggy) program runs OK on valgrind, but dies like this on cachegrind. A3. If valgrind shows any invalid reads, invalid writes and invalid frees in your program, the above may happen. Reason is that your program may trash valgrind's low-level memory manager, which then dies with the above assertion, or something like this. The cure is to fix your program so that it doesn't do any illegal memory accesses. The above failure will hopefully go away after that. ----------------------------------------------------------------- Q4. I'm running Red Hat Advanced Server. Valgrind always segfaults at startup. A4. Known issue with RHAS 2.1, due to funny stack permissions at startup. However, valgrind-1.9.4 and later automatically handle this correctly, and should not segfault. ----------------------------------------------------------------- Q5. I try running "valgrind my_program", but my_program runs normally, and Valgrind doesn't emit any output at all. A5. Is my_program statically linked? Valgrind doesn't work with statically linked binaries. my_program must rely on at least one shared object. To determine if a my_program is statically linked, run: ldd my_program It will show what shared objects my_program relies on, or say: not a dynamic executable if my_program is statically linked. ----------------------------------------------------------------- Q6. I try running "valgrind my_program" and get Valgrind's startup message, but I don't get any errors and I know my program has errors. A6. By default, Valgrind only traces the top-level process. So if your program spawns children, they won't be traced by Valgrind by default. Also, if your program is started by a shell script, Perl script, or something similar, Valgrind will trace the shell, or the Perl interpreter, or equivalent. To trace child processes, use the --trace-children=yes option. If you are tracing large trees of processes, it can be less disruptive to have the output sent over the network. Give valgrind the flag --logsocket=127.0.0.1:12345 (if you want logging output sent to port 12345 on localhost). You can use the valgrind-listener program to listen on that port: valgrind-listener 12345 Obviously you have to start the listener process first. See the documentation for more details. ----------------------------------------------------------------- Q7. My threaded server process runs unbelievably slowly on valgrind. So slowly, in fact, that at first I thought it had completely locked up. A7. We are not completely sure about this, but one possibility is that laptops with power management fool valgrind's timekeeping mechanism, which is (somewhat in error) based on the x86 RDTSC instruction. A "fix" which is claimed to work is to run some other cpu-intensive process at the same time, so that the laptop's power-management clock-slowing does not kick in. We would be interested in hearing more feedback on this. ----------------------------------------------------------------- Q8. My program dies (exactly) like this: REPE then 0xF valgrind: the `impossible' happened: Unhandled REPE case A8. Yeah ... that I believe is a P4 specific instruction. Are you building your app with -march=pentium4 or something like that? Others have reported that removing the flag works around this. In fact this is pretty easy to fix and I do have it on my to-do-for-1.9.6 list. I'd be interested to hear if you can get rid of it by changing your application build flags. ----------------------------------------------------------------- Q9. My program dies complaining that __libc_current_sigrtmin is unimplemented. A9. Try the following. It is an experiment, but it might work. We would very much appreciate you telling us if it does/ does not work for you. In vg_libpthread.c, add the 3 functions below. In vg_libpthread_unimp.c, remove the stubs for the same 3 functions. Let me know if it helps. Quite a lot of other valgrind users complain about this, but I have never been able to reproduce it, so fixing it isn't easy. So it's useful if you can try. int __libc_current_sigrtmin (void) { return -1; } int __libc_current_sigrtmax (void) { return -1; } int __libc_allocate_rtsig (int high) { return -1; } (Note: this fix is already in version 1.9.6 and later) ----------------------------------------------------------------- Q10. I upgraded to Red Hat 9 and threaded programs now act strange / deadlock when they didn't before. A10. Thread support on glibc 2.3.2+ with NPTL is not as good as on older LinuxThreads-based systems. We have this under consideration. Avoid Red Hat >= 8.1 for the time being, if you can. ----------------------------------------------------------------- Q11. I really need to use the NVidia libGL.so in my app. Help! A11. NVidia also noticed this it seems, and the "latest" drivers (version 4349, apparently) come with this text DISABLING CPU SPECIFIC FEATURES Setting the environment variable __GL_FORCE_GENERIC_CPU to a non-zero value will inhibit the use of CPU specific features such as MMX, SSE, or 3DNOW!. Use of this option may result in performance loss. This option may be useful in conjunction with software such as the Valgrind memory debugger. Set __GL_FORCE_GENERIC_CPU=1 and Valgrind should work. This has been confirmed by various people. Thanks NVidia! ----------------------------------------------------------------- Q12. My program dies like this (often at exit): VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH): internal error: (loads of text) A12. We're not entirely sure about this, and would appreciate someone sending a simple test case for us to look at. One possible cause is that your program modifies its environment variables, possibly including zeroing them all. Avoid this if you can. In any case, you may be able to work around it like this: Comment out the call to VG_(core_panic) at coregrind/vg_main.c:1647 and see if that helps. The text of coregrind/vg_main.c:1647 is as follows: VG_(core_panic)("VG_(mash_LD_PRELOAD_and_LD_LIBRARY_PATH) failed\n"); and so it's this call you want to comment out. ----------------------------------------------------------------- Q13. My program dies like this: error: /lib/librt.so.1: symbol __pthread_clock_settime, version GLIBC_PRIVATE not defined in file libpthread.so.0 with link time reference A13. This is a total swamp. Nevertheless there is a way out. It's a problem which is not easy to fix. Really the problem is that /lib/librt.so.1 refers to some symbols __pthread_clock_settime and __pthread_clock_gettime in /lib/libpthread.so which are not intended to be exported, ie they are private. Best solution is to ensure your program does not use /lib/librt.so.1. However .. since you're probably not using it directly, or even knowingly, that's hard to do. You might instead be able to fix it by playing around with coregrind/vg_libpthread.vs. Things to try: Remove this GLIBC_PRIVATE { __pthread_clock_gettime; __pthread_clock_settime; }; or maybe remove this GLIBC_2.2.3 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; or maybe add this GLIBC_2.2.4 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; GLIBC_2.2.5 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; or some combination of the above. After each change you need to delete coregrind/libpthread.so and do make && make install. I just don't know if any of the above will work. If you can find a solution which works, I would be interested to hear it. To which someone replied: I deleted this: GLIBC_2.2.3 { __pthread_clock_gettime; __pthread_clock_settime; } GLIBC_2.2; and it worked. ----------------------------------------------------------------- Q14. My program uses the C++ STL and string classes. Valgrind reports 'still reachable' memory leaks involving these classes at the exit of the program, but there should be none. A14. First of all: relax, it's probably not a bug, but a feature. Many implementations of the C++ standard libraries use their own memory pool allocators. Memory for quite a number of destructed objects is not immediately freed and given back to the OS, but kept in the pool(s) for later re-use. The fact that the pools are not freed at the exit() of the program cause valgrind to report this memory as still reachable. The behaviour not to free pools at the exit() could be called a bug of the library though. Using gcc, you can force the STL to use malloc and to free memory as soon as possible by globally disabling memory caching. Beware! Doing so will probably slow down your program, sometimes drastically. - With gcc 2.91, 2.95, 3.0 and 3.1, compile all source using the STL with -D__USE_MALLOC. Beware! This is removed from gcc starting with version 3.3. - With 3.2.2 and later, you should export the environment variable GLIBCPP_FORCE_NEW before running your program. There are other ways to disable memory pooling: using the malloc_alloc template with your objects (not portable, but should work for gcc) or even writing your own memory allocators. But all this goes beyond the scope of this FAQ. Start by reading http://gcc.gnu.org/onlinedocs/libstdc++/ext/howto.html#3 if you absolutely want to do that. But beware: 1) there are currently changes underway for gcc which are not totally reflected in the docs right now ("now" == 26 Apr 03) 2) allocators belong to the more messy parts of the STL and people went at great lengths to make it portable across platforms. Chances are good that your solution will work on your platform, but not on others. ----------------------------------------------------------------- (this is the end of the FAQ.)