This is (a) consistent with how the other containers are defined
and, more importantly, (b) allows the constification of the hash table API.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@14639
include/pub_tool_deduppoolalloc.h
coregrind/pub_core_deduppoolalloc.h
coregrind/m_deduppoolalloc.c
and uses it (currently only) for the strings in m_debuginfo/storage.c
The idea is that such ddup pool allocator will also be used for other
highly duplicated information (e.g. the DiCFSI information), where
significant gains can also be achieved.
The dedup pool for strings also decreases significantly the memory
needed by the read inline information (patch still to be committed,
see bug 278972).
When testing with a big executable (tacot_process),
this reduces the size of the dinfo arena from
trunk: 158941184/109760512 max/curr mmap'd, 156775944/107882728 max/curr,
to
ddup: 157892608/106614784 max/curr mmap'd, 156362160/101414712 max/curr
(so 3Mb less mmap-ed once debug info is read, 1Mb less mmap-ed in peak,
6Mb less allocated once debug info is read).
This is all gained due to the string which changes from:
trunk: 17,434,704 in 266: di.storage.addStr.1
to
ddup: 10,966,608 in 750: di.storage.addStr.1
(6.5Mb less memory used by strings)
The gain in mmap-ed memory is smaller due to fragmentation.
Probably one could decrease the fragmentation by using bigger
size for the dedup pool, but then we would lose memory on the last
allocated pool (and for small libraries, we often do not use much
of a big pool block).
Solution might be to increase the pool size but have a "shrink_block"
operation. To be looked at in the future.
In terms of performance, startup of a big executable (on an old pentium)
is not influenced significantly (something like 0.1 seconds on 15 seconds
startup for a big executable, on a slow pentium).
The dedup pool uses a hash table. The hash function used currently
is the VG_(adler32) check sum. It is reported (and visible also here)
that this checksum is not a very good hash function (many collisions).
To have statistics about collisions, use --stats -v -v -v
As an example of the collisions, on the strings in debug info of memcheck tool on x86,
one obtain:
--4789-- dedupPA:di.storage.addStr.1 9983 allocs (8174 uniq) 11 pools (4820 bytes free in last pool)
--4789-- nr occurences of chains of len N, N-plicated keys, N-plicated elts
--4789-- N: 0 : nr chain 6975, nr keys 0, nr elts 0
--4789-- N: 1 : nr chain 3670, nr keys 6410, nr elts 8174
--4789-- N: 2 : nr chain 1070, nr keys 226, nr elts 0
--4789-- N: 3 : nr chain 304, nr keys 100, nr elts 0
--4789-- N: 4 : nr chain 104, nr keys 84, nr elts 0
--4789-- N: 5 : nr chain 72, nr keys 42, nr elts 0
--4789-- N: 6 : nr chain 44, nr keys 34, nr elts 0
--4789-- N: 7 : nr chain 18, nr keys 13, nr elts 0
--4789-- N: 8 : nr chain 17, nr keys 8, nr elts 0
--4789-- N: 9 : nr chain 4, nr keys 6, nr elts 0
--4789-- N:10 : nr chain 9, nr keys 4, nr elts 0
--4789-- N:11 : nr chain 1, nr keys 0, nr elts 0
--4789-- N:13 : nr chain 1, nr keys 1, nr elts 0
--4789-- total nr of unique chains: 12289, keys 6928, elts 8174
which shows that on 8174 different strings, we have only 6410 strings which have
a unique hash value. As other examples, N:13 line shows we have 13 strings
mapping to the same key. N:14 line shows we have 4 groups of 10 strings mapping to the
same key, etc.
So, adler32 is definitely a bad hash function.
Trials have been done with another hash function, giving a much lower
collision rate. So, a better (but still fast) hash function would probably
be beneficial. To be looked at ...
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@14029
* new files include/pub_tool_groupalloc.h and coregrind/m_groupalloc.c
implementing a group allocator (based on helgrind group alloc).
* include/Makefile.am coregrind/Makefile.am : added pub_tool_groupalloc.h
and m_groupalloc.c
* helgrind/libhb_core.c : use pub_tool_groupalloc.h/m_groupalloc.c
instead of the local implementation.
* include/pub_tool_oset.h coregrind/m_oset.c : new function
allowing to create an oset that will use a pool allocator.
new function allowing to clone an oset (so as to share the pool alloc)
* memcheck/tests/unit_oset.c drd/tests/unit_bitmap.c : modified
so that it compiles with the new m_oset.c
* memcheck/mc_main.c : use group alloc for MC_Chunk
memcheck/mc_include.h : declare the MC_Chunk group alloc
* memcheck/mc_main.c : use group alloc for the nodes of the secVBitTable OSet
* include/pub_tool_hashtable.h coregrind/m_hashtable.c : pass the free node
function in the VG_(HT_destruct).
(needed as the hashtable user can allocate a node with its own alloc,
the hash table destroy must be able to free the nodes with the user
own free).
* coregrind/m_gdbserver/m_gdbserver.c : pass free function to VG_(HT_destruct)
* memcheck/mc_replace_strmem.c memcheck/mc_machine.c
memcheck/mc_malloc_wrappers.c memcheck/mc_leakcheck.c
memcheck/mc_errors.c memcheck/mc_translate.c : new include needed
due to group alloc.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@12341
there were a lot of loss records.
The fix was:
- Avoid the O(m * n) looping over the chunks when creating the loss
records, by putting loss records into an OSet instead of a list, which
makes duplicate detection for each chunk an O(log n) operation instead of
an O(n) operation.
- Avoid the looping over loss records which was used to do a poor
man's sort, but was O(n^2). Instead copy pointers to the loss records
from the OSet into an array and sort it normally with VG_(ssort) (n log n,
usually) before printing.
This approach was similar to that used in the patch Philippe attached to the
bug report.
Other changes:
- Added Philippe's test programs in the new memcheck/perf directory. It
used to take 57s on my machine, now it takes 1.6s.
- Cleaned up massif/perf/Makefile.am to be consistent with other Makefiles.
- Improved some comments relating to VgHashTable and OSet.
- Avoided a redundant traversal of the hash table in VG_(HT_to_array), also
identified by Philippe..
- Made memcheck/tests/mempool's results independent of the pointer size, and
thus was able to remove its .stderr.exp64 file.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@9781
Allow hashtables to dynamically resize (patch from Christoph
Bartoschek). Results in the following interface changes:
* HT_construct: no need to supply an initial table size.
Instead, supply a text string used to "name" the table, so
that debugging messages ("resizing the table") can say which
one they are resizing.
* Remove VG_(HT_get_node). This exposes the chain structure to
callers (via the next_ptr parameter), which is a problem since
callers could get some info about the chain structure which then
changes when the table is resized. Fortunately is not used.
* Remove VG_(HT_first_match) and VG_(HT_apply_to_all_nodes) as
they are unused.
* Make the iteration mechanism more paranoid, so any adding or
deleting of nodes part way through an iteration causes VG_(HT_next)
to assert.
* Fix the comment on VG_(HT_to_array) so it no longer speaks
specifically about MC's leak detector.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@6778
types in m_hashtable.c to void*. This requires no changes to code
already using VgHashTables, but it allows some previously-required casts
to be removed. I also changed Memcheck and Massif by removing some of
these now-unnecessary casts.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@4404
when it is created. Fortunately this didn't affect code outside this
module except for the calls to VG_(HT_construct)().
As a result, we save some memory because not all tables have to be as big
as the ones needed for malloc/free tracking.
git-svn-id: svn://svn.valgrind.org/valgrind/trunk@4100
