Due to the many benefits of using dynamically-loaded libraries (e.g.,
libc), most, if not all processes make use of them, either explicitly or
implicitly. Therefore, a substantial number of pages within each
process's address space may be shared through the use of DLLs.
As discussed above, this sharing inevitably causes pages
to be littered across memory, resulting in a drastic size-increase
of 
for each process 
.
The cause of this scattering effect is that we are trying to load
library pages into the preferred nodes of processes which initiated
read-in from disk, as if these were private pages. To alleviate the
scattering effect on the library pages, we need to treat them separately
in the NUMA management layer. We implement this simply by ignoring the
hint (
) that is passed down from the VM layer, and instead,
resorting to a sequential first-touch policy, where we try to allocate
pages linearly starting with node 0, and fill up each node before moving
onto the next node. This ensures that all DLL pages are aggregated
together, rather than scattered across a large number of nodes.
Table 3 shows a snapshot of the same set of processes
under the same workload as in Table 2, but with DLL
aggregation employed.
As expected, aggregating DLL pages reduces the number of active nodes per process. However, a new problem is introduced. Due to the extensive use of DLLs, by grouping pages used for libraries onto the earlier nodes, we allocate a large number of pages onto these nodes and quickly fill them. As a result, processes need several of these low address nodes in their active sets to access all of the needed libraries. In the two snapshots shown, this is clearly apparent: after aggregation (Table 3), both nodes 0 and 1 are mapped in all of the process active sets, whereas only node 0 was needed without aggregation (Table 2). With many libraries loaded, we would use up these earlier nodes fairly quickly, and may increase the memory footprint of processes. We explain this in more details in the next section and also describe how to alleviate the extra burden on these earlier nodes.
