It is possible to modify the guest OS to avoid using instructions that virtualize inefficiently. Going a step further, it is also possible to provide a safe call into the VMM from the guest OS to provide some semantic knowledge about the guest to the VMM, or to perform some operations on its behalf. This technique comes at the price of compatibility with off-the-shelf guest OSes.
An optimization we tried in this category is to alter the Linux kernel to avoid page table switches when switching to the idle task. An idle guest spends a significant amount of time context switching to and from its idle task. A guest context switch operation uses a number of privileged instructions and changes guest page tables. This requires VMM intervention to implement the guest context switch safely. In the experiments above, as optimizations are added to reduce CPU utilization, the virtual machine execution profiles show an increasing fraction of CPU overhead due to virtualizing guest context switches. The VMM in VM/PC-733 spends 8.5% of its time virtualizing page tables switches.
Linux's 2.2 kernels run the idle task as a kernel thread with the kernel's page table. The kernel's page table is a subset of every user application's page table. This implies that it is not necessary to switch page tables when switching to the idle thread. Further, if the idle thread runs with the page table of the last user process to run and the idle thread ends up yielding back to the same process, another page table switch can be avoided. This optimization relies on trusting the idle thread not to corrupt user memory, a reasonable requirement since the idle thread runs at a trusted kernel-level.
We prototyped the optimization of running the idle task with the prior user application's page table by modifying the Linux kernel's context switch function. This modification halves the MMU derived virtualization overhead, and almost all of the saved CPU cycles become CPU idle time. Besides reducing virtualization overhead, such an optimization may also benefit software-based IA-32 CPU implementations where the overhead of emulating the instructions involved in a context switch is significant.