Threads move between protection domains by performing a cross-domain call as shown in Figure 3. The cross domain call can be initiated by any control flow instruction, though it will usually be a standard subroutine call instruction. The target of the call is marked with a special permissions value known as a call gate. A call gate also has the PD-ID of the target protection domain stored in a special record type in the permissions table.
When call gate permissions are detected on a subroutine call, the hardware atomically pushes the return address and the caller's PD-ID onto a stack that resides in protected storage. This stack is called the cross-domain call stack and is implemented with some combination of an on-chip top-of-stack buffer, backed up by off-chip protected memory. The architecture then reads the new (callee's) PD-ID value from the permissions table and copies this into the CPU's PD-ID register. It looks up the protection table base pointer for the new PD-ID, and stores it in the table base register. At the end of this process, instructions are fetched from the context of the new protection domain.
MMP also uses return gates, which are the duals of call gates. They are also implemented using standard instructions and special MMP protection values. A return gate causes the architecture to pop the cross-domain call stack, finding the return address and protection domain of the last call. The architecture checks the return address against the return address being used by the return instruction. If they are different, a fault is generated which is handled by the memory supervisor. If the return addresses match, the hardware sets the protection domain to the PD-ID that was popped off the stack.
Call and return gates provide an efficient mechanism for mutually distrustful protection domains to safely call each other's services, without requiring new instructions in the ISA. Cross-domain calls are analogous to light-weight remote procedure calls, though cross-domain calls do not require copying data for protection, or an argument stack per domain pair, as LRPC does.
We expect cross-domain calls to be fast because the amount of on-chip state that needs to be changed is small. We believe CPU designers will be motivated to accelerate cross-domain calls to enable the benefits of protected execution. For example, traditional CPU microarchitectures flush pipelines on a context switch, imposing a large overhead. Domain switches can be made considerably faster by associating PD-ID values with each instruction in the pipeline, reducing the need to flush the pipeline.
If the called function needs an activation frame, it must request permissions for the stack space, and also make sure that permissions for the frame are exclusive to the current thread. This is done using the exclusive flag in a call to mmp_supr_set_perm. Because domains take exclusive access to a frame before executing in the frame, a frame's permissions do not need to be revoked at the end of a function for the caller's safety. A callee that is concerned about security could overwrite its activation frame before returning to avoid leaking information.
Calls to establish a frame will be frequent and could potentially be
expensive. Two special hardware registers can make the creation of a
frame fast, and can make permissions to read and write the frame
thread-local, closing the security loophole discussed in
Section ![[Stacks and threads]](crossref.png)
.
When the supervisor makes a stack current for a given CPU, it fills in two registers--frame base fb, and stack limit sl. The hardware allows read and writes to addresses between sl and fb (stacks grow down so sl fb). The fb value points to the base of the current activation frame. Its initial value for a given thread's quantum is specified (as a parameter to mmp_set_stack) when the thread manager starts the thread. The memory supervisor verifies the initial value of fb to make sure it is within the stack segment that is being activated. On a cross-domain call, the current fb is pushed onto the cross-domain call stack, and the current stack pointer is made the new fb. The hardware checks that the new fb value is smaller than the old value. Thus the hardware insures that the stack grows down, and the memory supervisor insures that it starts and ends in the right place, so the two registers can only be used to gain permission to read and write stack memory. The registers become part of the thread state which must be saved and restored.