Software Architecture

The binary-rewriting RAD tool comprises the following logical components: a disassembler, a core binary rewrite engine, a RAD component, and a PE (Portable Executable format specific) component. The disassembler functions in two main phases. In the first phase, it performs code/data and branch target identification covering all bytes in the code section, and in the second, it outputs the assembly instructions starting from the first byte in the code section. The core binary rewrite engine, independent of the binary format, hooks into the disassembler in the second phase to gain control at every instruction processed to look for 'interesting' function prologues and epilogues to instrument. This component handles all the issues involved in adding instrumentation code outlined in the section 3.2. Since the instructions that make up an 'interesting' pattern need not be contiguous, this component maintains a window of five instructions (current instruction and four previous ones), which is flushed whenever a branch target is encountered, so that we don't run over any jump target. The engine attempts to identify 'interesting' patterns in the window every time a new instruction is added. All the RAD code and its associated data are added to a new section at the end of the input binary. The RAD component implements the Return Address Defense mechanism. The prologue stub saves the return address on the stack to the Return Address Repository (RAR) and the epilogue stub keeps popping the RAR stack till it finds the return address currently on the top of the stack or till the RAR is empty in which case it flags an exception. This repeated popping ensures that the return from any of the caller's ancestors (from the current call stack), does not generate false security alarms. This scenario occurs in case of setjmp()/ longjmp() as well as compiler optimizations which cause functions to return straight to the caller's caller if the first return is to the caller's return section (e.g. tail recursion). The PE component initializes the binary rewrite process by adding a new section header in the section header table and setting up its fields appropriately; it also aligns the new section (called the .RAD section) that holds the RAD code, depending on where it should be loaded at run time (page boundary next to the end of the previous section) and where it should be stored in the binary file (file alignment boundary after the end of the previous section).