Fast IPSec: A High-Performance IPsec Implementation
Samuel J. Leffler
Errno Consulting
sam@errno.com

ABSTRACT

Fast IPsec is an implementation of the IPsec protocols[Kent & Atkinson, 1998a].for FreeBSD that was designed for high performance.In particular the protocols use theOpenBSD Cryptographic Framework, as portedto FreeBSD,[Leffler, 2003].so any cryptographic hardware is automatically usedto accelerate their operation.Fast IPsec, running on a uniprocessor system with a singleBroadcom BCM5822 cryptographic processor,has demonstrated throughput ofmore than 400 megabits/second when acting as an IPsec terminator.This is more than 50% higher than any other freely availableIPsec implementation.

Background and Introduction

The IP Security protocols (IPsec)are a suite of protocols[Kent & Atkinson, 1998a].standardized by the IETF[Kent, 1998; Kent & Atkinson, 1998b]for secure communication of IP datagrams.IPsec is comprised of three protocols: AH, ESP, and IPCOMP.AH provides authentication, ESP encryption and optionally authentication,and IPCOMP adds compression.Any or all of these protocols can be combined though most uses of IPsecemploy only the ESP protocol (with the optional authentication).Several freely available implementations of the IPsec protocols exist:the KAME Project distributes an IPsec suite that has beenintegrated into FreeBSD and NetBSD,OpenBSD has their own IPsec implementation,while the FreeS/WAN Project distributes an IPsec implementation for Linux.Of these implementations only the OpenBSD software includessupport for using cryptographic hardware to accelerate the protocols.Cryptographic hardware vendors sometimes providesoftware to integrate their productswith some of these IPsec implementationsbut these tend to be special-purposeand only a few are freely available.[Communications, 1999]

Fast IPsec is an implementation of the IPsec protocols forFreeBSD that was designed for high performance.In particular the protocols use theOpenBSD Cryptographic Framework, as portedto FreeBSD, so any cryptographic hardware is automatically usedto accelerate their operation.Fast IPsec, running on a uniprocessor system with a singleBroadcom BCM5822 cryptographic processor,has demonstrated throughput ofmore than 400 megabits/second when acting as an IPsec terminator.[Ambrisko, 2003]This is more than 50% higher than any other freely availableIPsec implementation.

Fast IPsec is derived from the KAME IPsec, but integrates manyof the lessons OpenBSD learned when adding hardwareacceleration to their IPsec implementation.The decision to start from the KAME implementationwas made for two reasons:

1. The KAME distribution is the most widely used IPsecimplementation for BSD systems.Providing a successor implementation that isfamiliar to existing users simplifies adoption.
2. The OpenBSD IPsec is tightly integrated with other facilitiesthat conflict with or duplicate existing facilities in FreeBSD.

While Fast IPsec has been designed as a successor to the KAME IPsecit has also been written to coexist with the KAME implementation in thesource tree.This permits users to evaluate Fast IPsec without giving up theirexisting IPsec implementation.Further, because Fast IPsec and KAME share APIs,users do not need to learn a new set of configuration tools.

The remainder of this paper is organized as follows.Section 2 describes the Fast IPsec implementation.Section 3 discusses performance issues and the techniquesused to attain its high performance.Section 4 describes performance results for FreeBSDand compares them to other freely available IPsec implementations.Section 5 outlines the the status and availability of this workand talks about future work.Section 6 gives conclusions.

Fast IPsec Implementation

Fast IPsec is comprised of four protocols(AH, ESP, IPIP, and IPCOMP),the security database,PF_KEYsocket support through which applications interact with the database,and glue code that resides in protocols such as IP, TCP, UDP, and ICMP.There is no cryptographic transformation or compression code inFast IPsec; this comes entirely from the FreeBSD cryptographic framework.

Protocols

The protocol implementations in Fast IPsecare completely different from those found in KAME.They borrow heavily from what is found in OpenBSD in several ways:

1. The protocols are structured in a ``continuation style''that permits the decoupling of cryptographic andprotocol processing.That is, each of the input and output processing paths for AH, ESP, and IPCOMP are broken up into ``before'' and ``after''portions that invoke the cryptographic framework and then continueprocessing on return through a callback mechanism.
2. Many code paths are unified to handle both IPv4 and IPv6protocols.This eliminates code duplication.
3. IP-in-IP encapsulation is implemented as a separate protocol.KAME handles the encapsulation/decapsulation of packets for tunnelsas special case code in the ESP and AH protocols.
4. Header data are retrieved from packets with``m_copydata''instead of forcing mbuf data to be contiguous.This eliminates many assumptions about the handling of data inlayers above and below.Performance measurements indicate doing this adds no noticeable costover the techniques used by KAME.

Otherwise, notable differences in the protocols are inareas like statistics.Fast IPsec is careful to record error statistics that uniquelyidentify each problem; this is critical for diagnosing systemswithout source code.

Packet Tags

Aside from the cryptographic framework that is described elsewhere,[Leffler, 2003]Fast IPsec required only the addition of ``packet tags'' to FreeBSD.Packet tags are used to associate typed variable-length data with a packet.The standard mbuf routines propagate these tagswhen packet headers are moved or duplicated and theyare automatically reclaimed when an mbuf chain is freed.Packet tags originated in OpenBSD[Keromytis, 2003].but they were changedin two important ways:

1. OpenBSD allocates all storage withM_NOWAIT(do not block to wait for memory).This is too inflexible for general use.In FreeBSD routines that allocate tag storage take a parameterthat specifies whether the caller is willing to block when allocating storage.
2. OpenBSD defines packet tag type values as a single 16-bit value thatmust be centrally administered to ensure uniqueness.In FreeBSD this 16-bit value was expanded to add a 32-bit .Tm cookiethat specifies an ABI or module ID.By convention cookies are defined as the date and time that a moduleis created, expressed as the number of seconds since the epoch (e.g.using the output of ``date -u +%s'').This scheme permits software modules to be developed without the needfor a central administrator.[Elischer, 2002].Developers define a unique cookie and then manage tag types privately.

When these changes were made, OpenBSD-compatible shims were provided sothat cross-platform compatibility could be maintained in code thatneeded to be portable.Packet tags were also used to replace the KAME ``auxiliary mbuf'' mechanism.

Security Database

The security database and policy management code is derivedfrom the KAME implementation.While numerous changes were done to reduce memory usageand improve performance, its design has been left mostly intact.

The most notable change was to replace the use ofsockaddr_storagefor recording network addresses with asockaddr_uniondata structure that is a union of the potential address formats.This reduces the storage for a network address from 128 bytes to 28.For systems with limited memory this is significant as each entry inthe security database records at least two addresses.This change also reduces the size of the runtimestack and the amount of data referenced by structurecopy and zeroing operations.

Otherwise the explicit algorithmspecifications and algorithm-related data were replacedwith references totransformation routines through which the protocols interactwith the cryptographic subsystem.This technique has been used successfully in other systems[Keromytis et al, 1997; Spencer et al, 2002].

Locking

Fast IPsec was initially developed for the 4.6 release of FreeBSD.This version of the operating system uses traditional synchronizationtechniques designed for a non-preemptive uniprocessor environment.Only one thread of execution is expected to be active at a time.Code that needs to synchronize access to data structures does thisby blocking interrupts so that asynchronous events are disabled.

FreeBSD 5.0 is the first release of FreeBSD to have a fullypreemptive kernel.In this environment it is undesirable to guard execution paths to ensure synchronization of data structures[Hsu, 2003].Instead locks are associated with data structures and concurrentthreads of execution are managed by preempting a thread when it encountersa locked data structure.This can simplify code and make proper locking more intuitive.

Fast IPsec has very few locking requirements.The protocols are expressly designed to function as separate threadsthat depend only on private data.The only central data structure that needs synchronizationis the security database.The database is referenced by code that is executed on behalf of systemcalls (applications sending data), by asynchronous activities thatoccur because of incoming network traffic,and by timer-driven threads that do things like expire securitypolicies.Prior to FreeBSD 5.0 it was possible to block asynchronous accessto the security database by raising the processor priority tosplnet,but with a fully preemptive kernel this no longer works.Instead a set of locks were added to guard accessesto each of the major data structuresand references to data structures are safeguarded with reference counters.These changes were straightforward and sufficiently fine-grained that there isno noticeable lock contention (as measured by the lock profiling facilities).The only real issue is the checking of security associations whentransmitting packets.Due to the structure of the database inherited from KAMEit is necessary to lock theipsecrequeststructure to ensure references to security associations are safely held.This turns out to be a ``hot spot'' that limits performance for bidirectionaltraffic flow.A redesign of the data structures to eliminate this lockingis in progress.

Performance Analysis

There are several major areas to study to understand the performance ofFast IPsec:the cryptographic subsystem,the protocol implementations,the security database,andthe network interface drivers.(There are other components such as mbuf and memory allocators buttheir individual performance tends to be less critical.)By far the majority of the overhead associated with IPsec is in thecryptographic processing.This is why it is so important to accelerate and offload the workfrom the main CPU.However the interactions between the various software components and thesystem architecture can have a noticeable effect on overall performance too.The next sections discuss these issues in more detail.

Crypto Subsystem

Fast IPsec uses the FreeBSD cryptographic framework todo all encryption and authentication work.This subsystem provides general-purpose device-independentsupport for a varietyof transformations, including the symmetric-key cryptographicoperations required by the ESP and AH protocols.The crypto support is comprised of a core set of code that managesrequests and a set of device drivers for cryptographic devices.In addition there is a software-only device driver that implements symmetric-key operations on the host CPU for systems that do nothave hardware devices.

The payload of each packet sent and received is passed to the cryptosubsystem.Prior to dispatching the crypto requestall the data needed to processthe packet on returnare collected and associated with the request.When the operation completes the crypto subsystem invokes acallback function stored in the request.This callback method completes the processing for the packet anddispatches the packet either ``up'' (for reception)or ``down'' (for transmission).

[Leffler, 2003]describes the work that was done to optimize the performance ofthe FreeBSD cryptographic subsystem.Increased performance of the crypto subsystem directly affectsthe performance of Fast IPsec.Of particular note is the work done to reduce latencyin processing cryptographic requests.Compared to other systems, the FreeBSD cryptographicsupport has significantly less overhead and lower latency.This is reflected in significantly higher performance, especially forembedded systems where CPU cycles spent on overhead are more noticeableand for high-end systems where keeping the cryptographic hardwarebusy is critical to optimal performance.

Data Handling, Alignment, and Fragmentation

The performance of network protocols is typically constrained by theefficiency with which data are moved (or not moved) and manipulated.Because IPsec requires significant computation to process each packet,inefficiencies in this area can be less noticeable.Nonetheless, while tuning the performance of Fast IPsecdata handling issues frequently appeared.

One issue was the need to properly align packetdata to ensure it is always processed with the ``fast path.''For example, some network interface drivers did not properly alignreceived Ethernet frames so that the IP header is aligned to a 32-bitboundary.This can force data to be copied by protocols as the packet is passed upthe stack.Further, when this data is passed to a crypto device driver,misaligned data can require additional copying.Several drivers with problems of this sort were fixed and theFast IPsec protocols take care to ensure data are optimally alignedfor ancillary operations.Fast IPsec and the FreeBSD cryptographic frameworkalso keep statistics on any misaligned or otherwise suboptimaldata manipulations.

Optimizing the input data path is simpler than optimizing the output path.For devices with a fixed-size link-level header,drivers can set up receive buffers so that dataare contiguous and well aligned.As packets work their way up the protocol stack protocol headers canbe efficiently removed.The only issue is ensuring proper alignment of data thatrequires cryptographic processing and this happens automaticallyif the IP header is aligned to a 32-bit boundary.

Optimizing the output path is a bit more involved.Headers must be prepended and packets must be rewritten with cryptographictransformations.Data that comes from a stream socket typically must be copied to createa writable version that can be transformed.Other data, such as packets being forwarded by an IPsec terminator,may be writable and not need to be copied.Fast IPsec creates a writable copy of an mbuf chain with the``m_clone''routine.This routine uses an.I "aggressive coalescing"technique that tries to compact the resulting mbuf chain andlinearize the data, but balancesthis goal against the cost of copying already writable data.Compacting an mbuf chain is good in that it reduces thenumber of individual segments that must be processed; especiallywhen arranging DMA to/from cryptographic hardware devices.Linearizing data is critical to the efficient use of cryptographichardware that does not support scatter/gather DMA;but it can also improve performance of software algorithms.The scheme employed by``m_clone''uses the following rules:.IP 1) 3``Inline mbufs'' are coalesced only when there is a clusterimmediately preceding that has space to hold the data..IP 2)Mbufs with writable external storage are left untouched..IP 3)Mbufs with read-only external storage must be copied.If there is space in the immediately preceding mbuf, then the data are copied.Otherwise, new storage is allocated and the data are copied.Data larger than a cluster is broken into multiple mbufs.

This scheme is designed for Ethernet traffic where the maximumframe size fits in a cluster.Further it depends on the ability to identify whether or not mbufs with external storage are writable.Finally, the ``busting of jumbograms'' is required because many driversassume packets can be directly mapped for DMA.

In practice many packets are coalesced into a single mbuf with allthe data linearized.Packets being forwarded are easily identified as writable in FreeBSD 5.0but require some special handling in 4.x versions of the system. [1]

[1] In FreeBSD 5.0 read-only mbuf chains are explicitly marked, but in4.x one must check a reference count that may be maintained in adriver-private data area.

Statistics kept on the operation of this scheme indicate about 45%of the packets that require copying are coalesced into a single mbuf;but these numbers are believed to be artificially low.(Most traffic is from performance benchmarks that run on the gatewaymachine. In this case traffic is TCP-based and the data are receivedlocally instead of being forwarded. Both these factors affect the results.)The issue with mapping packets for DMA is discussed below.

Network Interface Drivers

The operation of the network interface driver can strongly influenceperformance.Fast IPsec was tested with a wide variety of hardware and drivers.Performance varies significantly depending on load and frame size.The currently preferred device is the Intel PRO/1000 whichcomes in 32-bit and 64-bit PCI configurations.

Two issues with the driver in the handling of large packetswere identified and fixed.First, the driver did not support the ``bus dma'' API formapping mbuf chains on to the system bus for DMA.This meant that packets larger than a physical page had tobe broken up intomultiple mbufs (as described above for the``m_clone''function).The driver was redone to use the bus dma functions so that outboundpackets may be left intact (though they presently are still brokenup into clusters).

The second issue was that received jumboframes were written to multiplesegments and an mbuf chain was then constructed.An alternative approach is to allocate contiguous receive buffers.This however can require pre-allocation of a significant amount of memorybecause the device constrains receive buffer sizes to be a power of two(so a 9000-byte mtu requires 16 kilobyte receive buffers).The driver was changed to optionally allocate contiguous memory forreceiving large frames.

This revised driver performs noticeably better than the standard one.With a 9000-byte mtu set on both interfaces of systems connected bya cross-over cable,netperfperformance results with the standard driver drop by more than 210 Mb/swhile performance for the driver with contiguous receive buffers increasesby a modest amount(the increase is small because performance is already near peak andone of the machines is constrained by PCI bus bandwidth.)

System I/O Performance

Performance is influenced by many system-level issues.Cryptographic requests directed to hardware devices require twotrips across the I/O bus for each operation.This means that I/O performance is very important in understandinga system's capabilities.Bus width, latency, and device configuration can significantly affectperformance results.Kernel profiling indicates that most systems are limitedby their ability to field and process interrupts.In an IPsec terminator configuration each packet requires fourDMA operations (two for NIC DMA and two for hardware crypto DMA) andas many as four interrupts to be processed.This implies that interrupt overhead must be minimized for the highestpossible performance.In particular IRQ multiplexing must be avoided as well as system overheadlike harvesting ``IRQ entropy'' for the pseudo random number generator (PRNG).Polling techniques such those available in FreeBSD[Rizzo, 2001]can be useful in reducing this overhead.Some vendors of hardware crypto products claim tointelligently coalesce interrupts in their hardware but it is unclear ifthese are anything more than marketing hyperbole.

With regard to the PRNG, most cryptographic hardware includesa hardware random number generator (RNG)that can supply sufficient entropy to seed the system PRNG.While this can eliminate expensive techniques used to collect entropy data,care must be taken to evaluate the quality of the entropy suppliedby the hardware.Rndtestis a kernel module that was developed to testthe quality of random data sources[Wright & Leffler, 2002].It monitors data on the way to the system PRNG to see if it complieswith the FIPS 140-2 standard[Federal Information Processing
Standards, 2002].If data fail any of the FIPS 140-2 tests, they arediscarded and no data from the source are passed to the system PRNGuntil compliant data are seen.Testing can be done continuously or periodically and there are variouscontrols (e.g. whether to report problems to the system console)and statistics maintained.This facility has shown, for example, that the entropy data producedby a Broadcom BCM5822 device is sometimes unreliable.

Performance Results

The IPsec protocols are very flexible.There are three protocols that can be combinedto generate a variety of packet formats.Encapsulation may also vary:there is a.Tm transportmode in which the IPsec protocol headers are directly encapsulatedin IP and a.Tm tunnelmode in which an additional IP encapsulation is done when an intermediatemachine acts as an IPsec.Tm "terminator"or.Tm "gateway" .Tunnel mode is typical of IPsec-basedVirtual Private Network (VPN) applications..Fg "Typical VPN gateway configuration" "fig_1.eps"

For this paper we analyzed the performance of IPsec operatingin tunnel mode using only the ESP protocol, but doing bothencryption and authentication of the payload.When IPsec is deployed in this manner the network is typically configuredas a VPN gateway, as shown in Figure 1.Specifically, IPsec data are received from peers, decrypted and authenticated,and forwarded in the clear to trusted clients.This means each packet requires one pass through cryptographic processingbefore it is forwarded.This is in contrast to a point-to-point tunnel configuration wheredata are cryptographically processed twice.Throughput measures for an IPsec gateway will besignificantly higher than for a point-to-point tunnel.However collecting performance data for a gateway configuration is morecomplicated because it requires multiple clients and/or peers to saturate a Fast IPsec gateway.With multiple clients and/or peers it is also necessary to include a switchin the configuration and this device can become a bottleneckwhen evaluating performance.Therefore, to ensure others can easily reproduce the results presented here,we have chosen a network configuration in which two machines are physicallyconnected with a cross-over cable.Testing done on a multi-client terminator configuration confirmsthe result for this simpler configuration apply directly.

The test configuration has two systems, A and T,connected by a cross-over cable.Machine A has an Asus P4B533-V Intel845G motherboard with a1.8 GHz P4 processor.A dedicated Intel Pro/1000 (82540) NIC located in a 32-bit PCI slotwas used for testing.Machine T has a Tyan S2707G2N motherboard with a 1.8 GHz P4 processor.The Tyan system has dual Intel gigabitEthernet devices (82545 and 82551) on-board.The 82545 has a 64-bit PCI interface and the 82551 has a 32-bit PCIinterface.Testing was done using the 82545.The mtu on each interface was the default, 1500 bytes.

To establish baseline performance, tests were first run on anopen network.Table 1 shows data collected usingnetperfversion 2.2pl3.The data were collected with a 99% confidenceinterval (+/- 2.5%), identical buffering parameters,and a 32 kilobyte message size.

Table 1: Raw netperf results for fast testbed

A->T	T->A	Operating
Mb/s	%Sys	Mb/s	%Sys	System
627	75	55	799	84	78	FBSD 4.8 [2]
519	75	24	653	48	32	OBSD 3.3 [3]

[2] FBSD 4.8 is a July 4, 2003 snapshot of FreeBSD.

[3] OBSD 3.3 is a March 17, 2003 snapshot of OpenBSD.

Next a manually-keyed IPsec tunnel wasset up between the two machines.All traffic was encrypted with 3DES and authenticated with SHA1.Table 2 shows results for several configurations.The transfer rates reported by netperfare provided as well as the percentage of system timefor the sending and receiving machines, as reported by thevmstatprogram.Performance is different depending on which machine is initiating thetransfer; ``A->T'' results are for when netperfwas run on the Asus-based machine, while ``T->A'' is when netperfwas run on the Tyan-based machine.In one configuration a Broadcom BCM5822 was placed in each machine.Broadcom data sheets claim the 5822 can do 3DES+SHA1 calculations at 500 Mb/s.Testing described in[Leffler, 2003]measured a peak performance for a 5822of 470 Mb/s when located in a 64-bit PCI slotand about 410 Mb/s when located in a 32-bit PCI slot.In the other configurations crypto calculations were done by the host CPU;this is denoted by ``sw/''.

Table 2: IPsec tunnel results for fast testbed

A->T	T->A	Operating	Crypto
Mb/s	%Sys	Mb/s	%Sys	System	Support
160	80	73	170	70	94	FBSD 4.8	BCM5822
n/a	-	-	n/a	-	-	OBSD 3.3	BCM5822
43	100	93	44	89	97	FBSD 4.8	sw/FastIPsec
42	95	95	43	100	100	FBSD 4.8	sw/KAME
27	87	94	27	88	96	OBSD 3.3	sw/OpenBSD

The results show Fast IPsec is about 60% faster than OpenBSDwhen using software crypto and slightly faster thanthe KAME implementation.The higher performance relative to OpenBSD shows the value of FreeBSD'soptimized cryptographic subsystem.The comparison to KAME demonstrates that moving the crypto supportout of IPsec into a general-purpose framework can be done withoutlosing any performance.

Performance comparisons for Broadcom-accelerated configurationwere not possible.KAME does not support hardware crypto acceleration and whileOpenBSD claims support for the 5822 it did not work; trafficstalled and the tests never completed.However, based on the performance for raw cryptographic operationsreported in[Leffler, 2003]and the relative network performance over the unencrypted link,one can assume Fast IPsec will be significantly faster than OpenBSD.

Performance in this simple network configuration is limitedby the slowest component in the loop; in this case the CPU is thelimiting factor.Tests with faster processors and the 5822show performance of FreeBSD scales linearly with the CPU speed up to 218 Mb/sfor 2.53 GHz P4 processors.At that point I/O bandwidth to the 5822 in Machine Abecomes the limiting factor.64-bit PCI support on Machine A would makea significant difference in the results.

Because no direct comparison to OpenBSD was possible with the Broadcomhardware a second testbed was set up with different hardware.In this configuration two slower systems were used, each with aGTGI XL-Crypt card.The XL-Crypt uses a Hifn 7811 crypto part and can do 3DES+SHA1calculations at approximately 145 Mb/s.The two systems were purposely slower to match the performanceof the cryptographic hardware(using slow crypto hardware in a fast machine yields uninteresting resultsbecause the cryptographic hardware becomes the bottleneck.)Machine D was a Dell XPS 300 with 266 MHz PII processor.Machine E was an E-Machines etower 366i with a 366 MHz PIII processor.As before the systems were connected by a cross-over cablebut this time only 100 Mb/s Ethernet was used.Table 3 shows the results for running netperf over a manually-keyed IPsec tunnel on this hardware.

Table 3: IPsec performance results for slow testbed

D->E	E->D	Operating	Crypto
Mb/s	%Sys	Mb/s	%Sys	System	Support
45.9	81	20	51.4	92	22	FBSD 4.8	Hifn 7811
37.3	96	40	33.0	81	30	OBSD 3.3	Hifn 7811
9.8	100	93	9.4	99	92	FBSD 4.8	sw/FastIPsec
8.9	100	94	9.2	100	94	FBSD 4.8	sw/KAME
5.3	100	84	7.0	89	98	OBSD 3.3	sw/OpenBSD

FreeBSD was 23/55% faster than OpenBSD when using hardware acceleration.With crypto operations done in software the difference was 34/84%.As the system performance decreases the FreeBSD optimizations are lessnoticeable because other factors become significant.In this case the I/O performance of the Dell system is so poorthat it is the critical factor in determining overall performance.

Status and Future Work

An initial version of Fast IPsec was completed September 2001.Integration of this work into FreeBSD was completed November 2002 andcommitted to the stable branch in January 2003.The work described here, except for the fine-grained locking,is freely available as part of the FreeBSD 5.0 and 4.8 releases.Fast IPsec is currently being integrated into the NetBSD operating system[Stone, 2002].Several vendors have incorporated Fast IPsec in their products.

Fast IPsec lacks support for IPv6;this has yet to be done because of lack of interest.TheIPCOMPsupport is fully implemented but is not workingdue to an issue with the gzipcompression support in the FreeBSD kernel.

The security databaseimplementation needs to be redone.Much of the information stored in this database is better co-located withthe routing tables so that the routing table lookup algorithms can be used.In addition much of the data stored in the databaserequires two and three levels of indirection to access.Relocating this data will simplify the protocols and eliminate some ofthe locking currently required to ensure indirect pointers do not changewhile the protocol code follows indirect references.A redesigned databaseis likely to provide a noticeable performance improvement for systemsof any significant scale.

There is no support for the cryptographic algorithms required bythe legacy versions of the AH and ESP protocols.Correcting this requires changes to the cryptographic framework APIand may not be worthwhile.

ThePF_KEYsupport used to communicate with user-level applications is intertwinedwith the IPsec security database implementation.This needs to be changed by separatingthe pfkeyv2 protocol and database implementations.This would fix, for example, problems where changing aspects ofPF_KEYbreak user-mode applications such as racoon andsetkey.

Otherwise, the most significant deficiency in the current design isthe inability to use protocol-specific hardware operations.Most vendors of crypto hardware optimize their products for use as``all-in-one'' devices that take an IPsec packet and parse theprotocol and perform the cryptographic transforms in a single request.This is incompatible with the current general-purpose API provided bythe cryptographic framework.Supporting this kind of operation requires exposingIPsec state that is currently private.However adding this is the only way that some hardware devices can beused at all as they do not otherwise provide access to the cryptographictransformation hardware.

Conclusions

Fast IPsec is an implementation of the IPsec protocols that hasbeen designed as a plug-compatible successor to the KAMEIPsec protocols with high performance.The software has been demonstrated to have performance more than 50%better than any other publicly available IPsec implementation.

Acknowledgements

This work incorporates ideas, and some code, fromthe OpenBSD IPsec implementation; without it this work wouldprobably not have been done.Angelos Keromytis was helpful in understanding howbits and pieces of OpenBSD worked.

Vernier Networks supported this work; their contributionis greatly appreciated.Doug Ambrisko of Vernier Networks contributed many bug fixes and wasa great help in testing.

References

[Ambrisko, 2003].D. Ambrisko, Private email: 400mbs IPsec numbers(February 2003).

[Communications, 1999].RedCreek Communications, Inc., RedCreek IPSec VPN Card, Source for the driver is included in Linux starting with 2.4 (April 1999).

[Elischer, 2002]. J. Elischer, "Re: CFR: m_tag patch," freebsd-arch@freebsd.org (November 2002).

[Federal Information Processing Standards, 2002]. Federal Information Processing Standards, "Security Requirements for Cryptographic Modules" (FIPS PUB 140-2), National Institute of Standards and Technology (December 3, 2002).

[Hsu, 2003]. J. Hsu, "Reasoning about SMP in FreeBSD," BSDCon 2003 (September 2003).

[Kent, 1998]. S. Kent, "IP Authentication Header," RFC 2402 (November 1998).

[Kent & Atkinson, 1998a]. S. Kent & R. Atkinson, "Security Architecture for the Internet Protocol," RFC 2401 (August 1998).

[Kent & Atkinson, 1998b]. S. Kent & R. Atkinson, "IP Encapsulating Security Payload (ESP)," RFC 2402 (November 1998).

[Keromytis, 2003]. A. Keromytis, "Tagging Data In The Network Stack: mbuf_tags," BSDCon 2003 (September 2003).

[Keromytis et al, 1997]. A. Keromytis, J. Ioannidis, & J. Smith, "Implementing IPsec," Proceedings of Global Internet (GlobeCom) '97, pp. 1948-1952 (November 1997).

[Leffler, 2003]. S. Leffler, "Cryptographic Devices Support for FreeBSD," BSDCon 2003 (September 2003).

[Rizzo, 2001]. L. Rizzo, "Polling versus Interrupts in network device drivers," BSDConEurope 2001 (November 2001).

[Spencer et al, 2002]. H. Spencer, R. Briggs, D. Redelmeier, M. Richardson, S. Harris, & C. Shmeing, Linux FreeS/WAN (April 2002).

[Stone, 2002]. J. Stone, Private email: Re: Fast IPsec: patches for Fr eeBSD 4.x? (November 2002).

[Wright & Leffler, 2002. J. Wright & S. Leffler,

[rndtest (2002)]. https://www.freebsd.org/cgi/cvsweb.cgi/src/sys/dev/rndtest.

Biography

Sam Leffler has been actively working with UNIX since 1975when he first encountered it at Case Western Reserve University.While working for the Computer Systems Research Group (CSRG)at the University of California at Berkeley he helped withthe 4.1BSD release and was responsible for the release of 4.2BSD.He has contributed to almost every aspect of BSD systems;most recently working (again) on the networking subsystem.You can contact him via email at<sam@errno.com>.