One feature which was needed by dmfsd and the other utilities was an ability to perform arbitrary operations, such as start archive, finish restore, etc., on the file(s) which it was manipulating. At the same time as we were addressing this need, there was a desire to add the ability to add the capability to perform file/inode operations. One such example is adding the ability to manipulate access control lists on file systems which support them.
These operations would be similar at the VFS level to ioctl(2)s except that they would always reach the file system, rather than possibly being dispatched to device drivers as is the case for ioctl(2)s.
While there was no objection to adding this extension to the VFS interface, the form of its programmatic interface generated controversy and much discussion on NetBSD's kernel technology email list. The desired interface would consist of a file descriptor, a command code, and a data argument (void *). There were three options: add a new system call with this parameter signature, overload the fcntl(2) interface, or overload the ioctl(2) interface.
In the end, I opted for using the fcntl(2) system call. The main reasoning is that we should allocate the entire command space reserved for the new vnode operation now - it would not be advisable to reserve some now and then add more later. There are far fewer fcntl operations than ioctl operations in NetBSD, and they are less frequently added, so it is less likely that using fcntl(2) as the programmatic interface and reserving a sizable command space now will impede future kernel development. Additionally, in the case of overlay-type layered file systems, different layered file systems will need to choose unique codes, and pass codes they do not understand to underlying file systems, so that tools designed to operate on one particular file system type will operate regardless of the depth of layered file systems. This requirement is easier to satisfy with a spacious reservation of command space.
The fcntl(2) system call takes an integer as its command code. If the most significant bit is set, the operation is now considered a request for a file system-specific operation - a VOP_FCNTL() call. This division leaves approximately 2^31 commands available for traditional fcntl-type operations. The remaining 31 bits encode whether a value or a structure are read into or out of the kernel (3 bits), the size of data so transferred (12 bits), and the actual command code (16 bits). Half of the command space (32768 commands) is reserved for NetBSD use, while the remaining space is for a file system's private use.