My filter driver consists exclusively of global state, shared among all filter modules. When a [rare] configuration change needs to take place (coming in the form of an IOCTL from UM), I need to be capable of temporarily disconnecting the data-path (filter rx/tx) to be capable of carrying out the reconfiguration before then resuming it. A lock of any sort on the data-path is unacceptable.
While requesting a restart (via NdisFRestartFilter()) and performing the reconfiguration during the Attach transition seems like the correct approach, the problem is that this is only for an individual module. One way this could work is if it were permissible to synchronize all of the modules at their Restarting (or Pausing) transitions, using a semaphore (since I believe all these NDIS control routines, and certainly the IOCTL, will be in a context that is <= APC) as a fence to signal the originating thread handling the IOCTL to perform the reconfiguration, before then releasing the waiters.
The problem with this approach is that I don't know if it's permissible to synchronize during these filter state transition callbacks; given it's the NT kernel I would generally assume no, and it may even deadlock NDIS.
Other options I can see are:
a) Restart all modules and switching the data-path entries to dummys which just drop traffic, do the reconfigure, and switch back to the normal entry points.
b) Restart the driver (undesirable; it would require persisting some runtime state to disk first)
One additional option would be to invoke NdisFDeregisterFilterDriver and NdisFRegisterFilterDriver from your ioctl handler. This would lean on NDIS to do the heavy work of synchronizing all your filter modules. The downside is that there's a brief window of time where your filter is not attached to anything. If you have a Mandatory filter, this means that all traffic will be blackholed. If you have an Optional filter, this means that traffic will still flow, but your LWF won't participate in it. Whether this is acceptable depends on the nature of your LWF. If you do this, don't forget to wrap this in a critical region or kick the work off to a system worker thread; and protect it with a mutex or single-dispatch WDF queue.
Note that you do not need to run through DriverUnload -> DriverEntry just to be able to deregister/register the LWF with NDIS. NDIS is not really aware of your driver's state, and you can manipulate your LWF state independently. So I imagine this scheme is an improvement over option (b). It may be an improvement over option (a), if you're okay with NDIS's handling of traffic while your filter is not registered.
A lock of any sort on the data-path is unacceptable
That's a noble spirit, but you can go a little easier on yourself here. NdisAcquireRWLockRead is specifically designed to be used from the datapath. In the case where you call it from DISPATCH_LEVEL and there's no contention from any writers, you're paying 10 or 20 nanoseconds for the lock. That's obviously not ideal to do per-packet, but if you only do it per batch (per function call to FilterReceiveNbls or FilterSendNbls), it amortizes out okay. For comparison/CYA, several of the LWFs that are built into the OS acquire RW locks for Read on the datapath.
You can get a little further with lock-free techniques, but I wouldn't recommend them unless you are looking at an ETL that shows you spend too much CPU fiddling with locks. For example, you could keep an configuration "epoch" aka "generation number" as a global variable, and also in per-filtermodule state. Do a quick check if your local epoch is equal to the global epoch and skip all the synchronization if it is. This, plus a KeFlushQueuedDpcs in the configuration path, can really minimize the synchronization overhead on the datapath.
In general, you can't avoid having some synchronization overhead. The only real trick here is that you might be able to make someone else do it for you. If you can do everything with Pause/Detach, that's essentially pushing the synchronization problem out into the miniport and protocol. Any driver that originates NBLs is required to implement some synchronization to correctly handle Pause; if your LWF doesn't originate NBLs, you can lean on the drivers that do originate NBLs to synchronize the datapath for you, i.e. the miniport and protocol drivers. (NDIS itself does not have any synchronization of its own in the datapath, and merely trusts that drivers honor the contract correctly.)
I believe all these NDIS control routines, and certainly the IOCTL, will be in a context that is <= APC
Yes. For example, FilterPause has PASSIVE_LEVEL in its Requirements table. That's NDIS's guarantee to you that your FilterPause will only ever be invoked at PASSIVE_LEVEL. Some amount of waiting is acceptable there, and indeed is typical: your LWF is expected to wait for any NBLs that it originated to be returned to it.
The problem with this approach is that I don't know if it's permissible to synchronize during these filter state transition callbacks; given it's the NT kernel I would generally assume no, and it may even deadlock NDIS.
Your instinct is good: I would strongly discourage trying to synchronize the datapath states across all your LWF's filter modules. I'm certain it'll deadlock if you have a Monitoring filter, since you'd have multiple filter modules on the same network adapter. But even for Modifying LWFs, it'll probably also deadlock in special cases where there are subtle dependencies between miniport stacks.