Why softirq is used for highly threaded and high frequency uses?

What makes the softirq so special that we use it for high frequency uses., like in network drivers and block drivers.

SoftIrqs are typically used to complete queued work from a processed interrupt because they fit that need very well - they run with second-highest priority, but still run with hardware interrupts enabled.

Processing hw interrupts is the utmost priority, since if they are not processed quickly, then either too high of latency will be introduced and user experience suffers, or the hardware buffer will fill before the interrupt services the device, and data is lost. Dont service a network adapter fast enough? It's going to overwrite data in the fifo and you'll lose packets. Don't service a hard disk fast enough? The hard drive stalls queued read requests because it has nowhere to put the results.

SoftIrqs allow the critical part of servicing hardware interrupts to be as short as possible; instead of having to process the entire hw interrupt on the spot, the important data is read off the device into RAM or otherwise, and then a SoftIrq is started to finish the work. This keeps the hardware interrupts disabled for the shortest period of time, while still completing the work with high priority.

This article is a decent reference on the matter: https://lwn.net/Articles/520076/

Edits for questions:

SoftIrqs are re-entrant - they can be processed on any cpu. From the article I linked:

There are two places where software interrupts can "fire" and preempt the current thread. One of them is at the end of the processing for a hardware interrupt; it is common for interrupt handlers to raise softirqs, so it makes sense (for latency and optimal cache use) to process them as soon as hardware interrupts can be re-enabled

Emphasis added. They can be processed inline - I believe this means they can be processed without causing a context switch, which means as soon as hardware interrupts are enabled, we can jump straight to the SoftIrq right where we are with as little CPU cache abuse as possible. All of this contributes to SoftIrqs being lightweight but flexible, which makes them ideal for high-frequency processing.

• They can be pushed to another CPU if needed, which improves throughput.
• They can be processed immediately after hwints are enabled right in the current context, preserving processor state as much as possible, improving latency
• They allow hardware interrupts to keep processing, since those are our most important goal
• They can be rescheduled to the ksoftirqd process if load is too high and we need to take time from normal user processes.