linuxfilesystemsmipsjffs2

Reading a file kills NIC interrupts

I have a very strange problem with a board based on a MIPS processor and Linux 2.6. There are NIC interrupts for all incoming Ethernet packets. If I send 10.000 packets I could see that 10.000 NIC interrupts occurred.

START SYSTEM

SEND 10k PACKETS

/mnt/system # cat /proc/interrupts
          CPU0       
24:      10000         MIPS  NIC
29:       7192         MIPS  timer
30:          0         MIPS  UART1
31:       3092         MIPS  serial

ERR:          0

However, after I open and close a file (filled with zeros or regular) in the filesystem, there are much fewer NIC interrupts generated. For instance, just 2-7k interrupts for 10k packets. It has a detrimental effect on the system performance, but after rebooting everything with NIC interrupts is fine again.

START SYSTEM

std::fstream f;
f.open("/mnt/system/myfile");
f.close(); 

WAIT FOR SOME TIME

SEND 10k PACKETS

/mnt/system # cat /proc/interrupts
          CPU0       
24:       2045         MIPS  NIC
29:       7192         MIPS  timer
30:          0         MIPS  UART1
31:       3092         MIPS  serial

ERR:          0

The filesystem is jffs2 and the flash drive is 32M NOR serial device. Why reading a file kills NIC interrupts until reboot?

Solution

  • Caveat: This may not be a total solution, but I have some ideas. More information/testing may be necessary.

    When you do [the system does] nothing else, the NIC driver is fast enough to respond to an interrupt and the ISR will complete processing on the single packet before the next packet arrives. (i.e.) There is a one-to-one relationship between incoming packets and NIC interrupts.

    If you have other system activity, this may delay entry to the NIC driver's ISR. Also, this other activity may slow down processing in the NIC driver due to resource competition (e.g. locks, kmalloc, etc.)

    Meanwhile, more NIC packets arrive. When the NIC ISR is finally/ultimately entered, it sees that there are multiple packets pending. It will process all of them, without leaving the ISR. So, it might process (e.g.) 5 packets on each interrupt, so the interrupt count is reduced by 5x.

    This is what most "smart" NIC cards and drivers do. It actually improves throughput with heavy NIC traffic. Ordinarily, fewer NIC interrupts is a good thing because it reduces the wasted overhead of entering/exiting the ISR repeatedly.

    This really depends upon the packet arrival rate and [average] interval.

    So, the real question is: "When the interrupt count is reduced, do you lose packets/data or just see performance loss?"

    And, how are you measuring the system performance difference?

    Is the filesystem in question used otherwise (e.g. it is a root fs)? Or, is your program opening the file the only access to the fs [other than mounting it]

    I'm not familiar with jffs2 or your flash driver, so I'd ask if you suspect either of doing something that would disable interrupts for an extended period of time?

    UPDATE:

    I just noticed that you're using linux version 2.6 That kernel is approximately 10 years old. It's well past end-of-life, so it probably isn't getting bug fixes for drivers unless the platform vendor is providing them.

    So, another thing to consider is that any of the drivers may have a bug that causes the performance problem. There's a fair to good chance that the drivers have been fixed in more modern kernels.

    You might want to switch to a newer kernel if you can. If not, you may be left with the [unenviable] task of backporting the newer drivers to the older kernel [or at least cherry pick some of the bug fixes].