Rasberry Pi 4 USB serial port pyserial timing issue

I have a setup where Raspberry Pi 4 is connected to USB hub with 5 USB-Serial converters (https://www.dfrobot.com/product-104.html)

Each converter in connected to STM32 based device. The devices are also powered with the USB converter. The devices do not draw much power and USB converter also has external power supply connected to it.

I'm trying to use pytest to run automated tests on the devices. The tests write commands to the devices and wait for response. The devices interact with each other so the commands are sent to many devices roughly at the same time.

The setup seems to work fine if the tests are run manually one by one, but when running all tests together (pytest runs them still serially so race conditions on the serial ports shouldn't happen) the first one succeeds always but the rest seem to fail pretty randomly. The randomness is not completely random though, since if I change delays in the code the tests fail differently but if no changes are made then the tests fail always the same way.

All ports and all devices are proven to be working (some tests use all devices and ports and have passed many times), so wiring should be OK and the USB serial converters are working.

I did some debugging and noticed that the test does write to the serial port, but sometimes there is nothing on the physical lines (checked with oscilloscope). So in the pytest/python side the write seems to be successful but still no data flows in the wires. This happens only sometimes, in some tests/delay configurations there is data flowing so the scope setup should also be OK.

Here is a snippet of the initializaton and transmitting of the serial data:

class Link_Serial(linkcore.LinkCore):
    def __init__(self, comport, packet_queue, ack_queue, id_queue, link_address):
        super().__init__(packet_queue, ack_queue, id_queue, link_address)
        self.serial = serial.Serial(port=comport, baudrate=115200)
        self.rx_buffer = []
        self.received_packets = []
        self.events = []

    def on_transmit(self, tx_data):
        print("TX", self.serial.name, self.serial.is_open, tx_data)
        time.sleep(0.02)
        self.serial.write(tx_data)
        self.tx_done()
        return 1

    def reset(self):
        self.rx_buffer.clear()
        self.received_packets.clear()
        self.events.clear()

        self.serial.reset_input_buffer()
        self.serial.reset_output_buffer()

        super().reset()

Changing the time.sleep(...) value slightly causes different tests to fail

Here is how pytest initializes the object for the test:

def get_module_n(n):

    if n >= len(modules) or n < 0:
        raise

    if radios[n] is None:
        info = modules[n]
        radios[n] = Link_Serial(...)
    radios[n].reset()
    return radios[n]

def stop_module(radio):
    radio.power_off()
    radio.reset()


...

@pytest.fixture
def radio0():
    ctrl = rack.get_module_n(0)
    yield ctrl
    rack.stop_module(ctrl)

@pytest.fixture
def radio1():
    ctrl = rack.get_module_n(1)
    yield ctrl
    rack.stop_module(ctrl)

...

Here is an example test that is very sensitive to the timing:

def test_many_senders_one_receiver(radio0, radio1, radio2, radio3, radio4):
    time.sleep(0.1)
    radio0.power_on()
    radio1.power_on()
    radio2.power_on()
    radio3.power_on()
    radio4.power_on()
    time.sleep(0.1)

    message = [1, 2, 3, 4, 5, 6]

    for radio in [radio1, radio2, radio3, radio4]:
        radio.send_packet(radio0.id, message)

    for _ in range(0, 1000):
        rack.run_links([radio0, radio1, radio2, radio3, radio4], 10)
        if radio0.get_events().count(LINK_DATA_RECEIVED) == 4:
            break

    assert LINK_DATA_ACKED in radio1.get_events()
    print(radio2.get_events())
    assert LINK_DATA_ACKED in radio2.get_events()
    assert LINK_DATA_ACKED in radio3.get_events()
    assert LINK_DATA_ACKED in radio4.get_events()
    assert radio0.get_events().count(LINK_DATA_RECEIVED) == 4
    assert len(radio0.get_packets()) == 4

Initially I had it so that pytest initializes a new object for each test, but then only the first test succeeded (all tests on their own when run manually worked fine). This way the object is initialized once and only reset between tests.

Turns out that one if the serial converters was faulty. Worked for a while and then stopped working. Also a bug in my code caused some of the messages to be delayed for a long time. Together these issues made the problem very difficult to debug.