Introduction

I have my head scratching by one thing regarding the HWCDC and serial communication. It seems like serial.open() in various frameworks (reproducible with QtSerialPort and PySerial) does some initialization that needs to finish before serial.write() goes through, BUT only if the device reconnection happened during software runtime. So it needs some delay between open() and write(), which is unwanted in my project.

Code example

Let me explain it by code. Consider this minimal device code:

Config

platformio.ini

[env:tdisplay]
platform = espressif32
board = lilygo-t-display-s3
framework = arduino

Device code

It basically waits for 5B header and returns 256B of data.

#include <Arduino.h>

void setup()
{
    Serial.begin(115200);
}

void loop()
{
    // Set timeout to -1 so the "readBytes" waits forever for the first 5 bytes
    Serial.setTimeout(-1);

    // Read header
    uint8_t header[5];
    Serial.readBytes(header, 5);

    // Set timeout to 3 seconds in case something goes wrong, so we don't get stuck but handle the error
    Serial.setTimeout(3000);

    ... <handle the header here, header includes number of bytes that should follow> ...

    ... <in my test case, the code decides to return specific array of 256 bytes> ...

    Serial.write(responseBuffer, 256);
}

Python code

Now consider this Python code. It goes through infinite loop and just sends the command for obtaining said 256B of data and if the device returns less, print the "Response too short <number_of_bytes_read>" message.

import serial

ser = serial.Serial(
    port=None,
    baudrate=115200,
    timeout=0.5,
    write_timeout=None,
)

ser.port = <some specific port, "/dev/cu.usbmodem101" in my case>
command = <bytearray with command that expects 256B back>

while True:
    print("----------------------")
    try:
        # open the port
        ser.open() 
        print("Opened serial port")

        # CRUCIAL PART, WITHOUT THIS, THE DEVICE NEVER RECEIVES DATA AFTER RECONNECTION
        # time.sleep(1.5) # commented out now to introduce the problem in the next code block

        # write command that instruct the device to sent 256B back
        bw = ser.write(bytes(command))
        print("Wrote command " + str(bw))

        # read 256B of data
        br = ser.read(256)
        print("Read response 256")

        if len(br) != 256:
            print("Response too short " + str(len(br)))
    except serial.SerialException as ex:
        # We don't care in this test, most errors are due to device being disconnected from the PC (Mac in my case)
        print("Ex: " + str(ex))
    finally:
        # Close the port so it is ready for the next iteration
        ser.close()
        time.sleep(1)

Current behavior

Now, let me run the python code and print output here (I will describe my physical actions and thoughts in <>):

<I connect device to the PC>
<I start the python code>

----------------------
Opened serial port
Wrote command 7
Read response 256
----------------------
Opened serial port
Wrote command 7
Read response 256
----------------------
Opened serial port
Wrote command 7
Read response 256
----------------------   <So far so good, device is working as expected>
Opened serial port
Wrote command 7
Ex: read failed:         <I disconnect the device from the PC here>
[Errno 6] Device not
configured    
----------------------
Ex: [Errno 2] could not
open port /dev/cu.usbm
odem101: [Errno 2] No
such file or directory:
'/dev/cu.usbmodem101'
----------------------
Ex: [Errno 2] could not.  
open port /dev/cu.usbm
odem101: [Errno 2] No.   <These errors are expected as no device is connected to the PC >
such file or directory:
'/dev/cu.usbmodem101'
----------------------
Ex: [Errno 2] could not
open port /dev/cu.usbm
odem101: [Errno 2] No
such file or directory:
'/dev/cu.usbmodem101'
---------------------- 
Opened serial port       <I connect device back to the PC here>
Wrote command 7
Read response 256
Response too short 0     <Device is not returning any data?>
----------------------
Opened serial port
Wrote command 7
Read response 256
Response too short 0     <This continues indefinitely.>
----------------------
Opened serial port
Wrote command 7
Read response 256
Response too short 0
----------------------
Opened serial port
Wrote command 7
Read response 256
Response too short 0
----------------------   <How to fix this? Introduce delay after serial.open(). BUT WHY?>

Additional information

Here is some more relevant info I didn't put in code samples:

• I put LCD blinking at the end of setup() function. This tells me when exactly setup finished and loop() started
  • in DFRobot with basic HardwareSerial, it blinks almost immediately after introduction of power
  • in T-Display-S3 with HWCDC, it takes about 3 seconds to finish setup()
• I have some local flash storage logging on and I can see, that the device is always ready to receive commands after reconnection, but it never receives any, thus it has no reason to sent 256B of data back.
  • It basically call setup() only once after reconnection
  • This indicates problem on the OS level, not on the device level?
• The delay between serial.open() and serial.write() must be about 1.5-ish seconds or higher in order to "survive" device reconnection.
  • The delay itself was found out in this answer by user "r21"
• I suppose this is a problem related to HWCDC since I tested this with two devices:
  1. FireBeetle 2 ESP32-E:
     • using the most basic HardwareSerial class as Serial (but I need to install CH34xVCPDriver in order to see it in my Mac)
     • no problems at all
  2. LilyGO T-Display S3
     • using HWCDC class as Serial
     • reproducible

Questions

My main question is, why does this happen? Why do I have to introduce delay between serial.open() and serial.write()? The device may have not been ready in the first moments after reconnection, I understand that. But why does it happen indefinitely? The port must be ready at some point.

My secondary question is, why does this happen only when physical device reconnection during software runtime occurred? I mean, the device is in the same state (waiting for commands) some moments after reconnection as it would have been connected before the software started.

What is wrong?

One could argue something like "serial.open() needs some time to initialize before it can write, HWCDC is more complicated, needs more time", but why is then immediately ready in the first few iterations when the Python code is started and the device was connected beforehand?

There, it should also require some time to initialize, right? If answer would be "well, the device was connected before hand and was initialized on the OS level" or something like this, then why wasn't the conversation (5B <-> 256B) repaired after the device was reconnected and took some time to initialize on the OS level. You see my thought paradox?

Please, if you have answer to any of my questions, let me know.

Expected behavior

This is expected behavior, that occurs when trying the same thing with my first DFRobot device that uses HardwareSerial:

----------------------
Opened serial port
Wrote command 7
Read response 256
----------------------
Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
----------------------
Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
----------------------
Ex: [Errno 2] could not open port /dev/cu.wchusbserial110: [Errno 2] No such file or directory: '/dev/cu.wchusbserial110'
----------------------
Opened serial port        <first iteration after reconnection fails, this is fine>
Wrote command 7
Read response 256
Response too short 0
----------------------
Opened serial port        <second iteration after reconnection works>
Wrote command 7           <all other iterations work>
Read response 256
----------------------

All I want is that my device starts sending data again after physical reconnection. I understand some delay is needed to startup the device, but I don't understand why this delay is indefinite.

Solved by setting -DARDUINO_USB_MODE on 0 in platformio.ini:

board_build.extra_flags =
    -DARDUINO_LILYGO_T_DISPLAY_S3
    -DARDUINO_USB_MODE=0           <-- 1 is here by default
    -DARDUINO_USB_CDC_ON_BOOT=1
    -DARDUINO_RUNNING_CORE=1
    -DARDUINO_EVENT_RUNNING_CORE=1

Default values are from here