stream video and send response using python sockets

I modified the picamera script from https://picamera.readthedocs.io/en/latest/recipes2.html#rapid-capture-and-streaming to be able to stream video from my raspberry to my computer and at the same time send commands from the computer to the raspberry.

My client code looks like this:

while True:
        stream.seek(0)
        stream.truncate()

        camera.capture(stream, 'jpeg', use_video_port=True)

        connection.write(struct.pack('<L', stream.tell()))
        connection.flush()

        stream.seek(0)
        connection.write(stream.read())

        returnmessage = struct.unpack('<L', connection.read(struct.calcsize('<L')))[0]
        print(returnmessage)

        if returnmessage:
            if returnmessage == 1: 
                #do something
            else: 
                #do something else

and my server code:

while True:

    image_len = struct.unpack('<L', connection.read(struct.calcsize('<L')))[0]

    if not image_len:
        break

    image_stream = io.BytesIO()
    image_stream.write(connection.read(image_len))

    image_stream.seek(0)

    data = np.frombuffer(image_stream.getvalue(), dtype=np.uint8)
    image = cv2.imdecode(data, cv2.IMREAD_COLOR)

    cv2.imshow('stream',image)
    key = cv2.waitKey(1)

    if key != -1:
        if key == ord("g"):
            print("pressed g")
            connection.write(struct.pack('<L', 1))
            connection.flush()
        elif key == ord("h"):
            print("pressed a")
            connection.write(struct.pack('<L', 2))
            connection.flush()
    else:
        connection.write(struct.pack('<L', 0))
        connection.flush()

This works but does not feel right and can be really laggy from time to time. Depending on the streaming fps, I do not have to send a command after each frame so the waiting for the response is not always necessary but drops the streaming fps.

How can I approach this problem? Should I open another thread and another socket on each side for the responses?

For my purpose I found that adding the following to the client script seemed to do the job

reader, _, _ = select.select([connection], [], [], 0)

if reader: 
    returnmessage = struct.unpack('<L', connection.read(struct.calcsize('<L')))[0]

Thanks for your answer anyway.