pythonarduino-unoyoloyolov8

How to Script YOLO Detection From Python to Reactive LEDs on Arduino Broad


I have my own pre-trained YOLO model, and I want to have different colors of LEDs light up on Arduino Uno breadbroad when different labels are being detected from my webcam.

So, I was thinking of first assigning the command for different LEDs to light up. ("G" for green led, "R" for red led, "Y" for yellow led, "A" for turning both red and yellow led at the same time, and "0" to turn off all led. ) I started with my Arduino code as below:

char command;

void setup() {
    Serial.begin(9600);
    pinMode(2, OUTPUT);  // Green LED pin
    pinMode(3, OUTPUT);  // Red LED pin
    pinMode(4, OUTPUT);  // Yellow LED pin
}

void loop() {
    if (Serial.available() > 0) {
        command = Serial.read();
        if (command == 'G') {
            // Turn on green LED
            digitalWrite(2, HIGH);
            digitalWrite(3, LOW);  // Turn off red LED
            digitalWrite(4, LOW);  // Turn off yellow LED
        } else if (command == 'R') {
            // Turn on red LED
            digitalWrite(2, LOW);   // Turn off green LED
            digitalWrite(3, HIGH);
            digitalWrite(4, LOW);   // Turn off yellow LED
        } else if (command == 'Y') {
            // Turn on yellow LED
            digitalWrite(2, LOW);   // Turn off green LED
            digitalWrite(3, LOW);   // Turn off red LED
            digitalWrite(4, HIGH);
        } else if (command == 'A') {
            // Turn on both red and yellow LEDs
            digitalWrite(2, LOW);   // Turn off green LED
            digitalWrite(3, HIGH);
            digitalWrite(4, HIGH);
        } else if (command == '0') {
            // Turn off all LEDs
            digitalWrite(2, LOW);
            digitalWrite(3, LOW);
            digitalWrite(4, LOW);
        }
    }
}

In my YOLO model, there are three classes

#Classes
names:
  0: aaa
  1: bbb
  2: ccc

Therefore, for my python code I started with first turning on the webcam and loading then using serial library to connect to arduino.

from ultralytics import YOLO
import serial
import time
import sys

# Load my YOLO model using
model = YOLO("best.pt")

# Open the serial port for communication with Arduino
arduino = serial.Serial('COM3', 9600)  # Change 'COM3' to your Arduino port

after that, I want to assign the labels with command to Arduino, and I also try to do a little

# Map the class labels to corresponding Arduino commands
label_commands = {
    'aaa': 'G',
    'bbb': 'R',
    'ccc': 'Y',
    'aaa_bbb': 'A',
    'aaa_bbb_ccc': 'A',
    'none': '0'
}

while True:
    try:
        results = model.predict(source="0", show=True)  # assumes '0' is your source identifier

        # Extract the detected labels from results
        detected_labels = [item['label'] for item in results.xyxy[0].numpy()]

        # Determine the Arduino command based on detected labels
        if 'aaa' in detected_labels and 'bbb' in detected_labels and 'ccc' in detected_labels:
            command = label_commands['aaa_bbb_ccc']
        elif 'aaa' in detected_labels and 'bbb' in detected_labels:
            command = label_commands['aaa_bbb']
        elif 'aaa' in detected_labels:
            command = label_commands['aaa']
        elif 'bbb' in detected_labels and 'ccc' in detected_labels:
            command = label_commands['aaa_bbb_ccc']
        elif 'bbb' in detected_labels:
            command = label_commands['bbb']
        elif 'ccc' in detected_labels:
            command = label_commands['ccc']
        else:
            command = label_commands['none']

        # Print the command for debugging
        print(f"Sending command to Arduino: {command}")
        sys.stdout.flush()  # Flush the standard output

        # Send the command to Arduino
        arduino.write(command.encode())

    except Exception as e:
        print(f"Error: {e}")
        sys.stdout.flush()  # Flush the standard output

    # Wait for 5 seconds before sending the next command
    time.sleep(5)

BUT, this part of code doesn't work at all. I know my arduino is connected for sure, and I have also tested with from randoming sending the letter command to arduino from python, which worked perfectly.

Also when I run this code, the run window are showing only things like:

1/1: 0... Success ✅ (inf frames of shape 640x480 at 25.00 FPS)


WARNING ⚠️ inference results will accumulate in RAM unless `stream=True` is passed, causing potential out-of-memory
errors for large sources or long-running streams and videos. See https://docs.ultralytics.com/modes/predict/ for help.

Example:
    results = model(source=..., stream=True)  # generator of Results objects
    for r in results:
        boxes = r.boxes  # Boxes object for bbox outputs
        masks = r.masks  # Masks object for segment masks outputs
        probs = r.probs  # Class probabilities for classification outputs

0: 480x640 (no detections), 141.0ms
0: 480x640 1 aaa, 182.2ms
0: 480x640 1 aaa, 134.3ms
0: 480x640 1 aaa, 122.1ms
0: 480x640 (no detections), 121.4ms
0: 480x640 (no detections), 147.5ms
0: 480x640 (no detections), 131.5ms
0: 480x640 (no detections), 140.3ms
0: 480x640 1 aaa, 127.1ms
0: 480x640 1 aaa, 124.1ms
0: 480x640 1 aaa, 123.2ms
0: 480x640 1 aaa, 172.2ms
...

even with my debugging script, it doesn't tell me what command is being sent to the arduino broad

        # Print the command for debugging
        print(f"Sending command to Arduino: {command}")
        sys.stdout.flush()  # Flush the standard output

Solution

  • For the YOLO part, this can be helpful: first, as you predict a live stream, use stream=True to get a results generator. Next, to reach the class names of the detected objects, you first need to get their class ids and then refer to the result.names which is a list of model names, in your case it will be ['aaa', 'bbb', 'ccc']. Example for a COCO pre-trained YOLO model:

    model = YOLO(yolov8n.pt)
    results = model.predict(source="0", stream=True)
    # for each frame
    for result in results:
      class_ids = result.boxes.cls.tolist()
      detected_labels = [result.names[int(i)] for i in class_ids]
      #print(detected_labels)
      #['person', 'person', 'cat']
      #(if 2 person, 1 cat detected in a frame)
    

    More about how to use results generator and how to work with the prediction results.