pythonyolotensorrtyolov8

Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"

I am trying to export .engine from onnx for the pretrained yolov8m model but get into trtexec issue. Note that I am targeting for a model supporting dynamic batch-size.

I got the onnx by following the official instructions from ultralytics.

from ultralytics import YOLO

# Load a model
model = YOLO('yolov8m.pt')  # load an official model
model = YOLO('path/to/best.pt')  # load a custom trained

# Export the model
model.export(format='onnx',dynamic=True) # Note the dynamic arg

I get the corresponding onnx. Now when I try to run trtexec

trtexec --onnx=yolov8m.onnx --workspace=8144 --fp16 --minShapes=input:1x3x640x640 --optShapes=input:2x3x640x640 --maxShapes=input:10x3x640x640 --saveEngine=my.engine

I get

[08/10/2023-23:53:10] [I] TensorRT version: 8.2.5
[08/10/2023-23:53:11] [I] [TRT] [MemUsageChange] Init CUDA: CPU +336, GPU +0, now: CPU 348, GPU 4361 (MiB)
[08/10/2023-23:53:11] [I] [TRT] [MemUsageSnapshot] Begin constructing builder kernel library: CPU 348 MiB, GPU 4361 MiB
[08/10/2023-23:53:12] [I] [TRT] [MemUsageSnapshot] End constructing builder kernel library: CPU 483 MiB, GPU 4393 MiB
[08/10/2023-23:53:12] [I] Start parsing network model
[08/10/2023-23:53:12] [I] [TRT] ----------------------------------------------------------------
[08/10/2023-23:53:12] [I] [TRT] Input filename:   yolov8m.onnx
[08/10/2023-23:53:12] [I] [TRT] ONNX IR version:  0.0.8
[08/10/2023-23:53:12] [I] [TRT] Opset version:    17
[08/10/2023-23:53:12] [I] [TRT] Producer name:    pytorch
[08/10/2023-23:53:12] [I] [TRT] Producer version: 2.0.1
[08/10/2023-23:53:12] [I] [TRT] Domain:           
[08/10/2023-23:53:12] [I] [TRT] Model version:    0
[08/10/2023-23:53:12] [I] [TRT] Doc string:       
[08/10/2023-23:53:12] [I] [TRT] ----------------------------------------------------------------
[08/10/2023-23:53:12] [W] [TRT] onnx2trt_utils.cpp:366: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[08/10/2023-23:53:12] [E] [TRT] ModelImporter.cpp:773: While parsing node number 305 [Range -> "/model.22/Range_output_0"]:
[08/10/2023-23:53:12] [E] [TRT] ModelImporter.cpp:774: --- Begin node ---
[08/10/2023-23:53:12] [E] [TRT] ModelImporter.cpp:775: input: "/model.22/Constant_8_output_0"
input: "/model.22/Cast_output_0"
input: "/model.22/Constant_9_output_0"
output: "/model.22/Range_output_0"
name: "/model.22/Range"
op_type: "Range"

 

[08/10/2023-23:53:12] [E] [TRT] ModelImporter.cpp:776: --- End node ---
[08/10/2023-23:53:12] [E] [TRT] ModelImporter.cpp:779: ERROR: builtin_op_importers.cpp:3353 In function importRange:
[8] Assertion failed: inputs.at(0).isInt32() && "For range operator with dynamic inputs, this version of TensorRT only supports INT32!"
[08/10/2023-23:53:12] [E] Failed to parse onnx file
[08/10/2023-23:53:12] [I] Finish parsing network model
[08/10/2023-23:53:12] [E] Parsing model failed
[08/10/2023-23:53:12] [E] Failed to create engine from model.

I am aware that some people suggest upgrading to latest TRT version but I am looking for an alternate solution.

Solution

  • One alternative is to get a static model exported with the max batch-size required by my application (32 here).

    $ python3 export_yoloV8.py -w yolov8m.pt --batch 32 --simplify # note not using dynamic flag

    $ trtexec --onnx=dyolov8m-simple.onnx --workspace=8144 --fp16 --minShapes=input:1x3x640x640 --optShapes=input:2x3x640x640 --maxShapes=input:32x3x640x640 --saveEngine=yolov8m.engine