Hy,I work with faster_rcnn_resnet101_v1_1024x1024_coco17_tpu-8 pretrained model. I have problems when evaluating the model. The training went without any problems. I start the evaluation of the model with the command:
python model_main_tf2.py --pipeline_config_path=./training_outlook_action_ctx/training_1/pipeline.config --model_dir=./training_outlook_action_ctx/training_1 --checkpoint_dir=./training_outlook_action_ctx/training_1
After the first Loaded cuDNN version 8400, it starts throwing me the following error that repeats itself until it interrupts
WARNING:tensorflow:Ignoring ground truth with image id 1016176252 since it was previously added
W0810 10:17:12.131517 140545620840832 coco_evaluation.py:113] Ignoring ground truth with image id 1016176252 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1016176252 since it was previously added
W0810 10:17:12.131881 140545620840832 coco_evaluation.py:196] Ignoring detection with image id 1016176252 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1016176252 since it was previously added
W0810 10:17:12.652873 140545620840832 coco_evaluation.py:113] Ignoring ground truth with image id 1016176252 since it was previously added
WARNING:tensorflow:Ignoring detection with image id 1016176252 since it was previously added
W0810 10:17:12.653055 140545620840832 coco_evaluation.py:196] Ignoring detection with image id 1016176252 since it was previously added
WARNING:tensorflow:Ignoring ground truth with image id 1016176252 since it was previously added
here is my pipeline.config file
# Faster R-CNN with Resnet-50 (v1)
# Trained on COCO, initialized from Imagenet classification checkpoint
# This config is TPU compatible.
model {
faster_rcnn {
num_classes: 7
image_resizer {
fixed_shape_resizer {
width: 1024
height: 1024
}
}
feature_extractor {
type: 'faster_rcnn_resnet101_keras'
batch_norm_trainable: true
}
first_stage_anchor_generator {
grid_anchor_generator {
scales: [0.25, 0.5, 1.0, 2.0]
aspect_ratios: [0.5, 1.0, 2.0]
height_stride: 16
width_stride: 16
}
}
first_stage_box_predictor_conv_hyperparams {
op: CONV
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
truncated_normal_initializer {
stddev: 0.01
}
}
}
first_stage_nms_score_threshold: 0.0
first_stage_nms_iou_threshold: 0.7
first_stage_max_proposals: 300
first_stage_localization_loss_weight: 2.0
first_stage_objectness_loss_weight: 1.0
initial_crop_size: 14
maxpool_kernel_size: 2
maxpool_stride: 2
second_stage_box_predictor {
mask_rcnn_box_predictor {
use_dropout: false
dropout_keep_probability: 1.0
fc_hyperparams {
op: FC
regularizer {
l2_regularizer {
weight: 0.0
}
}
initializer {
variance_scaling_initializer {
factor: 1.0
uniform: true
mode: FAN_AVG
}
}
}
share_box_across_classes: true
}
}
second_stage_post_processing {
batch_non_max_suppression {
score_threshold: 0.0
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 300
}
score_converter: SOFTMAX
}
second_stage_localization_loss_weight: 2.0
second_stage_classification_loss_weight: 1.0
use_static_shapes: true
use_matmul_crop_and_resize: true
clip_anchors_to_image: true
use_static_balanced_label_sampler: true
use_matmul_gather_in_matcher: true
}
}
train_config: {
batch_size: 2
sync_replicas: true
startup_delay_steps: 0
replicas_to_aggregate: 8
num_steps: 200000
optimizer {
momentum_optimizer: {
learning_rate: {
cosine_decay_learning_rate {
learning_rate_base: .04
total_steps: 100000
warmup_learning_rate: .013333
warmup_steps: 2000
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}
fine_tune_checkpoint_version: V2
fine_tune_checkpoint: "/pretrained_models/faster_rcnn_resnet101_v1_1024x1024_coco17_tpu-8/checkpoint/ckpt-0"
fine_tune_checkpoint_type: "detection"
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
random_adjust_hue {
}
}
data_augmentation_options {
random_adjust_contrast {
}
}
data_augmentation_options {
random_adjust_saturation {
}
}
data_augmentation_options {
random_square_crop_by_scale {
scale_min: 0.6
scale_max: 1.3
}
}
max_number_of_boxes: 100
unpad_groundtruth_tensors: false
use_bfloat16: true # works only on TPUs
}
train_input_reader: {
label_map_path: "./training_outlook_action_ctx/data/label_map.pbtxt"
tf_record_input_reader {
input_path: "./training_outlook_action_ctx/data/train.records"
}
}
eval_config: {
metrics_set: "coco_detection_metrics"
use_moving_averages: false
batch_size: 2
}
eval_input_reader: {
label_map_path: "./training_outlook_action_ctx/data/label_map.pbtxt"
shuffle: false
tf_record_input_reader {
input_path: "./training_outlook_action_ctx/data/train.records"
}
}
OS: Debian GNU/Linux 11 (bullseye)
Python: 3.9.12
Tensorflow: 2.9.1
I tried to add num_examples and max_evals but failed. No matter how I adjust them, it still throws the same error
I must mention that the evaluation on the second dataset worked normally for me
Thanks in advance Edi
guys. I found a solution for how I used the script to create images and annotations. More precisely, I used a script that will crop my first-level annotations and create new XML files for cropped images. The filename and path were not good for me in the XML files (wrong path when programming the script).
After the change, the evaluation code error disappeared.
For all the questions I am available.