[SOLVED] Using tensorflow model maker how to save checkpoints during training

Using tensorflow model maker how to save checkpoints during training

I am using the TensorflowLite model maker and I would like to know how I can during training regularily save checkpoints so I can check the loss graphs with tensorboard. How is this done?

This function I call when I want to train but there seems to be no argument for additional settings like writing checkpoints or specifying a directory for it.

model = object_detector.create(train_data, model_spec=spec, batch_size=8, train_whole_model=True, validation_data=validation_data)

Solution

Ok I found out how it works! The param grad_checkpoint=true has to set in the spec object that is given to the object_detector.create() function! In the spec the model checkpoint dir can also be specified! This is how I got it working:

spec = object_detector.EfficientDetLite0Spec(
    model_name='efficientdet-lite0',
    model_dir='/home/alex/checkpoints_1/',
    hparams='grad_checkpoint=true,strategy=gpus',
    epochs=30, batch_size=8,
    steps_per_execution=1, moving_average_decay=0,
    var_freeze_expr='(efficientnet|fpn_cells|resample_p6)',
    tflite_max_detections=25
)

model = object_detector.create(train_data, model_spec=spec, batch_size=8, 
    train_whole_model=True, validation_data=validation_data)