pythonmachine-learningamazon-s3pytorchamazon-sagemaker

Loading and training data on Sagemaker with a Self-supervised pre-training model


I'm testing a Self-supervise pre-training model, independently of the model I'm getting the same Error. My environment is Jupiter Labs for AWS Sagemaker, the problem comes when, using args and trying to load the data (either from an S3 bucket or a local file on the instance) I keep getting this error:

usage: VICReg training script [-h] [--data_path DATA_PATH] [--exp-dir EXP_DIR]
                              [--log-freq-time LOG_FREQ_TIME] [--arch ARCH]
                              [--mlp MLP] [--epochs EPOCHS]
                              [--batch-size BATCH_SIZE] [--base-lr BASE_LR]
                              [--wd WD] [--sim-coeff SIM_COEFF]
                              [--std-coeff STD_COEFF] [--cov-coeff COV_COEFF]
                              [--num-workers NUM_WORKERS] [--device DEVICE]
                              [--world-size WORLD_SIZE]
                              [--local_rank LOCAL_RANK] [--dist-url DIST_URL]
                              [--rank RANK]
VICReg training script: error: unrecognized arguments: -f /home/sagemaker-user/.local/share/jupyter/runtime/kernel-299646e9-32c7-47f8-869a-72e80aa9271b.json

On more information about how I'm setting my code, this is where the Error most probably is:

def get_arguments():
    parser = argparse.ArgumentParser(description="Pretrain a resnet model with VICReg", add_help=False)

    # Data
    parser.add_argument("--data_path",
                        default='AnnualCrop/', type=Path,
                        help='dataset path')

    # Checkpoints
    parser.add_argument("--exp-dir", type=Path, default="./exp",
                        help='Path to the experiment folder, where all logs/checkpoints will be stored')
    parser.add_argument("--log-freq-time", type=int, default=60,
                        help='Print logs to the stats.txt file every [log-freq-time] seconds')

NOTE: I've tried with other models and codes as well but the same args.pars structure and keep getting the very same error.


Solution

  • I found the solution on a website indicating to add an empty space on the following lines of my code:

    if __name__ == '__main__':
        args = get_args_parser()
        args = args.parse_args("") # here the empty space is added as ""
        if args.output_dir:
            Path(args.output_dir).mkdir(parents=True, exist_ok=True)
        main(args)
    

    That actually solved my problem. You can check the following link for detailed information: https://medium.com/@data.scientist/ipython-trick-how-to-use-argparse-in-ipython-notebooks-a07423ab31fc