pythonyolov5clearml

ClearML, how to query the best performing model for a specific project and metric


I want to download the best performing model for a certain ClearlML project. I have the following content in my ClearML experiment platform:

enter image description here

According to: https://clear.ml/docs/latest/docs/clearml_sdk/model_sdk#querying-models I can get a list of models for a specific project:

model_list = Model.query_models(
    # Only models from `examples` project
    project_name='YOLOv5', 
    # Only models with input name
    model_name=None,
    # Only models with `demo` tag but without `TF` tag
    tags=['demo', '-TF'],
    # If `True`, only published models
    only_published=False,
    # If `True`, include archived models
    include_archived=True,
    # Maximum number of models returned
    max_results=5
)

print(model_list)

Which prints:

[<clearml.model.Model object at 0x7fefbaf22130>, <clearml.model.Model object at 0x7fefbaf22340>]

So I can run:

model_list[0].get_local_copy()

and get this specific model. But how do I download the best performing one for this project on a certain metric (in this case mAP_0.5:0.95 MAX)?


Solution

  • I ended up doing the following:

    try:
        import clearml
        from clearml import Dataset, Task, Model, OutputModel
        assert hasattr(clearml, '__version__')  # verify package import not local dir
    except (ImportError, AssertionError):
        clearml = None
    
    tasks = Task.get_tasks(project_name='YOLOv5', task_name='exp', task_filter={'status': ['completed']})
    
    results = {}
    best_task = None
    for task in tasks:
        results[task.id] = task.get_last_scalar_metrics()['metrics']['mAP_0.5:0.95']['max']
    
    best_model_task_id = max(results, key=results.get)
    model_list = Task.get_task(best_model_task_id).get_models()
    dest = model_list['output'][0].get_local_copy()
    print('Saved model at:', dest)