devopsclearmltrains

ClearML get max value from logged values


I use ClearML to track my tensorboard logs (from PyTorch Lightning) during training. At a point later I start another script which connects to existing task and do some testing.

But unfortenautly I do not have all information in the second script, so I want to query them from the logged values from ClearML server.

How would I do this?

I thought about something like this, but havn't found anything in documentation:

task = Task.init(project_name="Project", task_name="name", reuse_last_task_id="Task_id, continue_last_task=True)
x_value, y_value = task.get_value(key="val/acc", mode="max")
x_value2, y_value2 = task.get_value(key="epoch", mode="x", x=x_value)

Solution

  • Disclaimer I'm part of the ClearML (formerly Trains) team.

    To get an existing Task object for a running (or completed/failed) experiment, assuming we know Task ID:

    another_task = Task.get_task(task_id='aabbcc')
    

    If we only know the Task project/name

    another_task = Task.get_task(project_name='the project', task_name='the name')
    

    Notice that if you have multiple task under the same name it will return the most updated one. Once we have the Task object, we can do:

    latest_scalar_values_dict = another_task.get_last_scalar_metrics()
    

    Which would return all the scalars min/maxm/last values, for example:

    latest_scalar_values_dict = {
                'title': {
                    'series': {
                        'last': 0.5,
                        'min': 0.1,
                        'max': 0.9
                        }
                    }
                }
    

    documentation here

    If you need to get the entire graphs you can use task.get_reported_scalars() see docs