I use ClearML to track my tensorboard logs (from PyTorch Lightning) during training. At a point later I start another script which connects to existing task and do some testing.
But unfortenautly I do not have all information in the second script, so I want to query them from the logged values from ClearML server.
How would I do this?
I thought about something like this, but havn't found anything in documentation:
task = Task.init(project_name="Project", task_name="name", reuse_last_task_id="Task_id, continue_last_task=True)
x_value, y_value = task.get_value(key="val/acc", mode="max")
x_value2, y_value2 = task.get_value(key="epoch", mode="x", x=x_value)
x_value
would be my epoch or global stepy_value
the maximum value of plot "val/acc"x_value2
would be my epoch or global stepy_value2
the value of plot "epoch" at x_value
Disclaimer I'm part of the ClearML (formerly Trains) team.
To get an existing Task
object for a running (or completed/failed) experiment, assuming we know Task ID:
another_task = Task.get_task(task_id='aabbcc')
If we only know the Task project/name
another_task = Task.get_task(project_name='the project', task_name='the name')
Notice that if you have multiple task under the same name it will return the most updated one.
Once we have the Task
object, we can do:
latest_scalar_values_dict = another_task.get_last_scalar_metrics()
Which would return all the scalars min/maxm/last values, for example:
latest_scalar_values_dict = {
'title': {
'series': {
'last': 0.5,
'min': 0.1,
'max': 0.9
}
}
}
If you need to get the entire graphs you can use task.get_reported_scalars()
see docs