I have a number of experiments completed in AzureML, each one consisting of one pipeline with multiple jobs, and some of the jobs (2 in my case, say job X and job Y) having associated logged metrics. I would like to retrieve the name of the experiments which have one of the jobs (say job Y) associated metrics (say F1) bigger than some threshold.
First code snippet, slow if we have many experiments and runs (see below second code snippet):
from azure.ai.ml import MLClient
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
# Connect to AML
client = MLClient(
credential= InteractiveBrowserCredential(),
subscription_id="my-subscription-id",
resource_group_name="my-resource-group",
workspace_name="my-workspace"
)
# set tracking uri if run locally
mlflow_tracking_uri = client.workspaces.get(client.workspace_name).mlflow_tracking_uri
mlflow.set_tracking_uri(mlflow_tracking_uri)
# select all runs
# (slow, see alternative in second code snippet below)
all_runs = mlflow.search_runs(search_all_experiments=True)
# select runs whose my-metric-name is bigger than my_threshold
metric_name = "my-metric-name"
my_threshold = 0.9
selected_runs = all_runs.loc[all_runs[f"metrics.{metric_name}"] > my_threshold]
# get corresponding experiment IDs:
selected_experiment_ids = selected_runs["experiment_id"].unique()
# retrieve all experiments
# (slow, see alternative in second code snippet below)
exps = mlflow.search_experiments()
# show experiments whose experiment-id matches the selected runs
for exp in exps:
if exp.experiment_id in selected_experiment_ids:
print(f"Experiment name: {exp.name}\nExperiment ID: {exp.experiment_id}")
If we can narrow down our list of candidate experiments using either a list of experiment names we want to focus on, the user name who created the experiments, or both, we can get a faster response. This would be the resulting code after setting the tracking uri:
# restrict experiment name to be one of the following:
experiment_names = ["my-first-experiment", "my-second-experiment", "my-third-experiment"]
# restrict user id to be the following:
user_id = "my-user-id"
# select runs
all_runs = mlflow.search_runs(
experiment_names=experiment_names,
filter_string=f"tags.mlflow.user='{user_id}'",
)
# select runs whose my-metric-name is bigger than my_threshold
metric_name = "my-metric-name"
my_threshold = 0.9
selected_runs = all_runs.loc[all_runs[f"metrics.{metric_name}"] > my_threshold]
# get corresponding experiment IDs:
selected_experiment_ids = selected_runs["experiment_id"].unique()
# retrieve all experiments whose name is in the experiment_names list
filter_string = " or ".join([f"(name = {x})" for x in experiment_names])
exps = mlflow.search_experiments(filter_string=filter_string)
# show experiments whose experiment-id matches the selected runs
for exp in exps:
if exp.experiment_id in selected_experiment_ids:
print(f"Experiment name: {exp.name}\nExperiment ID: {exp.experiment_id}")