azure-machine-learning-serviceazureml-python-sdkazuremlsdk

How to programmatically get experiments with metrics above threshold azureml sdk v2


I have a number of experiments completed in AzureML, each one consisting of one pipeline with multiple jobs, and some of the jobs (2 in my case, say job X and job Y) having associated logged metrics. I would like to retrieve the name of the experiments which have one of the jobs (say job Y) associated metrics (say F1) bigger than some threshold.


Solution

  • First code snippet, slow if we have many experiments and runs (see below second code snippet):

    from azure.ai.ml import MLClient
    from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
    
    # Connect to AML
    client = MLClient(
            credential= InteractiveBrowserCredential(),
            subscription_id="my-subscription-id",
            resource_group_name="my-resource-group",
            workspace_name="my-workspace"
            )
    
    
    # set tracking uri if run locally
    mlflow_tracking_uri = client.workspaces.get(client.workspace_name).mlflow_tracking_uri
    mlflow.set_tracking_uri(mlflow_tracking_uri)
    
    
    # select all runs 
    # (slow, see alternative in second code snippet below)
    all_runs = mlflow.search_runs(search_all_experiments=True)
    
    # select runs whose my-metric-name is bigger than my_threshold
    metric_name = "my-metric-name"
    my_threshold = 0.9
    selected_runs = all_runs.loc[all_runs[f"metrics.{metric_name}"] > my_threshold]
    
    # get corresponding experiment IDs:
    selected_experiment_ids = selected_runs["experiment_id"].unique()
    
    # retrieve all experiments 
    # (slow, see alternative in second code snippet below)
    exps = mlflow.search_experiments()
    
    
    # show experiments whose experiment-id matches the selected runs
    for exp in exps:
        if exp.experiment_id in selected_experiment_ids:
            print(f"Experiment name: {exp.name}\nExperiment ID: {exp.experiment_id}")
    

    If we can narrow down our list of candidate experiments using either a list of experiment names we want to focus on, the user name who created the experiments, or both, we can get a faster response. This would be the resulting code after setting the tracking uri:

    # restrict experiment name to be one of the following:
    experiment_names = ["my-first-experiment", "my-second-experiment", "my-third-experiment"]
    
    # restrict user id to be the following:
    user_id = "my-user-id"
    
    # select runs 
    all_runs = mlflow.search_runs(
        experiment_names=experiment_names,
        filter_string=f"tags.mlflow.user='{user_id}'",
    )
    
    # select runs whose my-metric-name is bigger than my_threshold
    metric_name = "my-metric-name"
    my_threshold = 0.9
    selected_runs = all_runs.loc[all_runs[f"metrics.{metric_name}"] > my_threshold]
    
    # get corresponding experiment IDs:
    selected_experiment_ids = selected_runs["experiment_id"].unique()
    
    # retrieve all experiments whose name is in the experiment_names list
    filter_string = " or ".join([f"(name = {x})" for x in experiment_names])
    exps = mlflow.search_experiments(filter_string=filter_string)
    
    # show experiments whose experiment-id matches the selected runs
    for exp in exps:
        if exp.experiment_id in selected_experiment_ids:
            print(f"Experiment name: {exp.name}\nExperiment ID: {exp.experiment_id}")