Unable to test code-first gRPC service using AddCodeFirstGrpcClient and WebApplicationFactory

I'm trying to take advantage of Protobuf-net's client factory method AddCodeFirstGrpcClient to create client in a testing context using WebApplicationFactory<TEntryPoint>.

The issue is with the 2nd test (TestViaCodeFirst), it fails with the following error:

 Message: 
Grpc.Core.RpcException : Status(StatusCode="Unavailable", Detail="Error connecting to subchannel.", DebugException="System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it.")
---- System.Net.Sockets.SocketException : No connection could be made because the target machine actively refused it.

  Stack Trace: 
ConnectionManager.PickAsync(PickContext context, Boolean waitForReady, CancellationToken cancellationToken)
BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
Reshape.UnaryTaskAsyncImpl[TRequest,TResponse](AsyncUnaryCall`1 call, MetadataContext metadata, CancellationToken cancellationToken) line 554
GreeterServiceTests.TestViaCodeFirst() line 50
--- End of stack trace from previous location ---
----- Inner Stack Trace -----
AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
IValueTaskSource.GetResult(Int16 token)
Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken)
SocketConnectivitySubchannelTransport.TryConnectAsync(ConnectContext context, Int32 attempt)

My code

Shared code:

using ProtoBuf.Grpc;
using System.Runtime.Serialization;
using System.ServiceModel;

namespace GrpcExperiments.Shared;

[DataContract]
public class HelloReply
{
    [DataMember(Order = 1)]
    public string? Message { get; set; }
}

[DataContract]
public class HelloRequest
{
    [DataMember(Order = 1)]
    public string? Name { get; set; }
}

[ServiceContract]
public interface IGreeterService
{
    [OperationContract]
    Task<HelloReply> SayHelloAsync(HelloRequest request);
}

Server code:

using ProtoBuf.Grpc;
using GrpcExperiments.Shared;

public class GreeterService : IGreeterService
{
    public Task<HelloReply> SayHelloAsync(HelloRequest request)
    {
        return Task.FromResult(
            new HelloReply { Message = $"Hello {request.Name}" });
    }
}

Cli code:

using Grpc.Net.Client;
using GrpcExperiments.Shared;
using Microsoft.Extensions.DependencyInjection;
using ProtoBuf.Grpc.Client;
using ProtoBuf.Grpc.ClientFactory;
using Gncf = Grpc.Net.ClientFactory;

public class Program
{
    public static async Task Main()
    {
        await CreateClientOld();
        await CreateClientNew();

        Console.WriteLine("Press any key to exit...");
        Console.ReadKey();
    }

    static async Task CreateClientNew()
    {
        var services = new ServiceCollection();
        services.AddCodeFirstGrpcClient<IGreeterService>(o =>
        {
            o.Address = new Uri("https://localhost:7158");
        });
        using var serviceProvider = services.BuildServiceProvider();
        var factory = serviceProvider.GetRequiredService<Gncf.GrpcClientFactory>();
        var client = factory.CreateClient<IGreeterService>(nameof(IGreeterService));

        var reply = await client.SayHelloAsync(
            new HelloRequest { Name = "GreeterClient" });

        Console.WriteLine($"Greeting (new): {reply.Message}");
    }

    static async Task CreateClientOld()
    {
        using var channel = GrpcChannel.ForAddress("https://localhost:7158");
        var client = channel.CreateGrpcService<IGreeterService>();

        var reply = await client.SayHelloAsync(
            new HelloRequest { Name = "GreeterClient" });

        Console.WriteLine($"Greeting (old): {reply.Message}");
    }
}

Test code:

using Grpc.Net.Client;
using GrpcExperiments.Shared;
using Microsoft.AspNetCore.Mvc.Testing;
using Microsoft.Extensions.DependencyInjection;
using ProtoBuf.Grpc.Client;
using ProtoBuf.Grpc.ClientFactory;
using Gncf = Grpc.Net.ClientFactory;

namespace GrpcExperiements.Test;

public class GreeterServiceTests
{
    [Fact]
    public async Task TestViaGrpcChannel()
    {
        using var factory = new WebApplicationFactory<Program>();

        using var channel = GrpcChannel.ForAddress(factory.Server.BaseAddress, new GrpcChannelOptions
        {            
            HttpClient = factory.CreateClient()
        });

        var client = channel.CreateGrpcService<IGreeterService>();

        var reply = await client.SayHelloAsync(
            new HelloRequest { Name = "GreeterClient" });

        Assert.Equal("Hello GreeterClient", reply.Message);
    }

    [Fact]
    public async Task TestViaCodeFirst()
    {
        using var webApp = new WebApplicationFactory<Program>();

        using var services = new ServiceCollection();
        services
            .AddCodeFirstGrpcClient<IGreeterService>(nameof(IGreeterService), clientFactoryOptions =>
            {
                clientFactoryOptions.Address = webApp.Server.BaseAddress;
                //clientFactoryOptions.ChannelOptionsActions.Add(options => options.HttpClient = webApp.CreateClient());
            });
            //.ConfigureChannel(channel => 
            //    channel.HttpClient = webApp.CreateClient());

        using var serviceProvider = services.BuildServiceProvider();
        var factory = serviceProvider.GetRequiredService<Gncf.GrpcClientFactory>();
        var client = factory.CreateClient<IGreeterService>(nameof(IGreeterService));

        var reply = await client.SayHelloAsync(
            new HelloRequest { Name = "GreeterClient" });

        Assert.Equal("Hello GreeterClient", reply.Message);
    }
}

It was the handler that needed to be provided, not the client.
This worked.

[Fact]
public async Task TestViaCodeFirst()
{
    using var webApp = new WebApplicationFactory<Program>();

    var services = new ServiceCollection();
    services
        .AddCodeFirstGrpcClient<IGreeterService>(clientFactoryOptions =>
        {
            clientFactoryOptions.Address = webApp.Server.BaseAddress;
            clientFactoryOptions.ChannelOptionsActions.Add(option => 
                option.HttpHandler = webApp.Server.CreateHandler());
        });

    using var serviceProvider = services.BuildServiceProvider();
    var factory = serviceProvider.GetRequiredService<Grpc.Net.ClientFactory.GrpcClientFactory>();
    var client = factory.CreateClient<IGreeterService>(nameof(IGreeterService));

    var reply = await client.SayHelloAsync(
        new HelloRequest { Name = "GreeterClient" });

    Assert.Equal("Hello GreeterClient", reply.Message);
}