How to test a Phoenix application with Bypass?

Background

I have a Phoenix application, freshly created. This app has an endpoint that makes a request to an external service, and I would like to exercise/test the entire stack vertically.

To achieve this I have opted to try bypass to mock responses from the external service.

What I tried

To achieve this I have come up with the following code:

defmodule MyApp.ApplicationTest do
  @moduledoc false

  use ExUnit.Case, async: false

  alias HTTPoison

  @external_service_port 5112

  setup do
    bypass = Bypass.open(port: @external_service_port)

    {:ok, bypass: bypass}
  end

  test "client can handle an error response", %{bypass: bypass} do
    Bypass.expect_once(bypass, "GET", "api/users/validate", fn conn ->
      Plug.Conn.resp(conn, 429, ~s<{"errors": [{"code": 88, "message": "Rate limit exceeded"}]}>)
    end)

    MyApp.Application.start(nil, nil) |> IO.inspect(label: "APPLICATION STARTUP")

    HTTPoison.get("http://localhost:4000/api/users/validate") |> IO.inspect(label: "REQUEST")
  end
end

Problem

Unfortunately, when I run the test this is the output I get:

❯ mix test test/my_app/application_test.exs 
APPLICATION STARTUP: {:error, {:already_started, #PID<0.404.0>}}
REQUEST: {:error, %HTTPoison.Error{reason: :econnrefused, id: nil}}

My understanding is that the application is already started, but somehow I cannot reach it and I don't know why.

Questions

1. Is this the correct way to exercise the entire stack using ByPass with Phoenix?
2. Is there a better way of testing this?
3. Why can't I access my application?

Solution

As mentioned by some of the folks here, bypass merely adds a plug that intercepts requests. In my case, bypass is intercepting requests for both the external 3rd party, and the application server itself.

This is not how the library is supposed to work. Instead, the solution to my problem is to use something completely different, a different tool.

For this purpose, and following the suggestions from the community, I have decided to use Hammox (https://github.com/msz/hammox) to create a behaviour between my application and the 3rd party. Tests are then done a mock that implements a bhaviour, following Hammox doctrine instead.

You could technically use bypass to test the integration between the adapter (the implementation of the Behaviour that Hammox uses) and the 3rd party, by having a bypass server return answers that mimick that of the 3rd party. This in really looks a lot more like the documentation Bypass provides and is, imho, how bypass was supposed to be used to begin with.