[SOLVED] Chrome node: Unable to execute request: java.net.ConnectException: Connection refused after navigating to second URL

Currently running Selenium Grid with a Firefox & Chrome node on a Ubuntu EC2. Tests are ran with TestNG, tests ran through the Firefox node work perfectly however with the Chrome node I am always getting connection refused after navigating to a second URL. What I am finding confusing is that when running the grid locally through my Mac everything works perfectly.

Here is how it goes with Chrome tests:

Get Login page URL
Do Login
Attempt to Get new page URL
Connection refused

Has this issue been seen before? I have tried quite a few solutions for similar reported issues but to no avail. I am quite new to Selenium Grid so I am finding it hard to figure out what is going on.

My compose file:

version: "3"
services:
chrome:
image: selenium/node-chrome:4.0.0-rc-2-prerelease-20210908
shm_size: 2gb
volumes:

/dev/shm:/dev/shm
environment:
SE_EVENT_BUS_HOST=selenium-hub
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_NODE_SESSION_TIMEOUT=10
SE_SESSION_REQUEST_TIMEOUT=60
JAVA_OPTS=-Dwebdriver.chrome.whitelistedIps=
START_XVFB=false
ports:
"6900:5900"
firefox:
image: selenium/node-firefox:4.0.0-rc-2-prerelease-20210908
shm_size: 2gb
volumes:

/dev/shm:/dev/shm
environment:
SE_EVENT_BUS_HOST=selenium-hub
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_NODE_SESSION_TIMEOUT=10
SE_SESSION_REQUEST_TIMEOUT=60
START_XVFB=false
ports:
"6901:5900"
edge:
image: selenium/node-edge:4.0.0-rc-2-prerelease-20210908
shm_size: 2gb
volumes:

/dev/shm:/dev/shm
environment:
SE_EVENT_BUS_HOST=selenium-hub
SE_EVENT_BUS_PUBLISH_PORT=4442
SE_EVENT_BUS_SUBSCRIBE_PORT=4443
SE_NODE_SESSION_TIMEOUT=10
SE_SESSION_REQUEST_TIMEOUT=60
START_XVFB=false
ports:
"6902:5900"
selenium-hub:
image: selenium/hub:4.0.0-rc-2-prerelease-20210908
container_name: selenium-hub
ports:

"4442:4442"
"4443:4443"
"4444:4444"
depends_on:
chrome
edge
firefox

My ChromeDriver setup:

WebDriverManager.chromedriver().setup();

// chrome browser prefrences
HashMap<String, Object> prefs = new HashMap<>();
prefs.put("profile.default_content_settings.popups", 0);
prefs.put("download.prompt_for_download", false);
prefs.put("safebrowsing.enabled", false);
prefs.put("download.extensions_to_open", "application/xml");
prefs.put("download.extensions_to_open", "application/csv");
prefs.put("download.extensions_to_open", "application/pdf");
prefs.put("download.extensions_to_open", "application/zip");

// set chrome options
ChromeOptions chromeOptions = new ChromeOptions();
chromeOptions.setExperimentalOption("prefs", prefs);
chromeOptions.addArguments("--start-fullscreen");
chromeOptions.addArguments("--safebrowsing-disable-download-protection");
chromeOptions.addArguments("--no-sandbox");
chromeOptions.addArguments("--disable-gpu");
chromeOptions.addArguments ("--disable-setuid-sandbox");
chromeOptions.addArguments ("--no-first-run");
chromeOptions.addArguments ("--safebrowsing-disable-auto-update ");
chromeOptions.addArguments ("--disable-background-networking");
chromeOptions.addArguments("--headless", "--enable-logging", "--window-size=1920x1080");

// start remote session
driver = new RemoteWebDriver(new URL("http://localhost:4444/"),chromeOptions);

selenium/node-chrome logs:

Starting ChromeDriver 93.0.4577.15 (660fc11082ba57405eca2e8c49c3e1af756fbfae-refs/branch-heads/4577@{#203}) on port 22087 All remote connections are allowed. Use an allowlist instead! Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe. ChromeDriver was started successfully. 16:48:54.257 INFO [ProtocolHandshake.createSession] - Detected dialect: W3C Starting ChromeDriver 93.0.4577.15 (660fc11082ba57405eca2e8c49c3e1af756fbfae-refs/branch-heads/4577@{#203}) on port 6601 All remote connections are allowed. Use an allowlist instead! Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe. ChromeDriver was started successfully. 16:49:13.576 INFO [ProtocolHandshake.createSession] - Detected dialect: W3C 16:49:32.070 WARN [SpanWrappedHttpHandler.execute] - Unable to execute request: java.net.ConnectException: Connection refused: localhost/127.0.0.1:6601 java.io.UncheckedIOException: java.net.ConnectException: Connection refused: localhost/127.0.0.1:6601 at org.openqa.selenium.remote.http.netty.NettyHttpHandler.makeCall(NettyHttpHandler.java:80) at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42) at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56) at org.openqa.selenium.remote.http.netty.NettyHttpHandler.execute(NettyHttpHandler.java:51) at org.openqa.selenium.remote.http.AddSeleniumUserAgent.lambda$apply$0(AddSeleniumUserAgent.java:42) at org.openqa.selenium.remote.http.Filter.lambda$andFinally$1(Filter.java:56) at org.openqa.selenium.remote.http.netty.NettyClient.execute(NettyClient.java:119) at org.openqa.selenium.remote.tracing.TracedHttpClient.execute(TracedHttpClient.java:55) at org.openqa.selenium.grid.web.ReverseProxyHandler.execute(ReverseProxyHandler.java:100) at org.openqa.selenium.grid.node.ProtocolConvertingSession.execute(ProtocolConvertingSession.java:75) at org.openqa.selenium.grid.node.local.SessionSlot.execute(SessionSlot.java:123) at org.openqa.selenium.grid.node.local.LocalNode.executeWebDriverCommand(LocalNode.java:388) at org.openqa.selenium.grid.node.ForwardWebDriverCommand.execute(ForwardWebDriverCommand.java:35) at org.openqa.selenium.remote.http.Route$PredicatedRoute.handle(Route.java:373) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.grid.node.Node.execute(Node.java:240) at org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336) at org.openqa.selenium.remote.http.Route.execute(Route.java:68) at org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35) at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44) at org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64) at org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: java.net.ConnectException: Connection refused: localhost/127.0.0.1:6601 at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179) at org.asynchttpclient.netty.channel.NettyChannelConnector$1.onFailure(NettyChannelConnector.java:108) at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:28) at org.asynchttpclient.netty.SimpleChannelFutureListener.operationComplete(SimpleChannelFutureListener.java:20) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:707) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)