[SOLVED] Can not create shared-domain in Cloud Foundry

I can not create shared-domain in Cloud Foundry, any pushed apps get's health check connection refused.

I had working Cloud Foundry environment based on OpenStack IaaS. Everything worked as expected. I took my deployment files and after some time deployed it successfully in IaaS Vmware VSphere 7. The problem is, that every app that I push has problems with health check:

   2020-10-29T16:55:01.43+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a successfully destroyed container for instance 44e9c2a6-b54d-4fc4-4118-6d6d
   2020-10-29T16:55:36.55+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a creating container for instance 17f161a2-9788-426d-414d-6c33
   2020-10-29T16:55:37.18+0000 [CELL/0] OUT Cell 938b869c-5a68-40cc-9486-c5bc0d53a73a successfully created container for instance 17f161a2-9788-426d-414d-6c33
   2020-10-29T16:55:37.47+0000 [CELL/0] OUT Downloading droplet...
   2020-10-29T16:55:37.75+0000 [CELL/0] OUT Downloaded droplet
   2020-10-29T16:55:37.75+0000 [CELL/0] OUT Starting health monitoring of container
   2020-10-29T16:56:38.45+0000 [HEALTH/0] ERR Failed to make TCP connection to port 8080: connection refused
   2020-10-29T16:56:38.45+0000 [CELL/0] ERR Timed out after 1m0s: health check never passed.
   2020-10-29T16:56:38.46+0000 [CELL/SSHD/0] OUT Exit status 0
   2020-10-29T16:56:38.48+0000 [APP/PROC/WEB/0] OUT Exit status 143

I am also not able to create any shared domains:

bash-5.0# cf create-shared-domain tcp.cf.test-env.net --router-group default-tcp -v
REQUEST: [2020-10-29T17:03:33Z]
GET /v2/info HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)

RESPONSE: [2020-10-29T17:03:33Z]
HTTP/1.1 200 OK
Content-Length: 561
Content-Type: application/json;charset=utf-8
Date: Thu, 29 Oct 2020 17:03:33 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 4badb79b-2faf-4623-6c3c-ce5fa3223cd5::dc43d2c9-c902-4429-9d65-d9a0060983c5
{
  "api_version": "2.144.0",
  "app_ssh_endpoint": "ssh.cf.test-env.net:2222",
  "app_ssh_host_key_fingerprint": "ae:a3:ed:ad:37:d3:8a:7b:ed:b4:e5:d2:25:e5:8c:d0",
  "app_ssh_oauth_client": "ssh-proxy",
  "authorization_endpoint": "https://login.cf.test-env.net",
  "build": "",
  "description": "",
  "doppler_logging_endpoint": "wss://doppler.cf.test-env.net:443",
  "min_cli_version": null,
  "min_recommended_cli_version": null,
  "name": "",
  "osbapi_version": "2.15",
  "routing_endpoint": "https://api.cf.test-env.net/routing",
  "support": "",
  "token_endpoint": "https://uaa.cf.test-env.net",
  "version": 0
}


REQUEST: [2020-10-29T17:03:33Z]
GET /login HTTP/1.1
Host: login.cf.test-env.net
Accept: application/json
Connection: close
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)

RESPONSE: [2020-10-29T17:03:34Z]
HTTP/1.1 200 OK
Cache-Control: no-store
Content-Language: en-US
Content-Length: 384
Content-Type: application/json;charset=UTF-8
Date: Thu, 29 Oct 2020 17:03:34 GMT
Set-Cookie: X-Uaa-Csrf=NJlSPAjspn7m8oWuQdKsVD; Max-Age=86400; Expires=Fri, 30-Oct-2020 17:03:34 GMT; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: 577d4d31-ec30-477e-6f44-c0dd9306270d
X-Xss-Protection: 1; mode=block
{
  "app": {
    "version": "74.12.0"
  },
  "commit_id": "7311e68",
  "entityID": "login.cf.test-env.net",
  "idpDefinitions": {},
  "links": {
    "login": "https://login.cf.test-env.net",
    "passwd": "/forgot_password",
    "register": "/create_account",
    "uaa": "https://uaa.cf.test-env.net"
  },
  "prompts": {
    "password": "[PRIVATE DATA HIDDEN]",
    "username": [
      "text",
      "Email"
    ]
  },
  "timestamp": "2019-12-02T22:53:03+0000",
  "zone_name": "uaa"
}


Creating shared domain tcp.cf.test-env.net as admin...
REQUEST: [2020-10-29T17:03:34Z]
GET /routing/v1/router_groups?name=default-tcp HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Connection: close
Content-Type: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
[application/json Content Hidden]

RESPONSE: [2020-10-29T17:03:34Z]
HTTP/1.1 200 OK
Content-Length: 114
Content-Type: application/json
Date: Thu, 29 Oct 2020 17:03:34 GMT
X-Vcap-Request-Id: 9459b068-0987-4f5e-7dee-1efdb5ca6fb8
[
  {
    "guid": "343ba1e8-88a7-4003-6db6-4feabedd072b",
    "name": "default-tcp",
    "reservable_ports": "1024-2048",
    "type": "tcp"
  }
]


REQUEST: [2020-10-29T17:03:34Z]
POST /v2/shared_domains HTTP/1.1
Host: api.cf.test-env.net
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.47.2+d526c2cb3.2019-11-05 (go1.12.12; amd64 linux)
{
  "internal": false,
  "name": "tcp.cf.test-env.net",
  "router_group_guid": "343ba1e8-88a7-4003-6db6-4feabedd072b"
}


RESPONSE: [2020-10-29T17:04:04Z]
HTTP/1.0 504 Gateway Time-out
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>


Error unmarshalling the following into a cloud controller error: <html><body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>

FAILED

I suspect network configuration issue, that blocks some internal CF parts from connection. There is no any firewall or any rules found in VMware. I can also ping and make ssh connection between bosh created VM's.

Any ideas, what else can I do?

Can not create shared-domain in Cloud Foundry - 504 Gateway Time-out