Getting 'FAILED' when connecting to PCF Dev with CF CLI

I run pcf dev 0.15.0 on CentOS x64 7.2 with Vagrant 1.8.1 and VirtualBox 5.0.

When I do a vagrant up it seems (to me) that everything is booting up nicely (although it takes some minutes to get from 0 to 48 running services)

PCF Dev has reserved 32 GB out of 46 GB total system memory.
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Checking if box 'pcfdev/pcfdev' is up to date...
==> default: Clearing any previously set forwarded ports...
==> default: Clearing any previously set network interfaces...
==> default: Preparing network interfaces based on configuration...
    default: Adapter 1: nat
    default: Adapter 2: hostonly
==> default: Forwarding ports...
    default: 22 (guest) => 2222 (host) (adapter 1)
==> default: Running 'pre-boot' VM customizations...
==> default: Booting VM...
==> default: Waiting for machine to boot. This may take a few minutes...
    default: SSH address: 127.0.0.1:2222
    default: SSH username: vagrant
    default: SSH auth method: private key
    default: Warning: Remote connection disconnect. Retrying...
    default: Warning: Remote connection disconnect. Retrying...
==> default: Machine booted and ready!
==> default: Checking for guest additions in VM...
==> default: Configuring and enabling network interfaces...
==> default: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> default: flag to force provisioning. Provisioners marked to run always will still run.
==> default: Running provisioner: shell...
    default: Running: inline script
==> default: stdin: is not a tty
==> default: Waiting for services to start...
==> default: 0 out of 48 running
==> default: 0 out of 48 running
==> default: 0 out of 48 running
==> default: 0 out of 48 running
==> default: 0 out of 48 running
==> default: 37 out of 48 running
==> default: 48 out of 48 running
==> default: PCF Dev is now running.
==> default: To begin using PCF Dev, please run:
==> default:    cf login -a api.local.pcfdev.io --skip-ssl-validation
==> default: Email: admin
==> default: Password: admin

I run this unaltered, the only change is that I set VM_MEMORY because I want to use more memory (you can see the result on the first line of the logfile)

When I try to login using cf login -a api.local.pcfdev.io --skip-ssl-validation -v this can't be resolved so I use the IP address that ifconfig tells me after I vagrant ssh into the box:

eth0      Link encap:Ethernet  HWaddr 08:00:27:6c:38:a8
          inet addr:10.0.2.15  Bcast:10.0.2.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe6c:38a8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:823 errors:0 dropped:0 overruns:0 frame:0
          TX packets:669 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:82923 (82.9 KB)  TX bytes:83850 (83.8 KB)

eth1      Link encap:Ethernet  HWaddr 08:00:27:02:34:18
          inet addr:192.168.11.11  Bcast:192.168.11.255  Mask:255.255.255.0
          inet6 addr: fe80::a00:27ff:fe02:3418/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:32 errors:0 dropped:0 overruns:0 frame:0
          TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2955 (2.9 KB)  TX bytes:4782 (4.7 KB)

This is the result when I use 192.168.11.11 when connecting with cf:

API endpoint: 192.168.11.11

REQUEST: [2016-06-10T10:24:37+02:00]
GET /v2/info HTTP/1.1
Host: 192.168.11.11
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.18.1+a1103f0 / linux



RESPONSE: [2016-06-10T10:24:37+02:00]
HTTP/1.1 404 Not Found
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Date: Fri, 10 Jun 2016 08:24:37 GMT
X-Cf-Routererror: unknown_route
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: ba181cbd-aaf3-43cf-4169-e2d7e42d19a5

404 Not Found: Requested route ('192.168.11.11') does not exist.


REQUEST: [2016-06-10T10:24:37+02:00]
GET /v2/info HTTP/1.1
Host: 192.168.11.11
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.18.1+a1103f0 / linux



RESPONSE: [2016-06-10T10:24:37+02:00]
HTTP/1.1 404 Not Found
Content-Length: 65
Content-Type: text/plain; charset=utf-8
Date: Fri, 10 Jun 2016 08:24:37 GMT
X-Cf-Routererror: unknown_route
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 69d68148-17ca-4434-4562-4a6fd90d7984

404 Not Found: Requested route ('192.168.11.11') does not exist.

FAILED
Server error, status code: 404, error code: 0, message:

It looks likelike others are having similar problems in Windows too: https://github.com/pivotal-cf/pcfdev/issues/68

Solution

What you're trying won't work, because the IP isn't directly the IP of the API server, it's the IP of the router which will look at the requested host and determine where to proxy the request to (e.g. the system tells it to proxy requests for api.local.pcfdev.io to the Cloud Controller API running inside your PCF Dev box.

api.local.pcfdev.io should resolve to 192.168.11.11. Any *.local.pcfdev.io address will, this is a globally configured DNS for the convenience of people using PCF Dev. You don't even have to have PCF Dev installed on your machine for this to work, e.g. on my machine:

$ nslookup foo.local.pcfdev.io
Server:     208.69.43.23
Address:    208.69.43.23#53

Non-authoritative answer:
Name:   foo.local.pcfdev.io
Address: 192.168.11.11

If there is some limitation preventing your machine from resolving addresses that are registered out in the world, you may need to teach your local machine to do the right DNS resolution. One option is configuring your machine to resolve that wildcard domain: https://serverfault.com/questions/118378/in-my-etc-hosts-file-on-linux-osx-how-do-i-do-a-wildcard-subdomain

Another alternative is to just define the DNS resolutions for a finite list of system components in your /etc/hosts file. I'm not 100% sure what the full list would be, but I think this will probably cover it:

blobstore.local.pcfdev.io
uaa.local.pcfdev.io
login.local.pcfdev.io
api.local.pcfdev.io
doppler.local.pcfdev.io
loggregator.local.pcfdev.io

In addition to system components, you will need to make sure you have DNS set up for every app you push. I believe by default, your apps will be reachable via some-app-name.local.pcfdev.io which doesn't currently work for you. Here, you can either set up addtional /etc/hosts records for each app that you push, or you can choose a different wildcard apps domain that your machine can resolve to 192.168.11.11 (e.g. try 192.168.11.11.xip.io, though I suspect your machine won't like xip.io either). You can look more into how to tell PCF Dev to create a new domain to be used for apps here: https://docs.cloudfoundry.org/adminguide/manage-domains-routes.html. So in this case, your app would be reachable at some-app-name.192.168.11.11.xip.io.

Still, *.local.pcfdev.io DNS is registered with public DNS servers, so the simplest thing would just be to figure out how to get that resolving on your machine.