openstackopenstack-nova

NoValidHost: No valid host was found. There are not enough hosts available


When I create the instance in the dashboard, I get error:

No valid host was found. There are not enough hosts available.

In the /var/log/nova/nova-conductor.log file, there is the log:

2017-08-05 00:22:29.046 3834 WARNING nova.scheduler.utils [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available.
Traceback (most recent call last):

  File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 199, in inner
    return func(*args, **kwargs)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/manager.py", line 104, in select_destinations
    dests = self.driver.select_destinations(ctxt, spec_obj)

  File "/usr/lib/python2.7/site-packages/nova/scheduler/filter_scheduler.py", line 74, in select_destinations
    raise exception.NoValidHost(reason=reason)

NoValidHost: No valid host was found. There are not enough hosts available.

2017-08-05 00:22:29.048 3834 WARNING nova.scheduler.utils [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 2011e343-c8fc-4ed0-8148-b0d2b5ba37c3] Setting instance to ERROR state.
2017-08-05 00:22:30.785 3834 WARNING oslo_config.cfg [req-89c159c7-b40a-43eb-8f0d-9306eb73e83a 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] Option "auth_plugin" from group "neutron" is deprecated. Use option "auth_type" from group "neutron".

And I searched the SO, find a related post:Openstack-Devstack: Can't create instance, There are not enough hosts available

I checked the free_ram_mb in mysql:

MariaDB [nova]> select * from compute_nodes \G;
*************************** 1. row ***************************
           created_at: 2017-08-04 12:44:26
           updated_at: 2017-08-04 13:51:35
           deleted_at: NULL
                   id: 4
           service_id: NULL
                vcpus: 8
            memory_mb: 7808
             local_gb: 19
           vcpus_used: 0
       memory_mb_used: 512
        local_gb_used: 0
      hypervisor_type: QEMU
   hypervisor_version: 1005003
             cpu_info: {"vendor": "Intel", "model": "Broadwell", "arch": "x86_64", "features": ["smap", "avx", "clflush", "sep", "rtm", "vme", "invpcid", "msr", "fsgsbase", "xsave", "pge", "erms", "hle", "cmov", "tsc", "smep", "pcid", "pat", "lm", "abm", "adx", "3dnowprefetch", "nx", "fxsr", "syscall", "sse4.1", "pae", "sse4.2", "pclmuldq", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "rdtscp", "ht", "pse", "lahf_lm", "rdseed", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "ds", "invtsc", "pni", "aes", "avx2", "sse2", "ss", "hypervisor", "bmi1", "bmi2", "ssse3", "fpu", "cx16", "pse36", "mtrr", "movbe", "rdrand", "x2apic"], "topology": {"cores": 2, "cells": 1, "threads": 1, "sockets": 4}}
 disk_available_least: 18
          free_ram_mb: 7296
         free_disk_gb: 19
     current_workload: 0
          running_vms: 0
  hypervisor_hostname: ha-node1
              deleted: 0
              host_ip: 192.168.8.101
  supported_instances: [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"]]
            pci_stats: {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"}
              metrics: []
      extra_resources: NULL
                stats: {}
        numa_topology: NULL
                 host: ha-node1
 ram_allocation_ratio: 3
 cpu_allocation_ratio: 16
                 uuid: 9113940b-7ec9-462d-af06-6988dbb6b6cf
disk_allocation_ratio: 1
*************************** 2. row ***************************
           created_at: 2017-08-04 12:44:34
           updated_at: 2017-08-04 13:50:47
           deleted_at: NULL
                   id: 6
           service_id: NULL
                vcpus: 8
            memory_mb: 7808
             local_gb: 19
           vcpus_used: 0
       memory_mb_used: 512
        local_gb_used: 0
      hypervisor_type: QEMU
   hypervisor_version: 1005003
             cpu_info: {"vendor": "Intel", "model": "Broadwell", "arch": "x86_64", "features": ["smap", "avx", "clflush", "sep", "rtm", "vme", "invpcid", "msr", "fsgsbase", "xsave", "pge", "erms", "hle", "cmov", "tsc", "smep", "pcid", "pat", "lm", "abm", "adx", "3dnowprefetch", "nx", "fxsr", "syscall", "sse4.1", "pae", "sse4.2", "pclmuldq", "fma", "tsc-deadline", "mmx", "osxsave", "cx8", "mce", "de", "rdtscp", "ht", "pse", "lahf_lm", "rdseed", "popcnt", "mca", "pdpe1gb", "apic", "sse", "f16c", "ds", "invtsc", "pni", "aes", "avx2", "sse2", "ss", "hypervisor", "bmi1", "bmi2", "ssse3", "fpu", "cx16", "pse36", "mtrr", "movbe", "rdrand", "x2apic"], "topology": {"cores": 2, "cells": 1, "threads": 1, "sockets": 4}}
 disk_available_least: 18
          free_ram_mb: 7296
         free_disk_gb: 19
     current_workload: 0
          running_vms: 0
  hypervisor_hostname: ha-node2
              deleted: 0
              host_ip: 192.168.8.102
  supported_instances: [["i686", "qemu", "hvm"], ["x86_64", "qemu", "hvm"]]
            pci_stats: {"nova_object.version": "1.1", "nova_object.changes": ["objects"], "nova_object.name": "PciDevicePoolList", "nova_object.data": {"objects": []}, "nova_object.namespace": "nova"}
              metrics: []
      extra_resources: NULL
                stats: {}
        numa_topology: NULL
                 host: ha-node2
 ram_allocation_ratio: 3
 cpu_allocation_ratio: 16
                 uuid: 32b574df-52ac-43dc-87f8-353350449076
disk_allocation_ratio: 1
2 rows in set (0.00 sec)

You see the free_ram_mb: 7296, I just want to create a 512mb VM, but failed.


EDIT-1

The nova services all are up:

[root@ha-node1 ~]# nova service-list
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host     | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+
| 2  | nova-consoleauth | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:25.000000 | -               |
| 5  | nova-conductor   | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 7  | nova-cert        | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:23.000000 | -               |
| 15 | nova-scheduler   | ha-node3 | internal | enabled | up    | 2017-08-05T14:20:20.000000 | -               |
| 22 | nova-cert        | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:26.000000 | -               |
| 29 | nova-conductor   | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:22.000000 | -               |
| 32 | nova-consoleauth | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 33 | nova-consoleauth | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:29.000000 | -               |
| 36 | nova-scheduler   | ha-node1 | internal | enabled | up    | 2017-08-05T14:20:30.000000 | -               |
| 40 | nova-conductor   | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:26.000000 | -               |
| 44 | nova-cert        | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:27.000000 | -               |
| 46 | nova-scheduler   | ha-node2 | internal | enabled | up    | 2017-08-05T14:20:28.000000 | -               |
| 49 | nova-compute     | ha-node2 | nova     | enabled | up    | 2017-08-05T14:19:35.000000 | -               |
| 53 | nova-compute     | ha-node1 | nova     | enabled | up    | 2017-08-05T14:20:05.000000 | -               |
+----+------------------+----------+----------+---------+-------+----------------------------+-----------------+

The nova list:

[root@ha-node1 ~]# nova list 
+--------------------------------------+------+--------+------------+-------------+----------+
| ID                                   | Name | Status | Task State | Power State | Networks |
+--------------------------------------+------+--------+------------+-------------+----------+
| 20193e58-2c5b-44c6-a98f-a44e2001934f | vm1  | ERROR  | -          | NOSTATE     |          |

And the nova show instance:

[root@ha-node1 ~]# nova show 20193e58-2c5b-44c6-a98f-a44e2001934f
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                                                                                      |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                                                                                                                                                       |
| OS-EXT-AZ:availability_zone          | nova                                                                                                                                                                                                       |
| OS-EXT-SRV-ATTR:host                 | -                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:hostname             | vm1                                                                                                                                                                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000003                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                                                                                                                                                            |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                                                                                                                                                                          |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                                                                                                                                                            |
| OS-EXT-SRV-ATTR:reservation_id       | r-jct8kkcq                                                                                                                                                                                                 |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                                                                                                                                                                                                   |
| OS-EXT-SRV-ATTR:user_data            | -                                                                                                                                                                                                          |
| OS-EXT-STS:power_state               | 0                                                                                                                                                                                                          |
| OS-EXT-STS:task_state                | -                                                                                                                                                                                                          |
| OS-EXT-STS:vm_state                  | error                                                                                                                                                                                                      |
| OS-SRV-USG:launched_at               | -                                                                                                                                                                                                          |
| OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                          |
| accessIPv4                           |                                                                                                                                                                                                            |
| accessIPv6                           |                                                                                                                                                                                                            |
| config_drive                         |                                                                                                                                                                                                            |
| created                              | 2017-08-05T14:17:54Z                                                                                                                                                                                       |
| description                          | vm1                                                                                                                                                                                                        |
| fault                                | {"message": "No valid host was found. There are not enough hosts available.", "code": 500, "details": "  File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 496, in build_instances |
|                                      |     context, request_spec, filter_properties)                                                                                                                                                              |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 567, in _schedule_instances                                                                                                    |
|                                      |     hosts = self.scheduler_client.select_destinations(context, spec_obj)                                                                                                                                   |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 370, in wrapped                                                                                                                  |
|                                      |     return func(*args, **kwargs)                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 51, in select_destinations                                                                                             |
|                                      |     return self.queryclient.select_destinations(context, spec_obj)                                                                                                                                         |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method                                                                                                    |
|                                      |     return getattr(self.instance, __name)(*args, **kwargs)                                                                                                                                                 |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 32, in select_destinations                                                                                                |
|                                      |     return self.scheduler_rpcapi.select_destinations(context, spec_obj)                                                                                                                                    |
|                                      |   File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 126, in select_destinations                                                                                                     |
|                                      |     return cctxt.call(ctxt, 'select_destinations', **msg_args)                                                                                                                                             |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 169, in call                                                                                                                |
|                                      |     retry=self.retry)                                                                                                                                                                                      |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 97, in _send                                                                                                                 |
|                                      |     timeout=timeout, retry=retry)                                                                                                                                                                          |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 464, in send                                                                                                       |
|                                      |     retry=retry)                                                                                                                                                                                           |
|                                      |   File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 455, in _send                                                                                                      |
|                                      |     raise result                                                                                                                                                                                           |
|                                      | ", "created": "2017-08-05T14:18:14Z"}                                                                                                                                                                      |
| flavor                               | m1.tiny (1)                                                                                                                                                                                                |
| hostId                               |                                                                                                                                                                                                            |
| host_status                          |                                                                                                                                                                                                            |
| id                                   | 20193e58-2c5b-44c6-a98f-a44e2001934f                                                                                                                                                                       |
| image                                | cirros-0.3.4-x86_64 (202778cd-6b32-4486-9444-c167089d9082)                                                                                                                                                 |
| key_name                             | -                                                                                                                                                                                                          |
| locked                               | False                                                                                                                                                                                                      |
| metadata                             | {}                                                                                                                                                                                                         |
| name                                 | vm1                                                                                                                                                                                                        |
| os-extended-volumes:volumes_attached | []                                                                                                                                                                                                         |
| status                               | ERROR                                                                                                                                                                                                      |
| tags                                 | []                                                                                                                                                                                                         |
| tenant_id                            | 0d5998f2f7ec4c4892a32e06bafb19df                                                                                                                                                                           |
| updated                              | 2017-08-05T14:18:16Z                                                                                                                                                                                       |
| user_id                              | 2a5fa182fb1b459980db09cd1572850e                                                                                                                                                                           |
+--------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

EDIT-2

The nova-compute.log in the /var/log/nova/ useful information:

......
2017-08-05 22:17:42.669 103174 INFO nova.compute.resource_tracker [req-60a062ce-4b3d-4cb7-863e-2f9bba0bc6ec - - - - -] Compute_service record updated for ha-node1:ha-node1
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [req-7dded91f-7497-4d20-ba89-69f867a2a8fb 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Instance failed to spawn
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Traceback (most recent call last):
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2078, in _build_resources
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     yield resources
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1920, in _build_and_run_instance
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     block_device_info=block_device_info)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2584, in spawn
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     admin_pass=admin_password)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2959, in _create_image
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     fileutils.ensure_tree(libvirt_utils.get_instance_path(instance))
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib/python2.7/site-packages/oslo_utils/fileutils.py", line 40, in ensure_tree
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     os.makedirs(path, mode)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]     mkdir(name, mode)
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] OSError: [Errno 13] Permission denied: '/var/lib/nova/instances/20193e58-2c5b-44c6-a98f-a44e2001934f'
2017-08-05 22:18:03.357 103174 ERROR nova.compute.manager [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f]
2017-08-05 22:18:11.563 103174 INFO nova.compute.manager [req-7dded91f-7497-4d20-ba89-69f867a2a8fb 2a5fa182fb1b459980db09cd1572850e 0d5998f2f7ec4c4892a32e06bafb19df - - -] [instance: 20193e58-2c5b-44c6-a98f-a44e2001934f] Terminating instance
....

Solution

  • Finally I found I mount the /var/lib/nova/ to nfs directory /mnt/sdb/var/lib/nova/, but the /mnt/sdb/var/lib/nova/ permission is root:root, so I changed to the nova:nova(same to the /var/lib/nova/).

    command :

    chown -R nova:nova nova