When I delete a compute service with: openstack compute service delete a04910ef-1441-4949-8ffb-6393c22141b2, I can restart the nova-compute on the machine with sudo systemctl restart nova-compute to get back the service in openstack.
However, the node is not listed anymore in openstack hypervisor list (os-compute03.maas for instance in my case):
$ openstack compute service list --service nova-compute
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+
| ID | Binary | Host | Zone | Status | State | Updated At |
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+
| d1fadd40-6035-4f76-b8c1-5b981d003832 | nova-compute | os-compute08.maas | nova | disabled | down | 2024-09-25T14:02:55.000000 |
| 7e02a0bd-0e53-45bc-9680-99a33d98c05b | nova-compute | os-compute04.maas | nova | enabled | up | 2024-10-18T11:43:12.000000 |
| 7f47d65e-b041-44dc-927e-085effdf0ec9 | nova-compute | os-compute09.maas | nova | enabled | up | 2024-10-18T11:43:07.000000 |
| 3a7527b3-3664-4ae1-ac90-ac04e835ee5b | nova-compute | os-compute03.maas | nova | enabled | up | 2024-10-18T11:43:13.000000 |
+--------------------------------------+--------------+-------------------+------+----------+-------+----------------------------+
$ openstack hypervisor list
+----+---------------------+-----------------+----------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
+----+---------------------+-----------------+----------+-------+
| 1 | os-compute08.maas | QEMU | 10.0.1.8 | down |
| 2 | os-compute04.maas | QEMU | 10.0.1.4 | up |
| 3 | os-compute09.maas | QEMU | 10.0.1.9 | up |
+----+---------------------+-----------------+----------+-------+
^ no os-compute03.maas
$ sudo nova-manage cell_v2 list_hosts
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
+-----------+--------------------------------------+-------------------+
| Cell Name | Cell UUID | Hostname |
+-----------+--------------------------------------+-------------------+
| cell1 | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute04.maas |
| cell1 | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute08.maas |
| cell1 | d3dfd353-b4ee-4293-b362-1e0175ebe337 | os-compute09.maas |
+-----------+--------------------------------------+-------------------+
$ sudo nova-manage cell_v2 discover_hosts --verbose
Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code.
Deprecated: Option "logdir" from group "DEFAULT" is deprecated. Use option "log-dir" from group "DEFAULT".
Found 2 cell mappings.
Skipping cell0 since it does not contain hosts.
Getting computes from cell 'cell1': d3dfd353-b4ee-4293-b362-1e0175ebe337
Found 0 unmapped computes in cell: d3dfd353-b4ee-4293-b362-1e0175ebe337
I tried to restart nova controller:
$ sudo systemctl restart nova-scheduler
$ sudo systemctl restart nova-conductor
When restarting the nova controller, the logs does say the following:
2024-10-18 11:53:44.939 630048 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.939 630046 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.940 630045 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:53:44.968 630047 INFO nova.scheduler.host_manager [None req-eb8c8117-136c-4a13-ac57-d32d4099388a - - - - - -] Received a sync request from an unknown host 'os-compute09.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.539 630048 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.541 630047 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.542 630046 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:54:02.543 630045 INFO nova.scheduler.host_manager [None req-ad410200-f99f-4dc7-896d-637dd9620568 - - - - - -] Received a sync request from an unknown host 'os-compute04.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.432 630048 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.432 630048 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.435 630047 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.435 630046 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.436 630045 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Host mapping not found for host os-compute03.maas. Not tracking instance info for this host.
2024-10-18 11:55:06.436 630047 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.437 630045 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
2024-10-18 11:55:06.436 630046 INFO nova.scheduler.host_manager [None req-53ac9845-afb5-48ec-b24f-344d2afcfc3f - - - - - -] Received a sync request from an unknown host 'os-compute03.maas'. Re-created its InstanceList.
And particularly: Host mapping not found for host os-compute03.maas. Not tracking instance info for this host. I guess this is when the node should be added in the nova cell_v2? And this is indeed not the case.
I guess the following records in the database concerns the compute-nodes:
mysql> SELECT hypervisor_hostname, host_ip FROM nova.compute_nodes;
+---------------------+----------+
| hypervisor_hostname | host_ip |
+---------------------+----------+
| os-compute08.maas | 10.0.1.8 |
| os-compute04.maas | 10.0.1.4 |
| os-compute09.maas | 10.0.1.9 |
| os-compute03.maas | 10.0.1.3 |
+---------------------+----------+
4 rows in set (0.00 sec)
os-compute03.maas is indeed not listed in the nova_api.host_mappings table:
SELECT * FROM nova_api.host_mappings;
+---------------------+------------+----+---------+-------------------+
| created_at | updated_at | id | cell_id | host |
+---------------------+------------+----+---------+-------------------+
| 2024-07-02 21:23:10 | NULL | 1 | 2 | os-compute08.maas |
| 2024-07-02 21:23:11 | NULL | 2 | 2 | os-compute04.maas |
| 2024-07-02 21:23:11 | NULL | 3 | 2 | os-compute09.maas |
+---------------------+------------+----+---------+-------------------+
I did not tried to add the host manually in this table. I suppose this should be done automatically, but when? What did I miss?
How can we further debug this situation?
After 3 days of struggling, when I finally decided to look for help and write this post, I end up finding the solution the same day.
If anyone is in this situation, try to execute the following command:
$ sudo nova-manage cell_v2 map_cell_and_hosts
Ref: https://docs.openstack.org/nova/latest/cli/nova-manage.html#cell-v2-map-cell-and-hosts