“Host Key Verification Failed” Error in OpenStack Compute Node

Overview the Problem

If you see something like “Resize error: not able to execute ssh command” in your nova logs like on snippet log below, along with “Host key verification failed,” the issue is likely due to SSH strict host key checking. This happens when OpenStack tries to connect to another node but refuses due to an unverified SSH key.

2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server [req-70cbe032-df18-4fa8-b1d9-ec0c20d72755 1d1a3c1a90054db398f82efb23f6911e 0ab36a22526e4a37b18b6db397c41897 - 5f62e7b25d9e4412a7e3d08305987642 d34dd509881540aa88106d4b58c16521] Exception during message handling: nova.exception.ResizeError: Resize error: not able to execute ssh command: Unexpected error while running command.
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/driver.py", line 8341, in migrate_disk_and_power_off
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     self._remotefs.create_dir(dest, inst_base)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 95, in create_dir
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     on_completion=on_completion)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/virt/libvirt/volume/remotefs.py", line 185, in create_dir
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     on_execute=on_execute, on_completion=on_completion)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/utils.py", line 241, in ssh_execute
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     return execute(*ssh_cmd, **kwargs)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/nova/utils.py", line 233, in execute
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     return processutils.execute(*cmd, **kwargs)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server     cmd=sanitized_cmd)
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command.
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server Command: ssh -o BatchMode=yes 192.168.1.4 mkdir -p /var/lib/nova/instances/ae2c0793-a95b-4618-bcd6-f6a8eadf9087
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server Exit code: 255
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server Stdout: ''
2024-07-10 08:12:52.497 898673 ERROR oslo_messaging.rpc.server Stderr: 'Host key verification failed.\r\n'

Solving the Problem

To fix this, you can disable strict host key checking for the Nova user. Switch to the Nova user and modify the SSH config by adding StrictHostKeyChecking no. Then, set the correct permissions and make sure if compute node can remote SSH to another compute node. This ensures OpenStack can execute remote SSH commands without getting stuck on key verification.

# SSH to compute node
ssh <to_compute_node>

# Login as nova user
sudo su - nova

# Check into SSH config file, if doesn't exist you can create a new one
cat /var/lib/nova/.ssh/config

# Add disable strict host key checking configuration to SSH config file
cat <<EOF >>/var/lib/nova/.ssh/config
Host *
    StrictHostKeyChecking no
EOF

# Set permission to read only
chmod 400 /var/lib/nova/.ssh/config

After making this change, retry your operation, and things should work smoothly.