0

I'm failing to deploy an overcloud on 4 nodes using a TripleO undercloud instance. The journals on the overcloud nodes are inconclusive.

[root@director deploy]# grep -E '(overcloud-controller-2.*(ironic-python-agent|kernel:))|localhost\.localdomain.*iscsi' journal
Mar 18 14:36:55 localhost.localdomain kernel: iscsi: registered transport (tcp)
Mar 18 14:36:55 localhost.localdomain kernel: iscsi: registered transport (qla4xxx)
Mar 18 14:36:55 localhost.localdomain kernel: iscsi: registered transport (cxgb4i)
Mar 18 14:36:55 localhost.localdomain kernel: iscsi: registered transport (bnx2i)
Mar 18 14:36:55 localhost.localdomain kernel: iscsi: registered transport (be2iscsi)
Mar 18 14:36:55 localhost.localdomain kernel: In beiscsi_module_init, tt=(____ptrval____)
Mar 18 14:36:55 localhost.localdomain systemd-modules-load[504]: Inserted module 'iscsi_boot_sysfs'
Mar 18 14:36:55 localhost.localdomain systemd-modules-load[504]: Inserted module 'iscsi_ibft'
Mar 18 14:36:55 localhost.localdomain systemd-modules-load[504]: Inserted module 'iscsi_tcp'
Mar 18 14:36:55 localhost.localdomain systemd-modules-load[504]: Inserted module 'be2iscsi'
Mar 18 14:36:57 localhost.localdomain systemd[1]: Listening on Open-iSCSI iscsid Socket.
Mar 18 14:36:57 localhost.localdomain systemd[1]: Listening on Open-iSCSI iscsiuio Socket.
Mar 18 14:36:57 localhost.localdomain kernel: iscsi: registered transport (iser)
Mar 18 14:36:57 localhost.localdomain init-ibft-interfaces.sh[1054]: + iscsistart -v
Mar 18 14:36:57 localhost.localdomain init-ibft-interfaces.sh[1054]: iscsistart version 6.2.0.878-5
Mar 18 14:36:57 localhost.localdomain init-ibft-interfaces.sh[1054]: + iscsistart -f
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.134 1372 INFO ironic_python_agent.agent [-] Starting ironic-python-agent version: 6.1.2.dev19
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.246 1372 INFO root [-] No new RAID devices assembled during start-up
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.263 1372 INFO root [-] Picked root device /dev/sda for node None based on root device hints None
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.357 1372 INFO root [-] No new RAID devices assembled during start-up
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.373 1372 INFO root [-] Picked root device /dev/sda for node None based on root device hints None
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.374 1372 WARNING root [-] Can't find field vendor for device lo in device class net: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/lo/device/vendor'
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.387 1372 WARNING root [-] Can't find field vendor for device lo in device class net: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/lo/device/vendor'
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.485 1372 INFO root [-] No new RAID devices assembled during start-up
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.500 1372 INFO root [-] Picked root device /dev/sda for node None based on root device hints None
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.501 1372 INFO root [-] Hardware manager found: ironic_python_agent.hardware:GenericHardwareManager
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.513 1372 WARNING root [-] Can't find field vendor for device lo in device class net: FileNotFoundError: [Errno 2] No such file or directory: '/sys/class/net/lo/device/vendor'
Mar 18 14:37:29 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:29.581 1372 INFO root [-] Collecting full inventory
Mar 18 14:37:31 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:31.605 1372 INFO root [-] Inventory collected in 2.02 second(s)
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.160 1372 INFO ironic_python_agent.agent [-] Lookup succeeded, node UUID is 44503b21-394f-43e8-8ef9-5fdcba13dc57
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.160 1372 INFO root [-] Cached node 44503b21-394f-43e8-8ef9-5fdcba13dc57, waiting for its root device to appear
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.180 1372 INFO ironic_lib.utils [-] Root device found! The device "{'name': '/dev/sda', 'model': 'ST4000NM0033', 'size': 4000787030016, 'rotational': True, 'wwn': '0x5000c50067658b77', 'serial': 'Z1Z4KLAB', 'vendor': 'ATA', 'wwn_with_extension': '0x5000c50067658b77', 'wwn_vendor_extension': None, 'hctl': '0:0:0:0', 'by_path': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1'}" matches the root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.181 1372 INFO root [-] Picked root device /dev/sda for node 44503b21-394f-43e8-8ef9-5fdcba13dc57 based on root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.182 1372 INFO oslo.service.wsgi [-] ironic-python-agent listening on :::9999
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.183 1372 INFO ironic_python_agent.api.app [-] Started API service on port 9999
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.184 1372 INFO ironic_python_agent.agent [-] starting heartbeater
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.184 1372 INFO eventlet.wsgi.server [-] (1372) wsgi starting up on http://[::]:9999
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.352 1372 INFO ironic_python_agent.agent [-] heartbeat successful
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.353 1372 INFO ironic_python_agent.agent [-] sleeping before next heartbeat, interval: 165.44751401941707
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.603 1372 INFO root [-] No new RAID devices assembled during start-up
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.622 1372 INFO ironic_lib.utils [-] Root device found! The device "{'name': '/dev/sda', 'model': 'ST4000NM0033', 'size': 4000787030016, 'rotational': True, 'wwn': '0x5000c50067658b77', 'serial': 'Z1Z4KLAB', 'vendor': 'ATA', 'wwn_with_extension': '0x5000c50067658b77', 'wwn_vendor_extension': None, 'hctl': '0:0:0:0', 'by_path': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1'}" matches the root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.622 1372 INFO root [-] Picked root device /dev/sda for node 44503b21-394f-43e8-8ef9-5fdcba13dc57 based on root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.623 1372 INFO root [-] Synchronous command get_deploy_steps completed: {'deploy_steps': {'GenericHardwareManager': [{'step': 'apply_configuration', 'priority': 0, 'interface': 'raid', 'reboot_requested': False, 'argsinfo': {'raid_config': {'description': 'The RAID configuration to apply.', 'required': True}, 'delete_existing': {'description': "Setting this to 'True' indicates to delete existing RAID configuration prior to creating the new configuration. Default value is 'True'.", 'required': False}}}]}, 'hardware_manager_version': {'generic_hardware_manager': '1.1'}}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.624 1372 INFO eventlet.wsgi.server [-] ::ffff:10.100.4.5 "POST /v1/commands/?wait=true&agent_token=YBAoDxkuSrAuZ6yg5tCjFW7kIDxuVcq7GjJmjDwVDOo HTTP/1.1" status: 200  len: 4605 time: 0.1221805
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.754 1372 INFO ironic_lib.utils [-] Root device found! The device "{'name': '/dev/sda', 'model': 'ST4000NM0033', 'size': 4000787030016, 'rotational': True, 'wwn': '0x5000c50067658b77', 'serial': 'Z1Z4KLAB', 'vendor': 'ATA', 'wwn_with_extension': '0x5000c50067658b77', 'wwn_vendor_extension': None, 'hctl': '0:0:0:0', 'by_path': '/dev/disk/by-path/pci-0000:00:1f.2-ata-1'}" matches the root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:32 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:32.755 1372 INFO root [-] Picked root device /dev/sda for node 44503b21-394f-43e8-8ef9-5fdcba13dc57 based on root device hints {'wwn_with_extension': '0x5000c50067658b77'}
Mar 18 14:37:34 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:34.800 1372 INFO ironic_lib.disk_utils [-] Disk metadata on /dev/sda successfully destroyed for node 44503b21-394f-43e8-8ef9-5fdcba13dc57
Mar 18 14:37:34 overcloud-controller-2 kernel: Rounding down aligned max_sectors from 65535 to 65528
Mar 18 14:37:34 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:34.827 1372 INFO ironic_python_agent.extensions.iscsi [-] Created iSCSI target with iqn iqn.2008-10.org.openstack:44503b21-394f-43e8-8ef9-5fdcba13dc57, portal port 3260, on device /dev/sda using linux-io
Mar 18 14:37:34 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:34.827 1372 INFO root [-] Synchronous command start_iscsi_target completed: {'iscsi_target_iqn': 'iqn.2008-10.org.openstack:44503b21-394f-43e8-8ef9-5fdcba13dc57'}
Mar 18 14:37:34 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:34.829 1372 INFO eventlet.wsgi.server [-] ::ffff:10.100.4.5 "POST /v1/commands/?wait=true&agent_token=YBAoDxkuSrAuZ6yg5tCjFW7kIDxuVcq7GjJmjDwVDOo HTTP/1.1" status: 200  len: 519 time: 2.0919275
Mar 18 14:37:37 overcloud-controller-2 kernel: iSCSI/iqn.1994-05.com.redhat:297b558a33ee: Unsupported SCSI Opcode 0xa3, sending CHECK_CONDITION.
Mar 18 14:37:43 overcloud-controller-2 ironic-python-agent[1372]: 2021-03-18 14:37:43.123 1372 INFO ironic_python_agent.utils [-] Collecting system logs and debugging information

The ironic-conductor.log simply indicates the expected volume did not become available.

2021-03-19 10:37:59.290 7 ERROR ironic.drivers.modules.iscsi_deploy [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:37:59.292 7 ERROR ironic.drivers.modules.iscsi_deploy [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] Failed to login to an iSCSI target due to iSCSI connection was not seen by the file system after attempting to verify 3 times.: ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:00.998 7 ERROR ironic.drivers.modules.iscsi_deploy [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] Deploy of instance 5c2181bd-35e6-4fa0-969f-8740d631695b on node 44503b21-394f-43e8-8ef9-5fdcba13dc57 failed: iSCSI connection was not seen by the file system after attempting to verify 3 times.: ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:01.992 7 ERROR ironic.conductor.utils [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] Deploy failed for instance 5c2181bd-35e6-4fa0-969f-8740d631695b. Error: iSCSI connection was not seen by the file system after attempting to verify 3 times.: ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:02.094 7 ERROR ironic.conductor.task_manager [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] Node 44503b21-394f-43e8-8ef9-5fdcba13dc57 moved to provision state "deploy failed" from state "deploying"; target provision state is "active": ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base [req-26435ce3-96a9-4fe9-9124-16e7db2d9f36 - - - - -] Asynchronous exception for node 44503b21-394f-43e8-8ef9-5fdcba13dc57: Node failed to deploy. Error: iSCSI connection was not seen by the file system after attempting to verify 3 times.: ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base Traceback (most recent call last):
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/agent_base.py", line 507, in _heartbeat_deploy_wait
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     self.continue_deploy(task)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     result = f(*args, **kwargs)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/conductor/task_manager.py", line 148, in wrapper
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     return f(*args, **kwargs)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 621, in continue_deploy
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     uuid_dict_returned = do_agent_iscsi_deploy(task, self._client)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     result = f(*args, **kwargs)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 565, in do_agent_iscsi_deploy
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     conv_flags=conv_flags)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic_lib/metrics.py", line 59, in wrapped
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     result = f(*args, **kwargs)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 488, in continue_deploy
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     _fail_deploy(task, e, raise_exception=False)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     self.force_reraise()
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     six.reraise(self.type_, self.value, self.tb)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     raise value
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 482, in continue_deploy
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     uuid_dict_returned = deploy_partition_image(**params)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 294, in deploy_partition_image
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     with _iscsi_setup_and_handle_errors(address, port, iqn, lun) as dev:
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     return next(self.gen)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 222, in _iscsi_setup_and_handle_errors
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     login_iscsi(address, port, iqn)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 107, in login_iscsi
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     LOG.error("Failed to login to an iSCSI target due to %s", e)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     self.force_reraise()
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     six.reraise(self.type_, self.value, self.tb)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/six.py", line 703, in reraise
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     raise value
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 102, in login_iscsi
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     target_iqn)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base   File "/usr/lib/python3.6/site-packages/ironic/drivers/modules/iscsi_deploy.py", line 142, in check_file_system_for_iscsi_device
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base     raise exception.InstanceDeployFailure(msg)
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base ironic.common.exception.InstanceDeployFailure: iSCSI connection was not seen by the file system after attempting to verify 3 times.
2021-03-19 10:38:13.278 7 ERROR ironic.drivers.modules.agent_base

I'm using Openstack Release Ussuri.

The undercloud is built from an iPXE boot install using this kickstart:

#version=RHEL8
clearpart --all --initlabel
text

%include /tmp/network.ks

keyboard --vckeymap=us --xlayouts='us'
lang en_US.UTF-8

firstboot --disable
eula --agreed
reboot
skipx
services --enabled="chronyd"

url --url="http://redacted/centos/8/BaseOS/x86_64/os"
timezone --isUtc America/New_York
rootpw --iscrypted redacted
firewall --enabled --ssh
selinux --enforcing

part biosboot --fstype=biosboot --size=1
part /boot --fstype="ext4" --size=1024
part pv.01 --fstype="lvmpv" --grow
volgroup cl pv.01
logvol / --fstype="xfs" --name=root --vgname=cl --percent=100 --grow

%packages
@^minimal-environment
@standard

wget
sudo
%end

%addon com_redhat_kdump --disable --reserve-mb='auto'
%end

%pre
#!/bin/bash

DEVICE=$(ip r|grep default|grep  -oP '(?<=dev )\S*')
declare -A value
while IFS= read -r line; do
  kvp=${line##DHCP4.OPTION\[*\]:*[ ][ ]}
  value+=([${kvp% = *}]=${kvp#* = })
done <<< $(nmcli -f dhcp4 device show $DEVICE)

echo "network --device link --bootproto static --ip ${value[ip_address]} --netmask ${value[subnet_mask]} --gateway ${value[routers]} --hostname ${value[host_name]}.${value[domain_name]} --nameserver ${value[domain_name_servers]} --onboot yes --noipv6" > /tmp/network.ks

%end

%post --log=/root/install-post.log

PATH=/bin:/sbin:/usr/sbin:/usr/sbin
export PATH

cat << EOF > /etc/hosts
127.0.0.1   localhost.localdomain localhost
$(hostname -i)  $(hostname -f) $(hostname -s)

# The following lines are desirable for IPv6 capable hosts

::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
EOF

# Install authorized root access keys
mkdir /root/.ssh
chmod 700 /root/.ssh

cat <<EOF >/root/.ssh/authorized_keys
ssh-rsa redacted redacted@redacted
EOF
/bin/chown -R root:root /root/.ssh
/bin/chmod 400 /root/.ssh/*

# now fix roots SELinux context
restorecon -R -v /root

%end

The overcloud node kernels claim to be Linux version 4.18.0-240.15.1.el8_3.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Mon Mar 1 17:16:16 UTC 2021.

The undercloud, and the images, are constructed like so:

#!/bin/bash

useradd stack
passwd stack --stdin <<< "redacted"
echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
chmod 0440 /etc/sudoers.d/stack

su - stack <<"EOF"

OpnStkRel="ussuri"

sudo dnf install -y https://trunk.rdoproject.org/centos8/component/tripleo/current/python3-tripleo-repos-0.1.1-0.20210118183911.2cfaa48.el8.noarch.rpm
sudo -E tripleo-repos -b ${OpnStkRel} current ceph
sudo dnf install -y python3-tripleoclient
sudo dnf install -y ceph-ansible

cat <<EOFi >/home/stack/undercloud.conf
[DEFAULT]
clean_nodes = true
inspection_extras = true
inspection_runbench = true
local_interface = ens4
local_ip = 10.100.4.5/24
overcloud_domain_name = redacted
undercloud_admin_host = 10.100.4.6
undercloud_debug = false
undercloud_hostname = director.redacted
undercloud_nameservers = 192.168.2.137
undercloud_ntp_servers = 192.168.4.250
undercloud_public_host = 10.100.4.7
subnets = ctlplane-subnet

[ctlplane-subnet]
cidr = 10.100.4.0/24
dhcp_end = 10.100.4.89
dhcp_start = 10.100.4.70
dns_nameservers = 192.168.2.137
gateway = 10.100.4.1
inspection_iprange = 10.100.4.50,10.100.4.69
masquerade_network = 10.100.4.0/24
EOFi

mkdir ~/images
mkdir ~/templates
openstack tripleo container image prepare default --local-push-destination --output-env-file ~/templates/containers-prepare-parameter.yaml
openstack undercloud install
source ~/stackrc
openstack flavor create --id auto --ram 4096 --disk 40 --vcpus 1 computeHCI
openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "resources:CUSTOM_BAREMETAL"="1" --property "resources:DISK_GB"="0" --property "resources:MEMORY_MB"="0" --property "resources:VCPU"="0" computeHCI
openstack flavor set --property "capabilities:profile"="computeHCI" computeHCI
export OS_YAML="/usr/share/openstack-tripleo-common/image-yaml/overcloud-images-centos8.yaml"
export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean* /etc/yum.repos.d/tripleo-centos-*"
export STABLE_RELEASE=${OpnStkRel}
cd images/
openstack overcloud image build --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-images-python3.yaml --config-file $OS_YAML
openstack overcloud image upload
cd ..
wget http://repo.redacted/utils/openstack/config/instackenv_4node.json -O ~/instackenv.json
wget http://repo.redacted/utils/openstack/config/deploy.sh -O ~/deploy.sh
chmod +x deploy.sh
openstack overcloud roles generate -o ~/templates/roles_data.yaml Controller ComputeHCI
wget http://repo.redacted/utils/openstack/config/network_data.yaml -O ~/templates/network_data.yaml
wget http://repo.redacted/utils/openstack/config/node-info.yaml -O ~/templates/node-info.yaml
wget http://repo.redacted/utils/openstack/config/podman-ha.yaml -O ~/templates/podman-ha.yaml
wget http://repo.redacted/utils/openstack/config/inject-trust-anchor-hiera.yaml -O ~/templates/inject-trust-anchor-hiera.yaml
wget http://repo.redacted/utils/openstack/config/ceph-environment.yaml -O ~/templates/ceph-environment.yaml
wget http://repo.redacted/utils/openstack/config/network-isolation.yaml -O ~/templates/network-isolation.yaml
wget http://repo.redacted/utils/openstack/config/network-environment.yaml -O ~/templates/network-environment.yaml
wget http://repo.redacted/utils/openstack/config/firstboot-environment.yaml -O ~/templates/firstboot-environment.yaml
mkdir ~/templates/firstboot
mkdir ~/templates/nic-configs
wget http://repo.redacted/utils/openstack/config/firstboot-config.yaml -O ~/templates/firstboot/firstboot-config.yaml
wget http://repo.redacted/utils/openstack/config/computehci.yaml -O ~/templates/nic-configs/computehci.yaml
wget http://repo.redacted/utils/openstack/config/controller.yaml -O ~/templates/nic-configs/controller.yaml
EOF

I've been registering nodes manually, like so:

(undercloud) [stack@director ~]$ openstack overcloud node import ~/instackenv.json
Successfully registered node UUID 892a65c3-0aed-4f42-bcec-e2613b2bd5d3
Successfully registered node UUID 44503b21-394f-43e8-8ef9-5fdcba13dc57
Successfully registered node UUID 0b09454d-4092-42f7-abe7-80cb5414108e
Successfully registered node UUID af110119-3b85-47e3-a335-d0b25f4b849e
(undercloud) [stack@director ~]$ openstack overcloud node introspect --all-manageable
...

Successfully introspected nodes: ['892a65c3-0aed-4f42-bcec-e2613b2bd5d3', '44503b21-394f-43e8-8ef9-5fdcba13dc57', '0b09454d-4092-42f7-abe7-80cb5414108e', 'af110119-3b85-47e3-a335-d0b25f4b849e']
(undercloud) [stack@director ~]$ openstack overcloud node configure --all-manageable --root-device=sda
Successfully configured the nodes.
(undercloud) [stack@director ~]$ openstack overcloud node provide --all-manageable
...

Successfully provided nodes: ['892a65c3-0aed-4f42-bcec-e2613b2bd5d3', '44503b21-394f-43e8-8ef9-5fdcba13dc57', '0b09454d-4092-42f7-abe7-80cb5414108e', 'af110119-3b85-47e3-a335-d0b25f4b849e']
(undercloud) [stack@director ~]$ openstack baremetal node list
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| UUID                                 | Name      | Instance UUID | Power State | Provisioning State | Maintenance |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+
| 892a65c3-0aed-4f42-bcec-e2613b2bd5d3 | os1srvp01 | None          | power off   | available          | False       |
| 44503b21-394f-43e8-8ef9-5fdcba13dc57 | os1srvp03 | None          | power off   | available          | False       |
| 0b09454d-4092-42f7-abe7-80cb5414108e | os1srvp04 | None          | power off   | available          | False       |
| af110119-3b85-47e3-a335-d0b25f4b849e | os1srvp05 | None          | power off   | available          | False       |
+--------------------------------------+-----------+---------------+-------------+--------------------+-------------+

...and finally, the deployment is done like so:

#!/usr/bin/env bash

source ~/stackrc

time openstack overcloud deploy --templates \
  -r /home/stack/templates/roles_data.yaml \
  -n /home/stack/templates/network_data.yaml \
  -e /home/stack/templates/containers-prepare-parameter.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \
  -e /home/stack/templates/podman-ha.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/enable-swap.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
  -e /home/stack/templates/node-info.yaml \
  -e /home/stack/templates/inject-trust-anchor-hiera.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-dashboard.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml \
  -e /home/stack/templates/ceph-environment.yaml \
  -e /usr/share/openstack-tripleo-heat-templates/environments/services/neutron-ovn-dvr-ha.yaml \
  -e /home/stack/templates/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /home/stack/templates/firstboot-environment.yaml

Since everything is scripted there is no doubt about how the environemnt is built and configured, although, it builds from live repositories which are a moving target. The last rebuild was last night into this morning and this issue was still present as of 14:30 GMT.

Does anyone know the root cause or a workaround? I gave up on Victoria due to insoluble issue and fell back to Ussuri. Now it seems like I may need to fall back to Train unless I can get past this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.