I am receiving an error saying "Trace nova Stderr:'iscsiadm:No portal found. \n'. Trace nova Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack:volume-0000003 -p :3260 --rescan

Plz let me know if further info is req. Thanks in advance.


There are two servers
1. Controller node (hostname: cc)
2. Compute node (hostname: nc)

OS: Ubuntu 12.04 (Headless)
Openstack flavor: Essex

We had our set up working fine till last week. But, when we wanted to install Ubuntu desktop to move out from Headless and get some beautiful screens, we had type the command (in cc host)

#sudo apt-get install ubuntu-desktop

And then and there all our problems started. I could notice that, it is removing Apache, Openstack – dashboard and nova volume also with out any prior notice or confirmation. It didn’t even update the things.

So all our VM instances went down.

And then, when I ran #sudo nova-manage service list, it use to show me XXX for nova-volume and nova-compute. We reinstalled nova-volume which changed the status from XXX to :-).

And then, we tried all the things we know like reinstalling apahce2, reinstalling openstack-dashboard, apt-get update, apt-get upgrade, removing suspecting records from mysql, removing lvm.

After this some how we could get our openstack dashboard up and running. So now, our concern was about nova-compute service which was displaying XXX in service list.

Attempt 1: When we changed the nova.conf file with start_guests_on_host_boot=false and resume_guests_state_on_host_boot=false and rebooted the cc host, it displayed :-) for all services after reboot. Now when I checked dashboard, it was displaying status as Shut off for all vms with spinner rotating.

Attempt 2: When we changed the nova.conf file with start_guests_on_host_boot=true and resume_guests_state_on_host_boot=true and rebooted the cc host, it displayed XXX for nova-compute service after reboot. When we checked the /var/log/nova/nova-compute.log file, it displayed error like

iscsiadm: No portal found. \n'. Trace nova Command: sudo nova-rootwrap iscsiadm -m node -T iqn.2010-10.org.openstack: volume-0000003 -p: 3260 --rescan

Fix worked for above thing:
1.Changed the two flags in nova conf to true.
2.Executed #apt-get remove --purge iscsitarget
3.Rebooted the cc host.
4.Executed #/etc/init.d/open-iscsi restart.
5.Executed #sudo /etc/init.d/networking restart
6.Executed #sudo service ntp restart
7.Executed #sudo service tgt restart
8.Executed #sudo nova-manage db sync
9.Restarted nova-volume, nova-compute, nova-cert, nova-network, nova-scheduler, nova-consoleauth of cc host.

Result of the fix: We were able to get (we did not get this before) volumes list when we ran the below command
#iscsiadm -m discovery -t sendtargets -p :3260
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000007
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000005
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000014
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000003
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-0000000f
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000013
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000004
CC_HOST_IP:3260, 1 iqn.2010-10.org.openstack: volume-00000008

And still all the VMs were in status Shutoff and the spinner was spinning continuously for 1 whole day. We noticed that we were able to ping some of the VMs and also connect to them with MSTSC.

And now, we did not see the Portal not found error in the compute log. But we noticed error saying below.

libvirtError: Unable to allow access for disk path /dev/disk/by-path/ip-:3260-iscsi-iqn.2010-10.org.openstack:volume-00000017-lun-1: No such file or directory

And for your information, as part of my R&D, I had deleted (lvremove volume-17) the lvm related to that.

Current status: On cc host, Getting the error as

libvirtError: Unable to allow access for disk path /dev/disk/by-path/ip-:3260-iscsi-iqn.2010-10.org.openstack:volume-00000017-lun-1: No such file or directory

On nc host, getting error as

libvirtError: Unable to allow access for disk path /dev/disk/by-path/ip-:3260-iscsi-iqn.2010-10.org.openstack:volume-00000020-lun-1: No such file or directory

1. We don’t have cinder here. Only nova-volume, nova-compute, nova-cert, nova-network, nova-scheduler, nova-consoleauth.
2. Current status of services is

Binary Host Zone Status State Updated_At nova-network cc nova enabled :-) 2016-05-19 13:48:48
nova-compute cc nova enabled XXX 2016-05-19 11:37:23
nova-scheduler cc nova enabled :-) 2016-05-19 13:48:51
nova-volume cc nova enabled :-) 2016-05-19 13:48:48
nova-consoleauth cc nova enabled :-) 2016-05-19 13:48:50
nova-cert cc nova enabled :-) 2016-05-19 13:48:49
nova-compute nc nova enabled XXX 2016-05-19 11:37:13
nova-volume nc nova enabled :-) 2016-05-19 13:48:44

  1. Yes, tgt is running. root@cc:/dev/disk/by-path# sudo service tgt status tgt start/running, process 2583

  2. Config files are below.
    nova.conf on cc host: --dhcpbridge_flagfile=/etc/nova/nova.conf --my_ip= --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --debug = true --state_path=/var/lib/nova --lock_path=/run/lock/nova --allow_admin_api=true --use_deprecated_auth=false --auth_strategy=keystone --scheduler_driver=nova.scheduler.simple.SimpleScheduler --max_cores=106 --s3_host= --ec2_host= --rabbit_host= --cc_host= --nova_url=http://:8774/v1.1/ --routing_source_ip= --glance_api_servers=:9292 --image_service=nova.image.glance.GlanceImageService --iscsi_ip_prefix=192.168.4 --sql_connection=mysql://nova:nova@/nova --ec2_url=http://:8773/services/Cloud --keystone_ec2_url=http://:5000/v2.0/ec2tokens --api_paste_config=/etc/nova/api-paste.ini --libvirt_type=kvm --libvirt_use_virtio_for_bridges=true --start_guests_on_host_boot=true --resume_guests_state_on_host_boot=true --novnc_enabled=true --novncproxy_base_url=http://:6080/vnc_auto.html --vncserver_proxyclient_address= --vncserver_listen= --vncserver_listen= --network_manager=nova.network.manager.FlatDHCPManager --public_interface=eth0 --flat_interface=eth1 --flat_network_bridge=br100 --fixed_range=192.xxx.y.1/27 --floating_range=/27 --network_size=32 --flat_network_dhcp_start=192.xxx.y.33 --flat_injected=False --force_dhcp_release=True --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose

nova.conf on nc host:
--dhcpbridge_flagfile=/etc/nova/nova.conf --my_ip= --dhcpbridge=/usr/bin/nova-dhcpbridge --logdir=/var/log/nova --state_path=/var/lib/nova --lock_path=/run/lock/nova --allow_admin_api=true --use_deprecated_auth=false --auth_strategy=keystone --scheduler_driver=nova.scheduler.simple.SimpleScheduler --max_cores=106 --s3_host= --ec2_host= --rabbit_host= --cc_host= --nova_url=http://:8774/v1.1/ --routing_source_ip= --glance_api_servers=:9292 --image_service=nova.image.glance.GlanceImageService --iscsi_ip_prefix=192.xxx.y --sql_connection=mysql://nova:cyient#3@/nova --ec2_url=http://:8773/services/Cloud --keystone_ec2_url=http://:5000/v2.0/ec2tokens --api_paste_config=/etc/nova/api-paste.ini --libvirt_type=kvm --libvirt_use_virtio_for_bridges=true --start_guests_on_host_boot=true --resume_guests_state_on_host_boot=true --novnc_enabled=true --novncproxy_base_url=http://:6080/vnc_auto.html --vncserver_proxyclient_address= --vncserver_listen= --network_manager=nova.network.manager.FlatDHCPManager --public_interface=eth0 --flat_interface=eth1 --flat_network_bridge=br100 --fixed_range=192.xxx.y.1/27 --floating_range=/27 --network_size=32 --flat_network_dhcp_start=192.xxx.y.33 --flat_injected=false --force_dhcp_release=true --iscsi_helper=tgtadm --connection_type=libvirt --root_helper=sudo nova-rootwrap --verbose

Update: Got the above two errors fixed by below explained steps

  1. In cc host, I made the flags start_guests_on_host_boot=false and resume_guests_state_on_host_boot.
  2. Rebooted the cc and nc hosts.
  3. After reboot, all the nova services displayed :-).
  4. Deleted the records from mysql db whose deleted status is 1.
  5. Changed the status of the each and every vm to mysql: update instances set vm_state='active', task_state = NULL, power_state = 1 where uuid = INSTANCE_UUID
  6. Rebooted each and every vm using the command #nova reboot --hard INSTANCE_UUID
  7. This solved my problem and got all my VM up and running.
  8. Restored the above two boolean flags to true and rebooted cc host.
  9. After reboot, all nova services were :-) and all VMs are up and running.
  • When , where, what, why and how would be useful – Sum1sAdmin May 17 '16 at 8:18
  • I am trying to setup openstack essex on ubuntu 12.04. So when I restart my nova services, i get XXX for nova compute. And in logs, I found error saying "libvirtError:Unable to allow access for disk path /dev/disk/by-path/ip-<controller>:3260-iscsi-iqn.2010-10.org.openstack:volume-00000017-lun-1:No such file or directory" – Dilip Kumar May 19 '16 at 13:06
  • that's version 4 of openstack, version 13 is out now – Sum1sAdmin May 19 '16 at 13:26
  • Yeah, but my lead wants to set it up with essex only for some reason. – Dilip Kumar May 19 '16 at 13:30
  • well, you haven't supplied enough information to troubleshoot – Sum1sAdmin May 19 '16 at 13:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.