[lago-devel] Any ideas for root cause of this failure?

Running Lago/ovirt-system-tests on F24: # [Thread-1] Bootstrapping lago_basic_suite_3_6_engine: # [Thread-2] Bootstrapping lago_basic_suite_3_6_host1: # [Thread-3] Bootstrapping lago_basic_suite_3_6_host0: # [Thread-4] Bootstrapping lago_basic_suite_3_6_storage: # [Thread-1] Bootstrapping lago_basic_suite_3_6_engine: ERROR (in 0:00:04) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda> utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 265, in bootstrap return self.provider.bootstrap(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap if 'ip' in nic File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep ret.err, RuntimeError: Failed to bootstrap $LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2 command:"virt-sysprep" "-a" "$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2" "--selinux-relabel" "--hostname" "lago_basic_suite_3_6_engine" "--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod" "0700:/root/.ssh" "--upload" "/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys" "--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir" "/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write" "/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine" "--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write" "/etc/selinux/config:SELINUX=enforcing SELINUXTYPE=targeted " "--mkdir" "/etc/sysconfig/network-scripts" "--chmod" "0755:/etc/sysconfig/network-scripts" "--write" "/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:02" BOOTPROTO="dhcp" TYPE="Ethernet" ONBOOT="yes" NAME="eth0"" stdout:[ 0.0] Examining the guest ... stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed. This usually means the libguestfs appliance failed to start or crashed. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] # [Thread-4] Bootstrapping lago_basic_suite_3_6_storage: ERROR (in 0:00:04) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda> utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 265, in bootstrap return self.provider.bootstrap(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap if 'ip' in nic File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep ret.err, RuntimeError: Failed to bootstrap $LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_storage_root.qcow2 command:"virt-sysprep" "-a" "$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_storage_root.qcow2" "--selinux-relabel" "--hostname" "lago_basic_suite_3_6_storage" "--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod" "0700:/root/.ssh" "--upload" "/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys" "--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir" "/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write" "/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_storage" "--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write" "/etc/selinux/config:SELINUX=enforcing SELINUXTYPE=targeted " "--mkdir" "/etc/sysconfig/network-scripts" "--chmod" "0755:/etc/sysconfig/network-scripts" "--write" "/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:05" BOOTPROTO="dhcp" TYPE="Ethernet" ONBOOT="yes" NAME="eth0"" stdout:[ 0.0] Examining the guest ... stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed. This usually means the libguestfs appliance failed to start or crashed. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] # [Thread-2] Bootstrapping lago_basic_suite_3_6_host1: ERROR (in 0:00:04) # [Thread-3] Bootstrapping lago_basic_suite_3_6_host0: ERROR (in 0:00:04) Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda> utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 265, in bootstrap return self.provider.bootstrap(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap if 'ip' in nic File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep ret.err, RuntimeError: Failed to bootstrap $LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host1_root.qcow2 command:"virt-sysprep" "-a" "$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host1_root.qcow2" "--selinux-relabel" "--hostname" "lago_basic_suite_3_6_host1" "--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod" "0700:/root/.ssh" "--upload" "/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys" "--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir" "/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write" "/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_host1" "--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write" "/etc/selinux/config:SELINUX=enforcing SELINUXTYPE=targeted " "--mkdir" "/etc/sysconfig/network-scripts" "--chmod" "0755:/etc/sysconfig/network-scripts" "--write" "/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:03" BOOTPROTO="dhcp" TYPE="Ethernet" ONBOOT="yes" NAME="eth0"" stdout:[ 0.0] Examining the guest ... stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed. This usually means the libguestfs appliance failed to start or crashed. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] Error while running thread Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda> utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 265, in bootstrap return self.provider.bootstrap(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap if 'ip' in nic File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep ret.err, RuntimeError: Failed to bootstrap $LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host0_root.qcow2 command:"virt-sysprep" "-a" "$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host0_root.qcow2" "--selinux-relabel" "--hostname" "lago_basic_suite_3_6_host0" "--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod" "0700:/root/.ssh" "--upload" "/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys" "--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir" "/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write" "/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_host0" "--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write" "/etc/selinux/config:SELINUX=enforcing SELINUXTYPE=targeted " "--mkdir" "/etc/sysconfig/network-scripts" "--chmod" "0755:/etc/sysconfig/network-scripts" "--write" "/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:04" BOOTPROTO="dhcp" TYPE="Ethernet" ONBOOT="yes" NAME="eth0"" stdout:[ 0.0] Examining the guest ... stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed. This usually means the libguestfs appliance failed to start or crashed. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] @ Initialize and populate prefix: ERROR (in 0:00:08) Error occured, aborting Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 691, in main cli_plugins[args.verb].do_run(args) File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 180, in do_run self._do_run(**vars(args)) File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615, in wrapper return func(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 181, in do_init do_bootstrap=not skip_bootstrap, File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 816, in virt_conf_from_stream do_bootstrap=do_bootstrap, File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 943, in virt_conf self.virt_env.bootstrap() File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in bootstrap utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in invoke_in_parallel vt.join_all() File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in _ret_via_queue queue.put({'return': func()}) File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda> utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values()) File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line 265, in bootstrap return self.provider.bootstrap(*args, **kwargs) File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap if 'ip' in nic File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep ret.err, RuntimeError: Failed to bootstrap $LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2 command:"virt-sysprep" "-a" "$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2" "--selinux-relabel" "--hostname" "lago_basic_suite_3_6_engine" "--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod" "0700:/root/.ssh" "--upload" "/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys" "--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir" "/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write" "/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine" "--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write" "/etc/selinux/config:SELINUX=enforcing SELINUXTYPE=targeted " "--mkdir" "/etc/sysconfig/network-scripts" "--chmod" "0755:/etc/sysconfig/network-scripts" "--write" "/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:02" BOOTPROTO="dhcp" TYPE="Ethernet" ONBOOT="yes" NAME="eth0"" stdout:[ 0.0] Examining the guest ... stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed. This usually means the libguestfs appliance failed to start or crashed. Do: export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1 and run the command again. For further information, read: http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs You can also run 'libguestfs-test-tool' and post the *complete* output into a bug report or message to the libguestfs mailing list. If reporting bugs, run virt-sysprep with debugging enabled and include the complete output: virt-sysprep -v -x [...] lago-error.txt Open with Google Docs Displaying lago-error.txt.