[lago-devel] Any ideas for root cause of this failure?

[Yaniv Kaul]

Running Lago/ovirt-system-tests on F24:

  # [Thread-1] Bootstrapping lago_basic_suite_3_6_engine:
  # [Thread-2] Bootstrapping lago_basic_suite_3_6_host1:
  # [Thread-3] Bootstrapping lago_basic_suite_3_6_host0:
  # [Thread-4] Bootstrapping lago_basic_suite_3_6_storage:
  # [Thread-1] Bootstrapping lago_basic_suite_3_6_engine: ERROR (in 0:00:04)
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in
_ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda>
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
265, in bootstrap
    return self.provider.bootstrap(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap
    if 'ip' in nic
  File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep
    ret.err,
RuntimeError: Failed to bootstrap
$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2
command:"virt-sysprep" "-a"
"$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2"
"--selinux-relabel" "--hostname" "lago_basic_suite_3_6_engine"
"--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod"
"0700:/root/.ssh" "--upload"
"/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys"
"--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir"
"/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write"
"/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine"
"--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write"
"/etc/selinux/config:SELINUX=enforcing
SELINUXTYPE=targeted
" "--mkdir" "/etc/sysconfig/network-scripts" "--chmod"
"0755:/etc/sysconfig/network-scripts" "--write"
"/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:02"
BOOTPROTO="dhcp"
TYPE="Ethernet"
ONBOOT="yes"
NAME="eth0""
stdout:[   0.0] Examining the guest ...

stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]

  # [Thread-4] Bootstrapping lago_basic_suite_3_6_storage: ERROR (in 0:00:04)
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in
_ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda>
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
265, in bootstrap
    return self.provider.bootstrap(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap
    if 'ip' in nic
  File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep
    ret.err,
RuntimeError: Failed to bootstrap
$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_storage_root.qcow2
command:"virt-sysprep" "-a"
"$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_storage_root.qcow2"
"--selinux-relabel" "--hostname" "lago_basic_suite_3_6_storage"
"--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod"
"0700:/root/.ssh" "--upload"
"/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys"
"--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir"
"/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write"
"/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_storage"
"--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write"
"/etc/selinux/config:SELINUX=enforcing
SELINUXTYPE=targeted
" "--mkdir" "/etc/sysconfig/network-scripts" "--chmod"
"0755:/etc/sysconfig/network-scripts" "--write"
"/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:05"
BOOTPROTO="dhcp"
TYPE="Ethernet"
ONBOOT="yes"
NAME="eth0""
stdout:[   0.0] Examining the guest ...

stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]

  # [Thread-2] Bootstrapping lago_basic_suite_3_6_host1: ERROR (in 0:00:04)
  # [Thread-3] Bootstrapping lago_basic_suite_3_6_host0: ERROR (in 0:00:04)
Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in
_ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda>
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
265, in bootstrap
    return self.provider.bootstrap(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap
    if 'ip' in nic
  File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep
    ret.err,
RuntimeError: Failed to bootstrap
$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host1_root.qcow2
command:"virt-sysprep" "-a"
"$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host1_root.qcow2"
"--selinux-relabel" "--hostname" "lago_basic_suite_3_6_host1"
"--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod"
"0700:/root/.ssh" "--upload"
"/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys"
"--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir"
"/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write"
"/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_host1"
"--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write"
"/etc/selinux/config:SELINUX=enforcing
SELINUXTYPE=targeted
" "--mkdir" "/etc/sysconfig/network-scripts" "--chmod"
"0755:/etc/sysconfig/network-scripts" "--write"
"/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:03"
BOOTPROTO="dhcp"
TYPE="Ethernet"
ONBOOT="yes"
NAME="eth0""
stdout:[   0.0] Examining the guest ...

stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]

Error while running thread
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in
_ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda>
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
265, in bootstrap
    return self.provider.bootstrap(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap
    if 'ip' in nic
  File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep
    ret.err,
RuntimeError: Failed to bootstrap
$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host0_root.qcow2
command:"virt-sysprep" "-a"
"$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_host0_root.qcow2"
"--selinux-relabel" "--hostname" "lago_basic_suite_3_6_host0"
"--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod"
"0700:/root/.ssh" "--upload"
"/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys"
"--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir"
"/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write"
"/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_host0"
"--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write"
"/etc/selinux/config:SELINUX=enforcing
SELINUXTYPE=targeted
" "--mkdir" "/etc/sysconfig/network-scripts" "--chmod"
"0755:/etc/sysconfig/network-scripts" "--write"
"/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:04"
BOOTPROTO="dhcp"
TYPE="Ethernet"
ONBOOT="yes"
NAME="eth0""
stdout:[   0.0] Examining the guest ...

stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]

@ Initialize and populate prefix: ERROR (in 0:00:08)
Error occured, aborting
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 691, in main
    cli_plugins[args.verb].do_run(args)
  File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line
180, in do_run
    self._do_run(**vars(args))
  File "/usr/lib/python2.7/site-packages/lago/log_utils.py", line 615,
in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/cmd.py", line 181, in do_init
    do_bootstrap=not skip_bootstrap,
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 816, in
virt_conf_from_stream
    do_bootstrap=do_bootstrap,
  File "/usr/lib/python2.7/site-packages/lago/prefix.py", line 943, in virt_conf
    self.virt_env.bootstrap()
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in bootstrap
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 95, in
invoke_in_parallel
    vt.join_all()
  File "/usr/lib/python2.7/site-packages/lago/utils.py", line 53, in
_ret_via_queue
    queue.put({'return': func()})
  File "/usr/lib/python2.7/site-packages/lago/virt.py", line 156, in <lambda>
    utils.invoke_in_parallel(lambda vm: vm.bootstrap(), self._vms.values())
  File "/usr/lib/python2.7/site-packages/lago/plugins/vm.py", line
265, in bootstrap
    return self.provider.bootstrap(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/lago/vm.py", line 128, in bootstrap
    if 'ip' in nic
  File "/usr/lib/python2.7/site-packages/lago/sysprep.py", line 134, in sysprep
    ret.err,
RuntimeError: Failed to bootstrap
$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2
command:"virt-sysprep" "-a"
"$LAGO_PREFIX_PATH/images/lago_basic_suite_3_6_engine_root.qcow2"
"--selinux-relabel" "--hostname" "lago_basic_suite_3_6_engine"
"--root-password" "password:123456" "--mkdir" "/root/.ssh" "--chmod"
"0700:/root/.ssh" "--upload"
"/home/omer/ovirt-system-tests/deployment-basic_suite_3.6/default/id_rsa.pub:/root/.ssh/authorized_keys"
"--run-command" "chown root.root /root/.ssh/authorized_keys" "--mkdir"
"/etc/iscsi" "--chmod" "0755:/etc/iscsi" "--write"
"/etc/iscsi/initiatorname.iscsi:InitiatorName=iqn.2014-07.org.lago:lago_basic_suite_3_6_engine"
"--mkdir" "/etc/selinux" "--chmod" "0755:/etc/selinux" "--write"
"/etc/selinux/config:SELINUX=enforcing
SELINUXTYPE=targeted
" "--mkdir" "/etc/sysconfig/network-scripts" "--chmod"
"0755:/etc/sysconfig/network-scripts" "--write"
"/etc/sysconfig/network-scripts/ifcfg-eth0:HWADDR="54:52:c0:a8:c8:02"
BOOTPROTO="dhcp"
TYPE="Ethernet"
ONBOOT="yes"
NAME="eth0""
stdout:[   0.0] Examining the guest ...

stderr:virt-sysprep: error: libguestfs error: guestfs_launch failed.
This usually means the libguestfs appliance failed to start or crashed.
Do:
  export LIBGUESTFS_DEBUG=1 LIBGUESTFS_TRACE=1
and run the command again.  For further information, read:
  http://libguestfs.org/guestfs-faq.1.html#debugging-libguestfs
You can also run 'libguestfs-test-tool' and post the *complete* output
into a bug report or message to the libguestfs mailing list.

If reporting bugs, run virt-sysprep with debugging enabled and include the
complete output:

  virt-sysprep -v -x [...]



lago-error.txt
Open with Google Docs
Displaying lago-error.txt.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/lago-devel/attachments/20160825/8ac8d1d1/attachment-0001.html>