we have more failures with this error.

I checked latest failure and guest-agent is installed.
logs can be found here for latest failed job:

http://jenkins.ovirt.org/job/ovirt-4.3_change-queue-tester/371/artifact/basic-suite.el7.x86_64/test_logs/basic-suite-4.3/post-004_basic_sanity.py
2019-03-25 14:57:05,176::INFO::repoman.common.stores.RPM::Adding package /var/lib/lago/ovirt-4.3-snapshot-static-el7/noarch/ovirt-guest-agent-common-1.0.16-1.el7.noarch.rpm to repo Non persistent RPMStore

On Fri, Mar 22, 2019 at 7:52 PM Nir Soffer <nsoffer@redhat.com> wrote:
On Fri, Mar 22, 2019 at 7:12 PM Dafna Ron <dron@redhat.com> wrote:
Hi,

We are failing ovirt-engine master on test 004_basic_sanity.hotplug_cpu  
looking at the logs, we can see that the for some reason, libvirt reports a vm as none responsive which fails the test.

CQ first failure was for patch:
https://gerrit.ovirt.org/#/c/98553/ - core: Add display="on" for mdevs, use nodisplay to override
But I do not think this is the cause of failure.

Adding Marcin, Milan and Dan as well as I think it may be netwrok related.

You can see the libvirt log here:

you can see the full logs here:


Evgheni and I confirmed this is not an infra issue and the problem is ssh connection to the internal vm

Thanks,
Dafna


error: 
2019-03-22 15:08:22.658+0000: 22068: warning : qemuDomainObjTaint:7521 : Domain id=3 name='vm0' uuid=a9443d02-e054-40bb-8ea3-ae346e2d02a7 is tainted: hook-script
Why our vm is tainted?

2019-03-22 15:08:22.693+0000: 22068: error : virProcessRunInMountNamespace:1159 : internal error: child reported: unable to set security context 'system_u:object_r:virt_content_t:s0' on '/rhev/data-center/mnt/blockSD/91d97292-9ac3-4d77-a152-c7ea3250b065/images/e60dae48-ecc7-4171-8bfe-42bfc2190ffd/40243c76-a384-4497-8a2d-792a5e10d510': No such file or directory
This should not happen, libvirt is not adding labels to files in /rhev/data-center. It is using using its own mount
namespace and adding there the devices used by the VM. Since libvirt create the devices in its namespace
it should not complain about missing paths in /rhev/data-center.

I think we should file a libvirt bug for this.
 
2019-03-22 15:08:28.168+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
2019-03-22 15:08:58.193+0000: 22070: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
2019-03-22 15:13:58.179+0000: 22071: error : qemuDomainAgentAvailable:9133 : Guest agent is not responding: QEMU guest agent is not connected
Do we have guest agent in the test VMs?

Nir