Subject: [ OST Failure Report ] [ oVirt Master ] [ 15 Jan 2018 ] [ 004_basic_sanity.vm_run ]
Hi, We had a failure in OST on the upgrade suite. *Link and headline of suspected patches: Reported as the cause: https://gerrit.ovirt.org/#/c/86115/4 - <https://gerrit.ovirt.org/#/c/86115/4>vm: support error_policy='report' for CDROMs*Reported* as the root cause: *https://gerrit.ovirt.org/#/c/86114/4 net tests: Fix vlan creation name length in nettestlib <https://gerrit.ovirt.org/#/c/86114/4> * <https://gerrit.ovirt.org/#/c/86114/4>* *Link to Job:* *http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/>(Relevant) error snippet from the log: <error>* *vdsm: *type=file, path= threshold=unset at 0x7f57d8027d40>> watermarkLimit:536870912 (vm:2134) 2018-01-15 13:12:06,126-0500 ERROR (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') The vm start process failed (vm:917) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 846, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2735, in _run domxml = hooks.before_vm_start(self._buildDomainXML(), File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2191, in _buildDomainXML return self._make_domain_xml() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2236, in _make_domain_xml devices_xml = self._process_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2116, in _process_devices dev_xml = dev.getXML() File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 595, in getXML diskelem.appendChild(_getDriverXML(self)) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 957, in _getDriverXML if (drive['propagateErrors'] == 'on' or File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 292, in __getitem__ raise KeyError(key) KeyError: 'propagateErrors' 2018-01-15 13:12:06,129-0500 INFO (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') Changed state to Down: 'propagateErrors' (code=1) (vm:1636) engine: *2018-01-15 13:12:07,203-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-15) [] Rerun VM '75adb67b-4390-45f7-a677-8d4515cf4a2b'. Called from VDS 'lago-upgrade-from-release-suite-master-host0'2018-01-15 13:12:07,205-05 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-15) [] VMs initialization finished for Host: 'lago-upgrade-from-release-suite-master-host0:c98b0719-383b-4656-a28d-03005e2fb862'2018-01-15 13:12:07,256-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-19) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM vm0 on Host lago-upgrade-from-release-suite-master-host0.2018-01-15 13:12:07,274-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[75adb67b-4390-45f7-a677-8d4515cf4a2b=VM]', sharedLocks=''}'2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b'}), log id: 41c281dc2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 41c281dc2018-01-15 13:12:07,300-05 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS</error>*
We had a second failure on the same root cause patch. Dan, can you please have a look? On Tue, Jan 16, 2018 at 10:50 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We had a failure in OST on the upgrade suite.
*Link and headline of suspected patches: Reported as the cause: https://gerrit.ovirt.org/#/c/86115/4 - <https://gerrit.ovirt.org/#/c/86115/4>vm: support error_policy='report' for CDROMs*Reported* as the root cause: *https://gerrit.ovirt.org/#/c/86114/4 net tests: Fix vlan creation name length in nettestlib <https://gerrit.ovirt.org/#/c/86114/4> * <https://gerrit.ovirt.org/#/c/86114/4>*
*Link to Job:*
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/>(Relevant) error snippet from the log: <error>*
*vdsm: *type=file, path= threshold=unset at 0x7f57d8027d40>> watermarkLimit:536870912 (vm:2134) 2018-01-15 13:12:06,126-0500 ERROR (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') The vm start process failed (vm:917) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 846, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2735, in _run domxml = hooks.before_vm_start(self._buildDomainXML(), File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2191, in _buildDomainXML return self._make_domain_xml() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2236, in _make_domain_xml devices_xml = self._process_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2116, in _process_devices dev_xml = dev.getXML() File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 595, in getXML diskelem.appendChild(_getDriverXML(self)) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 957, in _getDriverXML if (drive['propagateErrors'] == 'on' or File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 292, in __getitem__ raise KeyError(key) KeyError: 'propagateErrors' 2018-01-15 13:12:06,129-0500 INFO (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') Changed state to Down: 'propagateErrors' (code=1) (vm:1636)
engine:
*2018-01-15 13:12:07,203-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-15) [] Rerun VM '75adb67b-4390-45f7-a677-8d4515cf4a2b'. Called from VDS 'lago-upgrade-from-release-suite-master-host0'2018-01-15 13:12:07,205-05 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-15) [] VMs initialization finished for Host: 'lago-upgrade-from-release-suite-master-host0:c98b0719-383b-4656-a28d-03005e2fb862'2018-01-15 13:12:07,256-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-19) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM vm0 on Host lago-upgrade-from-release-suite-master-host0.2018-01-15 13:12:07,274-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[75adb67b-4390-45f7-a677-8d4515cf4a2b=VM]', sharedLocks=''}'2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b'}), log id: 41c281dc2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 41c281dc2018-01-15 13:12:07,300-05 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS</error>*
This is a real error but it already fixed in master. Please update vdsm. בתאריך יום ג׳, 16 בינו׳ 2018, 12:57, מאת Dafna Ron <dron@redhat.com>:
We had a second failure on the same root cause patch. Dan, can you please have a look?
On Tue, Jan 16, 2018 at 10:50 AM, Dafna Ron <dron@redhat.com> wrote:
Hi,
We had a failure in OST on the upgrade suite.
*Link and headline of suspected patches: Reported as the cause: https://gerrit.ovirt.org/#/c/86115/4 - <https://gerrit.ovirt.org/#/c/86115/4>vm: support error_policy='report' for CDROMs*Reported* as the root cause: *https://gerrit.ovirt.org/#/c/86114/4 net tests: Fix vlan creation name length in nettestlib <https://gerrit.ovirt.org/#/c/86114/4> * <https://gerrit.ovirt.org/#/c/86114/4>*
*Link to Job:*
*http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/>Link to all logs:http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/ <http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4854/artifact/>(Relevant) error snippet from the log: <error>*
*vdsm: *type=file, path= threshold=unset at 0x7f57d8027d40>> watermarkLimit:536870912 (vm:2134) 2018-01-15 13:12:06,126-0500 ERROR (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') The vm start process failed (vm:917) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 846, in _startUnderlyingVm self._run() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2735, in _run domxml = hooks.before_vm_start(self._buildDomainXML(), File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2191, in _buildDomainXML return self._make_domain_xml() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2236, in _make_domain_xml devices_xml = self._process_devices() File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 2116, in _process_devices dev_xml = dev.getXML() File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 595, in getXML diskelem.appendChild(_getDriverXML(self)) File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 957, in _getDriverXML if (drive['propagateErrors'] == 'on' or File "/usr/lib/python2.7/site-packages/vdsm/virt/vmdevices/storage.py", line 292, in __getitem__ raise KeyError(key) KeyError: 'propagateErrors' 2018-01-15 13:12:06,129-0500 INFO (vm/75adb67b) [virt.vm] (vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b') Changed state to Down: 'propagateErrors' (code=1) (vm:1636)
engine:
*2018-01-15 13:12:07,203-05 ERROR [org.ovirt.engine.core.vdsbroker.monitoring.VmsMonitoring] (ForkJoinPool-1-worker-15) [] Rerun VM '75adb67b-4390-45f7-a677-8d4515cf4a2b'. Called from VDS 'lago-upgrade-from-release-suite-master-host0'2018-01-15 13:12:07,205-05 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ForkJoinPool-1-worker-15) [] VMs initialization finished for Host: 'lago-upgrade-from-release-suite-master-host0:c98b0719-383b-4656-a28d-03005e2fb862'2018-01-15 13:12:07,256-05 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-19) [] EVENT_ID: USER_INITIATED_RUN_VM_FAILED(151), Failed to run VM vm0 on Host lago-upgrade-from-release-suite-master-host0.2018-01-15 13:12:07,274-05 INFO [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Lock Acquired to object 'EngineLock:{exclusiveLocks='[75adb67b-4390-45f7-a677-8d4515cf4a2b=VM]', sharedLocks=''}'2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] START, IsVmDuringInitiatingVDSCommand( IsVmDuringInitiatingVDSCommandParameters:{vmId='75adb67b-4390-45f7-a677-8d4515cf4a2b'}), log id: 41c281dc2018-01-15 13:12:07,294-05 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 41c281dc2018-01-15 13:12:07,300-05 WARN [org.ovirt.engine.core.bll.RunVmOnceCommand] (EE-ManagedThreadFactory-engine-Thread-19) [] Validation of action 'RunVmOnce' failed for user admin@internal-authz. Reasons: VAR__ACTION__RUN,VAR__TYPE__VM,VAR__ACTION__RUN,VAR__TYPE__VM,SCHEDULING_NO_HOSTS</error>*
_______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (2)
-
Dafna Ron -
Nir Soffer