URGENT! OST broken! please stop merging patches to ovirt-engine!
We have two unresolved issues that are currently causing all master OST runs to fail and preventing us from effectively finding regressions. The 1st issue, which was already reported on Sunday, is a regression that is causing vm_run to fail and was probably introduced by this patch: https://gerrit.ovirt.org/#/c/79033/41 Here is a recent run that failed with this: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/ Logs are here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/artifact/... The 2nd issue is a new one, that seems to be causing add-host failures and I've just send another email about it. The gist of this is that I'm siing the following failures in the supervdsm logs, and vdsm is not loading: MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None) This failure does not seem to have been introduced by the platform and not oVirt code, because it also happens on code that already passed OST a few days ago. Please avoid merging any patches except to the purpose of resolving these issues. Nothing is making it into the 'tested' and nightly snapshot repos anyway ATM. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
On Wed, Aug 30, 2017 at 7:13 AM, Barak Korren <bkorren@redhat.com> wrote:
We have two unresolved issues that are currently causing all master OST runs to fail and preventing us from effectively finding regressions.
The 1st issue, which was already reported on Sunday, is a regression that is causing vm_run to fail and was probably introduced by this patch: https://gerrit.ovirt.org/#/c/79033/41
cc'd Shubham and Maor. Is anyone looking into this?
Here is a recent run that failed with this: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here: http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/2147/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/
The 2nd issue is a new one, that seems to be causing add-host failures and I've just send another email about it. The gist of this is that I'm siing the following failures in the supervdsm logs, and vdsm is not loading:
MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and not oVirt code, because it also happens on code that already passed OST a few days ago.
Please avoid merging any patches except to the purpose of resolving these issues. Nothing is making it into the 'tested' and nightly snapshot repos anyway ATM.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
Yes, there is already an acked patch that solves it. The weird thing is that the OST ran successfully when running with "ci please build" although it seems that the artifact was not added to it. On Wed, Aug 30, 2017 at 3:17 PM, Greg Sheremeta <gshereme@redhat.com> wrote:
On Wed, Aug 30, 2017 at 7:13 AM, Barak Korren <bkorren@redhat.com> wrote:
We have two unresolved issues that are currently causing all master OST runs to fail and preventing us from effectively finding regressions.
The 1st issue, which was already reported on Sunday, is a regression that is causing vm_run to fail and was probably introduced by this patch: https://gerrit.ovirt.org/#/c/79033/41
cc'd Shubham and Maor. Is anyone looking into this?
Here is a recent run that failed with this: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/artifact/...
The 2nd issue is a new one, that seems to be causing add-host failures and I've just send another email about it. The gist of this is that I'm siing the following failures in the supervdsm logs, and vdsm is not loading:
MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and not oVirt code, because it also happens on code that already passed OST a few days ago.
Please avoid merging any patches except to the purpose of resolving these issues. Nothing is making it into the 'tested' and nightly snapshot repos anyway ATM.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
The relevant path is most definitely the cause for Sunday's issue: 2017-08-30 04:38:52,542-04 ERROR [org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-18) [5c4b4916-9832-4460-a391-cac53ef8f19a] Error during ValidateFailure.: java.lang.ClassCastException: org.ovirt.engine.core.common.businessentities.storage.LunDisk cannot be cast to org.ovirt.engine.core.common.businessentities.storage.DiskImage at org.ovirt.engine.core.bll.RunVmCommand.lambda$checkDisksInBackupStorage$1(RunVmCommand.java:1105) [bll.jar:] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) [rt.jar:1.8.0_141] at java.util.HashMap$ValueSpliterator.tryAdvance(HashMap.java:1641) [rt.jar:1.8.0_141] at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) [rt.jar:1.8.0_141] at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498) [rt.jar:1.8.0_141] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485) [rt.jar:1.8.0_141] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) [rt.jar:1.8.0_141] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) [rt.jar:1.8.0_141] at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) [rt.jar:1.8.0_141] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) [rt.jar:1.8.0_141] at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449) [rt.jar:1.8.0_141] at org.ovirt.engine.core.bll.RunVmCommand.checkDisksInBackupStorage(RunVmCommand.java:1106) [bll.jar:] at org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:1020) [bll.jar:] at org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:87) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:848) [bll.jar:] at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:402) [bll.jar:] at org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:499) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:481) [bll.jar:] at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:434) [bll.jar:] On Wed, Aug 30, 2017 at 2:13 PM, Barak Korren <bkorren@redhat.com> wrote:
We have two unresolved issues that are currently causing all master OST runs to fail and preventing us from effectively finding regressions.
The 1st issue, which was already reported on Sunday, is a regression that is causing vm_run to fail and was probably introduced by this patch: https://gerrit.ovirt.org/#/c/79033/41
Here is a recent run that failed with this: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here: http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/2147/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-002_bootstrap.py/
The 2nd issue is a new one, that seems to be causing add-host failures and I've just send another email about it. The gist of this is that I'm siing the following failures in the supervdsm logs, and vdsm is not loading:
MainThread::ERROR::2017-08-30 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable LLDP on eth1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py", line 51, in _lldp_init Lldp.enable_lldp_on_iface(device) File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py", line 30, in enable_lldp_on_iface lldptool.enable_lldp_on_iface(iface, rx_only) File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py", line 46, in enable_lldp_on_iface raise EnableLldpError(rc, out, err, iface) EnableLldpError: (1, "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command timed out.\n", '', 'eth1') MainThread::DEBUG::2017-08-30 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and not oVirt code, because it also happens on code that already passed OST a few days ago.
Please avoid merging any patches except to the purpose of resolving these issues. Nothing is making it into the 'tested' and nightly snapshot repos anyway ATM.
-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel
participants (4)
-
Allon Mureinik -
Barak Korren -
Greg Sheremeta -
Maor Lipchuk