1:13 p.m.
We have two unresolved issues that are currently causing all master
OST runs to fail and preventing us from effectively finding
regressions.
The 1st issue, which was already reported on Sunday, is a regression
that is causing vm_run to fail and was probably introduced by this
patch:
https://gerrit.ovirt.org/#/c/79033/41
Here is a recent run that failed with this:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/artifa...
The 2nd issue is a new one, that seems to be causing add-host failures
and I've just send another email about it. The gist of this is that
I'm siing the following failures in the supervdsm logs, and vdsm is
not loading:
MainThread::ERROR::2017-08-30
05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable
LLDP on eth1
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py",
line 51, in _lldp_init
Lldp.enable_lldp_on_iface(device)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py",
line 30, in enable_lldp_on_iface
lldptool.enable_lldp_on_iface(iface, rx_only)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py",
line 46, in enable_lldp_on_iface
raise EnableLldpError(rc, out, err, iface)
EnableLldpError: (1,
"timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command
timed out.\n", '', 'eth1')
MainThread::DEBUG::2017-08-30
05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool
get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and
not oVirt code, because it also happens on code that already passed
OST a few days ago.
Please avoid merging any patches except to the purpose of resolving
these issues. Nothing is making it into the 'tested' and nightly
snapshot repos anyway ATM.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
2:17 p.m.
On Wed, Aug 30, 2017 at 7:13 AM, Barak Korren <bkorren(a)redhat.com> wrote:
We have two unresolved issues that are currently causing all master
OST runs to fail and preventing us from effectively finding
regressions.
The 1st issue, which was already reported on Sunday, is a regression
that is causing vm_run to fail and was probably introduced by this
patch:
https://gerrit.ovirt.org/#/c/79033/41
cc'd Shubham and Maor. Is anyone looking into this?
Here is a recent run that failed with this:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/2147/artifact/exported-artifacts/basic-suit-master-
el7/test_logs/basic-suite-master/post-002_bootstrap.py/
The 2nd issue is a new one, that seems to be causing add-host failures
and I've just send another email about it. The gist of this is that
I'm siing the following failures in the supervdsm logs, and vdsm is
not loading:
MainThread::ERROR::2017-08-30
05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable
LLDP on eth1
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py",
line 51, in _lldp_init
Lldp.enable_lldp_on_iface(device)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py",
line 30, in enable_lldp_on_iface
lldptool.enable_lldp_on_iface(iface, rx_only)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py",
line 46, in enable_lldp_on_iface
raise EnableLldpError(rc, out, err, iface)
EnableLldpError: (1,
"timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command
timed out.\n", '', 'eth1')
MainThread::DEBUG::2017-08-30
05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool
get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and
not oVirt code, because it also happens on code that already passed
OST a few days ago.
Please avoid merging any patches except to the purpose of resolving
these issues. Nothing is making it into the 'tested' and nightly
snapshot repos anyway ATM.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2:46 p.m.
Yes, there is already an acked patch that solves it.
The weird thing is that the OST ran successfully when running with "ci
please build" although it seems that the artifact was not added to it.
On Wed, Aug 30, 2017 at 3:17 PM, Greg Sheremeta <gshereme(a)redhat.com> wrote:
On Wed, Aug 30, 2017 at 7:13 AM, Barak Korren <bkorren(a)redhat.com> wrote:
>
> We have two unresolved issues that are currently causing all master
> OST runs to fail and preventing us from effectively finding
> regressions.
>
> The 1st issue, which was already reported on Sunday, is a regression
> that is causing vm_run to fail and was probably introduced by this
> patch:
> https://gerrit.ovirt.org/#/c/79033/41
cc'd Shubham and Maor. Is anyone looking into this?
>
>
>
> Here is a recent run that failed with this:
> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
>
> Logs are here:
>
>
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/artifa...
>
> The 2nd issue is a new one, that seems to be causing add-host failures
> and I've just send another email about it. The gist of this is that
> I'm siing the following failures in the supervdsm logs, and vdsm is
> not loading:
>
> MainThread::ERROR::2017-08-30
> 05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable
> LLDP on eth1
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py",
> line 51, in _lldp_init
> Lldp.enable_lldp_on_iface(device)
> File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py",
> line 30, in enable_lldp_on_iface
> lldptool.enable_lldp_on_iface(iface, rx_only)
> File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py",
> line 46, in enable_lldp_on_iface
> raise EnableLldpError(rc, out, err, iface)
> EnableLldpError: (1,
> "timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command
> timed out.\n", '', 'eth1')
> MainThread::DEBUG::2017-08-30
> 05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool
> get-lldp -i eth0 adminStatus (cwd None)
>
>
> This failure does not seem to have been introduced by the platform and
> not oVirt code, because it also happens on code that already passed
> OST a few days ago.
>
> Please avoid merging any patches except to the purpose of resolving
> these issues. Nothing is making it into the 'tested' and nightly
> snapshot repos anyway ATM.
>
> --
> Barak Korren
> RHV DevOps team , RHCE, RHCi
> Red Hat EMEA
> redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
> _______________________________________________
> Devel mailing list
> Devel(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
2:17 p.m.
The relevant path is most definitely the cause for Sunday's issue:
2017-08-30 04:38:52,542-04 ERROR
[org.ovirt.engine.core.bll.RunVmOnceCommand] (default task-18)
[5c4b4916-9832-4460-a391-cac53ef8f19a] Error during ValidateFailure.:
java.lang.ClassCastException:
org.ovirt.engine.core.common.businessentities.storage.LunDisk cannot
be cast to org.ovirt.engine.core.common.businessentities.storage.DiskImage
at
org.ovirt.engine.core.bll.RunVmCommand.lambda$checkDisksInBackupStorage$1(RunVmCommand.java:1105)
[bll.jar:]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
[rt.jar:1.8.0_141]
at java.util.HashMap$ValueSpliterator.tryAdvance(HashMap.java:1641)
[rt.jar:1.8.0_141]
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
[rt.jar:1.8.0_141]
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
[rt.jar:1.8.0_141]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
[rt.jar:1.8.0_141]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
[rt.jar:1.8.0_141]
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
[rt.jar:1.8.0_141]
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
[rt.jar:1.8.0_141]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
[rt.jar:1.8.0_141]
at java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:449)
[rt.jar:1.8.0_141]
at
org.ovirt.engine.core.bll.RunVmCommand.checkDisksInBackupStorage(RunVmCommand.java:1106)
[bll.jar:]
at org.ovirt.engine.core.bll.RunVmCommand.validate(RunVmCommand.java:1020)
[bll.jar:]
at org.ovirt.engine.core.bll.RunVmOnceCommand.validate(RunVmOnceCommand.java:87)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.internalValidate(CommandBase.java:848)
[bll.jar:]
at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:402)
[bll.jar:]
at
org.ovirt.engine.core.bll.executor.DefaultBackendActionExecutor.execute(DefaultBackendActionExecutor.java:13)
[bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:499) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:481) [bll.jar:]
at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:434) [bll.jar:]
On Wed, Aug 30, 2017 at 2:13 PM, Barak Korren <bkorren(a)redhat.com> wrote:
We have two unresolved issues that are currently causing all master
OST runs to fail and preventing us from effectively finding
regressions.
The 1st issue, which was already reported on Sunday, is a regression
that is causing vm_run to fail and was probably introduced by this
patch:
https://gerrit.ovirt.org/#/c/79033/41
Here is a recent run that failed with this:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2147/
Logs are here:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-
tester/2147/artifact/exported-artifacts/basic-suit-master-
el7/test_logs/basic-suite-master/post-002_bootstrap.py/
The 2nd issue is a new one, that seems to be causing add-host failures
and I've just send another email about it. The gist of this is that
I'm siing the following failures in the supervdsm logs, and vdsm is
not loading:
MainThread::ERROR::2017-08-30
05:55:59,476::initializer::53::root::(_lldp_init) Failed to enable
LLDP on eth1
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/network/initializer.py",
line 51, in _lldp_init
Lldp.enable_lldp_on_iface(device)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldp/lldpad.py",
line 30, in enable_lldp_on_iface
lldptool.enable_lldp_on_iface(iface, rx_only)
File "/usr/lib/python2.7/site-packages/vdsm/network/lldpad/lldptool.py",
line 46, in enable_lldp_on_iface
raise EnableLldpError(rc, out, err, iface)
EnableLldpError: (1,
"timeout\n'M00000001C3040000000c04eth1000badminStatus0002rx' command
timed out.\n", '', 'eth1')
MainThread::DEBUG::2017-08-30
05:55:59,477::cmdutils::133::root::(exec_cmd) /usr/sbin/lldptool
get-lldp -i eth0 adminStatus (cwd None)
This failure does not seem to have been introduced by the platform and
not oVirt code, because it also happens on code that already passed
OST a few days ago.
Please avoid merging any patches except to the purpose of resolving
these issues. Nothing is making it into the 'tested' and nightly
snapshot repos anyway ATM.
--
Barak Korren
RHV DevOps team , RHCE, RHCi
Red Hat EMEA
redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
2424
days inactive
2424
days old
3 comments
4 participants
participants (4)
-
Allon Mureinik -
Barak Korren -
Greg Sheremeta -
Maor Lipchuk