[ovirt-devel] OST Failure - Weekly update [17/03/2018-23/03/2018]

Yedidyah Bar David didi at redhat.com
Sun Mar 25 09:32:41 UTC 2018 

On Sun, Mar 25, 2018 at 12:04 PM, Yedidyah Bar David <didi at redhat.com>
wrote:

> basic suite failed for me too.
>
> /var/log/messages has[1]:
>
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
> vSwitch Database Unit...
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
> error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
> /etc/openvswitch/conf.db does not exist ... (warning).
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
> database /etc/openvswitch/conf.db runuser: System error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service: control process exited, code=exited status=1
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
> Open vSwitch Database Unit.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency failed
> for Open vSwitch.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job
> openvswitch.service/start failed with result 'dependency'.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency failed
> for Open vSwitch Forwarding Unit.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job
> ovs-vswitchd.service/start failed with result 'dependency'.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
> ovsdb-server.service entered failed state.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service failed.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed
> for Open vSwitch Delete Transient Ports.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service holdoff time over, scheduling restart.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
> vSwitch Database Unit...
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed
> session 14.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
> error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
> /etc/openvswitch/conf.db does not exist ... (warning).
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
> database /etc/openvswitch/conf.db runuser: System error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service: control process exited, code=exited status=1
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
> Open vSwitch Database Unit.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
> ovsdb-server.service entered failed state.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service failed.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed
> for Open vSwitch Delete Transient Ports.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service holdoff time over, scheduling restart.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
> vSwitch Database Unit...
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
> error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
> /etc/openvswitch/conf.db does not exist ... (warning).
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
> database /etc/openvswitch/conf.db runuser: System error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service: control process exited, code=exited status=1
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
> Open vSwitch Database Unit.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
> ovsdb-server.service entered failed state.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service failed.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed
> for Open vSwitch Delete Transient Ports.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Created slice User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: New session
> 17 of user root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Started Session 17
> of user root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Session
> 17 of user root.
> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated
> CCID 2 (TCP-like)
> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated
> CCID 3 (TCP-Friendly Rate Control)
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service holdoff time over, scheduling restart.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
> vSwitch Database Unit...
> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: sctp: Hash tables
> configured (bind 256/256)
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed
> session 17.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User
> Slice of root.
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
> error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
> /etc/openvswitch/conf.db does not exist ... (warning).
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
> database /etc/openvswitch/conf.db runuser: System error
> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
> ovsdb-server.service: control process exited, code=exited status=1
>
> [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_
> check-patch-el7-x86_64/4651/artifact/exported-artifacts/
> basic-suite-master__logs/test_logs/basic-suite-master/post-
> 001_initialize_engine.py/lago-basic-suite-master-engine/_
> var_log/messages/*view*/
>
>
Talked with danken, he asked to check if it's an selinux issue. It is.
audit lot has:

type=AVC msg=audit(1521967325.146:675): avc:  denied  { create } for
pid=3787 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0
tcontext=system_u:system_r:openvswitch_t:s0
tclass=netlink_audit_socket
type=SYSCALL msg=audit(1521967325.146:675): arch=c000003e syscall=41
success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffc4e12b930 items=0 ppid=3786
pid=3787 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser"
exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0
key=(null)
type=PROCTITLE msg=audit(1521967325.146:675):
proctitle=72756E75736572002D2D75736572006F70656E76737769746368002D2D006F767364622D746F6F6C002D76636F6E736F6C653A6F666600736368656D612D76657273696F6E002F7573722F73686172652F6F70656E767377697463682F767377697463682E6F7673736368656D61
type=AVC msg=audit(1521967325.150:676): avc:  denied  { create } for
pid=3789 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0
tcontext=system_u:system_r:openvswitch_t:s0
tclass=netlink_audit_socket
type=SYSCALL msg=audit(1521967325.150:676): arch=c000003e syscall=41
success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffe03060130 items=0 ppid=3755
pid=3789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0
sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser"
exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0
key=(null)
type=PROCTITLE msg=audit(1521967325.150:676):

http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/4651/artifact/exported-artifacts/basic-suite-master__logs/test_logs/basic-suite-master/post-001_initialize_engine.py/lago-basic-suite-master-engine/_var_log/audit/audit.log

And it's 2.9:

Mar 25 04:38:39 lago-basic-suite-master-engine yum[1183]: Installed:
1:python2-openvswitch-2.9.0-3.el7.noarch



>
> On Sun, Mar 25, 2018 at 10:07 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>
>> + Network team.
>> I'm not sure if we've moved to ovs 2.9 already?
>> Y.
>>
>> On Sat, Mar 24, 2018 at 8:19 PM, Greg Sheremeta <gshereme at redhat.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Is there an ongoing engine master OST failure blocking?
>>>
>>> [ INFO  ] Stage: Misc configuration
>>> [ INFO  ]  Stage: Package installation
>>> [ INFO  ]  Stage: Misc configuration
>>> [ ERROR ] Failed to execute stage \'Misc configuration\': Failed to
>>> start service \'openvswitch\'
>>> [ INFO  ] Yum Performing yum transaction rollback
>>>
>>>
>>> These are unrelated code changes:
>>>
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
>>> -patch-el7-x86_64/4644/
>>> https://gerrit.ovirt.org/#/c/89347/
>>>
>>> and
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
>>> -patch-el7-x86_64/4647/
>>> https://gerrit.ovirt.org/67166
>>>
>>> But they both die in 001, with exactly 1.24MB in the log and 'Failed to
>>> start service openvswitch'
>>> 001_initialize_engine.py.junit.xml    1.24 MB
>>>
>>> Full file: http://jenkins.ovirt.org/job/ovirt-system-tests_master
>>> _check-patch-el7-x86_64/4644/artifact/exported-artifacts/bas
>>> ic-suite-master__logs/001_initialize_engine.py.junit.xml
>>>
>>>
>>> On Fri, Mar 23, 2018 at 12:14 PM, Dafna Ron <dron at redhat.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I would like to update on this week's failures and OST current status.
>>>>
>>>> On 19-03-2018 - the CI team reported 3 different failures.
>>>>
>>>> On Master branch the failed changes reported were:
>>>>
>>>>
>>>> *core: fix removal of vm-host device -
>>>> https://gerrit.ovirt.org/#/c/89145/ <https://gerrit.ovirt.org/#/c/89145/>*
>>>>
>>>> *core: USB in osinfo configuration depends on chipset -
>>>> https://gerrit.ovirt.org/#/c/88777/ <https://gerrit.ovirt.org/#/c/88777/>*
>>>> *On 4.2 *branch, the reported change was:
>>>>
>>>>
>>>>
>>>> *core: Call endAction() of all child commands in ImportVmCommand -
>>>> https://gerrit.ovirt.org/#/c/89165/ <https://gerrit.ovirt.org/#/c/89165/>*
>>>> The fix's for the regressions were merged the following day
>>>> (20-03-2018)
>>>>
>>>> https://gerrit.ovirt.org/#/c/89250/- core: Replace generic unlockVm()
>>>> logic in ImportVmCommand
>>>> https://gerrit.ovirt.org/#/c/89187/ - core: Fix NPE when creating an
>>>> instance type
>>>>
>>>> On 20-03-2018 - the CI team discovered an issue on the job's cleanup
>>>> which caused random failures on changes testing due to failure in docker
>>>> cleanup. There is an open Jira on the issue:
>>>> https://ovirt-jira.atlassian.net/browse/OVIRT-1939
>>>>
>>>>
>>>>
>>>> *Below you can see the chart for this week's resolved issues but cause
>>>> of failure:*Code = regression of working components/functionalities
>>>> Configurations = package related issues
>>>> Other = failed build artifacts
>>>> Infra = infrastructure/OST/Lago related issues
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Below is a chart of resolved failures based on ovirt version*
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *Below is a chart showing failures by suite type: *
>>>> Thank you,
>>>> Dafna
>>>>
>>>>
>>>> _______________________________________________
>>>> Infra mailing list
>>>> Infra at ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> GREG SHEREMETA
>>>
>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>>>
>>> Red Hat NA
>>>
>>> <https://www.redhat.com/>
>>>
>>> gshereme at redhat.com    IRC: gshereme
>>> <https://red.ht/sig>
>>>
>>> _______________________________________________
>>> Devel mailing list
>>> Devel at ovirt.org
>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>
>>
>>
>> _______________________________________________
>> Infra mailing list
>> Infra at ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/infra
>>
>>
>
>
> --
> Didi
>



-- 
Didi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6237 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 27001 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 23605 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 5835 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 7163 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 26288 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20180325/8a51df92/attachment-0011.png>

More information about the Infra mailing list