[ovirt-devel] OST Failure - Weekly update [17/03/2018-23/03/2018]

Sun Mar 25 17:11:07 UTC 2018

Indeed, updating selinux-policy on both engine and hosts passes.

Change If671d938: [test do not merge] test selinux-policy update on engine
and hosts | gerrit.ovirt Code Review
https://gerrit.ovirt.org/#/c/89427/
http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/4676/consoleFull

So I guess the images need updates.

On Sun, Mar 25, 2018 at 11:20 AM, Greg Sheremeta <gshereme at redhat.com>
wrote:

> currently selinux-policy-3.13.1-166.el7_4.4.noarch
>
> updating selinux-policy on engine gets me past 001, and then 002 dies:
> http://jenkins.ovirt.org/job/ovirt-system-tests_master_
> check-patch-el7-x86_64/4663/console
>
> 15:05:45   # initialize_engine:
> rpm -qa:
> 15:05:46 CommandStatus(code=0, out='selinux-policy-3.13.1-166.el7_4.4.noarch\n',
> err='')
> 15:06:13 <snip> Package selinux-policy.noarch 0:3.13.1-166.el7_4.9 will be
> an update
> <snip> Updated:  selinux-policy.noarch 0:3.13.1-166.el7_4.9
> rpm -qa:
> 15:06:13 CommandStatus(code=0, out='selinux-policy-3.13.1-166.el7_4.9.noarch\n',
> err='')
>
> But later in 002
> 15:08:47 RuntimeError: 1 hosts failed installation:
> 15:08:47 lago-basic-suite-master-host-0: install_failed
>
> Perhaps selinux-policy needs to be updated on the hosts too? Not my area
> of expertise :)
>
> Greg
>
> On Sun, Mar 25, 2018 at 7:22 AM, Dan Kenigsberg <danken at redhat.com> wrote:
>
>> Which version of selinux-policy do we have on the Engine image?
>>
>> *Bug 1538936* <https://bugzilla.redhat.com/show_bug.cgi?id=1538936> - Open
>> vSwitch selinux policy needs updating [rhel-7.4.z]
>>
>> was fixed in selinux-policy-3.13.1-166.el7_4.9 which is available in
>> http://mirror.centos.org/centos-7/7/updates/x86_64/Packages/
>> selinux-policy-targeted-3.13.1-166.el7_4.9.noarch.rpm
>>
>>
>> On Sun, Mar 25, 2018 at 12:32 PM, Yedidyah Bar David <didi at redhat.com>
>> wrote:
>>
>>> On Sun, Mar 25, 2018 at 12:04 PM, Yedidyah Bar David <didi at redhat.com>
>>> wrote:
>>>
>>>> basic suite failed for me too.
>>>>
>>>> /var/log/messages has[1]:
>>>>
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
>>>> vSwitch Database Unit...
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
>>>> error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
>>>> /etc/openvswitch/conf.db does not exist ... (warning).
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
>>>> database /etc/openvswitch/conf.db runuser: System error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service: control process exited, code=exited status=1
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
>>>> Open vSwitch Database Unit.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency
>>>> failed for Open vSwitch.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job
>>>> openvswitch.service/start failed with result 'dependency'.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency
>>>> failed for Open vSwitch Forwarding Unit.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job
>>>> ovs-vswitchd.service/start failed with result 'dependency'.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
>>>> ovsdb-server.service entered failed state.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service failed.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion
>>>> failed for Open vSwitch Delete Transient Ports.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service holdoff time over, scheduling restart.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
>>>> vSwitch Database Unit...
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed
>>>> session 14.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice
>>>> User Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User
>>>> Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
>>>> error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
>>>> /etc/openvswitch/conf.db does not exist ... (warning).
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
>>>> database /etc/openvswitch/conf.db runuser: System error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service: control process exited, code=exited status=1
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
>>>> Open vSwitch Database Unit.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
>>>> ovsdb-server.service entered failed state.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service failed.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion
>>>> failed for Open vSwitch Delete Transient Ports.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service holdoff time over, scheduling restart.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
>>>> vSwitch Database Unit...
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
>>>> error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
>>>> /etc/openvswitch/conf.db does not exist ... (warning).
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
>>>> database /etc/openvswitch/conf.db runuser: System error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service: control process exited, code=exited status=1
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start
>>>> Open vSwitch Database Unit.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit
>>>> ovsdb-server.service entered failed state.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service failed.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion
>>>> failed for Open vSwitch Delete Transient Ports.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Created slice
>>>> User Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: New
>>>> session 17 of user root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting User
>>>> Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Started Session
>>>> 17 of user root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting
>>>> Session 17 of user root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated
>>>> CCID 2 (TCP-like)
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated
>>>> CCID 3 (TCP-Friendly Rate Control)
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service holdoff time over, scheduling restart.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open
>>>> vSwitch Database Unit...
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: sctp: Hash
>>>> tables configured (bind 256/256)
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed
>>>> session 17.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice
>>>> User Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User
>>>> Slice of root.
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System
>>>> error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl:
>>>> /etc/openvswitch/conf.db does not exist ... (warning).
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty
>>>> database /etc/openvswitch/conf.db runuser: System error
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED]
>>>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd:
>>>> ovsdb-server.service: control process exited, code=exited status=1
>>>>
>>>> [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
>>>> -patch-el7-x86_64/4651/artifact/exported-artifacts/basic-sui
>>>> te-master__logs/test_logs/basic-suite-master/post-001_initia
>>>> lize_engine.py/lago-basic-suite-master-engine/_var_log/messages/*view*/
>>>>
>>>>
>>> Talked with danken, he asked to check if it's an selinux issue. It is.
>>> audit lot has:
>>>
>>> type=AVC msg=audit(1521967325.146:675): avc:  denied  { create } for  pid=3787 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket
>>> type=SYSCALL msg=audit(1521967325.146:675): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffc4e12b930 items=0 ppid=3786 pid=3787 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null)
>>> type=PROCTITLE msg=audit(1521967325.146:675): proctitle=72756E75736572002D2D75736572006F70656E76737769746368002D2D006F767364622D746F6F6C002D76636F6E736F6C653A6F666600736368656D612D76657273696F6E002F7573722F73686172652F6F70656E767377697463682F767377697463682E6F7673736368656D61
>>> type=AVC msg=audit(1521967325.150:676): avc:  denied  { create } for  pid=3789 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket
>>> type=SYSCALL msg=audit(1521967325.150:676): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffe03060130 items=0 ppid=3755 pid=3789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null)
>>> type=PROCTITLE msg=audit(1521967325.150:676):
>>>
>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x86_64/4651/artifact/exported-artifacts/basic-suite-master__logs/test_logs/basic-suite-master/post-001_initialize_engine.py/lago-basic-suite-master-engine/_var_log/audit/audit.log
>>>
>>> And it's 2.9:
>>>
>>> Mar 25 04:38:39 lago-basic-suite-master-engine yum[1183]: Installed: 1:python2-openvswitch-2.9.0-3.el7.noarch
>>>
>>>
>>>
>>>>
>>>> On Sun, Mar 25, 2018 at 10:07 AM, Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>
>>>>> + Network team.
>>>>> I'm not sure if we've moved to ovs 2.9 already?
>>>>> Y.
>>>>>
>>>>> On Sat, Mar 24, 2018 at 8:19 PM, Greg Sheremeta <gshereme at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Is there an ongoing engine master OST failure blocking?
>>>>>>
>>>>>> [ INFO  ] Stage: Misc configuration
>>>>>> [ INFO  ]  Stage: Package installation
>>>>>> [ INFO  ]  Stage: Misc configuration
>>>>>> [ ERROR ] Failed to execute stage \'Misc configuration\': Failed to
>>>>>> start service \'openvswitch\'
>>>>>> [ INFO  ] Yum Performing yum transaction rollback
>>>>>>
>>>>>>
>>>>>> These are unrelated code changes:
>>>>>>
>>>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
>>>>>> -patch-el7-x86_64/4644/
>>>>>> https://gerrit.ovirt.org/#/c/89347/
>>>>>>
>>>>>> and
>>>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check
>>>>>> -patch-el7-x86_64/4647/
>>>>>> https://gerrit.ovirt.org/67166
>>>>>>
>>>>>> But they both die in 001, with exactly 1.24MB in the log and 'Failed
>>>>>> to start service openvswitch'
>>>>>> 001_initialize_engine.py.junit.xml    1.24 MB
>>>>>>
>>>>>> Full file: http://jenkins.ovirt.org/job/ovirt-system-tests_master
>>>>>> _check-patch-el7-x86_64/4644/artifact/exported-artifacts/bas
>>>>>> ic-suite-master__logs/001_initialize_engine.py.junit.xml
>>>>>>
>>>>>>
>>>>>> On Fri, Mar 23, 2018 at 12:14 PM, Dafna Ron <dron at redhat.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I would like to update on this week's failures and OST current
>>>>>>> status.
>>>>>>>
>>>>>>> On 19-03-2018 - the CI team reported 3 different failures.
>>>>>>>
>>>>>>> On Master branch the failed changes reported were:
>>>>>>>
>>>>>>>
>>>>>>> *core: fix removal of vm-host device -
>>>>>>> https://gerrit.ovirt.org/#/c/89145/ <https://gerrit.ovirt.org/#/c/89145/>*
>>>>>>>
>>>>>>> *core: USB in osinfo configuration depends on chipset -
>>>>>>> https://gerrit.ovirt.org/#/c/88777/ <https://gerrit.ovirt.org/#/c/88777/>*
>>>>>>> *On 4.2 *branch, the reported change was:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *core: Call endAction() of all child commands in ImportVmCommand -
>>>>>>> https://gerrit.ovirt.org/#/c/89165/ <https://gerrit.ovirt.org/#/c/89165/>*
>>>>>>> The fix's for the regressions were merged the following day
>>>>>>> (20-03-2018)
>>>>>>>
>>>>>>> https://gerrit.ovirt.org/#/c/89250/- core: Replace generic
>>>>>>> unlockVm() logic in ImportVmCommand
>>>>>>> https://gerrit.ovirt.org/#/c/89187/ - core: Fix NPE when creating
>>>>>>> an instance type
>>>>>>>
>>>>>>> On 20-03-2018 - the CI team discovered an issue on the job's cleanup
>>>>>>> which caused random failures on changes testing due to failure in docker
>>>>>>> cleanup. There is an open Jira on the issue:
>>>>>>> https://ovirt-jira.atlassian.net/browse/OVIRT-1939
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Below you can see the chart for this week's resolved issues but
>>>>>>> cause of failure:*Code = regression of working
>>>>>>> components/functionalities
>>>>>>> Configurations = package related issues
>>>>>>> Other = failed build artifacts
>>>>>>> Infra = infrastructure/OST/Lago related issues
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Below is a chart of resolved failures based on ovirt version*
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Below is a chart showing failures by suite type: *
>>>>>>> Thank you,
>>>>>>> Dafna
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Infra mailing list
>>>>>>> Infra at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> GREG SHEREMETA
>>>>>>
>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>>>>>>
>>>>>> Red Hat NA
>>>>>>
>>>>>> <https://www.redhat.com/>
>>>>>>
>>>>>> gshereme at redhat.com    IRC: gshereme
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Devel mailing list
>>>>>> Devel at ovirt.org
>>>>>> http://lists.ovirt.org/mailman/listinfo/devel
>>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Infra mailing list
>>>>> Infra at ovirt.org
>>>>> http://lists.ovirt.org/mailman/listinfo/infra
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Didi
>>>>
>>>
>>>
>>>
>>> --
>>> Didi
>>>
>>
>>
>
>
> --
>
> GREG SHEREMETA
>
> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX
>
> Red Hat NA
>
> <https://www.redhat.com/>
>
> gshereme at redhat.com    IRC: gshereme
> <https://red.ht/sig>
>

-- 

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

<https://www.redhat.com/>

gshereme at redhat.com    IRC: gshereme
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 7163 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 6237 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 5835 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 23605 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 26288 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 27001 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20180325/817f32a2/attachment-0011.png>