Re: [ovirt-devel] OST Failure - Weekly update [17/03/2018-23/03/2018]

Which version of selinux-policy do we have on the Engine image? *Bug 1538936* <https://bugzilla.redhat.com/show_bug.cgi?id=1538936> - Open vSwitch selinux policy needs updating [rhel-7.4.z] was fixed in selinux-policy-3.13.1-166.el7_4.9 which is available in http://mirror.centos.org/centos-7/7/updates/x86_64/ Packages/selinux-policy-targeted-3.13.1-166.el7_4.9.noarch.rpm On Sun, Mar 25, 2018 at 12:32 PM, Yedidyah Bar David <didi(a)redhat.com&gt; wrote: > On Sun, Mar 25, 2018 at 12:04 PM, Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > >> basic suite failed for me too. >> >> /var/log/messages has[1]: >> >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >> vSwitch Database Unit... >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >> error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >> /etc/openvswitch/conf.db does not exist ... (warning). >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >> database /etc/openvswitch/conf.db runuser: System error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service: control process exited, code=exited status=1 >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >> Open vSwitch Database Unit. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency >> failed for Open vSwitch. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job >> openvswitch.service/start failed with result 'dependency'. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency >> failed for Open vSwitch Forwarding Unit. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job >> ovs-vswitchd.service/start failed with result 'dependency'. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >> ovsdb-server.service entered failed state. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service failed. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed >> for Open vSwitch Delete Transient Ports. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service holdoff time over, scheduling restart. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >> vSwitch Database Unit... >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed >> session 14. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice >> User Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User >> Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >> error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >> /etc/openvswitch/conf.db does not exist ... (warning). >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >> database /etc/openvswitch/conf.db runuser: System error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service: control process exited, code=exited status=1 >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >> Open vSwitch Database Unit. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >> ovsdb-server.service entered failed state. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service failed. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed >> for Open vSwitch Delete Transient Ports. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service holdoff time over, scheduling restart. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >> vSwitch Database Unit... >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >> error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >> /etc/openvswitch/conf.db does not exist ... (warning). >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >> database /etc/openvswitch/conf.db runuser: System error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service: control process exited, code=exited status=1 >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >> Open vSwitch Database Unit. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >> ovsdb-server.service entered failed state. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service failed. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion failed >> for Open vSwitch Delete Transient Ports. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Created slice >> User Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: New >> session 17 of user root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting User >> Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Started Session >> 17 of user root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Session >> 17 of user root. >> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated >> CCID 2 (TCP-like) >> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated >> CCID 3 (TCP-Friendly Rate Control) >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service holdoff time over, scheduling restart. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >> vSwitch Database Unit... >> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: sctp: Hash tables >> configured (bind 256/256) >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed >> session 17. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice >> User Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User >> Slice of root. >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >> error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >> /etc/openvswitch/conf.db does not exist ... (warning). >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >> database /etc/openvswitch/conf.db runuser: System error >> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >> ovsdb-server.service: control process exited, code=exited status=1 >> >> [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >> -patch-el7-x86_64/4651/artifact/exported-artifacts/basic-sui >> te-master__logs/test_logs/basic-suite-master/post-001_initia >> lize_engine.py/lago-basic-suite-master-engine/_var_log/messages/*view*/ >> >> > Talked with danken, he asked to check if it's an selinux issue. It is. > audit lot has: > > type=AVC msg=audit(1521967325.146:675): avc: denied { create } for pid=3787 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket > type=SYSCALL msg=audit(1521967325.146:675): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffc4e12b930 items=0 ppid=3786 pid=3787 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null) > type=PROCTITLE msg=audit(1521967325.146:675): proctitle=72756E75736572002D2D75736572006F70656E76737769746368002D2D006F767364622D746F6F6C002D76636F6E736F6C653A6F666600736368656D612D76657273696F6E002F7573722F73686172652F6F70656E767377697463682F767377697463682E6F7673736368656D61 > type=AVC msg=audit(1521967325.150:676): avc: denied { create } for pid=3789 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket > type=SYSCALL msg=audit(1521967325.150:676): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffe03060130 items=0 ppid=3755 pid=3789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null) > type=PROCTITLE msg=audit(1521967325.150:676): > > http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x8... > > And it's 2.9: > > Mar 25 04:38:39 lago-basic-suite-master-engine yum[1183]: Installed: 1:python2-openvswitch-2.9.0-3.el7.noarch > > > >> >> On Sun, Mar 25, 2018 at 10:07 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >> >>> + Network team. >>> I'm not sure if we've moved to ovs 2.9 already? >>> Y. >>> >>> On Sat, Mar 24, 2018 at 8:19 PM, Greg Sheremeta <gshereme(a)redhat.com&gt; >>> wrote: >>> >>>> Hi, >>>> >>>> Is there an ongoing engine master OST failure blocking? >>>> >>>> [ INFO ] Stage: Misc configuration >>>> [ INFO ] Stage: Package installation >>>> [ INFO ] Stage: Misc configuration >>>> [ ERROR ] Failed to execute stage \'Misc configuration\': Failed to >>>> start service \'openvswitch\' >>>> [ INFO ] Yum Performing yum transaction rollback >>>> >>>> >>>> These are unrelated code changes: >>>> >>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >>>> -patch-el7-x86_64/4644/ >>>> https://gerrit.ovirt.org/#/c/89347/ >>>> >>>> and >>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >>>> -patch-el7-x86_64/4647/ >>>> https://gerrit.ovirt.org/67166 >>>> >>>> But they both die in 001, with exactly 1.24MB in the log and 'Failed >>>> to start service openvswitch' >>>> 001_initialize_engine.py.junit.xml 1.24 MB >>>> >>>> Full file: http://jenkins.ovirt.org/job/ovirt-system-tests_master >>>> _check-patch-el7-x86_64/4644/artifact/exported-artifacts/bas >>>> ic-suite-master__logs/001_initialize_engine.py.junit.xml >>>> >>>> >>>> On Fri, Mar 23, 2018 at 12:14 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>> >>>>> Hello, >>>>> >>>>> I would like to update on this week's failures and OST current >>>>> status. >>>>> >>>>> On 19-03-2018 - the CI team reported 3 different failures. >>>>> >>>>> On Master branch the failed changes reported were: >>>>> >>>>> >>>>> *core: fix removal of vm-host device - >>>>> https://gerrit.ovirt.org/#/c/89145/ <https://gerrit.ovirt.org/#/c/89145/>* >>>>> >>>>> *core: USB in osinfo configuration depends on chipset - >>>>> https://gerrit.ovirt.org/#/c/88777/ <https://gerrit.ovirt.org/#/c/88777/>* >>>>> *On 4.2 *branch, the reported change was: >>>>> >>>>> >>>>> >>>>> *core: Call endAction() of all child commands in ImportVmCommand - >>>>> https://gerrit.ovirt.org/#/c/89165/ <https://gerrit.ovirt.org/#/c/89165/>* >>>>> The fix's for the regressions were merged the following day >>>>> (20-03-2018) >>>>> >>>>> https://gerrit.ovirt.org/#/c/89250/- core: Replace generic >>>>> unlockVm() logic in ImportVmCommand >>>>> https://gerrit.ovirt.org/#/c/89187/ - core: Fix NPE when creating an >>>>> instance type >>>>> >>>>> On 20-03-2018 - the CI team discovered an issue on the job's cleanup >>>>> which caused random failures on changes testing due to failure in docker >>>>> cleanup. There is an open Jira on the issue: >>>>> https://ovirt-jira.atlassian.net/browse/OVIRT-1939 >>>>> >>>>> >>>>> >>>>> *Below you can see the chart for this week's resolved issues but >>>>> cause of failure:*Code = regression of working >>>>> components/functionalities >>>>> Configurations = package related issues >>>>> Other = failed build artifacts >>>>> Infra = infrastructure/OST/Lago related issues >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *Below is a chart of resolved failures based on ovirt version* >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *Below is a chart showing failures by suite type: * >>>>> Thank you, >>>>> Dafna >>>>> >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list >>>>> Infra(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> GREG SHEREMETA >>>> >>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>> >>>> Red Hat NA >>>> >>>> <https://www.redhat.com/> >>>> >>>> gshereme(a)redhat.com IRC: gshereme >>>> <https://red.ht/sig> >>>> >>>> _______________________________________________ >>>> Devel mailing list >>>> Devel(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/devel >>>> >>> >>> >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >>> >>> >> >> >> -- >> Didi >> > > > > -- > Didi >