Re: [ovirt-devel] OST Failure - Weekly update [17/03/2018-23/03/2018]

currently selinux-policy-3.13.1-166.el7_4.4.noarch updating selinux-policy on engine gets me past 001, and then 002 dies: http://jenkins.ovirt.org/job/ovirt-system-tests_master_ check-patch-el7-x86_64/4663/console 15:05:45 # initialize_engine: rpm -qa: 15:05:46 CommandStatus(code=0, out='selinux-policy-3.13.1-166.el7_4.4.noarch\n', err='') 15:06:13 <snip> Package selinux-policy.noarch 0:3.13.1-166.el7_4.9 will be an update <snip> Updated: selinux-policy.noarch 0:3.13.1-166.el7_4.9 rpm -qa: 15:06:13 CommandStatus(code=0, out='selinux-policy-3.13.1-166.el7_4.9.noarch\n', err='') But later in 002 15:08:47 RuntimeError: 1 hosts failed installation: 15:08:47 lago-basic-suite-master-host-0: install_failed Perhaps selinux-policy needs to be updated on the hosts too? Not my area of expertise :) Greg On Sun, Mar 25, 2018 at 7:22 AM, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > Which version of selinux-policy do we have on the Engine image? > > *Bug 1538936* <https://bugzilla.redhat.com/show_bug.cgi?id=1538936> - Open > vSwitch selinux policy needs updating [rhel-7.4.z] > > was fixed in selinux-policy-3.13.1-166.el7_4.9 which is available in > http://mirror.centos.org/centos-7/7/updates/x86_64/Packages/ > selinux-policy-targeted-3.13.1-166.el7_4.9.noarch.rpm > > > On Sun, Mar 25, 2018 at 12:32 PM, Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > >> On Sun, Mar 25, 2018 at 12:04 PM, Yedidyah Bar David <didi(a)redhat.com&gt; >> wrote: >> >>> basic suite failed for me too. >>> >>> /var/log/messages has[1]: >>> >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >>> vSwitch Database Unit... >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >>> error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >>> /etc/openvswitch/conf.db does not exist ... (warning). >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >>> database /etc/openvswitch/conf.db runuser: System error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service: control process exited, code=exited status=1 >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >>> Open vSwitch Database Unit. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency >>> failed for Open vSwitch. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job >>> openvswitch.service/start failed with result 'dependency'. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Dependency >>> failed for Open vSwitch Forwarding Unit. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Job >>> ovs-vswitchd.service/start failed with result 'dependency'. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >>> ovsdb-server.service entered failed state. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service failed. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion >>> failed for Open vSwitch Delete Transient Ports. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service holdoff time over, scheduling restart. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >>> vSwitch Database Unit... >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed >>> session 14. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice >>> User Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User >>> Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >>> error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >>> /etc/openvswitch/conf.db does not exist ... (warning). >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >>> database /etc/openvswitch/conf.db runuser: System error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service: control process exited, code=exited status=1 >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >>> Open vSwitch Database Unit. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >>> ovsdb-server.service entered failed state. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service failed. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion >>> failed for Open vSwitch Delete Transient Ports. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service holdoff time over, scheduling restart. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >>> vSwitch Database Unit... >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >>> error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >>> /etc/openvswitch/conf.db does not exist ... (warning). >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >>> database /etc/openvswitch/conf.db runuser: System error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service: control process exited, code=exited status=1 >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Failed to start >>> Open vSwitch Database Unit. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Unit >>> ovsdb-server.service entered failed state. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service failed. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Assertion >>> failed for Open vSwitch Delete Transient Ports. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Created slice >>> User Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: New >>> session 17 of user root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting User >>> Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Started Session >>> 17 of user root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting >>> Session 17 of user root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated >>> CCID 2 (TCP-like) >>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: DCCP: Activated >>> CCID 3 (TCP-Friendly Rate Control) >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service holdoff time over, scheduling restart. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Starting Open >>> vSwitch Database Unit... >>> Mar 25 04:42:05 lago-basic-suite-master-engine kernel: sctp: Hash >>> tables configured (bind 256/256) >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd-logind: Removed >>> session 17. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Removed slice >>> User Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: Stopping User >>> Slice of root. >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: runuser: System >>> error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: >>> /etc/openvswitch/conf.db does not exist ... (warning). >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: Creating empty >>> database /etc/openvswitch/conf.db runuser: System error >>> Mar 25 04:42:05 lago-basic-suite-master-engine ovs-ctl: [FAILED] >>> Mar 25 04:42:05 lago-basic-suite-master-engine systemd: >>> ovsdb-server.service: control process exited, code=exited status=1 >>> >>> [1] http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >>> -patch-el7-x86_64/4651/artifact/exported-artifacts/basic-sui >>> te-master__logs/test_logs/basic-suite-master/post-001_initia >>> lize_engine.py/lago-basic-suite-master-engine/_var_log/messages/*view*/ >>> >>> >> Talked with danken, he asked to check if it's an selinux issue. It is. >> audit lot has: >> >> type=AVC msg=audit(1521967325.146:675): avc: denied { create } for pid=3787 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket >> type=SYSCALL msg=audit(1521967325.146:675): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffc4e12b930 items=0 ppid=3786 pid=3787 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null) >> type=PROCTITLE msg=audit(1521967325.146:675): proctitle=72756E75736572002D2D75736572006F70656E76737769746368002D2D006F767364622D746F6F6C002D76636F6E736F6C653A6F666600736368656D612D76657273696F6E002F7573722F73686172652F6F70656E767377697463682F767377697463682E6F7673736368656D61 >> type=AVC msg=audit(1521967325.150:676): avc: denied { create } for pid=3789 comm="runuser" scontext=system_u:system_r:openvswitch_t:s0 tcontext=system_u:system_r:openvswitch_t:s0 tclass=netlink_audit_socket >> type=SYSCALL msg=audit(1521967325.150:676): arch=c000003e syscall=41 success=no exit=-13 a0=10 a1=3 a2=9 a3=7ffe03060130 items=0 ppid=3755 pid=3789 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="runuser" exe="/usr/sbin/runuser" subj=system_u:system_r:openvswitch_t:s0 key=(null) >> type=PROCTITLE msg=audit(1521967325.150:676): >> >> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check-patch-el7-x8... >> >> And it's 2.9: >> >> Mar 25 04:38:39 lago-basic-suite-master-engine yum[1183]: Installed: 1:python2-openvswitch-2.9.0-3.el7.noarch >> >> >> >>> >>> On Sun, Mar 25, 2018 at 10:07 AM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: >>> >>>> + Network team. >>>> I'm not sure if we've moved to ovs 2.9 already? >>>> Y. >>>> >>>> On Sat, Mar 24, 2018 at 8:19 PM, Greg Sheremeta <gshereme(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Is there an ongoing engine master OST failure blocking? >>>>> >>>>> [ INFO ] Stage: Misc configuration >>>>> [ INFO ] Stage: Package installation >>>>> [ INFO ] Stage: Misc configuration >>>>> [ ERROR ] Failed to execute stage \'Misc configuration\': Failed to >>>>> start service \'openvswitch\' >>>>> [ INFO ] Yum Performing yum transaction rollback >>>>> >>>>> >>>>> These are unrelated code changes: >>>>> >>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >>>>> -patch-el7-x86_64/4644/ >>>>> https://gerrit.ovirt.org/#/c/89347/ >>>>> >>>>> and >>>>> http://jenkins.ovirt.org/job/ovirt-system-tests_master_check >>>>> -patch-el7-x86_64/4647/ >>>>> https://gerrit.ovirt.org/67166 >>>>> >>>>> But they both die in 001, with exactly 1.24MB in the log and 'Failed >>>>> to start service openvswitch' >>>>> 001_initialize_engine.py.junit.xml 1.24 MB >>>>> >>>>> Full file: http://jenkins.ovirt.org/job/ovirt-system-tests_master >>>>> _check-patch-el7-x86_64/4644/artifact/exported-artifacts/bas >>>>> ic-suite-master__logs/001_initialize_engine.py.junit.xml >>>>> >>>>> >>>>> On Fri, Mar 23, 2018 at 12:14 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I would like to update on this week's failures and OST current >>>>>> status. >>>>>> >>>>>> On 19-03-2018 - the CI team reported 3 different failures. >>>>>> >>>>>> On Master branch the failed changes reported were: >>>>>> >>>>>> >>>>>> *core: fix removal of vm-host device - >>>>>> https://gerrit.ovirt.org/#/c/89145/ <https://gerrit.ovirt.org/#/c/89145/>* >>>>>> >>>>>> *core: USB in osinfo configuration depends on chipset - >>>>>> https://gerrit.ovirt.org/#/c/88777/ <https://gerrit.ovirt.org/#/c/88777/>* >>>>>> *On 4.2 *branch, the reported change was: >>>>>> >>>>>> >>>>>> >>>>>> *core: Call endAction() of all child commands in ImportVmCommand - >>>>>> https://gerrit.ovirt.org/#/c/89165/ <https://gerrit.ovirt.org/#/c/89165/>* >>>>>> The fix's for the regressions were merged the following day >>>>>> (20-03-2018) >>>>>> >>>>>> https://gerrit.ovirt.org/#/c/89250/- core: Replace generic >>>>>> unlockVm() logic in ImportVmCommand >>>>>> https://gerrit.ovirt.org/#/c/89187/ - core: Fix NPE when creating >>>>>> an instance type >>>>>> >>>>>> On 20-03-2018 - the CI team discovered an issue on the job's cleanup >>>>>> which caused random failures on changes testing due to failure in docker >>>>>> cleanup. There is an open Jira on the issue: >>>>>> https://ovirt-jira.atlassian.net/browse/OVIRT-1939 >>>>>> >>>>>> >>>>>> >>>>>> *Below you can see the chart for this week's resolved issues but >>>>>> cause of failure:*Code = regression of working >>>>>> components/functionalities >>>>>> Configurations = package related issues >>>>>> Other = failed build artifacts >>>>>> Infra = infrastructure/OST/Lago related issues >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Below is a chart of resolved failures based on ovirt version* >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *Below is a chart showing failures by suite type: * >>>>>> Thank you, >>>>>> Dafna >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Infra mailing list >>>>>> Infra(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/infra >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> GREG SHEREMETA >>>>> >>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>> >>>>> Red Hat NA >>>>> >>>>> <https://www.redhat.com/> >>>>> >>>>> gshereme(a)redhat.com IRC: gshereme >>>>> <https://red.ht/sig> >>>>> >>>>> _______________________________________________ >>>>> Devel mailing list >>>>> Devel(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/devel >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Infra mailing list >>>> Infra(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> >>>> >>> >>> >>> -- >>> Didi >>> >> >> >> >> -- >> Didi >> > > -- GREG SHEREMETA SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX Red Hat NA <https://www.redhat.com/> gshereme(a)redhat.com IRC: gshereme <https://red.ht/sig>