On Fri, May 15, 2020 at 12:24 PM Tomáš Golembiovský <tgolembi@redhat.com> wrote:
Today I tried to downgrade systemd to 239-18 and then upgrade again to
see what will happen.

Please do not upgrade again, systemd 239-18 is the version used in CentOS 8.1, and oVirt-4.4 is targeting CentOS 8.1 currently.
That openvswitch-2.11.1-5.el8.x86_64 does not work flawless on CentOS Stream is documented on
https://bugzilla.redhat.com/show_bug.cgi?id=1832725

If you like to test on CentOS Stream, there is an experimental repo in
https://copr.fedorainfracloud.org/coprs/dominik/OpenvSwitch/
which might change, disappear, or break your installation.



 
And yes, it is definitely caused by change in
systemd. Likely this change (BZ 1778384):

    https://github.com/systemd-rhel/rhel-8/pull/44

But I don't know where to report the problem for CentOS Stream. The
'openvswitch' component in Bugzilla is not active anymore. Does anyone
know what's the proper component?

    Tomas Golembiovsky

On Fri, 15 May 2020 00:02:00 +0200
Tomáš Golembiovský <tgolembi@redhat.com> wrote:

> Sorry for late reply, but I noticed the email just now. I had the same
> problem some time ago but could not figure out the source of it at that
> time. I noticed the behavior after some system update and reboot, but
> neither systemd nor openvswitch had been updated at that moment. Now I
> noticed that the host where things are broken contains
> systemd-239-21.el8 from CentOS Stream and host where everything works
> systemd-239-18.el8_1.5 from normal CentOS. so it is possible that
> systemd was updated on previous "dnf update" and I just forgot to reboot
> the host so the problem did not manifest itself. So the question is, are
> you perhaps using CentOS Stream on your host?
>
> As to why I suspect systemd, mostly because the unit file in openvswitch
> had not changed for a long time. The management of runtime directory is
> left to systemd (see the RuntimeDirectory* variables in unit file). As
> per documentation the owner is the same as the User in the unit file, or
> it defaults to root if not specified. That is our case here. The
> openvswitch unit file "cleverly" solves it by calling "chown" in
> ExecStartPre. The trouble is that the permission and owner of the
> runtime directory are "fixed" by systemd after every Exec* command so
> the chown does not do its job. It is likely that this behavior changed
> in systemd between 239-18 and 239-21.
>
> My workaround was simply to do that as part of ExecStart like this:
>
>     ExecStart=/bin/sh -x -c '/usr/bin/chown ${OVS_USER_ID} /var/run/openvswitch /var/log/openvswitch ; /usr/share/openvswitch/scripts/ovs-ctl \
>             --no-ovs-vswitchd --no-monitor --system-id=random \
>             ${OVSUSER} \
>             start $OPTIONS'
>
> Hope this helps,
>
>     Tomas Golebiovsky
>
>
> PS: The reason why you don't see the runtime directory on disk is that
> it is removed by systemd when the service stops. That's why creating it
> or changing the permission on it manually makes no sense. Or having it
> listed as RPM content.
>
>
> On Mon, 4 May 2020 14:51:57 +0100
> lejeczek <peljasz@yahoo.co.uk> wrote:
>
> > On 04/05/2020 13:59, Yedidyah Bar David wrote:
> > > On Mon, May 4, 2020 at 3:37 PM lejeczek <peljasz@yahoo.co.uk> wrote: 
> > >>
> > >>
> > >> On 04/05/2020 12:07, Yedidyah Bar David wrote: 
> > >>> On Mon, May 4, 2020 at 1:49 PM lejeczek <peljasz@yahoo.co.uk> wrote: 
> > >>>> hi devel,
> > >>>>
> > >>>> Some inconsistencies in systemd's daemon configs &/|| binaries?
> > >>>>
> > >>>> Starting Open vSwitch Database Unit...
> > >>>> ovs|00002|daemon_unix|EMER|/var/run/openvswitch/ovsdb-server.pid.tmp:
> > >>>> create failed (Permission denied)
> > >>>> Starting ovsdb-server ovsdb-server:
> > >>>> /var/run/openvswitch/ovsdb-server.pid.tmp: create failed
> > >>>> (Permission denied)
> > >>>> [FAILED]
> > >>>>
> > >>>> Any quick fix would mind to suggest? 
> > >>> Can you please check/share the output of:
> > >>>
> > >>> ls -la /var/run/openvswitch
> > >>>
> > >>> rpm -q openvswitch
> > >>>
> > >>> Best regards, 
> > >> $ ls -la /var/run/openvswitch
> > >> ls: cannot access '/var/run/openvswitch 
> > > So it's gone,
> > > 
> > >> $ rpm -ql openvswitch.x86_64
> > >> /etc/bash_completion.d/ovs-appctl-bashcomp.bash
> > >> /etc/bash_completion.d/ovs-vsctl-bashcomp.bash
> > >> /etc/logrotate.d/openvswitch
> > >> /etc/openvswitch
> > >> /etc/openvswitch/.conf.db.~lock~
> > >> /etc/openvswitch/conf.db
> > >> /etc/openvswitch/default.conf
> > >> /etc/openvswitch/system-id.conf
> > >> /etc/sysconfig/openvswitch
> > >> /run/openvswitch
> > >> /usr/bin/ovs-appctl
> > >> /usr/bin/ovs-dpctl
> > >> /usr/bin/ovs-ofctl
> > >> /usr/bin/ovs-pki
> > >> /usr/bin/ovs-vsctl
> > >> /usr/bin/ovsdb-client
> > >> /usr/bin/ovsdb-tool
> > >> /usr/bin/vtep-ctl
> > >> /usr/lib/.build-id
> > >> /usr/lib/.build-id/06
> > >> /usr/lib/.build-id/06/9fa09762648319fdda1761937dcd4f8e429828
> > >> /usr/lib/.build-id/14
> > >> /usr/lib/.build-id/14/f86ec775ffd588bd37fd61b932aa250103bfdb
> > >> /usr/lib/.build-id/1c
> > >> /usr/lib/.build-id/1c/2c3f9ede44b42a3659cccfc20b17bf51e08f95
> > >> /usr/lib/.build-id/34
> > >> /usr/lib/.build-id/34/7f712ade5dc04195c71b38c626e192dc2b078b
> > >> /usr/lib/.build-id/4f
> > >> /usr/lib/.build-id/4f/d442fdee55254ca3a7236e524536a8ffb73c17
> > >> /usr/lib/.build-id/55
> > >> /usr/lib/.build-id/55/b43b83a4ac745590014bbc7a4f16204ecb3cba
> > >> /usr/lib/.build-id/5d
> > >> /usr/lib/.build-id/5d/f9b2c0ac7c35a59464c9a910e89d71c3387d96
> > >> /usr/lib/.build-id/5e
> > >> /usr/lib/.build-id/5e/886639c0e4841d01cfa9d6a8aff210f297267e
> > >> /usr/lib/.build-id/6c
> > >> /usr/lib/.build-id/6c/ad54943c22a0ec765f339e4f4216b51a809dc9
> > >> /usr/lib/.build-id/6e
> > >> /usr/lib/.build-id/6e/cd97bce86b69ce1a9841b84540884ad7b7421f
> > >> /usr/lib/.build-id/78
> > >> /usr/lib/.build-id/78/a71d93990610e97e76a7b1b4acc105e64655a3
> > >> /usr/lib/.build-id/a9
> > >> /usr/lib/.build-id/a9/62872e830ac6c7db512b8fd17c7ae9d9227e7e
> > >> /usr/lib/.build-id/aa
> > >> /usr/lib/.build-id/aa/7a8f137a95a94eedb767aecbed27834ba906a9
> > >> /usr/lib/.build-id/dc
> > >> /usr/lib/.build-id/dc/a8f35e4acbbcf599b63369081d5643ce679ded
> > >> /usr/lib/.build-id/ee
> > >> /usr/lib/.build-id/ee/57eca75e3312ed9cad7fbad6977fc06bd4852f
> > >> /usr/lib/systemd/system/openvswitch.service
> > >> /usr/lib/systemd/system/ovs-delete-transient-ports.service
> > >> /usr/lib/systemd/system/ovs-vswitchd.service
> > >> /usr/lib/systemd/system/ovsdb-server.service
> > >> /usr/lib/udev/rules.d/91-vfio.rules
> > >> /usr/lib64/libofproto-2.11.so.0
> > >> /usr/lib64/libofproto-2.11.so.0.0.1
> > >> /usr/lib64/libopenvswitch-2.11.so.0
> > >> /usr/lib64/libopenvswitch-2.11.so.0.0.1
> > >> /usr/lib64/libovn-2.11.so.0
> > >> /usr/lib64/libovn-2.11.so.0.0.1
> > >> /usr/lib64/libovsdb-2.11.so.0
> > >> /usr/lib64/libovsdb-2.11.so.0.0.1
> > >> /usr/lib64/libsflow-2.11.so.0
> > >> /usr/lib64/libsflow-2.11.so.0.0.1
> > >> /usr/lib64/libvtep-2.11.so.0
> > >> /usr/lib64/libvtep-2.11.so.0.0.1
> > >> /usr/sbin/ovs-vswitchd
> > >> /usr/sbin/ovsdb-server
> > >> /usr/share/doc/openvswitch
> > >> /usr/share/doc/openvswitch/LICENSE
> > >> /usr/share/doc/openvswitch/NEWS
> > >> /usr/share/doc/openvswitch/NOTICE
> > >> /usr/share/doc/openvswitch/README.RHEL.rst
> > >> /usr/share/doc/openvswitch/README.rst
> > >> /usr/share/man/man1/ovsdb-client.1.gz
> > >> /usr/share/man/man1/ovsdb-server.1.gz
> > >> /usr/share/man/man1/ovsdb-tool.1.gz
> > >> /usr/share/man/man5/ovs-vswitchd.conf.db.5.gz
> > >> /usr/share/man/man5/ovsdb-server.5.gz
> > >> /usr/share/man/man5/ovsdb.5.gz
> > >> /usr/share/man/man5/vtep.5.gz
> > >> /usr/share/man/man7/ovs-actions.7.gz
> > >> /usr/share/man/man7/ovs-fields.7.gz
> > >> /usr/share/man/man7/ovsdb-server.7.gz
> > >> /usr/share/man/man7/ovsdb.7.gz
> > >> /usr/share/man/man8/ovs-appctl.8.gz
> > >> /usr/share/man/man8/ovs-ctl.8.gz
> > >> /usr/share/man/man8/ovs-dpctl.8.gz
> > >> /usr/share/man/man8/ovs-kmod-ctl.8.gz
> > >> /usr/share/man/man8/ovs-ofctl.8.gz
> > >> /usr/share/man/man8/ovs-parse-backtrace.8.gz
> > >> /usr/share/man/man8/ovs-pki.8.gz
> > >> /usr/share/man/man8/ovs-vsctl.8.gz
> > >> /usr/share/man/man8/ovs-vswitchd.8.gz
> > >> /usr/share/man/man8/vtep-ctl.8.gz
> > >> /usr/share/openvswitch/scripts/openvswitch.init
> > >> /usr/share/openvswitch/scripts/ovs-check-dead-ifs
> > >> /usr/share/openvswitch/scripts/ovs-ctl
> > >> /usr/share/openvswitch/scripts/ovs-kmod-ctl
> > >> /usr/share/openvswitch/scripts/ovs-lib
> > >> /usr/share/openvswitch/scripts/ovs-save
> > >> /usr/share/openvswitch/scripts/ovs-systemd-reload
> > >> /usr/share/openvswitch/scripts/ovs-vtep
> > >> /usr/share/openvswitch/vswitch.ovsschema
> > >> /usr/share/openvswitch/vtep.ovsschema
> > >> /var/lib/openvswitch
> > >> /var/lib/openvswitch/pki 
> > > ...although the package did include it.
> > >
> > > Perhaps check what/who removed it.
> > >
> > > To fix, you can try 'yum reinstall openvswitch'.
> > >
> > > Best regards, 
> > Nope, trivializing it won't help.
> > It's something in the either or both, systemd deamons
> > configs, binaries/configs.
> > Even a manual creation of the path:
> > $ mkdir /var/run/openvswitch/; chown openvswitch:hugetlbfs
> > /var/run/openvswitch/
> > Does nothing because very next attempt of:
> > $ systemctl restart openvswitch.service
> > and that end path is gone.
> >
> > regards, L
> > _______________________________________________
> > Devel mailing list -- devel@ovirt.org
> > To unsubscribe send an email to devel-leave@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IG3QDCTIUSSFZETZJ6UF4MYFFP33VKZP/
>
>
> --
> Tomáš Golembiovský <tgolembi@redhat.com>


--
Tomáš Golembiovský <tgolembi@redhat.com>
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/43QHQBTDCJZB4C3VIZYFDJACOPOBGFIZ/