[ovirt-devel] Re: 4.4 beta - systemd & pids - daemon fails

Sorry for late reply, but I noticed the email just now. I had the same problem some time ago but could not figure out the source of it at that time. I noticed the behavior after some system update and reboot, but neither systemd nor openvswitch had been updated at that moment. Now I noticed that the host where things are broken contains systemd-239-21.el8 from CentOS Stream and host where everything works systemd-239-18.el8_1.5 from normal CentOS. so it is possible that systemd was updated on previous "dnf update" and I just forgot to reboot the host so the problem did not manifest itself. So the question is, are you perhaps using CentOS Stream on your host? As to why I suspect systemd, mostly because the unit file in openvswitch had not changed for a long time. The management of runtime directory is left to systemd (see the RuntimeDirectory* variables in unit file). As per documentation the owner is the same as the User in the unit file, or it defaults to root if not specified. That is our case here. The openvswitch unit file "cleverly" solves it by calling "chown" in ExecStartPre. The trouble is that the permission and owner of the runtime directory are "fixed" by systemd after every Exec* command so the chown does not do its job. It is likely that this behavior changed in systemd between 239-18 and 239-21. My workaround was simply to do that as part of ExecStart like this: ExecStart=/bin/sh -x -c '/usr/bin/chown ${OVS_USER_ID} /var/run/openvswitch /var/log/openvswitch ; /usr/share/openvswitch/scripts/ovs-ctl \ --no-ovs-vswitchd --no-monitor --system-id=random \ ${OVSUSER} \ start $OPTIONS' Hope this helps, Tomas Golebiovsky PS: The reason why you don't see the runtime directory on disk is that it is removed by systemd when the service stops. That's why creating it or changing the permission on it manually makes no sense. Or having it listed as RPM content. On Mon, 4 May 2020 14:51:57 +0100 lejeczek <peljasz(a)yahoo.co.uk&gt; wrote: > On 04/05/2020 13:59, Yedidyah Bar David wrote: > > On Mon, May 4, 2020 at 3:37 PM lejeczek <peljasz(a)yahoo.co.uk&gt; wrote: > >> > >> > >> On 04/05/2020 12:07, Yedidyah Bar David wrote: > >>> On Mon, May 4, 2020 at 1:49 PM lejeczek <peljasz(a)yahoo.co.uk&gt; wrote: > >>>> hi devel, > >>>> > >>>> Some inconsistencies in systemd's daemon configs &/|| binaries? > >>>> > >>>> Starting Open vSwitch Database Unit... > >>>> ovs|00002|daemon_unix|EMER|/var/run/openvswitch/ovsdb-server.pid.tmp: > >>>> create failed (Permission denied) > >>>> Starting ovsdb-server ovsdb-server: > >>>> /var/run/openvswitch/ovsdb-server.pid.tmp: create failed > >>>> (Permission denied) > >>>> [FAILED] > >>>> > >>>> Any quick fix would mind to suggest? > >>> Can you please check/share the output of: > >>> > >>> ls -la /var/run/openvswitch > >>> > >>> rpm -q openvswitch > >>> > >>> Best regards, > >> $ ls -la /var/run/openvswitch > >> ls: cannot access '/var/run/openvswitch > > So it's gone, > > > >> $ rpm -ql openvswitch.x86_64 > >> /etc/bash_completion.d/ovs-appctl-bashcomp.bash > >> /etc/bash_completion.d/ovs-vsctl-bashcomp.bash > >> /etc/logrotate.d/openvswitch > >> /etc/openvswitch > >> /etc/openvswitch/.conf.db.~lock~ > >> /etc/openvswitch/conf.db > >> /etc/openvswitch/default.conf > >> /etc/openvswitch/system-id.conf > >> /etc/sysconfig/openvswitch > >> /run/openvswitch > >> /usr/bin/ovs-appctl > >> /usr/bin/ovs-dpctl > >> /usr/bin/ovs-ofctl > >> /usr/bin/ovs-pki > >> /usr/bin/ovs-vsctl > >> /usr/bin/ovsdb-client > >> /usr/bin/ovsdb-tool > >> /usr/bin/vtep-ctl > >> /usr/lib/.build-id > >> /usr/lib/.build-id/06 > >> /usr/lib/.build-id/06/9fa09762648319fdda1761937dcd4f8e429828 > >> /usr/lib/.build-id/14 > >> /usr/lib/.build-id/14/f86ec775ffd588bd37fd61b932aa250103bfdb > >> /usr/lib/.build-id/1c > >> /usr/lib/.build-id/1c/2c3f9ede44b42a3659cccfc20b17bf51e08f95 > >> /usr/lib/.build-id/34 > >> /usr/lib/.build-id/34/7f712ade5dc04195c71b38c626e192dc2b078b > >> /usr/lib/.build-id/4f > >> /usr/lib/.build-id/4f/d442fdee55254ca3a7236e524536a8ffb73c17 > >> /usr/lib/.build-id/55 > >> /usr/lib/.build-id/55/b43b83a4ac745590014bbc7a4f16204ecb3cba > >> /usr/lib/.build-id/5d > >> /usr/lib/.build-id/5d/f9b2c0ac7c35a59464c9a910e89d71c3387d96 > >> /usr/lib/.build-id/5e > >> /usr/lib/.build-id/5e/886639c0e4841d01cfa9d6a8aff210f297267e > >> /usr/lib/.build-id/6c > >> /usr/lib/.build-id/6c/ad54943c22a0ec765f339e4f4216b51a809dc9 > >> /usr/lib/.build-id/6e > >> /usr/lib/.build-id/6e/cd97bce86b69ce1a9841b84540884ad7b7421f > >> /usr/lib/.build-id/78 > >> /usr/lib/.build-id/78/a71d93990610e97e76a7b1b4acc105e64655a3 > >> /usr/lib/.build-id/a9 > >> /usr/lib/.build-id/a9/62872e830ac6c7db512b8fd17c7ae9d9227e7e > >> /usr/lib/.build-id/aa > >> /usr/lib/.build-id/aa/7a8f137a95a94eedb767aecbed27834ba906a9 > >> /usr/lib/.build-id/dc > >> /usr/lib/.build-id/dc/a8f35e4acbbcf599b63369081d5643ce679ded > >> /usr/lib/.build-id/ee > >> /usr/lib/.build-id/ee/57eca75e3312ed9cad7fbad6977fc06bd4852f > >> /usr/lib/systemd/system/openvswitch.service > >> /usr/lib/systemd/system/ovs-delete-transient-ports.service > >> /usr/lib/systemd/system/ovs-vswitchd.service > >> /usr/lib/systemd/system/ovsdb-server.service > >> /usr/lib/udev/rules.d/91-vfio.rules > >> /usr/lib64/libofproto-2.11.so.0 > >> /usr/lib64/libofproto-2.11.so.0.0.1 > >> /usr/lib64/libopenvswitch-2.11.so.0 > >> /usr/lib64/libopenvswitch-2.11.so.0.0.1 > >> /usr/lib64/libovn-2.11.so.0 > >> /usr/lib64/libovn-2.11.so.0.0.1 > >> /usr/lib64/libovsdb-2.11.so.0 > >> /usr/lib64/libovsdb-2.11.so.0.0.1 > >> /usr/lib64/libsflow-2.11.so.0 > >> /usr/lib64/libsflow-2.11.so.0.0.1 > >> /usr/lib64/libvtep-2.11.so.0 > >> /usr/lib64/libvtep-2.11.so.0.0.1 > >> /usr/sbin/ovs-vswitchd > >> /usr/sbin/ovsdb-server > >> /usr/share/doc/openvswitch > >> /usr/share/doc/openvswitch/LICENSE > >> /usr/share/doc/openvswitch/NEWS > >> /usr/share/doc/openvswitch/NOTICE > >> /usr/share/doc/openvswitch/README.RHEL.rst > >> /usr/share/doc/openvswitch/README.rst > >> /usr/share/man/man1/ovsdb-client.1.gz > >> /usr/share/man/man1/ovsdb-server.1.gz > >> /usr/share/man/man1/ovsdb-tool.1.gz > >> /usr/share/man/man5/ovs-vswitchd.conf.db.5.gz > >> /usr/share/man/man5/ovsdb-server.5.gz > >> /usr/share/man/man5/ovsdb.5.gz > >> /usr/share/man/man5/vtep.5.gz > >> /usr/share/man/man7/ovs-actions.7.gz > >> /usr/share/man/man7/ovs-fields.7.gz > >> /usr/share/man/man7/ovsdb-server.7.gz > >> /usr/share/man/man7/ovsdb.7.gz > >> /usr/share/man/man8/ovs-appctl.8.gz > >> /usr/share/man/man8/ovs-ctl.8.gz > >> /usr/share/man/man8/ovs-dpctl.8.gz > >> /usr/share/man/man8/ovs-kmod-ctl.8.gz > >> /usr/share/man/man8/ovs-ofctl.8.gz > >> /usr/share/man/man8/ovs-parse-backtrace.8.gz > >> /usr/share/man/man8/ovs-pki.8.gz > >> /usr/share/man/man8/ovs-vsctl.8.gz > >> /usr/share/man/man8/ovs-vswitchd.8.gz > >> /usr/share/man/man8/vtep-ctl.8.gz > >> /usr/share/openvswitch/scripts/openvswitch.init > >> /usr/share/openvswitch/scripts/ovs-check-dead-ifs > >> /usr/share/openvswitch/scripts/ovs-ctl > >> /usr/share/openvswitch/scripts/ovs-kmod-ctl > >> /usr/share/openvswitch/scripts/ovs-lib > >> /usr/share/openvswitch/scripts/ovs-save > >> /usr/share/openvswitch/scripts/ovs-systemd-reload > >> /usr/share/openvswitch/scripts/ovs-vtep > >> /usr/share/openvswitch/vswitch.ovsschema > >> /usr/share/openvswitch/vtep.ovsschema > >> /var/lib/openvswitch > >> /var/lib/openvswitch/pki > > ...although the package did include it. > > > > Perhaps check what/who removed it. > > > > To fix, you can try 'yum reinstall openvswitch'. > > > > Best regards, > Nope, trivializing it won't help. > It's something in the either or both, systemd deamons > configs, binaries/configs. > Even a manual creation of the path: > $ mkdir /var/run/openvswitch/; chown openvswitch:hugetlbfs > /var/run/openvswitch/ > Does nothing because very next attempt of: > $ systemctl restart openvswitch.service > and that end path is gone. > > regards, L > _______________________________________________ > Devel mailing list -- devel(a)ovirt.org > To unsubscribe send an email to devel-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/privacy-policy.html > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/IG3QDCTIUSS... -- Tomáš Golembiovský <tgolembi(a)redhat.com&gt;