[ovirt-devel] Re: [URGENT] - seems like regression: Re: Re: Misc configuration': Failed to start service 'rh-postgresql95-postgresql

Also, try to run the same in mock which would isolate your environment

On Mon, Feb 18, 2019 at 9:28 AM Dafna Ron <dron(a)redhat.com&gt; wrote: > are you running upgrade from release or basic? > > > On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <mperina(a)redhat.com&gt; wrote: > >> >> >> On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <eraviv(a)redhat.com&gt; wrote: >> >>> Martin, >>> Is following package also installed in you env? >>> >>> selinux-policy-targeted-3.13.1-229.el7_6.9.noarch >>> >>> It is installed in the OST failing environment. >>> >>> >> Yes, both selinux-policy and selinux-policy-targeted are installed and >> they didn't cause any issues >> >>> >>> >>> Eitan >>> >>> >>> On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina(a)redhat.com&gt; >>> wrote: >>> >>>> Hi, >>>> >>>> I'm unable to reproduce that issue outside OST, following scenarios >>>> worked without any issues: >>>> >>>> Scenario 1 >>>> 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed >>>> 2. Install and configure ovirt-engine 4.2.8 >>>> 3. Login to webadmin - everything works fine >>>> 4. Update to selinux-policy-*3.13.1-229.el7_6.9 >>>> 5. Login to webadmin - everything works fine >>>> 6. Try to restart ovirt-engine and rh-postgresql95-postgresql >>>> services >>>> 7. Login to webadmin - everything works fine >>>> 8. Upgrade all other available packages >>>> 9. Login to webadmin - everything works fine >>>> 10. Reboot the machine >>>> 11. Login to webadmin - everything works fine >>>> >>>> Senario 2 >>>> 1. Update CentOS to latest version and make sure that >>>> selinux-policy-*3.13.1-229.el7_6.9 is installed >>>> 2. Install and configure ovirt-engine 4.2.8 >>>> 3. Login to webadmin - everything works fine >>>> >>>> So continuing the investigation, but so far it seems to me related >>>> only to OST >>>> >>>> Martin >>>> >>>> >>>> On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv(a)redhat.com&gt; wrote: >>>> >>>>> Just to add some coal to the fire, here are my findings for failures >>>>> of the 4.2 OST network suite: >>>>> >>>>> Following the selinux update [0], engine setup fails because what >>>>> looks like failure of engine to communicate with postgresql. >>>>> In [1]: >>>>> >>>>> Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... >>>>> Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. >>>>> Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed. >>>>> >>>>> and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file: >>>>> >>>>> type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>> type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>> type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 >>>>> type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>> type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>> type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 >>>>> type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 >>>>> type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null) >>>>> >>>>> whereas in [3] - the build just before the selinux package update, these errors did not occur. >>>>> >>>>> Looks like alongside enabling selinux a policy update is required. >>>>> >>>>> thanks >>>>> >>>>> >>>>> [0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ >>>>> [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/ar... >>>>> [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/ar... >>>>> [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/ar... >>>>> >>>>> >>>>> On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >>>>> >>>>>> I think this is a regression causing rh-postgress to fail to start >>>>>> on selinux conf. >>>>>> the issue is probably with the selinux packages >>>>>> >>>>>> I ran lago locally to debug and ssh-ed to the vms and this is the >>>>>> output from the processes start: >>>>>> >>>>>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >>>>>> postfix/postdrop[9028]: warning: unable to look up public/pickup: No such >>>>>> file or directory >>>>>> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >>>>>> postfix/postdrop[9029]: warning: unable to look up public/pickup: No such >>>>>> file or directory >>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 >>>>>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob >>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Starting PostgreSQL database server... >>>>>> -- Subject: Unit rh-postgresql95-postgresql.service has begun >>>>>> start-up >>>>>> -- Defined-By: systemd >>>>>> -- Support: >>>>>> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >>>>>> -- >>>>>> -- Unit rh-postgresql95-postgresql.service has begun starting up. >>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>> postgresql-ctl[9041]: postgres cannot access the server configuration file >>>>>> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> postgresql-ctl[9041]: pg_ctl: could not start server >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> postgresql-ctl[9041]: Examine the log output. >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: rh-postgresql95-postgresql.service: control process exited, >>>>>> code=exited status=1 >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Failed to start PostgreSQL database server. >>>>>> -- Subject: Unit rh-postgresql95-postgresql.service has failed >>>>>> -- Defined-By: systemd >>>>>> -- Support: >>>>>> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >>>>>> -- >>>>>> -- Unit rh-postgresql95-postgresql.service has failed. >>>>>> -- >>>>>> -- The result is failed. >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: rh-postgresql95-postgresql.service failed. >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> polkitd[2720]: Unregistered Authentication Agent for >>>>>> unix-process:9033:93610 (system bus name :1.160, object path >>>>>> /org/freedesktop/PolicyKit1/Authent >>>>>> Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Started Session 51 of user root. >>>>>> -- Subject: Unit session-51.scope has finished start-up >>>>>> -- Defined-By: systemd >>>>>> >>>>>> >>>>>> >>>>>> Secure log: >>>>>> >>>>>> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >>>>>> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 >>>>>> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], >>>>>> object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale >>>>>> en_US.UTF-8) >>>>>> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >>>>>> polkitd[2720]: Unregistered Authentication Agent for >>>>>> unix-process:9033:93610 (system bus name :1.160, object path >>>>>> /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) >>>>>> (disconnected from bus) >>>>>> >>>>>> after setenforce: >>>>>> >>>>>> root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 >>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >>>>>> start rh-postgresql95-postgresql.service >>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >>>>>> status rh-postgresql95-postgresql.service >>>>>> ● rh-postgresql95-postgresql.service - PostgreSQL database server >>>>>> Loaded: loaded >>>>>> (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; >>>>>> vendor preset: disabled) >>>>>> Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago >>>>>> Process: 9137 >>>>>> ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D >>>>>> ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) >>>>>> Process: 9134 >>>>>> ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir >>>>>> %N (code=exited, status=0/SUCCESS) >>>>>> Main PID: 9143 (postgres) >>>>>> CGroup: /system.slice/rh-postgresql95-postgresql.service >>>>>> ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D >>>>>> /var/opt/rh/rh-postgresql95/lib/pgsql/data >>>>>> ├─9144 postgres: logger process >>>>>> ├─9146 postgres: checkpointer process >>>>>> ├─9147 postgres: writer process >>>>>> ├─9148 postgres: wal writer process >>>>>> ├─9149 postgres: autovacuum launcher process >>>>>> └─9150 postgres: stats collector process >>>>>> >>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Starting PostgreSQL database server... >>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>> postgresql-ctl[9137]: LOG: redirecting log output to logging collector >>>>>> process >>>>>> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >>>>>> postgresql-ctl[9137]: HINT: Future log output will appear in directory >>>>>> "pg_log". >>>>>> Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine >>>>>> systemd[1]: Started PostgreSQL database server. >>>>>> [root@lago-upgrade-from-release-suite-master-engine ~]# >>>>>> >>>>>> Not sure who deals with this configuration but this is a blocker as >>>>>> upgrade from release is failing for both ovirt-engine and vdsm. >>>>>> >>>>>> Thanks, >>>>>> Dafna >>>>>> >>>>>> >>>>>> On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal < >>>>>> grosenth(a)redhat.com&gt; wrote: >>>>>> >>>>>>> Thanks Greg >>>>>>> >>>>>>> I will check this >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta < >>>>>>> gshereme(a)redhat.com&gt; wrote: >>>>>>> >>>>>>>> Is there any way you can run >>>>>>>> "systemctl status rh-postgresql95-postgresql.service" and >>>>>>>> "journalctl -xe" >>>>>>>> like it suggests? >>>>>>>> The logs below don't give any indication why it failed to start, >>>>>>>> afaict. >>>>>>>> >>>>>>>> On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal < >>>>>>>> grosenth(a)redhat.com&gt; wrote: >>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> I receive this error message both in CQ and check_patch: >>>>>>>>> >>>>>>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql >>>>>>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None >>>>>>>>> 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 >>>>>>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout: >>>>>>>>> >>>>>>>>> >>>>>>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: >>>>>>>>> Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details. >>>>>>>>> >>>>>>>>> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' >>>>>>>>> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>>>>>> method['method']() >>>>>>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc >>>>>>>>> self._provisioning.provision() >>>>>>>>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision >>>>>>>>> self.restartPG() >>>>>>>>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG >>>>>>>>> state=state, >>>>>>>>> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state >>>>>>>>> service=name, >>>>>>>>> RuntimeError: Failed to start service 'rh-postgresql95-postgresql' >>>>>>>>> 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' >>>>>>>>> 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: >>>>>>>>> id uid local foreign state pid exe >>>>>>>>> >>>>>>>>> >>>>>>>>> What can cause it? >>>>>>>>> >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> >>>>>>>>> Galit >>>>>>>>> >>>>>>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_cha... >>>>>>>>> >>>>>>>>> >>>>>>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_s... >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Galit >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> GALIT ROSENTHAL >>>>>>>>> >>>>>>>>> SOFTWARE ENGINEER >>>>>>>>> >>>>>>>>> Red Hat >>>>>>>>> >>>>>>>>> <https://www.redhat.com/> >>>>>>>>> >>>>>>>>> galit(a)gmail.com T: 972-9-7692230 >>>>>>>>> <https://red.ht/sig> >>>>>>>>> _______________________________________________ >>>>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>>>> oVirt Code of Conduct: >>>>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>>>> List Archives: >>>>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPE... >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> GREG SHEREMETA >>>>>>>> >>>>>>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>>>>>> >>>>>>>> Red Hat NA >>>>>>>> >>>>>>>> <https://www.redhat.com/> >>>>>>>> >>>>>>>> gshereme(a)redhat.com IRC: gshereme >>>>>>>> <https://red.ht/sig> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> GALIT ROSENTHAL >>>>>>> >>>>>>> SOFTWARE ENGINEER >>>>>>> >>>>>>> Red Hat >>>>>>> >>>>>>> <https://www.redhat.com/> >>>>>>> >>>>>>> galit(a)gmail.com T: 972-9-7692230 >>>>>>> <https://red.ht/sig> >>>>>>> _______________________________________________ >>>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>>> oVirt Code of Conduct: >>>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>>> List Archives: >>>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTO... >>>>>>> >>>>>> _______________________________________________ >>>>>> Devel mailing list -- devel(a)ovirt.org >>>>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>>> oVirt Code of Conduct: >>>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>>> List Archives: >>>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6Z... >>>>>> >>>>> >>>> >>>> -- >>>> Martin Perina >>>> Associate Manager, Software Engineering >>>> Red Hat Czech s.r.o. >>>> >>> >> >> -- >> Martin Perina >> Associate Manager, Software Engineering >> Red Hat Czech s.r.o. >> >