On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:
Also, try to run the same in mock which would isolate your environment

My environment is minimal CentOS 7 VM installed from scratch, why would mock help me?

Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-(

I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

On Mon, Feb 18, 2019 at 9:28 AM Dafna Ron <dron@redhat.com> wrote:
are you running upgrade from release or basic?


On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <mperina@redhat.com> wrote:


On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <eraviv@redhat.com> wrote:
Martin,
Is following package also installed in you env?
selinux-policy-targeted-3.13.1-229.el7_6.9.noarch

 It is installed in the OST failing environment.

Yes, both selinux-policy and selinux-policy-targeted are installed and they didn't cause any issues


Eitan

On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina@redhat.com> wrote:
Hi,

I'm unable to reproduce that issue outside OST, following scenarios worked without any issues:

Scenario 1
  1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed
  2. Install and configure ovirt-engine 4.2.8
  3. Login to webadmin - everything works fine
  4. Update to selinux-policy-*3.13.1-229.el7_6.9
  5. Login to webadmin - everything works fine
  6. Try to restart ovirt-engine and rh-postgresql95-postgresql services
  7. Login to webadmin - everything works fine
  8. Upgrade all other available packages
  9. Login to webadmin - everything works fine
  10. Reboot the machine
  11. Login to webadmin - everything works fine

Senario 2
  1. Update CentOS to latest version and make sure that selinux-policy-*3.13.1-229.el7_6.9 is installed
  2. Install and configure ovirt-engine 4.2.8
  3. Login to webadmin - everything works fine

So continuing the investigation, but so far it seems to me related only to OST

Martin


On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv@redhat.com> wrote:
Just to add some coal to the fire, here are my findings for failures of the 4.2 OST network suite:

Following the selinux update [0], engine setup fails because what looks like failure of engine to communicate with postgresql.
In [1]:
Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server...
Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied
Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server
Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output.
Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1
Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server.
Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state.
Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed.

and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file:

type=AVC msg=audit(1550363215.978:1067): avc:  denied  { read } for  pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null)
type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730
type=AVC msg=audit(1550363215.978:1068): avc:  denied  { getattr } for  pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null)
type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730
type=AVC msg=audit(1550363215.994:1069): avc:  denied  { getattr } for  pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null)


whereas in [3] - the build just before the selinux package update, these errors did not occur.
Looks like alongside enabling selinux a policy update is required.
thanks

[0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/
[1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/
[2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/audit/audit.log/*view*/
[3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artifact/exported-artifacts/pre-tests/lago-network-suite-4-2-engine/_var_log/messages/*view*/

On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote:
I think this is a regression causing rh-postgress to fail to start on selinux conf.
the issue is probably with the selinux packages

I ran lago locally to debug and ssh-ed to the vms and this is the output from the processes start:

Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9028]: warning: unable to look up public/pickup: No such file or directory
Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9029]: warning: unable to look up public/pickup: No such file or directory
Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob
Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server...
-- Subject: Unit rh-postgresql95-postgresql.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rh-postgresql95-postgresql.service has begun starting up.
Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: pg_ctl: could not start server
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: Examine the log output.
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service: control process exited, code=exited status=1
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Failed to start PostgreSQL database server.
-- Subject: Unit rh-postgresql95-postgresql.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rh-postgresql95-postgresql.service has failed.
--
-- The result is failed.
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state.
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service failed.
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/Authent
Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]: Started Session 51 of user root.
-- Subject: Unit session-51.scope has finished start-up
-- Defined-By: systemd



Secure log:

Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

after setenforce:

root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0
[root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start rh-postgresql95-postgresql.service
[root@lago-upgrade-from-release-suite-master-engine ~]#
[root@lago-upgrade-from-release-suite-master-engine ~]#
[root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status rh-postgresql95-postgresql.service
● rh-postgresql95-postgresql.service - PostgreSQL database server
   Loaded: loaded (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago
  Process: 9137 ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS)
  Process: 9134 ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS)
 Main PID: 9143 (postgres)
   CGroup: /system.slice/rh-postgresql95-postgresql.service
           ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql95/lib/pgsql/data
           ├─9144 postgres: logger process  
           ├─9146 postgres: checkpointer process  
           ├─9147 postgres: writer process  
           ├─9148 postgres: wal writer process  
           ├─9149 postgres: autovacuum launcher process  
           └─9150 postgres: stats collector process  

Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server...
Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: LOG:  redirecting log output to logging collector process
Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: HINT:  Future log output will appear in directory "pg_log".
Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]: Started PostgreSQL database server.
[root@lago-upgrade-from-release-suite-master-engine ~]#

Not sure who deals with this configuration but this is a blocker as upgrade from release is failing for both ovirt-engine and vdsm.

Thanks,
Dafna


On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grosenth@redhat.com> wrote:
Thanks Greg

I will check this 


On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:
Is there any way you can run
"systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe"
like it suggests?
The logs below don't give any indication why it failed to start, afaict.

On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> wrote:
Hi 

I receive this error message both in CQ and check_patch:

2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql
2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None
2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1
2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout:


2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr:
Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details.

2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf''
2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod
    method['method']()
  File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc
    self._provisioning.provision()
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision
    self.restartPG()
  File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG
    state=state,
  File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state
    service=name,
RuntimeError: Failed to start service 'rh-postgresql95-postgresql'
2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql'
2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections:
id uid local foreign state pid exe

What can cause it?

Thanks
Galit
https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change-queue-tester/12916/testReport/junit/(root)/001_initialize_engine/running_tests___upgrade_from_release_suite_el7_x86_64___test_initialize_engine/

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_standard-check-patch/detail/ovirt-system-tests_standard-check-patch/3207/pipeline/101


Regards,
Galit

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat 

galit@gmail.com    T: 972-9-7692230    

_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTCT3HXORRTZ67RVXH653/


--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

gshereme@redhat.com    IRC: gshereme



--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat 

galit@gmail.com    T: 972-9-7692230    

_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKYMT2EL25CN3C26HOU2R/
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJGS5T4QQMRRBDGZG6J4L/


--
Martin Perina
Associate Manager, Software Engineering
Red Hat Czech s.r.o.


--
Martin Perina
Associate Manager, Software Engineering
Red Hat Czech s.r.o.


--
Martin Perina
Associate Manager, Software Engineering
Red Hat Czech s.r.o.