Thanks Greg I will check this On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:

Is there any way you can run "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" like it suggests? The logs below don't give any indication why it failed to start, afaict.

On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Hi

I receive this error message both in CQ and check_patch:

2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout:

2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details.

2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc self._provisioning.provision() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision self.restartPG() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG state=state, File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe

What can cause it?

Thanks

Galit

https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change...

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...

Regards,

Galit

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC...

--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

<https://www.redhat.com/>

gshereme@redhat.com IRC: gshereme <https://red.ht/sig>

-- GALIT ROSENTHAL SOFTWARE ENGINEER Red Hat <https://www.redhat.com/> galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>

Just to add some coal to the fire, here are my findings for failures of the 4.2 OST network suite: Following the selinux update [0], engine setup fails because what looks like failure of engine to communicate with postgresql. In [1]: Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed. and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file: type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null) whereas in [3] - the build just before the selinux package update, these errors did not occur. Looks like alongside enabling selinux a policy update is required. thanks [0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artif... On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote:

I think this is a regression causing rh-postgress to fail to start on selinux conf. the issue is probably with the selinux packages

I ran lago locally to debug and ssh-ed to the vms and this is the output from the processes start:

Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9028]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9029]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... -- Subject: Unit rh-postgresql95-postgresql.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has begun starting up. Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: pg_ctl: could not start server Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: Examine the log output. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Failed to start PostgreSQL database server. -- Subject: Unit rh-postgresql95-postgresql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has failed. -- -- The result is failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/Authent Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]: Started Session 51 of user root. -- Subject: Unit session-51.scope has finished start-up -- Defined-By: systemd

Secure log:

Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

after setenforce:

root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start rh-postgresql95-postgresql.service [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status rh-postgresql95-postgresql.service ● rh-postgresql95-postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago Process: 9137 ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) Process: 9134 ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS) Main PID: 9143 (postgres) CGroup: /system.slice/rh-postgresql95-postgresql.service ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql95/lib/pgsql/data ├─9144 postgres: logger process ├─9146 postgres: checkpointer process ├─9147 postgres: writer process ├─9148 postgres: wal writer process ├─9149 postgres: autovacuum launcher process └─9150 postgres: stats collector process

Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: LOG: redirecting log output to logging collector process Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: HINT: Future log output will appear in directory "pg_log". Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]: Started PostgreSQL database server. [root@lago-upgrade-from-release-suite-master-engine ~]#

Not sure who deals with this configuration but this is a blocker as upgrade from release is failing for both ovirt-engine and vdsm.

Thanks, Dafna

On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Thanks Greg

I will check this

On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:

...

Is there any way you can run "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" like it suggests? The logs below don't give any indication why it failed to start, afaict.

On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Hi

I receive this error message both in CQ and check_patch:

2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout:

2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details.

2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc self._provisioning.provision() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision self.restartPG() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG state=state, File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe

What can cause it?

Thanks

Galit

https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change...

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...

Regards,

Galit

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC...

--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

<https://www.redhat.com/>

gshereme@redhat.com IRC: gshereme <https://red.ht/sig>

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJG...

Martin, Is following package also installed in you env? selinux-policy-targeted-3.13.1-229.el7_6.9.noarch It is installed in the OST failing environment. Eitan On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina@redhat.com> wrote:

Hi,

I'm unable to reproduce that issue outside OST, following scenarios worked without any issues:

Scenario 1 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine 4. Update to selinux-policy-*3.13.1-229.el7_6.9 5. Login to webadmin - everything works fine 6. Try to restart ovirt-engine and rh-postgresql95-postgresql services 7. Login to webadmin - everything works fine 8. Upgrade all other available packages 9. Login to webadmin - everything works fine 10. Reboot the machine 11. Login to webadmin - everything works fine

Senario 2 1. Update CentOS to latest version and make sure that selinux-policy-*3.13.1-229.el7_6.9 is installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine

So continuing the investigation, but so far it seems to me related only to OST

Martin

On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Just to add some coal to the fire, here are my findings for failures of the 4.2 OST network suite:

Following the selinux update [0], engine setup fails because what looks like failure of engine to communicate with postgresql. In [1]:

Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed.

and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file:

type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null)

whereas in [3] - the build just before the selinux package update, these errors did not occur.

Looks like alongside enabling selinux a policy update is required.

thanks

[0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artif...

On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote:

...

I think this is a regression causing rh-postgress to fail to start on selinux conf. the issue is probably with the selinux packages

I ran lago locally to debug and ssh-ed to the vms and this is the output from the processes start:

Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9028]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9029]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... -- Subject: Unit rh-postgresql95-postgresql.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has begun starting up. Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: pg_ctl: could not start server Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: Examine the log output. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Failed to start PostgreSQL database server. -- Subject: Unit rh-postgresql95-postgresql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has failed. -- -- The result is failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/Authent Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]: Started Session 51 of user root. -- Subject: Unit session-51.scope has finished start-up -- Defined-By: systemd

Secure log:

Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

after setenforce:

root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start rh-postgresql95-postgresql.service [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status rh-postgresql95-postgresql.service ● rh-postgresql95-postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago Process: 9137 ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) Process: 9134 ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS) Main PID: 9143 (postgres) CGroup: /system.slice/rh-postgresql95-postgresql.service ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql95/lib/pgsql/data ├─9144 postgres: logger process ├─9146 postgres: checkpointer process ├─9147 postgres: writer process ├─9148 postgres: wal writer process ├─9149 postgres: autovacuum launcher process └─9150 postgres: stats collector process

Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: LOG: redirecting log output to logging collector process Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: HINT: Future log output will appear in directory "pg_log". Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]: Started PostgreSQL database server. [root@lago-upgrade-from-release-suite-master-engine ~]#

Not sure who deals with this configuration but this is a blocker as upgrade from release is failing for both ovirt-engine and vdsm.

Thanks, Dafna

On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Thanks Greg

I will check this

On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:

...

Is there any way you can run "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" like it suggests? The logs below don't give any indication why it failed to start, afaict.

On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Hi

I receive this error message both in CQ and check_patch:

2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout:

2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details.

2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc self._provisioning.provision() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision self.restartPG() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG state=state, File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe

What can cause it?

Thanks

Galit

https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change...

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...

Regards,

Galit

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC...

--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

<https://www.redhat.com/>

gshereme@redhat.com IRC: gshereme <https://red.ht/sig>

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJG...

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

are you running upgrade from release or basic? On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <mperina@redhat.com> wrote:

On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Martin, Is following package also installed in you env?

selinux-policy-targeted-3.13.1-229.el7_6.9.noarch

It is installed in the OST failing environment.

Yes, both selinux-policy and selinux-policy-targeted are installed and they didn't cause any issues

...

Eitan

On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina@redhat.com> wrote:

...

Hi,

I'm unable to reproduce that issue outside OST, following scenarios worked without any issues:

Scenario 1 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine 4. Update to selinux-policy-*3.13.1-229.el7_6.9 5. Login to webadmin - everything works fine 6. Try to restart ovirt-engine and rh-postgresql95-postgresql services 7. Login to webadmin - everything works fine 8. Upgrade all other available packages 9. Login to webadmin - everything works fine 10. Reboot the machine 11. Login to webadmin - everything works fine

Senario 2 1. Update CentOS to latest version and make sure that selinux-policy-*3.13.1-229.el7_6.9 is installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine

So continuing the investigation, but so far it seems to me related only to OST

Martin

On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Just to add some coal to the fire, here are my findings for failures of the 4.2 OST network suite:

Following the selinux update [0], engine setup fails because what looks like failure of engine to communicate with postgresql. In [1]:

Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed.

and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file:

type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null)

whereas in [3] - the build just before the selinux package update, these errors did not occur.

Looks like alongside enabling selinux a policy update is required.

thanks

[0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artif...

On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote:

...

I think this is a regression causing rh-postgress to fail to start on selinux conf. the issue is probably with the selinux packages

I ran lago locally to debug and ssh-ed to the vms and this is the output from the processes start:

Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9028]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9029]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... -- Subject: Unit rh-postgresql95-postgresql.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has begun starting up. Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: pg_ctl: could not start server Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: Examine the log output. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Failed to start PostgreSQL database server. -- Subject: Unit rh-postgresql95-postgresql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has failed. -- -- The result is failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/Authent Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]: Started Session 51 of user root. -- Subject: Unit session-51.scope has finished start-up -- Defined-By: systemd

Secure log:

Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

after setenforce:

root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start rh-postgresql95-postgresql.service [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status rh-postgresql95-postgresql.service ● rh-postgresql95-postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago Process: 9137 ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) Process: 9134 ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS) Main PID: 9143 (postgres) CGroup: /system.slice/rh-postgresql95-postgresql.service ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql95/lib/pgsql/data ├─9144 postgres: logger process ├─9146 postgres: checkpointer process ├─9147 postgres: writer process ├─9148 postgres: wal writer process ├─9149 postgres: autovacuum launcher process └─9150 postgres: stats collector process

Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: LOG: redirecting log output to logging collector process Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: HINT: Future log output will appear in directory "pg_log". Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]: Started PostgreSQL database server. [root@lago-upgrade-from-release-suite-master-engine ~]#

Not sure who deals with this configuration but this is a blocker as upgrade from release is failing for both ovirt-engine and vdsm.

Thanks, Dafna

On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Thanks Greg

I will check this

On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:

> Is there any way you can run > "systemctl status rh-postgresql95-postgresql.service" and > "journalctl -xe" > like it suggests? > The logs below don't give any indication why it failed to start, > afaict. > > On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> > wrote: > >> Hi >> >> I receive this error message both in CQ and check_patch: >> >> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql >> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None >> 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 >> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout: >> >> >> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: >> Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details. >> >> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' >> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >> method['method']() >> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc >> self._provisioning.provision() >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision >> self.restartPG() >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG >> state=state, >> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state >> service=name, >> RuntimeError: Failed to start service 'rh-postgresql95-postgresql' >> 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' >> 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: >> id uid local foreign state pid exe >> >> >> What can cause it? >> >> >> Thanks >> >> Galit >> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >> >> >> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... >> >> >> >> Regards, >> >> Galit >> >> >> -- >> >> GALIT ROSENTHAL >> >> SOFTWARE ENGINEER >> >> Red Hat >> >> <https://www.redhat.com/> >> >> galit@gmail.com T: 972-9-7692230 >> <https://red.ht/sig> >> _______________________________________________ >> Devel mailing list -- devel@ovirt.org >> To unsubscribe send an email to devel-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC... >> > > > -- > > GREG SHEREMETA > > SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX > > Red Hat NA > > <https://www.redhat.com/> > > gshereme@redhat.com IRC: gshereme > <https://red.ht/sig> >

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig> _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJG...

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

Also, try to run the same in mock which would isolate your environment On Mon, Feb 18, 2019 at 9:28 AM Dafna Ron <dron@redhat.com> wrote:

are you running upgrade from release or basic?

On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <mperina@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Martin, Is following package also installed in you env?

selinux-policy-targeted-3.13.1-229.el7_6.9.noarch

It is installed in the OST failing environment.

Yes, both selinux-policy and selinux-policy-targeted are installed and they didn't cause any issues

...

Eitan

On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina@redhat.com> wrote:

...

Hi,

I'm unable to reproduce that issue outside OST, following scenarios worked without any issues:

Scenario 1 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine 4. Update to selinux-policy-*3.13.1-229.el7_6.9 5. Login to webadmin - everything works fine 6. Try to restart ovirt-engine and rh-postgresql95-postgresql services 7. Login to webadmin - everything works fine 8. Upgrade all other available packages 9. Login to webadmin - everything works fine 10. Reboot the machine 11. Login to webadmin - everything works fine

Senario 2 1. Update CentOS to latest version and make sure that selinux-policy-*3.13.1-229.el7_6.9 is installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine

So continuing the investigation, but so far it seems to me related only to OST

Martin

On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Just to add some coal to the fire, here are my findings for failures of the 4.2 OST network suite:

Following the selinux update [0], engine setup fails because what looks like failure of engine to communicate with postgresql. In [1]:

Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed.

and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file:

type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null)

whereas in [3] - the build just before the selinux package update, these errors did not occur.

Looks like alongside enabling selinux a policy update is required.

thanks

[0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artif...

On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote:

...

I think this is a regression causing rh-postgress to fail to start on selinux conf. the issue is probably with the selinux packages

I ran lago locally to debug and ssh-ed to the vms and this is the output from the processes start:

Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9028]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine postfix/postdrop[9029]: warning: unable to look up public/pickup: No such file or directory Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... -- Subject: Unit rh-postgresql95-postgresql.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has begun starting up. Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: pg_ctl: could not start server Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9041]: Examine the log output. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Failed to start PostgreSQL database server. -- Subject: Unit rh-postgresql95-postgresql.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit rh-postgresql95-postgresql.service has failed. -- -- The result is failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine systemd[1]: rh-postgresql95-postgresql.service failed. Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/Authent Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine systemd[1]: Started Session 51 of user root. -- Subject: Unit session-51.scope has finished start-up -- Defined-By: systemd

Secure log:

Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine polkitd[2720]: Unregistered Authentication Agent for unix-process:9033:93610 (system bus name :1.160, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

after setenforce:

root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl start rh-postgresql95-postgresql.service [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl status rh-postgresql95-postgresql.service ● rh-postgresql95-postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; vendor preset: disabled) Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s ago Process: 9137 ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) Process: 9134 ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir %N (code=exited, status=0/SUCCESS) Main PID: 9143 (postgres) CGroup: /system.slice/rh-postgresql95-postgresql.service ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D /var/opt/rh/rh-postgresql95/lib/pgsql/data ├─9144 postgres: logger process ├─9146 postgres: checkpointer process ├─9147 postgres: writer process ├─9148 postgres: wal writer process ├─9149 postgres: autovacuum launcher process └─9150 postgres: stats collector process

Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine systemd[1]: Starting PostgreSQL database server... Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: LOG: redirecting log output to logging collector process Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine postgresql-ctl[9137]: HINT: Future log output will appear in directory "pg_log". Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine systemd[1]: Started PostgreSQL database server. [root@lago-upgrade-from-release-suite-master-engine ~]#

Not sure who deals with this configuration but this is a blocker as upgrade from release is failing for both ovirt-engine and vdsm.

Thanks, Dafna

On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal <grosenth@redhat.com> wrote:

> Thanks Greg > > I will check this > > > On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> > wrote: > >> Is there any way you can run >> "systemctl status rh-postgresql95-postgresql.service" and >> "journalctl -xe" >> like it suggests? >> The logs below don't give any indication why it failed to start, >> afaict. >> >> On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal < >> grosenth@redhat.com> wrote: >> >>> Hi >>> >>> I receive this error message both in CQ and check_patch: >>> >>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql >>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None >>> 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 >>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout: >>> >>> >>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: >>> Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details. >>> >>> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' >>> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception >>> Traceback (most recent call last): >>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>> method['method']() >>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc >>> self._provisioning.provision() >>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision >>> self.restartPG() >>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG >>> state=state, >>> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state >>> service=name, >>> RuntimeError: Failed to start service 'rh-postgresql95-postgresql' >>> 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' >>> 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: >>> id uid local foreign state pid exe >>> >>> >>> What can cause it? >>> >>> >>> Thanks >>> >>> Galit >>> >>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>> >>> >>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... >>> >>> >>> >>> Regards, >>> >>> Galit >>> >>> >>> -- >>> >>> GALIT ROSENTHAL >>> >>> SOFTWARE ENGINEER >>> >>> Red Hat >>> >>> <https://www.redhat.com/> >>> >>> galit@gmail.com T: 972-9-7692230 >>> <https://red.ht/sig> >>> _______________________________________________ >>> Devel mailing list -- devel@ovirt.org >>> To unsubscribe send an email to devel-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC... >>> >> >> >> -- >> >> GREG SHEREMETA >> >> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >> >> Red Hat NA >> >> <https://www.redhat.com/> >> >> gshereme@redhat.com IRC: gshereme >> <https://red.ht/sig> >> > > > -- > > GALIT ROSENTHAL > > SOFTWARE ENGINEER > > Red Hat > > <https://www.redhat.com/> > > galit@gmail.com T: 972-9-7692230 > <https://red.ht/sig> > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY... > _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJG...

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

Il giorno lun 18 feb 2019 alle ore 10:37 Martin Perina <mperina@redhat.com> ha scritto:

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:

...

Also, try to run the same in mock which would isolate your environment

My environment is minimal CentOS 7 VM installed from scratch, why would mock help me?

Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-(

I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

I guess we'll have to fallback to manual testing for 4.3.1 release since OST doesn't seem to be reliable right now.

...

On Mon, Feb 18, 2019 at 9:28 AM Dafna Ron <dron@redhat.com> wrote:

...

are you running upgrade from release or basic?

On Mon, Feb 18, 2019 at 8:46 AM Martin Perina <mperina@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 9:35 AM Eitan Raviv <eraviv@redhat.com> wrote:

...

Martin, Is following package also installed in you env?

selinux-policy-targeted-3.13.1-229.el7_6.9.noarch

It is installed in the OST failing environment.

Yes, both selinux-policy and selinux-policy-targeted are installed and they didn't cause any issues

...

Eitan

On Mon, Feb 18, 2019 at 9:56 AM Martin Perina <mperina@redhat.com> wrote:

...

Hi,

I'm unable to reproduce that issue outside OST, following scenarios worked without any issues:

Scenario 1 1. Make sure that selinux-policy-*3.13.1-229.el7_6.9 is not installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine 4. Update to selinux-policy-*3.13.1-229.el7_6.9 5. Login to webadmin - everything works fine 6. Try to restart ovirt-engine and rh-postgresql95-postgresql services 7. Login to webadmin - everything works fine 8. Upgrade all other available packages 9. Login to webadmin - everything works fine 10. Reboot the machine 11. Login to webadmin - everything works fine

Senario 2 1. Update CentOS to latest version and make sure that selinux-policy-*3.13.1-229.el7_6.9 is installed 2. Install and configure ovirt-engine 4.2.8 3. Login to webadmin - everything works fine

So continuing the investigation, but so far it seems to me related only to OST

Martin

On Mon, Feb 18, 2019 at 7:39 AM Eitan Raviv <eraviv@redhat.com> wrote:

> Just to add some coal to the fire, here are my findings for failures > of the 4.2 OST network suite: > > Following the selinux update [0], engine setup fails because what > looks like failure of engine to communicate with postgresql. > In [1]: > > Feb 16 19:26:55 lago-network-suite-4-2-engine systemd: Starting PostgreSQL database server... > Feb 16 19:26:55 lago-network-suite-4-2-engine postgresql-ctl: postgres cannot access the server configuration file "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission denied > Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: pg_ctl: could not start server > Feb 16 19:26:56 lago-network-suite-4-2-engine postgresql-ctl: Examine the log output. > Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service: control process exited, code=exited status=1 > Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Failed to start PostgreSQL database server. > Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: Unit rh-postgresql95-postgresql.service entered failed state. > Feb 16 19:26:56 lago-network-suite-4-2-engine systemd: rh-postgresql95-postgresql.service failed. > > and in [2] there are selinux access denials for pg_ctl to read the postgres.conf file: > > type=AVC msg=audit(1550363215.978:1067): avc: denied { read } for pid=8648 comm="pg_ctl" name="postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 > type=SYSCALL msg=audit(1550363215.978:1067): arch=c000003e syscall=2 success=no exit=-13 a0=7ffe611ff730 a1=0 a2=1b6 a3=24 items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) > type=PROCTITLE msg=audit(1550363215.978:1067): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 > type=AVC msg=audit(1550363215.978:1068): avc: denied { getattr } for pid=8648 comm="pg_ctl" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/PG_VERSION" dev="vda4" ino=888709 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 > type=SYSCALL msg=audit(1550363215.978:1068): arch=c000003e syscall=4 success=no exit=-13 a0=60a640 a1=7ffe611ffa50 a2=7ffe611ffa50 a3=2f62696c2f35396c items=0 ppid=1 pid=8648 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="pg_ctl" exe="/opt/rh/rh-postgresql95/root/usr/bin/pg_ctl" subj=system_u:system_r:postgresql_t:s0 key=(null) > type=PROCTITLE msg=audit(1550363215.978:1068): proctitle=2F6F70742F72682F72682D706F737467726573716C39352F726F6F742F7573722F62696E2F70675F63746C007374617274002D44002F7661722F6F70742F72682F72682D706F737467726573716C39352F6C69622F706773716C2F64617461002D73002D77002D7400323730 > type=AVC msg=audit(1550363215.994:1069): avc: denied { getattr } for pid=8654 comm="postgres" path="/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf" dev="vda4" ino=888710 scontext=system_u:system_r:postgresql_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file permissive=0 > type=SYSCALL msg=audit(1550363215.994:1069): arch=c000003e syscall=4 success=no exit=-13 a0=1d862b0 a1=7fff91968710 a2=7fff91968710 a3=2f62696c2f35396c items=0 ppid=8648 pid=8654 auid=4294967295 uid=26 gid=26 euid=26 suid=26 fsuid=26 egid=26 sgid=26 fsgid=26 tty=(none) ses=4294967295 comm="postgres" exe="/opt/rh/rh-postgresql95/root/usr/bin/postgres" subj=system_u:system_r:postgresql_t:s0 key=(null) > > whereas in [3] - the build just before the selinux package update, these errors did not occur. > > Looks like alongside enabling selinux a policy update is required. > > thanks > > > [0] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/900/ > [1] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... > [2] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/901/artif... > [3] https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/899/artif... > > > On Sun, Feb 17, 2019 at 11:16 PM Dafna Ron <dron@redhat.com> wrote: > >> I think this is a regression causing rh-postgress to fail to start >> on selinux conf. >> the issue is probably with the selinux packages >> >> I ran lago locally to debug and ssh-ed to the vms and this is the >> output from the processes start: >> >> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >> postfix/postdrop[9028]: warning: unable to look up public/pickup: No such >> file or directory >> Feb 17 16:02:01 lago-upgrade-from-release-suite-master-engine >> postfix/postdrop[9029]: warning: unable to look up public/pickup: No such >> file or directory >> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 >> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], ob >> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Starting PostgreSQL database server... >> -- Subject: Unit rh-postgresql95-postgresql.service has begun >> start-up >> -- Defined-By: systemd >> -- Support: >> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >> -- >> -- Unit rh-postgresql95-postgresql.service has begun starting up. >> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >> postgresql-ctl[9041]: postgres cannot access the server configuration file >> "/var/opt/rh/rh-postgresql95/lib/pgsql/data/postgresql.conf": Permission d >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> postgresql-ctl[9041]: pg_ctl: could not start server >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> postgresql-ctl[9041]: Examine the log output. >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> systemd[1]: rh-postgresql95-postgresql.service: control process exited, >> code=exited status=1 >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Failed to start PostgreSQL database server. >> -- Subject: Unit rh-postgresql95-postgresql.service has failed >> -- Defined-By: systemd >> -- Support: >> http://lists.freedesktop.org/mailman/listinfo/systemd-devel >> -- >> -- Unit rh-postgresql95-postgresql.service has failed. >> -- >> -- The result is failed. >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Unit rh-postgresql95-postgresql.service entered failed state. >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> systemd[1]: rh-postgresql95-postgresql.service failed. >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> polkitd[2720]: Unregistered Authentication Agent for >> unix-process:9033:93610 (system bus name :1.160, object path >> /org/freedesktop/PolicyKit1/Authent >> Feb 17 16:03:01 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Started Session 51 of user root. >> -- Subject: Unit session-51.scope has finished start-up >> -- Defined-By: systemd >> >> >> >> Secure log: >> >> Feb 17 16:02:34 lago-upgrade-from-release-suite-master-engine >> polkitd[2720]: Registered Authentication Agent for unix-process:9033:93610 >> (system bus name :1.160 [/usr/bin/pkttyagent --notify-fd 5 --fallback], >> object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale >> en_US.UTF-8) >> Feb 17 16:02:35 lago-upgrade-from-release-suite-master-engine >> polkitd[2720]: Unregistered Authentication Agent for >> unix-process:9033:93610 (system bus name :1.160, object path >> /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) >> (disconnected from bus) >> >> after setenforce: >> >> root@lago-upgrade-from-release-suite-master-engine ~]# setenforce 0 >> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >> start rh-postgresql95-postgresql.service >> [root@lago-upgrade-from-release-suite-master-engine ~]# >> [root@lago-upgrade-from-release-suite-master-engine ~]# >> [root@lago-upgrade-from-release-suite-master-engine ~]# systemctl >> status rh-postgresql95-postgresql.service >> ● rh-postgresql95-postgresql.service - PostgreSQL database server >> Loaded: loaded >> (/usr/lib/systemd/system/rh-postgresql95-postgresql.service; disabled; >> vendor preset: disabled) >> Active: active (running) since Sun 2019-02-17 16:08:18 EST; 7s >> ago >> Process: 9137 >> ExecStart=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-ctl start -D >> ${PGDATA} -s -w -t ${PGSTARTTIMEOUT} (code=exited, status=0/SUCCESS) >> Process: 9134 >> ExecStartPre=/opt/rh/rh-postgresql95/root/usr/libexec/postgresql-check-db-dir >> %N (code=exited, status=0/SUCCESS) >> Main PID: 9143 (postgres) >> CGroup: /system.slice/rh-postgresql95-postgresql.service >> ├─9143 /opt/rh/rh-postgresql95/root/usr/bin/postgres -D >> /var/opt/rh/rh-postgresql95/lib/pgsql/data >> ├─9144 postgres: logger process >> ├─9146 postgres: checkpointer process >> ├─9147 postgres: writer process >> ├─9148 postgres: wal writer process >> ├─9149 postgres: autovacuum launcher process >> └─9150 postgres: stats collector process >> >> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Starting PostgreSQL database server... >> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >> postgresql-ctl[9137]: LOG: redirecting log output to logging collector >> process >> Feb 17 16:08:17 lago-upgrade-from-release-suite-master-engine >> postgresql-ctl[9137]: HINT: Future log output will appear in directory >> "pg_log". >> Feb 17 16:08:18 lago-upgrade-from-release-suite-master-engine >> systemd[1]: Started PostgreSQL database server. >> [root@lago-upgrade-from-release-suite-master-engine ~]# >> >> Not sure who deals with this configuration but this is a blocker as >> upgrade from release is failing for both ovirt-engine and vdsm. >> >> Thanks, >> Dafna >> >> >> On Sun, Feb 17, 2019 at 10:55 AM Galit Rosenthal < >> grosenth@redhat.com> wrote: >> >>> Thanks Greg >>> >>> I will check this >>> >>> >>> On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta < >>> gshereme@redhat.com> wrote: >>> >>>> Is there any way you can run >>>> "systemctl status rh-postgresql95-postgresql.service" and >>>> "journalctl -xe" >>>> like it suggests? >>>> The logs below don't give any indication why it failed to start, >>>> afaict. >>>> >>>> On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal < >>>> grosenth@redhat.com> wrote: >>>> >>>>> Hi >>>>> >>>>> I receive this error message both in CQ and check_patch: >>>>> >>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql >>>>> 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None >>>>> 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 >>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout: >>>>> >>>>> >>>>> 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: >>>>> Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details. >>>>> >>>>> 2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' >>>>> 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod >>>>> method['method']() >>>>> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc >>>>> self._provisioning.provision() >>>>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision >>>>> self.restartPG() >>>>> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG >>>>> state=state, >>>>> File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state >>>>> service=name, >>>>> RuntimeError: Failed to start service 'rh-postgresql95-postgresql' >>>>> 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' >>>>> 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: >>>>> id uid local foreign state pid exe >>>>> >>>>> >>>>> What can cause it? >>>>> >>>>> >>>>> Thanks >>>>> >>>>> Galit >>>>> >>>>> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change... >>>>> >>>>> >>>>> https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan... >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> Galit >>>>> >>>>> >>>>> -- >>>>> >>>>> GALIT ROSENTHAL >>>>> >>>>> SOFTWARE ENGINEER >>>>> >>>>> Red Hat >>>>> >>>>> <https://www.redhat.com/> >>>>> >>>>> galit@gmail.com T: 972-9-7692230 >>>>> <https://red.ht/sig> >>>>> _______________________________________________ >>>>> Devel mailing list -- devel@ovirt.org >>>>> To unsubscribe send an email to devel-leave@ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: >>>>> https://www.ovirt.org/community/about/community-guidelines/ >>>>> List Archives: >>>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC... >>>>> >>>> >>>> >>>> -- >>>> >>>> GREG SHEREMETA >>>> >>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>> >>>> Red Hat NA >>>> >>>> <https://www.redhat.com/> >>>> >>>> gshereme@redhat.com IRC: gshereme >>>> <https://red.ht/sig> >>>> >>> >>> >>> -- >>> >>> GALIT ROSENTHAL >>> >>> SOFTWARE ENGINEER >>> >>> Red Hat >>> >>> <https://www.redhat.com/> >>> >>> galit@gmail.com T: 972-9-7692230 >>> <https://red.ht/sig> >>> _______________________________________________ >>> Devel mailing list -- devel@ovirt.org >>> To unsubscribe send an email to devel-leave@ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: >>> https://www.ovirt.org/community/about/community-guidelines/ >>> List Archives: >>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY... >>> >> _______________________________________________ >> Devel mailing list -- devel@ovirt.org >> To unsubscribe send an email to devel-leave@ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/CSNQENF4J6ZQJG... >> >

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

-- Martin Perina Associate Manager, Software Engineering Red Hat Czech s.r.o.

-- SANDRO BONAZZOLA MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo@redhat.com <https://red.ht/sig>

On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:

...

Also, try to run the same in mock which would isolate your environment

My environment is minimal CentOS 7 VM installed from scratch, why would mock help me?

Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-(

I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment: https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6 If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253). Best regards, -- Didi

On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote:

On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote:

...

On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com>

wrote:

...

...

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:

...

Also, try to run the same in mock which would isolate your

environment

...

My environment is minimal CentOS 7 VM installed from scratch, why

would mock help me?

...

Unfortunately I have FC29 on laptop and I haven't been able to run

OST on it so far :-(

...

I have been discussing with Galit over IRC and so far we haven't been

able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6

If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253).

Galit sent a WA which seems to work, do we want to merge it? [1]

Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites. I am working on a workaround in engine-setup.

or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue?

I think it's not related, not sure.

[1] https://gerrit.ovirt.org/#/c/97843/ [2] https://gerrit.ovirt.org/#/c/97704/

...

Best regards, -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ...

--

Eyal edri

MANAGER

RHV/CNV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- Didi

On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David <didi@redhat.com> wrote:

...

On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote:

...

On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote:

...

On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com>

wrote:

...

...

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote: > > Also, try to run the same in mock which would isolate your

environment

...

My environment is minimal CentOS 7 VM installed from scratch, why

would mock help me?

...

Unfortunately I have FC29 on laptop and I haven't been able to run

OST on it so far :-(

...

I have been discussing with Galit over IRC and so far we haven't

been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6

If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253).

Galit sent a WA which seems to work, do we want to merge it? [1]

Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites.

I am working on a workaround in engine-setup.

Seems like it would not be so easy, so perhaps for now we can update our images in CI to make sure they already have latest selinux packages. I think this will fix too.

So is this an selinux policy bug? Or is it on our side?

...

...

or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue?

I think it's not related, not sure.

...

[1] https://gerrit.ovirt.org/#/c/97843/ [2] https://gerrit.ovirt.org/#/c/97704/

...

Best regards, -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ...

--

Eyal edri

MANAGER

RHV/CNV DevOps

EMEA VIRTUALIZATION R&D

Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- Didi

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX...

Hello Tal, Sandro. Since engine from previous tag was failing and hence it is not yet build we might consider including this fix into the new tag that we need to do anyway (in case the bumps are not merged yet). Anton.

On 19 Feb 2019, at 10:40, Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Feb 19, 2019 at 10:34 AM Doron Fediuck <dfediuck@redhat.com> wrote:

On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote:

On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:

...

Also, try to run the same in mock which would isolate your environment

My environment is minimal CentOS 7 VM installed from scratch, why would mock help me?

Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-(

I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6

If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253).

Galit sent a WA which seems to work, do we want to merge it? [1]

Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites.

I am working on a workaround in engine-setup.

Seems like it would not be so easy, so perhaps for now we can update our images in CI to make sure they already have latest selinux packages. I think this will fix too.

So is this an selinux policy bug? Or is it on our side?

It's a bug in the selinux policy package, yes, not on our side.

Now pushed a workaround:

https://gerrit.ovirt.org/97870

or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue?

I think it's not related, not sure.

[1] https://gerrit.ovirt.org/#/c/97843/ [2] https://gerrit.ovirt.org/#/c/97704/

Best regards, -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ...

-- EYAL EDRI

MANAGER RHV/CNV DEVOPS EMEA VIRTUALIZATION R&D

Red Hat EMEA TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- Didi

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX...

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7RTHHBALWRSJAJ...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

Hi U/S CQ master passed after the use of an updated image Regards Galit בתאריך יום ג׳, 19 בפבר׳ 2019, 17:25, מאת Sandro Bonazzola < sbonazzo@redhat.com>:

Il giorno mar 19 feb 2019 alle ore 10:53 Anton Marchukov < amarchuk@redhat.com> ha scritto:

...

Hello Tal, Sandro.

Since engine from previous tag was failing and hence it is not yet build we might consider including this fix into the new tag that we need to do anyway (in case the bumps are not merged yet).

I'm trying to verify didi's patch, but it's kind of hard to reproduce on my laptop.

...

Anton.

...

On 19 Feb 2019, at 10:40, Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Feb 19, 2019 at 10:34 AM Doron Fediuck <dfediuck@redhat.com> wrote:

On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote:

On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com>

wrote:

...

...

On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote:

...

Also, try to run the same in mock which would isolate your

environment

...

My environment is minimal CentOS 7 VM installed from scratch, why

would mock help me?

...

Unfortunately I have FC29 on laptop and I haven't been able to run

OST on it so far :-(

...

I have been discussing with Galit over IRC and so far we haven't

been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6

If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253).

Galit sent a WA which seems to work, do we want to merge it? [1]

Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites.

I am working on a workaround in engine-setup.

Seems like it would not be so easy, so perhaps for now we can update our images in CI to make sure they already have latest selinux packages. I think this will fix too.

So is this an selinux policy bug? Or is it on our side?

It's a bug in the selinux policy package, yes, not on our side.

Now pushed a workaround:

https://gerrit.ovirt.org/97870

or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue?

I think it's not related, not sure.

[1] https://gerrit.ovirt.org/#/c/97843/ [2] https://gerrit.ovirt.org/#/c/97704/

Best regards, -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ...

-- EYAL EDRI

MANAGER RHV/CNV DEVOPS EMEA VIRTUALIZATION R&D

Red Hat EMEA TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- Didi

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX...

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7RTHHBALWRSJAJ...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

--

SANDRO BONAZZOLA

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/>

sbonazzo@redhat.com <https://red.ht/sig>

On Tue, 19 Feb 2019, 19:22 Eyal Edri <eedri@redhat.com wrote:

On Tue, Feb 19, 2019, 19:45 Galit Rosenthal <grosenth@redhat.com wrote:

...

Hi

U/S CQ master passed after the use of an updated image

+1, glad to hear that. are we going to apply also the other suggestion for workarounds for postgresql or abandon them?

I'd say let's use updated image to unblock OST, we discus further improvements later...

...

Regards Galit

בתאריך יום ג׳, 19 בפבר׳ 2019, 17:25, מאת Sandro Bonazzola < sbonazzo@redhat.com>:

...

Il giorno mar 19 feb 2019 alle ore 10:53 Anton Marchukov < amarchuk@redhat.com> ha scritto:

...

Hello Tal, Sandro.

Since engine from previous tag was failing and hence it is not yet build we might consider including this fix into the new tag that we need to do anyway (in case the bumps are not merged yet).

I'm trying to verify didi's patch, but it's kind of hard to reproduce on my laptop.

...

Anton.

...

On 19 Feb 2019, at 10:40, Yedidyah Bar David <didi@redhat.com> wrote:

On Tue, Feb 19, 2019 at 10:34 AM Doron Fediuck <dfediuck@redhat.com> wrote:

On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote:

On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote: On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote:

...

On Mon, Feb 18, 2019 at 11:39 AM Martin Perina <mperina@redhat.com>

wrote:

...

> > > > On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote: >> >> Also, try to run the same in mock which would isolate your environment > > > My environment is minimal CentOS 7 VM installed from scratch, why would mock help me? > > Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-( > > I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start).

Have you tried running OST 4.2 locally? When it fails, it should keep everything available for you to ssh into. It has been the 4th night that the out 4.2 suite is failing, which is not a healthy condition to be in.

Reproduced and added this comment:

https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6

If we do not get a solution soon, we can consider a workaround (as we did in bug 1518253).

Galit sent a WA which seems to work, do we want to merge it? [1]

Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites.

I am working on a workaround in engine-setup.

Seems like it would not be so easy, so perhaps for now we can update our images in CI to make sure they already have latest selinux packages. I think this will fix too.

So is this an selinux policy bug? Or is it on our side?

It's a bug in the selinux policy package, yes, not on our side.

Now pushed a workaround:

https://gerrit.ovirt.org/97870

or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue?

I think it's not related, not sure.

[1] https://gerrit.ovirt.org/#/c/97843/ [2] https://gerrit.ovirt.org/#/c/97704/

Best regards, -- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ...

-- EYAL EDRI

MANAGER RHV/CNV DEVOPS EMEA VIRTUALIZATION R&D

Red Hat EMEA TRIED. TESTED. TRUSTED. phone: +972-9-7692018 irc: eedri (on #tlv #rhev-dev #rhev-integ)

-- Didi

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX...

-- Didi _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7RTHHBALWRSJAJ...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

--

SANDRO BONAZZOLA

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/>

sbonazzo@redhat.com <https://red.ht/sig>

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GZ7ZNVNPOH3ZXT...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3QWUYU4TUN56R...

On Tue, Feb 19, 2019 at 8:51 PM Anton Marchukov <amarchuk@redhat.com> wrote:

I think having the new base image working confirms the assumption that the problem was related to some old selinux package.

How are the images built? I searched a bit and found [1]. Do we run it manually? In jenkins? Elsewhere? Do we keep logs? To investigate deeper, I'd like to understand how exactly we built the broken images. [1] https://github.com/lago-project/lago-images

Then we just need to estimate the probability of users having those package installed and not upgraded?

Not very probable, AFAICT. I *think* the flow is something like this, but this is not more than a (not-so-)wild guess: 1. Install el 7.2 (with selinux-policy 3.12) 2. Upgrade to later 3. Reboot the machine without a working /dev/console 4. Install scl pg and initdb (1.) or (2.) might be about how we create the images, (3.) might be specific to libvirt/lago.

Anton.

On Tue, Feb 19, 2019 at 7:47 PM Martin Perina <mperina@redhat.com> wrote:

...

On Tue, 19 Feb 2019, 19:22 Eyal Edri <eedri@redhat.com wrote:

...

On Tue, Feb 19, 2019, 19:45 Galit Rosenthal <grosenth@redhat.com wrote:

...

Hi

U/S CQ master passed after the use of an updated image

+1, glad to hear that. are we going to apply also the other suggestion for workarounds for postgresql or abandon them?

I'd say let's use updated image to unblock OST, we discus further improvements later...

...

...

Regards Galit

בתאריך יום ג׳, 19 בפבר׳ 2019, 17:25, מאת Sandro Bonazzola < sbonazzo@redhat.com>:

...

Il giorno mar 19 feb 2019 alle ore 10:53 Anton Marchukov < amarchuk@redhat.com> ha scritto:

...

Hello Tal, Sandro.

Since engine from previous tag was failing and hence it is not yet build we might consider including this fix into the new tag that we need to do anyway (in case the bumps are not merged yet).

I'm trying to verify didi's patch, but it's kind of hard to reproduce on my laptop.

...

Anton.

> On 19 Feb 2019, at 10:40, Yedidyah Bar David <didi@redhat.com> wrote: > > On Tue, Feb 19, 2019 at 10:34 AM Doron Fediuck <dfediuck@redhat.com> wrote: > > > On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> wrote: > On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David <didi@redhat.com> wrote: > On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> wrote: > > > On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David <didi@redhat.com> wrote: > On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> wrote: > > > > On Mon, Feb 18, 2019 at 11:39 AM Martin Perina < mperina@redhat.com> wrote: > > > > > > > > > > > > On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> wrote: > > >> > > >> Also, try to run the same in mock which would isolate your environment > > > > > > > > > My environment is minimal CentOS 7 VM installed from scratch, why would mock help me? > > > > > > Unfortunately I have FC29 on laptop and I haven't been able to run OST on it so far :-( > > > > > > I have been discussing with Galit over IRC and so far we haven't been able to identify any different package version on my CentOS (where everything works as expected) and her OST environment (where PG 9.5 failed to start). > > > > > > Have you tried running OST 4.2 locally? When it fails, it should keep > > everything available for you to ssh into. > > It has been the 4th night that the out 4.2 suite is failing, which is > > not a healthy condition to be in. > > Reproduced and added this comment: > > https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6 > > If we do not get a solution soon, we can consider a workaround (as we > did in bug 1518253). > > Galit sent a WA which seems to work, do we want to merge it? [1] > > Current version of this patch is only for the upgrade suites. I do not object to merging, we might want to handle somewhere for all suites. > > I am working on a workaround in engine-setup. > > Seems like it would not be so easy, so perhaps for now we can update our images in CI to make sure they already have latest selinux packages. I think this will fix too. > > So is this an selinux policy bug? Or is it on our side? > > It's a bug in the selinux policy package, yes, not on our side. > > Now pushed a workaround: > > https://gerrit.ovirt.org/97870 > > > > or maybe consider reverting the vmconsole patch? [2] which might have started this whole issue? > > I think it's not related, not sure. > > > > [1] https://gerrit.ovirt.org/#/c/97843/ > [2] https://gerrit.ovirt.org/#/c/97704/ > > > Best regards, > -- > Didi > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ... > > > -- > EYAL EDRI > > MANAGER > RHV/CNV DEVOPS > EMEA VIRTUALIZATION R&D > > Red Hat EMEA > TRIED. TESTED. TRUSTED. > phone: +972-9-7692018 > irc: eedri (on #tlv #rhev-dev #rhev-integ) > > > -- > Didi > > > -- > Didi > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX... > > > -- > Didi > _______________________________________________ > Devel mailing list -- devel@ovirt.org > To unsubscribe send an email to devel-leave@ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ > List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7RTHHBALWRSJAJ...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

--

SANDRO BONAZZOLA

MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV

Red Hat EMEA <https://www.redhat.com/>

sbonazzo@redhat.com <https://red.ht/sig>

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GZ7ZNVNPOH3ZXT...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3QWUYU4TUN56R...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YQUJCUP3PWC7YA...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/INV4VMSF3QNYSA...

-- Didi

On Wed, Feb 20, 2019 at 2:34 PM Galit Rosenthal <grosenth@redhat.com> wrote:

For building the centos/rhel images I'm using the information from this document

https://docs.google.com/document/d/1fxXBcgHnQr3r2qrSzXiSlpNMMLWnbq899s8rUP38...

Thanks, that's what I was looking for. I suggest to convert this to a jenkins job, so that image builds are controlled, logged, and more debuggable. Also: I now checked lago-images, and see there: $ grep -r centos7 . ./image-specs/el7-iscsi:#osinfo=centos7.2 ./image-specs/el7.5-base:#osinfo=centos7.5 ./image-specs/el7.3-base:#osinfo=centos7.3 ./image-specs/el7-nfs:#osinfo=centos7.2 ./image-specs/el7.4-base:#osinfo=centos7.4 ./image-specs/el7-base:#osinfo=centos7.2 Perhaps on the "newest" (rolling-release) el7-base, el7-nfs, el7-iscsi we start with centos7.2 and then update? And perhaps we run into current bug due to the way virt-builder works? Perhaps we can at least patch to use centos7.6, perhaps also create new files per each update and remove the existing ones (so that we do not forget to create/update). Best regards,

On Wed, Feb 20, 2019 at 11:29 AM Yedidyah Bar David <didi@redhat.com> wrote:

...

On Tue, Feb 19, 2019 at 8:51 PM Anton Marchukov <amarchuk@redhat.com> wrote:

...

I think having the new base image working confirms the assumption that the problem was related to some old selinux package.

How are the images built? I searched a bit and found [1]. Do we run it manually? In jenkins? Elsewhere? Do we keep logs? To investigate deeper, I'd like to understand how exactly we built the broken images.

[1] https://github.com/lago-project/lago-images

...

Then we just need to estimate the probability of users having those package installed and not upgraded?

Not very probable, AFAICT. I *think* the flow is something like this, but this is not more than a (not-so-)wild guess:

1. Install el 7.2 (with selinux-policy 3.12) 2. Upgrade to later 3. Reboot the machine without a working /dev/console 4. Install scl pg and initdb

(1.) or (2.) might be about how we create the images, (3.) might be specific to libvirt/lago.

...

Anton.

On Tue, Feb 19, 2019 at 7:47 PM Martin Perina <mperina@redhat.com> wrote:

...

On Tue, 19 Feb 2019, 19:22 Eyal Edri <eedri@redhat.com wrote:

...

On Tue, Feb 19, 2019, 19:45 Galit Rosenthal <grosenth@redhat.com wrote:

...

Hi

U/S CQ master passed after the use of an updated image

+1, glad to hear that. are we going to apply also the other suggestion for workarounds for postgresql or abandon them?

I'd say let's use updated image to unblock OST, we discus further improvements later...

...

...

Regards Galit

בתאריך יום ג׳, 19 בפבר׳ 2019, 17:25, מאת Sandro Bonazzola < sbonazzo@redhat.com>:

> > > Il giorno mar 19 feb 2019 alle ore 10:53 Anton Marchukov < > amarchuk@redhat.com> ha scritto: > >> Hello Tal, Sandro. >> >> Since engine from previous tag was failing and hence it is not yet >> build we might consider including this fix into the new tag that we need to >> do anyway (in case the bumps are not merged yet). >> > > I'm trying to verify didi's patch, but it's kind of hard to > reproduce on my laptop. > > > >> >> Anton. >> >> > On 19 Feb 2019, at 10:40, Yedidyah Bar David <didi@redhat.com> >> wrote: >> > >> > On Tue, Feb 19, 2019 at 10:34 AM Doron Fediuck < >> dfediuck@redhat.com> wrote: >> > >> > >> > On Tue, 19 Feb 2019 at 10:25, Yedidyah Bar David <didi@redhat.com> >> wrote: >> > On Tue, Feb 19, 2019 at 9:52 AM Yedidyah Bar David < >> didi@redhat.com> wrote: >> > On Tue, Feb 19, 2019 at 9:49 AM Eyal Edri <eedri@redhat.com> >> wrote: >> > >> > >> > On Tue, Feb 19, 2019 at 9:29 AM Yedidyah Bar David < >> didi@redhat.com> wrote: >> > On Tue, Feb 19, 2019 at 9:15 AM Dan Kenigsberg <danken@redhat.com> >> wrote: >> > > >> > > On Mon, Feb 18, 2019 at 11:39 AM Martin Perina < >> mperina@redhat.com> wrote: >> > > > >> > > > >> > > > >> > > > On Mon, Feb 18, 2019 at 10:32 AM Dafna Ron <dron@redhat.com> >> wrote: >> > > >> >> > > >> Also, try to run the same in mock which would isolate your >> environment >> > > > >> > > > >> > > > My environment is minimal CentOS 7 VM installed from scratch, >> why would mock help me? >> > > > >> > > > Unfortunately I have FC29 on laptop and I haven't been able >> to run OST on it so far :-( >> > > > >> > > > I have been discussing with Galit over IRC and so far we >> haven't been able to identify any different package version on my CentOS >> (where everything works as expected) and her OST environment (where PG 9.5 >> failed to start). >> > > >> > > >> > > Have you tried running OST 4.2 locally? When it fails, it >> should keep >> > > everything available for you to ssh into. >> > > It has been the 4th night that the out 4.2 suite is failing, >> which is >> > > not a healthy condition to be in. >> > >> > Reproduced and added this comment: >> > >> > https://bugzilla.redhat.com/show_bug.cgi?id=1518599#c6 >> > >> > If we do not get a solution soon, we can consider a workaround >> (as we >> > did in bug 1518253). >> > >> > Galit sent a WA which seems to work, do we want to merge it? [1] >> > >> > Current version of this patch is only for the upgrade suites. I >> do not object to merging, we might want to handle somewhere for all suites. >> > >> > I am working on a workaround in engine-setup. >> > >> > Seems like it would not be so easy, so perhaps for now we can >> update our images in CI to make sure they already have latest selinux >> packages. I think this will fix too. >> > >> > So is this an selinux policy bug? Or is it on our side? >> > >> > It's a bug in the selinux policy package, yes, not on our side. >> > >> > Now pushed a workaround: >> > >> > https://gerrit.ovirt.org/97870 >> > >> > >> > >> > or maybe consider reverting the vmconsole patch? [2] which might >> have started this whole issue? >> > >> > I think it's not related, not sure. >> > >> > >> > >> > [1] https://gerrit.ovirt.org/#/c/97843/ >> > [2] https://gerrit.ovirt.org/#/c/97704/ >> > >> > >> > Best regards, >> > -- >> > Didi >> > _______________________________________________ >> > Devel mailing list -- devel@ovirt.org >> > To unsubscribe send an email to devel-leave@ovirt.org >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/XYD4CJQGHAUKDQ... >> > >> > >> > -- >> > EYAL EDRI >> > >> > MANAGER >> > RHV/CNV DEVOPS >> > EMEA VIRTUALIZATION R&D >> > >> > Red Hat EMEA >> > TRIED. TESTED. TRUSTED. >> > phone: +972-9-7692018 >> > irc: eedri (on #tlv #rhev-dev #rhev-integ) >> > >> > >> > -- >> > Didi >> > >> > >> > -- >> > Didi >> > _______________________________________________ >> > Devel mailing list -- devel@ovirt.org >> > To unsubscribe send an email to devel-leave@ovirt.org >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7QF4RSTMYHZYIX... >> > >> > >> > -- >> > Didi >> > _______________________________________________ >> > Devel mailing list -- devel@ovirt.org >> > To unsubscribe send an email to devel-leave@ovirt.org >> > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> > oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> > List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/7RTHHBALWRSJAJ... >> >> -- >> Anton Marchukov >> Team Lead - Release Management - RHV DevOps - Red Hat >> >> >> >> >> >> > > -- > > SANDRO BONAZZOLA > > MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV > > Red Hat EMEA <https://www.redhat.com/> > > sbonazzo@redhat.com > <https://red.ht/sig> > _______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/GZ7ZNVNPOH3ZXT...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/J3QWUYU4TUN56R...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YQUJCUP3PWC7YA...

-- Anton Marchukov Team Lead - Release Management - RHV DevOps - Red Hat

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/INV4VMSF3QNYSA...

-- Didi

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

<https://www.redhat.com/>

galit@gmail.com T: 972-9-7692230 <https://red.ht/sig>

-- Didi

On Tue, Feb 19, 2019 at 8:47 PM Martin Perina <mperina@redhat.com> wrote:

On Tue, 19 Feb 2019, 19:22 Eyal Edri <eedri@redhat.com wrote:

...

On Tue, Feb 19, 2019, 19:45 Galit Rosenthal <grosenth@redhat.com wrote:

...

Hi

U/S CQ master passed after the use of an updated image

+1, glad to hear that. are we going to apply also the other suggestion for workarounds for postgresql or abandon them?

I'd say let's use updated image to unblock OST, we discus further improvements later...

I'm pleased to inform that after 4 red nights, https://jenkins.ovirt.org/job/ovirt-system-tests_network-suite-4.2/ is green again. Thanks for the workaround.

there is a parallel thread [ovirt-devel] [URGENT] - seems like regression: Re: Re: Misc configuration': Failed to start service 'rh-postgresql95-postgresql with some an idea. TL;DR: lago is not allocating a specific TTY number which postres's selinux policy requires. On Tue, Feb 19, 2019 at 12:20 PM Eli Mesika <emesika@redhat.com> wrote:

Galit

Did you find what went wrong?

On Sun, Feb 17, 2019 at 12:55 PM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Thanks Greg

I will check this

On Sun, Feb 17, 2019 at 12:51 PM Greg Sheremeta <gshereme@redhat.com> wrote:

...

Is there any way you can run "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" like it suggests? The logs below don't give any indication why it failed to start, afaict.

On Sun, Feb 17, 2019 at 4:59 AM Galit Rosenthal <grosenth@redhat.com> wrote:

...

Hi

I receive this error message both in CQ and check_patch:

2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd systemd.state:130 starting service rh-postgresql95-postgresql 2019-02-16 16:28:06,874-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:813 execute: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), executable='None', cwd='None', env=None 2019-02-16 16:28:07,913-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.executeRaw:863 execute-result: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service'), rc=1 2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:921 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stdout:

2019-02-16 16:28:07,914-0500 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:926 execute-output: ('/usr/bin/systemctl', 'start', 'rh-postgresql95-postgresql.service') stderr: Job for rh-postgresql95-postgresql.service failed because the control process exited with error code. See "systemctl status rh-postgresql95-postgresql.service" and "journalctl -xe" for details.

2019-02-16 16:28:07,915-0500 DEBUG otopi.transaction transaction.abort:119 aborting 'File transaction for '/var/opt/rh/rh-postgresql95/lib/pgsql/data/pg_hba.conf'' 2019-02-16 16:28:07,916-0500 DEBUG otopi.context context._executeMethod:143 method exception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 133, in _executeMethod method['method']() File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/provisioning/postgres.py", line 201, in _misc self._provisioning.provision() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 498, in provision self.restartPG() File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/postgres.py", line 399, in restartPG state=state, File "/usr/share/otopi/plugins/otopi/services/systemd.py", line 141, in state service=name, RuntimeError: Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,918-0500 ERROR otopi.context context._executeMethod:152 Failed to execute stage 'Misc configuration': Failed to start service 'rh-postgresql95-postgresql' 2019-02-16 16:28:07,958-0500 DEBUG otopi.plugins.otopi.debug.debug_failure.debug_failure debug_failure._notification:100 tcp connections: id uid local foreign state pid exe

What can cause it?

Thanks

Galit

https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-master_change...

https://jenkins.ovirt.org/blue/organizations/jenkins/ovirt-system-tests_stan...

Regards,

Galit

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

galit@gmail.com T: 972-9-7692230

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/QNDG65M6UPEXTC...

--

GREG SHEREMETA

SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX

Red Hat NA

gshereme@redhat.com IRC: gshereme

--

GALIT ROSENTHAL

SOFTWARE ENGINEER

Red Hat

galit@gmail.com T: 972-9-7692230

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/YROV4PLNBTOWKY...

_______________________________________________ Devel mailing list -- devel@ovirt.org To unsubscribe send an email to devel-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/TT5XORUC7P2PVQ...