Project: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/15... Build Number: 1536 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1536 [Marcin Sobczyk] basic: ansible: Port '003_00_metrics_bootstrap_pytest.py' to ansible ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 1000_analyze_postgresql.analyze_postgres Error Message: Failed to perform the postgres analysis. Exit code is 1 -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[0m?[0m lago.ssh: DEBUG: start task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine: lago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[32mSuccess?[0m (in 0:00:00) lago.ssh: DEBUG: start task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: Running 93da8eda on lago-performance-suite-master-engine: /root/analyze_postgresql.sh lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine returned with 1 lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine output: Last metadata expiration check: 0:11:09 ago on Fri 31 Jul 2020 10:59:32 PM EDT. pgdg-redhat-repo-42.0-11.noarch.rpm 12 kB/s | 11 kB 00:00 Dependencies resolved. ================================================================================ Package Architecture Version Repository Size ================================================================================ Installing: pgdg-redhat-repo noarch 42.0-11 @commandline 11 k Transaction Summary ================================================================================ Install 1 Package Total size: 11 k Installed size: 11 k Downloading Packages: Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 Installed: pgdg-redhat-repo-42.0-11.noarch Complete! PostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 190 kB/s | 376 kB 00:01 PostgreSQL 12 for RHEL/CentOS 8 - x86_64 112 kB/s | 247 kB 00:02 PostgreSQL 11 for RHEL/CentOS 8 - x86_64 217 kB/s | 424 kB 00:01 PostgreSQL 10 for RHEL/CentOS 8 - x86_64 161 kB/s | 308 kB 00:01 PostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 127 kB/s | 300 kB 00:02 PostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 111 kB/s | 259 kB 00:02 Dependencies resolved. ================================================================================ Package Arch Version Repository Size ================================================================================ Installing: pgbadger noarch 11.3-1.rhel8 pgdg-common 348 k pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k Installing dependencies: perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k Transaction Summary ================================================================================ Install 4 Packages Total download size: 874 k Installed size: 3.5 M Downloading Packages: (1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 (2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 523 kB/s | 24 kB 00:00 (3/4): pgbadger-11.3-1.rhel8.noarch.rpm 252 kB/s | 348 kB 00:01 (4/4): pgcluu-3.1-2.rhel8.noarch.rpm 235 kB/s | 356 kB 00:01 -------------------------------------------------------------------------------- Total 561 kB/s | 874 kB 00:01 PostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 Key imported successfully Extra Packages for Enterprise Linux 8 - x86_64 7.4 kB/s | 1.6 kB 00:00 Key imported successfully Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 Installing : pgbadger-11.3-1.rhel8.noarch 3/4 Installing : pgcluu-3.1-2.rhel8.noarch 4/4 Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 Verifying : pgbadger-11.3-1.rhel8.noarch 1/4 Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 Installed: perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch pgbadger-11.3-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch Complete! lago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine errors: + pg_datadir=/var/lib/pgsql/data + pg_service=postgresql.service + pgdg= + scl_enable= ++ get_engine_pg_scl ++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf) ++ local engine_pg_scl_conf ++ local res= ++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]] ++ echo '' + engine_pg_scl= + [[ -n '' ]] ++ rpm -q postgresql + pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64 + case "${pgver}" in + pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-r... + [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-r... ]] + yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-r... + yum install -y pgbadger pgcluu warning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.3-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY Importing GPG key 0x442DF0F8: Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers@pgfoundry.org>" Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8 From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG warning: /var/cache/dnf/epel-el8-fe1ce5303067b1fb/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY Importing GPG key 0x2F86D6A1: Userid : "Fedora EPEL (8) <epel@fedoraproject.org>" Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1 From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8 ++ su - postgres -c 'psql postgres postgres -t -c '\''show log_directory'\''' ++ sed 2d + pglogdir=' log' + pglogdir=log + cd /var/lib/pgsql/data/log/ ++ su - postgres -c 'psql postgres postgres -t -c '\''show log_filename'\''' ++ sed 2d + pglog=' postgresql-%m.log' ++ date +postgresql-%m.log + pglog=postgresql-07.log + mkdir pgbadger + pgbadger -f stderr postgresql-07.log --outdir pgbadger FATAL: logfile "postgresql-07.log" must exist! - Error at line 2410 --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 142, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 60, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/1000_analyze_postgresql.py", line 52, in analyze_postgres ' Exit code is %s' % result.code File "/usr/lib/python2.7/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) 'Failed to perform the postgres analysis. Exit code is 1\n-------------------- >> begin captured logging << --------------------\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[0m\x1b[0m\nlago.ssh: DEBUG: start task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:1c6ed586-2943-4d3c-841a-73107c9668bd:Get ssh client for lago-performance-suite-master-engine:\nlago.plugins.vm: INFO: * Copy /home/jenkins/agent/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: \x1b[32mSuccess\x1b[0m (in 0:00:00)\nlago.ssh: DEBUG: start task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: end task:ede6f1b0-4462-4dde-a985-63e60a9deabc:Get ssh client for lago-performance-suite-master-engine:\nlago.ssh: DEBUG: Running 93da8eda on lago-performance-suite-master-engine: /root/analyze_postgresql.sh\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine returned with 1\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine output:\n Last metadata expiration check: 0:11:09 ago on Fri 31 Jul 2020 10:59:32 PM EDT.\npgdg-redhat-repo-42.0-11.noarch.rpm 12 kB/s | 11 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling:\n pgdg-redhat-repo noarch 42.0-11 @commandline 11 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size: 11 k\nInstalled size: 11 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 \n Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 \n\nInstalled:\n pgdg-redhat-repo-42.0-11.noarch \n\nComplete!\nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 190 kB/s | 376 kB 00:01 \nPostgreSQL 12 for RHEL/CentOS 8 - x86_64 112 kB/s | 247 kB 00:02 \nPostgreSQL 11 for RHEL/CentOS 8 - x86_64 217 kB/s | 424 kB 00:01 \nPostgreSQL 10 for RHEL/CentOS 8 - x86_64 161 kB/s | 308 kB 00:01 \nPostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 127 kB/s | 300 kB 00:02 \nPostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 111 kB/s | 259 kB 00:02 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n pgbadger noarch 11.3-1.rhel8 pgdg-common 348 k\n pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k\nInstalling dependencies:\n perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k\n perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k\n\nTransaction Summary\n================================================================================\nInstall 4 Packages\n\nTotal download size: 874 k\nInstalled size: 3.5 M\nDownloading Packages:\n(1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 1.4 MB/s | 145 kB 00:00 \n(2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 523 kB/s | 24 kB 00:00 \n(3/4): pgbadger-11.3-1.rhel8.noarch.rpm 252 kB/s | 348 kB 00:01 \n(4/4): pgcluu-3.1-2.rhel8.noarch.rpm 235 kB/s | 356 kB 00:01 \n--------------------------------------------------------------------------------\nTotal 561 kB/s | 874 kB 00:01 \nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 \nKey imported successfully\nExtra Packages for Enterprise Linux 8 - x86_64 7.4 kB/s | 1.6 kB 00:00 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 \n Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 \n Installing : pgbadger-11.3-1.rhel8.noarch 3/4 \n Installing : pgcluu-3.1-2.rhel8.noarch 4/4 \n Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 \n Verifying : pgbadger-11.3-1.rhel8.noarch 1/4 \n Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 \n Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 \n Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 \n\nInstalled:\n perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch\n pgbadger-11.3-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch \n\nComplete!\n\nlago.ssh: DEBUG: Command 93da8eda on lago-performance-suite-master-engine errors:\n + pg_datadir=/var/lib/pgsql/data\n+ pg_service=postgresql.service\n+ pgdg=\n+ scl_enable=\n++ get_engine_pg_scl\n++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf)\n++ local engine_pg_scl_conf\n++ local res=\n++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]]\n++ echo \'\'\n+ engine_pg_scl=\n+ [[ -n \'\' ]]\n++ rpm -q postgresql\n+ pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64\n+ case "${pgver}" in\n+ pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-redhat-repo-42.0-11.noarch.rpm\n+ [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-r... ]]\n+ yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgdg-redhat-repo-42.0-11.noarch.rpm\n+ yum install -y pgbadger pgcluu\nwarning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.3-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY\nImporting GPG key 0x442DF0F8:\n Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers@pgfoundry.org>"\n Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG\nwarning: /var/cache/dnf/epel-el8-fe1ce5303067b1fb/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY\nImporting GPG key 0x2F86D6A1:\n Userid : "Fedora EPEL (8) <epel@fedoraproject.org>"\n Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1\n From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_directory\'\\\'\'\'\n++ sed 2d\n+ pglogdir=\' log\'\n+ pglogdir=log\n+ cd /var/lib/pgsql/data/log/\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_filename\'\\\'\'\'\n++ sed 2d\n+ pglog=\' postgresql-%m.log\'\n++ date +postgresql-%m.log\n+ pglog=postgresql-07.log\n+ mkdir pgbadger\n+ pgbadger -f stderr postgresql-07.log --outdir pgbadger\nFATAL: logfile "postgresql-07.log" must exist!\n - Error at line 2410\n\n--------------------- >> end captured logging << ---------------------'