Project: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master/ Build: https://jenkins.ovirt.org/job/ovirt-system-tests_performance-suite-master... Build Number: 1693 Build Status: Failure Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #1693 [hbraha] network: test invocation event propagted to vdsm ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 1000_analyze_postgresql.analyze_postgres Error Message: Failed to perform the postgres analysis. Exit code is 1 -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[0m?[0m lago.ssh: DEBUG: start task:14c53eaf-faa7-42a1-afcd-836d8ed96496:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:14c53eaf-faa7-42a1-afcd-836d8ed96496:Get ssh client for lago-performance-suite-master-engine: lago.plugins.vm: INFO: * Copy /home/jenkins/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[32mSuccess?[0m (in 0:00:00) lago.ssh: DEBUG: start task:a05e4a4e-5079-4230-ba72-e60f0858d088:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:a05e4a4e-5079-4230-ba72-e60f0858d088:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: Running 710b9982 on lago-performance-suite-master-engine: /root/analyze_postgresql.sh lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine returned with 1 lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine output: b'Last metadata expiration check: 0:10:42 ago on Thu 31 Dec 2020 09:56:30 PM EST.\npgdg-redhat-repo-42.0-11.noarch.rpm 51 kB/s | 11 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling:\n pgdg-redhat-repo noarch 42.0-11 @commandline 11 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size: 11 k\nInstalled size: 11 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 \n Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 \n\nInstalled:\n pgdg-redhat-repo-42.0-11.noarch \n\nComplete!\nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 548 kB/s | 533 kB 00:00 \nPostgreSQL 12 for RHEL/CentOS 8 - x86_64 212 kB/s | 408 kB 00:01 \nPostgreSQL 11 for RHEL/CentOS 8 - x86_64 282 kB/s | 604 kB 00:02 \nPostgreSQL 10 for RHEL/CentOS 8 - x86_64 208 kB/s | 422 kB 00:02 \nPostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 201 kB/s | 409 kB 00:02 \nPostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 182 kB/s | 365 kB 00:02 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n pgbadger noarch 11.4-1.rhel8 pgdg-common 350 k\n pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k\nInstalling dependencies:\n perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k\n perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k\n\nTransaction Summary\n================================================================================\nInstall 4 Packages\n\nTotal download size: 876 k\nInstalled size: 3.6 M\nDownloading Packages:\n(1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 2.0 MB/s | 145 kB 00:00 \n(2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 747 kB/s | 24 kB 00:00 \n(3/4): pgcluu-3.1-2.rhel8.noarch.rpm 286 kB/s | 356 kB 00:01 \n(4/4): pgbadger-11.4-1.rhel8.noarch.rpm 280 kB/s | 350 kB 00:01 \n--------------------------------------------------------------------------------\nTotal 698 kB/s | 876 kB 00:01 \nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 \nKey imported successfully\nExtra Packages for Enterprise Linux 8 - x86_64 7.9 kB/s | 1.6 kB 00:00 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 \n Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 \n Installing : pgbadger-11.4-1.rhel8.noarch 3/4 \n Installing : pgcluu-3.1-2.rhel8.noarch 4/4 \n Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 \n Verifying : pgbadger-11.4-1.rhel8.noarch 1/4 \n Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 \n Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 \n Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 \n\nInstalled:\n perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch\n pgbadger-11.4-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch \n\nComplete!\n' lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine errors: b'+ pg_datadir=/var/lib/pgsql/data\n+ pg_service=postgresql.service\n+ pgdg=\n+ scl_enable=\n++ get_engine_pg_scl\n++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf)\n++ local engine_pg_scl_conf\n++ local res=\n++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]]\n++ echo \'\'\n+ engine_pg_scl=\n+ [[ -n \'\' ]]\n++ rpm -q postgresql\n+ pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64\n+ case "${pgver}" in\n+ pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_6... [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgd... ]]\n+ yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgd... yum install -y pgbadger pgcluu\nwarning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.4-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY\nImporting GPG key 0x442DF0F8:\n Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers(a)pgfoundry.org&gt;&quot;\n Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG\nwarning: /var/cache/dnf/epel-el8-5635746ae2054ad8/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY\nImporting GPG key 0x2F86D6A1:\n Userid : "Fedora EPEL (8) <epel(a)fedoraproject.org&gt;&quot;\n Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1\n From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8\n++ sed 2d\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_directory\'\\\'\'\'\n+ pglogdir=\' log\'\n+ pglogdir=log\n+ cd /var/lib/pgsql/data/log/\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_filename\'\\\'\'\'\n++ sed 2d\n+ pglog=\' postgresql-%m.log\'\n++ date +postgresql-%m.log\n+ pglog=postgresql-12.log\n+ mkdir pgbadger\n+ pgbadger -f stderr postgresql-12.log --outdir pgbadger\nFATAL: logfile "postgresql-12.log" must exist!\n - Error at line 2418\n' --------------------- >> end captured logging << --------------------- Stack Trace: File "/usr/lib64/python3.6/unittest/case.py", line 60, in testPartExecutor yield File "/usr/lib64/python3.6/unittest/case.py", line 622, in run testMethod() File "/usr/lib/python3.6/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python3.6/site-packages/ovirtlago/testlib.py", line 144, in wrapped_test test() File "/usr/lib/python3.6/site-packages/ovirtlago/testlib.py", line 62, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/test-scenarios/1000_analyze_postgresql.py", line 52, in analyze_postgres ' Exit code is %s' % result.code File "/usr/lib/python3.6/site-packages/nose/tools/trivial.py", line 29, in eq_ raise AssertionError(msg or "%r != %r" % (a, b)) Failed to perform the postgres analysis. Exit code is 1 -------------------- >> begin captured logging << -------------------- lago.plugins.vm: INFO: * Copy /home/jenkins/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[0m?[0m lago.ssh: DEBUG: start task:14c53eaf-faa7-42a1-afcd-836d8ed96496:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:14c53eaf-faa7-42a1-afcd-836d8ed96496:Get ssh client for lago-performance-suite-master-engine: lago.plugins.vm: INFO: * Copy /home/jenkins/workspace/ovirt-system-tests_performance-suite-master/ovirt-system-tests/performance-suite-master/../common/test-scenarios-files/analyze_postgresql.sh to lago-performance-suite-master-engine:/root: ?[32mSuccess?[0m (in 0:00:00) lago.ssh: DEBUG: start task:a05e4a4e-5079-4230-ba72-e60f0858d088:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: end task:a05e4a4e-5079-4230-ba72-e60f0858d088:Get ssh client for lago-performance-suite-master-engine: lago.ssh: DEBUG: Running 710b9982 on lago-performance-suite-master-engine: /root/analyze_postgresql.sh lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine returned with 1 lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine output: b'Last metadata expiration check: 0:10:42 ago on Thu 31 Dec 2020 09:56:30 PM EST.\npgdg-redhat-repo-42.0-11.noarch.rpm 51 kB/s | 11 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling:\n pgdg-redhat-repo noarch 42.0-11 @commandline 11 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package\n\nTotal size: 11 k\nInstalled size: 11 k\nDownloading Packages:\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : pgdg-redhat-repo-42.0-11.noarch 1/1 \n Verifying : pgdg-redhat-repo-42.0-11.noarch 1/1 \n\nInstalled:\n pgdg-redhat-repo-42.0-11.noarch \n\nComplete!\nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 548 kB/s | 533 kB 00:00 \nPostgreSQL 12 for RHEL/CentOS 8 - x86_64 212 kB/s | 408 kB 00:01 \nPostgreSQL 11 for RHEL/CentOS 8 - x86_64 282 kB/s | 604 kB 00:02 \nPostgreSQL 10 for RHEL/CentOS 8 - x86_64 208 kB/s | 422 kB 00:02 \nPostgreSQL 9.6 for RHEL/CentOS 8 - x86_64 201 kB/s | 409 kB 00:02 \nPostgreSQL 9.5 for RHEL/CentOS 8 - x86_64 182 kB/s | 365 kB 00:02 \nDependencies resolved.\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n pgbadger noarch 11.4-1.rhel8 pgdg-common 350 k\n pgcluu noarch 3.1-2.rhel8 pgdg-common 356 k\nInstalling dependencies:\n perl-Text-CSV_XS x86_64 1.40-1.el8 epel-el8 145 k\n perl-UNIVERSAL-isa noarch 1.20171012-4.el8 epel-el8 24 k\n\nTransaction Summary\n================================================================================\nInstall 4 Packages\n\nTotal download size: 876 k\nInstalled size: 3.6 M\nDownloading Packages:\n(1/4): perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm 2.0 MB/s | 145 kB 00:00 \n(2/4): perl-UNIVERSAL-isa-1.20171012-4.el8.noar 747 kB/s | 24 kB 00:00 \n(3/4): pgcluu-3.1-2.rhel8.noarch.rpm 286 kB/s | 356 kB 00:01 \n(4/4): pgbadger-11.4-1.rhel8.noarch.rpm 280 kB/s | 350 kB 00:01 \n--------------------------------------------------------------------------------\nTotal 698 kB/s | 876 kB 00:01 \nPostgreSQL common RPMs for RHEL/CentOS 8 - x86_ 1.6 MB/s | 1.7 kB 00:00 \nKey imported successfully\nExtra Packages for Enterprise Linux 8 - x86_64 7.9 kB/s | 1.6 kB 00:00 \nKey imported successfully\nRunning transaction check\nTransaction check succeeded.\nRunning transaction test\nTransaction test succeeded.\nRunning transaction\n Preparing : 1/1 \n Installing : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 1/4 \n Installing : perl-Text-CSV_XS-1.40-1.el8.x86_64 2/4 \n Installing : pgbadger-11.4-1.rhel8.noarch 3/4 \n Installing : pgcluu-3.1-2.rhel8.noarch 4/4 \n Running scriptlet: pgcluu-3.1-2.rhel8.noarch 4/4 \n Verifying : pgbadger-11.4-1.rhel8.noarch 1/4 \n Verifying : pgcluu-3.1-2.rhel8.noarch 2/4 \n Verifying : perl-Text-CSV_XS-1.40-1.el8.x86_64 3/4 \n Verifying : perl-UNIVERSAL-isa-1.20171012-4.el8.noarch 4/4 \n\nInstalled:\n perl-Text-CSV_XS-1.40-1.el8.x86_64 perl-UNIVERSAL-isa-1.20171012-4.el8.noarch\n pgbadger-11.4-1.rhel8.noarch pgcluu-3.1-2.rhel8.noarch \n\nComplete!\n' lago.ssh: DEBUG: Command 710b9982 on lago-performance-suite-master-engine errors: b'+ pg_datadir=/var/lib/pgsql/data\n+ pg_service=postgresql.service\n+ pgdg=\n+ scl_enable=\n++ get_engine_pg_scl\n++ engine_pg_scl_conf=(/etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf)\n++ local engine_pg_scl_conf\n++ local res=\n++ [[ -e /etc/ovirt-engine/engine.conf.d/*scl-postgres*.conf ]]\n++ echo \'\'\n+ engine_pg_scl=\n+ [[ -n \'\' ]]\n++ rpm -q postgresql\n+ pgver=postgresql-12.1-2.module_el8.1.0+273+979c16e6.x86_64\n+ case "${pgver}" in\n+ pgdg=https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_6... [[ -n https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgd... ]]\n+ yum install -y https://download.postgresql.org/pub/repos/yum/12/redhat/rhel-8-x86_64/pgd... yum install -y pgbadger pgcluu\nwarning: /var/cache/dnf/pgdg-common-38b6c5045045f841/packages/pgbadger-11.4-1.rhel8.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID 442df0f8: NOKEY\nImporting GPG key 0x442DF0F8:\n Userid : "PostgreSQL RPM Building Project <pgsqlrpms-hackers(a)pgfoundry.org&gt;&quot;\n Fingerprint: 68C9 E2B9 1A37 D136 FE74 D176 1F16 D2E1 442D F0F8\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-PGDG\nwarning: /var/cache/dnf/epel-el8-5635746ae2054ad8/packages/perl-Text-CSV_XS-1.40-1.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 2f86d6a1: NOKEY\nImporting GPG key 0x2F86D6A1:\n Userid : "Fedora EPEL (8) <epel(a)fedoraproject.org&gt;&quot;\n Fingerprint: 94E2 79EB 8D8F 25B2 1810 ADF1 21EA 45AB 2F86 D6A1\n From : https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-8\n++ sed 2d\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_directory\'\\\'\'\'\n+ pglogdir=\' log\'\n+ pglogdir=log\n+ cd /var/lib/pgsql/data/log/\n++ su - postgres -c \'psql postgres postgres -t -c \'\\\'\'show log_filename\'\\\'\'\'\n++ sed 2d\n+ pglog=\' postgresql-%m.log\'\n++ date +postgresql-%m.log\n+ pglog=postgresql-12.log\n+ mkdir pgbadger\n+ pgbadger -f stderr postgresql-12.log --outdir pgbadger\nFATAL: logfile "postgresql-12.log" must exist!\n - Error at line 2418\n' --------------------- >> end captured logging << ---------------------