Project: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/ Build: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/113/ Build Number: 113 Build Status: Still Failing Triggered By: Started by timer ------------------------------------- Changes Since Last Success: ------------------------------------- Changes for Build #111 [Roy Golan] Add metrics to basic suite [Sandro Bonazzola] ovirt-release: add fc26 support [Sandro Bonazzola] cockpit-ovirt: add fedora 26 Changes for Build #112 [Roy Golan] Add metrics to basic suite Changes for Build #113 [Benny Zlotnik] Add live storage migration test ----------------- Failed Tests: ----------------- 1 tests failed. FAILED: 002_bootstrap.wait_engine Error Message: None != True after 600 seconds Stack Trace: Traceback (most recent call last): File "/usr/lib64/python2.7/unittest/case.py", line 369, in run testMethod() File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest self.test(*self.arg) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 129, in wrapped_test test() File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, in wrapper return func(get_test_prefix(), *args, **kwargs) File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ ovirt-system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", line 102, in wait_engine testlib.assert_true_within(_engine_is_up, timeout=10 * 60) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 256, in assert_true_within assert_equals_within(func, True, timeout, allowed_exceptions) File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 230, in assert_equals_within '%s != %s after %s seconds' % (res, value, timeout) AssertionError: None != True after 600 seconds

On Tue, May 30, 2017 at 5:53 AM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: > Project: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/ > Build: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/115/ I'm not able to debug this. As far as I can tell, engine-setup logs are truncated before engine-setup termination.

> > Build Number: 115 > Build Status: Still Failing > Triggered By: Started by timer > > ------------------------------------- > Changes Since Last Success: > ------------------------------------- > Changes for Build #111 > [Roy Golan] Add metrics to basic suite > > [Sandro Bonazzola] ovirt-release: add fc26 support > > [Sandro Bonazzola] cockpit-ovirt: add fedora 26 > > > Changes for Build #112 > [Roy Golan] Add metrics to basic suite > > > Changes for Build #113 > [Benny Zlotnik] Add live storage migration test > > > Changes for Build #114 > [Daniel Belenky] Adding change resolver to check-patch for better changes > detection. > > [Sandro Bonazzola] ovirt-engine-nodejs: add fedora 26 > > [Sandro Bonazzola] ovirt-engine-nodejs-modules: add fedora 26 > > [Sandro Bonazzola] ovirt-engine-yarn: add fedora 26 > > [Eyal Edri] remove ovirt-web-ui project > > [Barak Korren] Add PyYAML package to all slaves > > [Barak Korren] Add retries to mirror_client on JSON errors > > [Barak Korren] Enabled loading mirrors data from local file > > [Barak Korren] Added GerritMergedChange class > > > Changes for Build #115 > [Daniel Belenky] Adding change resolver to check-patch for better changes > detection. > > > > > ----------------- > Failed Tests: > ----------------- > 1 tests failed. > FAILED: 002_bootstrap.wait_engine > > Error Message: > None != True after 600 seconds > > Stack Trace: > Traceback (most recent call last): > File "/usr/lib64/python2.7/unittest/case.py", line 369, in run > testMethod() > File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in > runTest > self.test(*self.arg) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line > 129, in wrapped_test > test() > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line 59, > in wrapper > return func(get_test_prefix(), *args, **kwargs) > File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ovirt- > system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", > line 102, in wait_engine > testlib.assert_true_within(_engine_is_up, timeout=10 * 60) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line > 256, in assert_true_within > assert_equals_within(func, True, timeout, allowed_exceptions) > File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line > 230, in assert_equals_within > '%s != %s after %s seconds' % (res, value, timeout) > AssertionError: None != True after 600 seconds > _______________________________________________ > Infra mailing list > Infra(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/infra > > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

On Tue, May 30, 2017 at 9:50 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > > > On Tue, May 30, 2017 at 12:30 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; > wrote: > >> >> >> On Tue, May 30, 2017 at 5:53 AM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >> >>> Project: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/ >>> Build: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/115/ >> >> >> I'm not able to debug this. >> As far as I can tell, engine-setup logs are truncated before >> engine-setup termination. >> > > postgresql not running? > > 2017-05-29 23:11:03,016-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1179 dbenv: {'OVESETUP_DB/database': 'engine', 'OVESETUP_DB/host': 'localhost', 'OVESETUP_DB/user': 'engine', 'OVESETUP_DB/secured': False, 'OVESETUP_DB/securedHostValidation': False, 'OVESETUP_DB/password': '**FILTERED**', 'OVESETUP_DB/port': 5432} > 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:212 Database: 'None', Statement: ' > select count(*) as count > from pg_catalog.pg_tables > where schemaname = 'public'; > ', args: {} > 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:217 Creating own connection > 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1211 database connection failed > Traceback (most recent call last): > File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 1209, in getCredentials > ] = self.isNewDatabase() > File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 405, in isNewDatabase > transaction=False, > File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 226, in execute > database=database, > File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 160, in connect > sslmode=sslmode, > File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect > conn = _connect(dsn, connection_factory=connection_factory, async=async) > OperationalError: could not connect to server: Connection refused > Is the server running on host "localhost" (::1) and accepting > TCP/IP connections on port 5432? > could not connect to server: Connection refused > Is the server running on host "localhost" (127.0.0.1) and accepting > TCP/IP connections on port 5432? > > Not sure. postgresql not running shouldn't truncate engine-setup logs. Adding Didi and Simone

> > >> >> >> >> >>> >>> Build Number: 115 >>> Build Status: Still Failing >>> Triggered By: Started by timer >>> >>> ------------------------------------- >>> Changes Since Last Success: >>> ------------------------------------- >>> Changes for Build #111 >>> [Roy Golan] Add metrics to basic suite >>> >>> [Sandro Bonazzola] ovirt-release: add fc26 support >>> >>> [Sandro Bonazzola] cockpit-ovirt: add fedora 26 >>> >>> >>> Changes for Build #112 >>> [Roy Golan] Add metrics to basic suite >>> >>> >>> Changes for Build #113 >>> [Benny Zlotnik] Add live storage migration test >>> >>> >>> Changes for Build #114 >>> [Daniel Belenky] Adding change resolver to check-patch for better >>> changes detection. >>> >>> [Sandro Bonazzola] ovirt-engine-nodejs: add fedora 26 >>> >>> [Sandro Bonazzola] ovirt-engine-nodejs-modules: add fedora 26 >>> >>> [Sandro Bonazzola] ovirt-engine-yarn: add fedora 26 >>> >>> [Eyal Edri] remove ovirt-web-ui project >>> >>> [Barak Korren] Add PyYAML package to all slaves >>> >>> [Barak Korren] Add retries to mirror_client on JSON errors >>> >>> [Barak Korren] Enabled loading mirrors data from local file >>> >>> [Barak Korren] Added GerritMergedChange class >>> >>> >>> Changes for Build #115 >>> [Daniel Belenky] Adding change resolver to check-patch for better >>> changes detection. >>> >>> >>> >>> >>> ----------------- >>> Failed Tests: >>> ----------------- >>> 1 tests failed. >>> FAILED: 002_bootstrap.wait_engine >>> >>> Error Message: >>> None != True after 600 seconds >>> >>> Stack Trace: >>> Traceback (most recent call last): >>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >>> testMethod() >>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in >>> runTest >>> self.test(*self.arg) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>> 129, in wrapped_test >>> test() >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>> 59, in wrapper >>> return func(get_test_prefix(), *args, **kwargs) >>> File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ovirt- >>> system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", >>> line 102, in wait_engine >>> testlib.assert_true_within(_engine_is_up, timeout=10 * 60) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>> 256, in assert_true_within >>> assert_equals_within(func, True, timeout, allowed_exceptions) >>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>> 230, in assert_equals_within >>> '%s != %s after %s seconds' % (res, value, timeout) >>> AssertionError: None != True after 600 seconds >>> _______________________________________________ >>> Infra mailing list >>> Infra(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/infra >>> >>> >> >> >> -- >> >> SANDRO BONAZZOLA >> >> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >> >> Red Hat EMEA <https://www.redhat.com/> >> <https://red.ht/sig> >> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >> > > -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

On Tue, May 30, 2017 at 9:57 AM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; wrote: > > > On Tue, May 30, 2017 at 9:50 AM, Sahina Bose <sabose(a)redhat.com&gt; wrote: > >> >> >> On Tue, May 30, 2017 at 12:30 PM, Sandro Bonazzola <sbonazzo(a)redhat.com&gt; >> wrote: >> >>> >>> >>> On Tue, May 30, 2017 at 5:53 AM, <jenkins(a)jenkins.phx.ovirt.org&gt; wrote: >>> >>>> Project: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/ >>>> Build: http://jenkins.ovirt.org/job/ovirt_master_hc-system-tests/115/ >>> >>> >>> I'm not able to debug this. >>> As far as I can tell, engine-setup logs are truncated before >>> engine-setup termination. >>> >> >> postgresql not running? >> >> 2017-05-29 23:11:03,016-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1179 dbenv: {'OVESETUP_DB/database': 'engine', 'OVESETUP_DB/host': 'localhost', 'OVESETUP_DB/user': 'engine', 'OVESETUP_DB/secured': False, 'OVESETUP_DB/securedHostValidation': False, 'OVESETUP_DB/password': '**FILTERED**', 'OVESETUP_DB/port': 5432} >> 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:212 Database: 'None', Statement: ' >> select count(*) as count >> from pg_catalog.pg_tables >> where schemaname = 'public'; >> ', args: {} >> 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:217 Creating own connection >> 2017-05-29 23:11:03,017-0400 DEBUG otopi.ovirt_engine_setup.engine_common.database database.getCredentials:1211 database connection failed >> Traceback (most recent call last): >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 1209, in getCredentials >> ] = self.isNewDatabase() >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 405, in isNewDatabase >> transaction=False, >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 226, in execute >> database=database, >> File "/usr/share/ovirt-engine/setup/ovirt_engine_setup/engine_common/database.py", line 160, in connect >> sslmode=sslmode, >> File "/usr/lib64/python2.7/site-packages/psycopg2/__init__.py", line 164, in connect >> conn = _connect(dsn, connection_factory=connection_factory, async=async) >> OperationalError: could not connect to server: Connection refused >> Is the server running on host "localhost" (::1) and accepting >> TCP/IP connections on port 5432? >> could not connect to server: Connection refused >> Is the server running on host "localhost" (127.0.0.1) and accepting >> TCP/IP connections on port 5432? >> >> > Not sure. postgresql not running shouldn't truncate engine-setup logs. > Adding Didi and Simone > That one is failing at customization but, if I'm not wrong, we are not going to start the DB till misc stage. On my opinion is a regression somewhere. But I don't expect it's going to make engine-setup crash/truncate the logs just by itself.

That job is failing since http://jenkins.ovirt.org/job/ovirt_master_hc- system-tests/111/ from May 27th and it seams it's always hanging at: 2017-05-26 23:10:54,739-0400 DEBUG otopi.plugins.otopi.network.firewalld plugin.executeRaw:813 execute: ('/bin/firewall-cmd', '--list-all-zones'), executable='None', cwd='None', env=None > > > > >> >> >>> >>> >>> >>> >>>> >>>> Build Number: 115 >>>> Build Status: Still Failing >>>> Triggered By: Started by timer >>>> >>>> ------------------------------------- >>>> Changes Since Last Success: >>>> ------------------------------------- >>>> Changes for Build #111 >>>> [Roy Golan] Add metrics to basic suite >>>> >>>> [Sandro Bonazzola] ovirt-release: add fc26 support >>>> >>>> [Sandro Bonazzola] cockpit-ovirt: add fedora 26 >>>> >>>> >>>> Changes for Build #112 >>>> [Roy Golan] Add metrics to basic suite >>>> >>>> >>>> Changes for Build #113 >>>> [Benny Zlotnik] Add live storage migration test >>>> >>>> >>>> Changes for Build #114 >>>> [Daniel Belenky] Adding change resolver to check-patch for better >>>> changes detection. >>>> >>>> [Sandro Bonazzola] ovirt-engine-nodejs: add fedora 26 >>>> >>>> [Sandro Bonazzola] ovirt-engine-nodejs-modules: add fedora 26 >>>> >>>> [Sandro Bonazzola] ovirt-engine-yarn: add fedora 26 >>>> >>>> [Eyal Edri] remove ovirt-web-ui project >>>> >>>> [Barak Korren] Add PyYAML package to all slaves >>>> >>>> [Barak Korren] Add retries to mirror_client on JSON errors >>>> >>>> [Barak Korren] Enabled loading mirrors data from local file >>>> >>>> [Barak Korren] Added GerritMergedChange class >>>> >>>> >>>> Changes for Build #115 >>>> [Daniel Belenky] Adding change resolver to check-patch for better >>>> changes detection. >>>> >>>> >>>> >>>> >>>> ----------------- >>>> Failed Tests: >>>> ----------------- >>>> 1 tests failed. >>>> FAILED: 002_bootstrap.wait_engine >>>> >>>> Error Message: >>>> None != True after 600 seconds >>>> >>>> Stack Trace: >>>> Traceback (most recent call last): >>>> File "/usr/lib64/python2.7/unittest/case.py", line 369, in run >>>> testMethod() >>>> File "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in >>>> runTest >>>> self.test(*self.arg) >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>>> 129, in wrapped_test >>>> test() >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>>> 59, in wrapper >>>> return func(get_test_prefix(), *args, **kwargs) >>>> File "/home/jenkins/workspace/ovirt_master_hc-system-tests/ovirt- >>>> system-tests/hc-basic-suite-master/test-scenarios/002_bootstrap.py", >>>> line 102, in wait_engine >>>> testlib.assert_true_within(_engine_is_up, timeout=10 * 60) >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>>> 256, in assert_true_within >>>> assert_equals_within(func, True, timeout, allowed_exceptions) >>>> File "/usr/lib/python2.7/site-packages/ovirtlago/testlib.py", line >>>> 230, in assert_equals_within >>>> '%s != %s after %s seconds' % (res, value, timeout) >>>> AssertionError: None != True after 600 seconds >>>> _______________________________________________ >>>> Infra mailing list >>>> Infra(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/infra >>>> >>>> >>> >>> >>> -- >>> >>> SANDRO BONAZZOLA >>> >>> ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D >>> >>> Red Hat EMEA <https://www.redhat.com/> >>> <https://red.ht/sig> >>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >>> >> >> > > > -- > > SANDRO BONAZZOLA > > ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D > > Red Hat EMEA <https://www.redhat.com/> > <https://red.ht/sig> > TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> >