Adding ovn maintainers. On Sun, Dec 24, 2017 at 9:25 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: > Test failed: [ 098_ovirt_provider_ovn.use_ovn_provider ] > > Link to suspected patches: > > - Linked test failed on: > https://gerrit.ovirt.org/#/c/85703/3 > - It seems OVN patches had been failing tests ever since: > https://gerrit.ovirt.org/#/c/85645/2 > > Link to Job: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4492/ > > Link to all logs: > http://jenkins.ovirt.org/job/ovirt-master_change-queue- > tester/4492/artifact/exported-artifacts/basic-suit-master- > el7/test_logs/basic-suite-master/post-098_ovirt_provider_ovn.py/ > > Error snippet from log: > > <error> > > Fault reason is "Operation Failed". Fault detail is "Failed to > communicate with the external provider, see log for additional > details.". HTTP response code is 400. > -------------------- >> begin captured logging << > -------------------- requests.packages.urllib3.connectionpool: > INFO: * Starting new HTTPS connection (1): 192.168.201.4 > py.warnings: WARNING: * Unverified HTTPS request is being made. > Adding certificate verification is strongly advised. See: > https://urllib3.readthedocs.org/en/latest/security.html > requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/tokens/ > HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: > "GET /v2.0/networks/ HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/ports/ > HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/subnets/ > HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: > "POST /v2.0/networks/ HTTP/1.1" 201 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: > "POST /v2.0/subnets/ HTTP/1.1" 201 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: "POST /v2.0/ports/ > HTTP/1.1" 201 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: > "GET /v2.0/networks/ HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/ports/ > HTTP/1.1" 200 None > requests.packages.urllib3.connectionpool: INFO: * Starting new > HTTPS connection (1): 192.168.201.4 > requests.packages.urllib3.connectionpool: DEBUG: "GET /v2.0/subnets/ > HTTP/1.1" 200 None > --------------------- >> end captured logging << > --------------------- > > > > </error> > > > > -- > Barak Korren > RHV DevOps team , RHCE, RHCi > Red Hat EMEA > redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel > > >

On Mon, Dec 25, 2017 at 2:09 PM, Dominik Holler <dholler(a)redhat.com&gt; wrote: > A helpful hint is in > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4492/artifa... : > Caused by: org.jboss.resteasy.spi.ReaderException: > org.codehaus.jackson.map.JsonMappingException: Can not construct > instance of java.util.Calendar from String value '2017-12-27 > 13:19:51Z': not a valid representation (error: Can not parse date > "2017-12-27 13:19:51Z": not compatible with any of standard forms > ("yyyy-MM-dd'T'HH:mm:ss.SSSZ", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", > "EEE, dd MMM yyyy HH:mm:ss zzz", "yyyy-MM-dd")) at [Source: > org.jboss.resteasy.client.core.BaseClientResponse$InputStreamWrapper@72c184c5; > line: 1, column: 23] (through reference chain: > com.woorea.openstack.keystone.model.Access["token"]->com.woorea.openstack.keystone.model.Token["expires"]) > > > This problem was introduced by > https://gerrit.ovirt.org/#/c/85702/ > > I created a fix: > https://gerrit.ovirt.org/85734 Thanks for the quick fix. Is the new format accpetable to other users of the keystone-like API (such at the neutron cli)?

On 25 December 2017 at 14:14, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: > On Mon, Dec 25, 2017 at 2:09 PM, Dominik Holler <dholler(a)redhat.com&gt; wrote: >> A helpful hint is in >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4492/artifa... : >> Caused by: org.jboss.resteasy.spi.ReaderException: org.codehaus.jackson.map.JsonMappingException: Can not construct instance of java.util.Calendar from String value '2017-12-27 13:19:51Z': not a valid representation (error: Can not parse date "2017-12-27 13:19:51Z": not compatible with any of standard forms ("yyyy-MM-dd'T'HH:mm:ss.SSSZ", "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "EEE, dd MMM yyyy HH:mm:ss zzz", "yyyy-MM-dd")) >> at [Source: org.jboss.resteasy.client.core.BaseClientResponse$InputStreamWrapper@72c184c5; line: 1, column: 23] (through reference chain: com.woorea.openstack.keystone.model.Access["token"]->com.woorea.openstack.keystone.model.Token["expires"]) >> >> >> This problem was introduced by >> https://gerrit.ovirt.org/#/c/85702/ >> >> I created a fix: >> https://gerrit.ovirt.org/85734 > > Thanks for the quick fix. > > Is the new format accpetable to other users of the keystone-like API > (such at the neutron cli)? It seems the fix patch itself failed as well: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4539/ The failed test is: 006_migrations.prepare_migration_attachments_ipv6 It seems engine has lost the ability to talk to the host. Logs are here: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4539/artifa...

On Mon, Dec 25, 2017 at 2:09 PM, Dominik Holler <dholler(a)redhat.com&gt;

> A helpful hint is in > > http://jenkins.ovirt.org/job/ovirt-master_change-queue-

> Caused by: org.jboss.resteasy.spi.ReaderException:

> at [Source: org.jboss.resteasy.client.core.BaseClientResponse$

> > > This problem was introduced by > https://gerrit.ovirt.org/#/c/85702/ > > I created a fix: > https://gerrit.ovirt.org/85734 Thanks for the quick fix. Is the new format accpetable to other users of the keystone-like API (such at the neutron cli)?

On Thu, Dec 28, 2017 at 5:24 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > On Dec 27, 2017 12:35 PM, "Barak Korren" <bkorren(a)redhat.com&gt; wrote: > > On 25 December 2017 at 14:14, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> On Mon, Dec 25, 2017 at 2:09 PM, Dominik Holler <dholler(a)redhat.com&gt; >> wrote: >>> A helpful hint is in >>> >>> >>> http://jenkins.ovirt.org/job/ovirt-master_change-queue- tester/4492/artifact/exported-artifacts/basic-suit-master- el7/test_logs/basic-suite-master/post-098_ovirt_ provider_ovn.py/lago-basic-suite-master-engine/_var_log/ ovirt-engine/engine.log >>> : >>> Caused by: org.jboss.resteasy.spi.ReaderException: >>> org.codehaus.jackson.map.JsonMappingException: Can not construct instance of >>> java.util.Calendar from String value '2017-12-27 13:19:51Z': not a valid >>> representation (error: Can not parse date "2017-12-27 13:19:51Z": not >>> compatible with any of standard forms ("yyyy-MM-dd'T'HH:mm:ss.SSSZ", >>> "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "EEE, dd MMM yyyy HH:mm:ss zzz", >>> "yyyy-MM-dd")) >>> at [Source: >>> org.jboss.resteasy.client.core.BaseClientResponse$ InputStreamWrapper@72c184c5; >>> line: 1, column: 23] (through reference chain: >>> com.woorea.openstack.keystone.model.Access["token"]->com. woorea.openstack.keystone.model.Token["expires"]) >>> >>> >>> This problem was introduced by >>> https://gerrit.ovirt.org/#/c/85702/ >>> >>> I created a fix: >>> https://gerrit.ovirt.org/85734 >> >> Thanks for the quick fix. >> >> Is the new format accpetable to other users of the keystone-like API >> (such at the neutron cli)? > > It seems the fix patch itself failed as well: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4539/ > > The failed test is: 006_migrations.prepare_migration_attachments_ipv6 > > > Any updates on this failure? There's a versioning mess in ovirt-provider-ovn, which might cause the fixed not to be tested. Fixing the master branch with https://gerrit.ovirt.org/#/c/85797 (and possibly other patches) is our utmost task for today. Dan.

Yet http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4559/ (which is the gating job for https://gerrit.ovirt.org/#/c/85797/2 ) still fails. Could you look into why, Marcin? The failure seems unrelated to ovn, as it is about a *host* loosing connectivity. But it reproduces too much, so we need to get to the bottom of it.

On 29 December 2017 at 00:22, Barak Korren <bkorren(a)redhat.com&gt; wrote: > On 28 December 2017 at 20:02, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> Yet http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4559/ >> (which is the gating job for https://gerrit.ovirt.org/#/c/85797/2 ) >> still fails. >> Could you look into why, Marcin? >> The failure seems unrelated to ovn, as it is about a *host* loosing >> connectivity. But it reproduces too much, so we need to get to the >> bottom of it. >> > > Re sending the change through the gate yielded a different error: > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4563/ > > If this is still unrelated, we need to think seriously what is raising > this large amount of unrelated failures. We cannot do any accurate > reporting when failures are sporadic. > And here is yet another host connectivity issue failing a test for a change that should have no effect whatsoever (its a tox patch for vdsm): http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4565/

-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On Thu, Dec 28, 2017 at 11:48 PM, Yaniv Kaul <ykaul(a)redhat.com&gt; wrote: > > > > On Fri, Dec 29, 2017 at 12:26 AM, Barak Korren <bkorren(a)redhat.com&gt; wrote: >> >> On 29 December 2017 at 00:22, Barak Korren <bkorren(a)redhat.com&gt; wrote: >> > On 28 December 2017 at 20:02, Dan Kenigsberg <danken(a)redhat.com&gt; wrote: >> >> Yet >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4559/ >> >> (which is the gating job for https://gerrit.ovirt.org/#/c/85797/2 ) >> >> still fails. >> >> Could you look into why, Marcin? >> >> The failure seems unrelated to ovn, as it is about a *host* loosing >> >> connectivity. But it reproduces too much, so we need to get to the >> >> bottom of it. >> >> >> > >> > Re sending the change through the gate yielded a different error: >> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4563/ >> > >> > If this is still unrelated, we need to think seriously what is raising >> > this large amount of unrelated failures. We cannot do any accurate >> > reporting when failures are sporadic. >> > >> >> And here is yet another host connectivity issue failing a test for a >> change that should have no effect whatsoever (its a tox patch for >> vdsm): >> >> http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4565/ > > > I've added a fair number of changes this week. I doubt they are related, > but the one that stands out > is the addition of a fence-agent to one of the hosts. > https://gerrit.ovirt.org/#/c/85817/ disables this specific test, just in > case. > > I don't think it causes an issue, but it's the only one looking at the git > log I can suspect.

Trying to rebuild Barak's build resulted in another fail: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4571/ (with the same problem as Dan's build) Engine log contains a few of "IOException: Broken pipe" which seem to correspond to a vdsm restart: "[vds] Exiting (vdsmd:170)" yet looking at my local successful run, I see the same issues in the log. I don't see any other obvious reasons for the problem so far.