Subject: [ OST Failure Report ] [ master ] [ 19-09-2017E ] [ 004_basic_sanity ]

Wednesday, 20 September 2017

This is a multi-part message in MIME format.
--------------38C81434D2056F5745CD1AD2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Hi, 

Lago reports add_console was the failed action:

<testcase classname="004_basic_sanity" name="add_console"
time="1.765">...</testcase>

In the engine log I can see that both hosts fail to respond and that we
have also have errors in ansible for invalid services in fedora.

CQ reports a network patch as the probable cause.

See further info below

**

*Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/*

*

Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/

Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifa...

(Relevant) error snippet from the log:

<error>

*

**Here is the ansible error: ***
*

host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"},
"msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"},
"msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}

**Here is the engine.log error**

*017-09-19 12:28:59,191-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true}
2017-09-19 12:28:59,191-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9"
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:80
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23

{"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
"result": true}^@
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}
2017-09-19 12:28:59,194-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"
2017-09-19 12:28:59,234-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:8534
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:279e9c67-7fc4-4ace-8709-2534f4d53241

*

*</error>*

**

Thanks.

Dafna

--------------38C81434D2056F5745CD1AD2
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit

<html>
  <head>

    <meta http-equiv="content-type" content="text/html;
charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <p>Hi, </p>
    <div class="collapsible-content" style="color: rgb(0, 0, 0);
      font-family: &quot;Times New Roman&quot;; font-size: medium;
      font-style: normal; font-variant-ligatures: normal;
      font-variant-caps: normal; font-weight: normal; letter-spacing:
      normal; orphans: 2; text-align: start; text-indent: 0px;
      text-transform: none; white-space: normal; widows: 2;
      word-spacing: 0px; -webkit-text-stroke-width: 0px;
      text-decoration-style: initial; text-decoration-color: initial;">
      <div class="collapsible" id="collapsible1">
        <div class="collapsed hidden">
          <div class="line"><span class="html-tag">Lago
reports
              add_console was the failed action: <br>
              <br>
            </span><span class="html-tag"><span
class="html-tag">&lt;testcase<span
                  class="html-attribute"><span> </span><span

class="html-attribute-name">classname</span>="<span

class="html-attribute-value">004_basic_sanity</span>"</span><span
                  class="html-attribute"><span> </span><span

class="html-attribute-name">name</span>="<span

class="html-attribute-value">add_console</span>"</span><span
                  class="html-attribute"><span> </span><span

class="html-attribute-name">time</span>="<span

class="html-attribute-value">1.765</span>"</span>&gt;</span><span
                class="text">...</span><span
class="html-tag">&lt;/testcase&gt;<br>
              </span> <br>
              In the engine log I can see that both hosts fail to
              respond and that we have also have errors in ansible for
              invalid services in fedora. <br>
              <br>
              CQ reports a network patch as the probable cause. <br>
              <br>
              See further info below<br>
              <br>
              <br class="Apple-interchange-newline">
            </span></div>
        </div>
      </div>
    </div>
    <p><b style="font-weight:normal;"
        id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: <a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ov...
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">&lt;error&gt;</span></p>
      </b></p>
    <p><b style="font-weight:normal;"

id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
          is the ansible error: </b></b><b
style="font-weight:normal;"

id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><br>
      </b></p>
    <p><span
style="font-weight:normal;">host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
        [lago-basic-suite-master-host-0] (item={u'service':
        u'ovirt-vmconsole'}) =&gt; {"failed": true,
"item": {"service":
        "ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'ovirt-vmconsole' not among existing services Services are
        defined by port/tcp relationship and named as they are in
        /etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
        [lago-basic-suite-master-host-0] (item={u'service':
        u'glusterfs'}) =&gt; {"failed": true, "item":
{"service":
        "glusterfs"}, "msg": "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'glusterfs' not among existing services Services are defined by
        port/tcp relationship and named as they are in /etc/services (on
        most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
        [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'})
        =&gt; {"failed": true, "item": {"service":
"ctdb"}, "msg":
        "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
        not among existing services Services are defined by port/tcp
        relationship and named as they are in /etc/services (on most
        systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
        [lago-basic-suite-master-host-0] (item={u'service':
        u'ovirt-storageconsole'}) =&gt; {"failed": true,
"item":
        {"service": "ovirt-storageconsole"}, "msg":
"ERROR: Exception
        caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'ovirt-storageconsole' not among existing services Services are
        defined by port/tcp relationship and named as they are in
        /etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
        [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'})
        =&gt; {"failed": true, "item": {"service":
"nrpe"}, "msg":
        "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
        not among existing services Services are defined by port/tcp
        relationship and named as they are in /etc/services (on most
        systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
        [lago-basic-suite-master-host-1] (item={u'service':
        u'ovirt-vmconsole'}) =&gt; {"failed": true,
"item": {"service":
        "ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'ovirt-vmconsole' not among existing services Services are
        defined by port/tcp relationship and named as they are in
        /etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
        [lago-basic-suite-master-host-1] (item={u'service':
        u'glusterfs'}) =&gt; {"failed": true, "item":
{"service":
        "glusterfs"}, "msg": "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'glusterfs' not among existing services Services are defined by
        port/tcp relationship and named as they are in /etc/services (on
        most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
        [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'})
        =&gt; {"failed": true, "item": {"service":
"ctdb"}, "msg":
        "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
        not among existing services Services are defined by port/tcp
        relationship and named as they are in /etc/services (on most
        systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
        [lago-basic-suite-master-host-1] (item={u'service':
        u'ovirt-storageconsole'}) =&gt; {"failed": true,
"item":
        {"service": "ovirt-storageconsole"}, "msg":
"ERROR: Exception
        caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
        'ovirt-storageconsole' not among existing services Services are
        defined by port/tcp relationship and named as they are in
        /etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
        [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'})
        =&gt; {"failed": true, "item": {"service":
"nrpe"}, "msg":
        "ERROR: Exception caught:
        org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
        not among existing services Services are defined by port/tcp
        relationship and named as they are in /etc/services (on most
        systems)"}<br>
        <br>
      </span></p>
    <p><b style="font-weight:normal;"

id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
          is the engine.log error</b></b></p>
    <p><b style="font-weight:normal;"

id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">017-09-19
        12:28:59,191-04 DEBUG
        [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
        (ResponseWorker) [] Message received: {"jsonrpc": "2.0",
"id":
        "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result":
true}<br>
        2017-09-19 12:28:59,191-04 ERROR
        [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
        [] Not able to update response for
        "634fae71-5f8f-49c1-9d5f-ebc24239cfa9"<br>
        2017-09-19 12:28:59,194-04 DEBUG
        [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
        Stomp Reactor) [5438f56c] MESSAGE<br>
        content-length:80<br>
        destination:jms.topic.vdsm_responses<br>
        content-type:application/json<br>
        subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23<br>
        <br>
        {"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
        "result": true}^@<br>
        2017-09-19 12:28:59,194-04 DEBUG
        [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
        (ResponseWorker) [] Message received: {"jsonrpc": "2.0",
"id":
        "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result":
true}<br>
        2017-09-19 12:28:59,194-04 ERROR
        [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
        [] Not able to update response for
        "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"<br>
        2017-09-19 12:28:59,234-04 DEBUG
        [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
        Stomp Reactor) [5438f56c] MESSAGE<br>
        content-length:8534<br>
        destination:jms.topic.vdsm_responses<br>
        content-type:application/json<br>
        subscription:279e9c67-7fc4-4ace-8709-2534f4d53241<br>
        <br>
        <p dir="ltr"
          style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">&lt;/error&gt;</span></p>
      </b></p>
    <p><br>
    </p>
    <p>Thanks. <br>
    </p>
    <p>Dafna</p>
    <p><br class="Apple-interchange-newline">
    </p>
  </body>
</html>

--------------38C81434D2056F5745CD1AD2--

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011