4:04 a.m.
This is a multi-part message in MIME format.
--------------38C81434D2056F5745CD1AD2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi,
Lago reports add_console was the failed action:
<testcase classname="004_basic_sanity" name="add_console"
time="1.765">...</testcase>
In the engine log I can see that both hosts fail to respond and that we
have also have errors in ansible for invalid services in fedora.
CQ reports a network patch as the probable cause.
See further info below
**
*Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/*
*
Link to Job:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/
Link to all logs:
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifa...
(Relevant) error snippet from the log:
<error>
*
**Here is the ansible error: ***
*
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"},
"msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'})
=> {"failed": true, "item": {"service":
"ovirt-vmconsole"}, "msg":
"ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services
are defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) =>
{"failed": true, "item": {"service": "glusterfs"},
"msg": "ERROR:
Exception caught: org.fedoraproject.FirewallD1.Exception:
INVALID_SERVICE: 'glusterfs' not among existing services Services are
defined by port/tcp relationship and named as they are in /etc/services
(on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) =>
{"failed": true, "item": {"service": "ctdb"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true, "item":
{"service":
"ovirt-storageconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are defined
by port/tcp relationship and named as they are in /etc/services (on most
systems)"}
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) =>
{"failed": true, "item": {"service": "nrpe"},
"msg": "ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most systems)"}
**Here is the engine.log error**
*017-09-19 12:28:59,191-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true}
2017-09-19 12:28:59,191-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9"
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:80
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23
{"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
"result": true}^@
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker)
[] Message received: {"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}
2017-09-19 12:28:59,194-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not
able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"
2017-09-19 12:28:59,234-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp
Reactor) [5438f56c] MESSAGE
content-length:8534
destination:jms.topic.vdsm_responses
content-type:application/json
subscription:279e9c67-7fc4-4ace-8709-2534f4d53241
*
*</error>*
**
Thanks.
Dafna
--------------38C81434D2056F5745CD1AD2
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="content-type" content="text/html;
charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi, </p>
<div class="collapsible-content" style="color: rgb(0, 0, 0);
font-family: "Times New Roman"; font-size: medium;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-weight: normal; letter-spacing:
normal; orphans: 2; text-align: start; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
text-decoration-style: initial; text-decoration-color: initial;">
<div class="collapsible" id="collapsible1">
<div class="collapsed hidden">
<div class="line"><span class="html-tag">Lago
reports
add_console was the failed action: <br>
<br>
</span><span class="html-tag"><span
class="html-tag"><testcase<span
class="html-attribute"><span> </span><span
class="html-attribute-name">classname</span>="<span
class="html-attribute-value">004_basic_sanity</span>"</span><span
class="html-attribute"><span> </span><span
class="html-attribute-name">name</span>="<span
class="html-attribute-value">add_console</span>"</span><span
class="html-attribute"><span> </span><span
class="html-attribute-name">time</span>="<span
class="html-attribute-value">1.765</span>"</span>></span><span
class="text">...</span><span
class="html-tag"></testcase><br>
</span> <br>
In the engine log I can see that both hosts fail to
respond and that we have also have errors in ansible for
invalid services in fedora. <br>
<br>
CQ reports a network patch as the probable cause. <br>
<br>
See further info below<br>
<br>
<br class="Apple-interchange-newline">
</span></div>
</div>
</div>
</div>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to suspected patches: <a class="moz-txt-link-freetext"
href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ov...
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to Job: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link
to all logs: <a class="moz-txt-link-freetext"
href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/...
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant)
error snippet from the log: </span></p>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p>
</b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
is the ansible error: </b></b><b
style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><br>
</b></p>
<p><span
style="font-weight:normal;">host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-vmconsole'}) => {"failed": true,
"item": {"service":
"ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-vmconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'glusterfs'}) => {"failed": true, "item":
{"service":
"glusterfs"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'glusterfs' not among existing services Services are defined by
port/tcp relationship and named as they are in /etc/services (on
most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'ctdb'})
=> {"failed": true, "item": {"service":
"ctdb"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true,
"item":
{"service": "ovirt-storageconsole"}, "msg":
"ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed:
[lago-basic-suite-master-host-0] (item={u'service': u'nrpe'})
=> {"failed": true, "item": {"service":
"nrpe"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-vmconsole'}) => {"failed": true,
"item": {"service":
"ovirt-vmconsole"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-vmconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'glusterfs'}) => {"failed": true, "item":
{"service":
"glusterfs"}, "msg": "ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'glusterfs' not among existing services Services are defined by
port/tcp relationship and named as they are in /etc/services (on
most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'ctdb'})
=> {"failed": true, "item": {"service":
"ctdb"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service':
u'ovirt-storageconsole'}) => {"failed": true,
"item":
{"service": "ovirt-storageconsole"}, "msg":
"ERROR: Exception
caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE:
'ovirt-storageconsole' not among existing services Services are
defined by port/tcp relationship and named as they are in
/etc/services (on most systems)"}<br>
host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed:
[lago-basic-suite-master-host-1] (item={u'service': u'nrpe'})
=> {"failed": true, "item": {"service":
"nrpe"}, "msg":
"ERROR: Exception caught:
org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe'
not among existing services Services are defined by port/tcp
relationship and named as they are in /etc/services (on most
systems)"}<br>
<br>
</span></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here
is the engine.log error</b></b></p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">017-09-19
12:28:59,191-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
(ResponseWorker) [] Message received: {"jsonrpc": "2.0",
"id":
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result":
true}<br>
2017-09-19 12:28:59,191-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
[] Not able to update response for
"634fae71-5f8f-49c1-9d5f-ebc24239cfa9"<br>
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
Stomp Reactor) [5438f56c] MESSAGE<br>
content-length:80<br>
destination:jms.topic.vdsm_responses<br>
content-type:application/json<br>
subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23<br>
<br>
{"jsonrpc": "2.0", "id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b",
"result": true}^@<br>
2017-09-19 12:28:59,194-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker]
(ResponseWorker) [] Message received: {"jsonrpc": "2.0",
"id":
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result":
true}<br>
2017-09-19 12:28:59,194-04 ERROR
[org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker)
[] Not able to update response for
"b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"<br>
2017-09-19 12:28:59,234-04 DEBUG
[org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL
Stomp Reactor) [5438f56c] MESSAGE<br>
content-length:8534<br>
destination:jms.topic.vdsm_responses<br>
content-type:application/json<br>
subscription:279e9c67-7fc4-4ace-8709-2534f4d53241<br>
<br>
<p dir="ltr"
style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span
style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p>
</b></p>
<p><br>
</p>
<p>Thanks. <br>
</p>
<p>Dafna</p>
<p><br class="Apple-interchange-newline">
</p>
</body>
</html>
--------------38C81434D2056F5745CD1AD2--