This is a multi-part message in MIME format. --------------38C81434D2056F5745CD1AD2 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, Lago reports add_console was the failed action: <testcase classname="004_basic_sanity" name="add_console" time="1.765">...</testcase> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. CQ reports a network patch as the probable cause. See further info below ** *Link to suspected patches: https://gerrit.ovirt.org/#/c/81921/* * Link to Job: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/ Link to all logs: http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact/... (Relevant) error snippet from the log: <error> * **Here is the ansible error: *** * host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"} **Here is the engine.log error** *017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true} 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9" 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:80 destination:jms.topic.vdsm_responses content-type:application/json subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23 {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@ 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true} 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b" 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE content-length:8534 destination:jms.topic.vdsm_responses content-type:application/json subscription:279e9c67-7fc4-4ace-8709-2534f4d53241 * *</error>* ** Thanks. Dafna --------------38C81434D2056F5745CD1AD2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi, </p> <div class="collapsible-content" style="color: rgb(0, 0, 0); font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial;"> <div class="collapsible" id="collapsible1"> <div class="collapsed hidden"> <div class="line"><span class="html-tag">Lago reports add_console was the failed action: <br> <br> </span><span class="html-tag"><span class="html-tag"><testcase<span class="html-attribute"><span> </span><span class="html-attribute-name">classname</span>="<span class="html-attribute-value">004_basic_sanity</span>"</span><span class="html-attribute"><span> </span><span class="html-attribute-name">name</span>="<span class="html-attribute-value">add_console</span>"</span><span class="html-attribute"><span> </span><span class="html-attribute-name">time</span>="<span class="html-attribute-value">1.765</span>"</span>></span><span class="text">...</span><span class="html-tag"></testcase><br> </span> <br> In the engine log I can see that both hosts fail to respond and that we have also have errors in ansible for invalid services in fedora. <br> <br> CQ reports a network patch as the probable cause. <br> <br> See further info below<br> <br> <br class="Apple-interchange-newline"> </span></div> </div> </div> </div> <p><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to suspected patches: <a class="moz-txt-link-freetext" href="https://gerrit.ovirt.org/#/c/81921/">https://gerrit.ovirt.org/#/c/81921/</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to Job: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">Link to all logs: <a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact/exported-artifacts/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/2719/artifact/exported-artifacts/</a></span></p> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;">(Relevant) error snippet from the log: </span></p> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"><error></span></p> </b></p> <p><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here is the ansible error: </b></b><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><br> </b></p> <p><span style="font-weight:normal;">host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122826-lago-basic-suite-master-host-0-7cf4e7c2.log:failed: [lago-basic-suite-master-host-0] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-vmconsole'}) => {"failed": true, "item": {"service": "ovirt-vmconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-vmconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'glusterfs'}) => {"failed": true, "item": {"service": "glusterfs"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'glusterfs' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ctdb'}) => {"failed": true, "item": {"service": "ctdb"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ctdb' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'ovirt-storageconsole'}) => {"failed": true, "item": {"service": "ovirt-storageconsole"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'ovirt-storageconsole' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> host-deploy/ovirt-host-deploy-ansible-20170919122827-lago-basic-suite-master-host-1-1b1135b0.log:failed: [lago-basic-suite-master-host-1] (item={u'service': u'nrpe'}) => {"failed": true, "item": {"service": "nrpe"}, "msg": "ERROR: Exception caught: org.fedoraproject.FirewallD1.Exception: INVALID_SERVICE: 'nrpe' not among existing services Services are defined by port/tcp relationship and named as they are in /etc/services (on most systems)"}<br> <br> </span></p> <p><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271"><b>Here is the engine.log error</b></b></p> <p><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-9e7f-17f6-7959-14c05b062271">017-09-19 12:28:59,191-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "634fae71-5f8f-49c1-9d5f-ebc24239cfa9", "result": true}<br> 2017-09-19 12:28:59,191-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "634fae71-5f8f-49c1-9d5f-ebc24239cfa9"<br> 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE<br> content-length:80<br> destination:jms.topic.vdsm_responses<br> content-type:application/json<br> subscription:f7138b74-fed3-4eb2-9fdc-957bc4af8f23<br> <br> {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}^@<br> 2017-09-19 12:28:59,194-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b", "result": true}<br> 2017-09-19 12:28:59,194-04 ERROR [org.ovirt.vdsm.jsonrpc.client.JsonRpcClient] (ResponseWorker) [] Not able to update response for "b4842d9b-38f1-4fd4-a4de-a6b42f25d10b"<br> 2017-09-19 12:28:59,234-04 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (SSL Stomp Reactor) [5438f56c] MESSAGE<br> content-length:8534<br> destination:jms.topic.vdsm_responses<br> content-type:application/json<br> subscription:279e9c67-7fc4-4ace-8709-2534f4d53241<br> <br> <p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:11pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap;"></error></span></p> </b></p> <p><br> </p> <p>Thanks. <br> </p> <p>Dafna</p> <p><br class="Apple-interchange-newline"> </p> </body> </html> --------------38C81434D2056F5745CD1AD2--