<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Milan,</div><div class=""><br class=""></div>log [1], VM b3962e5c-08e3-444e-910e-ea675fa1a5c7<div class="">migration away finished at 2017-12-05 06:26:24,515-0500</div><div class="">incoming migration of the same VM back, at 2017-12-05 06:26:46,614-0500</div><div class=""><br class=""></div><div class="">seems to me the migration away didn’t really properly clean up the VM. Milane, can you check that if logs matches? Perhaps orphaned libvirt’s xml?</div><div class="">Otherwise it would need reproduction in CI and some more logs….</div><div class=""><br class=""></div><div class="">[1] <a href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log" class="">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log</a></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On 5 Dec 2017, at 13:26, Dafna Ron <<a href="mailto:dron@redhat.com" class="">dron@redhat.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">
<meta http-equiv="content-type" content="text/html; charset=utf-8" class="">
<div text="#000000" bgcolor="#FFFFFF" class=""><p class="">Hi, <br class="">
</p><p class="">We had a failure for test 006_migrations.migrate_vm on master. </p><p class="">There was a libvirt disruption in the migration src and after
that vdsm reported the migration as failed because the vm already
exists which makes me suspect a split bran case. <br class="">
</p><p class="">The patch reported has nothing to do with this issue and I think
we simply stumbled on a race condition which can cause a split
brain. <br class="">
</p><p class="">Please note that there are several metrics related issues
reported in vdsm logs as well. <br class="">
</p><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class="">
</b></p><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link and headline of suspected patches:
</span></b></div><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">
</span></p><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""><b class="">Not related </b>
</span></div>
<br class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to Job:</span></div>
<br class="">
</b><div class=""><br class="webkit-block-placeholder"></div><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><b class=""><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/</a></b></b></p><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><br class="">
</b></p><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">Link to all logs:</span></b></div><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class="">
<br class="">
</b><div class=""><br class="webkit-block-placeholder"></div><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><b class=""><a class="moz-txt-link-freetext" href="http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-006_migrations.py/</a></b></b></p><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><br class="">
</b></p><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">(Relevant) error snippet from the log: </span></b></div><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""><error></span></b></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">
</span></b></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""><b class="">Engine: </b>
</span></b></div><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class="">2017-12-05 06:26:48,546-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (EE-ManagedThreadFactory-engine-Thread-385) [] EVENT_ID: VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: vm0, Source: lago-
basic-suite-master-host-1, Destination: lago-basic-suite-master-host-0).
</span></b></div>
<br class="">
<b class="">dst:</b><b class=""><br class="">
</b></b><div class=""><br class="webkit-block-placeholder"></div><p class=""><b class=""><span style="font-weight:normal;" class="">2017-12-05 06:26:46,615-0500
WARN (jsonrpc/6) [vds] vm
b3962e5c-08e3-444e-910e-ea675fa1a5c7 already exists (API:179)<br class="">
2017-12-05 06:26:46,615-0500 ERROR (jsonrpc/6) [api] FINISH
create error=Virtual machine already exists (api:124)<br class="">
Traceback (most recent call last):<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line 117, in method<br class="">
ret = func(*args, **kwargs)<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line
180, in create<br class="">
raise exception.VMExists()<br class="">
VMExists: Virtual machine already exists<br class="">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt]
FINISH create return={'status': {'message': 'Virtual machine
already exists', 'code': 4}} from=::ffff:192.168.201.3,50394
(api:52)<br class="">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt]
FINISH migrationCreate return={'status': {'message': 'Virtual
machine already exists', 'code': 4}}
from=::ffff:192.168.201.3,50394 (api:52)<br class="">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6)
[jsonrpc.JsonRpcServer] RPC call VM.migrationCreate failed
(error 4) in 0.03 seconds (__init__:573)<br class="">
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [api.virt]
START destroy(gracefulAttempts=1)
from=::ffff:192.168.201.3,50394 (api:46)<br class="">
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM
resources (vm:4967)<br class="">
2017-12-05 06:26:46,625-0500 WARN (jsonrpc/2) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') trying to set
state to Powering down when already Down (vm:575)<br class="">
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping
connection (guestagent:435)<br class="">
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7')
_destroyVmGraceful attempt #0 (vm:5004)<br class="">
2017-12-05 06:26:46,626-0500 WARN (jsonrpc/2) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') VM
'b3962e5c-08e3-444e-910e-ea675fa1a5c7' couldn't be destroyed
in libvirt: Requested operation is not valid: domain is not
running (vm<br class="">
:5025)<br class="">
2017-12-05 06:26:46,627-0500 INFO (jsonrpc/2) [vdsm.api]
START
teardownImage(sdUUID='952bb427-b88c-4fbe-99ef-49970d3aaf70',
spUUID='9dcfeaaf-96b7-4e26-a327-5570e0e39261',
imgUUID='e6eadbae-ec7a-48f4-8832-64a622a12bef', volUUID=None)
from<br class="">
=::ffff:192.168.201.3,50394,
task_id=2da93725-5533-4354-a369-751eb44f9ef2 (api:46)<br class="">
</span></b></p><p class=""><b class=""><span style="font-weight:normal;" class=""><b class="">scr </b><br class="">
</span></b></p><p class=""><b class=""><span style="font-weight:normal;" class="">2017-12-05 06:26:46,623-0500
ERROR (migsrc/b3962e5c) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') migration
destination error: Virtual machine already exists
(migration:290)<br class="">
</span></b></p><p class=""><b class=""><span style="font-weight:normal;" class=""><b class="">disruption on </b><b class="">dst</b><b class="">:
</b><br class="">
</span></b></p><p class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><b class="">2017-12-05
06:20:04,662-0500 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer]
RPC call VM.shutdown succeeded in 0.00 seconds (__init__:573)<br class="">
2017-12-05 06:20:04,676-0500 ERROR (Thread-1) [root] Shutdown
by QEMU Guest Agent failed (vm:5097)<br class="">
Traceback (most recent call last):<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 5088, in qemuGuestAgentShutdown<br class="">
self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)<br class="">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line 98, in f<br class="">
ret = attr(*args, **kwargs)<br class="">
File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 126, in wrapper<br class="">
ret = f(*args, **kwargs)<br class="">
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line
512, in wrapper<br class="">
return func(inst, *args, **kwargs)<br class="">
File "/usr/lib64/python2.7/site-packages/libvirt.py", line
2403, in shutdownFlags<br class="">
if ret == -1: raise libvirtError
('virDomainShutdownFlags() failed', dom=self)<br class="">
libvirtError: Guest agent is not responding: QEMU guest agent
is not connected<br class="">
2017-12-05 06:20:04,697-0500 INFO (libvirt/events) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') block threshold
1 exceeded on 'vda'
(/rhev/data-center/mnt/blockSD/952bb427-b88c-4fbe-99ef-49970d3aaf70/images/e6eadbae-ec7a-48f4-<br class="">
8832-64a622a12bef/1f063c73-f5cd-44e8-a84f-7810857f82df)
(drivemonitor:162)<br class="">
2017-12-05 06:20:04,698-0500 INFO (libvirt/events) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') drive 'vda'
threshold exceeded (storage:872)<br class="">
2017-12-05 06:20:05,889-0500 INFO (jsonrpc/7) [api.host]
START getAllVmStats() from=::1,41118 (api:46)<br class="">
2017-12-05 06:20:05,891-0500 INFO (jsonrpc/7) [api.host]
FINISH getAllVmStats return={'status': {'message': 'Done',
'code': 0}, 'statsList': (suppressed)} from=::1,41118 (api:52)<br class="">
2017-12-05 06:20:05,892-0500 INFO (jsonrpc/7)
[jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded
in 0.00 seconds (__init__:573)<br class="">
2017-12-05 06:20:07,466-0500 INFO (libvirt/events) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') underlying
process disconnected (vm:1024)<br class="">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM
resources (vm:4967)<br class="">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [virt.vm]
(vmId='b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping
connection (guestagent:435)<br class="">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [vdsm.api]
START
teardownImage(sdUUID='952bb427-b88c-4fbe-99ef-49970d3aaf70',
spUUID='9dcfeaaf-96b7-4e26-a327-5570e0e39261',
imgUUID='e6eadbae-ec7a-48f4-8832-64a622a12bef', volUUID=None)<br class="">
from=internal, task_id=9efedf46-d3be-4e41-b7f9-a074ed6344f6
(api:46)<br class="">
</b><br class="">
</b></p><div style="line-height: 1.38; margin-top: 0pt; margin-bottom: 0pt;" class=""><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class=""><span style="font-size: 11pt; font-family: Arial; background-color: transparent; font-weight: 400; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-variant-east-asian: normal; font-variant-position: normal; text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" class=""></error></span></b></div><b style="font-weight:normal;" id="docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" class="">
</b><br class="Apple-interchange-newline"><div class=""><br class="webkit-block-placeholder"></div><p class=""><br class="">
</p><p class=""><br class="">
</p>
</div>
_______________________________________________<br class="">Devel mailing list<br class=""><a href="mailto:Devel@ovirt.org" class="">Devel@ovirt.org</a><br class="">http://lists.ovirt.org/mailman/listinfo/devel</div></blockquote></div><br class=""></div></body></html>