5:26 p.m.
--Apple-Mail=_9A4D98BA-C601-4816-8587-5700FE4F5348
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=utf-8
Milan,
log [1], VM b3962e5c-08e3-444e-910e-ea675fa1a5c7
migration away finished at 2017-12-05 06:26:24,515-0500
incoming migration of the same VM back, at 2017-12-05 06:26:46,614-0500
seems to me the migration away didn=E2=80=99t really properly clean up =
the VM. Milane, can you check that if logs matches? Perhaps orphaned =
libvirt=E2=80=99s xml?
Otherwise it would need reproduction in CI and some more logs=E2=80=A6.
[1] =
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifac=
t/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/po=
st-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.log=
=
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifa=
ct/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/p=
ost-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vdsm/vdsm.lo=
g>
On 5 Dec 2017, at 13:26, Dafna Ron <dron(a)redhat.com> wrote:
=20
Hi,=20
We had a failure for test 006_migrations.migrate_vm on master.=20
=20
There was a libvirt disruption in the migration src and after that =
vdsm reported
the migration as failed because the vm already exists =
which makes me suspect a split bran case.=20
The patch reported has nothing to do with this issue and I think we =
simply stumbled on a race condition which can cause a split brain.=20
Please note that there are several metrics related issues reported in
=
vdsm logs as well.=20
=20
Link and headline of suspected patches:=20
=20
Not related=20
=20
Link to Job:
=20
=20
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/ =
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/>
=20
Link to all logs:
=20
=20
=
http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifac=
t/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/po=
st-006_migrations.py/ =
<http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/artifa=
ct/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/p=
ost-006_migrations.py/>
=20
(Relevant) error snippet from the log:=20
<error>
=20
Engine:=20
2017-12-05 06:26:48,546-05 ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(EE-ManagedThreadFactory-engine-Thread-385) [] EVENT_ID: =
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: vm0, Source: =
lago-
basic-suite-master-host-1, Destination: =
lago-basic-suite-master-host-0).
=20
=20
dst:
=20
2017-12-05 06:26:46,615-0500 WARN (jsonrpc/6) [vds] vm =
b3962e5c-08e3-444e-910e-ea675fa1a5c7 already exists (API:179)
2017-12-05 06:26:46,615-0500 ERROR (jsonrpc/6) [api] FINISH create =
error=3DVirtual machine already exists (api:124)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line =
117,
in method
ret =3D func(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 180, in =
create
raise exception.VMExists()
VMExists: Virtual machine already exists
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt] FINISH =
create
return=3D{'status': {'message': 'Virtual machine already exists',
=
'code': 4}} from=3D::ffff:192.168.201.3,50394 (api:52)
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt] FINISH =
migrationCreate return=3D{'status': {'message': 'Virtual machine
already =
exists', 'code': 4}} from=3D::ffff:192.168.201.3,50394 (api:52)
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6)
[jsonrpc.JsonRpcServer] =
RPC call VM.migrationCreate failed (error 4) in 0.03
seconds =
(__init__:573)
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [api.virt] START =
destroy(gracefulAttempts=3D1) from=3D::ffff:192.168.201.3,50394 (api:46)
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM resources =
(vm:4967)
2017-12-05 06:26:46,625-0500 WARN (jsonrpc/2) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') trying to set state to =
Powering down when already Down (vm:575)
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping connection =
(guestagent:435)
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') _destroyVmGraceful =
attempt #0 (vm:5004)
2017-12-05 06:26:46,626-0500 WARN (jsonrpc/2) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') VM =
'b3962e5c-08e3-444e-910e-ea675fa1a5c7' couldn't be destroyed in libvirt: =
Requested operation is not valid: domain is not running (vm
:5025)
2017-12-05 06:26:46,627-0500 INFO (jsonrpc/2) [vdsm.api] START =
teardownImage(sdUUID=3D'952bb427-b88c-4fbe-99ef-49970d3aaf70', =
spUUID=3D'9dcfeaaf-96b7-4e26-a327-5570e0e39261', =
imgUUID=3D'e6eadbae-ec7a-48f4-8832-64a622a12bef', volUUID=3DNone) from
=3D::ffff:192.168.201.3,50394, =
task_id=3D2da93725-5533-4354-a369-751eb44f9ef2 (api:46)
=20
scr=20
=20
2017-12-05 06:26:46,623-0500 ERROR (migsrc/b3962e5c) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') migration destination =
error: Virtual machine already exists (migration:290)
=20
disruption on dst:=20
=20
2017-12-05 06:20:04,662-0500 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] =
RPC call
VM.shutdown succeeded in 0.00 seconds (__init__:573)
2017-12-05 06:20:04,676-0500 ERROR (Thread-1) [root] Shutdown by QEMU
=
Guest Agent failed (vm:5097)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/virt/vm.py", line 5088, =
in qemuGuestAgentShutdown
self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)
File "/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py", line =
98, in f
ret =3D attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", =
line 126, in wrapper
ret =3D f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 512, in =
wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 2403, in =
shutdownFlags
if ret =3D=3D -1: raise libvirtError
('virDomainShutdownFlags() =
failed', dom=3Dself)
libvirtError: Guest agent is not responding: QEMU guest agent is not
=
connected
2017-12-05 06:20:04,697-0500 INFO (libvirt/events) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') block threshold 1 =
exceeded on 'vda' =
(/rhev/data-center/mnt/blockSD/952bb427-b88c-4fbe-99ef-49970d3aaf70/images=
/e6eadbae-ec7a-48f4-
8832-64a622a12bef/1f063c73-f5cd-44e8-a84f-7810857f82df) =
(drivemonitor:162)
2017-12-05 06:20:04,698-0500 INFO (libvirt/events) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') drive 'vda'
threshold =
exceeded (storage:872)
2017-12-05 06:20:05,889-0500 INFO (jsonrpc/7) [api.host] START =
getAllVmStats() from=3D::1,41118 (api:46)
2017-12-05 06:20:05,891-0500 INFO (jsonrpc/7) [api.host] FINISH =
getAllVmStats return=3D{'status': {'message': 'Done',
'code': 0}, =
'statsList': (suppressed)} from=3D::1,41118 (api:52)
2017-12-05 06:20:05,892-0500 INFO (jsonrpc/7)
[jsonrpc.JsonRpcServer] =
RPC call Host.getAllVmStats succeeded in 0.00 seconds
(__init__:573)
2017-12-05 06:20:07,466-0500 INFO (libvirt/events) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') underlying process =
disconnected (vm:1024)
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM resources =
(vm:4967)
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [virt.vm] =
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping connection =
(guestagent:435)
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) [vdsm.api] START
=
teardownImage(sdUUID=3D'952bb427-b88c-4fbe-99ef-49970d3aaf70', =
spUUID=3D'9dcfeaaf-96b7-4e26-a327-5570e0e39261', =
imgUUID=3D'e6eadbae-ec7a-48f4-8832-64a622a12bef', volUUID=3DNone)
from=3Dinternal, task_id=3D9efedf46-d3be-4e41-b7f9-a074ed6344f6 =
(api:46)
=20
=20
</error>
=20
=20
=20
=20
_______________________________________________
Devel mailing list
Devel(a)ovirt.org
http://lists.ovirt.org/mailman/listinfo/devel
--Apple-Mail=_9A4D98BA-C601-4816-8587-5700FE4F5348
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=utf-8
<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html =
charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div class=3D"">Milan,</div><div
class=3D""><br =
class=3D""></div>log [1], VM
b3962e5c-08e3-444e-910e-ea675fa1a5c7<div =
class=3D"">migration away finished at 2017-12-05 =
06:26:24,515-0500</div><div class=3D"">incoming migration of the
same VM =
back, at 2017-12-05 06:26:46,614-0500</div><div class=3D""><br =
class=3D""></div><div class=3D"">seems to me the
migration away didn=E2=80=
=99t really properly clean up the VM. Milane, can you check that if logs =
matches? Perhaps orphaned libvirt=E2=80=99s xml?</div><div =
class=3D"">Otherwise it would need reproduction in CI and some more =
logs=E2=80=A6.</div><div class=3D""><br
class=3D""></div><div =
class=3D"">[1] <a =
href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste...
/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-m=
aster/post-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vdsm/=
vdsm.log" =
class=3D"">http://jenkins.ovirt.org/job/ovirt-master_change-...
278/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suit=
e-master/post-006_migrations.py/lago-basic-suite-master-host-0/_var_log/vd=
sm/vdsm.log</a></div><div class=3D""><br
class=3D""><div><blockquote =
type=3D"cite" class=3D""><div class=3D"">On 5 Dec
2017, at 13:26, Dafna =
Ron <<a href=3D"mailto:dron@redhat.com" =
class=3D"">dron(a)redhat.com</a>&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div class=3D"">
=20
<meta http-equiv=3D"content-type" content=3D"text/html; =
charset=3Dutf-8" class=3D"">
=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"
class=3D""><p class=3D"">Hi, =
<br class=3D"">
</p><p class=3D"">We had a failure for test =
006_migrations.migrate_vm on master. </p><p
class=3D"">There was a =
libvirt disruption in the migration src and after
that vdsm reported the migration as failed because the vm already
exists which makes me suspect a split bran case. <br class=3D"">
</p><p class=3D"">The patch reported has nothing to do with this
=
issue and I think
we simply stumbled on a race condition which can cause a split
brain. <br class=3D"">
</p><p class=3D"">Please note that there are several metrics
related =
issues
reported in vdsm logs as well. <br class=3D"">
</p><p class=3D""><b style=3D"font-weight:normal;"
=
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550"
class=3D"">=
</b></p><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><span style=3D"font-size: 11pt; font-family: Arial; =
background-color: transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link and headline of suspected patches:=20
</span></b></div><b style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><p dir=3D"ltr" =
style=3D"line-height:1.38;margin-top:0pt;margin-bottom:0pt;" =
class=3D""><span style=3D"font-size: 11pt; font-family: Arial; =
background-color: transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">
</span></p><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><span style=3D"font-size: 11pt;
=
font-family: Arial; background-color: transparent; font-weight: 400; =
font-style: normal; font-variant-ligatures: normal; font-variant-caps: =
normal; font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D""><b class=3D"">Not related </b>
</span></div>
<br class=3D""><div style=3D"line-height: 1.38;
margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><span style=3D"font-size: 11pt;
=
font-family: Arial; background-color: transparent; font-weight: 400; =
font-style: normal; font-variant-ligatures: normal; font-variant-caps: =
normal; font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link to Job:</span></div>
<br class=3D"">
</b><div class=3D""><br
class=3D"webkit-block-placeholder"></div><p =
class=3D""><b style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><b class=3D""><a
class=3D"moz-txt-link-freetext" =
href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste...
/">http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/4278/</a>=
</b></b></p><p class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><br class=3D"">
</b></p><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><span style=3D"font-size: 11pt; font-family: Arial; =
background-color: transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">Link to all logs:</span></b></div><b =
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550"
class=3D"">=
<br class=3D"">
</b><div class=3D""><br
class=3D"webkit-block-placeholder"></div><p =
class=3D""><b style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><b class=3D""><a
class=3D"moz-txt-link-freetext" =
href=3D"http://jenkins.ovirt.org/job/ovirt-master_change-queue-teste...
/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-m=
aster/post-006_migrations.py/">http://jenkins.ovirt.org/job/ovirt...
hange-queue-tester/4278/artifact/exported-artifacts/basic-suit-master-el7/=
test_logs/basic-suite-master/post-006_migrations.py/</a></b></b></p><p
=
class=3D""><b style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><br class=3D"">
</b></p><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><span style=3D"font-size: 11pt; font-family: Arial; =
background-color: transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">(Relevant) error snippet from the log:
</span></b></div><b =
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><b class=3D""><span
style=3D"font-size: =
11pt; font-family: Arial; background-color: transparent; font-weight: =
400; font-style: normal; font-variant-ligatures: normal; =
font-variant-caps: normal; font-variant-east-asian: normal; =
font-variant-position: normal; text-decoration: none; vertical-align: =
baseline; white-space: pre-wrap;" =
class=3D""><error></span></b></div><div
style=3D"line-height: =
1.38; margin-top: 0pt; margin-bottom: 0pt;" class=3D""><b
class=3D""><span=
style=3D"font-size: 11pt; font-family: Arial; background-color: =
transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D"">
</span></b></div><div style=3D"line-height: 1.38; margin-top:
0pt; =
margin-bottom: 0pt;" class=3D""><b class=3D""><span
style=3D"font-size: =
11pt; font-family: Arial; background-color: transparent; font-weight: =
400; font-style: normal; font-variant-ligatures: normal; =
font-variant-caps: normal; font-variant-east-asian: normal; =
font-variant-position: normal; text-decoration: none; vertical-align: =
baseline; white-space: pre-wrap;" class=3D""><b
class=3D"">Engine: </b>
</span></b></div><div style=3D"line-height: 1.38; margin-top:
0pt; =
margin-bottom: 0pt;" class=3D""><b class=3D""><span
style=3D"font-size: =
11pt; font-family: Arial; background-color: transparent; font-weight: =
400; font-style: normal; font-variant-ligatures: normal; =
font-variant-caps: normal; font-variant-east-asian: normal; =
font-variant-position: normal; text-decoration: none; vertical-align: =
baseline; white-space: pre-wrap;" class=3D"">2017-12-05 06:26:48,546-05
=
ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(EE-ManagedThreadFactory-engine-Thread-385) [] EVENT_ID: =
VM_MIGRATION_TO_SERVER_FAILED(120), Migration failed (VM: vm0, Source: =
lago-
basic-suite-master-host-1, Destination: lago-basic-suite-master-host-0).
</span></b></div>
<br class=3D"">
<b class=3D"">dst:</b><b class=3D""><br
class=3D"">
</b></b><div class=3D""><br =
class=3D"webkit-block-placeholder"></div><p
class=3D""><b class=3D""><span=
style=3D"font-weight:normal;" class=3D"">2017-12-05
06:26:46,615-0500
WARN (jsonrpc/6) [vds] vm
b3962e5c-08e3-444e-910e-ea675fa1a5c7 already exists =
(API:179)<br class=3D"">
2017-12-05 06:26:46,615-0500 ERROR (jsonrpc/6) [api] FINISH
create error=3DVirtual machine already exists (api:124)<br =
class=3D"">
Traceback (most recent call last):<br class=3D"">
File =
"/usr/lib/python2.7/site-packages/vdsm/common/api.py",
line 117, in method<br class=3D"">
ret =3D func(*args, **kwargs)<br
class=3D"">
File "/usr/lib/python2.7/site-packages/vdsm/API.py", =
line
180, in create<br class=3D"">
raise exception.VMExists()<br
class=3D"">
VMExists: Virtual machine already exists<br class=3D"">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt]
FINISH create return=3D{'status': {'message': 'Virtual
machine
already exists', 'code': 4}} from=3D::ffff:192.168.201.3,50394
(api:52)<br class=3D"">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6) [api.virt]
FINISH migrationCreate return=3D{'status': {'message': =
'Virtual
machine already exists', 'code': 4}}
from=3D::ffff:192.168.201.3,50394 (api:52)<br class=3D"">
2017-12-05 06:26:46,620-0500 INFO (jsonrpc/6)
[jsonrpc.JsonRpcServer] RPC call VM.migrationCreate failed
(error 4) in 0.03 seconds (__init__:573)<br class=3D"">
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [api.virt]
START destroy(gracefulAttempts=3D1)
from=3D::ffff:192.168.201.3,50394 (api:46)<br class=3D"">
2017-12-05 06:26:46,624-0500 INFO (jsonrpc/2) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM
resources (vm:4967)<br class=3D"">
2017-12-05 06:26:46,625-0500 WARN (jsonrpc/2) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') trying to set
state to Powering down when already Down (vm:575)<br
class=3D"">=
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping
connection (guestagent:435)<br class=3D"">
2017-12-05 06:26:46,625-0500 INFO (jsonrpc/2) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7')
_destroyVmGraceful attempt #0 (vm:5004)<br class=3D"">
2017-12-05 06:26:46,626-0500 WARN (jsonrpc/2) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') VM
'b3962e5c-08e3-444e-910e-ea675fa1a5c7' couldn't be destroyed
in libvirt: Requested operation is not valid: domain is not
running (vm<br class=3D"">
:5025)<br class=3D"">
2017-12-05 06:26:46,627-0500 INFO (jsonrpc/2) [vdsm.api]
START
teardownImage(sdUUID=3D'952bb427-b88c-4fbe-99ef-49970d3aaf70',
spUUID=3D'9dcfeaaf-96b7-4e26-a327-5570e0e39261',
imgUUID=3D'e6eadbae-ec7a-48f4-8832-64a622a12bef', =
volUUID=3DNone)
from<br class=3D"">
=3D::ffff:192.168.201.3,50394,
task_id=3D2da93725-5533-4354-a369-751eb44f9ef2 (api:46)<br =
class=3D"">
</span></b></p><p class=3D""><b
class=3D""><span =
style=3D"font-weight:normal;" class=3D""><b
class=3D"">scr </b><br =
class=3D"">
</span></b></p><p class=3D""><b
class=3D""><span =
style=3D"font-weight:normal;" class=3D"">2017-12-05
06:26:46,623-0500
ERROR (migsrc/b3962e5c) [virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') migration
destination error: Virtual machine already exists
(migration:290)<br class=3D"">
</span></b></p><p class=3D""><b
class=3D""><span =
style=3D"font-weight:normal;" class=3D""><b
class=3D"">disruption on =
</b><b class=3D"">dst</b><b class=3D"">:
</b><br class=3D"">
</span></b></p><p class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><b class=3D"">2017-12-05
06:20:04,662-0500 INFO (jsonrpc/3) =
[jsonrpc.JsonRpcServer]
RPC call VM.shutdown succeeded in 0.00 seconds =
(__init__:573)<br class=3D"">
2017-12-05 06:20:04,676-0500 ERROR (Thread-1) [root] Shutdown
by QEMU Guest Agent failed (vm:5097)<br class=3D"">
Traceback (most recent call last):<br class=3D"">
File =
"/usr/lib/python2.7/site-packages/vdsm/virt/vm.py",
line 5088, in qemuGuestAgentShutdown<br class=3D"">
=
self._dom.shutdownFlags(libvirt.VIR_DOMAIN_SHUTDOWN_GUEST_AGENT)<br =
class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/virt/virdomain.py",
line 98, in f<br class=3D"">
ret =3D attr(*args, **kwargs)<br
class=3D"">
File
"/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 126, in wrapper<br class=3D"">
ret =3D f(*args, **kwargs)<br
class=3D"">
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", =
line
512, in wrapper<br class=3D"">
return func(inst, *args, **kwargs)<br =
class=3D"">
File "/usr/lib64/python2.7/site-packages/libvirt.py", =
line
2403, in shutdownFlags<br class=3D"">
if ret =3D=3D -1: raise libvirtError
('virDomainShutdownFlags() failed', dom=3Dself)<br
class=3D"">
libvirtError: Guest agent is not responding: QEMU guest agent
is not connected<br class=3D"">
2017-12-05 06:20:04,697-0500 INFO (libvirt/events) =
[virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') block =
threshold
1 exceeded on 'vda'
=
(/rhev/data-center/mnt/blockSD/952bb427-b88c-4fbe-99ef-49970d3aaf70/images=
/e6eadbae-ec7a-48f4-<br class=3D"">
8832-64a622a12bef/1f063c73-f5cd-44e8-a84f-7810857f82df)
(drivemonitor:162)<br class=3D"">
2017-12-05 06:20:04,698-0500 INFO (libvirt/events) =
[virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') drive 'vda'
threshold exceeded (storage:872)<br class=3D"">
2017-12-05 06:20:05,889-0500 INFO (jsonrpc/7) [api.host]
START getAllVmStats() from=3D::1,41118 (api:46)<br class=3D"">
2017-12-05 06:20:05,891-0500 INFO (jsonrpc/7) [api.host]
FINISH getAllVmStats return=3D{'status': {'message':
'Done',
'code': 0}, 'statsList': (suppressed)} from=3D::1,41118 =
(api:52)<br class=3D"">
2017-12-05 06:20:05,892-0500 INFO (jsonrpc/7)
[jsonrpc.JsonRpcServer] RPC call Host.getAllVmStats succeeded
in 0.00 seconds (__init__:573)<br class=3D"">
2017-12-05 06:20:07,466-0500 INFO (libvirt/events) =
[virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') underlying
process disconnected (vm:1024)<br class=3D"">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) =
[virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Release VM
resources (vm:4967)<br class=3D"">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) =
[virt.vm]
(vmId=3D'b3962e5c-08e3-444e-910e-ea675fa1a5c7') Stopping
connection (guestagent:435)<br class=3D"">
2017-12-05 06:20:07,469-0500 INFO (libvirt/events) =
[vdsm.api]
START
teardownImage(sdUUID=3D'952bb427-b88c-4fbe-99ef-49970d3aaf70',
spUUID=3D'9dcfeaaf-96b7-4e26-a327-5570e0e39261',
imgUUID=3D'e6eadbae-ec7a-48f4-8832-64a622a12bef', =
volUUID=3DNone)<br class=3D"">
from=3Dinternal, =
task_id=3D9efedf46-d3be-4e41-b7f9-a074ed6344f6
(api:46)<br class=3D"">
</b><br class=3D"">
</b></p><div style=3D"line-height: 1.38; margin-top: 0pt; =
margin-bottom: 0pt;" class=3D""><b
style=3D"font-weight:normal;" =
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D""><span style=3D"font-size: 11pt; font-family: Arial; =
background-color: transparent; font-weight: 400; font-style: normal; =
font-variant-ligatures: normal; font-variant-caps: normal; =
font-variant-east-asian: normal; font-variant-position: normal; =
text-decoration: none; vertical-align: baseline; white-space: pre-wrap;" =
class=3D""></error></span></b></div><b
style=3D"font-weight:normal;"=
id=3D"docs-internal-guid-5859b7a1-269d-3fbb-6886-821a181fd550" =
class=3D"">
</b><br class=3D"Apple-interchange-newline"><div
class=3D""><br =
class=3D"webkit-block-placeholder"></div><p
class=3D""><br class=3D"">
</p><p class=3D""><br class=3D"">
</p>
</div>
_______________________________________________<br class=3D"">Devel =
mailing list<br class=3D""><a href=3D"mailto:Devel@ovirt.org"
=
class=3D"">Devel(a)ovirt.org</a><br =
class=3D"">http://lists.ovirt.org/mailman/listinfo/devel<...
</div><br
class=3D""></div></body></html>=
--Apple-Mail=_9A4D98BA-C601-4816-8587-5700FE4F5348--