Clones from snapshots on Cinder RBD problem

19 Sep 2016

      --Apple-Mail=_B6DC913E-B61D-41DC-8111-D2CABDDAB2E1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Hello all,=20

I already opened bug =
<https://bugzilla.redhat.com/show_bug.cgi?id=3D1377442> on this, but I =
thought I'd post to the list to see if anyone has seen this. I've got =
oVirt 4.0.3 using a Cinder external provider for Ceph (Cinder version is =
7.0.1 from RDO Liberty, Ceph is Jewel 10.2.2) to get access to RBD =
images. I am able to boot VMs from these RBD images, create snapshots of =
them, both live and when the VM is offline, but when I try to clone a =
new VM from those snapshots, the task never completes and I see errors =
like these <http://pastebin.com/ZLqRDSNW> in =
/var/log/ovirt-engine/engine.log:
=20
...
ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (default =
task-93) [50af631a] Command =
'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' failed: null
=20
ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (default =
task-93) [50af631a] Exception: java.lang.NullPointerException
...which is then followed repeatedly by:
...
ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] =
(DefaultQuartzScheduler8) [50af631a] Ending command =
'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' with failure.
=20
ERROR =
[org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskComman=
d] (DefaultQuartzScheduler8) [50af631a] Ending command =
'org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskComman=
d' with failure.
=20
INFO  [org.ovirt.engine.core.utils.transaction.TransactionSupport] =
(DefaultQuartzScheduler8) [] transaction rolled back
=20
ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler8) [] Correlation ID: 284e62be, Job ID: =
1899e73a-e5f8-4cda-b166-fbd5d444dc7c, Call Stack: null, Custom Event ID: =
-1, Message: Failed to complete VM testRBDclone1 creation.
=20
ERROR [org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler8) [] Failed invoking callback end method =
'onFailed' for command '81f6cbd0-6298-4033-99a2-6a740e10c61f' with =
exception 'null', the callback is marked for end method retries
The only way I've been able to back out of this state is to use =
engine-cleanup and then restore from a backup. No amount of stopping and =
restarting ovirt-engine.service seemed to get the task to fail =
completely so that it could be cleaned up.

Is anyone successfully using Cinder snapshots? Or is this a known issue =
for which I have not yet found a bug report? My review of the trace =
under that java.lang.NullPointerException did not lead me anywhere, but =
I might not know what I'm looking for there.=20

Thanks,
Andrew Richards
Senior Systems Engineer
keepertechnology

--Apple-Mail=_B6DC913E-B61D-41DC-8111-D2CABDDAB2E1
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=us-ascii

<html><head><meta http-equiv=3D"Content-Type" content=3D"text/html =
charset=3Dus-ascii"></head><body style=3D"word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D"">Hello all, <div class=3D""><br class=3D""></div><div =
class=3D"">I already opened <a =
href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1377442" =
class=3D"">bug</a> on this, but I thought I'd post to the list to =
see if anyone has seen this. I've got oVirt 4.0.3 using a Cinder =
external provider for Ceph (Cinder version is 7.0.1 from RDO =
Liberty, Ceph is Jewel 10.2.2) to get access to RBD images. I am able to =
boot VMs from these RBD images, create snapshots of them, both live and =
when the VM is offline, but when I try to clone a new VM from those =
snapshots, the task never completes and I see <a =
href=3D"http://pastebin.com/ZLqRDSNW" class=3D"">errors like =
these</a> in /var/log/ovirt-engine/engine.log:<div =
class=3D""> </div></div><blockquote type=3D"cite" class=3D"">ERROR =
[org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] (default task-93) =
[50af631a] Command 'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' =
failed: null<br class=3D""><br class=3D""><div class=3D""><div =
class=3D"">ERROR [org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] =
(default task-93) [50af631a] Exception: =
java.lang.NullPointerException</div></div></blockquote><div =
class=3D""></div><div class=3D""><div class=3D""><br class=3D""></div><div=
 class=3D"">...which is then followed repeatedly by:</div><div =
class=3D""><br class=3D""></div><div class=3D""></div><blockquote =
type=3D"cite" class=3D""><div class=3D"">ERROR =
[org.ovirt.engine.core.bll.AddVmFromSnapshotCommand] =
(DefaultQuartzScheduler8) [50af631a] Ending command =
'org.ovirt.engine.core.bll.AddVmFromSnapshotCommand' with =
failure.</div><div class=3D""><br class=3D""></div><div class=3D"">ERROR =
[org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskComman=
d] (DefaultQuartzScheduler8) [50af631a] Ending command =
'org.ovirt.engine.core.bll.storage.disk.cinder.CloneSingleCinderDiskComman=
d' with failure.</div><div class=3D""><br class=3D""></div><div =
class=3D"">INFO =
 [org.ovirt.engine.core.utils.transaction.TransactionSupport] =
(DefaultQuartzScheduler8) [] transaction rolled back</div><div =
class=3D""><br class=3D""></div><div class=3D"">ERROR =
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] =
(DefaultQuartzScheduler8) [] Correlation ID: 284e62be, Job ID: =
1899e73a-e5f8-4cda-b166-fbd5d444dc7c, Call Stack: null, Custom Event ID: =
-1, Message: Failed to complete VM testRBDclone1 creation.</div><div =
class=3D""><br class=3D""></div><div class=3D"">ERROR =
[org.ovirt.engine.core.bll.tasks.CommandCallbacksPoller] =
(DefaultQuartzScheduler8) [] Failed invoking callback end method =
'onFailed' for command '81f6cbd0-6298-4033-99a2-6a740e10c61f' with =
exception 'null', the callback is marked for end method =
retries</div></blockquote><div class=3D""><br =
class=3D"webkit-block-placeholder"></div><div class=3D"">The only way =
I've been able to back out of this state is to use engine-cleanup and =
then restore from a backup. No amount of stopping and restarting =
ovirt-engine.service seemed to get the task to fail completely so that =
it could be cleaned up.</div><div class=3D""><br class=3D""></div><div =
class=3D"">Is anyone successfully using Cinder snapshots? Or is this a =
known issue for which I have not yet found a bug report? My review of =
the trace under that java.lang.NullPointerException did not lead me =
anywhere, but I might not know what I'm looking for =
there. </div><div class=3D"">
<div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; orphans: =
auto; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; =
orphans: auto; text-align: start; text-indent: 0px; text-transform: =
none; white-space: normal; widows: auto; word-spacing: 0px; =
-webkit-text-stroke-width: 0px; word-wrap: break-word; =
-webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" =
class=3D""><div style=3D"color: rgb(0, 0, 0); font-variant: normal; =
letter-spacing: normal; line-height: normal; orphans: 2; text-align: =
-webkit-auto; text-indent: 0px; text-transform: none; white-space: =
normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: =
after-white-space;" class=3D""><span class=3D"Apple-style-span" =
style=3D"border-collapse: separate; font-variant-ligatures: normal; =
font-variant-position: normal; font-variant-numeric: normal; =
font-variant-alternates: normal; font-variant-east-asian: normal; =
line-height: normal; border-spacing: 0px;"><div style=3D"color: rgb(0, =
0, 0);" class=3D""><br =
class=3D"Apple-interchange-newline">Thanks,</div><div style=3D"color: =
rgb(0, 0, 0);" class=3D"">Andrew Richards</div><div style=3D"color: =
rgb(0, 0, 0);" class=3D""><font class=3D"Apple-style-span" =
style=3D"font-size: 11px;">Senior Systems Engineer</font></div><div =
style=3D"color: rgb(0, 0, 0);" class=3D""><span style=3D"text-align: =
center; font-family: Calibri; letter-spacing: -0.6px; font-size: large; =
color: rgb(16, 78, 127);" class=3D""><b class=3D"">keeper</b></span><span =
style=3D"color: rgb(134, 136, 132); text-align: center; font-family: =
Calibri; font-size: large; letter-spacing: -1.4px;" =
class=3D"">technology</span></div></span></div></div></div>
</div>
<br class=3D""></div></body></html>=

--Apple-Mail=_B6DC913E-B61D-41DC-8111-D2CABDDAB2E1--

Andrew Richards

tags

participants (1)