[ovirt-users] [HELP] unmapping a deleted storage domain triggers crash ovirt

Thu Nov 17 15:48:08 UTC 2016

Hello again,

Just to provide an update for future reference, I've been unable to sort this problem out.
However, Ovirt4 does not show this failure.

My current approach to move VMs from one installation to another is:
1)            Shutdown VM in ovirt3.
2)            Maintenance VM storage
3)            Clone VM storage at LUN level
4)            Import cloned storage in ovirt4
5)            Import VM in ovirt4
If everything is ok:
6a)          detach + remove original VM storage in ovirt3
7a)          remove mappings for original VM storage LUN in ovirt3; this imply cluster instability (sanlock crash on hosts);
8a)          recover cluster; this implies shutting down host (and its VM) one by one
9a)          delete unmapped original LUN from storage manager

When Sanlock freeze on host, the only solution found so far is to reboot it. This has to be done on all host experiencing this problem (which basically means: all hosts in the cluster)

Cheers
AG

From: Andrea Ghelardi
Sent: Tuesday, November 08, 2016 16:40
To: users at ovirt.org
Subject: [HELP] unmapping a deleted storage domain triggers crash ovirt

Hello people,

something's not right in my ovirt infrastructure.
I currently have two different ovirt installation:
Ovirt3: 7 hosts linked to Compellent iscsi storage running ovirt 3.5.6
Ovirt4: 4 hosts linked to (same) Compellent iscsi storage running ovirt 4.0.4

I'm currently moving my guests from ovirt3 to ovirt4.
Since iscsi storage is linked to both installations, my high level approach is:

1)      Shutdown VM in ovirt3.

2)      Maintenance + detach + remove VM storage in ovirt3

3)      Change LUN mapping via iscsi storage manager from ovirt3 to ovirt4

4)      Import storage in ovirt4

5)      Import VM in ovirt4

6)      Run and cheers with high grade liquor. GOTO step 1 for different VM.

Now, as soon as I perform step 3 (remove mappings from LUN), ovirt3 goes crazy and eventually forces me to reboot all hosts one by one.
I tried different low level approaches to unmap LUN mpath'ed from hosts with inconsistent results.
A notable error log extract is:
Nov  8 15:30:52 sovana vdsm root ERROR Process failed with rc=1 out='\nudevadm settle - timeout of 5 seconds reached, the event queue contains:\n  /sys/devices/virtual/block/dm-39 (8603)\n  /sys/devices/virtual/block/dm-39 (8604)\n  /sys
/devices/virtual/block/dm-39 (8605)\n  /sys/devices/virtual/block/dm-39 (8606)\n  /sys/devices/virtual/block/dm-39 (8607)\n  /sys/devices/virtual/block/dm-39 (8608)\n  /sys/devices/virtual/block/dm-39 (8609)\n  /sys/devices/virtual/block
/dm-39 (8610)\n  /sys/devices/virtual/block/dm-39 (8611)\n  /sys/devices/virtual/block/dm-39 (8612)\n  /sys/devices/virtual/block/dm-39 (8613)\n  /sys/devices/virtual/block/dm-39 (8614)\n  /sys/devices/virtual/block/dm-39 (8615)\n  /sys/
devices/virtual/block/dm-39 (8616)\n  /sys/devices/virtual/block/dm-39 (8617)\n  /sys/devices/virtual/block/dm-39 (8618)\n  /sys/devices/virtual/block/dm-39 (8619)\n  /sys/devices/virtual/block/dm-39 (8620)\n  /sys/devices/virtual/block/
dm-39 (8621)\n  /sys/devices/virtual/block/dm-39 (8622)\n  /sys/devices/virtual/block/dm-39 (8623)\n  /sys/devices/virtual/block/dm-39 (8624)\n  /sys/devices/virtual/block/dm-39 (8625)\n  /sys/devices/virtual/block/dm-39 (8626)\n  /sys/d
evices/virtual/block/dm-39 (8627)\n  /sys/devices/virtual/block/dm-39 (8628)\n  /sys/devices/virtual/block/dm-39 (8629)\n  /sys/devices/virtual/block/dm-39 (8630)\n  /sys/devices/virtual/block/dm-39 (8631)\n  /sys/devices/virtual/block/d
m-39 (8632)\n  /sys/devices/virtual/block/dm-39 (8633)\n  /sys/devices/virtual/block/dm-39 (8634)\n  /sys/devices/virtual/block/dm-39 (8635)\n  /sys/devices/virtual/block/dm-39 (8636)\n  /sys/devices/virtual/block/dm-39 (8637)\n  /sys/de
vices/virtual/block/dm-39 (8638)\n  /sys/devices/virtual/block/dm-39 (8639)\n  /sys/devices/virtual/block/dm-39 (8640)\n  /sys/devices/virtual/block/dm-39 (8641)\n  /sys/devices/virtual/block/dm-39 (8642)\n  /sys/devices/virtual/block/dm
-39 (8643)\n  /sys/devices/virtual/block/dm-39 (8644)\n  /sys/devices/virtual/block/dm-39 (8645)\n  /sys/devices/virtual/block/dm-39 (8646)\n  /sys/devices/virtual/block/d

I really need your help to sort this out as I'm actually blocked in my task.

Why mapping changes triggers ovirt crash on a storage which ovirt should not care about?
Thanks

Andrea Ghelardi

+39 050 2203 71 | www.iongroup.com<http://www.iongroup.com/> | a.ghelardi at iontrading.com<mailto:a.ghelardi at iontrading.com>
Via San Martino, 52 - 56125 Pisa - ITALY

This email and any attachments may contain information which is confidential and/or privileged. The information is intended exclusively for the addressee and the views expressed may not be official policy, but the personal views of the originator. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents is prohibited. If you have received this email and any file transmitted with it in error, please notify the sender by telephone or return email immediately and delete the material from your computer. Internet communications are not secure and ION Trading is not responsible for their abuse by third parties, nor for any alteration or corruption in transmission, nor for any damage or loss caused by any virus or other defect. ION Trading accepts no liability or responsibility arising out of or in any way connected to this email.

[iON_HBlu_small]
Automation through innovation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161117/1061137d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 5300 bytes
Desc: image001.png
URL: <http://lists.ovirt.org/pipermail/users/attachments/20161117/1061137d/attachment-0001.png>