<div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Wed, Feb 22, 2017 at 9:27 AM Nir Soffer <<a href="mailto:nsoffer@redhat.com">nsoffer@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Wed, Feb 22, 2017 at 9:03 AM, Nelson Lameiras<br class="gmail_msg">
<<a href="mailto:nelson.lameiras@lyra-network.com" class="gmail_msg" target="_blank">nelson.lameiras@lyra-network.com</a>> wrote:<br class="gmail_msg">
> Hello,<br class="gmail_msg">
><br class="gmail_msg">
> Not sure it is the same issue, but we have had a "major" issue recently in our production system when removing a ISCSI volume from oVirt, and then removing it from SAN.<br class="gmail_msg">
<br class="gmail_msg">
What version? OS version?<br class="gmail_msg">
<br class="gmail_msg">
The order must be:<br class="gmail_msg">
<br class="gmail_msg">
1. remove the LUN from storage domain<br class="gmail_msg">
will be available in next 4.1 release. in older versions you have<br class="gmail_msg">
to remove the storage domain<br class="gmail_msg">
<br class="gmail_msg">
2. unzone the LUN on the server<br class="gmail_msg">
<br class="gmail_msg">
3. remove the multipath devices and the paths on the nodes<br class="gmail_msg">
<br class="gmail_msg">
> The issue being that each host was still trying to access regularly to the SAN volume in spite of not being completely removed from oVirt.<br class="gmail_msg">
<br class="gmail_msg">
What do you mean by "not being completely removed"?<br class="gmail_msg">
<br class="gmail_msg">
Who was accessing the volume?<br class="gmail_msg">
<br class="gmail_msg">
> This led to an massive increase of error logs, which filled completely /var/log partition,<br class="gmail_msg">
<br class="gmail_msg">
Which log was full with errors?<br class="gmail_msg">
<br class="gmail_msg">
> which snowballed into crashing vdsm and other nasty consequences.<br class="gmail_msg">
<br class="gmail_msg">
You should have big enough /var/log to avoid such issues.<br class="gmail_msg"></blockquote><div><br></div><div>- Log rotation should be set better not to consume excessive amounts of space.</div><div>I'm seeing /etc/vdsm/logrotate/vdsm - not sure why it's not under /etc/logrotate.d . Looking at the file, seems like there's a 15M limit and 100 files, which translates to 1.5GB - and it is supposed to be compressed (not sure XZ is a good choice - it's very CPU intensive).</div><div><br></div><div>Others (Gluster?) do not seem to have a size limit, just weekly. Need to look at other components as well.</div><div>- At least on ovirt-node, we'd like to separate some directories to different partitions. So for example core dumps (which should be limited as well) on /var/core do not fill the same partition as /var/log is and thus render the host unusable. </div><div>And again, looking at file, we have a 'size 0' on /var/log/core/*.dump - and 'rotate 1' - not sure what it means - but it should not be in /var/log/core, but /var/core, I reckon.</div><div>Y.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br class="gmail_msg">
><br class="gmail_msg">
> Anyway, the solution was to manually logout from SAN (in each host) with iscsiadm and manually remove iscsi targets (again in each host). It was not difficult once the problem was found because currently we only have 3 hosts in this cluster, but I'm wondering what would happen if we had hundreds of hosts ?<br class="gmail_msg">
><br class="gmail_msg">
> Maybe I'm being naive but shouldn't this be "oVirt job" ? Is there a RFE still waiting to be included on this subject or should I write one ?<br class="gmail_msg">
<br class="gmail_msg">
We have RFE for this here:<br class="gmail_msg">
<a href="https://bugzilla.redhat.com/1310330" rel="noreferrer" class="gmail_msg" target="_blank">https://bugzilla.redhat.com/1310330</a><br class="gmail_msg">
<br class="gmail_msg">
But you must understand that ovirt does not control your storage server,<br class="gmail_msg">
you are responsible to add devices on the storage server, and remove<br class="gmail_msg">
them. We are only consuming the devices.<br class="gmail_msg">
<br class="gmail_msg">
Even we we provide a way to remove devices on all hosts, you will have<br class="gmail_msg">
to remove the device on the storage server before removing it from<br class="gmail_msg">
hosts. If not, ovirt will find the removed devices again in the next<br class="gmail_msg">
scsi rescan,<br class="gmail_msg">
and we do lot of these to support automatic discovery of new devices<br class="gmail_msg">
or resized devices.<br class="gmail_msg">
<br class="gmail_msg">
Nir<br class="gmail_msg">
<br class="gmail_msg">
><br class="gmail_msg">
> cordialement, regards,<br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> Nelson LAMEIRAS<br class="gmail_msg">
> Ingénieur Systèmes et Réseaux / Systems and Networks engineer<br class="gmail_msg">
> Tel: <a href="tel:+33%205%2032%2009%2009%2070" value="+33532090970" class="gmail_msg" target="_blank">+33 5 32 09 09 70</a><br class="gmail_msg">
> <a href="mailto:nelson.lameiras@lyra-network.com" class="gmail_msg" target="_blank">nelson.lameiras@lyra-network.com</a><br class="gmail_msg">
><br class="gmail_msg">
> <a href="http://www.lyra-network.com" rel="noreferrer" class="gmail_msg" target="_blank">www.lyra-network.com</a> | <a href="http://www.payzen.eu" rel="noreferrer" class="gmail_msg" target="_blank">www.payzen.eu</a><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
><br class="gmail_msg">
> Lyra Network, 109 rue de l'innovation, 31670 Labège, FRANCE<br class="gmail_msg">
><br class="gmail_msg">
> ----- Original Message -----<br class="gmail_msg">
> From: "Nir Soffer" <<a href="mailto:nsoffer@redhat.com" class="gmail_msg" target="_blank">nsoffer@redhat.com</a>><br class="gmail_msg">
> To: "Gianluca Cecchi" <<a href="mailto:gianluca.cecchi@gmail.com" class="gmail_msg" target="_blank">gianluca.cecchi@gmail.com</a>>, "Adam Litke" <<a href="mailto:alitke@redhat.com" class="gmail_msg" target="_blank">alitke@redhat.com</a>><br class="gmail_msg">
> Cc: "users" <<a href="mailto:users@ovirt.org" class="gmail_msg" target="_blank">users@ovirt.org</a>><br class="gmail_msg">
> Sent: Tuesday, February 21, 2017 6:32:18 PM<br class="gmail_msg">
> Subject: Re: [ovirt-users] best way to remove SAN lun<br class="gmail_msg">
><br class="gmail_msg">
> On Tue, Feb 21, 2017 at 7:25 PM, Gianluca Cecchi<br class="gmail_msg">
> <<a href="mailto:gianluca.cecchi@gmail.com" class="gmail_msg" target="_blank">gianluca.cecchi@gmail.com</a>> wrote:<br class="gmail_msg">
>> On Tue, Feb 21, 2017 at 6:10 PM, Nir Soffer <<a href="mailto:nsoffer@redhat.com" class="gmail_msg" target="_blank">nsoffer@redhat.com</a>> wrote:<br class="gmail_msg">
>>><br class="gmail_msg">
>>> This is caused by active lvs on the remove storage domains that were not<br class="gmail_msg">
>>> deactivated during the removal. This is a very old known issue.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> You have remove the remove device mapper entries - you can see the devices<br class="gmail_msg">
>>> using:<br class="gmail_msg">
>>><br class="gmail_msg">
>>> dmsetup status<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Then you can remove the mapping using:<br class="gmail_msg">
>>><br class="gmail_msg">
>>> dmsetup remove device-name<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Once you removed the stale lvs, you will be able to remove the multipath<br class="gmail_msg">
>>> device and the underlying paths, and lvm will not complain about read<br class="gmail_msg">
>>> errors.<br class="gmail_msg">
>>><br class="gmail_msg">
>>> Nir<br class="gmail_msg">
>><br class="gmail_msg">
>><br class="gmail_msg">
>> OK Nir, thanks for advising.<br class="gmail_msg">
>><br class="gmail_msg">
>> So what I run with success on the 2 hosts<br class="gmail_msg">
>><br class="gmail_msg">
>> [root@ovmsrv05 vdsm]# for dev in $(dmsetup status | grep<br class="gmail_msg">
>> 900b1853--e192--4661--a0f9--7c7c396f6f49 | cut -d ":" -f 1)<br class="gmail_msg">
>> do<br class="gmail_msg">
>> dmsetup remove $dev<br class="gmail_msg">
>> done<br class="gmail_msg">
>> [root@ovmsrv05 vdsm]#<br class="gmail_msg">
>><br class="gmail_msg">
>> and now I can run<br class="gmail_msg">
>><br class="gmail_msg">
>> [root@ovmsrv05 vdsm]# multipath -f 3600a0b80002999020000cd3c5501458f<br class="gmail_msg">
>> [root@ovmsrv05 vdsm]#<br class="gmail_msg">
>><br class="gmail_msg">
>> Also, with related names depending on host,<br class="gmail_msg">
>><br class="gmail_msg">
>> previous maps to single devices were for example in ovmsrv05:<br class="gmail_msg">
>><br class="gmail_msg">
>> 3600a0b80002999020000cd3c5501458f dm-4 IBM ,1814 FAStT<br class="gmail_msg">
>> size=2.0T features='2 pg_init_retries 50' hwhandler='1 rdac' wp=rw<br class="gmail_msg">
>> |-+- policy='service-time 0' prio=0 status=enabled<br class="gmail_msg">
>> | |- 0:0:0:2 sdb 8:16 failed undef running<br class="gmail_msg">
>> | `- 1:0:0:2 sdh 8:112 failed undef running<br class="gmail_msg">
>> `-+- policy='service-time 0' prio=0 status=enabled<br class="gmail_msg">
>> |- 0:0:1:2 sdg 8:96 failed undef running<br class="gmail_msg">
>> `- 1:0:1:2 sdn 8:208 failed undef running<br class="gmail_msg">
>><br class="gmail_msg">
>> And removal of single path devices:<br class="gmail_msg">
>><br class="gmail_msg">
>> [root@ovmsrv05 root]# for dev in sdb sdh sdg sdn<br class="gmail_msg">
>> do<br class="gmail_msg">
>> echo 1 > /sys/block/${dev}/device/delete<br class="gmail_msg">
>> done<br class="gmail_msg">
>> [root@ovmsrv05 vdsm]#<br class="gmail_msg">
>><br class="gmail_msg">
>> All clean now... ;-)<br class="gmail_msg">
><br class="gmail_msg">
> Great!<br class="gmail_msg">
><br class="gmail_msg">
> I think we should have a script doing all these steps.<br class="gmail_msg">
><br class="gmail_msg">
> Nir<br class="gmail_msg">
> _______________________________________________<br class="gmail_msg">
> Users mailing list<br class="gmail_msg">
> <a href="mailto:Users@ovirt.org" class="gmail_msg" target="_blank">Users@ovirt.org</a><br class="gmail_msg">
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br class="gmail_msg">
_______________________________________________<br class="gmail_msg">
Users mailing list<br class="gmail_msg">
<a href="mailto:Users@ovirt.org" class="gmail_msg" target="_blank">Users@ovirt.org</a><br class="gmail_msg">
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" class="gmail_msg" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br class="gmail_msg">
</blockquote></div></div>