<div dir="ltr"><div>Nir,<br></div>See attached the repoplot output.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 14, 2016 at 12:18 PM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Apr 14, 2016 at 12:02 PM, Fred Rolland <<a href="mailto:frolland@redhat.com">frolland@redhat.com</a>> wrote:<br>
> From the log, we can see that the lvextend command took 18 sec, which is<br>
> quite long.<br>
<br>
</span>Fred, can you run repoplot on this log file? it will may explain why this lvm<br>
call took 18 seconds.<br>
<span class="HOEnZb"><font color="#888888"><br>
Nir<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
><br>
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13<br>
> 10:52:06,759::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset<br>
> --cpu-list 0-23 /usr/bin/sudo -n /usr/sbin/lvm lvextend --config ' devices {<br>
> preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1<br>
> write_cache_state=0 disable_after_error_count=3 filter = [<br>
> '\''a|/dev/mapper/36000eb3a4f1acbc20000000000000043|'\'', '\''r|.*|'\'' ] }<br>
> global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1<br>
> use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --autobackup<br>
> n --size 6016m<br>
> 5de4a000-a9c4-489c-8eee-10368647c413/721d09bc-60e7-4310-9ba2-522d2a4b03d0<br>
> (cwd None)<br>
> ....<br>
> 60decf0c-6d9a-4c3b-bee6-de9d2ff05e85::DEBUG::2016-04-13<br>
> 10:52:22,217::lvm::290::Storage.Misc.excCmd::(cmd) SUCCESS: <err> = '<br>
> WARNING: lvmetad is running but disabled. Restart lvmetad before enabling<br>
> it!\n WARNING: This metadata update is NOT backed up\n'; <rc> = 0<br>
><br>
><br>
> The watermark can be configured by the following value:<br>
><br>
> 'volume_utilization_percent', '50',<br>
> 'Together with volume_utilization_chunk_mb, set the minimal free '<br>
> 'space before a thin provisioned block volume is extended. Use '<br>
> 'lower values to extend earlier.')<br>
><br>
> On Thu, Apr 14, 2016 at 11:42 AM, Michal Skrivanek<br>
> <<a href="mailto:michal.skrivanek@redhat.com">michal.skrivanek@redhat.com</a>> wrote:<br>
>><br>
>><br>
>> > On 14 Apr 2016, at 09:57, <a href="mailto:nicolas@devels.es">nicolas@devels.es</a> wrote:<br>
>> ><br>
>> > Ok, that makes sense, thanks for the insight both Alex and Fred. I'm<br>
>> > attaching the VDSM log of the SPM node at the time of the pause. I couldn't<br>
>> > find anything that would clearly identify the problem, but maybe you'll be<br>
>> > able to.<br>
>><br>
>> In extreme conditions it will happen. When your storage is slow to respond<br>
>> to extension request, and when your write rate is very high then it may<br>
>> happen, as it is happening to you, that you run out space sooner than the<br>
>> extension finishes. You can change the watermark value I guess(right,<br>
>> Fred?), but better would be to plan a bit more ahead and either use<br>
>> preallocated or create thin and then allocate expected size in advance<br>
>> before the operation causing it (typically it only happens during untarring<br>
>> gigabytes of data, or huge database dump/restore)<br>
>> Even then, the VM should always be automatially resumed once the disk<br>
>> space is allocated<br>
>><br>
>> Thanks,<br>
>> michal<br>
>><br>
>> ><br>
>> > Thanks.<br>
>> ><br>
>> > Regards.<br>
>> ><br>
>> > El 2016-04-13 13:09, Fred Rolland escribió:<br>
>> >> Hi,<br>
>> >> Yes, just as Alex explained, if the disk has been created as thin<br>
>> >> provisioning, the vdsm will extends once a watermark is reached.<br>
>> >> Usually it should not get to the state the Vm is paused.<br>
>> >> From the log, you can see that the request for extension has been sent<br>
>> >> before the VM got to the No Space Error.<br>
>> >> Later, we can see the VM resuming.<br>
>> >> INFO::2016-04-13<br>
>> >> 10:52:04,182::vm::1026::virt.vm::(extendDrivesIfNeeded)<br>
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::Requesting extension for<br>
>> >> volume<br>
>> >> ....<br>
>> >> INFO::2016-04-13 10:52:29,360::vm::3728::virt.vm::(onIOError)<br>
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::abnormal vm stop device<br>
>> >> virtio-disk0 error enospc<br>
>> >> ....<br>
>> >> INFO::2016-04-13 10:52:54,317::vm::5084::virt.vm::(_logGuestCpuStatus)<br>
>> >> vmId=`f9cd282e-110a-4896-98d3-6d320662744d`::CPU running: onResume<br>
>> >> Note that the extension is done on the SPM host, so it would be<br>
>> >> interesting to see the vdsm log from the host that was in SPM role at<br>
>> >> this timeframe.<br>
>> >> Regards,<br>
>> >> Fred<br>
>> >> On Wed, Apr 13, 2016 at 2:43 PM, Alex Crow <<a href="mailto:acrow@integrafin.co.uk">acrow@integrafin.co.uk</a>><br>
>> >> wrote:<br>
>> >>> Hi,<br>
>> >>> If you have set up VM disks as Thin Provisioned, the VM has to<br>
>> >>> pause when the disk image needs to expand. You won't see this on VMs<br>
>> >>> with preallocated storage.<br>
>> >>> It's not the SAN that's running out of space, it's the VM image<br>
>> >>> needing to be expanded incrementally each time.<br>
>> >>> Cheers<br>
>> >>> Alex<br>
>> >>> On 13/04/16 12:04, <a href="mailto:nicolas@devels.es">nicolas@devels.es</a> wrote:<br>
>> >>> Hi Fred,<br>
>> >>> This is an iSCSI storage. I'm attaching the VDSM logs from the host<br>
>> >>> where this machine has been running. Should you need any further<br>
>> >>> info, don't hesitate to ask.<br>
>> >>> Thanks.<br>
>> >>> Regards.<br>
>> >>> El 2016-04-13 11:54, Fred Rolland escribió:<br>
>> >>> Hi,<br>
>> >>> What kind of storage do you have ? (ISCSI,FC,NFS...)<br>
>> >>> Can you provide the vdsm logs from the host where this VM runs ?<br>
>> >>> Thanks,<br>
>> >>> Freddy<br>
>> >>> On Wed, Apr 13, 2016 at 1:02 PM, <<a href="mailto:nicolas@devels.es">nicolas@devels.es</a>> wrote:<br>
>> >>> Hi,<br>
>> >>> We're running oVirt 3.6.4.1-1. Lately we're seeing a bunch of<br>
>> >>> events like these:<br>
>> >>> 2016-04-13 10:52:30,735 INFO<br>
>> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]<br>
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] VM<br>
>> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(<a href="http://vm.domain.com" rel="noreferrer" target="_blank">vm.domain.com</a> [1] [1]) moved<br>
>> >>> from<br>
>> >>> 'Up' --> 'Paused'<br>
>> >>> 2016-04-13 10:52:30,815 INFO<br>
>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]<br>
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,<br>
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM <a href="http://vm.domain.com" rel="noreferrer" target="_blank">vm.domain.com</a><br>
>> >>> [1] [1]<br>
>> >>> has been paused.<br>
>> >>> 2016-04-13 10:52:30,898 ERROR<br>
>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]<br>
>> >>> (DefaultQuartzScheduler_Worker-86) [60dea18f] Correlation ID: null,<br>
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM <a href="http://vm.domain.com" rel="noreferrer" target="_blank">vm.domain.com</a><br>
>> >>> [1] [1]<br>
>> >>> has been paused due to no Storage space error.<br>
>> >>> 2016-04-13 10:52:52,320 WARN<br>
>> >>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxyData]<br>
>> >>> (org.ovirt.thread.pool-8-thread-38) [] domain<br>
>> >>> '5de4a000-a9c4-489c-8eee-10368647c413:iscsi01' in problem. vds:<br>
>> >>> '<a href="http://host6.domain.com" rel="noreferrer" target="_blank">host6.domain.com</a> [2] [2]'<br>
>> >>> 2016-04-13 10:52:55,183 INFO<br>
>> >>> [org.ovirt.engine.core.vdsbroker.VmAnalyzer]<br>
>> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] VM<br>
>> >>> 'f9cd282e-110a-4896-98d3-6d320662744d'(<a href="http://vm.domain.com" rel="noreferrer" target="_blank">vm.domain.com</a> [1] [1]) moved<br>
>> >>> from<br>
>> >>> 'Paused' --> 'Up'<br>
>> >>> 2016-04-13 10:52:55,318 INFO<br>
>> >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]<br>
>> >>> (DefaultQuartzScheduler_Worker-70) [3da0f3d4] Correlation ID: null,<br>
>> >>> Call Stack: null, Custom Event ID: -1, Message: VM <a href="http://vm.domain.com" rel="noreferrer" target="_blank">vm.domain.com</a><br>
>> >>> [1] [1]<br>
>> >>> has recovered from paused back to up.<br>
>> >>> The storage domain is far from being full, though (400+ G available<br>
>> >>> right now). Could this be related to this other issue [1]? If not,<br>
>> >>> how could I debug what's going on?<br>
>> >>> Thanks.<br>
>> >>> [1]: <a href="https://www.mail-archive.com/users@ovirt.org/msg32079.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/users@ovirt.org/msg32079.html</a><br>
>> >>> [3]<br>
>> >>> [3]<br>
>> >>> _______________________________________________<br>
>> >>> Users mailing list<br>
>> >>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> >>> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> [4] [4]<br>
>> >>> Links:<br>
>> >>> ------<br>
>> >>> [1] <a href="http://vm.domain.com" rel="noreferrer" target="_blank">http://vm.domain.com</a> [1]<br>
>> >>> [2] <a href="http://host6.domain.com" rel="noreferrer" target="_blank">http://host6.domain.com</a> [2]<br>
>> >>> [3] <a href="https://www.mail-archive.com/users@ovirt.org/msg32079.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/users@ovirt.org/msg32079.html</a> [3]<br>
>> >>> [4] <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> [4]<br>
>> >> _______________________________________________<br>
>> >> Users mailing list<br>
>> >> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> >> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> [4]<br>
>> >> --<br>
>> >> This message is intended only for the addressee and may contain<br>
>> >> confidential information. Unless you are that person, you may not<br>
>> >> disclose its contents or use it in any way and are requested to delete<br>
>> >> the message along with any attachments and notify us immediately.<br>
>> >> This email is not intended to, nor should it be taken to, constitute<br>
>> >> advice.<br>
>> >> The information provided is correct to our knowledge & belief and must<br>
>> >> not<br>
>> >> be used as a substitute for obtaining tax, regulatory, investment,<br>
>> >> legal or<br>
>> >> any other appropriate advice.<br>
>> >> "Transact" is operated by Integrated Financial Arrangements Ltd.<br>
>> >> 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020)<br>
>> >> 7608 5300.<br>
>> >> (Registered office: as above; Registered in England and Wales under<br>
>> >> number: 3727592). Authorised and regulated by the Financial Conduct<br>
>> >> Authority (entered on the Financial Services Register; no. 190856).<br>
>> >> _______________________________________________<br>
>> >> Users mailing list<br>
>> >> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> >> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a> [4]<br>
>> >> Links:<br>
>> >> ------<br>
>> >> [1] <a href="http://vm.domain.com" rel="noreferrer" target="_blank">http://vm.domain.com</a><br>
>> >> [2] <a href="http://host6.domain.com" rel="noreferrer" target="_blank">http://host6.domain.com</a><br>
>> >> [3] <a href="https://www.mail-archive.com/users@ovirt.org/msg32079.html" rel="noreferrer" target="_blank">https://www.mail-archive.com/users@ovirt.org/msg32079.html</a><br>
>> >> [4] <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>> >> _______________________________________________<br>
>> >> Users mailing list<br>
>> >> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> >> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>> > <vdsm.log.gz>_______________________________________________<br>
>> > Users mailing list<br>
>> > <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
>> > <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman/listinfo/users</a><br>
><br>
</div></div></blockquote></div><br></div>