VM has paused due to no storage space error
Hi users, I have this problem that sometimes 1 to 3 VM just automatically paused with user interaction and getting this error "VM has paused due to no storage space error". any inputs from you guys are very appreciated. TIA Sandvik
On Sun, Oct 2, 2016 at 12:06 PM, Sandvik Agustin <agustinsandvik@gmail.com> wrote:
Hi users,
I have this problem that sometimes 1 to 3 VM just automatically paused with user interaction and getting this error "VM has paused due to no storage space error". any inputs from you guys are very appreciated.
This is expected - when there is no storage space :-) The vm is paused when there are some io pending io requests that could not be fulfilled since you don't have enough space. In a real machine the io requests would fail. In a vm, the vm can pause, you can fix the issue (extend the storage domain), and resume the vm. But I guess there is storage space available, otherwise you would not spend the time sending this mail. This can happen when using thin provisioned disks on block storage (iSCSI, FC). We provision such disk with 1G, and and extend the disk (add 1G) when it becomes too full (by default, free space < 0.5G). If we fail to extend the disk quick enough, the vm will pause before the extend was completed. Once the extend was completed, we resume the vm. So you may see very short pauses, but they should be rare. To understand the issue, we need to inspect vdsm logs from the host running the vm that paused, showing the timeframe when the vm was paused. You should see this message in the log each time a vm pauses: abnormal vm stop device <devname> error ENOSPC Nir
hi, thanks for the reply, *without user interaction.", yes, I still have free space of 26.7TB out of 50TB, i'll try to attach some of the vdsm logs related to this, but is there any chance that this can be cause of network problem and, or system time not sync for each hypervisor and ovirt-engein? Thanks Again, Sandvik On Sun, Oct 2, 2016 at 6:55 PM, Nir Soffer <nsoffer@redhat.com> wrote:
On Sun, Oct 2, 2016 at 12:06 PM, Sandvik Agustin <agustinsandvik@gmail.com> wrote:
Hi users,
I have this problem that sometimes 1 to 3 VM just automatically paused with user interaction and getting this error "VM has paused due to no storage space error". any inputs from you guys are very appreciated.
This is expected - when there is no storage space :-)
The vm is paused when there are some io pending io requests that could not be fulfilled since you don't have enough space.
In a real machine the io requests would fail. In a vm, the vm can pause, you can fix the issue (extend the storage domain), and resume the vm.
But I guess there is storage space available, otherwise you would not spend the time sending this mail.
This can happen when using thin provisioned disks on block storage (iSCSI, FC). We provision such disk with 1G, and and extend the disk (add 1G) when it becomes too full (by default, free space < 0.5G).
If we fail to extend the disk quick enough, the vm will pause before the extend was completed. Once the extend was completed, we resume the vm.
So you may see very short pauses, but they should be rare.
To understand the issue, we need to inspect vdsm logs from the host running the vm that paused, showing the timeframe when the vm was paused.
You should see this message in the log each time a vm pauses:
abnormal vm stop device <devname> error ENOSPC
Nir
Hi Nir, hi Sandvik, As I saw this issue lots of times and as I'm using thin prov. + block storage, I feel concerned. Read my question below. Le 02/10/2016 à 12:55, Nir Soffer a écrit :
On Sun, Oct 2, 2016 at 12:06 PM, Sandvik Agustin <agustinsandvik@gmail.com> wrote:
Hi users,
I have this problem that sometimes 1 to 3 VM just automatically paused with user interaction and getting this error "VM has paused due to no storage space error". any inputs from you guys are very appreciated. This is expected - when there is no storage space :-)
The vm is paused when there are some io pending io requests that could not be fulfilled since you don't have enough space.
In a real machine the io requests would fail. In a vm, the vm can pause, you can fix the issue (extend the storage domain), and resume the vm.
But I guess there is storage space available, otherwise you would not spend the time sending this mail.
This can happen when using thin provisioned disks on block storage (iSCSI, FC). We provision such disk with 1G, and and extend the disk (add 1G) when it becomes too full (by default, free space < 0.5G).
If we fail to extend the disk quick enough,
"quick enough" -> Is there some place where this threshold can be configured?
the vm will pause before the extend was completed. Once the extend was completed, we resume the vm.
So you may see very short pauses, but they should be rare.
To understand the issue, we need to inspect vdsm logs from the host running the vm that paused, showing the timeframe when the vm was paused.
You should see this message in the log each time a vm pauses:
abnormal vm stop device <devname> error ENOSPC
Nir _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- IMPORTANT! This message has been scanned for viruses and phishing links. However, it is your responsibility to evaluate the links and attachments you choose to click. If you are uncertain, we always try to help. Greetings helpdesk@actnet.se
-- IMPORTANT! This message has been scanned for viruses and phishing links. However, it is your responsibility to evaluate the links and attachments you choose to click. If you are uncertain, we always try to help. Greetings helpdesk@actnet.se
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5MAYP4SZZQC5BB...
-- Nicolas ECARNOT
On 14 May 2019, at 22:06, Nicolas Ecarnot <nicolas@ecarnot.net> wrote:
Hi Nir, hi Sandvik,
As I saw this issue lots of times and as I'm using thin prov. + block storage, I feel concerned. Read my question below.
Le 02/10/2016 à 12:55, Nir Soffer a écrit :
On Sun, Oct 2, 2016 at 12:06 PM, Sandvik Agustin <agustinsandvik@gmail.com> wrote:
Hi users,
I have this problem that sometimes 1 to 3 VM just automatically paused with user interaction and getting this error "VM has paused due to no storage space error". any inputs from you guys are very appreciated. This is expected - when there is no storage space :-)
The vm is paused when there are some io pending io requests that could not be fulfilled since you don't have enough space.
In a real machine the io requests would fail. In a vm, the vm can pause, you can fix the issue (extend the storage domain), and resume the vm.
But I guess there is storage space available, otherwise you would not spend the time sending this mail.
This can happen when using thin provisioned disks on block storage (iSCSI, FC). We provision such disk with 1G, and and extend the disk (add 1G) when it becomes too full (by default, free space < 0.5G).
If we fail to extend the disk quick enough,
"quick enough" -> Is there some place where this threshold can be configured?
vdsm.conf ('volume_utilization_percent', '50', 'Together with volume_utilization_chunk_mb, set the minimal free ' 'space before a thin provisioned block volume is extended. Use ' 'lower values to extend earlier.'), ('volume_utilization_chunk_mb', '1024', 'Size of extension chunk in megabytes, and together with ' 'volume_utilization_percent, set the free space limit. Use higher ' 'values to extend in bigger chunks.’), HTH, michal
the vm will pause before the extend was completed. Once the extend was completed, we resume the vm.
So you may see very short pauses, but they should be rare.
To understand the issue, we need to inspect vdsm logs from the host running the vm that paused, showing the timeframe when the vm was paused.
You should see this message in the log each time a vm pauses:
abnormal vm stop device <devname> error ENOSPC
Nir _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- IMPORTANT! This message has been scanned for viruses and phishing links. However, it is your responsibility to evaluate the links and attachments you choose to click. If you are uncertain, we always try to help. Greetings helpdesk@actnet.se
-- IMPORTANT! This message has been scanned for viruses and phishing links. However, it is your responsibility to evaluate the links and attachments you choose to click. If you are uncertain, we always try to help. Greetings helpdesk@actnet.se
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/5MAYP4SZZQC5BB...
-- Nicolas ECARNOT _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/KF4SVQOE7U7ELL...
On Wed, May 15, 2019 at 2:32 PM Nicolas Ecarnot <nicolas@ecarnot.net> wrote:
This can happen when using thin provisioned disks on block storage (iSCSI, FC). We provision such disk with 1G, and and extend the disk (add 1G) when it becomes too full (by default, free space < 0.5G).
If we fail to extend the disk quick enough,
"quick enough" -> Is there some place where this threshold can be configured?
Hi, it is some time I don't go into it, because I anyway tend to use preallocated disks on block storage, but at 4.0, 4.1 time, when using thin dsks, the parameters and their tested values were these in vdsm.conf section [irs] or in some other file such as /etc/vdsm/vdsm.conf.d/50_thin_block_extension_rules.conf [irs] # Together with volume_utilization_chunk_mb, set the minimal free # space before a thin provisioned block volume is extended. Use lower # values to extend earlier. volume_utilization_percent = 20 # Size of extension chunk in megabytes, and together with # volume_utilization_percent, set the free space limit. Use higher # values to extend in bigger chunks. volume_utilization_chunk_mb = 8192 Crosscheck in 4.2/4.3. Also, I remember I had problems in RHV 4.2 and RHVH hosts trying to customize vdsm.conf, but I didn't deal much with it because I decided then to use preallocated and not use anymore those parameters. HIH investigating more, Gianluca
participants (5)
-
Gianluca Cecchi -
Michal Skrivanek -
Nicolas Ecarnot -
Nir Soffer -
Sandvik Agustin