VM has been paused due to NO STORAGE SPACE ERROR ?!?!?!?!
Dear All, Does someone had seen that error ? When I run this command from my virtual machine: # time dd if=/dev/zero of=enrico.dd bs=4k count=10000000 VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state. Inside events from ovirt web interface I see this: "VM has been paused due to lack of storage space" but no ERROR found in /var/log/vdsm.log. My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O. Any ideas ? Thanks a lot. Best Regards Enrico -- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
This is a multi-part message in MIME format. --------------BED635C9A6D1FFD50A694810 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit yes ... it's a thin provisioning , in fact with preallocated disk type I haven't any problem. Thanks you so much Best Regards Enrico Il 16/03/2018 13:28, Karli Sjöberg ha scritto:
Den 16 mars 2018 12:26 skrev Enrico Becchetti <enrico.becchetti@pg.infn.it>:
Dear All, Does someone had seen that error ? When I run this command from my virtual machine:
# time dd if=/dev/zero of=enrico.dd bs=4k count=10000000
VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state.
Inside events from ovirt web interface I see this:
"VM has been paused due to lack of storage space"
but no ERROR found in /var/log/vdsm.log.
My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O.
Any ideas ? Thanks a lot. Best Regards Enrico
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it ______________________________________________________________________
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?
/K Dear All, Does someone had seen that error ? When I run this command from my virtual machine:
# time dd if=/dev/zero of=enrico.dd bs=4k count=10000000
VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state.
Inside events from ovirt web interface I see this:
"VM has been paused due to lack of storage space"
but no ERROR found in /var/log/vdsm.log.
My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O.
Any ideas ? Thanks a lot. Best Regards Enrico
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it ______________________________________________________________________
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________ --------------BED635C9A6D1FFD50A694810 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> <div class="moz-cite-prefix">yes ... it's a thin provisioning , in fact with preallocated disk type I haven't any problem.<br> Thanks you so much <br> Best Regards<br> Enrico<br> <br> Il 16/03/2018 13:28, Karli Sjöberg ha scritto:<br> </div> <blockquote type="cite" cite="mid:e14510c4-501c-46e6-bdf9-bc65d20f0bb7@email.android.com"> <meta name="Generator" content="Z-Push"> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> <span style="font-size: 11pt"> <div dir="auto"> <div><br> <div class="gmail_extra"><br> <div class="gmail_quote">Den 16 mars 2018 12:26 skrev Enrico Becchetti <a class="moz-txt-link-rfc2396E" href="mailto:enrico.becchetti@pg.infn.it"><enrico.becchetti@pg.infn.it></a>:<br type="attribution"> <blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div> Dear All,<br> Does someone had seen that error ? When I run this command from my <br> virtual machine:<br> <br> # time dd if=/dev/zero of=enrico.dd bs=4k count=10000000<br> <br> VM was paused due to kind a storage error/problem. Strange message<br> because tell about "no storage space error" but ovirt puts virtual <br> machine in<br> a paused state.<br> <br> Inside events from ovirt web interface I see this:<br> <br> "VM has been paused due to lack of storage space"<br> <br> but no ERROR found in /var/log/vdsm.log.<br> <br> My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and <br> before now<br> I haven't see any other problem during the normal functioning of the vm <br> , it's seem<br> that this error occurs only when there is massive I/O.<br> <br> Any ideas ?<br> Thanks a lot.<br> Best Regards<br> Enrico<br> <br> <br> -- <br> _______________________________________________________________________<br> <br> Enrico Becchetti Servizio di Calcolo e Reti<br> <br> Istituto Nazionale di Fisica Nucleare - Sezione di Perugia<br> Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)<br> Phone:+39 075 5852777 Mail: Enrico.Becchettipg.infn.it<br> ______________________________________________________________________<br> <br> _______________________________________________<br> Users mailing list<br> <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> </div> </blockquote> </div> <br> </div> </div> <div class="gmail_extra" dir="auto">I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?</div> <div class="gmail_extra" dir="auto"><br> </div> <div class="gmail_extra" dir="auto">/K</div> </div> </span> Dear All,<br> Does someone had seen that error ? When I run this command from my <br> virtual machine:<br> <br> # time dd if=/dev/zero of=enrico.dd bs=4k count=10000000<br> <br> VM was paused due to kind a storage error/problem. Strange message<br> because tell about "no storage space error" but ovirt puts virtual <br> machine in<br> a paused state.<br> <br> Inside events from ovirt web interface I see this:<br> <br> "VM has been paused due to lack of storage space"<br> <br> but no ERROR found in /var/log/vdsm.log.<br> <br> My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and <br> before now<br> I haven't see any other problem during the normal functioning of the vm <br> , it's seem<br> that this error occurs only when there is massive I/O.<br> <br> Any ideas ?<br> Thanks a lot.<br> Best Regards<br> Enrico<br> <br> -- <br> _______________________________________________________________________<br> <br> Enrico Becchetti Servizio di Calcolo e Reti<br> <br> Istituto Nazionale di Fisica Nucleare - Sezione di Perugia<br> Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY)<br> Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it<br> ______________________________________________________________________<br> <br> _______________________________________________<br> Users mailing list<br> <a class="moz-txt-link-abbreviated" href="mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br> </at></blockquote> <p><br> </p> <pre class="moz-signature" cols="72">-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________ </pre> </body> </html> --------------BED635C9A6D1FFD50A694810--
Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :
Den 16 mars 2018 12:26 skrev Enrico Becchetti <enrico.becchetti@pg.infn.it>:
Dear All, Does someone had seen that error ?
Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space.
Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?
All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx). Enrico, do you know if a bug has been filed about this? -- Nicolas ECARNOT
On 16/03/18 13:46, Nicolas Ecarnot wrote:
Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :
Den 16 mars 2018 12:26 skrev Enrico Becchetti <enrico.becchetti@pg.infn.it>:
Dear All, Does someone had seen that error ?
Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space.
Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?
All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx).
Enrico, do you know if a bug has been filed about this?
Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour. If you need high write performance your VM disks should always be preallocated. We only use Thin Provision for VMs where we know that disk writes are low (eg network services, CPU-bound apps, etc). Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice. "Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856).
Le 16/03/2018 à 15:48, Alex Crow a écrit :
On 16/03/18 13:46, Nicolas Ecarnot wrote:
Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :
Den 16 mars 2018 12:26 skrev Enrico Becchetti <enrico.becchetti@pg.infn.it>:
Dear All, Does someone had seen that error ?
Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space.
Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?
All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx).
Enrico, do you know if a bug has been filed about this?
Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour.
AFAIR, most of them went back up and running by themselves (we had to manually some of them from times to times). The storage side weakness is an interesting trail to follow. We also experienced this behavior when migrating lots of VMs at once, yet using a dedicated storage network. Being on this mailing list since long, I remember we already discussed several times about how some users feel how oVirt can appear sensitive to storage latencies. On my side, the site where most of our workload resides is still in 3.6, so I can not yet witness the efforts oVirt devs have made to cope with this in 4.2 but I'm sure they did. -- Nicolas ECARNOT
Il 16/03/2018 15:48, Alex Crow ha scritto:
On 16/03/18 13:46, Nicolas Ecarnot wrote:
Le 16/03/2018 à 13:28, Karli Sjöberg a écrit :
Den 16 mars 2018 12:26 skrev Enrico Becchetti <enrico.becchetti@pg.infn.it>:
Dear All, Does someone had seen that error ?
Yes, I experienced it dozens of times on 3.6 (my 4.2 setup has insufficient workload to trigger such event). And in every case, there was no actual lack of space.
Enrico Becchetti Servizio di Calcolo e Reti I think I remember something to do with thin provisioning and not being able to grow fast enough, so out of space. Are the VM's disk thick or thin?
All our storage domains are thin-prov. and served by iSCSI (Equallogic PS6xxx and 4xxx).
Enrico, do you know if a bug has been filed about this?
Did the VM remain paused? In my experience the VM just gets temporarily paused while the storage is expanded. RH confirmed to me in a ticket that this is expected behaviour.
If you need high write performance your VM disks should always be preallocated. We only use Thin Provision for VMs where we know that disk writes are low (eg network services, CPU-bound apps, etc).
Thanks a lot !!! Best Regards Enrico
Alex -- This message is intended only for the addressee and may contain confidential information. Unless you are that person, you may not disclose its contents or use it in any way and are requested to delete the message along with any attachments and notify us immediately. This email is not intended to, nor should it be taken to, constitute advice. The information provided is correct to our knowledge & belief and must not be used as a substitute for obtaining tax, regulatory, investment, legal or any other appropriate advice.
"Transact" is operated by Integrated Financial Arrangements Ltd. 29 Clement's Lane, London EC4N 7AE. Tel: (020) 7608 4900 Fax: (020) 7608 5300. (Registered office: as above; Registered in England and Wales under number: 3727592). Authorised and regulated by the Financial Conduct Authority (entered on the Financial Services Register; no. 190856). _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- _______________________________________________________________________ Enrico Becchetti Servizio di Calcolo e Reti Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
On Fri, Mar 16, 2018 at 1:25 PM, Enrico Becchetti < enrico.becchetti@pg.infn.it> wrote:
Dear All, Does someone had seen that error ? When I run this command from my virtual machine:
# time dd if=/dev/zero of=enrico.dd bs=4k count=10000000
I don't think it's a very interesting test case for IO performance, but in any case, it may cause the VM to try to write faster than its thin provisioned disk can be extended. A simple workaround would be to change in VDSM the threshold of when it gets extended and by how much. For example: [irs] volume_utilization_percent = 15 volume_utilization_chunk_mb = 4048 Y.
VM was paused due to kind a storage error/problem. Strange message because tell about "no storage space error" but ovirt puts virtual machine in a paused state.
Inside events from ovirt web interface I see this:
"VM has been paused due to lack of storage space"
but no ERROR found in /var/log/vdsm.log.
My oVirt enviroment 4.2.1 has three hypervivosr with FC storage and before now I haven't see any other problem during the normal functioning of the vm , it's seem that this error occurs only when there is massive I/O.
Any ideas ? Thanks a lot. Best Regards Enrico
-- _______________________________________________________________________
Enrico Becchetti Servizio di Calcolo e Reti
Istituto Nazionale di Fisica Nucleare - Sezione di Perugia Via Pascoli,c/o Dipartimento di Fisica 06123 Perugia (ITALY) Phone:+39 075 5852777 Mail: Enrico.Becchetti<at>pg.infn.it ______________________________________________________________________
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
participants (5)
-
Alex Crow -
Enrico Becchetti -
Karli Sjöberg -
Nicolas Ecarnot -
Yaniv Kaul