[Users] cluster.min-free-disk option on gluster 3.4 Was: Re: how to temporarily solve low disk space problem
1:29 p.m.
Il 07/01/2014 18:00, users-request(a)ovirt.org ha scritto:
Date: Tue, 7 Jan 2014 16:21:37 +0100
From: Gianluca Cecchi<gianluca.cecchi(a)gmail.com>
To: Gadi Ickowicz<gickowic(a)redhat.com>
Cc: users<users(a)ovirt.org>
Subject: Re: [Users] how to temporarily solve low disk space problem
preventing start of VM
Message-ID:
<CAG2kNCxw54+dANqGX79j1HzjLYgPbxujxkb83GwVzZMfj=jNfA(a)mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
On Tue, Jan 7, 2014 at 4:08 PM, Gadi Ickowicz wrote:
> >Hi,
> >
> >The low disk space is controlled by a parameter in the engine called
FreeSpaceCriticalLowInGB, and can be configured using the cli tool engine-config.
> >
> >you should be able to set it to something like 4 GB or maybe 3GB so you can start
your vms with the following command:
> >
> >engine-config -s FreeSpaceCriticalLowInGB=3
> >
> >After doing this you have to restart the engine service (service ovirt-engine
restart) for the changes to take effect.
> >
> >If you want to see what the current value is use engine-config -g
FreeSpaceCriticalLowInGB.
> >
> >
> >To see a list of all configurable values you can use engine-config --list
> >
> >Thanks,
> >Gadi Ickowicz
It did the trick, thanks!
Could this parameter impact in general only start of new VMs or in any
way also already running VMs?
Gianluca
added gluster-users as they can responde to us questions.
Gianluca, as you are using glusterfs, and as I can see on your df output:
/dev/mapper/fedora-DATA_GLUSTER 30G 23G 7.8G 75%
/gluster/DATA_GLUSTER
node01.mydomain:gvdata 30G 26G 4.6G 85%
/rhev/data-center/mnt/glusterSD/node01.mydomain:gvdata
be careful to gluster cluster.min-free-disk option, that on gluster 3.1
and 3.2 it's default option is 0% (good for you!)
http://gluster.org/community/documentation//index.php/Gluster_3.2:_Settin...
but I can't find the same documentation for gluster 3.4, that I suppose
you're using this gluster version on ovirt; otherwise on red hat storage
documentation cluster.min-free-disk default option is 10% (bad for you):
https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/ht...
so you fill your gvdata volume up to 90% and let we know if it's stop
I/O (or only write) or we can wait for a clarification from gluster guy :-)
best regards
a
--
Amedeo Salvati
RHC{DS,E,VA} - LPIC-3 - UCP - NCLA 11
email: amedeo(a)oscert.net
email: amedeo(a)linux.com
http://plugcomputing.it/redhatcert.php
http://plugcomputing.it/lpicert.php
4:07 a.m.
On Tue, Jan 7, 2014 at 8:29 PM, Amedeo Salvati wrote:
>
> Could this parameter impact in general only start of new VMs or in any
> way also already running VMs?
> Gianluca
added gluster-users as they can responde to us questions.
Gianluca, as you are using glusterfs, and as I can see on your df output:
/dev/mapper/fedora-DATA_GLUSTER 30G 23G 7.8G 75%
/gluster/DATA_GLUSTER
node01.mydomain:gvdata 30G 26G 4.6G 85%
/rhev/data-center/mnt/glusterSD/node01.mydomain:gvdata
be careful to gluster cluster.min-free-disk option, that on gluster 3.1 and
3.2 it's default option is 0% (good for you!)
http://gluster.org/community/documentation//index.php/Gluster_3.2:_Settin...
but I can't find the same documentation for gluster 3.4, that I suppose
you're using this gluster version on ovirt; otherwise on red hat storage
documentation cluster.min-free-disk default option is 10% (bad for you):
https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.1/ht...
so you fill your gvdata volume up to 90% and let we know if it's stop I/O
(or only write) or we can wait for a clarification from gluster guy :-)
best regards
a
My environment is based on OVirt 3.3.2 with Fedora 19 oVirt stable repo.
Plus GlusterFS upgraded to 3.4.2-1.fc19 from updates-testing f19 repo
At this moment I have 4Gb free on xfs filesystem, that is the base for
gluster mount point.
# engine-config -g FreeSpaceCriticalLowInGB
FreeSpaceCriticalLowInGB: 2 version: general
--> all ok
Then
# engine-config -s FreeSpaceCriticalLowInGB=6
# systemctl restart ovirt-engine
--> all ok
Then
Tried "yum update" of a F20 VM that goes without problem (about 50Mb
involved in transaction)
Reboot on VM (so no power off)
--> all ok
Then
Shutdown of VM and attempt to power on it
--> fail
In webadmin gui I get:
"
Error while executing action:
f20:
Cannot run VM. Low disk space on target Storage Domain gvdata.
"
In engine.log I get
2014-01-10 10:37:42,610 WARN [org.ovirt.engine.core.bll.RunVmCommand]
(ajp--127.0.0.1-8702-1) [12c2f039] CanDoAction of action RunVm failed.
Reasons:VAR__ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_TARGET_STORAGE_DOMAIN,$storageName
gvdata
, sharedLocks= ]
Strangely no output in webadmin gui events pane (see also below the
list of them)
Then
# engine-config -s FreeSpaceCriticalLowInGB=1
# systemctl restart ovirt-engine
--> all ok I can power on the f20 VM again
HIH,
Gianluca
Regarding GlusterFS i went through several updates from 3.x to 4.x
version for gluster and in 3.x for engine/hosts, so I don't know in a
clean install starting with oVirt 3.3.2 and GlusterFS 3.4.2 what
engine would have put as a value for FreeSpaceCriticalLowInGB, if a
fixed one (absolute or percentage) or variable depending on total
initial size of XFS filesystem.
Still one strange thing I notice in engine events sequence is this
(please read from last line going up); are normal/expected the
datacenter outputs messages when you restart engine?
2014-Jan-10, 10:50 user admin@internal initiated console session for VM f20
2014-Jan-10, 10:50 VM f20 started on Host f18ovn03
2014-Jan-10, 10:50 user admin@internal initiated console session for VM f20
--> here VM starts ok now
2014-Jan-10, 10:49 VM f20 was started by admin@internal (Host: f18ovn03).
2014-Jan-10, 10:49 User admin@internal logged in.
---> here I have done login again to webadmin gui after engine restart
2014-Jan-10, 10:49 User admin@internal logged in.
2014-Jan-10, 10:47 Storage Pool Manager runs on Host f18ovn03
(Address: 10.4.4.59).
2014-Jan-10, 10:47 Invalid status on Data Center Gluster. Setting
status to Non Responsive.
2014-Jan-10, 10:47 State was set to Up for host f18ovn01.
2014-Jan-10, 10:47 State was set to Up for host f18ovn03.
--> reset FreeSpaceCriticalLowInGB to 1 and restart of engine
2014-Jan-10, 10:47 User admin@internal logged out.
---> shutdown and power off of VM
2014-Jan-10, 10:37 VM f20 is down. Exit message: User shut down
2014-Jan-10, 10:36 user admin@internal initiated console session for VM f20
2014-Jan-10, 10:35 user admin@internal initiated console session for VM f20
2014-Jan-10, 10:31 User admin@internal logged in.
---> here I have done login again to webadmin gui after engine restart
2014-Jan-10, 10:31 User admin@internal logged in.
2014-Jan-10, 10:29 Warning, Low disk space.gvdata domain has 4 GB of free space
2014-Jan-10, 10:29 Storage Pool Manager runs on Host f18ovn03
(Address: 10.4.4.59).
2014-Jan-10, 10:29 Invalid status on Data Center Gluster. Setting
status to Non Responsive.
2014-Jan-10, 10:29 State was set to Up for host f18ovn03.
2014-Jan-10, 10:29 State was set to Up for host f18ovn01.
--> here I have set new higher value for FreeSpaceCriticalLowInGB that
will cause problems and issued restart of engine
2014-Jan-10, 10:29 User admin@internal logged out.
2014-Jan-10, 10:23 User admin@internal logged in.
3766
days inactive
3769
days old
1 comments
2 participants
participants (2)
-
Amedeo Salvati -
Gianluca Cecchi