local on host storage domain full, how to clean up
Hello, I have a local based storage domain that has got full, so that it is now inactive and virtual machines that are on it are paused (vm paused due to lack of storage space). Any advice on how to clean up, eventually deleting some of them? # df -h /2t_2 Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_2t_2-2t_2 1.8T 1.7T 0 100% /2t_2 [root@ovirt01 fdf9546c-68fa-42c9-8a10-78ef3ee534b8]# Is there any easy way to map disk-to-vm of directories inside /2t_2/images/dbf9611d-9090-42d6-81e0-58105bc20011/images/ so that I can "sacrifice" some VMs deleting the corresponding disks' directories, to be able at least to activate the storage domain again and make a cleaner check? Or any other suggestions? I'm not able to expand it. It is not directly managed by me and I suppose too much storage over-provisioning has been done. Thanks in advance. Gianluca
On Fri, May 27, 2022 at 11:04 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, I have a local based storage domain that has got full, so that it is now inactive and virtual machines that are on it are paused (vm paused due to lack of storage space). Any advice on how to clean up, eventually deleting some of them?
# df -h /2t_2 Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_2t_2-2t_2 1.8T 1.7T 0 100% /2t_2
Why is the size (1.8T) different from Used (1.7T) despite Use%=100%? If this is because it's an ext4 filesystem, you can try reducing the amount reserved to root ('tune2fs -m 1' or so). I think the fs should be unmounted for this.
[root@ovirt01 fdf9546c-68fa-42c9-8a10-78ef3ee534b8]#
Is there any easy way to map disk-to-vm of directories inside /2t_2/images/dbf9611d-9090-42d6-81e0-58105bc20011/images/ so that I can "sacrifice" some VMs deleting the corresponding disks' directories, to be able at least to activate the storage domain again and make a cleaner check?
I don't believe severe damage can happen by doing this, but really have no idea.
Or any other suggestions? I'm not able to expand it.
Perhaps you can also temporarily move some VMs' disks to some other local storage and symlink there - no idea what vdsm might do. I think qemu will be happy, if it can access them (you might need to play with permissions/selinux/etc). I'd personally try any of the above on a test setup first.
It is not directly managed by me and I suppose too much storage over-provisioning has been done.
Good luck and best regards, -- Didi
On Mon, May 30, 2022 at 7:56 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Fri, May 27, 2022 at 11:04 AM Gianluca Cecchi <gianluca.cecchi@gmail.com> wrote:
Hello, I have a local based storage domain that has got full, so that it is now
inactive and virtual machines that are on it are paused (vm paused due to lack of storage space).
Any advice on how to clean up, eventually deleting some of them?
# df -h /2t_2 Filesystem Size Used Avail Use% Mounted on /dev/mapper/vg_2t_2-2t_2 1.8T 1.7T 0 100% /2t_2
Why is the size (1.8T) different from Used (1.7T) despite Use%=100%?
If this is because it's an ext4 filesystem, you can try reducing the amount reserved to root ('tune2fs -m 1' or so). I think the fs should be unmounted for this.
Yes, I think the reserved default space. In the meantime I selected a not important VM where I had a 100Gb data disk (it was the only one of this type of disk size, so it was easy to find it on filesystem) and executed cat /dev/null > disk_file so that I didn't have any logical inconsistency from oVirt point of view and 100Gb of space immediately released. Then I was able to activate the storage domain and delete the VM and go on with further analysis on disk usage. Anyway it would be useful to have a way to search through the engine database to have an easier mapping between disk files and VMs. Gianluca
Do you mean something like this ? /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select disk_id,image_guid,disk_alias,disk_profile_name from all_disks where disk_content_type='0' order by disk_alias" > /root/disk_ids-to_vms.txt isi
On Mon, May 30, 2022 at 4:44 PM _isi _ <uli.iske@elkb.de> wrote:
Do you mean something like this ?
/usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select disk_id,image_guid,disk_alias,disk_profile_name from all_disks where disk_content_type='0' order by disk_alias" > /root/disk_ids-to_vms.txt
isi _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3FBFDABUYYTMVW...
partially, because in my case if I take a VM with VM ID 711e5dc6-04a1-42a4-b52c-96685bc19b1f It has 3 disks, one of which has the alias ostack-ceph1_Disk3 the query for this disk gives select disk_id,image_guid,disk_alias,disk_profile_name from all_disks where disk_content_type='0' order by disk_alias; . . . 1093ada3-65b2-407a-a6b8-fd9529f4a0db | ded7c7c7-477f-44e3-8ac5-8908bde6340c | ostack-ceph1_Disk3 | 2t_1 And I get under /2t_1 /2t_1/images/caa6fd69-e371-4476-89eb-cbbae82c2add/images # ll | grep 1093 drwxr-xr-x. 2 vdsm kvm 4096 Apr 20 2021 1093ada3-65b2-407a-a6b8-fd9529f4a0db # # cd 1093ada3-65b2-407a-a6b8-fd9529f4a0db/ # ll total 156872396 -rw-rw----. 1 vdsm kvm 161061273600 Apr 30 2021 ded7c7c7-477f-44e3-8ac5-8908bde6340c -rw-r--r--. 1 vdsm kvm 289 Apr 20 2021 ded7c7c7-477f-44e3-8ac5-8908bde6340c.meta # but I miss the link between the id of the VM and the disk, that is what I would like to get
You can use the all_disks_for_vms view, instead of all_disks. Then add the vm_id and / or vm_names columns. For example: /usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select disk_id,image_guid,disk_alias,disk_profile_name,vm_id,vm_names from all_disks_for_vms where disk_content_type='0' order by disk_alias" > /root/disk_ids-to_vms.txt -Patrick Hibbs On Sun, 2022-06-05 at 16:16 +0200, Gianluca Cecchi wrote:
On Mon, May 30, 2022 at 4:44 PM _isi _ <uli.iske@elkb.de> wrote:
Do you mean something like this ?
/usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select disk_id,image_guid,disk_alias,disk_profile_name from all_disks where disk_content_type='0' order by disk_alias" > /root/disk_ids- to_vms.txt
isi _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3FBFDABUYYTMVW...
partially, because in my case if I take a VM with VM ID 711e5dc6- 04a1-42a4-b52c-96685bc19b1f
It has 3 disks, one of which has the alias ostack-ceph1_Disk3
the query for this disk gives select disk_id,image_guid,disk_alias,disk_profile_name from all_disks where disk_content_type='0' order by disk_alias; . . . 1093ada3-65b2-407a-a6b8-fd9529f4a0db | ded7c7c7-477f-44e3-8ac5- 8908bde6340c | ostack-ceph1_Disk3 | 2t_1
And I get under /2t_1
/2t_1/images/caa6fd69-e371-4476-89eb-cbbae82c2add/images
# ll | grep 1093 drwxr-xr-x. 2 vdsm kvm 4096 Apr 20 2021 1093ada3-65b2-407a-a6b8- fd9529f4a0db #
# cd 1093ada3-65b2-407a-a6b8-fd9529f4a0db/ # ll total 156872396 -rw-rw----. 1 vdsm kvm 161061273600 Apr 30 2021 ded7c7c7-477f-44e3- 8ac5-8908bde6340c -rw-r--r--. 1 vdsm kvm 289 Apr 20 2021 ded7c7c7-477f-44e3- 8ac5-8908bde6340c.meta #
but I miss the link between the id of the VM and the disk, that is what I would like to get
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/BJ5J4KNMMZ4PNW...
On Sun, Jun 5, 2022 at 7:00 PM Patrick Hibbs <hibbsncc1701@gmail.com> wrote:
You can use the all_disks_for_vms view, instead of all_disks. Then add the vm_id and / or vm_names columns.
For example:
/usr/share/ovirt-engine/dbscripts/engine-psql.sh -c "select disk_id,image_guid,disk_alias,disk_profile_name,vm_id,vm_names from all_disks_for_vms where disk_content_type='0' order by disk_alias" > /root/disk_ids-to_vms.txt
-Patrick Hibbs
Thas seems indeed better, thanks! engine=# select disk_id,image_guid,disk_alias,disk_profile_name,vm_id,vm_names from all_disks_for_vms where disk_content_type='0' and disk_alias='ostack-ceph1_Disk3'; disk_id | image_guid | disk_alias | disk_profile_name | vm_id | vm_names --------------------------------------+--------------------------------------+--------------------+- ------------------+--------------------------------------+-------------- 1093ada3-65b2-407a-a6b8-fd9529f4a0db | ded7c7c7-477f-44e3-8ac5-8908bde6340c | ostack-ceph1_Disk3 | 2t_1 | 711e5dc6-04a1-42a4-b52c-96685bc19b1f | ostack-ceph1 (1 row) so that I can link the directory and disk file under /2t_1/images/caa6fd69-e371-4476-89eb-cbbae82c2add/images 1093ada3-65b2-407a-a6b8-fd9529f4a0db/ded7c7c7-477f-44e3-8ac5-8908bde6340c with the disk alias ostack-ceph1_Disk3 and with the corresponding vm id as seen from GUI 711e5dc6-04a1-42a4-b52c-96685bc19b1f and its name ostack-ceph1 I'm going to crosscheck in a SAN based configuration if any sort of"translation" is to be done, due to the layout based on LVM and not on filesystem Gianluca
uczestnicy (4)
-
_isi _ -
Gianluca Cecchi -
Patrick Hibbs -
Yedidyah Bar David