On my oVirt 4.1.9 cluster, the hosted engine filled up its disk last night (it was deployed from the engine appliance image and had a 20G root and 5G swap). I gained a little space by trimming down swap, but I need to clean it up. It looks like the problem is the Postgres database - /var/lib/pgsql/data is using 16G. Is there some cleanup I can do (and then some way to keep it from happening again)? Also, is there a way to extend the hosted engine disk? It's on iSCSI storage, and the LUN has free space. -- Chris Adams <cma@cmadams.net>
On October 11, 2019 3:43:55 PM GMT+03:00, Chris Adams <cma@cmadams.net> wrote:
On my oVirt 4.1.9 cluster, the hosted engine filled up its disk last night (it was deployed from the engine appliance image and had a 20G root and 5G swap). I gained a little space by trimming down swap, but I need to clean it up.
It looks like the problem is the Postgres database - /var/lib/pgsql/data is using 16G. Is there some cleanup I can do (and then some way to keep it from happening again)?
Also, is there a way to extend the hosted engine disk? It's on iSCSI storage, and the LUN has free space.
I have noticed during engine upgrade that the tool proposes database cleanup, but I have never used it. Best Regards, Strahil Nikolov
Hi! We have similar problem and it looks like psql vacuuming isn't working properly. I expect that this problem will be gone when we will upgrade ovirt to latest version. Until then we are performing weekly vacuuming of top N (usualy 4) tables by "external size" in ovirt_engine_history database. A little bit more about vacuuming: https://www.postgresql.org/docs/9.1/sql-vacuum.html External size: https://wiki-bsse.ethz.ch/display/ITDOC/Check+size+of+tables+and+objects+in+... , https://matjaz.it/tutorial-sizes-of-tables-and-databases-in-postgresql/ Example which queries we are running in our case: To get top 4 tables ordered by "external size": SELECT relname, pg_size_pretty(pg_total_relation_size(relid)), pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC limit 4; We use names from previous query and perform vacuum table by table: VACUUM (VERBOSE, FULL) ${TABLE}; To get new size of table after vaccuming: SELECT pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables WHERE relname::text like '${TABLE}'; Note that ${TABLE} is variable with value for example: "vm_interface_samples_history" I suggest to run few queries and check "external size" of your tables. In our case vacuuming has huge effect on releasing disk space and it takes jus few seconds for database. Oh yes, I almost forgot. As Strahi said, there is some ovirt tool (script) for cleaning up psql database but in our case it didn't have such huge effect as it didn't perform full vacuum. Regards, Pavel On 11/10/2019 15:50, Strahil Nikolov wrote:
On October 11, 2019 3:43:55 PM GMT+03:00, Chris Adams <cma@cmadams.net> wrote:
On my oVirt 4.1.9 cluster, the hosted engine filled up its disk last night (it was deployed from the engine appliance image and had a 20G root and 5G swap). I gained a little space by trimming down swap, but I need to clean it up.
It looks like the problem is the Postgres database - /var/lib/pgsql/data is using 16G. Is there some cleanup I can do (and then some way to keep it from happening again)?
Also, is there a way to extend the hosted engine disk? It's on iSCSI storage, and the LUN has free space. I have noticed during engine upgrade that the tool proposes database cleanup, but I have never used it.
Best Regards, Strahil Nikolov _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YU6Y3M5EL6LO2R...
-- -- Pavel Sipos, Arnes <pavel.sipos@arnes.si> ARNES, p.p. 7, SI-1001 Ljubljana, Slovenia T: +386 1 479 88 00 W: www.arnes.si, aai.arnes.si
Hi Pavel, you can use engine tools: engine-vacuum and dwh-vacuum. It can do full vacuum with -f option and also it runs with engine-setup when you choose it. On 10/11/19 4:36 PM, Pavel Šipoš wrote:
Hi!
We have similar problem and it looks like psql vacuuming isn't working properly. I expect that this problem will be gone when we will upgrade ovirt to latest version. Until then we are performing weekly vacuuming of top N (usualy 4) tables by "external size" in ovirt_engine_history database.
A little bit more about vacuuming: https://www.postgresql.org/docs/9.1/sql-vacuum.html External size: https://wiki-bsse.ethz.ch/display/ITDOC/Check+size+of+tables+and+objects+in+... , https://matjaz.it/tutorial-sizes-of-tables-and-databases-in-postgresql/
Example which queries we are running in our case:
To get top 4 tables ordered by "external size": SELECT relname, pg_size_pretty(pg_total_relation_size(relid)), pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables ORDER BY pg_total_relation_size(relid) DESC limit 4;
We use names from previous query and perform vacuum table by table: VACUUM (VERBOSE, FULL) ${TABLE};
To get new size of table after vaccuming: SELECT pg_size_pretty(pg_total_relation_size(relid) - pg_relation_size(relid)) FROM pg_catalog.pg_statio_user_tables WHERE relname::text like '${TABLE}';
Note that ${TABLE} is variable with value for example: "vm_interface_samples_history"
I suggest to run few queries and check "external size" of your tables.
In our case vacuuming has huge effect on releasing disk space and it takes jus few seconds for database.
Oh yes, I almost forgot. As Strahi said, there is some ovirt tool (script) for cleaning up psql database but in our case it didn't have such huge effect as it didn't perform full vacuum.
Regards, Pavel
On 11/10/2019 15:50, Strahil Nikolov wrote:
On October 11, 2019 3:43:55 PM GMT+03:00, Chris Adams <cma@cmadams.net> wrote:
On my oVirt 4.1.9 cluster, the hosted engine filled up its disk last night (it was deployed from the engine appliance image and had a 20G root and 5G swap). I gained a little space by trimming down swap, but I need to clean it up.
It looks like the problem is the Postgres database - /var/lib/pgsql/data is using 16G. Is there some cleanup I can do (and then some way to keep it from happening again)?
Also, is there a way to extend the hosted engine disk? It's on iSCSI storage, and the LUN has free space. I have noticed during engine upgrade that the tool proposes database cleanup, but I have never used it.
Best Regards, Strahil Nikolov _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/YU6Y3M5EL6LO2R...
Best regards. -- Lucie Leistnerova Senior Quality Engineer, QE Cloud, RHVM Red Hat EMEA IRC: lleistne @ #rhev-qe
Once upon a time, Lucie Leistnerova <lleistne@redhat.com> said:
you can use engine tools: engine-vacuum and dwh-vacuum. It can do full vacuum with -f option and also it runs with engine-setup when you choose it.
engine-vacuum doesn't appear to touch the DWB DB, and I don't have a dwh-vacuum (on 4.1). After reading a RHBZ, the issue was the various *_samples_* tables, so I did: ######################## systemctl stop ovirt-engine-dwhd.service . /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf PGPASSWORD="$DWH_DB_PASSWORD" psql \ -h "$DWH_DB_HOST" \ -p "$DWH_DB_PORT" \ -U "$DWH_DB_USER" \ -d "$DWH_DB_DATABASE" \ <<'EOF' vacuum full host_interface_samples_history; vacuum full vm_disk_samples_history; vacuum full vm_disks_usage_samples_history; vacuum full vm_interface_samples_history; vacuum full vm_samples_history; EOF systemctl start ovirt-engine-dwhd.service ######################## That reduced my Postgres DB size from 15G to 1.4G. -- Chris Adams <cma@cmadams.net>
On 10/14/19 3:07 PM, Chris Adams wrote:
Once upon a time, Lucie Leistnerova <lleistne@redhat.com> said:
you can use engine tools: engine-vacuum and dwh-vacuum. It can do full vacuum with -f option and also it runs with engine-setup when you choose it. engine-vacuum doesn't appear to touch the DWB DB, and I don't have a dwh-vacuum (on 4.1). Ah, you are right. dwh-vacuum was added in 4.2. After reading a RHBZ, the issue was the various *_samples_* tables, so I did:
######################## systemctl stop ovirt-engine-dwhd.service
. /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf PGPASSWORD="$DWH_DB_PASSWORD" psql \ -h "$DWH_DB_HOST" \ -p "$DWH_DB_PORT" \ -U "$DWH_DB_USER" \ -d "$DWH_DB_DATABASE" \ <<'EOF' vacuum full host_interface_samples_history; vacuum full vm_disk_samples_history; vacuum full vm_disks_usage_samples_history; vacuum full vm_interface_samples_history; vacuum full vm_samples_history; EOF
systemctl start ovirt-engine-dwhd.service ########################
That reduced my Postgres DB size from 15G to 1.4G.
-- Lucie Leistnerova Senior Quality Engineer, QE Cloud, RHVM Red Hat EMEA IRC: lleistne @ #rhev-qe
Once upon a time, Lucie Leistnerova <lleistne@redhat.com> said:
Ah, you are right. dwh-vacuum was added in 4.2.
Just for interest, I tried this on my 4.3.6 dev setup, and it just gives: [root@engine ~]# dwh-vacuum /usr/bin/dwh-vacuum: line 47: vacuumdb: command not found Looks like it needs the SCL enablement that engine-vacuum has. -- Chris Adams <cma@cmadams.net>
On 10/14/19 3:36 PM, Chris Adams wrote:
Once upon a time, Lucie Leistnerova <lleistne@redhat.com> said:
Ah, you are right. dwh-vacuum was added in 4.2. Just for interest, I tried this on my 4.3.6 dev setup, and it just gives:
[root@engine ~]# dwh-vacuum /usr/bin/dwh-vacuum: line 47: vacuumdb: command not found
Looks like it needs the SCL enablement that engine-vacuum has.
My mistake, I thought it was fixed already but it was only for engine-vacuum. I've created new BZ https://bugzilla.redhat.com/show_bug.cgi?id=1761494 Thanks for catching this! -- Lucie Leistnerova Senior Quality Engineer, QE Cloud, RHVM Red Hat EMEA IRC: lleistne @ #rhev-qe
participants (4)
-
Chris Adams -
Lucie Leistnerova -
Pavel Šipoš -
Strahil Nikolov