Hosted Engine ansible runner artifacts cleanup

Hey, I can't get the artifacts to cleanup, i have configured the ansible runner config.yml but it seems not to be running. Should i restart the ovirt-engine service? What is the best way to manually cleanup the artifacts (i can't get the correct ansible-runner command to clean it up) as it is starting to fill up disk space Thanks

You mean you want to remove the Hosted engine if yes you can simply run *engine-cleanup* on one of the host. On Sun, Oct 3, 2021 at 3:22 PM Erez Zarum <erezz@nanosek.com> wrote:
Hey, I can't get the artifacts to cleanup, i have configured the ansible runner config.yml but it seems not to be running. Should i restart the ovirt-engine service?
What is the best way to manually cleanup the artifacts (i can't get the correct ansible-runner command to clean it up) as it is starting to fill up disk space
Thanks _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/QAEPEFAQQUPRJR...

Hey, the ovirt-engine runs ansible playbooks and uses the ansible-runner project, what happens is that if you have a lot of tasks the artifacts of ansible-runner fills up the disk space of the ovirt-engine. They are located in /usr/share/ovirt-engine/ansible-runner There's a config in /etc/ansible-runner-service that controls the artifacts rotation, for some reason it does not kick in, in general there's a way to manually run the ansible-runner and let it rotate the artifacts, i want to understand how to do this without messing up the ovirt-engine. Thanks

On Tue, Oct 5, 2021 at 3:54 PM Erez Zarum <erezz@nanosek.com> wrote:
Hey, the ovirt-engine runs ansible playbooks and uses the ansible-runner project, what happens is that if you have a lot of tasks the artifacts of ansible-runner fills up the disk space of the ovirt-engine. They are located in /usr/share/ovirt-engine/ansible-runner There's a config in /etc/ansible-runner-service that controls the artifacts rotation, for some reason it does not kick in, in general there's a way to manually run the ansible-runner and let it rotate the artifacts, i want to understand how to do this without messing up the ovirt-engine.
I think the only thing out-of-the-box that should clean these logs is logrotate, for which we indeed supply a conf file. Does logrotate run on your system? Should run daily by default, by cron. The configuration was changed recently, see also: https://bugzilla.redhat.com/show_bug.cgi?id=1941507 Best regards, -- Didi

Hi, if you are talking about the `/var/log/ovirt-engine/ansible-runner-service.log`. The logrotate is located in `/etc/logrotate.d/ansible-runner-service` and should archive weekly. Should work automatically from ansible-runner-service 1.0.7. The logroate includes daily the `/etc/logrotate.d` in `/etc/cron.daily/logrotate` so it should be loaded. It is possible that you have broken the logrotate/cron. Martin Necas On Tue, Oct 5, 2021 at 4:19 PM Yedidyah Bar David <didi@redhat.com> wrote:
On Tue, Oct 5, 2021 at 3:54 PM Erez Zarum <erezz@nanosek.com> wrote:
Hey, the ovirt-engine runs ansible playbooks and uses the ansible-runner
project, what happens is that if you have a lot of tasks the artifacts of ansible-runner fills up the disk space of the ovirt-engine.
They are located in /usr/share/ovirt-engine/ansible-runner There's a config in /etc/ansible-runner-service that controls the artifacts rotation, for some reason it does not kick in, in general there's a way to manually run the ansible-runner and let it rotate the artifacts, i want to understand how to do this without messing up the ovirt-engine.
I think the only thing out-of-the-box that should clean these logs is logrotate, for which we indeed supply a conf file. Does logrotate run on your system? Should run daily by default, by cron.
The configuration was changed recently, see also:
https://bugzilla.redhat.com/show_bug.cgi?id=1941507
Best regards, -- Didi _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/LKFBMJT5SPTKQZ...

Hey, No, i am talking about the ansible-runner artifacts, when the ovirt-engine runs a playbook (which is a task in the webui for example), it creates a folder in /usr/share/ovirt-engine/ansible-runner/artifacts for each run. There's a config.yml in /etc/ansible-runner-service but for some reason it does not affect anything. Thanks.

On Tue, Oct 5, 2021 at 8:50 PM Erez Zarum <erezz@nanosek.com> wrote:
Hey, No, i am talking about the ansible-runner artifacts, when the ovirt-engine runs a playbook (which is a task in the webui for example), it creates a folder in /usr/share/ovirt-engine/ansible-runner/artifacts for each run.
Thanks for the clarification. I don't have /usr/share/ovirt-engine/ansible-runner/artifacts on my engine machine. Which version do you use? Perhaps file a bug in bugzilla and attach a sosreport.
There's a config.yml in /etc/ansible-runner-service but for some reason it does not affect anything.
What do you mean by that? Did you try changing it and saw that this does not affect it at all? Or a specific setting? Something else? Thanks and best regards, -- Didi

On Wed, Oct 6, 2021 at 7:50 AM Yedidyah Bar David <didi@redhat.com> wrote:
On Tue, Oct 5, 2021 at 8:50 PM Erez Zarum <erezz@nanosek.com> wrote:
Hey, No, i am talking about the ansible-runner artifacts, when the
ovirt-engine runs a playbook (which is a task in the webui for example), it creates a folder in /usr/share/ovirt-engine/ansible-runner/artifacts for each run.
Thanks for the clarification. I don't have /usr/share/ovirt-engine/ansible-runner/artifacts on my engine machine. Which version do you use? Perhaps file a bug in bugzilla and attach a sosreport.
Periodic artifact clean up has been added to ansible-runner-service 1.0.6: https://github.com/ansible/ansible-runner-service/releases But I highly recommend to upgrade to the latest version 1.0.7
There's a config.yml in /etc/ansible-runner-service but for some reason
it does not affect anything.
What do you mean by that? Did you try changing it and saw that this does not affect it at all? Or a specific setting? Something else?
Thanks and best regards, -- Didi _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RUTJT53TQWJD4C...
-- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

I can't just upgrade, it's part of the ovirt-engine (hosted/appliance). Anyhow, it is indeed version 1.06 already. What i am asking is how do i run the cleanup process on the ovirt-engine, even if manually, what is the best practice for that?

Hey, I was wrong with the path. [root@ovirthe artifacts]# pwd /usr/share/ovirt-engine/ansible-runner-service-project/artifacts [root@ovirthe artifacts]# ls | wc -l 1851 [root@ovirthe artifacts]# du -hs . 2.5G . [root@ovirthe artifacts]# What is the best way to run ansible-runner and let it clean the artifacts?

Thank you all. Answering myself. cd /usr/share/ovirt-engine/ansible-runner-service-project ansible-runner --rotate-artifacts 20 --artifact-dir /usr/share/ovirt-engine/ansible-runner-service-project/artifacts -m ping --hosts all[0] run /usr/share/ovirt-engine/ansible-runner-service-project This will run the "ping" module on the first host, it will fail because i did not provide any private key but it will cleanup artifacts and will leave only 20 of them. Solved :)

On Thu, Oct 7, 2021 at 11:49 AM Erez Zarum <erezz@nanosek.com> wrote:
Thank you all. Answering myself.
cd /usr/share/ovirt-engine/ansible-runner-service-project ansible-runner --rotate-artifacts 20 --artifact-dir /usr/share/ovirt-engine/ansible-runner-service-project/artifacts -m ping --hosts all[0] run /usr/share/ovirt-engine/ansible-runner-service-project
This will run the "ping" module on the first host, it will fail because i did not provide any private key but it will cleanup artifacts and will leave only 20 of them.
Solved :)
Thanks for the report! Martin, IMO these artifacts should go somewhere under /var . Can't find any EL-specific policy for this right now, but see e.g.: https://refspecs.linuxfoundation.org/FHS_2.3/fhs-2.3.html#THEUSRHIERARCHY Thanks and best regards, -- Didi

I second that, i believe most of questions here regarding expanding self hosted engine disk space is because of those artifacts filling up the disk space. the "/var" is quite empty but also has most of the disk space percentage divided into it so i believe this will be the best solution. Thanks!
participants (5)
-
Erez Zarum
-
Martin Necas
-
Martin Perina
-
Ritesh Chikatwar
-
Yedidyah Bar David