Re: broker.log not rotating
by Strahil Nikolov
I would recommend you to try to logrotate it.
I had similar issue with corrupted logrotate state file which led to vdsm growing to 20GB
You can also use 'truncate -s 0 your.log' to wipe without removing it.
Best Regards,
Strahil Nikolov
На 2 юли 2020 г. 8:00:27 GMT+03:00, Anton Louw via Users <users(a)ovirt.org> написа:
>Hi All,
>
>I had a space alert on one of my nodes this morning, and when I looked
>around, I saw that var/log/ovirt-hosted-engine-ha/broker.log was
>sitting at around 30GB. Does anybody know if it is safe to delete the
>log file? Or is there another process that I should follow?
>
>I had a look at my other nodes, and the broker.log file does not exceed
>4GB.
>
>Thank you
>
>
>Anton Louw
>Cloud Engineer: Storage and Virtualization
>______________________________________
>D: 087 805 1572 | M: N/A
>A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
>anton.louw(a)voxtelecom.co.za
>
>www.vox.co.za
4 years, 12 months
Re: broker.log not rotating
by Sandro Bonazzola
Il giorno gio 2 lug 2020 alle ore 07:05 Anton Louw via Users <
users(a)ovirt.org> ha scritto:
>
>
> Hi All,
>
>
>
> I had a space alert on one of my nodes this morning, and when I looked
> around, I saw that var/log/ovirt-hosted-engine-ha/broker.log was sitting
> at around 30GB. Does anybody know if it is safe to delete the log file? Or
> is there another process that I should follow?
>
>
>
> I had a look at my other nodes, and the broker.log file does not exceed
> 4GB.
>
Hi, broker logs should be rotating daily keeping 7 days in total unless you
manually changed the logging configuration at broker-log.conf.
If on this host it's growing so fast there's probably something broken on
the host.
Can you please inspect the log files and check what's got logged causing so
many log entries?
>
>
> Thank you
>
> *Anton Louw*
> *Cloud Engineer: Storage and Virtualization* at *Vox*
> ------------------------------
> *T:* 087 805 0000 | *D:* 087 805 1572
> *M:* N/A
> *E:* anton.louw(a)voxtelecom.co.za
> *A:* Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
> www.vox.co.za
>
> [image: F] <https://www.facebook.com/voxtelecomZA>
> [image: T] <https://www.twitter.com/voxtelecom>
> [image: I] <https://www.instagram.com/voxtelecomza/>
> [image: L] <https://www.linkedin.com/company/voxtelecom>
> [image: Y] <https://www.youtube.com/user/VoxTelecom>
>
> [image: #VoxBrand]
> <https://www.vox.co.za/fibre/fibre-to-the-home/?prod=HOME>
> *Disclaimer*
>
> The contents of this email are confidential to the sender and the intended
> recipient. Unless the contents are clearly and entirely of a personal
> nature, they are subject to copyright in favour of the holding company of
> the Vox group of companies. Any recipient who receives this email in error
> should immediately report the error to the sender and permanently delete
> this email from all storage devices.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> <https://www.voxtelecom.co.za/security/mimecast/?prod=Enterprise>.
>
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/4L6OOVVW6KO...
>
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
4 years, 12 months
Re: ETL service aggregation error
by Shirly Radco
Hi,
I was unable to view your log file. But if its indeed an the issue please
run:
systemctl stop ovirt-engine-dwhd
ovirt_engine_history=# UPDATE history_configuration set var_datetime =
date_trunc('hour', now())- interval '2 hour' WHERE var_name =
'lastHourAggr';
ovirt_engine_history=# UPDATE history_configuration set var_datetime =
date_trunc('hour', now())- interval '1 day' WHERE var_name = 'lastDayAggr';
systemctl restart ovirt-engine-dwhd
If you would like me to review the errors please attach the dwh log file.
Best regards,
--
Shirly Radco
BI Principal Software Engineer
Red Hat <https://www.redhat.com/>
<https://www.redhat.com/>
On Fri, Jun 26, 2020 at 10:28 PM Staniforth, Paul <
P.Staniforth(a)leedsbeckett.ac.uk> wrote:
> Hello Ayansh,
> It looks like the lastHourAgg is wrong is should
> be 2 hours before the runTime but is 1:30 hours before runtime.
>
> Try
>
> systemctl stop ovirt-engine-dwhd
>
> in ovirt_engine_history database
>
> UPDATE history_configuration set var_datetime = var_datetime - interval '24 hour' WHERE var_name = 'lastHourAggr';
>
> UPDATE history_configuration set var_datetime = var_datetime - interval '1 day' WHERE var_name = 'lastDayAggr';
>
> systemctl stop ovirt-engine-dwhd
>
>
> see https://www.mail-archive.com/users@ovirt.org/msg53055.html
>
>
> Regards,
>
> Paul Staniforth
>
> School of Built Environment Engineering and Computing.
>
> Leeds Beckett University
>
> Networked Systems Analyst, Research and Engineering Support.
> Technical Lead Architect IMS and oVirt Virtualization
>
> tel: +44 (0)113 28123754
> email: p.staniforth(a)leedsbeckett.ac.uk
>
> <https://www.gofundme.com/f/bytes-for-heroes>
>
> ------------------------------
> *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com>
> *Sent:* 26 June 2020 17:10
> *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users <
> users(a)ovirt.org>; devel(a)ovirt.org <devel(a)ovirt.org>
> *Subject:* Re: [ovirt-users] Re: ETL service aggregation error
>
>
> *Caution External Mail:* Do not click any links or open any attachments
> unless you trust the sender and know that the content is safe.
> Can we fix this any how guys.
>
> On Thu, Jun 25, 2020 at 5:53 PM Ayansh Rocks <shashank123rastogi(a)gmail.com>
> wrote:
>
> Is it a bug ?
>
>
> On Thu, Jun 25, 2020 at 4:11 PM Ayansh Rocks <shashank123rastogi(a)gmail.com>
> wrote:
>
> See below as well if it helps -
>
> [root@iondelvm149 ~]# date
> Thu Jun 25 16:09:45 IST 2020
> [root@iondelvm149 ~]# psql -h localhost -U ovirt_engine_history -d
> ovirt_engine_history
> Password for user ovirt_engine_history:
> psql (9.2.24, server 10.6)
> WARNING: psql version 9.2, server version 10.0.
> Some psql features might not work.
> Type "help" for help.
>
> ovirt_engine_history=> select * from history_configuration;
> var_name | var_value | var_datetime
> -------------------+-----------+------------------------
> default_language | en_US |
> firstSync | false | 2018-03-23 12:27:00-04
> lastHourAggr | | 2020-06-25 05:00:00-04
> HourlyAggFailed | false |
> lastDayAggr | | 2020-06-24 01:00:00-04
> MinimalETLVersion | 4.3.0 |
> (6 rows)
>
> ovirt_engine_history=>
>
> On Thu, Jun 25, 2020 at 4:06 PM Staniforth, Paul <
> P.Staniforth(a)leedsbeckett.ac.uk> wrote:
>
> Hello Shashank,
> is it on the hour? it sounds like it got a
> problem aggregating the samples to create the hourly samples.
>
> I had to update the java memory usage as it was running out of heap memory.
>
> The defaults are
>
> DWH_HEAP_MIN=1g
> DWH_HEAP_MAX=1g
>
> see
>
> /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.conf
>
> for settings.
>
> Regards,
> Paul S.
>
>
> ------------------------------
> *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com>
> *Sent:* 25 June 2020 10:40
> *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users <
> users(a)ovirt.org>
> *Subject:* Re: [ovirt-users] Re: ETL service aggregation error
>
>
> *Caution External Mail:* Do not click any links or open any attachments
> unless you trust the sender and know that the content is safe.
> Hi Paul,
>
> Yes i am able to connect to the database from ovirt engine machine.
>
> port is already open. what could be issue, i am getting this error in
> every hour.
>
> Thanks
>
> On Wed, Jun 24, 2020 at 6:30 PM Staniforth, Paul <
> P.Staniforth(a)leedsbeckett.ac.uk> wrote:
>
> Hello Shashank,
> it looks like it's had a problem for over 2
> years, is the Data Warehouse database local or remote? is there a firewall
> port open?
>
> Can you connect to the database from the engine machine?
>
> The credentials should be in
> /etc/ovirt-engine-dwh/ovirt-engine-dwhd.conf.d/10-setup-database.conf
>
> as your using 4.3 the version of postgresql is 10 and using scl
>
> scl enable rh-postgresql10 "psql -h localhost -U ovirt_engine_history
> -d ovirt_engine_history"
>
> if it's using the standard user, database and is local.
>
>
> Regards,
> Paul S.
> .
>
> ------------------------------
> *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com>
> *Sent:* 24 June 2020 12:47
> *To:* Staniforth, Paul <P.Staniforth(a)leedsbeckett.ac.uk>; users <
> users(a)ovirt.org>
> *Subject:* Re: [ovirt-users] Re: ETL service aggregation error
>
>
> *Caution External Mail:* Do not click any links or open any attachments
> unless you trust the sender and know that the content is safe.
> Please find the attached error logs.
>
> On Thu, Jun 4, 2020 at 8:17 PM Staniforth, Paul <
> P.Staniforth(a)leedsbeckett.ac.uk> wrote:
>
> Hello Shashank,
> I can't see any of your images and also it
> would be better to have the log file as text.
>
> Regards,
> Paul S.
>
>
>
> ------------------------------
> *From:* Ayansh Rocks <shashank123rastogi(a)gmail.com>
> *Sent:* 04 June 2020 15:16
> *To:* users <users(a)ovirt.org>
> *Subject:* [ovirt-users] Re: ETL service aggregation error
>
>
> *Caution External Mail:* Do not click any links or open any attachments
> unless you trust the sender and know that the content is safe.
> Any update on this ?
>
> On Tue, May 26, 2020 at 1:41 PM Ayansh Rocks <shashank123rastogi(a)gmail.com>
> wrote:
>
> Hi,
>
> I am using 4.3.7 self hosted engine. From Few days i am getting regular
> below error messages :-
> [image: image.png]
>
> Logs in /var/log/ovirt-engine-dwh/ovirt-engine-dwhd.log
> [image: image.png]
>
> What could be the reason for this?
>
> Thanks
> Shashank
>
> To view the terms under which this email is distributed, please go to:-
> http://leedsbeckett.ac.uk/disclaimer/email/
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbec...>
>
> To view the terms under which this email is distributed, please go to:-
> http://leedsbeckett.ac.uk/disclaimer/email/
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbec...>
>
> To view the terms under which this email is distributed, please go to:-
> http://leedsbeckett.ac.uk/disclaimer/email/
> <https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fleedsbec...>
>
> To view the terms under which this email is distributed, please go to:-
> http://leedsbeckett.ac.uk/disclaimer/email/
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BCU7ZTSVHZR...
>
5 years
broker.log not rotating
by Anton Louw
Hi All,
I had a space alert on one of my nodes this morning, and when I looked around, I saw that var/log/ovirt-hosted-engine-ha/broker.log was sitting at around 30GB. Does anybody know if it is safe to delete the log file? Or is there another process that I should follow?
I had a look at my other nodes, and the broker.log file does not exceed 4GB.
Thank you
Anton Louw
Cloud Engineer: Storage and Virtualization
______________________________________
D: 087 805 1572 | M: N/A
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
anton.louw(a)voxtelecom.co.za
www.vox.co.za
5 years
VMs shutdown mysteriously
by Bobby
Hello,
All 4 VMs on one of my oVirt cluster node shutdown for an unknown reason
almost simultaneously.
Please help me to find the root cause.
Thanks.
Please note the host seems doing fine and never crash or hangs and I can
migrate VMs back to it later.
Here is the exact timeline of all the related events combined from the host
and the VM(s):
On oVirt host:
/var/log/vdsm/vdsm.log:
2020-06-25 15:25:16,944-0500 WARN (qgapoller/3)
[virt.periodic.VmDispatcher] could not run <function <lambda> at
0x7f4ed2f9f5f0> on ['e0257b06-28fd-4d41-83a9-adf1904d3622'] (periodic:289)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.ovirt-guest-agent.0
already removed (fileutils:54)
2020-06-25 15:25:19,203-0500 WARN (libvirt/events) [root] File:
/var/lib/libvirt/qemu/channels/e0257b06-28fd-4d41-83a9-adf1904d3622.org.qemu.guest_agent.0
already removed (fileutils:54)
[root@athos log]# journalctl -u NetworkManager --since=today
-- Logs begin at Wed 2020-05-20 22:07:33 CDT, end at Thu 2020-06-25
16:36:05 CDT. --
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
/var/log/messages:
Jun 25 15:25:18 athos kernel: SRV-VL: port 2(vnet0) entered disabled state
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1136]
device (vnet0): state change: disconnected -> unmanaged (reason
'unmanaged', sys-iface-state: 'removed')
Jun 25 15:25:18 athos NetworkManager[1600]: <info> [1593116718.1146]
device (vnet0): released from master device SRV-VL
Jun 25 15:25:18 athos libvirtd: 2020-06-25 20:25:18.122+0000: 2713: error :
qemuMonitorIO:718 : internal error: End of file from qemu monitor
/var/log/libvirt/qemu/aries.log:
2020-06-25T20:25:28.353975Z qemu-kvm: terminating on signal 15 from pid
2713 (/usr/sbin/libvirtd)
2020-06-25 20:25:28.584+0000: shutting down, reason=shutdown
=============================================================================================
On the first VM effected (same thing on others):
/var/log/ovirt-guest-agent/ovirt-guest-agent.log:
MainThread::INFO::2020-06-25
15:25:20,270::ovirt-guest-agent::104::root::Stopping oVirt guest agent
CredServer::INFO::2020-06-25
15:25:20,626::CredServer::262::root::CredServer has stopped.
MainThread::INFO::2020-06-25
15:25:21,150::ovirt-guest-agent::78::root::oVirt guest agent is down.
=============================================================================================
Packages version installated:
Host OS version: CentOS 7.7.1908:
ovirt-hosted-engine-ha-2.3.5-1.el7.noarch
ovirt-provider-ovn-driver-1.2.22-1.el7.noarch
ovirt-release43-4.3.6-1.el7.noarch
ovirt-imageio-daemon-1.5.2-0.el7.noarch
ovirt-vmconsole-1.0.7-2.el7.noarch
ovirt-imageio-common-1.5.2-0.el7.x86_64
ovirt-engine-sdk-python-3.6.9.1-1.el7.noarch
ovirt-vmconsole-host-1.0.7-2.el7.noarch
ovirt-host-4.3.4-1.el7.x86_64
libvirt-4.5.0-23.el7_7.1.x86_64
libvirt-daemon-4.5.0-23.el7_7.1.x86_6
qemu-kvm-ev-2.12.0-33.1.el7.x86_64
qemu-kvm-common-ev-2.12.0-33.1.el7.x86_64
On guest VM:
ovirt-guest-agent-1.0.13-1.el6.noarch
qemu-guest-agent-0.12.1.2-2.491.el6_8.3.x86_64
5 years
Re: Some nodes periodically display as Non Responsive
by Martin Perina
Hi Anton,
to diagnose the issue we would need to have logs from both engine and
affected host.
Regards,
Martin
On Wed, Jul 1, 2020 at 6:51 AM Anton Louw via Users <users(a)ovirt.org> wrote:
>
>
> Hi Everybody,
>
>
>
> I am got some strange things happening. I have got two data centers, DC1
> and DC2, in DC1, some of my nodes (Not all the time and not all the nodes)
> go into a “not responding” state. I can still ping the hosts, and I can
> still access the VMs on the hosts. My Engine sits in DC2, and this does not
> happen to any of the hosts in DC2.
>
>
>
> It seems like the Engine loses connectivity to the hosts in DC1, and then
> cannot re-establish the connection.
>
>
>
> Is there anywhere I can check to get more insight into what is actually
> happening?
>
>
>
> Thanks
>
>
>
> *Anton Louw*
> *Cloud Engineer: Storage and Virtualization* at *Vox*
> ------------------------------
> *T:* 087 805 0000 | *D:* 087 805 1572
> *M:* N/A
> *E:* anton.louw(a)voxtelecom.co.za
> *A:* Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
> www.vox.co.za
>
> [image: F] <https://www.facebook.com/voxtelecomZA>
> [image: T] <https://www.twitter.com/voxtelecom>
> [image: I] <https://www.instagram.com/voxtelecomza/>
> [image: L] <https://www.linkedin.com/company/voxtelecom>
> [image: Y] <https://www.youtube.com/user/VoxTelecom>
>
> [image: #VoxBrand]
> <https://www.vox.co.za/fibre/fibre-to-the-home/?prod=HOME>
> *Disclaimer*
>
> The contents of this email are confidential to the sender and the intended
> recipient. Unless the contents are clearly and entirely of a personal
> nature, they are subject to copyright in favour of the holding company of
> the Vox group of companies. Any recipient who receives this email in error
> should immediately report the error to the sender and permanently delete
> this email from all storage devices.
>
> This email has been scanned for viruses and malware, and may have been
> automatically archived by *Mimecast Ltd*, an innovator in Software as a
> Service (SaaS) for business. Providing a *safer* and *more useful* place
> for your human generated data. Specializing in; Security, archiving and
> compliance. To find out more Click Here
> <https://www.voxtelecom.co.za/security/mimecast/?prod=Enterprise>.
>
>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/TRMWV4Q6AFH...
>
--
Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.
5 years
Some nodes periodically display as Non Responsive
by Anton Louw
Hi Everybody,
I am got some strange things happening. I have got two data centers, DC1 and DC2, in DC1, some of my nodes (Not all the time and not all the nodes) go into a "not responding" state. I can still ping the hosts, and I can still access the VMs on the hosts. My Engine sits in DC2, and this does not happen to any of the hosts in DC2.
It seems like the Engine loses connectivity to the hosts in DC1, and then cannot re-establish the connection.
Is there anywhere I can check to get more insight into what is actually happening?
Thanks
Anton Louw
Cloud Engineer: Storage and Virtualization
______________________________________
D: 087 805 1572 | M: N/A
A: Rutherford Estate, 1 Scott Street, Waverley, Johannesburg
anton.louw(a)voxtelecom.co.za
www.vox.co.za
5 years
