[Users] VSDM´s logrotate makes Hosts fill up var eventually

Thu Jan 9 12:01:12 EST 2014

________________________________________
Från: users-bounces at ovirt.org [users-bounces at ovirt.org] f&#246;r Karli Sjöberg [Karli.Sjoberg at slu.se]
Skickat: den 9 januari 2014 17:53
Till: Dan Kenigsberg; Sven Kieske
Kopia: users at ovirt.org
Ämne: Re: [Users] VSDM´s logrotate makes Hosts fill up var eventually

________________________________________
Från: users-bounces at ovirt.org [users-bounces at ovirt.org] f&#246;r Dan Kenigsberg [danken at redhat.com]
Skickat: den 9 januari 2014 17:33
Till: Sven Kieske
Kopia: users at ovirt.org
Ämne: Re: [Users] VSDM´s logrotate makes Hosts fill up var eventually

On Thu, Jan 09, 2014 at 01:39:08PM +0000, Sven Kieske wrote:
> Hi,
>
> I also guess this gets so large because of the loglevels in
> /etc/vdsm/logger.conf
>
> this seems to be the default:
>
> [logger_root]
> level=DEBUG
> handlers=syslog,logfile
> propagate=0
>
> [logger_vds]
> level=DEBUG
> handlers=syslog,logfile
> qualname=vds
> propagate=0
>
> [logger_Storage]
> level=DEBUG
> handlers=logfile
> qualname=Storage
> propagate=0
>
> [logger_metadata]
> level=WARNING
> handlers=metadata
> qualname=irs.metadata
> propagate=0
>
> [handler_syslog]
> level=WARNING
> class=handlers.SysLogHandler
> formatter=sysform
> args=('/dev/log', handlers.SysLogHandler.LOG_USER)
>
> [handler_logfile]
> class=logging.handlers.WatchedFileHandler
> args=('/var/log/vdsm/vdsm.log',)
> filters=storage.misc.TracebackRepeatFilter
> level=DEBUG
> formatter=long
>
> [handler_metadata]
> class=logging.handlers.WatchedFileHandler
> args=('/var/log/vdsm/metadata.log',)
> level=WARNING
> formatter=long
>
>
> which is "debug" level for most loggers.
>
> Question to the devs:
>
> Is this really needed as a default in a production
> environment?
>
> my vdsm is a little bit older btw:
>
> vdsm-4.12.1-4.el6.x86_64
> vdsm-cli-4.12.1-4.el6.noarch
> vdsm-python-4.12.1-4.el6.x86_64
> vdsm-python-cpopen-4.12.1-4.el6.x86_64
> vdsm-xmlrpc-4.12.1-4.el6.noarch
>
> did this change in vdsm 4.13. ?

No change yet.

>
> Am 09.01.2014 14:26, schrieb Karli Sjöberg:
> > Hi!
> >
> > I just noticed my Hypervisor nodes starting to complain about disks
> > almost being full. I started investigation and noticed that:
> > # du -h /var/log/libvirtd.log
> > 100G        /var/log/libvirtd.log
> >
> > And many Hosts system partition had indeed become full:S
> >
> > Why weren´t the file rotated? Well:
> > # ls -lah /var/log/libvirtd.log.* | wc -l
> > 100
> >
> > And the rotate policy says:
> > /etc/logrotate.d/libvirtd
> > ## beginning of configuration section by vdsm
> > /var/log/libvirt/libvirtd.log {
> >     rotate 100
> >     missingok
> >     copytruncate
> >     size 15M
> >     compress
> >     compresscmd /usr/bin/xz
> >     uncompresscmd /usr/bin/unxz
> >     compressext .xz
> > }
> >
> > Now, I just handled it by changing "100" to "1000" but I think that a

I do not understand this issue, Karli. After 100 log files have been
created, the oldest one should have been removed and replaced by the
newest one. logrotate is expected to be called every 15 minutes, so it
should not have stayed above 15M for so long. Do you see any error when
running `/usr/sbin/logrotate /etc/logrotate.d/libvirtd` as root?

What the funk, it doesn´t do anything, complains nothing like it´s done what it´s supposed to, and no errors in "/var/log/messages" either...

Oh, and here´s the output:
# du -h /var/log/libvirtd.log
1.1G	/var/log/libvirtd.log
# /usr/sbin/logrotate -d -v /etc/logrotate.d/libvirtd
reading config file /etc/logrotate.d/libvirtd
reading config info for /var/log/libvirt/libvirtd.log 
compress_prog is now /usr/bin/xz
uncompress_prog is now /usr/bin/unxz
compress_ext is now .xz

Handling 1 logs

rotating pattern: /var/log/libvirt/libvirtd.log  15728640 bytes (1000 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/libvirt/libvirtd.log
  log does not need rotating

Really, are you sure about that?

/K

> > better default needs to be placed in general, don´t you?
> > # rpm -qa | grep vdsm
> > vdsm-python-4.13.0-11.el6.x86_64
> > vdsm-python-cpopen-4.13.0-11.el6.x86_64
> > vdsm-4.13.0-11.el6.x86_64
> > vdsm-xmlrpc-4.13.0-11.el6.noarch
> > vdsm-cli-4.13.0-11.el6.noarch

The question of how much logging we should keep is a tough one. I, as a
developer, would like to have as much as possible. For long-running busy
systems, it has happened to me that the core bug was spotted in
vdsm.log.67 or so.

However, I understand that verbosity has its price. To understand
whether we are stable enough to change the defaults, I need volunteers:
people who are willing to change their log level to INFO or WARNING, and
see if they miss useful information from their logs.

When you make you log level higher, you can lower the number of kept
log files, as they would not be filled as quick.

Would you, users@, help me with hard data?

Well, I understand that having more data of course helps when there´s an issue, however having lots of data isn´t helping in this particular case;P
Seriously though, it´s not bothering me having lots of logs. I don´t mind the DEBUG mode, and as long as rotate _does what it´s supposed to_ (but that´s besides the point) it´s not taking up that much space to argue about either so for me it´s cool (normally).

Dan.
_______________________________________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
_______________________________________________
Users mailing list
Users at ovirt.org
http://lists.ovirt.org/mailman/listinfo/users