[Users] VSDM´s logrotate makes Hosts fill up var eventually

[Dan Kenigsberg]

On Thu, Jan 09, 2014 at 01:39:08PM +0000, Sven Kieske wrote:
> Hi,
> 
> I also guess this gets so large because of the loglevels in
> /etc/vdsm/logger.conf
> 
> this seems to be the default:
> 
> [logger_root]
> level=DEBUG
> handlers=syslog,logfile
> propagate=0
> 
> [logger_vds]
> level=DEBUG
> handlers=syslog,logfile
> qualname=vds
> propagate=0
> 
> [logger_Storage]
> level=DEBUG
> handlers=logfile
> qualname=Storage
> propagate=0
> 
> [logger_metadata]
> level=WARNING
> handlers=metadata
> qualname=irs.metadata
> propagate=0
> 
> [handler_syslog]
> level=WARNING
> class=handlers.SysLogHandler
> formatter=sysform
> args=('/dev/log', handlers.SysLogHandler.LOG_USER)
> 
> [handler_logfile]
> class=logging.handlers.WatchedFileHandler
> args=('/var/log/vdsm/vdsm.log',)
> filters=storage.misc.TracebackRepeatFilter
> level=DEBUG
> formatter=long
> 
> [handler_metadata]
> class=logging.handlers.WatchedFileHandler
> args=('/var/log/vdsm/metadata.log',)
> level=WARNING
> formatter=long
> 
> 
> which is "debug" level for most loggers.
> 
> Question to the devs:
> 
> Is this really needed as a default in a production
> environment?
> 
> my vdsm is a little bit older btw:
> 
> vdsm-4.12.1-4.el6.x86_64
> vdsm-cli-4.12.1-4.el6.noarch
> vdsm-python-4.12.1-4.el6.x86_64
> vdsm-python-cpopen-4.12.1-4.el6.x86_64
> vdsm-xmlrpc-4.12.1-4.el6.noarch
> 
> did this change in vdsm 4.13. ?

No change yet.

> 
> Am 09.01.2014 14:26, schrieb Karli Sjöberg:
> > Hi!
> > 
> > I just noticed my Hypervisor nodes starting to complain about disks
> > almost being full. I started investigation and noticed that:
> > # du -h /var/log/libvirtd.log
> > 100G	/var/log/libvirtd.log
> > 
> > And many Hosts system partition had indeed become full:S
> > 
> > Why weren´t the file rotated? Well:
> > # ls -lah /var/log/libvirtd.log.* | wc -l
> > 100
> > 
> > And the rotate policy says:
> > /etc/logrotate.d/libvirtd
> > ## beginning of configuration section by vdsm
> > /var/log/libvirt/libvirtd.log {
> >     rotate 100
> >     missingok
> >     copytruncate
> >     size 15M
> >     compress
> >     compresscmd /usr/bin/xz
> >     uncompresscmd /usr/bin/unxz
> >     compressext .xz
> > }
> > 
> > Now, I just handled it by changing "100" to "1000" but I think that a

I do not understand this issue, Karli. After 100 log files have been
created, the oldest one should have been removed and replaced by the
newest one. logrotate is expected to be called every 15 minutes, so it
should not have stayed above 15M for so long. Do you see any error when
running `/usr/sbin/logrotate /etc/logrotate.d/libvirtd` as root?

> > better default needs to be placed in general, don´t you?
> > # rpm -qa | grep vdsm
> > vdsm-python-4.13.0-11.el6.x86_64
> > vdsm-python-cpopen-4.13.0-11.el6.x86_64
> > vdsm-4.13.0-11.el6.x86_64
> > vdsm-xmlrpc-4.13.0-11.el6.noarch
> > vdsm-cli-4.13.0-11.el6.noarch

The question of how much logging we should keep is a tough one. I, as a
developer, would like to have as much as possible. For long-running busy
systems, it has happened to me that the core bug was spotted in
vdsm.log.67 or so.

However, I understand that verbosity has its price. To understand
whether we are stable enough to change the defaults, I need volunteers:
people who are willing to change their log level to INFO or WARNING, and
see if they miss useful information from their logs.

When you make you log level higher, you can lower the number of kept
log files, as they would not be filled as quick.

Would you, users@, help me with hard data?

Dan.