[Users] VSDM´s logrotate makes Hosts fill up var eventually

[Karli Sjöberg]

I´m resending an email sent yersterday that was probably overlooked due
to a lack of indentation.

On Thu, 2014-01-09 at 16:33 +0000, Dan Kenigsberg wrote:
> On Thu, Jan 09, 2014 at 01:39:08PM +0000, Sven Kieske wrote:
> > Hi,
> > 
> > I also guess this gets so large because of the loglevels in
> > /etc/vdsm/logger.conf
> > 
> > this seems to be the default:
> > 
> > [logger_root]
> > level=DEBUG
> > handlers=syslog,logfile
> > propagate=0
> > 
> > [logger_vds]
> > level=DEBUG
> > handlers=syslog,logfile
> > qualname=vds
> > propagate=0
> > 
> > [logger_Storage]
> > level=DEBUG
> > handlers=logfile
> > qualname=Storage
> > propagate=0
> > 
> > [logger_metadata]
> > level=WARNING
> > handlers=metadata
> > qualname=irs.metadata
> > propagate=0
> > 
> > [handler_syslog]
> > level=WARNING
> > class=handlers.SysLogHandler
> > formatter=sysform
> > args=('/dev/log', handlers.SysLogHandler.LOG_USER)
> > 
> > [handler_logfile]
> > class=logging.handlers.WatchedFileHandler
> > args=('/var/log/vdsm/vdsm.log',)
> > filters=storage.misc.TracebackRepeatFilter
> > level=DEBUG
> > formatter=long
> > 
> > [handler_metadata]
> > class=logging.handlers.WatchedFileHandler
> > args=('/var/log/vdsm/metadata.log',)
> > level=WARNING
> > formatter=long
> > 
> > 
> > which is "debug" level for most loggers.
> > 
> > Question to the devs:
> > 
> > Is this really needed as a default in a production
> > environment?
> > 
> > my vdsm is a little bit older btw:
> > 
> > vdsm-4.12.1-4.el6.x86_64
> > vdsm-cli-4.12.1-4.el6.noarch
> > vdsm-python-4.12.1-4.el6.x86_64
> > vdsm-python-cpopen-4.12.1-4.el6.x86_64
> > vdsm-xmlrpc-4.12.1-4.el6.noarch
> > 
> > did this change in vdsm 4.13. ?
> 
> No change yet.
> 
> > 
> > Am 09.01.2014 14:26, schrieb Karli Sjöberg:
> > > Hi!
> > > 
> > > I just noticed my Hypervisor nodes starting to complain about disks
> > > almost being full. I started investigation and noticed that:
> > > # du -h /var/log/libvirtd.log
> > > 100G	/var/log/libvirtd.log
> > > 
> > > And many Hosts system partition had indeed become full:S
> > > 
> > > Why weren´t the file rotated? Well:
> > > # ls -lah /var/log/libvirtd.log.* | wc -l
> > > 100
> > > 
> > > And the rotate policy says:
> > > /etc/logrotate.d/libvirtd
> > > ## beginning of configuration section by vdsm
> > > /var/log/libvirt/libvirtd.log {
> > >     rotate 100
> > >     missingok
> > >     copytruncate
> > >     size 15M
> > >     compress
> > >     compresscmd /usr/bin/xz
> > >     uncompresscmd /usr/bin/unxz
> > >     compressext .xz
> > > }
> > > 
> > > Now, I just handled it by changing "100" to "1000" but I think that a
> 
> I do not understand this issue, Karli. After 100 log files have been
> created, the oldest one should have been removed and replaced by the
> newest one. logrotate is expected to be called every 15 minutes, so it
> should not have stayed above 15M for so long. Do you see any error when
> running `/usr/sbin/logrotate /etc/logrotate.d/libvirtd` as root?

What the funk, it doesn´t do anything, complains nothing like it´s done
what it´s supposed to, and no errors in "/var/log/messages" either...

Oh, and here´s the output:
# du -h /var/log/libvirtd.log
1.1G    /var/log/libvirtd.log
# /usr/sbin/logrotate -d -v /etc/logrotate.d/libvirtd
reading config file /etc/logrotate.d/libvirtd
reading config info for /var/log/libvirt/libvirtd.log 
compress_prog is now /usr/bin/xz
uncompress_prog is now /usr/bin/unxz
compress_ext is now .xz

Handling 1 logs

rotating pattern: /var/log/libvirt/libvirtd.log  15728640 bytes (1000
rotations)
empty log files are rotated, old logs are removed
considering log /var/log/libvirt/libvirtd.log
  log does not need rotating

Really, are you sure about that?

> 
> > > better default needs to be placed in general, don´t you?
> > > # rpm -qa | grep vdsm
> > > vdsm-python-4.13.0-11.el6.x86_64
> > > vdsm-python-cpopen-4.13.0-11.el6.x86_64
> > > vdsm-4.13.0-11.el6.x86_64
> > > vdsm-xmlrpc-4.13.0-11.el6.noarch
> > > vdsm-cli-4.13.0-11.el6.noarch
> 
> The question of how much logging we should keep is a tough one. I, as a
> developer, would like to have as much as possible. For long-running busy
> systems, it has happened to me that the core bug was spotted in
> vdsm.log.67 or so.
> 
> However, I understand that verbosity has its price. To understand
> whether we are stable enough to change the defaults, I need volunteers:
> people who are willing to change their log level to INFO or WARNING, and
> see if they miss useful information from their logs.
> 
> When you make you log level higher, you can lower the number of kept
> log files, as they would not be filled as quick.
> 
> Would you, users@, help me with hard data?

Well, I understand that having more data of course helps when there´s an
issue, however having lots of data isn´t exactly helping in this
particular case;P

Seriously though, it´s not bothering me having lots of logs. I don´t
mind the DEBUG mode, and as long as rotate _does what it´s supposed to_
it´s not taking up that much space to argue about either so for me it´s
cool (normally).

/K

> 
> Dan.
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users



-- 

Med Vänliga Hälsningar

-------------------------------------------------------------------------------
Karli Sjöberg
Swedish University of Agricultural Sciences Box 7079 (Visiting Address
Kronåsvägen 8)
S-750 07 Uppsala, Sweden
Phone:  +46-(0)18-67 15 66
karli.sjoberg at slu.se