[node-devel] Logrotate seem broken

[Mike Burns]

On 04/25/2013 01:35 PM, Jonas Israelsson wrote:
> On 25/04/13 17:52, Mike Burns wrote:
>> Can you provide the output of the following:
>>
>> du -sh /var/log
>> df -h /var/log
>>
> I sure can, I'll even give you two..
>
> Bellow you find the output with about an hour between, during this the
> the log should have been rotated four times, right ?
> The /var/log/libvirtd.log grows about a MB/min
>
>
> # First -- Thu Apr 25 16:33:36 UTC 2013
>
> [root at fat-tony admin]# df -h /var/log/
> Filesystem                  Size  Used Avail Use% Mounted on
> /dev/mapper/HostVG-Logging  2.0G  374M  1.6G  20% /var/log
>
> [root at fat-tony admin]# du -sh /var/log/
> 168M /var/log/
> [root at fat-tony admin]#
>
>
> # Second -- Thu Apr 25 17:25:45 UTC 2013
>
> [root at fat-tony admin]# df -h /var/log/
> Filesystem                  Size  Used Avail Use% Mounted on
> /dev/mapper/HostVG-Logging  2.0G  443M  1.5G  24% /var/log
>
> [root at fat-tony admin]# du -sh /var/log/
> 231M    /var/log/
>
>
>
> All rotated logs below are when running logrotate manually.
>
> [root at fat-tony admin]# ll /var/log
> -rw-------. 1 root root 144427492 Apr 25 17:30 libvirtd.log
> -rw-------. 1 root root   1177724 Apr 25 15:30 libvirtd.log.1.xz
> -rw-------. 1 root root    845740 Apr 25 15:02 libvirtd.log.2.xz
> -rw-------. 1 root root     81856 Apr 25 14:42 libvirtd.log.3.xz
> -rw-------. 1 root root     66488 Apr 25 14:40 libvirtd.log.4.xz

That's what I expected.  There is an issue with vdsm that the logs 
aren't being cleaned up after log rotate.

If you run lsof /var/log, you'll see something like:

python    20829      root    4r      REG             253,10    25471139 
        280 /var/log/vdsm/vdsm.log.1 (deleted)

This can be resolved by doing the following:

Put the host in maintenance mode (recommended, but possible to skip)
edit /etc/logrotate.d/vdsm to add copytrucate to first entry[1].
restart vdsmd (systemctl restart vdsmd)

Now, the logs shouldn't fill up.

Mike