On Thu, Nov 22, 2018, 13:43 Fabrice Bacchella <fabrice.bacchella@icloud.com wrote:

My vdsm log files are huge:

-rw-r--r-- 1 vdsm kvm 1.8G Nov 22 11:32 vdsm.log

And this is juste half an hour of logs:

$ head -1 vdsm.log
2018-11-22 11:01:12,132+0100 ERROR (mailbox-spm) [storage.MailBox.SpmMailMonitor] mailbox 2 checksum failed, not clearing mailbox, clearing new mail (data='...lots of data', expected='\xa4\x06\x08\x00') (mailbox:612)

Are you sure this is the log line? the error from line 612 should be:

mailbox %s checksum failed, not clearing mailbox, clearing new mail (data=%r, checksum=%r, expected=%r)

Please open a bug and attach the interesting part of the log - from starting the SPM

until the first error was seen, and some errors after that.

It would be useful if you cam share the compressed log somehow.

We also need the contents of the inbox and outbox:

For iSCSI/FC domain, the logical volumes

/dev/domain-uuid/{inbox,outbox}

For NFS/Gluster, the files:

/rhev/data-center/mnt/server:_path/domain-uuid/dom_md/{inbox,outbox}

You can copy them with dd, compress, and attach to the bug.

I just upgraded vdsm:
$ rpm -qi vdsm
Name : vdsm
Version : 4.20.43

This started after the upgrade?

As first aid fix you can clear the inbox and outbox files like this:

1. Stop vdsm on the SPM host

systemctl stop vdsmd

2. Clear the mailboxes

For iSCSI/FC:

blkdiscard -z /dev/domain-uuid/{inbox,outbox}

For NFS/Gluster:

dd if=/dev/zero of=/rhev/data-center/mnt/server:_path/domain-uuid/dom_md/inbox bs=1M count=16 oflag=direct conv=fsync

dd if=/dev/zero of=/rhev/data-center/mnt/server:_path/domain-uuid/dom_md/outbox bs=1M count=16 oflag=direct conv=fsync

3. Start vdsm on the SPM host

systemctl start vdsmd

Nir