<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 26, 2017 at 10:41 AM, Nir Soffer <span dir="ltr"><<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On Thu, Jan 26, 2017 at 9:46 AM, Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br>
> I'm seeing a failure in 4.1, which I've never seen in the past:<br>
> 2017-01-26 02:21:00,124 ERROR (mailbox-spm) [storage.MailBox.<wbr>SpmMailMonitor]<br>
> SPM_MailMonitor: mailbox 2 checksum failed, not clearing mailbox, clearing<br>
> newMail. (storage_mailbox:618)<br>
<br>
</span>I have seen this in the past, this is a very rare error.<br>
<br>
I'm not clear why this happens, if it has any effect or how we can avoid it.<br>
<br>
Best open a bug about it, we should investigate it.<br></blockquote><div><br></div><div>Not sure if its the same error, but we have open bug on the same test:</div><div><br></div><div><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1416386">https://bugzilla.redhat.com/show_bug.cgi?id=1416386</a><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-"><br>
> (from host1).<br>
><br>
> There are also some new warnings around monitoring, such as:<br>
> 2017-01-26 02:20:58,035 WARN (libvirt/events) [virt.vm]<br>
> (vmId='85e8df3c-8bdf-470f-<wbr>8c28-baa521d663de') unknown eventid 8 args<br>
> ('/rhev/data-center/27d461cd-<wbr>03b2-4b2b-a1f8-157c27bb710e/<wbr>2e10aa41-24f1-4c83-8e29-<wbr>15f772a2d04b/images/304c3f03-<wbr>61a6-4b79-a748-174a4cd24f84/<wbr>3f96ca05-e8f3-41f2-8b9c-<wbr>c7d170b24f92',<br>
> 3, 0, 8) (clientIF:540)<br>
<br>
</span>This means that libvirt sent an event that we are not handling, there is nothing<br>
to warn about.<br>
<br>
Since this event is a change in system state, having an info level log<br>
would be nice.<br>
<br>
The number 8 is not very helpful, we need to map the number to the event name<br>
as documented in libvirt.<br>
<a href="https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainEventID" rel="noreferrer" target="_blank">https://libvirt.org/html/<wbr>libvirt-libvirt-domain.html#<wbr>virDomainEventID</a><br>
<br>
Ala is working on handling this and other events during live merge for 4.2.<br>
<div class="gmail-HOEnZb"><div class="gmail-h5"><br>
> On Wed, Jan 25, 2017 at 2:17 PM, Gil Shinar <<a href="mailto:gshinar@redhat.com">gshinar@redhat.com</a>> wrote:<br>
>><br>
>> Link to job:<br>
>> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4951/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/4951/</a><br>
>><br>
>> Sorry for the mistake<br>
>><br>
>> On Wed, Jan 25, 2017 at 1:43 PM, Gil Shinar <<a href="mailto:gshinar@redhat.com">gshinar@redhat.com</a>> wrote:<br>
>>><br>
>>> Hi,<br>
>>><br>
>>> The test in $subject has failed. Below please find exception I have found<br>
>>> in engine.log.<br>
>>><br>
>>> {"jsonrpc": "2.0", "id": "4b8eb810-c52d-4c92-a792-<wbr>e554f87c9493", "error":<br>
>>> {"message": "Cannot deactivate Logical Volume: ('General Storage Exception:<br>
>>> (\"5 [] [\\' WARNING: Not using lvmetad because config setting<br>
>>> use_lvmetad=0.\\', \\' WARNING: To avoid corruption, rescan devices to make<br>
>>> changes visible (pvscan --cache).\\', \\' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/778cbc5b-a9df-<wbr>46d7-bc80-1a66f7d3e2b5 in<br>
>>> use.\\', \\' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/ab4e8962-6196-<wbr>485e-be2a-d5791a38eaeb in<br>
>>> use.\\']\\\\nf9dce023-0282-<wbr>4185-9ad9-fe71c3975106/[\\'<wbr>778cbc5b-a9df-46d7-bc80-<wbr>1a66f7d3e2b5\\',<br>
>>> \\'ab4e8962-6196-485e-be2a-<wbr>d5791a38eaeb\\']\",)',)", "code": 552}}�<br>
>>><br>
>>> 2017-01-25 05:05:49,037-05 DEBUG<br>
>>> [org.ovirt.vdsm.jsonrpc.<wbr>client.internal.<wbr>ResponseWorker] (ResponseWorker) []<br>
>>> Message received: {"jsonrpc": "2.0", "id":<br>
>>> "4b8eb810-c52d-4c92-a792-<wbr>e554f87c9493", "error": {"message": "Cannot<br>
>>> deactivate Logical Volume: ('General Storage Exception: (\"5 [] [\\'<br>
>>> WARNING: Not using lvmetad because config setting use_lvmetad=0.\\', \\'<br>
>>> WARNING: To avoid corruption, rescan devices to make changes visible (pvscan<br>
>>> --cache).\\', \\' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/778cbc5b-a9df-<wbr>46d7-bc80-1a66f7d3e2b5 in<br>
>>> use.\\', \\' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/ab4e8962-6196-<wbr>485e-be2a-d5791a38eaeb in<br>
>>> use.\\']\\\\nf9dce023-0282-<wbr>4185-9ad9-fe71c3975106/[\\'<wbr>778cbc5b-a9df-46d7-bc80-<wbr>1a66f7d3e2b5\\',<br>
>>> \\'ab4e8962-6196-485e-be2a-<wbr>d5791a38eaeb\\']\",)',)", "code": 552}}<br>
>>> 2017-01-25 05:05:49,047-05 ERROR<br>
>>> [org.ovirt.engine.core.dal.<wbr>dbbroker.auditloghandling.<wbr>AuditLogDirector]<br>
>>> (DefaultQuartzScheduler5) [59ab00f1] EVENT_ID:<br>
>>> VDS_BROKER_COMMAND_FAILURE(10,<wbr>802), Correlation ID: null, Call Stack: null,<br>
>>> Custom Event ID: -1, Message: VDSM lago-basic-suite-master-host0 command<br>
>>> TeardownImageVDS failed: Cannot deactivate Logical Volume: ('General Storage<br>
>>> Exception: ("5 [] [\' WARNING: Not using lvmetad because config setting<br>
>>> use_lvmetad=0.\', \' WARNING: To avoid corruption, rescan devices to make<br>
>>> changes visible (pvscan --cache).\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/778cbc5b-a9df-<wbr>46d7-bc80-1a66f7d3e2b5 in<br>
>>> use.\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/ab4e8962-6196-<wbr>485e-be2a-d5791a38eaeb in<br>
>>> use.\']\\nf9dce023-0282-4185-<wbr>9ad9-fe71c3975106/[\'778cbc5b-<wbr>a9df-46d7-bc80-1a66f7d3e2b5\',<br>
>>> \'ab4e8962-6196-485e-be2a-<wbr>d5791a38eaeb\']",)',)<br>
>>> 2017-01-25 05:05:49,047-05 ERROR<br>
>>> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>TeardownImageVDSCommand]<br>
>>> (DefaultQuartzScheduler5) [59ab00f1] Command<br>
>>> 'TeardownImageVDSCommand(<wbr>HostName = lago-basic-suite-master-host0,<br>
>>> ImageActionsVDSCommandParamete<wbr>rs:{runAsync='true',<br>
>>> hostId='60e52527-b637-445c-<wbr>b408-0275d347e76a'})' execution failed:<br>
>>> VDSGenericException: VDSErrorException: Failed in vdscommand to<br>
>>> TeardownImageVDS, error = Cannot deactivate Logical Volume: ('General<br>
>>> Storage Exception: ("5 [] [\' WARNING: Not using lvmetad because config<br>
>>> setting use_lvmetad=0.\', \' WARNING: To avoid corruption, rescan devices<br>
>>> to make changes visible (pvscan --cache).\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/778cbc5b-a9df-<wbr>46d7-bc80-1a66f7d3e2b5 in<br>
>>> use.\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/ab4e8962-6196-<wbr>485e-be2a-d5791a38eaeb in<br>
>>> use.\']\\nf9dce023-0282-4185-<wbr>9ad9-fe71c3975106/[\'778cbc5b-<wbr>a9df-46d7-bc80-1a66f7d3e2b5\',<br>
>>> \'ab4e8962-6196-485e-be2a-<wbr>d5791a38eaeb\']",)',)<br>
>>> 2017-01-25 05:05:49,047-05 DEBUG<br>
>>> [org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>TeardownImageVDSCommand]<br>
>>> (DefaultQuartzScheduler5) [59ab00f1] Exception:<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>VDSErrorException:<br>
>>> VDSGenericException: VDSErrorException: Failed in vdscommand to<br>
>>> TeardownImageVDS, error = Cannot deactivate Logical Volume: ('General<br>
>>> Storage Exception: ("5 [] [\' WARNING: Not using lvmetad because config<br>
>>> setting use_lvmetad=0.\', \' WARNING: To avoid corruption, rescan devices<br>
>>> to make changes visible (pvscan --cache).\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/778cbc5b-a9df-<wbr>46d7-bc80-1a66f7d3e2b5 in<br>
>>> use.\', \' Logical volume<br>
>>> f9dce023-0282-4185-9ad9-<wbr>fe71c3975106/ab4e8962-6196-<wbr>485e-be2a-d5791a38eaeb in<br>
>>> use.\']\\nf9dce023-0282-4185-<wbr>9ad9-fe71c3975106/[\'778cbc5b-<wbr>a9df-46d7-bc80-1a66f7d3e2b5\',<br>
>>> \'ab4e8962-6196-485e-be2a-<wbr>d5791a38eaeb\']",)',)<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>BrokerCommandBase.<wbr>proceedProxyReturnValue(<wbr>BrokerCommandBase.java:182)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>ImageActionsVDSCommandBase.<wbr>executeVdsBrokerCommand(<wbr>ImageActionsVDSCommandBase.<wbr>java:20)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>VdsBrokerCommand.<wbr>executeVDSCommand(<wbr>VdsBrokerCommand.java:111)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.VDSCommandBase.<wbr>executeCommand(VDSCommandBase.<wbr>java:73)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.dal.<wbr>VdcCommandBase.execute(<wbr>VdcCommandBase.java:33)<br>
>>> [dal.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.vdsbroker.<wbr>DefaultVdsCommandExecutor.<wbr>execute(<wbr>DefaultVdsCommandExecutor.<wbr>java:14)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.<wbr>vdsbroker.ResourceManager.<wbr>runVdsCommand(ResourceManager.<wbr>java:407)<br>
>>> [vdsbroker.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>VDSBrokerFrontendImpl.<wbr>runVdsCommand(<wbr>VDSBrokerFrontendImpl.java:33)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>storage.disk.image.<wbr>ImagesHandler.teardownImage(<wbr>ImagesHandler.java:1007)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>storage.disk.image.<wbr>ImagesHandler.<wbr>getQemuImageInfoFromVdsm(<wbr>ImagesHandler.java:856)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>storage.disk.image.<wbr>BaseImagesCommand.<wbr>endSuccessfully(<wbr>BaseImagesCommand.java:367)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>CommandBase.<wbr>internalEndSuccessfully(<wbr>CommandBase.java:736)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>CommandBase.<wbr>endActionInTransactionScope(<wbr>CommandBase.java:694)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.bll.<wbr>CommandBase.runInTransaction(<wbr>CommandBase.java:2057)<br>
>>> [bll.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.utils.<wbr>transaction.<wbr>TransactionSupport.<wbr>executeInSuppressed(<wbr>TransactionSupport.java:164)<br>
>>> [utils.jar:]<br>
>>> at<br>
>>> org.ovirt.engine.core.utils.<wbr>transaction.<wbr>TransactionSupport.<wbr>executeInScope(<wbr>TransactionSupport.java:103)<br>
>>> [utils.jar:]<br>
>>> at org.ovirt.engine.core.bll.<wbr>CommandBase.endAction(<wbr>CommandBase.java:559)<br>
>>> [bll.jar:]<br>
>>> at org.ovirt.engine.core.bll.<wbr>Backend.endAction(Backend.<wbr>java:536)<br>
>>> [bll.jar:]<br>
>>><br>
>>><br>
>>> Link to suspected patches: <a href="https://gerrit.ovirt.org/#/c/71132/" rel="noreferrer" target="_blank">https://gerrit.ovirt.org/#/c/<wbr>71132/</a><br>
>>> Link to Job:<br>
>>> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4950/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/4950/</a><br>
>>> Link to all logs:<br>
>>> <a href="http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_master/4951/artifact/exported-artifacts/basic_suite_master.sh-el7/test_logs/basic-suite-master/post-004_basic_sanity.py/" rel="noreferrer" target="_blank">http://jenkins.ovirt.org/job/<wbr>test-repo_ovirt_experimental_<wbr>master/4951/artifact/exported-<wbr>artifacts/basic_suite_master.<wbr>sh-el7/test_logs/basic-suite-<wbr>master/post-004_basic_sanity.<wbr>py/</a><br>
>>><br>
>>> Thanks<br>
>>> Gil<br>
>><br>
>><br>
>><br>
>> ______________________________<wbr>_________________<br>
>> Devel mailing list<br>
>> <a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
>> <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
><br>
><br>
______________________________<wbr>_________________<br>
Devel mailing list<br>
<a href="mailto:Devel@ovirt.org">Devel@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/devel</a><br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Eyal Edri<br>Associate Manager</div><div>RHV DevOps<br>EMEA ENG Virtualization R&D<br>Red Hat Israel<br><br>phone: +972-9-7692018<br>irc: eedri (on #tlv #rhev-dev #rhev-integ)</div></div></div></div></div></div></div>
</div></div>