[ovirt-devel] [ OST Failure Report ] [ oVirt master ] [ 25.1.17 ] [hotplug_disk]

Thu Jan 26 07:46:12 UTC 2017

I'm seeing a failure in 4.1, which I've never seen in the past:
2017-01-26 02:21:00,124 ERROR (mailbox-spm)
[storage.MailBox.SpmMailMonitor] SPM_MailMonitor: mailbox 2 checksum
failed, not clearing mailbox, clearing newMail. (storage_mailbox:618)

(from host1).

There are also some new warnings around monitoring, such as:
2017-01-26 02:20:58,035 WARN  (libvirt/events) [virt.vm]
(vmId='85e8df3c-8bdf-470f-8c28-baa521d663de') unknown eventid 8 args
('/rhev/data-center/27d461cd-03b2-4b2b-a1f8-157c27bb710e/2e10aa41-24f1-4c83-8e29-15f772a2d04b/images/304c3f03-61a6-4b79-a748-174a4cd24f84/3f96ca05-e8f3-41f2-8b9c-c7d170b24f92',
3, 0, 8) (clientIF:540)

On Wed, Jan 25, 2017 at 2:17 PM, Gil Shinar <gshinar at redhat.com> wrote:

> Link to job: http://jenkins.ovirt.org/job/test-repo_ovirt_
> experimental_master/4951/
>
> Sorry for the mistake
>
> On Wed, Jan 25, 2017 at 1:43 PM, Gil Shinar <gshinar at redhat.com> wrote:
>
>> Hi,
>>
>> The test in $subject has failed. Below please find exception I have found
>> in engine.log.
>>
>> {"jsonrpc": "2.0", "id": "4b8eb810-c52d-4c92-a792-e554f87c9493", "error": {"message": "Cannot deactivate Logical Volume: ('General Storage Exception: (\"5 [] [\\'  WARNING: Not using lvmetad because config setting use_lvmetad=0.\\', \\'  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\\', \\'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5 in use.\\', \\'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/ab4e8962-6196-485e-be2a-d5791a38eaeb in use.\\']\\\\nf9dce023-0282-4185-9ad9-fe71c3975106/[\\'778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5\\', \\'ab4e8962-6196-485e-be2a-d5791a38eaeb\\']\",)',)", "code": 552}}�
>>
>> 2017-01-25 05:05:49,037-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.internal.ResponseWorker] (ResponseWorker) [] Message received: {"jsonrpc": "2.0", "id": "4b8eb810-c52d-4c92-a792-e554f87c9493", "error": {"message": "Cannot deactivate Logical Volume: ('General Storage Exception: (\"5 [] [\\'  WARNING: Not using lvmetad because config setting use_lvmetad=0.\\', \\'  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\\', \\'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5 in use.\\', \\'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/ab4e8962-6196-485e-be2a-d5791a38eaeb in use.\\']\\\\nf9dce023-0282-4185-9ad9-fe71c3975106/[\\'778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5\\', \\'ab4e8962-6196-485e-be2a-d5791a38eaeb\\']\",)',)", "code": 552}}
>> 2017-01-25 05:05:49,047-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler5) [59ab00f1] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VDSM lago-basic-suite-master-host0 command TeardownImageVDS failed: Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\'  WARNING: Not using lvmetad because config setting use_lvmetad=0.\', \'  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5 in use.\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/ab4e8962-6196-485e-be2a-d5791a38eaeb in use.\']\\nf9dce023-0282-4185-9ad9-fe71c3975106/[\'778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5\', \'ab4e8962-6196-485e-be2a-d5791a38eaeb\']",)',)
>> 2017-01-25 05:05:49,047-05 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand] (DefaultQuartzScheduler5) [59ab00f1] Command 'TeardownImageVDSCommand(HostName = lago-basic-suite-master-host0, ImageActionsVDSCommandParameters:{runAsync='true', hostId='60e52527-b637-445c-b408-0275d347e76a'})' execution failed: VDSGenericException: VDSErrorException: Failed in vdscommand to TeardownImageVDS, error = Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\'  WARNING: Not using lvmetad because config setting use_lvmetad=0.\', \'  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5 in use.\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/ab4e8962-6196-485e-be2a-d5791a38eaeb in use.\']\\nf9dce023-0282-4185-9ad9-fe71c3975106/[\'778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5\', \'ab4e8962-6196-485e-be2a-d5791a38eaeb\']",)',)
>> 2017-01-25 05:05:49,047-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.TeardownImageVDSCommand] (DefaultQuartzScheduler5) [59ab00f1] Exception: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed in vdscommand to TeardownImageVDS, error = Cannot deactivate Logical Volume: ('General Storage Exception: ("5 [] [\'  WARNING: Not using lvmetad because config setting use_lvmetad=0.\', \'  WARNING: To avoid corruption, rescan devices to make changes visible (pvscan --cache).\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5 in use.\', \'  Logical volume f9dce023-0282-4185-9ad9-fe71c3975106/ab4e8962-6196-485e-be2a-d5791a38eaeb in use.\']\\nf9dce023-0282-4185-9ad9-fe71c3975106/[\'778cbc5b-a9df-46d7-bc80-1a66f7d3e2b5\', \'ab4e8962-6196-485e-be2a-d5791a38eaeb\']",)',)
>> 	at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:182) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.vdsbroker.vdsbroker.ImageActionsVDSCommandBase.executeVdsBrokerCommand(ImageActionsVDSCommandBase.java:20) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:111) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:73) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:33) [dal.jar:]
>> 	at org.ovirt.engine.core.vdsbroker.vdsbroker.DefaultVdsCommandExecutor.execute(DefaultVdsCommandExecutor.java:14) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:407) [vdsbroker.jar:]
>> 	at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.runVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.storage.disk.image.ImagesHandler.teardownImage(ImagesHandler.java:1007) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.storage.disk.image.ImagesHandler.getQemuImageInfoFromVdsm(ImagesHandler.java:856) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.storage.disk.image.BaseImagesCommand.endSuccessfully(BaseImagesCommand.java:367) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.CommandBase.internalEndSuccessfully(CommandBase.java:736) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.CommandBase.endActionInTransactionScope(CommandBase.java:694) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:2057) [bll.jar:]
>> 	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:164) [utils.jar:]
>> 	at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:103) [utils.jar:]
>> 	at org.ovirt.engine.core.bll.CommandBase.endAction(CommandBase.java:559) [bll.jar:]
>> 	at org.ovirt.engine.core.bll.Backend.endAction(Backend.java:536) [bll.jar:]
>>
>>
>> Link to suspected patches: https://gerrit.ovirt.org/#/c/71132/
>> Link to Job: http://jenkins.ovirt.org/job/test-repo_ovirt_experimental_ma
>> ster/4950/
>> Link to all logs: http://jenkins.ovirt.org/job/t
>> est-repo_ovirt_experimental_master/4951/artifact/exported-ar
>> tifacts/basic_suite_master.sh-el7/test_logs/basic-suite-mast
>> er/post-004_basic_sanity.py/
>>
>> Thanks
>> Gil
>>
>
>
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170126/9b2ba35d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 4_1_hotplug_disk.tgz
Type: application/x-gzip
Size: 1646561 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/devel/attachments/20170126/9b2ba35d/attachment-0001.tgz>