Hi, we have a failure on basic suite for test check_snapshot_with. _memory. I am actually not seeing any reason the patch would cause that specific issue but its consistently failing on this change. I am seeing a vm memory saved to the iscsi domain but aside from that nothing is failing before tto cause a cleanup issue or create low space on the storage). can some one please take a look to see if anything in the patch is causing this? Link and headline of suspected patches: https://gerrit.ovirt.org/#/c/95436/1 - webadmin: network operation in progress - sync host interfaces Link to Job: http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/ Link to all logs: https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact... (Relevant) error snippet from the log: <error> 2018-11-15 08:19:33,409-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runVdsCommand, params: [IsVmDuringInitiating , IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], timeElapsed: 1ms 2018-11-15 08:19:33,430-05 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID: USER_FAILED_RUN_VM(54), Failed to run VM vm0 due to a failed validation: [Cannot run VM. Low disk space on Storage Domain iscsi.] (User: admin@internal-authz). 2018-11-15 08:19:33,430-05 WARN [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of action 'RunVm' failed for user admin@internal-authz. Reasons: VAR__ ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName iscsi 2018-11-15 08:19:33,431-05 INFO [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to object 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e 48ecb4=VM]', sharedLocks=''}' 2018-11-15 08:19:33,438-05 DEBUG [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction, params: [RunVm, RunVmParams:{comm andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null', commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], timeElapsed: 115ms 2018-11-15 08:19:33,441-05 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage Domain iscsi.] 2018-11-15 08:19:34,044-05 DEBUG [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START, GetAllVmStatsVDSCommand(HostName = lago-basic-su ite-4-2-host-0, VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}), log id: 65651e2b 2018-11-15 08:19:34,045-05 DEBUG [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND destination:jms.topic.vdsm_requests reply-to:jms.topic.vdsm_responses content-length:103 </error> _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZIG4VQKIFFA...

I am seeing a second project to fail on this: https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change... I am not seeing any changes in OST merged in the last 2 days that could have caused this and I am also not seeing any infrastructure commonality (not running on same hosts). However, something has changed to cause the iscsi domain to be filled and fail on run vm. Milan, can you please double check that this change is not causing any leftovers in the iscsi domain? https://gerrit.ovirt.org/#/c/95333/ Thanks, Dafna On Fri, Nov 16, 2018 at 11:32 AM Greg Sheremeta <gshereme(a)redhat.com&gt; wrote: > It's definitely not that patch - the patch is UI / GWT only and isn't > tested outside of the selenium tests in 008. Also, this is a backport and > we didn't see any issues on master. > > Greg > > On Fri, Nov 16, 2018, 6:24 AM Dafna Ron <dron(a)redhat.com wrote: > >> Hi, >> we have a failure on basic suite for test check_snapshot_with. _memory. >> >> I am actually not seeing any reason the patch would cause that specific >> issue but its consistently failing on this change. >> I am seeing a vm memory saved to the iscsi domain but aside from that >> nothing is failing before tto cause a cleanup issue or create low space on >> the storage). >> can some one please take a look to see if anything in the patch is >> causing this? >> >> >> Link and headline of suspected patches: >> >> https://gerrit.ovirt.org/#/c/95436/1 - >> webadmin: network operation in progress - sync host interfaces >> >> Link to Job: >> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/ >> >> Link to all logs: >> >> >> https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact... >> >> (Relevant) error snippet from the log: >> >> <error> >> >> 2018-11-15 08:19:33,409-05 DEBUG >> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: >> runVdsCommand, params: [IsVmDuringInitiating >> , >> IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], >> timeElapsed: 1ms >> 2018-11-15 08:19:33,430-05 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID: >> USER_FAILED_RUN_VM(54), Failed to run VM >> vm0 due to a failed validation: [Cannot run VM. Low disk space on >> Storage Domain iscsi.] (User: admin@internal-authz). >> 2018-11-15 08:19:33,430-05 WARN >> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) >> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of action 'RunVm' failed >> for user admin@internal-authz. Reasons: VAR__ >> ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName >> iscsi >> 2018-11-15 08:19:33,431-05 INFO >> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) >> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to object >> 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e >> 48ecb4=VM]', sharedLocks=''}' >> 2018-11-15 08:19:33,438-05 DEBUG >> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction, >> params: [RunVm, RunVmParams:{comm >> andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null', >> commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], >> timeElapsed: 115ms >> 2018-11-15 08:19:33,441-05 ERROR >> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >> task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage >> Domain iscsi.] >> 2018-11-15 08:19:34,044-05 DEBUG >> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] >> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START, >> GetAllVmStatsVDSCommand(HostName = lago-basic-su >> ite-4-2-host-0, >> VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}), >> log id: 65651e2b >> 2018-11-15 08:19:34,045-05 DEBUG >> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] >> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND >> destination:jms.topic.vdsm_requests >> reply-to:jms.topic.vdsm_responses >> content-length:103 >> >> </error> >> >> >> _______________________________________________ >> Devel mailing list -- devel(a)ovirt.org >> To unsubscribe send an email to devel-leave(a)ovirt.org >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> List Archives: >> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZIG4VQKIFFA... >> >

this has happened in the not so distant past: [ovirt-devel] [ OST Failure Report ] [ oVirt 4.2 (imgbased) ] [ 09-08-2018 ] [ 004_basic_sanity.check_snapshot_with_memory ] On Fri, Nov 16, 2018 at 3:54 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > Other projects are passing so it cannot be ost > we have ovirt-engine 4.2 that passed though: > https://gerrit.ovirt.org/#/c/95447/ > > logging a ticket to look into these failures. > > On Fri, Nov 16, 2018 at 1:30 PM Dafna Ron <dron(a)redhat.com&gt; wrote: > >> I am seeing a second project to fail on this: >> >> https://jenkins.ovirt.org/view/Change%20queue%20jobs/job/ovirt-4.2_change... >> >> I am not seeing any changes in OST merged in the last 2 days that could >> have caused this and I am also not seeing any infrastructure commonality >> (not running on same hosts). >> >> However, something has changed to cause the iscsi domain to be filled >> and fail on run vm. >> >> Milan, can you please double check that this change is not causing any >> leftovers in the iscsi domain? >> https://gerrit.ovirt.org/#/c/95333/ >> >> Thanks, >> Dafna >> >> >> >> On Fri, Nov 16, 2018 at 11:32 AM Greg Sheremeta <gshereme(a)redhat.com&gt; >> wrote: >> >>> It's definitely not that patch - the patch is UI / GWT only and isn't >>> tested outside of the selenium tests in 008. Also, this is a backport and >>> we didn't see any issues on master. >>> >>> Greg >>> >>> On Fri, Nov 16, 2018, 6:24 AM Dafna Ron <dron(a)redhat.com wrote: >>> >>>> Hi, >>>> we have a failure on basic suite for test check_snapshot_with. >>>> _memory. >>>> >>>> I am actually not seeing any reason the patch would cause that >>>> specific issue but its consistently failing on this change. >>>> I am seeing a vm memory saved to the iscsi domain but aside from that >>>> nothing is failing before tto cause a cleanup issue or create low space on >>>> the storage). >>>> can some one please take a look to see if anything in the patch is >>>> causing this? >>>> >>>> >>>> Link and headline of suspected patches: >>>> >>>> https://gerrit.ovirt.org/#/c/95436/1 - >>>> webadmin: network operation in progress - sync host interfaces >>>> >>>> Link to Job: >>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/ >>>> >>>> Link to all logs: >>>> >>>> >>>> https://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/3487/artifact... >>>> >>>> (Relevant) error snippet from the log: >>>> >>>> <error> >>>> >>>> 2018-11-15 08:19:33,409-05 DEBUG >>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: >>>> runVdsCommand, params: [IsVmDuringInitiating >>>> , >>>> IsVmDuringInitiatingVDSCommandParameters:{vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], >>>> timeElapsed: 1ms >>>> 2018-11-15 08:19:33,430-05 ERROR >>>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] EVENT_ID: >>>> USER_FAILED_RUN_VM(54), Failed to run VM >>>> vm0 due to a failed validation: [Cannot run VM. Low disk space on >>>> Storage Domain iscsi.] (User: admin@internal-authz). >>>> 2018-11-15 08:19:33,430-05 WARN >>>> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) >>>> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Validation of action 'RunVm' failed >>>> for user admin@internal-authz. Reasons: VAR__ >>>> ACTION__RUN,VAR__TYPE__VM,ACTION_TYPE_FAILED_DISK_SPACE_LOW_ON_STORAGE_DOMAIN,$storageName >>>> iscsi >>>> 2018-11-15 08:19:33,431-05 INFO >>>> [org.ovirt.engine.core.bll.RunVmCommand] (default task-5) >>>> [dce28fd0-bebc-4aba-80d5-ffc081b1f591] Lock freed to object >>>> 'EngineLock:{exclusiveLocks='[9144eb88-8f0a-4e77-9d70-3b761e >>>> 48ecb4=VM]', sharedLocks=''}' >>>> 2018-11-15 08:19:33,438-05 DEBUG >>>> [org.ovirt.engine.core.common.di.interceptor.DebugLoggingInterceptor] >>>> (default task-5) [dce28fd0-bebc-4aba-80d5-ffc081b1f591] method: runAction, >>>> params: [RunVm, RunVmParams:{comm >>>> andId='69628467-56d3-4d93-aed1-a2807b7dbc45', user='null', >>>> commandType='Unknown', vmId='9144eb88-8f0a-4e77-9d70-3b761e48ecb4'}], >>>> timeElapsed: 115ms >>>> 2018-11-15 08:19:33,441-05 ERROR >>>> [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default >>>> task-5) [] Operation Failed: [Cannot run VM. Low disk space on Storage >>>> Domain iscsi.] >>>> 2018-11-15 08:19:34,044-05 DEBUG >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetAllVmStatsVDSCommand] >>>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] START, >>>> GetAllVmStatsVDSCommand(HostName = lago-basic-su >>>> ite-4-2-host-0, >>>> VdsIdVDSCommandParametersBase:{hostId='063dfef9-bc9d-44c3-8ba3-8142e3eb129c'}), >>>> log id: 65651e2b >>>> 2018-11-15 08:19:34,045-05 DEBUG >>>> [org.ovirt.vdsm.jsonrpc.client.reactors.stomp.impl.Message] >>>> (EE-ManagedThreadFactory-engineScheduled-Thread-25) [] SEND >>>> destination:jms.topic.vdsm_requests >>>> reply-to:jms.topic.vdsm_responses >>>> content-length:103 >>>> >>>> </error> >>>> >>>> >>>> _______________________________________________ >>>> Devel mailing list -- devel(a)ovirt.org >>>> To unsubscribe send an email to devel-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: >>>> https://www.ovirt.org/community/about/community-guidelines/ >>>> List Archives: >>>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/ZIG4VQKIFFA... >>>> >>> _______________________________________________ Devel mailing list -- devel(a)ovirt.org To unsubscribe send an email to devel-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/LXXKMSAFSWA...