[ovirt-devel] Re: [CQ Failure Report] [Ovirt 4.2] [4.7.18]

Jira opened: https://ovirt-jira.atlassian.net/browse/OVIRT-2286 Tal, can you help fix the test? thanks, Dafna On Wed, Jul 4, 2018 at 7:41 PM, Dafna Ron <dron(a)redhat.com&gt; wrote: > The vdsm restart is a test in basic_sanity.py > The task that is stuck is downloadImageFromStream > > 2018-07-04 10:12:52,659-04 WARN [org.ovirt.engine.core.bll.sto > rage.domain.DeactivateStorageDomainWithOvfUpdateCommand] (default > task-1) [ce1c28ba-1550-457f-b5e3-ad051488f897] There are running tasks: > 'AsyncTask:{commandId='0b18c13f-0fce- > 4303-8f85-ae5a2b051991', rootCommandId='0b18c13f-0fce-4303-8f85-ae5a2b051991', > storagePoolId='8dd2fe5a-9dca-42e2-8593-1de1b18b4887', > taskId='f2af86fb-dbbb-430c-afd9-2f25131583b1', > vdsmTaskId='45ee0fc8-830d-47d5-9c4a-6d4ed72ae6a1', stepId= > 'null', taskType='downloadImageFromStream', status='running'} > > The task succeeded but after the deactivate storage domain attempt: > > 2018-07-04 10:13:01,579-04 INFO [org.ovirt.engine.core.bll.Ser > ialChildCommandsExecutionCallback] (EE-ManagedThreadFactory-engineScheduled-Thread-98) > [1c611686] Command 'ProcessOvfUpdateForStorageDomain' id: > '6342fce4-96ff-4ae3-8b40-8155a > 5509761' child commands '[0b18c13f-0fce-4303-8f85-ae5a2b051991, > 8e16826f-93dd-442a-8a74-13d14222d45e]' executions were completed, status > 'SUCCEEDED' > > > we should have a solution in ost for locked objects failing jobs... will > open a Jira to follow > > > > On Wed, Jul 4, 2018 at 6:16 PM, Nir Soffer <nsoffer(a)redhat.com&gt; wrote: > >> On Wed, Jul 4, 2018 at 6:46 PM Dafna Ron <dron(a)redhat.com&gt; wrote: >> >>> The actual test has failed with error: >>> >>> 2018-07-04 10:12:52,665-04 ERROR [org.ovirt.engine.api.restapi. >>> resource.AbstractBackendResource] (default task-1) [] Operation >>> Failed: [Cannot deactivate Storage while there are running tasks on this >>> Storage. >>> >>> -Please wait until tasks will finish and try again.] >>> >>> However, there is a problem with vdsm on host-1. it restarts which may >>> cause the issue with the running tasks. >>> >> >> Who restarted vdsm? why? >> >> >>> 2018-07-04 10:08:46,603-0400 INFO (ioprocess/5191) [IOProcessClient] >>> shutdown requested (__init__:108) >>> 2018-07-04 10:08:46,604-0400 INFO (MainThread) [storage.udev] Stopping >>> multipath event listener (udev:149) >>> 2018-07-04 10:08:46,604-0400 INFO (MainThread) [vdsm.api] FINISH >>> prepareForShutdown return=None from=internal, task_id=bf3b87e4-febf-4cb6-8bfa-5840fc926b49 >>> (api:52) >>> 2018-07-04 10:08:46,605-0400 INFO (MainThread) [vds] Stopping threads >>> (vdsmd:160) >>> 2018-07-04 10:08:46,605-0400 INFO (MainThread) [vds] Exiting >>> (vdsmd:171) >>> 2018-07-04 10:10:00,145-0400 INFO (MainThread) [vds] (PID: 14034) I am >>> the actual vdsm 4.20.32-1.el7 lago-basic-suite-4-2-host-1 >>> (3.10.0-862.2.3.el7.x86_64) (vdsmd:149) >>> 2018-07-04 10:10:00,146-0400 INFO (MainThread) [vds] VDSM will run >>> with cpu affinity: frozenset([1]) (vdsmd:262) >>> 2018-07-04 10:10:00,151-0400 INFO (MainThread) [storage.HSM] START HSM >>> init (hsm:366) >>> 2018-07-04 10:10:00,154-0400 INFO (MainThread) [storage.HSM] Creating >>> data-center mount directory '/rhev/data-center/mnt' (hsm:373) >>> 2018-07-04 10:10:00,154-0400 INFO (MainThread) [storage.fileUtils] >>> Creating directory: /rhev/data-center/mnt mode: None (fileUtils:197) >>> 2018-07-04 10:10:00,265-0400 INFO (MainThread) [storage.HSM] Unlinking >>> file '/rhev/data-center/8dd2fe5a-9dca-42e2-8593-1de1b18b4887/44eb >>> a8db-3a9c-4fbe-ba33-a039fcd561e1' (hsm:523) >>> 2018-07-04 10:10:00,266-0400 INFO (MainThread) [storage.HSM] Unlinking >>> file '/rhev/data-center/8dd2fe5a-9dca-42e2-8593-1de1b18b4887/mastersd' >>> (hsm:523) >>> 2018-07-04 10:10:00,266-0400 INFO (MainThread) [storage.HSM] Unlinking >>> file '/rhev/data-center/8dd2fe5a-9dca-42e2-8593-1de1b18b4887/c798 >>> 0a1e-91ef-4095-82eb-37ec03da9b3f' (hsm:523) >>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.HSM] Unlinking >>> file '/rhev/data-center/8dd2fe5a-9dca-42e2-8593-1de1b18b4887/4fc6 >>> 2763-d8a5-4c36-8687-91870a92ff05' (hsm:523) >>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.HSM] Unlinking >>> file '/rhev/data-center/8dd2fe5a-9dca-42e2-8593-1de1b18b4887/0236 >>> 3608-01b9-4176-b7a1-e9ee235f792a' (hsm:523) >>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.udev] >>> Registering multipath event monitor <vdsm.storage.mpathhealth.Monitor >>> object at 0x7f9aa4548150> (udev:182) >>> 2018-07-04 10:10:00,267-0400 INFO (MainThread) [storage.udev] Starting >>> multipath event listener (udev:116) >>> 2018-07-04 10:10:00,298-0400 INFO (MainThread) [storage.check] >>> Starting check service (check:91) >>> 2018-07-04 10:10:00,303-0400 INFO (MainThread) [storage.Dispatcher] >>> Starting StorageDispatcher... (di >>> >>> On Wed, Jul 4, 2018 at 4:12 PM, Greg Sheremeta <gshereme(a)redhat.com&gt; >>> wrote: >>> >>>> """ >>>> Error: Fault reason is "Operation Failed". Fault detail is "[Cannot >>>> deactivate Storage while there are running tasks on this Storage. >>>> -Please wait until tasks will finish and try again.]". HTTP response >>>> code is 409. >>>> """ >>>> >>>> >>>> On Wed, Jul 4, 2018 at 10:56 AM Ehud Yonasi <eyonasi(a)redhat.com&gt; >>>> wrote: >>>> >>>>> Link to job: >>>>> http://jenkins.ovirt.org/job/ovirt-4.2_change-queue-tester/2571/ >>>>> >>>>> Suspected error: >>>>> >>>>> Error occured, aborting >>>>> Traceback (most recent call last): >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 362, in do_run >>>>> self.cli_plugins[args.ovirtverb].do_run(args) >>>>> File "/usr/lib/python2.7/site-packages/lago/plugins/cli.py", line 184, in do_run >>>>> self._do_run(**vars(args)) >>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 505, in wrapper >>>>> return func(*args, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/lago/utils.py", line 516, in wrapper >>>>> return func(*args, prefix=prefix, **kwargs) >>>>> File "/usr/lib/python2.7/site-packages/ovirtlago/cmd.py", line 99, in do_ovirt_runtest >>>>> raise RuntimeError('Some tests failed') >>>>> RuntimeError: Some tests failed >>>>> + res=1 >>>>> + cd - >>>>> /home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests >>>>> + return 1 >>>>> + failed=true >>>>> + env_collect /home/jenkins/workspace/ovirt-4.2_change-queue-tester/ovirt-system-tests/test_logs/basic-suite-4.2/post-007_sd_reattach.py >>>>> >>>>> _______________________________________________ >>>>> Infra mailing list -- infra(a)ovirt.org >>>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>>> y/about/community-guidelines/ >>>>> List Archives: https://lists.ovirt.org/archiv >>>>> es/list/infra(a)ovirt.org/message/JVGNSGB3HDB3E7Y7CTQR2FCJ3KEXMLFW/ >>>>> >>>> >>>> >>>> -- >>>> >>>> GREG SHEREMETA >>>> >>>> SENIOR SOFTWARE ENGINEER - TEAM LEAD - RHV UX >>>> >>>> Red Hat NA >>>> >>>> <https://www.redhat.com/> >>>> >>>> gshereme(a)redhat.com IRC: gshereme >>>> <https://red.ht/sig> >>>> >>>> _______________________________________________ >>>> Infra mailing list -- infra(a)ovirt.org >>>> To unsubscribe send an email to infra-leave(a)ovirt.org >>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>>> oVirt Code of Conduct: https://www.ovirt.org/communit >>>> y/about/community-guidelines/ >>>> List Archives: https://lists.ovirt.org/archiv >>>> es/list/infra(a)ovirt.org/message/GRWM4DDAGDV52QM3AWMKLWJP7MNI7S5O/ >>>> >>>> >>> _______________________________________________ >>> Infra mailing list -- infra(a)ovirt.org >>> To unsubscribe send an email to infra-leave(a)ovirt.org >>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >>> oVirt Code of Conduct: https://www.ovirt.org/communit >>> y/about/community-guidelines/ >>> List Archives: https://lists.ovirt.org/archiv >>> es/list/infra(a)ovirt.org/message/EPWWZ3WCO43XEDB53XPJYZL4PGTKOYDQ/ >>> >> >