- Users - oVirt List Archives

dhw-error after HE restore
by Christoph Köhler 16 Jan '20

16 Jan '20

1 0

Re: upgrade issue
by Jingjie Jiang 16 Jan '20

16 Jan '20

Hi Paul, Thanks for your reply. I did install ovirt-release43 before I install the ovirt-node-ng-image-update package. Here is the brief procedure: 1. Install the 4.2.8 ovirt host on CentOS7.6 by adding the node from AdminPortal. 2. Upgrade ovirt 4.2.8 engine to 4.3.6. 3. Set the host to maintainence from AdminPortal. 4. Yum install ovirt-release43 rpm # yum install http://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm 5.Yum install node update package # yum install https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-imag… 6. reboot vdsm did not get update. -Jingjie On 1/15/20 6:14 AM, Staniforth, Paul wrote: > Hel# yum install > https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-imag… > Jingjie, > I think you need to install > https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm > <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresources…> > so yum updates to the 4.3 release otherwise it will stay on the 4.2 > release (4.2.8 being the last version). > > > Regards, > Paul S. > ------------------------------------------------------------------------ > *From:* Jingjie Jiang <jingjie.jiang(a)oracle.com> > *Sent:* 14 January 2020 15:30 > *To:* Parth Dhanjal <dparth(a)redhat.com> > *Cc:* users <users(a)ovirt.org> > *Subject:* [ovirt-users] Re: upgrade issue > > *Caution External Mail:* Do not click any links or open any > attachments unless you trust the sender and know that the content is safe. > > Hi Path, > > Thanks for your rely. > > Please check my inline comments > > -Jingjie > > On 1/14/20 2:28 AM, Parth Dhanjal wrote: >> Hello Jingjie! >> >> You can follow >> https://www.ovirt.org/documentation/upgrade-guide/chap-Upgrading_from_4.1_t… >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> > > I tried the above link before and it worked. > > But, I'd like to know how the upgrade with ovirt-node-ng-image-update > package works. > > Any suggestion about the issue I reported in previous email? > > Also, about the link you pointed, "imgbase check" command crashed when > I ran from the ovirt host based on CentOS: > > # imgbase check > Traceback (most recent call last): > File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main > "__main__", fname, loader, pkg_name) > File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code > exec code in run_globals > File "/usr/lib/python2.7/site-packages/imgbased/__main__.py", line > 53, in <module> > CliApplication() > File "/usr/lib/python2.7/site-packages/imgbased/__init__.py", line > 82, in CliApplication > app.hooks.emit("post-arg-parse", args) > File "/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120, > in emit > cb(self.context, *args) > File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", > line 185, in post_argparse > run_check(app) > File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", > line 225, in run_check > status = Health(app).status() > File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", > line 358, in status > status.results.append(group().run()) > File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py", > line 385, in check_thin > pool = self.app.imgbase._thinpool() > File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line > 120, in _thinpool > return LVM.Thinpool.from_tag(self.thinpool_tag) > File "/usr/lib/python2.7/site-packages/imgbased/lvm.py", line 227, > in from_tag > assert len(lvs) == 1 > AssertionError > > > >> Under this link you can find a tab to upgrade the hosts. Instead of >> https://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresources…> >> You can use >> https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresources…> >> >> Regards >> Parth Dhanjal >> >> On Mon, Jan 13, 2020 at 11:44 PM <jingjie.jiang(a)oracle.com >> <mailto:jingjie.jiang@oracle.com>> wrote: >> >> Hello, >> >> By following the instruction at >> https://www.ovirt.org/documentation/upgrade-guide/appe-Manually_Updating_Ho… >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…>, >> I tried to upgrade host from 4.2.8(CentOS 7.6) to 4.3.7. >> After the reboot, the host still running 4.2.8 vdsm. >> # rpm -qa | grep vdsm >> vdsm-common-4.20.46-1.el7.noarch >> vdsm-yajsonrpc-4.20.46-1.el7.noarch >> vdsm-hook-vhostmd-4.20.46-1.el7.noarch >> vdsm-4.20.46-1.el7.x86_64 >> vdsm-client-4.20.46-1.el7.noarch >> vdsm-python-4.20.46-1.el7.noarch >> vdsm-hook-fcoe-4.20.46-1.el7.noarch >> vdsm-network-4.20.46-1.el7.x86_64 >> vdsm-jsonrpc-4.20.46-1.el7.noarch >> vdsm-hook-vmfex-dev-4.20.46-1.el7.noarch >> vdsm-http-4.20.46-1.el7.noarch >> vdsm-api-4.20.46-1.el7.noarch >> vdsm-hook-ethtool-options-4.20.46-1.el7.noarch >> vdsm-hook-openstacknet-4.20.46-1.el7.noarch >> [root@ovs260 ~]# >> >> >> Looks like the upgrade did not take place as expected. >> Any clue? >> >> Thanks in advance, >> Jingjie >> _______________________________________________ >> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org> >> To unsubscribe send an email to users-leave(a)ovirt.org >> <mailto:users-leave@ovirt.org> >> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >> oVirt Code of Conduct: >> https://www.ovirt.org/community/about/community-guidelines/ >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovirt…> >> List Archives: >> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BLE3M55EMIDEC… >> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.ovi…> >> > To view the terms under which this email is distributed, please go to:- > http://leedsbeckett.ac.uk/disclaimer/email/

3 2

ovirt hosted engine deploy
by Nazan CENGİZ 16 Jan '20

16 Jan '20

Hi all, Ovirt node was successfully installed.But hosted engine deployment failed. I have been shared log files. [cid:6fda892b-465e-461f-b04c-f00344af3c5c] Could you please help me,please? Thanks. [cid:imageeb37b7.PNG@67cdf97c.449a1217]<http://www.havelsan.com.tr> [cid:imagee7cd94.JPG@4c2a019b.45ac6e31] Nazan CENGİZ AR-GE MÜHENDİSİ Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE [cid:imagebfc44b.PNG@4e621161.4ea98136] +90 312 219 57 87 [cid:image732ed1.PNG@b5060b4a.47ab90b8] +90 312 219 57 97 [cid:image48a8ed.JPG@9f956a02.4bad7f08] YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> LEGAL NOTICE: This e-mail is subject to the Terms and Conditions document which can be accessed with this link. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> [http://www.havelsan.com.tr/Library/images/mail/email.jpg] Lütfen gerekmedikçe bu sayfa ve eklerini yazdırmayınız / Please consider the environment before printing this email

2 2

Power management and storage domains error when running Ansible tasks!
by m.skrzetuski＠gmail.com 15 Jan '20

15 Jan '20

Hello everyone, when I create a host via Ansible role "ovirt.infra" I get an error that power management is not enabled (red exclamation sign in the UI). My vars in the playbook look as follows. hosts: - name: delirium address: delirium.home cluster: Matrix public_key: true power_management_enabled: true Why is power management not enabled? What is it used for and how do I enable it with Ansible? I am trying to create a storage domain as follows. storages: vmdata: domain_function: data state: present data_center: Matrix localfs: path: /vmisos host: delirium.home vmisos: domain_function: data state: present data_center: Matrix localfs: path: /vmdata host: delirium.home However I get the following error. msg: Fault reason is "Incomplete parameters". Fault detail is "StorageDomain [storage] required for add". HTTP response code is 400. An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Error: Fault reason is "Incomplete parameters". Fault detail is "StorageDomain [storage] required for add". HTTP response code is 400. failed: [delirium.home] (item={'key': 'vmisos', 'value': {'name': 'isos', 'domain_function': 'data', 'state': 'present', 'data_center': 'Matrix', 'localfs': {'path': '/vmdata'}, 'host': 'delirium.home'}}) => changed=false What is "storage" here? I don't see "storage" as parameter in https://docs.ansible.com/ansible/latest/modules/ovirt_storage_domain_module…. Kind regards skrzetuski

3 6

upgrade issue
by jingjie.jiang＠oracle.com 15 Jan '20

15 Jan '20

Hello, By following the instruction at https://www.ovirt.org/documentation/upgrade-guide/appe-Manually_Updating_Ho…, I tried to upgrade host from 4.2.8(CentOS 7.6) to 4.3.7. After the reboot, the host still running 4.2.8 vdsm. # rpm -qa | grep vdsm vdsm-common-4.20.46-1.el7.noarch vdsm-yajsonrpc-4.20.46-1.el7.noarch vdsm-hook-vhostmd-4.20.46-1.el7.noarch vdsm-4.20.46-1.el7.x86_64 vdsm-client-4.20.46-1.el7.noarch vdsm-python-4.20.46-1.el7.noarch vdsm-hook-fcoe-4.20.46-1.el7.noarch vdsm-network-4.20.46-1.el7.x86_64 vdsm-jsonrpc-4.20.46-1.el7.noarch vdsm-hook-vmfex-dev-4.20.46-1.el7.noarch vdsm-http-4.20.46-1.el7.noarch vdsm-api-4.20.46-1.el7.noarch vdsm-hook-ethtool-options-4.20.46-1.el7.noarch vdsm-hook-openstacknet-4.20.46-1.el7.noarch [root@ovs260 ~]# Looks like the upgrade did not take place as expected. Any clue? Thanks in advance, Jingjie

4 4

ovirt-provider-ovn external network provider
by p.staniforth＠leedsbeckett.ac.uk 14 Jan '20

14 Jan '20

Hello I was testing the ovirt-provider-ovn which was working but at some point seems to have stopped. The logical networks are created and VMs have their ports created and attached on the hosts they run or migrate to but their are no geneve tunnels created and on the engine host "ovn-sbctl show" produces no output. Any help appreciated. Thanks, Paul S.

3 2

Ovirt installation with USB
by Nazan CENGİZ 14 Jan '20

14 Jan '20

Hi all, I am trying install ovirt-node-ng-installer-4.3.5-2019080513.el7<https://resources.ovirt.org/pub/ovirt-4.3/iso/ovirt-node-ng-installer/4.3.5…> .I want setup on USB.I used UltraISO and USB Converter vs for converting.But it failed on below; "dracut-initqueue Warning dracut timeout": dracut/: Do you have any suggestions? Thanks. [cid:image56c8fc.PNG@b74488ec.4c807a5f]<http://www.havelsan.com.tr> [cid:image1ba961.JPG@c094eb48.40ac6f8b] Nazan CENGİZ AR-GE MÜHENDİSİ Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE [cid:image3092b8.PNG@a1ffaf9d.43a701c4] +90 312 219 57 87 [cid:image93a440.PNG@3f130358.4c891e67] +90 312 219 57 97 [cid:image7b8ddf.JPG@3fa3e4a7.40a7c68c] YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> LEGAL NOTICE: This e-mail is subject to the Terms and Conditions document which can be accessed with this link. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari> [http://www.havelsan.com.tr/Library/images/mail/email.jpg] Lütfen gerekmedikçe bu sayfa ve eklerini yazdırmayınız / Please consider the environment before printing this email

3 2

Re: Upload ISO to data domain - checksum mismatch on the brick
by Strahil 14 Jan '20

14 Jan '20

Hello Community, Just a short update - uploading on the next day works without issues. I will check it again ... Maybe too many isos in a row makes it crazy. Note: I have opened an RFE , so we can provide the ISO checksum, so the proxy can verify it: https://bugzilla.redhat.com/show_bug.cgi?id=1787906 Best Regards, Strahil NikolovOn Jan 12, 2020 12:56, Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote: > > Hello Community, > > I'm currently migrating all my ISOs from the ISO domain to the DATA domain but it seems that the checksum (on brick level) mismatches. > The problem is that I have managed to upload several ISOs (one of them 4.9GB) without issues. > > Restarting the ovirt-imageio-proxy.service and later the HostedEngine VM doesn't help. > > The logs from the imageio proxy indicate no issues. > > I thought that the gluster could be the problem ,but this is not the case - I tested via copying (using ovirt1) between the 2 mountpoints and the sha512sum matches. > > Trying to upload via ovirt2 also doesn't help. > > Any ideas are appreciated. > > PS: I'm running latest oVirt. > > Best Regards, > Strahil Nikolov

1 0

Re: Ovirt installation with USB
by Strahil 14 Jan '20

14 Jan '20

It's CentOS based -> either use the tools in the CentOS wiki (or anything in the Fedora) or use 'dd' in any Linux/UNIX distribution. Best Regards, Strahil NikolovOn Jan 14, 2020 13:43, Nazan CENGİZ <nazancengiz(a)havelsan.com.tr> wrote: > > Hi all, > > I am trying install ovirt-node-ng-installer-4.3.5-2019080513.el7 .I want setup on USB.I used UltraISO and USB Converter vs for converting.But it failed on below; > > "dracut-initqueue Warning dracut timeout": > > > dracut/: > > > Do you have any suggestions? > > Thanks. > > > > Nazan CENGİZ > AR-GE MÜHENDİSİ > Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE > +90 312 219 57 87 > +90 312 219 57 97 > YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. > LEGAL NOTICE: <

1 0

SPM contending loop
by Jayme 14 Jan '20

14 Jan '20

My cluster appears to be experiencing an SPM problem. I recently placed each host in maintenance to move the ovirt management network to another interface. All was successful and all VMs are currently running. However, I'm not facing an SPM contending loop with data center going in and out of responsive status. I have a 3 server HCI setup and all volumes are active and healed, there are no unsynced entries or split brains. Does anyone know how I could diagnose the SPM issue? engine.log: 2020-01-13 22:24:54,777-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] START, GlusterTasksListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 349f80a9 2020-01-13 22:24:55,231-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand] (DefaultQuartzScheduler2) [213adf4f] FINISH, GlusterTasksListVDSCommand, return: [], log id: 349f80a9 2020-01-13 22:24:58,245-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard0, VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 7b04f110 2020-01-13 22:24:58,887-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand' return value ' TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'} ' 2020-01-13 22:24:58,888-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] HostName = Orchard1 2020-01-13 22:24:58,888-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command 'HSMGetAllTasksStatusesVDSCommand(HostName = Orchard1, VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'})' execution failed: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM 2020-01-13 22:24:59,034-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.220/24:CONNECTED, orchard1.grove.silverorange.com:CONNECTED, orchard2.grove.silverorange.com:DISCONNECTED], log id: 7b04f110 2020-01-13 22:24:59,049-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterServersListVDSCommand(HostName = Orchard2, VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}), log id: 43f1dd82 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, ConnectStoragePoolVDSCommand(HostName = Orchard1, ConnectStoragePoolVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', vdsId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 2b397b31 2020-01-13 22:24:59,099-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} 2020-01-13 22:24:59,850-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand, return: [10.12.0.222/24:CONNECTED, 10.11.0.220:CONNECTED, orchard1.grove.silverorange.com:CONNECTED], log id: 43f1dd82 2020-01-13 22:24:59,852-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] START, GlusterVolumesListVDSCommand(HostName = Orchard0, GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}), log id: 263be6f8 2020-01-13 22:25:00,019-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, ConnectStoragePoolVDSCommand, return: , log id: 2b397b31 2020-01-13 22:25:00,036-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] hostFromVds::selectedVds - 'Orchard1', spmStatus 'Free', storage pool 'Default', storage pool version '4.3' 2020-01-13 22:25:00,056-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] starting spm on vds 'Orchard1', storage pool 'Default', prevId '-1', LVER '-1' 2020-01-13 22:25:00,057-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, SpmStartVDSCommand(HostName = Orchard1, SpmStartVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', prevId='-1', prevLVER='-1', storagePoolFormatType='V4', recoveryMode='Manual', SCSIFencing='false'}), log id: 3dea111a 2020-01-13 22:25:00,065-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling started: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' 2020-01-13 22:25:00,500-04 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterVolumesListVDSCommand, return: {3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@743f4102, 71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@cad5f8f4, 752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f9f55499, c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7a2c16da, 79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@33e1ab37, 095fd8fc-5322-4741-8805-fc0bb64b554f=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b28195c0}, log id: 263be6f8 2020-01-13 22:25:03,089-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Failed in 'HSMGetTaskStatusVDS' method 2020-01-13 22:25:03,090-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' task status 'finished' 2020-01-13 22:25:03,090-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Start SPM Task failed - result: 'cleanSuccess', message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',), code = 100 2020-01-13 22:25:03,104-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling ended, spm status: Free 2020-01-13 22:25:03,105-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START, HSMClearTaskVDSCommand(HostName = Orchard1, HSMTaskGuidBaseVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d', taskId='671d5904-e062-4d45-9eb4-83a6f13657fe'}), log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, HSMClearTaskVDSCommand, return: , log id: 5335606f 2020-01-13 22:25:03,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH, SpmStartVDSCommand, return: SpmStatusResult:{SPM Id='-1', SPM LVER='-1', SPM Status='Free'}, log id: 3dea111a 2020-01-13 22:25:03,117-04 INFO [org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: a45e442e-9989-11e8-b0e4-00163e4bf18a Type: StoragePool 2020-01-13 22:25:03,179-04 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException: IRSErrorException: SpmStart failed 2020-01-13 22:25:03,205-04 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Irs placed on server 'fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d' failed. Proceed Failover 2020-01-13 22:25:03,223-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] START, ConnectStoragePoolVDSCommand(HostName = Orchard2, ConnectStoragePoolVDSCommandParameters:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', vdsId='fd0752d8-2d41-45b0-887a-0ffacbb8a237', storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}), log id: 6a7302e9 2020-01-13 22:25:03,224-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Executing with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active, 23c22a0f-0482-425e-8ada-730cf8ec0751=active, 390c0320-e843-4ff3-a4bb-a9973058447f=active, fb43d33a-82c8-44cb-8169-090cd0d8f56e=active, d70b171e-7488-4d52-8cad-bbc581dbf16e=active, 1f2e9989-9ab3-43d5-971d-568b8feca918=active} ----------------------------- vdsm.log from one of my hosts: 2020-01-13 22:26:12,434-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,435-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 88772af7-8cf5-433e-8be3-8d0adf0bbf04 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/88772af7-8cf5-433e-8be3-8d0adf0bbf04/88772af7-8cf5-433e-8be3-8d0adf0bbf04.job.0',) 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a9b11e33-9b93-46a0-a36e-85063fd53ebe (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe/a9b11e33-9b93-46a0-a36e-85063fd53ebe.recover.0',) 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 650f2df4-6489-47e2-af5d-db86a22f01c0 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/650f2df4-6489-47e2-af5d-db86a22f01c0/650f2df4-6489-47e2-af5d-db86a22f01c0.recover.0',) 2020-01-13 22:26:12,487-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,488-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e.job.0',) 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743.task',) 2020-01-13 22:26:12,505-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,506-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 75318d21-45b2-4dbd-985c-a7851a10a463 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/75318d21-45b2-4dbd-985c-a7851a10a463/75318d21-45b2-4dbd-985c-a7851a10a463.job.0',) 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 1779b352-022c-49a3-9388-2f688d33cdab (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/1779b352-022c-49a3-9388-2f688d33cdab/1779b352-022c-49a3-9388-2f688d33cdab.job.0',) 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 0302036a-7d99-4685-befb-6fee1602feaf (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746, in _load self._loadJobMetaFile(taskDir, jn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696, in _loadJobMetaFile self._loadMetaFile(taskFile, self.jobs[n], Job.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/0302036a-7d99-4685-befb-6fee1602feaf/0302036a-7d99-4685-befb-6fee1602feaf.job.0',) 2020-01-13 22:26:12,545-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,546-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 24a825b9-d48d-4134-8aa1-b4db7a9c6ab1 (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737, in _load self._loadTaskMetaFile(taskDir) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688, in _loadTaskMetaFile self._loadMetaFile(taskFile, self, Task.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.backup/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.task',) 2020-01-13 22:26:12,560-0400 ERROR (tasks/5) [storage.TaskManager.Task] Unexpected error (task:652) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631, in _loadMetaFile for line in getProcPool().readLines(filename): File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py", line 334, in readLines return ioproc.readlines(path) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555, in readlines return self.readfile(path, direct).splitlines() File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543, in readfile "direct": direct}, self.timeout) File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448, in _sendCommand raise OSError(errcode, errstr) OSError: [Errno 2] No such file or directory 2020-01-13 22:26:12,561-0400 ERROR (tasks/5) [storage.TaskManager] taskManager: Skipping directory: 90d88529-a051-4acd-bef2-d0aa034c15de (taskManager:222) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 214, in loadDumpedTasks t = Task.loadTask(store, taskID) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162, in loadTask t._load(store, ext) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751, in _load self._loadRecoveryMetaFile(taskDir, rn) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705, in _loadRecoveryMetaFile self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653, in _loadMetaFile raise se.TaskMetaDataLoadError(filename) TaskMetaDataLoadError: Can't load Task Metadata: ('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/90d88529-a051-4acd-bef2-d0aa034c15de/90d88529-a051-4acd-bef2-d0aa034c15de.recover.0',) 2020-01-13 22:26:12,607-0400 ERROR (tasks/5) [storage.StoragePool] Unexpected error (sp:383) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) 2020-01-13 22:26:12,608-0400 ERROR (tasks/5) [storage.StoragePool] failed: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) (sp:384) 2020-01-13 22:26:12,635-0400 ERROR (tasks/5) [storage.TaskManager.Task] (Task='3c7de2a0-597c-4ebe-b4de-689dba26045b') Unexpected error (task:875) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run return fn(*args, **kargs) File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336, in run return self.cmd(*self.argslist, **self.argsdict) File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in startSpm self.taskMng.recoverDumpedTasks() File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 227, in recoverDumpedTasks self.queueRecovery(task) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 48, in queueRecovery return self._queueTask(task, task.recover) File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line 54, in _queueTask 'Task id already in use: {0}'.format(task.id)) AddTaskError: TaskManager error, unable to add task: ('Task id already in use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)

3 3