dhw-error after HE restore
by Christoph Köhler
Hey,
since we had moved the HE to another storage with
hosted-engine --delpoy --restore-from-file=...
we have a running engine (4.3.7.2-1.el7) - fine!
But I get every 6 minutes an error from dhw:
2020-01-16
14:33:10|bXli2A|p4VZox|jabi49|OVIRT_ENGINE_DWH|SampleTimeKeepingJob|Default|5|tWarn|tWarn_1|Can
not sample data, oVirt Engine is not updating the statistics. Please
check your oVirt Engine status.|9704
I don't no what to do.
Restarting ovirt-engine-dwhw does not help.
Any ideas?
Chris
4 years, 3 months
Re: upgrade issue
by Jingjie Jiang
Hi Paul,
Thanks for your reply.
I did install ovirt-release43 before I install the
ovirt-node-ng-image-update package.
Here is the brief procedure:
1. Install the 4.2.8 ovirt host on CentOS7.6 by adding the node from
AdminPortal.
2. Upgrade ovirt 4.2.8 engine to 4.3.6.
3. Set the host to maintainence from AdminPortal.
4. Yum install ovirt-release43 rpm
# yum install
http://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm
5.Yum install node update package
# yum install
https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-im...
6. reboot
vdsm did not get update.
-Jingjie
On 1/15/20 6:14 AM, Staniforth, Paul wrote:
> Hel# yum install
> https://resources.ovirt.org/pub/ovirt-4.3/rpm/el7/noarch/ovirt-node-ng-im...
> Jingjie,
> I think you need to install
> https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm
> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresourc...>
> so yum updates to the 4.3 release otherwise it will stay on the 4.2
> release (4.2.8 being the last version).
>
>
> Regards,
> Paul S.
> ------------------------------------------------------------------------
> *From:* Jingjie Jiang <jingjie.jiang(a)oracle.com>
> *Sent:* 14 January 2020 15:30
> *To:* Parth Dhanjal <dparth(a)redhat.com>
> *Cc:* users <users(a)ovirt.org>
> *Subject:* [ovirt-users] Re: upgrade issue
>
> *Caution External Mail:* Do not click any links or open any
> attachments unless you trust the sender and know that the content is safe.
>
> Hi Path,
>
> Thanks for your rely.
>
> Please check my inline comments
>
> -Jingjie
>
> On 1/14/20 2:28 AM, Parth Dhanjal wrote:
>> Hello Jingjie!
>>
>> You can follow
>> https://www.ovirt.org/documentation/upgrade-guide/chap-Upgrading_from_4.1...
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>
>
> I tried the above link before and it worked.
>
> But, I'd like to know how the upgrade with ovirt-node-ng-image-update
> package works.
>
> Any suggestion about the issue I reported in previous email?
>
> Also, about the link you pointed, "imgbase check" command crashed when
> I ran from the ovirt host based on CentOS:
>
> # imgbase check
> Traceback (most recent call last):
> File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
> "__main__", fname, loader, pkg_name)
> File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
> exec code in run_globals
> File "/usr/lib/python2.7/site-packages/imgbased/__main__.py", line
> 53, in <module>
> CliApplication()
> File "/usr/lib/python2.7/site-packages/imgbased/__init__.py", line
> 82, in CliApplication
> app.hooks.emit("post-arg-parse", args)
> File "/usr/lib/python2.7/site-packages/imgbased/hooks.py", line 120,
> in emit
> cb(self.context, *args)
> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py",
> line 185, in post_argparse
> run_check(app)
> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py",
> line 225, in run_check
> status = Health(app).status()
> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py",
> line 358, in status
> status.results.append(group().run())
> File "/usr/lib/python2.7/site-packages/imgbased/plugins/core.py",
> line 385, in check_thin
> pool = self.app.imgbase._thinpool()
> File "/usr/lib/python2.7/site-packages/imgbased/imgbase.py", line
> 120, in _thinpool
> return LVM.Thinpool.from_tag(self.thinpool_tag)
> File "/usr/lib/python2.7/site-packages/imgbased/lvm.py", line 227,
> in from_tag
> assert len(lvs) == 1
> AssertionError
>
>
>
>> Under this link you can find a tab to upgrade the hosts. Instead of
>> https://resources.ovirt.org/pub/yum-repo/ovirt-release42.rpm
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresourc...>
>> You can use
>> https://resources.ovirt.org/pub/yum-repo/ovirt-release43.rpm
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fresourc...>
>>
>> Regards
>> Parth Dhanjal
>>
>> On Mon, Jan 13, 2020 at 11:44 PM <jingjie.jiang(a)oracle.com
>> <mailto:jingjie.jiang@oracle.com>> wrote:
>>
>> Hello,
>>
>> By following the instruction at
>> https://www.ovirt.org/documentation/upgrade-guide/appe-Manually_Updating_...
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>,
>> I tried to upgrade host from 4.2.8(CentOS 7.6) to 4.3.7.
>> After the reboot, the host still running 4.2.8 vdsm.
>> # rpm -qa | grep vdsm
>> vdsm-common-4.20.46-1.el7.noarch
>> vdsm-yajsonrpc-4.20.46-1.el7.noarch
>> vdsm-hook-vhostmd-4.20.46-1.el7.noarch
>> vdsm-4.20.46-1.el7.x86_64
>> vdsm-client-4.20.46-1.el7.noarch
>> vdsm-python-4.20.46-1.el7.noarch
>> vdsm-hook-fcoe-4.20.46-1.el7.noarch
>> vdsm-network-4.20.46-1.el7.x86_64
>> vdsm-jsonrpc-4.20.46-1.el7.noarch
>> vdsm-hook-vmfex-dev-4.20.46-1.el7.noarch
>> vdsm-http-4.20.46-1.el7.noarch
>> vdsm-api-4.20.46-1.el7.noarch
>> vdsm-hook-ethtool-options-4.20.46-1.el7.noarch
>> vdsm-hook-openstacknet-4.20.46-1.el7.noarch
>> [root@ovs260 ~]#
>>
>>
>> Looks like the upgrade did not take place as expected.
>> Any clue?
>>
>> Thanks in advance,
>> Jingjie
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> <mailto:users-leave@ovirt.org>
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ovi...>
>> List Archives:
>> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BLE3M55EMID...
>> <https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.o...>
>>
> To view the terms under which this email is distributed, please go to:-
> http://leedsbeckett.ac.uk/disclaimer/email/
4 years, 3 months
ovirt hosted engine deploy
by Nazan CENGİZ
Hi all,
Ovirt node was successfully installed.But hosted engine deployment failed.
I have been shared log files.
[cid:6fda892b-465e-461f-b04c-f00344af3c5c]
Could you please help me,please?
Thanks.
[cid:imageeb37b7.PNG@67cdf97c.449a1217]<http://www.havelsan.com.tr> [cid:imagee7cd94.JPG@4c2a019b.45ac6e31]
Nazan CENGİZ
AR-GE MÜHENDİSİ
Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE
[cid:imagebfc44b.PNG@4e621161.4ea98136] +90 312 219 57 87 [cid:image732ed1.PNG@b5060b4a.47ab90b8] +90 312 219 57 97
[cid:image48a8ed.JPG@9f956a02.4bad7f08]
YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari>
LEGAL NOTICE: This e-mail is subject to the Terms and Conditions document which can be accessed with this link. <http://havelsan.com.tr/tr/news/e-posta-yasal-uyari>
[http://www.havelsan.com.tr/Library/images/mail/email.jpg] Lütfen gerekmedikçe bu sayfa ve eklerini yazdırmayınız / Please consider the environment before printing this email
4 years, 3 months
Power management and storage domains error when running Ansible tasks!
by m.skrzetuski＠gmail.com
Hello everyone,
when I create a host via Ansible role "ovirt.infra" I get an error that power management is not enabled (red exclamation sign in the UI). My vars in the playbook look as follows.
hosts:
- name: delirium
address: delirium.home
cluster: Matrix
public_key: true
power_management_enabled: true
Why is power management not enabled?
What is it used for and how do I enable it with Ansible?
I am trying to create a storage domain as follows.
storages:
vmdata:
domain_function: data
state: present
data_center: Matrix
localfs:
path: /vmisos
host: delirium.home
vmisos:
domain_function: data
state: present
data_center: Matrix
localfs:
path: /vmdata
host: delirium.home
However I get the following error.
msg: Fault reason is "Incomplete parameters". Fault detail is "StorageDomain [storage] required for add". HTTP response code is 400.
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: Error: Fault reason is "Incomplete parameters". Fault detail is "StorageDomain [storage] required for add". HTTP response code is 400.
failed: [delirium.home] (item={'key': 'vmisos', 'value': {'name': 'isos', 'domain_function': 'data', 'state': 'present', 'data_center': 'Matrix', 'localfs': {'path': '/vmdata'}, 'host': 'delirium.home'}}) => changed=false
What is "storage" here? I don't see "storage" as parameter in https://docs.ansible.com/ansible/latest/modules/ovirt_storage_domain_modu....
Kind regards
skrzetuski
4 years, 3 months
upgrade issue
by jingjie.jiang＠oracle.com
Hello,
By following the instruction at https://www.ovirt.org/documentation/upgrade-guide/appe-Manually_Updating_..., I tried to upgrade host from 4.2.8(CentOS 7.6) to 4.3.7.
After the reboot, the host still running 4.2.8 vdsm.
# rpm -qa | grep vdsm
vdsm-common-4.20.46-1.el7.noarch
vdsm-yajsonrpc-4.20.46-1.el7.noarch
vdsm-hook-vhostmd-4.20.46-1.el7.noarch
vdsm-4.20.46-1.el7.x86_64
vdsm-client-4.20.46-1.el7.noarch
vdsm-python-4.20.46-1.el7.noarch
vdsm-hook-fcoe-4.20.46-1.el7.noarch
vdsm-network-4.20.46-1.el7.x86_64
vdsm-jsonrpc-4.20.46-1.el7.noarch
vdsm-hook-vmfex-dev-4.20.46-1.el7.noarch
vdsm-http-4.20.46-1.el7.noarch
vdsm-api-4.20.46-1.el7.noarch
vdsm-hook-ethtool-options-4.20.46-1.el7.noarch
vdsm-hook-openstacknet-4.20.46-1.el7.noarch
[root@ovs260 ~]#
Looks like the upgrade did not take place as expected.
Any clue?
Thanks in advance,
Jingjie
4 years, 3 months
ovirt-provider-ovn external network provider
by p.staniforth＠leedsbeckett.ac.uk
Hello I was testing the ovirt-provider-ovn which was working but at some point seems to have stopped.
The logical networks are created and VMs have their ports created and attached on the hosts they run or migrate to but their are no geneve tunnels created and on the engine host "ovn-sbctl show" produces no output.
Any help appreciated.
Thanks,
Paul S.
4 years, 3 months
Re: Upload ISO to data domain - checksum mismatch on the brick
by Strahil
Hello Community,
Just a short update - uploading on the next day works without issues.
I will check it again ... Maybe too many isos in a row makes it crazy.
Note: I have opened an RFE , so we can provide the ISO checksum, so the proxy can verify it:
https://bugzilla.redhat.com/show_bug.cgi?id=1787906
Best Regards,
Strahil NikolovOn Jan 12, 2020 12:56, Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
>
> Hello Community,
>
> I'm currently migrating all my ISOs from the ISO domain to the DATA domain but it seems that the checksum (on brick level) mismatches.
> The problem is that I have managed to upload several ISOs (one of them 4.9GB) without issues.
>
> Restarting the ovirt-imageio-proxy.service and later the HostedEngine VM doesn't help.
>
> The logs from the imageio proxy indicate no issues.
>
> I thought that the gluster could be the problem ,but this is not the case - I tested via copying (using ovirt1) between the 2 mountpoints and the sha512sum matches.
>
> Trying to upload via ovirt2 also doesn't help.
>
> Any ideas are appreciated.
>
> PS: I'm running latest oVirt.
>
> Best Regards,
> Strahil Nikolov
4 years, 3 months
Re: Ovirt installation with USB
by Strahil
It's CentOS based -> either use the tools in the CentOS wiki (or anything in the Fedora) or use 'dd' in any Linux/UNIX distribution.
Best Regards,
Strahil NikolovOn Jan 14, 2020 13:43, Nazan CENGİZ <nazancengiz(a)havelsan.com.tr> wrote:
>
> Hi all,
>
> I am trying install ovirt-node-ng-installer-4.3.5-2019080513.el7 .I want setup on USB.I used UltraISO and USB Converter vs for converting.But it failed on below;
>
> "dracut-initqueue Warning dracut timeout":
>
>
> dracut/:
>
>
> Do you have any suggestions?
>
> Thanks.
>
>
>
> Nazan CENGİZ
> AR-GE MÜHENDİSİ
> Mustafa Kemal Mahallesi 2120 Cad. No:39 06510 Çankaya Ankara TÜRKİYE
> +90 312 219 57 87
> +90 312 219 57 97
> YASAL UYARI: Bu elektronik posta işbu linki kullanarak ulaşabileceğiniz Koşul ve Şartlar dokümanına tabidir.
> LEGAL NOTICE: <
4 years, 3 months
SPM contending loop
by Jayme
My cluster appears to be experiencing an SPM problem. I recently placed
each host in maintenance to move the ovirt management network to another
interface. All was successful and all VMs are currently running. However,
I'm not facing an SPM contending loop with data center going in and out of
responsive status.
I have a 3 server HCI setup and all volumes are active and healed, there
are no unsynced entries or split brains.
Does anyone know how I could diagnose the SPM issue?
engine.log:
2020-01-13 22:24:54,777-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler2) [213adf4f] START,
GlusterTasksListVDSCommand(HostName = Orchard0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 349f80a9
2020-01-13 22:24:55,231-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterTasksListVDSCommand]
(DefaultQuartzScheduler2) [213adf4f] FINISH, GlusterTasksListVDSCommand,
return: [], log id: 349f80a9
2020-01-13 22:24:58,245-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] START,
GlusterServersListVDSCommand(HostName = Orchard0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 7b04f110
2020-01-13 22:24:58,887-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command
'org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand'
return value '
TaskStatusListReturn:{status='Status [code=654, message=Not SPM]'}
'
2020-01-13 22:24:58,888-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] HostName = Orchard1
2020-01-13 22:24:58,888-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Command
'HSMGetAllTasksStatusesVDSCommand(HostName = Orchard1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'})'
execution failed: IRSGenericException: IRSErrorException:
IRSNonOperationalException: Not SPM
2020-01-13 22:24:59,034-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand,
return: [10.12.0.220/24:CONNECTED, orchard1.grove.silverorange.com:CONNECTED,
orchard2.grove.silverorange.com:DISCONNECTED], log id: 7b04f110
2020-01-13 22:24:59,049-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] START,
GlusterServersListVDSCommand(HostName = Orchard2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 43f1dd82
2020-01-13 22:24:59,099-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
ConnectStoragePoolVDSCommand(HostName = Orchard1,
ConnectStoragePoolVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
vdsId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}),
log id: 2b397b31
2020-01-13 22:24:59,099-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Executing with
domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active,
23c22a0f-0482-425e-8ada-730cf8ec0751=active,
390c0320-e843-4ff3-a4bb-a9973058447f=active,
fb43d33a-82c8-44cb-8169-090cd0d8f56e=active,
d70b171e-7488-4d52-8cad-bbc581dbf16e=active,
1f2e9989-9ab3-43d5-971d-568b8feca918=active}
2020-01-13 22:24:59,850-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterServersListVDSCommand,
return: [10.12.0.222/24:CONNECTED, 10.11.0.220:CONNECTED,
orchard1.grove.silverorange.com:CONNECTED], log id: 43f1dd82
2020-01-13 22:24:59,852-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] START,
GlusterVolumesListVDSCommand(HostName = Orchard0,
GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 263be6f8
2020-01-13 22:25:00,019-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
ConnectStoragePoolVDSCommand, return: , log id: 2b397b31
2020-01-13 22:25:00,036-04 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) []
hostFromVds::selectedVds - 'Orchard1', spmStatus 'Free', storage pool
'Default', storage pool version '4.3'
2020-01-13 22:25:00,056-04 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] starting spm on vds
'Orchard1', storage pool 'Default', prevId '-1', LVER '-1'
2020-01-13 22:25:00,057-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
SpmStartVDSCommand(HostName = Orchard1,
SpmStartVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', prevId='-1',
prevLVER='-1', storagePoolFormatType='V4', recoveryMode='Manual',
SCSIFencing='false'}), log id: 3dea111a
2020-01-13 22:25:00,065-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
started: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe'
2020-01-13 22:25:00,500-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [4f66c75b] FINISH, GlusterVolumesListVDSCommand,
return:
{3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@743f4102,
71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@cad5f8f4,
752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@f9f55499,
c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@7a2c16da,
79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@33e1ab37,
095fd8fc-5322-4741-8805-fc0bb64b554f=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@b28195c0},
log id: 263be6f8
2020-01-13 22:25:03,089-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Failed in
'HSMGetTaskStatusVDS' method
2020-01-13 22:25:03,090-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
ended: taskId '671d5904-e062-4d45-9eb4-83a6f13657fe' task status 'finished'
2020-01-13 22:25:03,090-04 ERROR
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] Start SPM Task
failed - result: 'cleanSuccess', message: VDSGenericException:
VDSErrorException: Failed to HSMGetTaskStatusVDS, error = TaskManager
error, unable to add task: ('Task id already in use:
7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',), code = 100
2020-01-13 22:25:03,104-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] spmStart polling
ended, spm status: Free
2020-01-13 22:25:03,105-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] START,
HSMClearTaskVDSCommand(HostName = Orchard1,
HSMTaskGuidBaseVDSCommandParameters:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d',
taskId='671d5904-e062-4d45-9eb4-83a6f13657fe'}), log id: 5335606f
2020-01-13 22:25:03,110-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
HSMClearTaskVDSCommand, return: , log id: 5335606f
2020-01-13 22:25:03,110-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [] FINISH,
SpmStartVDSCommand, return: SpmStatusResult:{SPM Id='-1', SPM LVER='-1',
SPM Status='Free'}, log id: 3dea111a
2020-01-13 22:25:03,117-04 INFO
[org.ovirt.engine.core.bll.storage.pool.SetStoragePoolStatusCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Running
command: SetStoragePoolStatusCommand internal: true. Entities affected :
ID: a45e442e-9989-11e8-b0e4-00163e4bf18a Type: StoragePool
2020-01-13 22:25:03,179-04 ERROR
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac]
IrsBroker::Failed::GetStoragePoolInfoVDS: IRSGenericException:
IRSErrorException: SpmStart failed
2020-01-13 22:25:03,205-04 INFO
[org.ovirt.engine.core.vdsbroker.irsbroker.IrsProxy]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Irs placed
on server 'fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d' failed. Proceed Failover
2020-01-13 22:25:03,223-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] START,
ConnectStoragePoolVDSCommand(HostName = Orchard2,
ConnectStoragePoolVDSCommandParameters:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237',
vdsId='fd0752d8-2d41-45b0-887a-0ffacbb8a237',
storagePoolId='a45e442e-9989-11e8-b0e4-00163e4bf18a', masterVersion='1'}),
log id: 6a7302e9
2020-01-13 22:25:03,224-04 INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand]
(EE-ManagedThreadFactory-engineScheduled-Thread-72) [1c8884ac] Executing
with domain map: {edc68a7c-7604-47e6-89bc-3738d727e8fc=active,
23c22a0f-0482-425e-8ada-730cf8ec0751=active,
390c0320-e843-4ff3-a4bb-a9973058447f=active,
fb43d33a-82c8-44cb-8169-090cd0d8f56e=active,
d70b171e-7488-4d52-8cad-bbc581dbf16e=active,
1f2e9989-9ab3-43d5-971d-568b8feca918=active}
-----------------------------
vdsm.log from one of my hosts:
2020-01-13 22:26:12,434-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,435-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 88772af7-8cf5-433e-8be3-8d0adf0bbf04
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
in _load
self._loadJobMetaFile(taskDir, jn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
in _loadJobMetaFile
self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/88772af7-8cf5-433e-8be3-8d0adf0bbf04/88772af7-8cf5-433e-8be3-8d0adf0bbf04.job.0',)
2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,462-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: a9b11e33-9b93-46a0-a36e-85063fd53ebe
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
in _load
self._loadRecoveryMetaFile(taskDir, rn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
in _loadRecoveryMetaFile
self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a9b11e33-9b93-46a0-a36e-85063fd53ebe/a9b11e33-9b93-46a0-a36e-85063fd53ebe.recover.0',)
2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,476-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 650f2df4-6489-47e2-af5d-db86a22f01c0
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
in _load
self._loadRecoveryMetaFile(taskDir, rn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
in _loadRecoveryMetaFile
self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/650f2df4-6489-47e2-af5d-db86a22f01c0/650f2df4-6489-47e2-af5d-db86a22f01c0.recover.0',)
2020-01-13 22:26:12,487-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,488-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
in _load
self._loadJobMetaFile(taskDir, jn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
in _loadJobMetaFile
self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e/a2e86fcc-8e7e-4e6d-bf5e-5ac61a98169e.job.0',)
2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,493-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737,
in _load
self._loadTaskMetaFile(taskDir)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688,
in _loadTaskMetaFile
self._loadMetaFile(taskFile, self, Task.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743/eb8b4f7a-9b5c-46f6-aaa7-7ef05dbf1743.task',)
2020-01-13 22:26:12,505-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,506-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 75318d21-45b2-4dbd-985c-a7851a10a463
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
in _load
self._loadJobMetaFile(taskDir, jn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
in _loadJobMetaFile
self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/75318d21-45b2-4dbd-985c-a7851a10a463/75318d21-45b2-4dbd-985c-a7851a10a463.job.0',)
2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,517-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 1779b352-022c-49a3-9388-2f688d33cdab
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
in _load
self._loadJobMetaFile(taskDir, jn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
in _loadJobMetaFile
self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/1779b352-022c-49a3-9388-2f688d33cdab/1779b352-022c-49a3-9388-2f688d33cdab.job.0',)
2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,532-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 0302036a-7d99-4685-befb-6fee1602feaf
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 746,
in _load
self._loadJobMetaFile(taskDir, jn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 696,
in _loadJobMetaFile
self._loadMetaFile(taskFile, self.jobs[n], Job.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/0302036a-7d99-4685-befb-6fee1602feaf/0302036a-7d99-4685-befb-6fee1602feaf.job.0',)
2020-01-13 22:26:12,545-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,546-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 24a825b9-d48d-4134-8aa1-b4db7a9c6ab1
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 737,
in _load
self._loadTaskMetaFile(taskDir)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 688,
in _loadTaskMetaFile
self._loadMetaFile(taskFile, self, Task.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.backup/24a825b9-d48d-4134-8aa1-b4db7a9c6ab1.task',)
2020-01-13 22:26:12,560-0400 ERROR (tasks/5) [storage.TaskManager.Task]
Unexpected error (task:652)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 631,
in _loadMetaFile
for line in getProcPool().readLines(filename):
File "/usr/lib/python2.7/site-packages/vdsm/storage/outOfProcess.py",
line 334, in readLines
return ioproc.readlines(path)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 555,
in readlines
return self.readfile(path, direct).splitlines()
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 543,
in readfile
"direct": direct}, self.timeout)
File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 448,
in _sendCommand
raise OSError(errcode, errstr)
OSError: [Errno 2] No such file or directory
2020-01-13 22:26:12,561-0400 ERROR (tasks/5) [storage.TaskManager]
taskManager: Skipping directory: 90d88529-a051-4acd-bef2-d0aa034c15de
(taskManager:222)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
214, in loadDumpedTasks
t = Task.loadTask(store, taskID)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 1162,
in loadTask
t._load(store, ext)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 751,
in _load
self._loadRecoveryMetaFile(taskDir, rn)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 705,
in _loadRecoveryMetaFile
self._loadMetaFile(taskFile, self.recoveries[n], Recovery.fields)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 653,
in _loadMetaFile
raise se.TaskMetaDataLoadError(filename)
TaskMetaDataLoadError: Can't load Task Metadata:
('/rhev/data-center/a45e442e-9989-11e8-b0e4-00163e4bf18a/mastersd/master/tasks/90d88529-a051-4acd-bef2-d0aa034c15de/90d88529-a051-4acd-bef2-d0aa034c15de.recover.0',)
2020-01-13 22:26:12,607-0400 ERROR (tasks/5) [storage.StoragePool]
Unexpected error (sp:383)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in
startSpm
self.taskMng.recoverDumpedTasks()
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
227, in recoverDumpedTasks
self.queueRecovery(task)
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
48, in queueRecovery
return self._queueTask(task, task.recover)
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
54, in _queueTask
'Task id already in use: {0}'.format(task.id))
AddTaskError: TaskManager error, unable to add task: ('Task id already in
use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)
2020-01-13 22:26:12,608-0400 ERROR (tasks/5) [storage.StoragePool] failed:
TaskManager error, unable to add task: ('Task id already in use:
7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',) (sp:384)
2020-01-13 22:26:12,635-0400 ERROR (tasks/5) [storage.TaskManager.Task]
(Task='3c7de2a0-597c-4ebe-b4de-689dba26045b') Unexpected error (task:875)
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882,
in _run
return fn(*args, **kargs)
File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 336,
in run
return self.cmd(*self.argslist, **self.argsdict)
File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 378, in
startSpm
self.taskMng.recoverDumpedTasks()
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
227, in recoverDumpedTasks
self.queueRecovery(task)
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
48, in queueRecovery
return self._queueTask(task, task.recover)
File "/usr/lib/python2.7/site-packages/vdsm/storage/taskManager.py", line
54, in _queueTask
'Task id already in use: {0}'.format(task.id))
AddTaskError: TaskManager error, unable to add task: ('Task id already in
use: 7df9eeb4-f7a8-4de4-b3f7-5e5607d48dda',)
4 years, 3 months
