9:29 a.m.
Hi!
This is from ansible logs:
2021-04-27 22:20:38,286+0300 ERROR ansible failed {
"ansible_host": "localhost",
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
"ansible_result": {
"_ansible_no_log": false,
"changed": false,
"exception": "Traceback (most recent call last):\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 804, in main\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
line 651, in create\n **kwargs\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
26182, in add\n return self._internal_add(storage_domain, headers,
query, wait)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232,
in _internal_add\n return future.wait() if wait else future\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
in wait\n return self._code(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229,
in callback\n self._check_fault(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132,
in _check_fault\n self._raise_error(response, body)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118,
in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
\"Operation Failed\". Fault detail is \"[Physical device
initialization failed. Please check that the device is empty and
accessible by the host.]\". HTTP response code is 400.\n",
"invocation": {
"module_args": {
"backup": null,
"comment": null,
"critical_space_action_blocker": null,
"data_center": "Default",
"description": null,
"destroy": null,
"discard_after_delete": true,
"domain_function": "data",
"fcp": {
"lun_id": "36000d31005b4f6000000000000000029"
},
"fetch_nested": false,
"format": null,
"glusterfs": null,
"host": "ovirt02.virt.local",
"id": null,
"iscsi": null,
"localfs": null,
"managed_block_storage": null,
"name": "hosted_storage",
"nested_attributes": [],
"nfs": null,
"poll_interval": 3,
"posixfs": null,
"state": "unattached",
"timeout": 180,
"wait": true,
"warning_low_space": null,
"wipe_after_delete": null
}
},
"msg": "Fault reason is \"Operation Failed\". Fault
detail is
\"[Physical device initialization failed. Please check that the device
is empty and accessible by the host.]\". HTTP response code is 400."
},
"ansible_task": "Add Fibre Channel storage domain",
"ansible_type": "task",
"status": "FAILED",
This seem to happen same time:
2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
getDeviceList return={'devList': [{'GUID':
'36000d31005b4f6000000000000000029', 'capacity': '536870912000',
'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
'pathlist': [], 'pathstatus': [{'physdev': 'sdu',
'state': 'active',
'capacity': '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev':
'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev': 'sdv', 'state':
'active', 'capacity':
'536870912000', 'lun': '3', 'type': 'FCP'},
{'physdev': 'sdt',
'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
'FCP'}], 'physicalblocksize': '4096', 'productID':
'Compellent Vol',
'pvUUID': '', 'pvsize': '', 'serial':
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
devlist=['36000d31005b4f6000000000000000029'], force=False,
options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
with specific filter failed or returned no data, retrying with a wider
filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[]
err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
exclusively. Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:511)
2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices
{ preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"] hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:538)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
failed with rc=5 (lvm:1142)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:1143)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",)
from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
(task:880)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
aborted: 'value=Failed to initialize physical device:
("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
abortedcode=601' (task:1190)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
FINISH createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83)
I tried clearing the disk but from above it seems that something else
is wonky here.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Story of the problems continues. Finally shut everything down, got
> > storage domains to maintenance and then this happens:
> >
> > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
> > "[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]". HTTP response code is 400.
> > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > device initialization failed. Please check that the device is empty
> > and accessible by the host.]\". HTTP response code is 400."}
>
> Can you please check/share relevant parts of vdsm.log when this happens?
> Thanks.
>
> >
> > No amount zeroing device helped.
> >
> > Next plan: Find out if I can restore backup to standalone server. If
> > that fails that's the end of oVirt for me. It worked fine for a couple
> > of years but this update hassle is too much. Should have gone until
> > the end with 4.3.
> >
> > Thanks,
> > Juhani
> >
> > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > Hmm. Is it possible that when the other node is still running v4.3
> > > this operation can't be completed as it doesn't know how to do it.
> > >
> > > Thanks,
> > > Juhani
> > >
> > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > It seems that it is not supported in oVirt yet? I got this response
> > > > when I tried to change master with those storage domain that I have:
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > <fault>
> > > > <detail>[Cannot switch master storage domain. Switch master
> > > > storage domain operation is not supported.]</detail>
> > > > <reason>Operation Failed</reason>
> > > > </fault>
> > > >
> > > > So is this really the only way to do this: shutdown everything and
put
> > > > other storage domains into maintenance? It would have been nice if
> > > > this information was in the upgrade guide. It made this seem so easy
> > > > and simple...
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Thanks this looks like what I'm looking for. I'm still
wondering how
> > > > > to use this. I have LUN just for new hosted storage. Ansible
created
> > > > > storage domain to it correctly but just can't activate it. So
is the
> > > > > idea that I activate this unattached hosted_storage domain and
try to
> > > > > use API to make it master? I attached a screen shot how it looks
> > > > > currently.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > To continue. I noticed that another storage domain took
the data
> > > > > > > (master) now. I saw one advice that you can force
change by putting
> > > > > > > the storage domain to maintenance mode. Problem is that
there are VM's
> > > > > > > running on these domains. How is this supposed to work
during the
> > > > > > > restore?
> > > > > >
> > > > > > There is a recent change [1] by Shani (Cced) that should
allow you
> > > > > > to choose another storage domain as master. So you can
create a new
> > > > > > (temporary?) SD with the correct compatibility level and
then set it
> > > > > > to master.
> > > > > >
> > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Juhani
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I started the upgrade from 4.3->4.4. Now
I'm stuck as restoring the
> > > > > > > > backup fails to create the correct storage domain
for Hosted Engine.
> > > > > > > > How can I create one? Error from ansible task is:
> > > > > > > >
> > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is
"Operation Failed". Fault
> > > > > > > > detail is "[Domain format is different from
master storage domain
> > > > > > > > format]". HTTP response code is 400.
> > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > format is different from master storage domain
format]\". HTTP
> > > > > > > > response code is 400."}
> > > > > > > >
> > > > > > > > And from UI I can see that what has been created
is a data domain. Not
> > > > > > > > data (master) domain as old one was. Now I'm
stuck here. This is a
> > > > > > > > fibre channel system where I'm trying to do
this if it is relevant.
> > > > > > > >
> > > > > > > > What I could find from the logs is this:
> > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'name': 'hosted_storage',
> > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > > > > 'data', 'used': 4294967296,
'vms': [], 'warning_low_space_indicator':
> > > > > > > > 10, 'wipe_after_delete': False}],
'failed': False}
> > > > > > > >
> > > > > > > > 'master': False? I'm not sure if this
creation or check. I tried this
> > > > > > > > opration twice. I removed the new hosted_storage
and remove also old
> > > > > > > > hosted_engine domain on second try to make sure
that it doesn't
> > > > > > > > prevent creation of another master. No luck with
that.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Juhani
> > > > > > > _______________________________________________
> > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Didi
> > > > > >
> >
>
>
> --
> Didi
>