3:16 a.m.
I found these using dmsetup ls -tree:
6db20b74--512d--4a70--994e--8923d9e1e50b-master (253:21)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
6db20b74--512d--4a70--994e--8923d9e1e50b-inbox (253:17)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
db64924d--aa94--48ad--809a--3d235e84686b-ids (253:22)
└─36000d31005b4f600000000000000001d (253:12)
├─ (8:112)
├─ (8:96)
├─ (8:64)
└─ (8:48)
df259f5a--e1bd--4a93--bd6e--bace450625dc-master (253:35)
└─36000d31005b4f600000000000000001a (253:13)
├─ (8:144)
├─ (8:128)
├─ (8:80)
└─ (8:32)
6db20b74--512d--4a70--994e--8923d9e1e50b-outbox (253:18)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
ID matches the one in ansible error and vdsm log. Are these done by
oVirt? I mean it's pretty hard to know what to free if you don't know
what should be and should not be there. And if these can be cleared
how do you do that?
By the way does anyone know how I can connect back to that HE that is
hosted locally during installation (URL like
https://ovirt02.virt.local:6900/ovirt-engine/). It seems the
connection disappeared this morning. Now I'm totally blind what's
going on in both nodes... Maybe I have to clear this SHE recovery and
start from the scratch.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 9:57 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Hi!
> >
> > This is from ansible logs:
> > 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
> > "ansible_host": "localhost",
> > "ansible_playbook":
> > "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
> > "ansible_result": {
> > "_ansible_no_log": false,
> > "changed": false,
> > "exception": "Traceback (most recent call last):\n File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
> > line 804, in main\n File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
> > line 651, in create\n **kwargs\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> > 26182, in add\n return self._internal_add(storage_domain, headers,
> > query, wait)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
232,
> > in _internal_add\n return future.wait() if wait else future\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
> > in wait\n return self._code(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
229,
> > in callback\n self._check_fault(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
132,
> > in _check_fault\n self._raise_error(response, body)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
118,
> > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> > \"Operation Failed\". Fault detail is \"[Physical device
> > initialization failed. Please check that the device is empty and
> > accessible by the host.]\". HTTP response code is 400.\n",
> > "invocation": {
> > "module_args": {
> > "backup": null,
> > "comment": null,
> > "critical_space_action_blocker": null,
> > "data_center": "Default",
> > "description": null,
> > "destroy": null,
> > "discard_after_delete": true,
> > "domain_function": "data",
> > "fcp": {
> > "lun_id":
"36000d31005b4f6000000000000000029"
> > },
> > "fetch_nested": false,
> > "format": null,
> > "glusterfs": null,
> > "host": "ovirt02.virt.local",
> > "id": null,
> > "iscsi": null,
> > "localfs": null,
> > "managed_block_storage": null,
> > "name": "hosted_storage",
> > "nested_attributes": [],
> > "nfs": null,
> > "poll_interval": 3,
> > "posixfs": null,
> > "state": "unattached",
> > "timeout": 180,
> > "wait": true,
> > "warning_low_space": null,
> > "wipe_after_delete": null
> > }
> > },
> > "msg": "Fault reason is \"Operation Failed\".
Fault detail is
> > \"[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]\". HTTP response code is 400."
> > },
> > "ansible_task": "Add Fibre Channel storage domain",
> > "ansible_type": "task",
> > "status": "FAILED",
> >
> > This seem to happen same time:
> >
> > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> > getDeviceList return={'devList': [{'GUID':
> > '36000d31005b4f6000000000000000029', 'capacity':
'536870912000',
> > 'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
> > 'pathlist': [], 'pathstatus': [{'physdev':
'sdu', 'state': 'active',
> > 'capacity': '536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev':
> > 'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
> > 'type': 'FCP'}, {'physdev': 'sdv',
'state': 'active', 'capacity':
> > '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev': 'sdt',
> > 'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
> > 'FCP'}], 'physicalblocksize': '4096',
'productID': 'Compellent Vol',
> > 'pvUUID': '', 'pvsize': '', 'serial':
> > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
> > 'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
> > 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
> > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> > devlist=['36000d31005b4f6000000000000000029'], force=False,
> > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> > with specific filter failed or returned no data, retrying with a wider
> > filter, cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> > hints="none" obtain_device_list_from_udev=0 } global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
out=[]
> > err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
> > exclusively. Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:511)
>
> It seems like this ^^^ is the problem. I suggest to check around e.g.
> with lsof etc., and/or search the net for something like
>
> "pvcreate" "Can't open" "exclusively. Mounted
filesystem?"
>
> Best regards,
>
> > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> > tries have failed: cmd=['/sbin/lvm', 'pvcreate',
'--config', 'devices
> > { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> >
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
> > "r|.*|"] hints="none" obtain_device_list_from_udev=0 }
global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
err=["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:538)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> > failed with rc=5 (lvm:1142)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:1143)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> > createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> > from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
> > (task:880)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
> > aborted: 'value=Failed to initialize physical device:
> > ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> > abortedcode=601' (task:1190)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> > FINISH createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
(dispatcher:83)
> >
> > I tried clearing the disk but from above it seems that something else
> > is wonky here.
> >
> > Thanks,
> > Juhani
> >
> > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > >
> > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > Story of the problems continues. Finally shut everything down, got
> > > > storage domains to maintenance and then this happens:
> > > >
> > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is
> > > > "[Physical device initialization failed. Please check that the
device
> > > > is empty and accessible by the host.]". HTTP response code is
400.
> > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
false, "msg":
> > > > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > > > device initialization failed. Please check that the device is empty
> > > > and accessible by the host.]\". HTTP response code is
400."}
> > >
> > > Can you please check/share relevant parts of vdsm.log when this happens?
> > > Thanks.
> > >
> > > >
> > > > No amount zeroing device helped.
> > > >
> > > > Next plan: Find out if I can restore backup to standalone server. If
> > > > that fails that's the end of oVirt for me. It worked fine for a
couple
> > > > of years but this update hassle is too much. Should have gone until
> > > > the end with 4.3.
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Hmm. Is it possible that when the other node is still running
v4.3
> > > > > this operation can't be completed as it doesn't know how
to do it.
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > It seems that it is not supported in oVirt yet? I got this
response
> > > > > > when I tried to change master with those storage domain that
I have:
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > > > <fault>
> > > > > > <detail>[Cannot switch master storage domain.
Switch master
> > > > > > storage domain operation is not supported.]</detail>
> > > > > > <reason>Operation Failed</reason>
> > > > > > </fault>
> > > > > >
> > > > > > So is this really the only way to do this: shutdown
everything and put
> > > > > > other storage domains into maintenance? It would have been
nice if
> > > > > > this information was in the upgrade guide. It made this seem
so easy
> > > > > > and simple...
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks this looks like what I'm looking for.
I'm still wondering how
> > > > > > > to use this. I have LUN just for new hosted storage.
Ansible created
> > > > > > > storage domain to it correctly but just can't
activate it. So is the
> > > > > > > idea that I activate this unattached hosted_storage
domain and try to
> > > > > > > use API to make it master? I attached a screen shot how
it looks
> > > > > > > currently.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani
Rautiainen
> > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > To continue. I noticed that another storage
domain took the data
> > > > > > > > > (master) now. I saw one advice that you can
force change by putting
> > > > > > > > > the storage domain to maintenance mode.
Problem is that there are VM's
> > > > > > > > > running on these domains. How is this
supposed to work during the
> > > > > > > > > restore?
> > > > > > > >
> > > > > > > > There is a recent change [1] by Shani (Cced) that
should allow you
> > > > > > > > to choose another storage domain as master. So you
can create a new
> > > > > > > > (temporary?) SD with the correct compatibility
level and then set it
> > > > > > > > to master.
> > > > > > > >
> > > > > > > > [1]
https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Juhani
> > > > > > > > >
> > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani
Rautiainen
> > > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi!
> > > > > > > > > >
> > > > > > > > > > I started the upgrade from 4.3->4.4.
Now I'm stuck as restoring the
> > > > > > > > > > backup fails to create the correct
storage domain for Hosted Engine.
> > > > > > > > > > How can I create one? Error from ansible
task is:
> > > > > > > > > >
> > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason
is "Operation Failed". Fault
> > > > > > > > > > detail is "[Domain format is
different from master storage domain
> > > > > > > > > > format]". HTTP response code is
400.
> > > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED!
=> {"changed": false, "msg":
> > > > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > > > format is different from master storage
domain format]\". HTTP
> > > > > > > > > > response code is 400."}
> > > > > > > > > >
> > > > > > > > > > And from UI I can see that what has been
created is a data domain. Not
> > > > > > > > > > data (master) domain as old one was. Now
I'm stuck here. This is a
> > > > > > > > > > fibre channel system where I'm
trying to do this if it is relevant.
> > > > > > > > > >
> > > > > > > > > > What I could find from the logs is
this:
> > > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > > >
otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage',
> > > > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > > > 'permissions': [],
'status': 'unattached', 'storage': {'type':
'fcp',
> > > > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > > >
'36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths':
0,
> > > > > > > > > > 'product_id': 'Compellent
Vol', 'serial':
> > > > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > > > [], 'storage_format':
'v5', 'supports_discard': True,
> > > > > > > > > > 'supports_discard_zeroes_data':
False, 'templates': [], 'type':
> > > > > > > > > > 'data', 'used':
4294967296, 'vms': [], 'warning_low_space_indicator':
> > > > > > > > > > 10, 'wipe_after_delete':
False}], 'failed': False}
> > > > > > > > > >
> > > > > > > > > > 'master': False? I'm not
sure if this creation or check. I tried this
> > > > > > > > > > opration twice. I removed the new
hosted_storage and remove also old
> > > > > > > > > > hosted_engine domain on second try to
make sure that it doesn't
> > > > > > > > > > prevent creation of another master. No
luck with that.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Juhani
> > > > > > > > >
_______________________________________________
> > > > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > > > To unsubscribe send an email to
users-leave(a)ovirt.org
> > > > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Didi
> > > > > > > >
> > > >
> > >
> > >
> > > --
> > > Didi
> > >
> >
>
>
> --
> Didi
>