[ovirt-users] Re: Restoring hosted engine from backup fails on new FC storage domain creation

On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen <juhani.rautiainen(a)gmail.com&gt; wrote: > > > Hi! > > > > This is from ansible logs: > 2021-04-27 22:20:38,286+0300 ERROR ansible failed { > > "ansible_host": "localhost", > "ansible_playbook": > > "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml", > > "ansible_result": { > > "_ansible_no_log": false, > "changed": false, > "exception": "Traceback (most recent call last):\n File > > \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage > _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_ > storage_domain.py\",

> \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage > _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/o > virt.py\",

> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line > 26182, in add\n return self._internal_add(storage_domain, headers, > query, wait)\n File > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232, > in _internal_add\n return future.wait() if wait else future\n File > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55, > in wait\n return self._code(response)\n File > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229, > in callback\n self._check_fault(response)\n File > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132, > in _check_fault\n self._raise_error(response, body)\n File > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118, > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is > \"Operation Failed\". Fault detail is \"[Physical device > initialization failed. Please check that the device is empty and > accessible by the host.]\". HTTP response code is 400.\n", > > "invocation": { > > "module_args": { > > "backup": null, > "comment": null, > "critical_space_action_blocker": null, > "data_center": "Default", > "description": null, > "destroy": null, > "discard_after_delete": true, > "domain_function": "data", > "fcp": { > > "lun_id": "36000d31005b4f6000000000000000029" > > }, > "fetch_nested": false, > "format": null, > "glusterfs": null, > "host": "ovirt02.virt.local", > "id": null, > "iscsi": null, > "localfs": null, > "managed_block_storage": null, > "name": "hosted_storage", > "nested_attributes": [], > "nfs": null, > "poll_interval": 3, > "posixfs": null, > "state": "unattached", > "timeout": 180, > "wait": true, > "warning_low_space": null, > "wipe_after_delete": null > > } > > }, > "msg": "Fault reason is \"Operation Failed\". Fault detail is > > \"[Physical device initialization failed. Please check that the device > is empty and accessible by the host.]\". HTTP response code is 400." > > }, > "ansible_task": "Add Fibre Channel storage domain", > "ansible_type": "task", > "status": "FAILED", > > > > This seem to happen same time: > > > > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH > getDeviceList return={'devList': [{'GUID': > '36000d31005b4f6000000000000000029', 'capacity': '536870912000', > 'devtype': 'FCP', 'fwrev': '0704', 'logicalblocksize': '512', > 'pathlist': [], 'pathstatus': [{'physdev': 'sdu', 'state': 'active', > 'capacity': '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': > 'sds', 'state': 'active', 'capacity': '536870912000', 'lun': '3', > 'type': 'FCP'}, {'physdev': 'sdv', 'state': 'active', 'capacity': > '536870912000', 'lun': '3', 'type': 'FCP'}, {'physdev': 'sdt', > 'state': 'active', 'capacity': '536870912000', 'lun': '3', 'type': > 'FCP'}], 'physicalblocksize': '4096', 'productID': 'Compellent Vol', > 'pvUUID': '', 'pvsize': '', 'serial': > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID': 'COMPELNT', > 'vgUUID': '', 'discard_max_bytes': 268435456, 'discard_zeroes_data': > 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472, > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b, > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54) > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5', > devlist=['36000d31005b4f6000000000000000029'], force=False, > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780, > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48) > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command > with specific filter failed or returned no data, retrying with a wider > filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices { > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 > filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|", "r|.*|"] > > hints="none" obtain_device_list_from_udev=0 } global { > > locking_type=1 prioritise_write_locks=1 wait_for_locks=1 > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', > '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore', > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[] > err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029 > exclusively. Mounted filesystem?", " Can't open > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted > filesystem?"] (lvm:511) It seems like this ^^^ is the problem. I suggest to check around e.g. with lsof etc., and/or search the net for something like

"pvcreate" "Can't open" "exclusively. Mounted filesystem?" Best regards, > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2 > tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices > { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1 > write_cache_state=0 disable_after_error_count=3 > filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36 > 000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f60000000000000 > 0001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",

> hints="none" obtain_device_list_from_udev=0 } global { locking_type=1 > prioritise_write_locks=1 wait_for_locks=1 > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }', > '--metadatasize', '128m', '--metadatacopies', '2', '--metadataignore', > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=[" > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. > Mounted filesystem?", " Can't open > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted > filesystem?"] (lvm:538) > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate > failed with rc=5 (lvm:1142) > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], [" > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively. > Mounted filesystem?", " Can't open > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted > filesystem?"] (lvm:1143) > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH > createVG error=Failed to initialize physical device: > ("['/dev/mapper/36000d31005b4f6000000000000000029']",) > from=::ffff:192.168.222.89,57472, flow_id=6dce1780, > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52) > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) > [storage.TaskManager.Task] > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error > (task:880) > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) > [storage.TaskManager.Task] > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is > aborted: 'value=Failed to initialize physical device: > ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",) > abortedcode=601' (task:1190) > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher] > FINISH createVG error=Failed to initialize physical device: > ("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83) > > > > I tried clearing the disk but from above it seems that something else > is wonky here. > > > > Thanks, > Juhani > > > > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com&gt; > wrote: > > > > > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > Story of the problems continues. Finally shut everything down, got > > > storage domains to maintenance and then this happens: > > > > > > > > > > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is > > > "[Physical device initialization failed. Please check that the device > > > is empty and accessible by the host.]". HTTP response code is 400. > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": > > > "Fault reason is \"Operation Failed\". Fault detail is \"[Physical > > > device initialization failed. Please check that the device is empty > > > and accessible by the host.]\". HTTP response code is 400."} > > > > > > > > Can you please check/share relevant parts of vdsm.log when this > > happens? > > Thanks. > > > > > > > > > > > > > > > No amount zeroing device helped. > > > > > > > > > > > > Next plan: Find out if I can restore backup to standalone server. If > > > that fails that's the end of oVirt for me. It worked fine for a > > > couple > > > of years but this update hassle is too much. Should have gone until > > > the end with 4.3. > > > > > > > > > > > > Thanks, > > > Juhani > > > > > > > > > > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen > > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > Hmm. Is it possible that when the other node is still running v4.3 > > > > this operation can't be completed as it doesn't know how to do it. > > > > > > > > > > > > > > > > Thanks, > > > > Juhani > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen > > > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > > > > > It seems that it is not supported in oVirt yet? I got this > > > > > response > > > > > when I tried to change master with those storage domain that I > > > > > have: > > > > > > > > > > > > > > > > > > > > <?xml version="1.0" encoding="UTF-8" standalone="yes"?> > > > > > <fault> > > > > > > > > > > <detail>[Cannot switch master storage domain. Switch master > > > > > > > > > > storage domain operation is not supported.]</detail> > > > > > > > > > > <reason>Operation Failed</reason> > > > > > > > > > > </fault> > > > > > > > > > > > > > > > > > > > > So is this really the only way to do this: shutdown everything and > > > > > put > > > > > other storage domains into maintenance? It would have been nice > > > > > if > > > > > this information was in the upgrade guide. It made this seem so > > > > > easy > > > > > and simple... > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > Juhani > > > > > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen > > > > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > > > > > > > > > Thanks this looks like what I'm looking for. I'm still wondering > > > > > > how > > > > > > to use this. I have LUN just for new hosted storage. Ansible > > > > > > created > > > > > > storage domain to it correctly but just can't activate it. So is > > > > > > the > > > > > > idea that I activate this unattached hosted_storage domain and > > > > > > try to > > > > > > use API to make it master? I attached a screen shot how it > > > > > > looks > > > > > > currently. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David > > > > > > <didi(a)redhat.com&gt; wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen > > > > > > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > To continue. I noticed that another storage domain took the > > > > > > > > data > > > > > > > > (master) now. I saw one advice that you can force change by > > > > > > > > putting > > > > > > > > the storage domain to maintenance mode. Problem is that > > > > > > > > there are VM's > > > > > > > > running on these domains. How is this supposed to work > > > > > > > > during the > > > > > > > > restore? > > > > > > > > > > > > > > > > > > > > > > > > > > > > There is a recent change [1] by Shani (Cced) that should allow > > > > > > > you > > > > > > > to choose another storage domain as master. So you can create > > > > > > > a new > > > > > > > (temporary?) SD with the correct compatibility level and then > > > > > > > set it > > > > > > > to master. > > > > > > > > > > > > > > > > > > > > > > > > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923 > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Juhani > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen > > > > > > > > <juhani.rautiainen(a)gmail.com&gt; wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I started the upgrade from 4.3->4.4. Now I'm stuck as > > > > > > > > > restoring the > > > > > > > > > backup fails to create the correct storage domain for > > > > > > > > > Hosted Engine. > > > > > > > > > How can I create one? Error from ansible task is: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation > > > > > > > > > Failed". Fault > > > > > > > > > detail is "[Domain format is different from master storage > > > > > > > > > domain > > > > > > > > > format]". HTTP response code is 400. > > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": > > > > > > > > > false, "msg": > > > > > > > > > "Fault reason is \"Operation Failed\". Fault detail is > > > > > > > > > \"[Domain > > > > > > > > > format is different from master storage domain format]\". > > > > > > > > > HTTP > > > > > > > > > response code is 400."} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And from UI I can see that what has been created is a data > > > > > > > > > domain. Not > > > > > > > > > data (master) domain as old one was. Now I'm stuck here. > > > > > > > > > This is a > > > > > > > > > fibre channel system where I'm trying to do this if it is > > > > > > > > > relevant. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > What I could find from the logs is this: > > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG > > > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils > > > > > > > > > ansible_utils._process_output:105 storage_domain_details: > > > > > > > > > {'changed': > > > > > > > > > False, 'ovirt_storage_domains': [{'href': > > > > > > > > > '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534- > > > > > > > > > 6f1a4084fdf4', > > > > > > > > > 'comment': '', 'description': '', 'id': > > > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': > > > > > > > > > 'hosted_storage', > > > > > > > > > 'available': 531502202880, 'backup': False, 'block_size': > > > > > > > > > 512, > > > > > > > > > 'committed': 0, 'critical_space_action_blocker': 5, > > > > > > > > > 'discard_after_delete': True, 'disk_profiles': [], > > > > > > > > > 'disk_snapshots': > > > > > > > > > [], 'disks': [], 'external_status': 'ok', 'master': > > > > > > > > > False, > > > > > > > > > 'permissions': [], 'status': 'unattached', 'storage': > > > > > > > > > {'type': 'fcp', > > > > > > > > > 'volume_group': {'id': > > > > > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit', > > > > > > > > > 'logical_units': [{'discard_max_size': 268435456, > > > > > > > > > 'discard_zeroes_data': False, 'id': > > > > > > > > > '36000d31005b4f6000000000000000029', 'lun_mapping': 3, > > > > > > > > > 'paths': 0, > > > > > > > > > 'product_id': 'Compellent Vol', 'serial': > > > > > > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': > > > > > > > > > 536870912000, > > > > > > > > > 'storage_domain_id': > > > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', > > > > > > > > > 'vendor_id': 'COMPELNT', 'volume_group_id': > > > > > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, > > > > > > > > > 'storage_connections': > > > > > > > > > [], 'storage_format': 'v5', 'supports_discard': True, > > > > > > > > > 'supports_discard_zeroes_data': False, 'templates': [], > > > > > > > > > 'type': > > > > > > > > > 'data', 'used': 4294967296, 'vms': [], > > > > > > > > > 'warning_low_space_indicator': > > > > > > > > > 10, 'wipe_after_delete': False}], 'failed': False} > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 'master': False? I'm not sure if this creation or check. I > > > > > > > > > tried this > > > > > > > > > opration twice. I removed the new hosted_storage and > > > > > > > > > remove also old > > > > > > > > > hosted_engine domain on second try to make sure that it > > > > > > > > > doesn't > > > > > > > > > prevent creation of another master. No luck with that. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Juhani > > > > > > > > > > > > > > > > _______________________________________________ > > > > > > > > Users mailing list -- users(a)ovirt.org > > > > > > > > To unsubscribe send an email to users-leave(a)ovirt.org > > > > > > > > Privacy Statement: > > > > > > > > https://www.ovirt.org/privacy-policy.html > > > > > > > > oVirt Code of Conduct: > > > > > > > > https://www.ovirt.org/community/about/community-guidelines/ > > > > > > > >

> > > > > > > > https://lists.ovirt.org/archives/list/users@ovirt.org/messa > > > > > > > > ge/TZHLFDIYRBQFVDLEVG4SWVZPXSVO7SKK/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Didi > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Didi > > > > > > -- Didi _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/privacy-policy.html oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2R53V3IMD5JKK B22A3G6OWME3GXDDUFW/