1:58 a.m.
Hi!
I started the upgrade from 4.3->4.4. Now I'm stuck as restoring the
backup fails to create the correct storage domain for Hosted Engine.
How can I create one? Error from ansible task is:
[ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is "[Domain format is different from master storage domain
format]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"Fault reason is \"Operation Failed\". Fault detail is \"[Domain
format is different from master storage domain format]\". HTTP
response code is 400."}
And from UI I can see that what has been created is a data domain. Not
data (master) domain as old one was. Now I'm stuck here. This is a
fibre channel system where I'm trying to do this if it is relevant.
What I could find from the logs is this:
2021-04-27 09:36:06,925+0300 DEBUG
otopi.ovirt_hosted_engine_setup.ansible_utils
ansible_utils._process_output:105 storage_domain_details: {'changed':
False, 'ovirt_storage_domains': [{'href':
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
'comment': '', 'description': '', 'id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage',
'available': 531502202880, 'backup': False, 'block_size': 512,
'committed': 0, 'critical_space_action_blocker': 5,
'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
[], 'disks': [], 'external_status': 'ok', 'master':
False,
'permissions': [], 'status': 'unattached', 'storage':
{'type': 'fcp',
'volume_group': {'id': 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
'logical_units': [{'discard_max_size': 268435456,
'discard_zeroes_data': False, 'id':
'36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths':
0,
'product_id': 'Compellent Vol', 'serial':
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'vendor_id': 'COMPELNT', 'volume_group_id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
[], 'storage_format': 'v5', 'supports_discard': True,
'supports_discard_zeroes_data': False, 'templates': [], 'type':
'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
10, 'wipe_after_delete': False}], 'failed': False}
'master': False? I'm not sure if this creation or check. I tried this
opration twice. I removed the new hosted_storage and remove also old
hosted_engine domain on second try to make sure that it doesn't
prevent creation of another master. No luck with that.
Thanks,
Juhani
2:13 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
To continue. I noticed that another storage domain took the data
(master) now. I saw one advice that you can force change by putting
the storage domain to maintenance mode. Problem is that there are VM's
running on these domains. How is this supposed to work during the
restore?
Thanks,
Juhani
On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> Hi!
>
> I started the upgrade from 4.3->4.4. Now I'm stuck as restoring the
> backup fails to create the correct storage domain for Hosted Engine.
> How can I create one? Error from ansible task is:
>
> [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
> detail is "[Domain format is different from master storage domain
> format]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> "Fault reason is \"Operation Failed\". Fault detail is \"[Domain
> format is different from master storage domain format]\". HTTP
> response code is 400."}
>
> And from UI I can see that what has been created is a data domain. Not
> data (master) domain as old one was. Now I'm stuck here. This is a
> fibre channel system where I'm trying to do this if it is relevant.
>
> What I could find from the logs is this:
> 2021-04-27 09:36:06,925+0300 DEBUG
> otopi.ovirt_hosted_engine_setup.ansible_utils
> ansible_utils._process_output:105 storage_domain_details: {'changed':
> False, 'ovirt_storage_domains': [{'href':
> '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> 'comment': '', 'description': '', 'id':
> 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
'hosted_storage',
> 'available': 531502202880, 'backup': False, 'block_size':
512,
> 'committed': 0, 'critical_space_action_blocker': 5,
> 'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
> [], 'disks': [], 'external_status': 'ok', 'master':
False,
> 'permissions': [], 'status': 'unattached', 'storage':
{'type': 'fcp',
> 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> 'logical_units': [{'discard_max_size': 268435456,
> 'discard_zeroes_data': False, 'id':
> '36000d31005b4f6000000000000000029', 'lun_mapping': 3,
'paths': 0,
> 'product_id': 'Compellent Vol', 'serial':
> 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> 'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> 'vendor_id': 'COMPELNT', 'volume_group_id':
> 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> [], 'storage_format': 'v5', 'supports_discard': True,
> 'supports_discard_zeroes_data': False, 'templates': [],
'type':
> 'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
> 10, 'wipe_after_delete': False}], 'failed': False}
>
> 'master': False? I'm not sure if this creation or check. I tried this
> opration twice. I removed the new hosted_storage and remove also old
> hosted_engine domain on second try to make sure that it doesn't
> prevent creation of another master. No luck with that.
>
> Thanks,
> Juhani
2:40 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
To continue. I noticed that another storage domain took the data
(master) now. I saw one advice that you can force change by putting
the storage domain to maintenance mode. Problem is that there are VM's
running on these domains. How is this supposed to work during the
restore?
There is a recent change [1] by Shani (Cced) that should allow you
to choose another storage domain as master. So you can create a new
(temporary?) SD with the correct compatibility level and then set it
to master.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
Best regards,
Thanks,
Juhani
On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> Hi!
>
> I started the upgrade from 4.3->4.4. Now I'm stuck as restoring the
> backup fails to create the correct storage domain for Hosted Engine.
> How can I create one? Error from ansible task is:
>
> [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
> detail is "[Domain format is different from master storage domain
> format]". HTTP response code is 400.
> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> "Fault reason is \"Operation Failed\". Fault detail is
\"[Domain
> format is different from master storage domain format]\". HTTP
> response code is 400."}
>
> And from UI I can see that what has been created is a data domain. Not
> data (master) domain as old one was. Now I'm stuck here. This is a
> fibre channel system where I'm trying to do this if it is relevant.
>
> What I could find from the logs is this:
> 2021-04-27 09:36:06,925+0300 DEBUG
> otopi.ovirt_hosted_engine_setup.ansible_utils
> ansible_utils._process_output:105 storage_domain_details: {'changed':
> False, 'ovirt_storage_domains': [{'href':
> '/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> 'comment': '', 'description': '', 'id':
> 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
'hosted_storage',
> 'available': 531502202880, 'backup': False, 'block_size':
512,
> 'committed': 0, 'critical_space_action_blocker': 5,
> 'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
> [], 'disks': [], 'external_status': 'ok', 'master':
False,
> 'permissions': [], 'status': 'unattached',
'storage': {'type': 'fcp',
> 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> 'logical_units': [{'discard_max_size': 268435456,
> 'discard_zeroes_data': False, 'id':
> '36000d31005b4f6000000000000000029', 'lun_mapping': 3,
'paths': 0,
> 'product_id': 'Compellent Vol', 'serial':
> 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> 'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> 'vendor_id': 'COMPELNT', 'volume_group_id':
> 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> [], 'storage_format': 'v5', 'supports_discard': True,
> 'supports_discard_zeroes_data': False, 'templates': [],
'type':
> 'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
> 10, 'wipe_after_delete': False}], 'failed': False}
>
> 'master': False? I'm not sure if this creation or check. I tried this
> opration twice. I removed the new hosted_storage and remove also old
> hosted_engine domain on second try to make sure that it doesn't
> prevent creation of another master. No luck with that.
>
> Thanks,
> Juhani
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
--
Didi
2:47 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
Thanks this looks like what I'm looking for. I'm still wondering how
to use this. I have LUN just for new hosted storage. Ansible created
storage domain to it correctly but just can't activate it. So is the
idea that I activate this unattached hosted_storage domain and try to
use API to make it master? I attached a screen shot how it looks
currently.
On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> To continue. I noticed that another storage domain took the data
> (master) now. I saw one advice that you can force change by putting
> the storage domain to maintenance mode. Problem is that there are VM's
> running on these domains. How is this supposed to work during the
> restore?
There is a recent change [1] by Shani (Cced) that should allow you
to choose another storage domain as master. So you can create a new
(temporary?) SD with the correct compatibility level and then set it
to master.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
Best regards,
>
> Thanks,
> Juhani
>
> On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Hi!
> >
> > I started the upgrade from 4.3->4.4. Now I'm stuck as restoring the
> > backup fails to create the correct storage domain for Hosted Engine.
> > How can I create one? Error from ansible task is:
> >
> > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
> > detail is "[Domain format is different from master storage domain
> > format]". HTTP response code is 400.
> > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> > "Fault reason is \"Operation Failed\". Fault detail is
\"[Domain
> > format is different from master storage domain format]\". HTTP
> > response code is 400."}
> >
> > And from UI I can see that what has been created is a data domain. Not
> > data (master) domain as old one was. Now I'm stuck here. This is a
> > fibre channel system where I'm trying to do this if it is relevant.
> >
> > What I could find from the logs is this:
> > 2021-04-27 09:36:06,925+0300 DEBUG
> > otopi.ovirt_hosted_engine_setup.ansible_utils
> > ansible_utils._process_output:105 storage_domain_details: {'changed':
> > False, 'ovirt_storage_domains': [{'href':
> >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > 'comment': '', 'description': '',
'id':
> > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
'hosted_storage',
> > 'available': 531502202880, 'backup': False,
'block_size': 512,
> > 'committed': 0, 'critical_space_action_blocker': 5,
> > 'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
> > [], 'disks': [], 'external_status': 'ok',
'master': False,
> > 'permissions': [], 'status': 'unattached',
'storage': {'type': 'fcp',
> > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > 'logical_units': [{'discard_max_size': 268435456,
> > 'discard_zeroes_data': False, 'id':
> > '36000d31005b4f6000000000000000029', 'lun_mapping': 3,
'paths': 0,
> > 'product_id': 'Compellent Vol', 'serial':
> > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size':
536870912000,
> > 'storage_domain_id': 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > 'vendor_id': 'COMPELNT', 'volume_group_id':
> > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> > [], 'storage_format': 'v5', 'supports_discard': True,
> > 'supports_discard_zeroes_data': False, 'templates': [],
'type':
> > 'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
> > 10, 'wipe_after_delete': False}], 'failed': False}
> >
> > 'master': False? I'm not sure if this creation or check. I tried
this
> > opration twice. I removed the new hosted_storage and remove also old
> > hosted_engine domain on second try to make sure that it doesn't
> > prevent creation of another master. No luck with that.
> >
> > Thanks,
> > Juhani
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
--
Didi
3:07 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
It seems that it is not supported in oVirt yet? I got this response
when I tried to change master with those storage domain that I have:
<?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
<fault>
<detail>[Cannot switch master storage domain. Switch master
storage domain operation is not supported.]</detail>
<reason>Operation Failed</reason>
</fault>
So is this really the only way to do this: shutdown everything and put
other storage domains into maintenance? It would have been nice if
this information was in the upgrade guide. It made this seem so easy
and simple...
Thanks,
Juhani
On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> Thanks this looks like what I'm looking for. I'm still wondering how
> to use this. I have LUN just for new hosted storage. Ansible created
> storage domain to it correctly but just can't activate it. So is the
> idea that I activate this unattached hosted_storage domain and try to
> use API to make it master? I attached a screen shot how it looks
> currently.
>
>
>
> On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
> >
> > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > To continue. I noticed that another storage domain took the data
> > > (master) now. I saw one advice that you can force change by putting
> > > the storage domain to maintenance mode. Problem is that there are VM's
> > > running on these domains. How is this supposed to work during the
> > > restore?
> >
> > There is a recent change [1] by Shani (Cced) that should allow you
> > to choose another storage domain as master. So you can create a new
> > (temporary?) SD with the correct compatibility level and then set it
> > to master.
> >
> > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> >
> > Best regards,
> >
> > >
> > > Thanks,
> > > Juhani
> > >
> > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > Hi!
> > > >
> > > > I started the upgrade from 4.3->4.4. Now I'm stuck as restoring
the
> > > > backup fails to create the correct storage domain for Hosted Engine.
> > > > How can I create one? Error from ansible task is:
> > > >
> > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault
> > > > detail is "[Domain format is different from master storage
domain
> > > > format]". HTTP response code is 400.
> > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
false, "msg":
> > > > "Fault reason is \"Operation Failed\". Fault detail is
\"[Domain
> > > > format is different from master storage domain format]\". HTTP
> > > > response code is 400."}
> > > >
> > > > And from UI I can see that what has been created is a data domain.
Not
> > > > data (master) domain as old one was. Now I'm stuck here. This is
a
> > > > fibre channel system where I'm trying to do this if it is
relevant.
> > > >
> > > > What I could find from the logs is this:
> > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > ansible_utils._process_output:105 storage_domain_details:
{'changed':
> > > > False, 'ovirt_storage_domains': [{'href':
> > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > 'comment': '', 'description': '',
'id':
> > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
'hosted_storage',
> > > > 'available': 531502202880, 'backup': False,
'block_size': 512,
> > > > 'committed': 0, 'critical_space_action_blocker': 5,
> > > > 'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
> > > > [], 'disks': [], 'external_status': 'ok',
'master': False,
> > > > 'permissions': [], 'status': 'unattached',
'storage': {'type': 'fcp',
> > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > 'logical_units': [{'discard_max_size': 268435456,
> > > > 'discard_zeroes_data': False, 'id':
> > > > '36000d31005b4f6000000000000000029', 'lun_mapping': 3,
'paths': 0,
> > > > 'product_id': 'Compellent Vol', 'serial':
> > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size':
536870912000,
> > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > 'vendor_id': 'COMPELNT', 'volume_group_id':
> > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > 'supports_discard_zeroes_data': False, 'templates':
[], 'type':
> > > > 'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
> > > > 10, 'wipe_after_delete': False}], 'failed': False}
> > > >
> > > > 'master': False? I'm not sure if this creation or check. I
tried this
> > > > opration twice. I removed the new hosted_storage and remove also old
> > > > hosted_engine domain on second try to make sure that it doesn't
> > > > prevent creation of another master. No luck with that.
> > > >
> > > > Thanks,
> > > > Juhani
> > > _______________________________________________
> > > Users mailing list -- users(a)ovirt.org
> > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> >
> >
> >
> > --
> > Didi
> >
3:47 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
Hmm. Is it possible that when the other node is still running v4.3
this operation can't be completed as it doesn't know how to do it.
Thanks,
Juhani
On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> It seems that it is not supported in oVirt yet? I got this response
> when I tried to change master with those storage domain that I have:
>
> <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> <fault>
> <detail>[Cannot switch master storage domain. Switch master
> storage domain operation is not supported.]</detail>
> <reason>Operation Failed</reason>
> </fault>
>
> So is this really the only way to do this: shutdown everything and put
> other storage domains into maintenance? It would have been nice if
> this information was in the upgrade guide. It made this seem so easy
> and simple...
>
> Thanks,
> Juhani
>
> On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Thanks this looks like what I'm looking for. I'm still wondering how
> > to use this. I have LUN just for new hosted storage. Ansible created
> > storage domain to it correctly but just can't activate it. So is the
> > idea that I activate this unattached hosted_storage domain and try to
> > use API to make it master? I attached a screen shot how it looks
> > currently.
> >
> >
> >
> > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > >
> > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > To continue. I noticed that another storage domain took the data
> > > > (master) now. I saw one advice that you can force change by putting
> > > > the storage domain to maintenance mode. Problem is that there are
VM's
> > > > running on these domains. How is this supposed to work during the
> > > > restore?
> > >
> > > There is a recent change [1] by Shani (Cced) that should allow you
> > > to choose another storage domain as master. So you can create a new
> > > (temporary?) SD with the correct compatibility level and then set it
> > > to master.
> > >
> > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > >
> > > Best regards,
> > >
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Hi!
> > > > >
> > > > > I started the upgrade from 4.3->4.4. Now I'm stuck as
restoring the
> > > > > backup fails to create the correct storage domain for Hosted
Engine.
> > > > > How can I create one? Error from ansible task is:
> > > > >
> > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault
> > > > > detail is "[Domain format is different from master storage
domain
> > > > > format]". HTTP response code is 400.
> > > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
false, "msg":
> > > > > "Fault reason is \"Operation Failed\". Fault
detail is \"[Domain
> > > > > format is different from master storage domain format]\".
HTTP
> > > > > response code is 400."}
> > > > >
> > > > > And from UI I can see that what has been created is a data
domain. Not
> > > > > data (master) domain as old one was. Now I'm stuck here. This
is a
> > > > > fibre channel system where I'm trying to do this if it is
relevant.
> > > > >
> > > > > What I could find from the logs is this:
> > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > ansible_utils._process_output:105 storage_domain_details:
{'changed':
> > > > > False, 'ovirt_storage_domains': [{'href':
> > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > 'comment': '', 'description': '',
'id':
> > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
'hosted_storage',
> > > > > 'available': 531502202880, 'backup': False,
'block_size': 512,
> > > > > 'committed': 0, 'critical_space_action_blocker':
5,
> > > > > 'discard_after_delete': True, 'disk_profiles':
[], 'disk_snapshots':
> > > > > [], 'disks': [], 'external_status': 'ok',
'master': False,
> > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > 'logical_units': [{'discard_max_size':
268435456,
> > > > > 'discard_zeroes_data': False, 'id':
> > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029',
'size': 536870912000,
> > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > 'data', 'used': 4294967296, 'vms': [],
'warning_low_space_indicator':
> > > > > 10, 'wipe_after_delete': False}], 'failed':
False}
> > > > >
> > > > > 'master': False? I'm not sure if this creation or
check. I tried this
> > > > > opration twice. I removed the new hosted_storage and remove also
old
> > > > > hosted_engine domain on second try to make sure that it
doesn't
> > > > > prevent creation of another master. No luck with that.
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > _______________________________________________
> > > > Users mailing list -- users(a)ovirt.org
> > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > >
> > >
> > >
> > > --
> > > Didi
> > >
2:59 p.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
Story of the problems continues. Finally shut everything down, got
storage domains to maintenance and then this happens:
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
"[Physical device initialization failed. Please check that the device
is empty and accessible by the host.]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg":
"Fault reason is \"Operation Failed\". Fault detail is \"[Physical
device initialization failed. Please check that the device is empty
and accessible by the host.]\". HTTP response code is 400."}
No amount zeroing device helped.
Next plan: Find out if I can restore backup to standalone server. If
that fails that's the end of oVirt for me. It worked fine for a couple
of years but this update hassle is too much. Should have gone until
the end with 4.3.
Thanks,
Juhani
On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> Hmm. Is it possible that when the other node is still running v4.3
> this operation can't be completed as it doesn't know how to do it.
>
> Thanks,
> Juhani
>
> On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > It seems that it is not supported in oVirt yet? I got this response
> > when I tried to change master with those storage domain that I have:
> >
> > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > <fault>
> > <detail>[Cannot switch master storage domain. Switch master
> > storage domain operation is not supported.]</detail>
> > <reason>Operation Failed</reason>
> > </fault>
> >
> > So is this really the only way to do this: shutdown everything and put
> > other storage domains into maintenance? It would have been nice if
> > this information was in the upgrade guide. It made this seem so easy
> > and simple...
> >
> > Thanks,
> > Juhani
> >
> > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > Thanks this looks like what I'm looking for. I'm still wondering
how
> > > to use this. I have LUN just for new hosted storage. Ansible created
> > > storage domain to it correctly but just can't activate it. So is the
> > > idea that I activate this unattached hosted_storage domain and try to
> > > use API to make it master? I attached a screen shot how it looks
> > > currently.
> > >
> > >
> > >
> > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > > >
> > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > To continue. I noticed that another storage domain took the data
> > > > > (master) now. I saw one advice that you can force change by
putting
> > > > > the storage domain to maintenance mode. Problem is that there are
VM's
> > > > > running on these domains. How is this supposed to work during
the
> > > > > restore?
> > > >
> > > > There is a recent change [1] by Shani (Cced) that should allow you
> > > > to choose another storage domain as master. So you can create a new
> > > > (temporary?) SD with the correct compatibility level and then set it
> > > > to master.
> > > >
> > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > >
> > > > Best regards,
> > > >
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > Hi!
> > > > > >
> > > > > > I started the upgrade from 4.3->4.4. Now I'm stuck as
restoring the
> > > > > > backup fails to create the correct storage domain for Hosted
Engine.
> > > > > > How can I create one? Error from ansible task is:
> > > > > >
> > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault
> > > > > > detail is "[Domain format is different from master
storage domain
> > > > > > format]". HTTP response code is 400.
> > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > "Fault reason is \"Operation Failed\". Fault
detail is \"[Domain
> > > > > > format is different from master storage domain
format]\". HTTP
> > > > > > response code is 400."}
> > > > > >
> > > > > > And from UI I can see that what has been created is a data
domain. Not
> > > > > > data (master) domain as old one was. Now I'm stuck here.
This is a
> > > > > > fibre channel system where I'm trying to do this if it
is relevant.
> > > > > >
> > > > > > What I could find from the logs is this:
> > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > ansible_utils._process_output:105 storage_domain_details:
{'changed':
> > > > > > False, 'ovirt_storage_domains': [{'href':
> > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > 'comment': '', 'description':
'', 'id':
> > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'name': 'hosted_storage',
> > > > > > 'available': 531502202880, 'backup': False,
'block_size': 512,
> > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > [], 'disks': [], 'external_status':
'ok', 'master': False,
> > > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > 'logical_units': [{'discard_max_size':
268435456,
> > > > > > 'discard_zeroes_data': False, 'id':
> > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029',
'size': 536870912000,
> > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > > 'data', 'used': 4294967296, 'vms':
[], 'warning_low_space_indicator':
> > > > > > 10, 'wipe_after_delete': False}], 'failed':
False}
> > > > > >
> > > > > > 'master': False? I'm not sure if this creation
or check. I tried this
> > > > > > opration twice. I removed the new hosted_storage and remove
also old
> > > > > > hosted_engine domain on second try to make sure that it
doesn't
> > > > > > prevent creation of another master. No luck with that.
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > _______________________________________________
> > > > > Users mailing list -- users(a)ovirt.org
> > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > >
> > > >
> > > >
> > > > --
> > > > Didi
> > > >
12:30 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
Story of the problems continues. Finally shut everything down, got
storage domains to maintenance and then this happens:
ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
"[Physical device initialization failed. Please check that the device
is empty and accessible by the host.]". HTTP response code is 400.
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
"Fault reason is \"Operation Failed\". Fault detail is \"[Physical
device initialization failed. Please check that the device is empty
and accessible by the host.]\". HTTP response code is 400."}
Can you please check/share relevant parts of vdsm.log when this happens?
Thanks.
No amount zeroing device helped.
Next plan: Find out if I can restore backup to standalone server. If
that fails that's the end of oVirt for me. It worked fine for a couple
of years but this update hassle is too much. Should have gone until
the end with 4.3.
Thanks,
Juhani
On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
> Hmm. Is it possible that when the other node is still running v4.3
> this operation can't be completed as it doesn't know how to do it.
>
> Thanks,
> Juhani
>
> On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > It seems that it is not supported in oVirt yet? I got this response
> > when I tried to change master with those storage domain that I have:
> >
> > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > <fault>
> > <detail>[Cannot switch master storage domain. Switch master
> > storage domain operation is not supported.]</detail>
> > <reason>Operation Failed</reason>
> > </fault>
> >
> > So is this really the only way to do this: shutdown everything and put
> > other storage domains into maintenance? It would have been nice if
> > this information was in the upgrade guide. It made this seem so easy
> > and simple...
> >
> > Thanks,
> > Juhani
> >
> > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > Thanks this looks like what I'm looking for. I'm still wondering
how
> > > to use this. I have LUN just for new hosted storage. Ansible created
> > > storage domain to it correctly but just can't activate it. So is the
> > > idea that I activate this unattached hosted_storage domain and try to
> > > use API to make it master? I attached a screen shot how it looks
> > > currently.
> > >
> > >
> > >
> > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > >
> > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > To continue. I noticed that another storage domain took the
data
> > > > > (master) now. I saw one advice that you can force change by
putting
> > > > > the storage domain to maintenance mode. Problem is that there
are VM's
> > > > > running on these domains. How is this supposed to work during
the
> > > > > restore?
> > > >
> > > > There is a recent change [1] by Shani (Cced) that should allow you
> > > > to choose another storage domain as master. So you can create a new
> > > > (temporary?) SD with the correct compatibility level and then set it
> > > > to master.
> > > >
> > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > >
> > > > Best regards,
> > > >
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > Hi!
> > > > > >
> > > > > > I started the upgrade from 4.3->4.4. Now I'm stuck
as restoring the
> > > > > > backup fails to create the correct storage domain for
Hosted Engine.
> > > > > > How can I create one? Error from ansible task is:
> > > > > >
> > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault
> > > > > > detail is "[Domain format is different from master
storage domain
> > > > > > format]". HTTP response code is 400.
> > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > "Fault reason is \"Operation Failed\". Fault
detail is \"[Domain
> > > > > > format is different from master storage domain
format]\". HTTP
> > > > > > response code is 400."}
> > > > > >
> > > > > > And from UI I can see that what has been created is a data
domain. Not
> > > > > > data (master) domain as old one was. Now I'm stuck
here. This is a
> > > > > > fibre channel system where I'm trying to do this if it
is relevant.
> > > > > >
> > > > > > What I could find from the logs is this:
> > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > ansible_utils._process_output:105 storage_domain_details:
{'changed':
> > > > > > False, 'ovirt_storage_domains': [{'href':
> > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > 'comment': '', 'description':
'', 'id':
> > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'name': 'hosted_storage',
> > > > > > 'available': 531502202880, 'backup': False,
'block_size': 512,
> > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > [], 'disks': [], 'external_status':
'ok', 'master': False,
> > > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > 'logical_units': [{'discard_max_size':
268435456,
> > > > > > 'discard_zeroes_data': False, 'id':
> > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029',
'size': 536870912000,
> > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
'storage_connections':
> > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > > 'data', 'used': 4294967296, 'vms':
[], 'warning_low_space_indicator':
> > > > > > 10, 'wipe_after_delete': False}], 'failed':
False}
> > > > > >
> > > > > > 'master': False? I'm not sure if this creation
or check. I tried this
> > > > > > opration twice. I removed the new hosted_storage and remove
also old
> > > > > > hosted_engine domain on second try to make sure that it
doesn't
> > > > > > prevent creation of another master. No luck with that.
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > _______________________________________________
> > > > > Users mailing list -- users(a)ovirt.org
> > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > >
> > > >
> > > >
> > > > --
> > > > Didi
> > > >
--
Didi
1:29 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
Hi!
This is from ansible logs:
2021-04-27 22:20:38,286+0300 ERROR ansible failed {
"ansible_host": "localhost",
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
"ansible_result": {
"_ansible_no_log": false,
"changed": false,
"exception": "Traceback (most recent call last):\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 804, in main\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
line 651, in create\n **kwargs\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
26182, in add\n return self._internal_add(storage_domain, headers,
query, wait)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232,
in _internal_add\n return future.wait() if wait else future\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
in wait\n return self._code(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229,
in callback\n self._check_fault(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132,
in _check_fault\n self._raise_error(response, body)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118,
in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
\"Operation Failed\". Fault detail is \"[Physical device
initialization failed. Please check that the device is empty and
accessible by the host.]\". HTTP response code is 400.\n",
"invocation": {
"module_args": {
"backup": null,
"comment": null,
"critical_space_action_blocker": null,
"data_center": "Default",
"description": null,
"destroy": null,
"discard_after_delete": true,
"domain_function": "data",
"fcp": {
"lun_id": "36000d31005b4f6000000000000000029"
},
"fetch_nested": false,
"format": null,
"glusterfs": null,
"host": "ovirt02.virt.local",
"id": null,
"iscsi": null,
"localfs": null,
"managed_block_storage": null,
"name": "hosted_storage",
"nested_attributes": [],
"nfs": null,
"poll_interval": 3,
"posixfs": null,
"state": "unattached",
"timeout": 180,
"wait": true,
"warning_low_space": null,
"wipe_after_delete": null
}
},
"msg": "Fault reason is \"Operation Failed\". Fault
detail is
\"[Physical device initialization failed. Please check that the device
is empty and accessible by the host.]\". HTTP response code is 400."
},
"ansible_task": "Add Fibre Channel storage domain",
"ansible_type": "task",
"status": "FAILED",
This seem to happen same time:
2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
getDeviceList return={'devList': [{'GUID':
'36000d31005b4f6000000000000000029', 'capacity': '536870912000',
'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
'pathlist': [], 'pathstatus': [{'physdev': 'sdu',
'state': 'active',
'capacity': '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev':
'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev': 'sdv', 'state':
'active', 'capacity':
'536870912000', 'lun': '3', 'type': 'FCP'},
{'physdev': 'sdt',
'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
'FCP'}], 'physicalblocksize': '4096', 'productID':
'Compellent Vol',
'pvUUID': '', 'pvsize': '', 'serial':
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
devlist=['36000d31005b4f6000000000000000029'], force=False,
options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
with specific filter failed or returned no data, retrying with a wider
filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[]
err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
exclusively. Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:511)
2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices
{ preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"] hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:538)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
failed with rc=5 (lvm:1142)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:1143)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",)
from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
(task:880)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
aborted: 'value=Failed to initialize physical device:
("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
abortedcode=601' (task:1190)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
FINISH createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83)
I tried clearing the disk but from above it seems that something else
is wonky here.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Story of the problems continues. Finally shut everything down, got
> > storage domains to maintenance and then this happens:
> >
> > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
> > "[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]". HTTP response code is 400.
> > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > device initialization failed. Please check that the device is empty
> > and accessible by the host.]\". HTTP response code is 400."}
>
> Can you please check/share relevant parts of vdsm.log when this happens?
> Thanks.
>
> >
> > No amount zeroing device helped.
> >
> > Next plan: Find out if I can restore backup to standalone server. If
> > that fails that's the end of oVirt for me. It worked fine for a couple
> > of years but this update hassle is too much. Should have gone until
> > the end with 4.3.
> >
> > Thanks,
> > Juhani
> >
> > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > Hmm. Is it possible that when the other node is still running v4.3
> > > this operation can't be completed as it doesn't know how to do it.
> > >
> > > Thanks,
> > > Juhani
> > >
> > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > It seems that it is not supported in oVirt yet? I got this response
> > > > when I tried to change master with those storage domain that I have:
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > <fault>
> > > > <detail>[Cannot switch master storage domain. Switch master
> > > > storage domain operation is not supported.]</detail>
> > > > <reason>Operation Failed</reason>
> > > > </fault>
> > > >
> > > > So is this really the only way to do this: shutdown everything and
put
> > > > other storage domains into maintenance? It would have been nice if
> > > > this information was in the upgrade guide. It made this seem so easy
> > > > and simple...
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Thanks this looks like what I'm looking for. I'm still
wondering how
> > > > > to use this. I have LUN just for new hosted storage. Ansible
created
> > > > > storage domain to it correctly but just can't activate it. So
is the
> > > > > idea that I activate this unattached hosted_storage domain and
try to
> > > > > use API to make it master? I attached a screen shot how it looks
> > > > > currently.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > To continue. I noticed that another storage domain took
the data
> > > > > > > (master) now. I saw one advice that you can force
change by putting
> > > > > > > the storage domain to maintenance mode. Problem is that
there are VM's
> > > > > > > running on these domains. How is this supposed to work
during the
> > > > > > > restore?
> > > > > >
> > > > > > There is a recent change [1] by Shani (Cced) that should
allow you
> > > > > > to choose another storage domain as master. So you can
create a new
> > > > > > (temporary?) SD with the correct compatibility level and
then set it
> > > > > > to master.
> > > > > >
> > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Juhani
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I started the upgrade from 4.3->4.4. Now
I'm stuck as restoring the
> > > > > > > > backup fails to create the correct storage domain
for Hosted Engine.
> > > > > > > > How can I create one? Error from ansible task is:
> > > > > > > >
> > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is
"Operation Failed". Fault
> > > > > > > > detail is "[Domain format is different from
master storage domain
> > > > > > > > format]". HTTP response code is 400.
> > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > format is different from master storage domain
format]\". HTTP
> > > > > > > > response code is 400."}
> > > > > > > >
> > > > > > > > And from UI I can see that what has been created
is a data domain. Not
> > > > > > > > data (master) domain as old one was. Now I'm
stuck here. This is a
> > > > > > > > fibre channel system where I'm trying to do
this if it is relevant.
> > > > > > > >
> > > > > > > > What I could find from the logs is this:
> > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'name': 'hosted_storage',
> > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > > > > 'data', 'used': 4294967296,
'vms': [], 'warning_low_space_indicator':
> > > > > > > > 10, 'wipe_after_delete': False}],
'failed': False}
> > > > > > > >
> > > > > > > > 'master': False? I'm not sure if this
creation or check. I tried this
> > > > > > > > opration twice. I removed the new hosted_storage
and remove also old
> > > > > > > > hosted_engine domain on second try to make sure
that it doesn't
> > > > > > > > prevent creation of another master. No luck with
that.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Juhani
> > > > > > > _______________________________________________
> > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Didi
> > > > > >
> >
>
>
> --
> Didi
>
1:57 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
Hi!
This is from ansible logs:
2021-04-27 22:20:38,286+0300 ERROR ansible failed {
"ansible_host": "localhost",
"ansible_playbook":
"/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
"ansible_result": {
"_ansible_no_log": false,
"changed": false,
"exception": "Traceback (most recent call last):\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
line 804, in main\n File
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
line 651, in create\n **kwargs\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
26182, in add\n return self._internal_add(storage_domain, headers,
query, wait)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232,
in _internal_add\n return future.wait() if wait else future\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
in wait\n return self._code(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229,
in callback\n self._check_fault(response)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132,
in _check_fault\n self._raise_error(response, body)\n File
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118,
in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
\"Operation Failed\". Fault detail is \"[Physical device
initialization failed. Please check that the device is empty and
accessible by the host.]\". HTTP response code is 400.\n",
"invocation": {
"module_args": {
"backup": null,
"comment": null,
"critical_space_action_blocker": null,
"data_center": "Default",
"description": null,
"destroy": null,
"discard_after_delete": true,
"domain_function": "data",
"fcp": {
"lun_id": "36000d31005b4f6000000000000000029"
},
"fetch_nested": false,
"format": null,
"glusterfs": null,
"host": "ovirt02.virt.local",
"id": null,
"iscsi": null,
"localfs": null,
"managed_block_storage": null,
"name": "hosted_storage",
"nested_attributes": [],
"nfs": null,
"poll_interval": 3,
"posixfs": null,
"state": "unattached",
"timeout": 180,
"wait": true,
"warning_low_space": null,
"wipe_after_delete": null
}
},
"msg": "Fault reason is \"Operation Failed\". Fault
detail is
\"[Physical device initialization failed. Please check that the device
is empty and accessible by the host.]\". HTTP response code is 400."
},
"ansible_task": "Add Fibre Channel storage domain",
"ansible_type": "task",
"status": "FAILED",
This seem to happen same time:
2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
getDeviceList return={'devList': [{'GUID':
'36000d31005b4f6000000000000000029', 'capacity': '536870912000',
'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
'pathlist': [], 'pathstatus': [{'physdev': 'sdu',
'state': 'active',
'capacity': '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev':
'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev': 'sdv', 'state':
'active', 'capacity':
'536870912000', 'lun': '3', 'type': 'FCP'},
{'physdev': 'sdt',
'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
'FCP'}], 'physicalblocksize': '4096', 'productID':
'Compellent Vol',
'pvUUID': '', 'pvsize': '', 'serial':
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
devlist=['36000d31005b4f6000000000000000029'], force=False,
options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
with specific filter failed or returned no data, retrying with a wider
filter, cmd=['/sbin/lvm', 'pvcreate', '--config', 'devices {
preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[]
err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
exclusively. Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:511)
It seems like this ^^^ is the problem. I suggest to check around e.g.
with lsof etc., and/or search the net for something like
"pvcreate" "Can't open" "exclusively. Mounted
filesystem?"
Best regards,
2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices
{ preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
write_cache_state=0 disable_after_error_count=3
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"] hints="none" obtain_device_list_from_udev=0 } global {
locking_type=1 prioritise_write_locks=1 wait_for_locks=1
use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
'--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 err=["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:538)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
failed with rc=5 (lvm:1142)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
Mounted filesystem?", " Can't open
/dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
filesystem?"] (lvm:1143)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",)
from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
(task:880)
2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
[storage.TaskManager.Task]
(Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
aborted: 'value=Failed to initialize physical device:
("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
abortedcode=601' (task:1190)
2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
FINISH createVG error=Failed to initialize physical device:
("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83)
I tried clearing the disk but from above it seems that something else
is wonky here.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Story of the problems continues. Finally shut everything down, got
> > storage domains to maintenance and then this happens:
> >
> > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault detail is
> > "[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]". HTTP response code is 400.
> > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > device initialization failed. Please check that the device is empty
> > and accessible by the host.]\". HTTP response code is 400."}
>
> Can you please check/share relevant parts of vdsm.log when this happens?
> Thanks.
>
> >
> > No amount zeroing device helped.
> >
> > Next plan: Find out if I can restore backup to standalone server. If
> > that fails that's the end of oVirt for me. It worked fine for a couple
> > of years but this update hassle is too much. Should have gone until
> > the end with 4.3.
> >
> > Thanks,
> > Juhani
> >
> > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > Hmm. Is it possible that when the other node is still running v4.3
> > > this operation can't be completed as it doesn't know how to do
it.
> > >
> > > Thanks,
> > > Juhani
> > >
> > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > It seems that it is not supported in oVirt yet? I got this response
> > > > when I tried to change master with those storage domain that I have:
> > > >
> > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > <fault>
> > > > <detail>[Cannot switch master storage domain. Switch
master
> > > > storage domain operation is not supported.]</detail>
> > > > <reason>Operation Failed</reason>
> > > > </fault>
> > > >
> > > > So is this really the only way to do this: shutdown everything and
put
> > > > other storage domains into maintenance? It would have been nice if
> > > > this information was in the upgrade guide. It made this seem so easy
> > > > and simple...
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Thanks this looks like what I'm looking for. I'm still
wondering how
> > > > > to use this. I have LUN just for new hosted storage. Ansible
created
> > > > > storage domain to it correctly but just can't activate it.
So is the
> > > > > idea that I activate this unattached hosted_storage domain and
try to
> > > > > use API to make it master? I attached a screen shot how it
looks
> > > > > currently.
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > To continue. I noticed that another storage domain
took the data
> > > > > > > (master) now. I saw one advice that you can force
change by putting
> > > > > > > the storage domain to maintenance mode. Problem is
that there are VM's
> > > > > > > running on these domains. How is this supposed to work
during the
> > > > > > > restore?
> > > > > >
> > > > > > There is a recent change [1] by Shani (Cced) that should
allow you
> > > > > > to choose another storage domain as master. So you can
create a new
> > > > > > (temporary?) SD with the correct compatibility level and
then set it
> > > > > > to master.
> > > > > >
> > > > > > [1] https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Juhani
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani Rautiainen
> > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi!
> > > > > > > >
> > > > > > > > I started the upgrade from 4.3->4.4. Now
I'm stuck as restoring the
> > > > > > > > backup fails to create the correct storage domain
for Hosted Engine.
> > > > > > > > How can I create one? Error from ansible task
is:
> > > > > > > >
> > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is
"Operation Failed". Fault
> > > > > > > > detail is "[Domain format is different from
master storage domain
> > > > > > > > format]". HTTP response code is 400.
> > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > format is different from master storage domain
format]\". HTTP
> > > > > > > > response code is 400."}
> > > > > > > >
> > > > > > > > And from UI I can see that what has been created
is a data domain. Not
> > > > > > > > data (master) domain as old one was. Now I'm
stuck here. This is a
> > > > > > > > fibre channel system where I'm trying to do
this if it is relevant.
> > > > > > > >
> > > > > > > > What I could find from the logs is this:
> > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > 'dd52022b-7616-47f6-9534-6f1a4084fdf4',
'name': 'hosted_storage',
> > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > 'permissions': [], 'status':
'unattached', 'storage': {'type': 'fcp',
> > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3, 'paths': 0,
> > > > > > > > 'product_id': 'Compellent Vol',
'serial':
> > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > > > 'supports_discard_zeroes_data': False,
'templates': [], 'type':
> > > > > > > > 'data', 'used': 4294967296,
'vms': [], 'warning_low_space_indicator':
> > > > > > > > 10, 'wipe_after_delete': False}],
'failed': False}
> > > > > > > >
> > > > > > > > 'master': False? I'm not sure if this
creation or check. I tried this
> > > > > > > > opration twice. I removed the new hosted_storage
and remove also old
> > > > > > > > hosted_engine domain on second try to make sure
that it doesn't
> > > > > > > > prevent creation of another master. No luck with
that.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Juhani
> > > > > > > _______________________________________________
> > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > To unsubscribe send an email to users-leave(a)ovirt.org
> > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Didi
> > > > > >
> >
>
>
> --
> Didi
>
--
Didi
3:16 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
I found these using dmsetup ls -tree:
6db20b74--512d--4a70--994e--8923d9e1e50b-master (253:21)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
6db20b74--512d--4a70--994e--8923d9e1e50b-inbox (253:17)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
db64924d--aa94--48ad--809a--3d235e84686b-ids (253:22)
└─36000d31005b4f600000000000000001d (253:12)
├─ (8:112)
├─ (8:96)
├─ (8:64)
└─ (8:48)
df259f5a--e1bd--4a93--bd6e--bace450625dc-master (253:35)
└─36000d31005b4f600000000000000001a (253:13)
├─ (8:144)
├─ (8:128)
├─ (8:80)
└─ (8:32)
6db20b74--512d--4a70--994e--8923d9e1e50b-outbox (253:18)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
ID matches the one in ansible error and vdsm log. Are these done by
oVirt? I mean it's pretty hard to know what to free if you don't know
what should be and should not be there. And if these can be cleared
how do you do that?
By the way does anyone know how I can connect back to that HE that is
hosted locally during installation (URL like
https://ovirt02.virt.local:6900/ovirt-engine/). It seems the
connection disappeared this morning. Now I'm totally blind what's
going on in both nodes... Maybe I have to clear this SHE recovery and
start from the scratch.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 9:57 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Hi!
> >
> > This is from ansible logs:
> > 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
> > "ansible_host": "localhost",
> > "ansible_playbook":
> > "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
> > "ansible_result": {
> > "_ansible_no_log": false,
> > "changed": false,
> > "exception": "Traceback (most recent call last):\n File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
> > line 804, in main\n File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
> > line 651, in create\n **kwargs\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> > 26182, in add\n return self._internal_add(storage_domain, headers,
> > query, wait)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
232,
> > in _internal_add\n return future.wait() if wait else future\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
> > in wait\n return self._code(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
229,
> > in callback\n self._check_fault(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
132,
> > in _check_fault\n self._raise_error(response, body)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
118,
> > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> > \"Operation Failed\". Fault detail is \"[Physical device
> > initialization failed. Please check that the device is empty and
> > accessible by the host.]\". HTTP response code is 400.\n",
> > "invocation": {
> > "module_args": {
> > "backup": null,
> > "comment": null,
> > "critical_space_action_blocker": null,
> > "data_center": "Default",
> > "description": null,
> > "destroy": null,
> > "discard_after_delete": true,
> > "domain_function": "data",
> > "fcp": {
> > "lun_id":
"36000d31005b4f6000000000000000029"
> > },
> > "fetch_nested": false,
> > "format": null,
> > "glusterfs": null,
> > "host": "ovirt02.virt.local",
> > "id": null,
> > "iscsi": null,
> > "localfs": null,
> > "managed_block_storage": null,
> > "name": "hosted_storage",
> > "nested_attributes": [],
> > "nfs": null,
> > "poll_interval": 3,
> > "posixfs": null,
> > "state": "unattached",
> > "timeout": 180,
> > "wait": true,
> > "warning_low_space": null,
> > "wipe_after_delete": null
> > }
> > },
> > "msg": "Fault reason is \"Operation Failed\".
Fault detail is
> > \"[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]\". HTTP response code is 400."
> > },
> > "ansible_task": "Add Fibre Channel storage domain",
> > "ansible_type": "task",
> > "status": "FAILED",
> >
> > This seem to happen same time:
> >
> > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> > getDeviceList return={'devList': [{'GUID':
> > '36000d31005b4f6000000000000000029', 'capacity':
'536870912000',
> > 'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
> > 'pathlist': [], 'pathstatus': [{'physdev':
'sdu', 'state': 'active',
> > 'capacity': '536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev':
> > 'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
> > 'type': 'FCP'}, {'physdev': 'sdv',
'state': 'active', 'capacity':
> > '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev': 'sdt',
> > 'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
> > 'FCP'}], 'physicalblocksize': '4096',
'productID': 'Compellent Vol',
> > 'pvUUID': '', 'pvsize': '', 'serial':
> > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
> > 'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
> > 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
> > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> > devlist=['36000d31005b4f6000000000000000029'], force=False,
> > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> > with specific filter failed or returned no data, retrying with a wider
> > filter, cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> > hints="none" obtain_device_list_from_udev=0 } global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
out=[]
> > err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
> > exclusively. Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:511)
>
> It seems like this ^^^ is the problem. I suggest to check around e.g.
> with lsof etc., and/or search the net for something like
>
> "pvcreate" "Can't open" "exclusively. Mounted
filesystem?"
>
> Best regards,
>
> > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> > tries have failed: cmd=['/sbin/lvm', 'pvcreate',
'--config', 'devices
> > { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> >
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
> > "r|.*|"] hints="none" obtain_device_list_from_udev=0 }
global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
err=["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:538)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> > failed with rc=5 (lvm:1142)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:1143)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> > createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> > from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
> > (task:880)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
> > aborted: 'value=Failed to initialize physical device:
> > ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> > abortedcode=601' (task:1190)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> > FINISH createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
(dispatcher:83)
> >
> > I tried clearing the disk but from above it seems that something else
> > is wonky here.
> >
> > Thanks,
> > Juhani
> >
> > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > >
> > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > Story of the problems continues. Finally shut everything down, got
> > > > storage domains to maintenance and then this happens:
> > > >
> > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is
> > > > "[Physical device initialization failed. Please check that the
device
> > > > is empty and accessible by the host.]". HTTP response code is
400.
> > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
false, "msg":
> > > > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > > > device initialization failed. Please check that the device is empty
> > > > and accessible by the host.]\". HTTP response code is
400."}
> > >
> > > Can you please check/share relevant parts of vdsm.log when this happens?
> > > Thanks.
> > >
> > > >
> > > > No amount zeroing device helped.
> > > >
> > > > Next plan: Find out if I can restore backup to standalone server. If
> > > > that fails that's the end of oVirt for me. It worked fine for a
couple
> > > > of years but this update hassle is too much. Should have gone until
> > > > the end with 4.3.
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Hmm. Is it possible that when the other node is still running
v4.3
> > > > > this operation can't be completed as it doesn't know how
to do it.
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > It seems that it is not supported in oVirt yet? I got this
response
> > > > > > when I tried to change master with those storage domain that
I have:
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > > > <fault>
> > > > > > <detail>[Cannot switch master storage domain.
Switch master
> > > > > > storage domain operation is not supported.]</detail>
> > > > > > <reason>Operation Failed</reason>
> > > > > > </fault>
> > > > > >
> > > > > > So is this really the only way to do this: shutdown
everything and put
> > > > > > other storage domains into maintenance? It would have been
nice if
> > > > > > this information was in the upgrade guide. It made this seem
so easy
> > > > > > and simple...
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks this looks like what I'm looking for.
I'm still wondering how
> > > > > > > to use this. I have LUN just for new hosted storage.
Ansible created
> > > > > > > storage domain to it correctly but just can't
activate it. So is the
> > > > > > > idea that I activate this unattached hosted_storage
domain and try to
> > > > > > > use API to make it master? I attached a screen shot how
it looks
> > > > > > > currently.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani
Rautiainen
> > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > To continue. I noticed that another storage
domain took the data
> > > > > > > > > (master) now. I saw one advice that you can
force change by putting
> > > > > > > > > the storage domain to maintenance mode.
Problem is that there are VM's
> > > > > > > > > running on these domains. How is this
supposed to work during the
> > > > > > > > > restore?
> > > > > > > >
> > > > > > > > There is a recent change [1] by Shani (Cced) that
should allow you
> > > > > > > > to choose another storage domain as master. So you
can create a new
> > > > > > > > (temporary?) SD with the correct compatibility
level and then set it
> > > > > > > > to master.
> > > > > > > >
> > > > > > > > [1]
https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Juhani
> > > > > > > > >
> > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani
Rautiainen
> > > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi!
> > > > > > > > > >
> > > > > > > > > > I started the upgrade from 4.3->4.4.
Now I'm stuck as restoring the
> > > > > > > > > > backup fails to create the correct
storage domain for Hosted Engine.
> > > > > > > > > > How can I create one? Error from ansible
task is:
> > > > > > > > > >
> > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason
is "Operation Failed". Fault
> > > > > > > > > > detail is "[Domain format is
different from master storage domain
> > > > > > > > > > format]". HTTP response code is
400.
> > > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED!
=> {"changed": false, "msg":
> > > > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > > > format is different from master storage
domain format]\". HTTP
> > > > > > > > > > response code is 400."}
> > > > > > > > > >
> > > > > > > > > > And from UI I can see that what has been
created is a data domain. Not
> > > > > > > > > > data (master) domain as old one was. Now
I'm stuck here. This is a
> > > > > > > > > > fibre channel system where I'm
trying to do this if it is relevant.
> > > > > > > > > >
> > > > > > > > > > What I could find from the logs is
this:
> > > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > > >
otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage',
> > > > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > > > 'permissions': [],
'status': 'unattached', 'storage': {'type':
'fcp',
> > > > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > > >
'36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths':
0,
> > > > > > > > > > 'product_id': 'Compellent
Vol', 'serial':
> > > > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > > > [], 'storage_format':
'v5', 'supports_discard': True,
> > > > > > > > > > 'supports_discard_zeroes_data':
False, 'templates': [], 'type':
> > > > > > > > > > 'data', 'used':
4294967296, 'vms': [], 'warning_low_space_indicator':
> > > > > > > > > > 10, 'wipe_after_delete':
False}], 'failed': False}
> > > > > > > > > >
> > > > > > > > > > 'master': False? I'm not
sure if this creation or check. I tried this
> > > > > > > > > > opration twice. I removed the new
hosted_storage and remove also old
> > > > > > > > > > hosted_engine domain on second try to
make sure that it doesn't
> > > > > > > > > > prevent creation of another master. No
luck with that.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Juhani
> > > > > > > > >
_______________________________________________
> > > > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > > > To unsubscribe send an email to
users-leave(a)ovirt.org
> > > > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Didi
> > > > > > > >
> > > >
> > >
> > >
> > > --
> > > Didi
> > >
> >
>
>
> --
> Didi
>
4:44 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Wed, Apr 28, 2021 at 11:16 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
I found these using dmsetup ls -tree:
6db20b74--512d--4a70--994e--8923d9e1e50b-master (253:21)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
6db20b74--512d--4a70--994e--8923d9e1e50b-inbox (253:17)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
db64924d--aa94--48ad--809a--3d235e84686b-ids (253:22)
└─36000d31005b4f600000000000000001d (253:12)
├─ (8:112)
├─ (8:96)
├─ (8:64)
└─ (8:48)
df259f5a--e1bd--4a93--bd6e--bace450625dc-master (253:35)
└─36000d31005b4f600000000000000001a (253:13)
├─ (8:144)
├─ (8:128)
├─ (8:80)
└─ (8:32)
6db20b74--512d--4a70--994e--8923d9e1e50b-outbox (253:18)
└─36000d31005b4f6000000000000000029 (253:36)
├─ (65:64)
├─ (65:80)
├─ (65:32)
└─ (65:48)
ID matches the one in ansible error and vdsm log. Are these done by
oVirt? I mean it's pretty hard to know what to free if you don't know
what should be and should not be there. And if these can be cleared
how do you do that?
Storage is not my expertise, but I think that's not the point. Yes, oVirt
creates/created them. But this does not mean that pvcreate should fail.
It most likely fails because something else is using the device somehow.
If this "something" is an oVirt process, that's likely a bug in oVirt.
Can you please try checking this?
By the way does anyone know how I can connect back to that HE that is
hosted locally during installation (URL like
https://ovirt02.virt.local:6900/ovirt-engine/). It seems the
connection disappeared this morning. Now I'm totally blind what's
going on in both nodes... Maybe I have to clear this SHE recovery and
start from the scratch.
If the VM is still running (you can check with e.g. 'ps auxww | grep qemu'),
you can try connecting to it. You should be able to find its (local) IP address
in the setup logs, search for "local_vm_ip".
Best regards,
Thanks,
Juhani
On Wed, Apr 28, 2021 at 9:57 AM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > Hi!
> >
> > This is from ansible logs:
> > 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
> > "ansible_host": "localhost",
> > "ansible_playbook":
> > "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
> > "ansible_result": {
> > "_ansible_no_log": false,
> > "changed": false,
> > "exception": "Traceback (most recent call last):\n
File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
> > line 804, in main\n File
> >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
> > line 651, in create\n **kwargs\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> > 26182, in add\n return self._internal_add(storage_domain, headers,
> > query, wait)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
232,
> > in _internal_add\n return future.wait() if wait else future\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
55,
> > in wait\n return self._code(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
229,
> > in callback\n self._check_fault(response)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
132,
> > in _check_fault\n self._raise_error(response, body)\n File
> > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line
118,
> > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> > \"Operation Failed\". Fault detail is \"[Physical device
> > initialization failed. Please check that the device is empty and
> > accessible by the host.]\". HTTP response code is 400.\n",
> > "invocation": {
> > "module_args": {
> > "backup": null,
> > "comment": null,
> > "critical_space_action_blocker": null,
> > "data_center": "Default",
> > "description": null,
> > "destroy": null,
> > "discard_after_delete": true,
> > "domain_function": "data",
> > "fcp": {
> > "lun_id":
"36000d31005b4f6000000000000000029"
> > },
> > "fetch_nested": false,
> > "format": null,
> > "glusterfs": null,
> > "host": "ovirt02.virt.local",
> > "id": null,
> > "iscsi": null,
> > "localfs": null,
> > "managed_block_storage": null,
> > "name": "hosted_storage",
> > "nested_attributes": [],
> > "nfs": null,
> > "poll_interval": 3,
> > "posixfs": null,
> > "state": "unattached",
> > "timeout": 180,
> > "wait": true,
> > "warning_low_space": null,
> > "wipe_after_delete": null
> > }
> > },
> > "msg": "Fault reason is \"Operation Failed\".
Fault detail is
> > \"[Physical device initialization failed. Please check that the device
> > is empty and accessible by the host.]\". HTTP response code is 400."
> > },
> > "ansible_task": "Add Fibre Channel storage domain",
> > "ansible_type": "task",
> > "status": "FAILED",
> >
> > This seem to happen same time:
> >
> > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> > getDeviceList return={'devList': [{'GUID':
> > '36000d31005b4f6000000000000000029', 'capacity':
'536870912000',
> > 'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
> > 'pathlist': [], 'pathstatus': [{'physdev':
'sdu', 'state': 'active',
> > 'capacity': '536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev':
> > 'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
> > 'type': 'FCP'}, {'physdev': 'sdv',
'state': 'active', 'capacity':
> > '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev': 'sdt',
> > 'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
> > 'FCP'}], 'physicalblocksize': '4096',
'productID': 'Compellent Vol',
> > 'pvUUID': '', 'pvsize': '', 'serial':
> > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
> > 'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
> > 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
> > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> > devlist=['36000d31005b4f6000000000000000029'], force=False,
> > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> > with specific filter failed or returned no data, retrying with a wider
> > filter, cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices {
> > preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> > filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> > hints="none" obtain_device_list_from_udev=0 } global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
out=[]
> > err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
> > exclusively. Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:511)
>
> It seems like this ^^^ is the problem. I suggest to check around e.g.
> with lsof etc., and/or search the net for something like
>
> "pvcreate" "Can't open" "exclusively. Mounted
filesystem?"
>
> Best regards,
>
> > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> > tries have failed: cmd=['/sbin/lvm', 'pvcreate',
'--config', 'devices
> > { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> > write_cache_state=0 disable_after_error_count=3
> >
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
> > "r|.*|"] hints="none" obtain_device_list_from_udev=0 }
global {
> > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
err=["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:538)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> > failed with rc=5 (lvm:1142)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
> > Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> > Mounted filesystem?", " Can't open
> > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > filesystem?"] (lvm:1143)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> > createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> > from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
> > (task:880)
> > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> > [storage.TaskManager.Task]
> > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
> > aborted: 'value=Failed to initialize physical device:
> > ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> > abortedcode=601' (task:1190)
> > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> > FINISH createVG error=Failed to initialize physical device:
> > ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
(dispatcher:83)
> >
> > I tried clearing the disk but from above it seems that something else
> > is wonky here.
> >
> > Thanks,
> > Juhani
> >
> > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > >
> > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > Story of the problems continues. Finally shut everything down, got
> > > > storage domains to maintenance and then this happens:
> > > >
> > > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is
> > > > "[Physical device initialization failed. Please check that the
device
> > > > is empty and accessible by the host.]". HTTP response code is
400.
> > > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed":
false, "msg":
> > > > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > > > device initialization failed. Please check that the device is empty
> > > > and accessible by the host.]\". HTTP response code is
400."}
> > >
> > > Can you please check/share relevant parts of vdsm.log when this happens?
> > > Thanks.
> > >
> > > >
> > > > No amount zeroing device helped.
> > > >
> > > > Next plan: Find out if I can restore backup to standalone server. If
> > > > that fails that's the end of oVirt for me. It worked fine for a
couple
> > > > of years but this update hassle is too much. Should have gone until
> > > > the end with 4.3.
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > Hmm. Is it possible that when the other node is still running
v4.3
> > > > > this operation can't be completed as it doesn't know how
to do it.
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > It seems that it is not supported in oVirt yet? I got this
response
> > > > > > when I tried to change master with those storage domain
that I have:
> > > > > >
> > > > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > > > <fault>
> > > > > > <detail>[Cannot switch master storage domain.
Switch master
> > > > > > storage domain operation is not supported.]</detail>
> > > > > > <reason>Operation Failed</reason>
> > > > > > </fault>
> > > > > >
> > > > > > So is this really the only way to do this: shutdown
everything and put
> > > > > > other storage domains into maintenance? It would have been
nice if
> > > > > > this information was in the upgrade guide. It made this
seem so easy
> > > > > > and simple...
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > Thanks this looks like what I'm looking for.
I'm still wondering how
> > > > > > > to use this. I have LUN just for new hosted storage.
Ansible created
> > > > > > > storage domain to it correctly but just can't
activate it. So is the
> > > > > > > idea that I activate this unattached hosted_storage
domain and try to
> > > > > > > use API to make it master? I attached a screen shot
how it looks
> > > > > > > currently.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani
Rautiainen
> > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > To continue. I noticed that another storage
domain took the data
> > > > > > > > > (master) now. I saw one advice that you can
force change by putting
> > > > > > > > > the storage domain to maintenance mode.
Problem is that there are VM's
> > > > > > > > > running on these domains. How is this
supposed to work during the
> > > > > > > > > restore?
> > > > > > > >
> > > > > > > > There is a recent change [1] by Shani (Cced) that
should allow you
> > > > > > > > to choose another storage domain as master. So
you can create a new
> > > > > > > > (temporary?) SD with the correct compatibility
level and then set it
> > > > > > > > to master.
> > > > > > > >
> > > > > > > > [1]
https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Juhani
> > > > > > > > >
> > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani
Rautiainen
> > > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > Hi!
> > > > > > > > > >
> > > > > > > > > > I started the upgrade from 4.3->4.4.
Now I'm stuck as restoring the
> > > > > > > > > > backup fails to create the correct
storage domain for Hosted Engine.
> > > > > > > > > > How can I create one? Error from
ansible task is:
> > > > > > > > > >
> > > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason
is "Operation Failed". Fault
> > > > > > > > > > detail is "[Domain format is
different from master storage domain
> > > > > > > > > > format]". HTTP response code is
400.
> > > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED!
=> {"changed": false, "msg":
> > > > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is \"[Domain
> > > > > > > > > > format is different from master storage
domain format]\". HTTP
> > > > > > > > > > response code is 400."}
> > > > > > > > > >
> > > > > > > > > > And from UI I can see that what has
been created is a data domain. Not
> > > > > > > > > > data (master) domain as old one was.
Now I'm stuck here. This is a
> > > > > > > > > > fibre channel system where I'm
trying to do this if it is relevant.
> > > > > > > > > >
> > > > > > > > > > What I could find from the logs is
this:
> > > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > > >
otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > > ansible_utils._process_output:105
storage_domain_details: {'changed':
> > > > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage',
> > > > > > > > > > 'available': 531502202880,
'backup': False, 'block_size': 512,
> > > > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > > > 'discard_after_delete': True,
'disk_profiles': [], 'disk_snapshots':
> > > > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > > > 'permissions': [],
'status': 'unattached', 'storage': {'type':
'fcp',
> > > > > > > > > > 'volume_group': {'id':
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > > >
'36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths':
0,
> > > > > > > > > > 'product_id': 'Compellent
Vol', 'serial':
> > > > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > 'vendor_id':
'COMPELNT', 'volume_group_id':
> > > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > > > [], 'storage_format':
'v5', 'supports_discard': True,
> > > > > > > > > > 'supports_discard_zeroes_data':
False, 'templates': [], 'type':
> > > > > > > > > > 'data', 'used':
4294967296, 'vms': [], 'warning_low_space_indicator':
> > > > > > > > > > 10, 'wipe_after_delete':
False}], 'failed': False}
> > > > > > > > > >
> > > > > > > > > > 'master': False? I'm not
sure if this creation or check. I tried this
> > > > > > > > > > opration twice. I removed the new
hosted_storage and remove also old
> > > > > > > > > > hosted_engine domain on second try to
make sure that it doesn't
> > > > > > > > > > prevent creation of another master. No
luck with that.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Juhani
> > > > > > > > >
_______________________________________________
> > > > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > > > To unsubscribe send an email to
users-leave(a)ovirt.org
> > > > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Didi
> > > > > > > >
> > > >
> > >
> > >
> > > --
> > > Didi
> > >
> >
>
>
> --
> Didi
>
--
Didi
6:02 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
Hi!
I know something about multipathing and LVM but I'm not quite familiar
how this is supposed to look. I compared 4.4 node to 4.3 node and it
looks totally different from lvm perpective. I'm not sure if this is
because of changes or problems in this node. Multipath shows same
disks but for example pvdisplay on 4.4 node shows only local disk PV:
[root@ovirt02 tmp]# pvdisplay
--- Physical volume ---
PV Name /dev/sda3
VG Name onn
On old node:
[root@ovirt01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/36000d31005b4f600000000000000001a
--- Physical volume ---
PV Name /dev/mapper/36000d31005b4f600000000000000001d
--- Physical volume ---
PV Name /dev/mapper/360002ac00000000000000261000057b9
--- Physical volume ---
PV Name /dev/mapper/3600508b1001cce98eb718bd780426566p3
VG Name onn_ovirt01
This worries me as I don't have an idea how to fix this.
Thanks,
Juhani
On Wed, Apr 28, 2021 at 12:45 PM Yedidyah Bar David <didi(a)redhat.com> wrote:
>
> On Wed, Apr 28, 2021 at 11:16 AM Juhani Rautiainen
> <juhani.rautiainen(a)gmail.com> wrote:
> >
> > I found these using dmsetup ls -tree:
> >
> > 6db20b74--512d--4a70--994e--8923d9e1e50b-master (253:21)
> > └─36000d31005b4f6000000000000000029 (253:36)
> > ├─ (65:64)
> > ├─ (65:80)
> > ├─ (65:32)
> > └─ (65:48)
> > 6db20b74--512d--4a70--994e--8923d9e1e50b-inbox (253:17)
> > └─36000d31005b4f6000000000000000029 (253:36)
> > ├─ (65:64)
> > ├─ (65:80)
> > ├─ (65:32)
> > └─ (65:48)
> > db64924d--aa94--48ad--809a--3d235e84686b-ids (253:22)
> > └─36000d31005b4f600000000000000001d (253:12)
> > ├─ (8:112)
> > ├─ (8:96)
> > ├─ (8:64)
> > └─ (8:48)
> > df259f5a--e1bd--4a93--bd6e--bace450625dc-master (253:35)
> > └─36000d31005b4f600000000000000001a (253:13)
> > ├─ (8:144)
> > ├─ (8:128)
> > ├─ (8:80)
> > └─ (8:32)
> > 6db20b74--512d--4a70--994e--8923d9e1e50b-outbox (253:18)
> > └─36000d31005b4f6000000000000000029 (253:36)
> > ├─ (65:64)
> > ├─ (65:80)
> > ├─ (65:32)
> > └─ (65:48)
> >
> > ID matches the one in ansible error and vdsm log. Are these done by
> > oVirt? I mean it's pretty hard to know what to free if you don't know
> > what should be and should not be there. And if these can be cleared
> > how do you do that?
>
> Storage is not my expertise, but I think that's not the point. Yes, oVirt
> creates/created them. But this does not mean that pvcreate should fail.
> It most likely fails because something else is using the device somehow.
> If this "something" is an oVirt process, that's likely a bug in oVirt.
> Can you please try checking this?
>
> >
> > By the way does anyone know how I can connect back to that HE that is
> > hosted locally during installation (URL like
> > https://ovirt02.virt.local:6900/ovirt-engine/). It seems the
> > connection disappeared this morning. Now I'm totally blind what's
> > going on in both nodes... Maybe I have to clear this SHE recovery and
> > start from the scratch.
>
> If the VM is still running (you can check with e.g. 'ps auxww | grep qemu'),
> you can try connecting to it. You should be able to find its (local) IP address
> in the setup logs, search for "local_vm_ip".
>
> Best regards,
>
> >
> > Thanks,
> > Juhani
> >
> > On Wed, Apr 28, 2021 at 9:57 AM Yedidyah Bar David <didi(a)redhat.com>
wrote:
> > >
> > > On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > Hi!
> > > >
> > > > This is from ansible logs:
> > > > 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
> > > > "ansible_host": "localhost",
> > > > "ansible_playbook":
> > > >
"/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
> > > > "ansible_result": {
> > > > "_ansible_no_log": false,
> > > > "changed": false,
> > > > "exception": "Traceback (most recent call
last):\n File
> > > >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_storage_domain.py\",
> > > > line 804, in main\n File
> > > >
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage_domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/ovirt.py\",
> > > > line 651, in create\n **kwargs\n File
> > > >
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> > > > 26182, in add\n return self._internal_add(storage_domain, headers,
> > > > query, wait)\n File
> > > > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\",
line 232,
> > > > in _internal_add\n return future.wait() if wait else future\n
File
> > > > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\",
line 55,
> > > > in wait\n return self._code(response)\n File
> > > > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\",
line 229,
> > > > in callback\n self._check_fault(response)\n File
> > > > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\",
line 132,
> > > > in _check_fault\n self._raise_error(response, body)\n File
> > > > \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\",
line 118,
> > > > in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> > > > \"Operation Failed\". Fault detail is \"[Physical
device
> > > > initialization failed. Please check that the device is empty and
> > > > accessible by the host.]\". HTTP response code is 400.\n",
> > > > "invocation": {
> > > > "module_args": {
> > > > "backup": null,
> > > > "comment": null,
> > > > "critical_space_action_blocker": null,
> > > > "data_center": "Default",
> > > > "description": null,
> > > > "destroy": null,
> > > > "discard_after_delete": true,
> > > > "domain_function": "data",
> > > > "fcp": {
> > > > "lun_id":
"36000d31005b4f6000000000000000029"
> > > > },
> > > > "fetch_nested": false,
> > > > "format": null,
> > > > "glusterfs": null,
> > > > "host": "ovirt02.virt.local",
> > > > "id": null,
> > > > "iscsi": null,
> > > > "localfs": null,
> > > > "managed_block_storage": null,
> > > > "name": "hosted_storage",
> > > > "nested_attributes": [],
> > > > "nfs": null,
> > > > "poll_interval": 3,
> > > > "posixfs": null,
> > > > "state": "unattached",
> > > > "timeout": 180,
> > > > "wait": true,
> > > > "warning_low_space": null,
> > > > "wipe_after_delete": null
> > > > }
> > > > },
> > > > "msg": "Fault reason is \"Operation
Failed\". Fault detail is
> > > > \"[Physical device initialization failed. Please check that the
device
> > > > is empty and accessible by the host.]\". HTTP response code is
400."
> > > > },
> > > > "ansible_task": "Add Fibre Channel storage
domain",
> > > > "ansible_type": "task",
> > > > "status": "FAILED",
> > > >
> > > > This seem to happen same time:
> > > >
> > > > 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> > > > getDeviceList return={'devList': [{'GUID':
> > > > '36000d31005b4f6000000000000000029', 'capacity':
'536870912000',
> > > > 'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
> > > > 'pathlist': [], 'pathstatus': [{'physdev':
'sdu', 'state': 'active',
> > > > 'capacity': '536870912000', 'lun':
'3', 'type': 'FCP'}, {'physdev':
> > > > 'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
> > > > 'type': 'FCP'}, {'physdev': 'sdv',
'state': 'active', 'capacity':
> > > > '536870912000', 'lun': '3', 'type':
'FCP'}, {'physdev': 'sdt',
> > > > 'state': 'active', 'capacity':
'536870912000', 'lun': '3', 'type':
> > > > 'FCP'}], 'physicalblocksize': '4096',
'productID': 'Compellent Vol',
> > > > 'pvUUID': '', 'pvsize': '',
'serial':
> > > > 'SCOMPELNTCompellent_Vol_0005b4f6-00000029',
'vendorID': 'COMPELNT',
> > > > 'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
> > > > 0, 'status': 'unknown'}]}
from=::ffff:192.168.222.89,57472,
> > > > flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> > > > task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> > > > 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> > > > createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> > > > devlist=['36000d31005b4f6000000000000000029'], force=False,
> > > > options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > > > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> > > > 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> > > > with specific filter failed or returned no data, retrying with a
wider
> > > > filter, cmd=['/sbin/lvm', 'pvcreate',
'--config', 'devices {
> > > > preferred_names=["^/dev/mapper/"]
ignore_suspended_devices=1
> > > > write_cache_state=0 disable_after_error_count=3
> > > >
filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> > > > hints="none" obtain_device_list_from_udev=0 } global {
> > > > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > > > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > > > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > > > 'y', '/dev/mapper/36000d31005b4f6000000000000000029']
rc=5 out=[]
> > > > err=[" Can't open
/dev/mapper/36000d31005b4f6000000000000000029
> > > > exclusively. Mounted filesystem?", " Can't open
> > > > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > > > filesystem?"] (lvm:511)
> > >
> > > It seems like this ^^^ is the problem. I suggest to check around e.g.
> > > with lsof etc., and/or search the net for something like
> > >
> > > "pvcreate" "Can't open" "exclusively. Mounted
filesystem?"
> > >
> > > Best regards,
> > >
> > > > 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> > > > tries have failed: cmd=['/sbin/lvm', 'pvcreate',
'--config', 'devices
> > > > { preferred_names=["^/dev/mapper/"]
ignore_suspended_devices=1
> > > > write_cache_state=0 disable_after_error_count=3
> > > >
filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f600000000000000001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
> > > > "r|.*|"] hints="none"
obtain_device_list_from_udev=0 } global {
> > > > locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> > > > use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> > > > '--metadatasize', '128m', '--metadatacopies',
'2', '--metadataignore',
> > > > 'y', '/dev/mapper/36000d31005b4f6000000000000000029']
rc=5 err=["
> > > > Can't open /dev/mapper/36000d31005b4f6000000000000000029
exclusively.
> > > > Mounted filesystem?", " Can't open
> > > > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > > > filesystem?"] (lvm:538)
> > > > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> > > > failed with rc=5 (lvm:1142)
> > > > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [],
["
> > > > Can't open /dev/mapper/36000d31005b4f6000000000000000029
exclusively.
> > > > Mounted filesystem?", " Can't open
> > > > /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> > > > filesystem?"] (lvm:1143)
> > > > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> > > > createVG error=Failed to initialize physical device:
> > > >
("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> > > > from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> > > > task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> > > > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> > > > [storage.TaskManager.Task]
> > > > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected
error
> > > > (task:880)
> > > > 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> > > > [storage.TaskManager.Task]
> > > > (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task
is
> > > > aborted: 'value=Failed to initialize physical device:
> > > >
("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> > > > abortedcode=601' (task:1190)
> > > > 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> > > > FINISH createVG error=Failed to initialize physical device:
> > > >
("['/dev/mapper/36000d31005b4f6000000000000000029']",) (dispatcher:83)
> > > >
> > > > I tried clearing the disk but from above it seems that something else
> > > > is wonky here.
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > > On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David
<didi(a)redhat.com> wrote:
> > > > >
> > > > > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > >
> > > > > > Story of the problems continues. Finally shut everything
down, got
> > > > > > storage domains to maintenance and then this happens:
> > > > > >
> > > > > > ovirtsdk4.Error: Fault reason is "Operation
Failed". Fault detail is
> > > > > > "[Physical device initialization failed. Please check
that the device
> > > > > > is empty and accessible by the host.]". HTTP response
code is 400.
> > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed": false, "msg":
> > > > > > "Fault reason is \"Operation Failed\". Fault
detail is \"[Physical
> > > > > > device initialization failed. Please check that the device
is empty
> > > > > > and accessible by the host.]\". HTTP response code is
400."}
> > > > >
> > > > > Can you please check/share relevant parts of vdsm.log when this
happens?
> > > > > Thanks.
> > > > >
> > > > > >
> > > > > > No amount zeroing device helped.
> > > > > >
> > > > > > Next plan: Find out if I can restore backup to standalone
server. If
> > > > > > that fails that's the end of oVirt for me. It worked
fine for a couple
> > > > > > of years but this update hassle is too much. Should have
gone until
> > > > > > the end with 4.3.
> > > > > >
> > > > > > Thanks,
> > > > > > Juhani
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > Hmm. Is it possible that when the other node is still
running v4.3
> > > > > > > this operation can't be completed as it doesn't
know how to do it.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Juhani
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > It seems that it is not supported in oVirt yet? I
got this response
> > > > > > > > when I tried to change master with those storage
domain that I have:
> > > > > > > >
> > > > > > > > <?xml version="1.0"
encoding="UTF-8" standalone="yes"?>
> > > > > > > > <fault>
> > > > > > > > <detail>[Cannot switch master storage
domain. Switch master
> > > > > > > > storage domain operation is not
supported.]</detail>
> > > > > > > > <reason>Operation Failed</reason>
> > > > > > > > </fault>
> > > > > > > >
> > > > > > > > So is this really the only way to do this:
shutdown everything and put
> > > > > > > > other storage domains into maintenance? It would
have been nice if
> > > > > > > > this information was in the upgrade guide. It made
this seem so easy
> > > > > > > > and simple...
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Juhani
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani
Rautiainen
> > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Thanks this looks like what I'm looking
for. I'm still wondering how
> > > > > > > > > to use this. I have LUN just for new hosted
storage. Ansible created
> > > > > > > > > storage domain to it correctly but just
can't activate it. So is the
> > > > > > > > > idea that I activate this unattached
hosted_storage domain and try to
> > > > > > > > > use API to make it master? I attached a
screen shot how it looks
> > > > > > > > > currently.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar
David <didi(a)redhat.com> wrote:
> > > > > > > > > >
> > > > > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani
Rautiainen
> > > > > > > > > > <juhani.rautiainen(a)gmail.com>
wrote:
> > > > > > > > > > >
> > > > > > > > > > > To continue. I noticed that another
storage domain took the data
> > > > > > > > > > > (master) now. I saw one advice that
you can force change by putting
> > > > > > > > > > > the storage domain to maintenance
mode. Problem is that there are VM's
> > > > > > > > > > > running on these domains. How is
this supposed to work during the
> > > > > > > > > > > restore?
> > > > > > > > > >
> > > > > > > > > > There is a recent change [1] by Shani
(Cced) that should allow you
> > > > > > > > > > to choose another storage domain as
master. So you can create a new
> > > > > > > > > > (temporary?) SD with the correct
compatibility level and then set it
> > > > > > > > > > to master.
> > > > > > > > > >
> > > > > > > > > > [1]
https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Juhani
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM
Juhani Rautiainen
> > > > > > > > > > > <juhani.rautiainen(a)gmail.com>
wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > Hi!
> > > > > > > > > > > >
> > > > > > > > > > > > I started the upgrade from
4.3->4.4. Now I'm stuck as restoring the
> > > > > > > > > > > > backup fails to create the
correct storage domain for Hosted Engine.
> > > > > > > > > > > > How can I create one? Error
from ansible task is:
> > > > > > > > > > > >
> > > > > > > > > > > > [ ERROR ] ovirtsdk4.Error:
Fault reason is "Operation Failed". Fault
> > > > > > > > > > > > detail is "[Domain format
is different from master storage domain
> > > > > > > > > > > > format]". HTTP response
code is 400.
> > > > > > > > > > > > [ ERROR ] fatal: [localhost]:
FAILED! => {"changed": false, "msg":
> > > > > > > > > > > > "Fault reason is
\"Operation Failed\". Fault detail is \"[Domain
> > > > > > > > > > > > format is different from
master storage domain format]\". HTTP
> > > > > > > > > > > > response code is 400."}
> > > > > > > > > > > >
> > > > > > > > > > > > And from UI I can see that
what has been created is a data domain. Not
> > > > > > > > > > > > data (master) domain as old
one was. Now I'm stuck here. This is a
> > > > > > > > > > > > fibre channel system where
I'm trying to do this if it is relevant.
> > > > > > > > > > > >
> > > > > > > > > > > > What I could find from the
logs is this:
> > > > > > > > > > > > 2021-04-27 09:36:06,925+0300
DEBUG
> > > > > > > > > > > >
otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > > > >
ansible_utils._process_output:105 storage_domain_details: {'changed':
> > > > > > > > > > > > False,
'ovirt_storage_domains': [{'href':
> > > > > > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name': 'hosted_storage',
> > > > > > > > > > > > 'available':
531502202880, 'backup': False, 'block_size': 512,
> > > > > > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > > > > >
'discard_after_delete': True, 'disk_profiles': [],
'disk_snapshots':
> > > > > > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master': False,
> > > > > > > > > > > > 'permissions': [],
'status': 'unattached', 'storage': {'type':
'fcp',
> > > > > > > > > > > > 'volume_group':
{'id': 'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > > > > > 'discard_zeroes_data':
False, 'id':
> > > > > > > > > > > >
'36000d31005b4f6000000000000000029', 'lun_mapping': 3, 'paths':
0,
> > > > > > > > > > > > 'product_id':
'Compellent Vol', 'serial':
> > > > > > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size': 536870912000,
> > > > > > > > > > > > 'storage_domain_id':
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > > > > 'vendor_id':
'COMPELNT', 'volume_group_id':
> > > > > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}}, 'storage_connections':
> > > > > > > > > > > > [], 'storage_format':
'v5', 'supports_discard': True,
> > > > > > > > > > > >
'supports_discard_zeroes_data': False, 'templates': [], 'type':
> > > > > > > > > > > > 'data',
'used': 4294967296, 'vms': [], 'warning_low_space_indicator':
> > > > > > > > > > > > 10,
'wipe_after_delete': False}], 'failed': False}
> > > > > > > > > > > >
> > > > > > > > > > > > 'master': False?
I'm not sure if this creation or check. I tried this
> > > > > > > > > > > > opration twice. I removed the
new hosted_storage and remove also old
> > > > > > > > > > > > hosted_engine domain on second
try to make sure that it doesn't
> > > > > > > > > > > > prevent creation of another
master. No luck with that.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Juhani
> > > > > > > > > > >
_______________________________________________
> > > > > > > > > > > Users mailing list --
users(a)ovirt.org
> > > > > > > > > > > To unsubscribe send an email to
users-leave(a)ovirt.org
> > > > > > > > > > > Privacy Statement:
https://www.ovirt.org/privacy-policy.html
> > > > > > > > > > > oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > > > > > List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TZHLFDIYRBQ...
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Didi
> > > > > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Didi
> > > > >
> > > >
> > >
> > >
> > > --
> > > Didi
> > >
> >
>
>
> --
> Didi
>
8:22 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Wednesday, 28 April 2021 13:02:01 CEST Juhani Rautiainen wrote:
Hi!
I know something about multipathing and LVM but I'm not quite familiar
how this is supposed to look. I compared 4.4 node to 4.3 node and it
looks totally different from lvm perpective. I'm not sure if this is
because of changes
there are no change in disk layout between 4.3 and 4.4 AFAIK
or problems in this node. Multipath shows same
disks but for example pvdisplay on 4.4 node shows only local disk PV:
showing all PVs means you don't have configured lvm filter on the hosts, you
should run
vdsm-tool config-lvm-filter
on each host, see [1] for more details.
Not showing any PVs on may be caused by installed lvm filter (you can run it
again with --config 'devices {filter=["a|.*|"]}' to find out) or because
there
are no PVs created yet, which is more likely as in previous log there was an
error during pvcreate.
[1] https://blogs.ovirt.org/2017/12/lvm-configuration-the-easy-way/
[root@ovirt02 tmp]# pvdisplay
--- Physical volume ---
PV Name /dev/sda3
VG Name onn
On old node:
[root@ovirt01 ~]# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/36000d31005b4f600000000000000001a
--- Physical volume ---
PV Name /dev/mapper/36000d31005b4f600000000000000001d
--- Physical volume ---
PV Name /dev/mapper/360002ac00000000000000261000057b9
--- Physical volume ---
PV Name /dev/mapper/3600508b1001cce98eb718bd780426566p3
VG Name onn_ovirt01
This worries me as I don't have an idea how to fix this.
Thanks,
Juhani
8:15 a.m.
New subject: Restoring hosted engine from backup fails on new FC storage domain creation
On Wednesday, 28 April 2021 08:57:32 CEST Yedidyah Bar David wrote:
On Wed, Apr 28, 2021 at 9:29 AM Juhani Rautiainen
<juhani.rautiainen(a)gmail.com> wrote:
>
>
> Hi!
>
>
>
> This is from ansible logs:
> 2021-04-27 22:20:38,286+0300 ERROR ansible failed {
>
> "ansible_host": "localhost",
> "ansible_playbook":
>
> "/usr/share/ovirt-hosted-engine-setup/ansible/trigger_role.yml",
>
> "ansible_result": {
>
> "_ansible_no_log": false,
> "changed": false,
> "exception": "Traceback (most recent call last):\n File
>
> \"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage
> _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/modules/ovirt_
> storage_domain.py\",
line 804, in main\n File
>
\"/tmp/ansible_ovirt_storage_domain_payload_hlz7j9sk/ansible_ovirt_storage
> _domain_payload.zip/ansible_collections/ovirt/ovirt/plugins/module_utils/o
> virt.py\",
line 651, in create\n **kwargs\n File
>
\"/usr/lib64/python3.6/site-packages/ovirtsdk4/services.py\", line
> 26182, in add\n return self._internal_add(storage_domain, headers,
> query, wait)\n File
> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 232,
> in _internal_add\n return future.wait() if wait else future\n File
> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 55,
> in wait\n return self._code(response)\n File
> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 229,
> in callback\n self._check_fault(response)\n File
> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 132,
> in _check_fault\n self._raise_error(response, body)\n File
> \"/usr/lib64/python3.6/site-packages/ovirtsdk4/service.py\", line 118,
> in _raise_error\n raise error\novirtsdk4.Error: Fault reason is
> \"Operation Failed\". Fault detail is \"[Physical device
> initialization failed. Please check that the device is empty and
> accessible by the host.]\". HTTP response code is 400.\n",
>
> "invocation": {
>
> "module_args": {
>
> "backup": null,
> "comment": null,
> "critical_space_action_blocker": null,
> "data_center": "Default",
> "description": null,
> "destroy": null,
> "discard_after_delete": true,
> "domain_function": "data",
> "fcp": {
>
> "lun_id":
"36000d31005b4f6000000000000000029"
>
> },
> "fetch_nested": false,
> "format": null,
> "glusterfs": null,
> "host": "ovirt02.virt.local",
> "id": null,
> "iscsi": null,
> "localfs": null,
> "managed_block_storage": null,
> "name": "hosted_storage",
> "nested_attributes": [],
> "nfs": null,
> "poll_interval": 3,
> "posixfs": null,
> "state": "unattached",
> "timeout": 180,
> "wait": true,
> "warning_low_space": null,
> "wipe_after_delete": null
>
> }
>
> },
> "msg": "Fault reason is \"Operation Failed\". Fault
detail is
>
> \"[Physical device initialization failed. Please check that the device
> is empty and accessible by the host.]\". HTTP response code is 400."
>
> },
> "ansible_task": "Add Fibre Channel storage domain",
> "ansible_type": "task",
> "status": "FAILED",
>
>
>
> This seem to happen same time:
>
>
>
> 2021-04-27 22:20:37,576+0300 INFO (jsonrpc/5) [vdsm.api] FINISH
> getDeviceList return={'devList': [{'GUID':
> '36000d31005b4f6000000000000000029', 'capacity':
'536870912000',
> 'devtype': 'FCP', 'fwrev': '0704',
'logicalblocksize': '512',
> 'pathlist': [], 'pathstatus': [{'physdev': 'sdu',
'state': 'active',
> 'capacity': '536870912000', 'lun': '3',
'type': 'FCP'}, {'physdev':
> 'sds', 'state': 'active', 'capacity':
'536870912000', 'lun': '3',
> 'type': 'FCP'}, {'physdev': 'sdv', 'state':
'active', 'capacity':
> '536870912000', 'lun': '3', 'type': 'FCP'},
{'physdev': 'sdt',
> 'state': 'active', 'capacity': '536870912000',
'lun': '3', 'type':
> 'FCP'}], 'physicalblocksize': '4096', 'productID':
'Compellent Vol',
> 'pvUUID': '', 'pvsize': '', 'serial':
> 'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'vendorID':
'COMPELNT',
> 'vgUUID': '', 'discard_max_bytes': 268435456,
'discard_zeroes_data':
> 0, 'status': 'unknown'}]} from=::ffff:192.168.222.89,57472,
> flow_id=676e5ced-7a4f-4a36-8842-cb98a38f5f4b,
> task_id=792a93ab-5129-4bf2-8ff0-70b5cdf19221 (api:54)
> 2021-04-27 22:20:37,641+0300 INFO (jsonrpc/4) [vdsm.api] START
> createVG(vgname='bd3efbef-0178-4a21-9a13-2a293482fdd5',
> devlist=['36000d31005b4f6000000000000000029'], force=False,
> options=None) from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:48)
> 2021-04-27 22:20:37,729+0300 WARN (jsonrpc/4) [storage.LVM] Command
> with specific filter failed or returned no data, retrying with a wider
> filter, cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices {
> preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3
> filter=["a|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
>
> hints="none" obtain_device_list_from_udev=0 } global {
>
> locking_type=1 prioritise_write_locks=1 wait_for_locks=1
> use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> '--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
> 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5 out=[]
> err=[" Can't open /dev/mapper/36000d31005b4f6000000000000000029
> exclusively. Mounted filesystem?", " Can't open
> /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> filesystem?"] (lvm:511)
It seems like this ^^^ is the problem. I suggest to check around e.g.
with lsof etc., and/or search the net for something like
This is more likely caused by old signatures on the device. You should try to
wipe all the signatures with
wipefs -a /dev/mapper/36000d31005b4f6000000000000000029
of with
kpartx -d /dev/mapper/36000d31005b4f6000000000000000029
and try again
"pvcreate" "Can't open" "exclusively.
Mounted filesystem?"
Best regards,
> 2021-04-27 22:20:37,830+0300 WARN (jsonrpc/4) [storage.LVM] All 2
> tries have failed: cmd=['/sbin/lvm', 'pvcreate', '--config',
'devices
> { preferred_names=["^/dev/mapper/"] ignore_suspended_devices=1
> write_cache_state=0 disable_after_error_count=3
> filter=["a|^/dev/mapper/360002ac00000000000000261000057b9$|^/dev/mapper/36
> 000d31005b4f600000000000000001a$|^/dev/mapper/36000d31005b4f60000000000000
> 0001d$|^/dev/mapper/36000d31005b4f6000000000000000029$|",
"r|.*|"]
> hints="none" obtain_device_list_from_udev=0 } global
{ locking_type=1
> prioritise_write_locks=1 wait_for_locks=1
> use_lvmetad=0 } backup { retain_min=50 retain_days=0 }',
> '--metadatasize', '128m', '--metadatacopies', '2',
'--metadataignore',
> 'y', '/dev/mapper/36000d31005b4f6000000000000000029'] rc=5
err=["
> Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> Mounted filesystem?", " Can't open
> /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> filesystem?"] (lvm:538)
> 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] pvcreate
> failed with rc=5 (lvm:1142)
> 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.LVM] [], ["
> Can't open /dev/mapper/36000d31005b4f6000000000000000029 exclusively.
> Mounted filesystem?", " Can't open
> /dev/mapper/36000d31005b4f6000000000000000029 exclusively. Mounted
> filesystem?"] (lvm:1143)
> 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4) [vdsm.api] FINISH
> createVG error=Failed to initialize physical device:
> ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
> from=::ffff:192.168.222.89,57472, flow_id=6dce1780,
> task_id=4b269432-9f27-406b-a03b-6dfaaa2d1f67 (api:52)
> 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4)
> [storage.TaskManager.Task]
> (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') Unexpected error
> (task:880)
> 2021-04-27 22:20:37,830+0300 INFO (jsonrpc/4)
> [storage.TaskManager.Task]
> (Task='4b269432-9f27-406b-a03b-6dfaaa2d1f67') aborting: Task is
> aborted: 'value=Failed to initialize physical device:
> ("[\'/dev/mapper/36000d31005b4f6000000000000000029\']",)
> abortedcode=601' (task:1190)
> 2021-04-27 22:20:37,830+0300 ERROR (jsonrpc/4) [storage.Dispatcher]
> FINISH createVG error=Failed to initialize physical device:
> ("['/dev/mapper/36000d31005b4f6000000000000000029']",)
(dispatcher:83)
>
>
>
> I tried clearing the disk but from above it seems that something else
> is wonky here.
>
>
>
> Thanks,
> Juhani
>
>
>
> On Wed, Apr 28, 2021 at 8:30 AM Yedidyah Bar David <didi(a)redhat.com>
> wrote:
> >
> >
> > On Tue, Apr 27, 2021 at 10:59 PM Juhani Rautiainen
> > <juhani.rautiainen(a)gmail.com> wrote:
> >
> > >
> > >
> > > Story of the problems continues. Finally shut everything down, got
> > > storage domains to maintenance and then this happens:
> > >
> > >
> > >
> > > ovirtsdk4.Error: Fault reason is "Operation Failed". Fault
detail is
> > > "[Physical device initialization failed. Please check that the
device
> > > is empty and accessible by the host.]". HTTP response code is 400.
> > > [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false,
"msg":
> > > "Fault reason is \"Operation Failed\". Fault detail is
\"[Physical
> > > device initialization failed. Please check that the device is empty
> > > and accessible by the host.]\". HTTP response code is 400."}
> >
> >
> >
> > Can you please check/share relevant parts of vdsm.log when this
> > happens?
> > Thanks.
> >
> >
> >
> > >
> > >
> > > No amount zeroing device helped.
> > >
> > >
> > >
> > > Next plan: Find out if I can restore backup to standalone server. If
> > > that fails that's the end of oVirt for me. It worked fine for a
> > > couple
> > > of years but this update hassle is too much. Should have gone until
> > > the end with 4.3.
> > >
> > >
> > >
> > > Thanks,
> > > Juhani
> > >
> > >
> > >
> > > On Tue, Apr 27, 2021 at 11:47 AM Juhani Rautiainen
> > > <juhani.rautiainen(a)gmail.com> wrote:
> > >
> > > >
> > > >
> > > > Hmm. Is it possible that when the other node is still running v4.3
> > > > this operation can't be completed as it doesn't know how to
do it.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > > Juhani
> > > >
> > > >
> > > >
> > > > On Tue, Apr 27, 2021 at 11:07 AM Juhani Rautiainen
> > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > >
> > > > >
> > > > >
> > > > > It seems that it is not supported in oVirt yet? I got this
> > > > > response
> > > > > when I tried to change master with those storage domain that I
> > > > > have:
> > > > >
> > > > >
> > > > >
> > > > > <?xml version="1.0" encoding="UTF-8"
standalone="yes"?>
> > > > > <fault>
> > > > >
> > > > > <detail>[Cannot switch master storage domain. Switch
master
> > > > >
> > > > > storage domain operation is not supported.]</detail>
> > > > >
> > > > > <reason>Operation Failed</reason>
> > > > >
> > > > > </fault>
> > > > >
> > > > >
> > > > >
> > > > > So is this really the only way to do this: shutdown everything
and
> > > > > put
> > > > > other storage domains into maintenance? It would have been nice
> > > > > if
> > > > > this information was in the upgrade guide. It made this seem so
> > > > > easy
> > > > > and simple...
> > > > >
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Juhani
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Apr 27, 2021 at 10:47 AM Juhani Rautiainen
> > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > >
> > > > > >
> > > > > >
> > > > > > Thanks this looks like what I'm looking for. I'm
still wondering
> > > > > > how
> > > > > > to use this. I have LUN just for new hosted storage.
Ansible
> > > > > > created
> > > > > > storage domain to it correctly but just can't activate
it. So is
> > > > > > the
> > > > > > idea that I activate this unattached hosted_storage domain
and
> > > > > > try to
> > > > > > use API to make it master? I attached a screen shot how it
> > > > > > looks
> > > > > > currently.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Apr 27, 2021 at 10:41 AM Yedidyah Bar David
> > > > > > <didi(a)redhat.com> wrote:
> > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Apr 27, 2021 at 10:15 AM Juhani Rautiainen
> > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > To continue. I noticed that another storage
domain took the
> > > > > > > > data
> > > > > > > > (master) now. I saw one advice that you can force
change by
> > > > > > > > putting
> > > > > > > > the storage domain to maintenance mode. Problem
is that
> > > > > > > > there are VM's
> > > > > > > > running on these domains. How is this supposed to
work
> > > > > > > > during the
> > > > > > > > restore?
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > There is a recent change [1] by Shani (Cced) that
should allow
> > > > > > > you
> > > > > > > to choose another storage domain as master. So you can
create
> > > > > > > a new
> > > > > > > (temporary?) SD with the correct compatibility level
and then
> > > > > > > set it
> > > > > > > to master.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > [1]
https://bugzilla.redhat.com/show_bug.cgi?id=1576923
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Juhani
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Apr 27, 2021 at 9:58 AM Juhani
Rautiainen
> > > > > > > > <juhani.rautiainen(a)gmail.com> wrote:
> > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Hi!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I started the upgrade from 4.3->4.4. Now
I'm stuck as
> > > > > > > > > restoring the
> > > > > > > > > backup fails to create the correct storage
domain for
> > > > > > > > > Hosted Engine.
> > > > > > > > > How can I create one? Error from ansible
task is:
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > [ ERROR ] ovirtsdk4.Error: Fault reason is
"Operation
> > > > > > > > > Failed". Fault
> > > > > > > > > detail is "[Domain format is different
from master storage
> > > > > > > > > domain
> > > > > > > > > format]". HTTP response code is 400.
> > > > > > > > > [ ERROR ] fatal: [localhost]: FAILED! =>
{"changed":
> > > > > > > > > false, "msg":
> > > > > > > > > "Fault reason is \"Operation
Failed\". Fault detail is
> > > > > > > > > \"[Domain
> > > > > > > > > format is different from master storage
domain format]\".
> > > > > > > > > HTTP
> > > > > > > > > response code is 400."}
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > And from UI I can see that what has been
created is a data
> > > > > > > > > domain. Not
> > > > > > > > > data (master) domain as old one was. Now
I'm stuck here.
> > > > > > > > > This is a
> > > > > > > > > fibre channel system where I'm trying
to do this if it is
> > > > > > > > > relevant.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > What I could find from the logs is this:
> > > > > > > > > 2021-04-27 09:36:06,925+0300 DEBUG
> > > > > > > > >
otopi.ovirt_hosted_engine_setup.ansible_utils
> > > > > > > > > ansible_utils._process_output:105
storage_domain_details:
> > > > > > > > > {'changed':
> > > > > > > > > False, 'ovirt_storage_domains':
[{'href':
> > > > > > > > >
'/ovirt-engine/api/storagedomains/dd52022b-7616-47f6-9534-
> > > > > > > > > 6f1a4084fdf4',
> > > > > > > > > 'comment': '',
'description': '', 'id':
> > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4', 'name':
> > > > > > > > > 'hosted_storage',
> > > > > > > > > 'available': 531502202880,
'backup': False, 'block_size':
> > > > > > > > > 512,
> > > > > > > > > 'committed': 0,
'critical_space_action_blocker': 5,
> > > > > > > > > 'discard_after_delete': True,
'disk_profiles': [],
> > > > > > > > > 'disk_snapshots':
> > > > > > > > > [], 'disks': [],
'external_status': 'ok', 'master':
> > > > > > > > > False,
> > > > > > > > > 'permissions': [], 'status':
'unattached', 'storage':
> > > > > > > > > {'type': 'fcp',
> > > > > > > > > 'volume_group': {'id':
> > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit',
> > > > > > > > > 'logical_units':
[{'discard_max_size': 268435456,
> > > > > > > > > 'discard_zeroes_data': False,
'id':
> > > > > > > > > '36000d31005b4f6000000000000000029',
'lun_mapping': 3,
> > > > > > > > > 'paths': 0,
> > > > > > > > > 'product_id': 'Compellent
Vol', 'serial':
> > > > > > > > >
'SCOMPELNTCompellent_Vol_0005b4f6-00000029', 'size':
> > > > > > > > > 536870912000,
> > > > > > > > > 'storage_domain_id':
> > > > > > > > >
'dd52022b-7616-47f6-9534-6f1a4084fdf4',
> > > > > > > > > 'vendor_id': 'COMPELNT',
'volume_group_id':
> > > > > > > > >
'HRLDCn-p7X2-5X2O-vm4h-1Wb9-wAMu-WkIwit'}]}},
> > > > > > > > > 'storage_connections':
> > > > > > > > > [], 'storage_format': 'v5',
'supports_discard': True,
> > > > > > > > > 'supports_discard_zeroes_data':
False, 'templates': [],
> > > > > > > > > 'type':
> > > > > > > > > 'data', 'used': 4294967296,
'vms': [],
> > > > > > > > > 'warning_low_space_indicator':
> > > > > > > > > 10, 'wipe_after_delete': False}],
'failed': False}
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 'master': False? I'm not sure if
this creation or check. I
> > > > > > > > > tried this
> > > > > > > > > opration twice. I removed the new
hosted_storage and
> > > > > > > > > remove also old
> > > > > > > > > hosted_engine domain on second try to make
sure that it
> > > > > > > > > doesn't
> > > > > > > > > prevent creation of another master. No luck
with that.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Juhani
> > > > > > > >
> > > > > > > > _______________________________________________
> > > > > > > > Users mailing list -- users(a)ovirt.org
> > > > > > > > To unsubscribe send an email to
users-leave(a)ovirt.org
> > > > > > > > Privacy Statement:
> > > > > > > > https://www.ovirt.org/privacy-policy.html
> > > > > > > > oVirt Code of Conduct:
> > > > > > > >
https://www.ovirt.org/community/about/community-guidelines/
> > > > > > > >
List Archives:
> > > > > > > >
https://lists.ovirt.org/archives/list/users@ovirt.org/messa
> > > > > > > > ge/TZHLFDIYRBQFVDLEVG4SWVZPXSVO7SKK/> >
> > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Didi
> > > > > > >
> > > > > > >
> > >
> > >
> >
> >
> >
> >
> > --
> > Didi
> >
> >
>
>
--
Didi
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/ List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/2R53V3IMD5JKK
B22A3G6OWME3GXDDUFW/
1088
days inactive
1089
days old
14 comments
3 participants
participants (3)
-
Juhani Rautiainen -
Vojtech Juranek -
Yedidyah Bar David