HostedEngine Deployment fails activating NFS Storage Domain hosted_storage via GUI Step 4
Hello, I'm deploying HostedEngine to a NFS Storage. HostedEngineLocal ist setup and running already. But Step 4 (Moving to hosted_storage Domain on NFS) fails. The Host ist Node-NG 4.3.3.1 based. The intended NFS Domain gets mounted in the host but activation (I think via EngineAPI fails): [ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."} mount in host shows: storage.rxmgmt.databay.de:/ovirt/hosted_storage on /rhev/data-center/mnt/storage.rxmgmt.databay.de:_ovirt_hosted__storage type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=172.16.252.231,local_lock=none,addr=172.16.252.3) I also sshd into the locally running engine vi 192.168.122.XX and the VM can mount the storage domain, too: [root@engine01 ~]# mount storage.rxmgmt.databay.de:/ovirt/hosted_storage /mnt/ -o vers=4.1 [root@engine01 ~]# mount | grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) storage.rxmgmt.databay.de:/ovirt/hosted_storage on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.71,local_lock=none,addr=172.16.252.3) [root@engine01 ~]# ls -al /mnt/ total 18 drwxrwxr-x. 3 vdsm kvm 4 Apr 30 12:59 . dr-xr-xr-x. 17 root root 224 Apr 16 14:31 .. drwxr-xr-x. 4 vdsm kvm 4 Apr 30 12:40 4dc42146-b3fb-47ec-bf06-8d9bf7cdf893 -rwxr-xr-x. 1 vdsm kvm 0 Apr 30 12:55 __DIRECT_IO_TEST__ Anything I can do ? Log-Extract of ovirt-hosted-engine-setup-ansible-create_storage_domain included. -- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------
On Tue, Apr 30, 2019 at 1:35 PM Ralf Schenk <rs@databay.de> wrote:
Hello,
I'm deploying HostedEngine to a NFS Storage. HostedEngineLocal ist setup and running already. But Step 4 (Moving to hosted_storage Domain on NFS) fails. The Host ist Node-NG 4.3.3.1 based.
The intended NFS Domain gets mounted in the host but activation (I think via EngineAPI fails):
[ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
mount in host shows:
storage.rxmgmt.databay.de:/ovirt/hosted_storage on /rhev/data-center/mnt/storage.rxmgmt.databay.de:_ovirt_hosted__storage type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=172.16.252.231,local_lock=none,addr=172.16.252.3)
I also sshd into the locally running engine vi 192.168.122.XX and the VM can mount the storage domain, too:
[root@engine01 ~]# mount storage.rxmgmt.databay.de:/ovirt/hosted_storage /mnt/ -o vers=4.1 [root@engine01 ~]# mount | grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) storage.rxmgmt.databay.de:/ovirt/hosted_storage on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.71,local_lock=none,addr=172.16.252.3) [root@engine01 ~]# ls -al /mnt/ total 18 drwxrwxr-x. 3 vdsm kvm 4 Apr 30 12:59 . dr-xr-xr-x. 17 root root 224 Apr 16 14:31 .. drwxr-xr-x. 4 vdsm kvm 4 Apr 30 12:40 4dc42146-b3fb-47ec-bf06-8d9bf7cdf893 -rwxr-xr-x. 1 vdsm kvm 0 Apr 30 12:55 __DIRECT_IO_TEST__
Anything I can do ?
99% that folder was dirty (it already contained something) when you started the deployment. I can only suggest to clean that folder and start from scratch.
Log-Extract of ovirt-hosted-engine-setup-ansible-create_storage_domain included.
--
*Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <rs@databay.de>
*Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------ _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37WY4NUSJYMA7P...
Hello, that is definitely not my problem. Did a complete new deployment (after rebooting host) Before deploying on my storage: root@storage-rx:/srv/nfs/ovirt/hosted_storage# ls -al total 17 drwxrwxr-x 2 vdsm vdsm 2 Apr 30 13:53 . drwxr-xr-x 8 root root 8 Apr 2 18:02 .. While deploying in late stage: root@storage-rx:/srv/nfs/ovirt/hosted_storage# ls -al total 18 drwxrwxr-x 3 vdsm vdsm 4 Apr 30 14:51 . drwxr-xr-x 8 root root 8 Apr 2 18:02 .. drwxr-xr-x 4 vdsm vdsm 4 Apr 30 14:51 d26e4a31-8d73-449d-bebc-f2ce7a979e5d -rwxr-xr-x 1 vdsm vdsm 0 Apr 30 14:51 __DIRECT_IO_TEST__ Immediately the error occurs in GUI: [ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."} Am 30.04.2019 um 13:48 schrieb Simone Tiraboschi:
On Tue, Apr 30, 2019 at 1:35 PM Ralf Schenk <rs@databay.de <mailto:rs@databay.de>> wrote:
Hello,
I'm deploying HostedEngine to a NFS Storage. HostedEngineLocal ist setup and running already. But Step 4 (Moving to hosted_storage Domain on NFS) fails. The Host ist Node-NG 4.3.3.1 based.
The intended NFS Domain gets mounted in the host but activation (I think via EngineAPI fails):
[ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
mount in host shows:
storage.rxmgmt.databay.de:/ovirt/hosted_storage on /rhev/data-center/mnt/storage.rxmgmt.databay.de:_ovirt_hosted__storage type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=172.16.252.231,local_lock=none,addr=172.16.252.3)
I also sshd into the locally running engine vi 192.168.122.XX and the VM can mount the storage domain, too:
[root@engine01 ~]# mount storage.rxmgmt.databay.de:/ovirt/hosted_storage /mnt/ -o vers=4.1 [root@engine01 ~]# mount | grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) storage.rxmgmt.databay.de:/ovirt/hosted_storage on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.71,local_lock=none,addr=172.16.252.3) [root@engine01 ~]# ls -al /mnt/ total 18 drwxrwxr-x. 3 vdsm kvm 4 Apr 30 12:59 . dr-xr-xr-x. 17 root root 224 Apr 16 14:31 .. drwxr-xr-x. 4 vdsm kvm 4 Apr 30 12:40 4dc42146-b3fb-47ec-bf06-8d9bf7cdf893 -rwxr-xr-x. 1 vdsm kvm 0 Apr 30 12:55 __DIRECT_IO_TEST__
Anything I can do ?
99% that folder was dirty (it already contained something) when you started the deployment. I can only suggest to clean that folder and start from scratch.
Log-Extract of ovirt-hosted-engine-setup-ansible-create_storage_domain included.
--
*Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------ _______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37WY4NUSJYMA7P...
-- *Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <mailto:rs@databay.de> *Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de> Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------------------------------------------------
On Tue, Apr 30, 2019 at 2:55 PM Ralf Schenk <rs@databay.de> wrote:
Hello,
that is definitely not my problem. Did a complete new deployment (after rebooting host)
Before deploying on my storage: root@storage-rx:/srv/nfs/ovirt/hosted_storage# ls -al total 17 drwxrwxr-x 2 vdsm vdsm 2 Apr 30 13:53 . drwxr-xr-x 8 root root 8 Apr 2 18:02 ..
While deploying in late stage: root@storage-rx:/srv/nfs/ovirt/hosted_storage# ls -al total 18 drwxrwxr-x 3 vdsm vdsm 4 Apr 30 14:51 . drwxr-xr-x 8 root root 8 Apr 2 18:02 .. drwxr-xr-x 4 vdsm vdsm 4 Apr 30 14:51 d26e4a31-8d73-449d-bebc-f2ce7a979e5d -rwxr-xr-x 1 vdsm vdsm 0 Apr 30 14:51 __DIRECT_IO_TEST__
Immediately the error occurs in GUI: [ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
Can you please share engine.log and vdsm.log?
Am 30.04.2019 um 13:48 schrieb Simone Tiraboschi:
On Tue, Apr 30, 2019 at 1:35 PM Ralf Schenk <rs@databay.de> wrote:
Hello,
I'm deploying HostedEngine to a NFS Storage. HostedEngineLocal ist setup and running already. But Step 4 (Moving to hosted_storage Domain on NFS) fails. The Host ist Node-NG 4.3.3.1 based.
The intended NFS Domain gets mounted in the host but activation (I think via EngineAPI fails):
[ INFO ] TASK [ovirt.hosted_engine_setup : Activate storage domain] [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is "[]". HTTP response code is 400. [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400."}
mount in host shows:
storage.rxmgmt.databay.de:/ovirt/hosted_storage on /rhev/data-center/mnt/storage.rxmgmt.databay.de:_ovirt_hosted__storage type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,soft,nosharecache,proto=tcp,timeo=600,retrans=6,sec=sys,clientaddr=172.16.252.231,local_lock=none,addr=172.16.252.3)
I also sshd into the locally running engine vi 192.168.122.XX and the VM can mount the storage domain, too:
[root@engine01 ~]# mount storage.rxmgmt.databay.de:/ovirt/hosted_storage /mnt/ -o vers=4.1 [root@engine01 ~]# mount | grep nfs sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) storage.rxmgmt.databay.de:/ovirt/hosted_storage on /mnt type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.71,local_lock=none,addr=172.16.252.3) [root@engine01 ~]# ls -al /mnt/ total 18 drwxrwxr-x. 3 vdsm kvm 4 Apr 30 12:59 . dr-xr-xr-x. 17 root root 224 Apr 16 14:31 .. drwxr-xr-x. 4 vdsm kvm 4 Apr 30 12:40 4dc42146-b3fb-47ec-bf06-8d9bf7cdf893 -rwxr-xr-x. 1 vdsm kvm 0 Apr 30 12:55 __DIRECT_IO_TEST__
Anything I can do ?
99% that folder was dirty (it already contained something) when you started the deployment. I can only suggest to clean that folder and start from scratch.
Log-Extract of ovirt-hosted-engine-setup-ansible-create_storage_domain included.
--
*Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <rs@databay.de>
*Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------ _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/37WY4NUSJYMA7P...
--
*Ralf Schenk* fon +49 (0) 24 05 / 40 83 70 fax +49 (0) 24 05 / 40 83 759 mail *rs@databay.de* <rs@databay.de>
*Databay AG* Jens-Otto-Krag-Straße 11 D-52146 Würselen *www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202 Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm. Philipp Hermanns Aufsichtsratsvorsitzender: Wilhelm Dohmen ------------------------------
I am having a similar problem - upgrade from 4.2.7 to 4.3.3 ... My Data Center would not activate, and I was getting all sorts of errors on the UI. I ended up shutting down the existing engine VM using --shutdown-vm ... trying to restart it, the console would report that the volume could not be found and would not start. ugh. so, ssh into another host of the cluster, same thing. grr. --deploy goes through most of the settings up until what I am assuming is probably 4 on the UI ... errors at activating storage domain. I created a separate nfs share so that I can hopefully import my machines that are still limping along, since I havent rebooted the fileserver 2019-05-17 09:03:59,698-0400 DEBUG var changed: host "localhost" var "otopi_storage_domain_details" type "<type 'dict'>" value: "{ "changed": false, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_storage_domain_payload_4b8QQJ/__main__.py\", line 664, in main\n storage_domains_module.post_create_check(sd_id)\n File \"/tmp/ansible_ovirt_storage_domain_payload_4b8QQJ/__main__.py\", line 526, in post_create_check\n id=storage_domain.id,\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 3053, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n return future.wait() if wait else future\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n return self._code(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n self._check_fault(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n self._raise_error(response , body)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400.\n", "failed": true, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400." }" 400 - Bad request?
On Fri, May 17, 2019 at 4:10 PM <dan.midthun@gmail.com> wrote:
I am having a similar problem - upgrade from 4.2.7 to 4.3.3 ... My Data Center would not activate, and I was getting all sorts of errors on the UI. I ended up shutting down the existing engine VM using --shutdown-vm ... trying to restart it, the console would report that the volume could not be found and would not start. ugh.
so, ssh into another host of the cluster, same thing. grr. --deploy goes through most of the settings up until what I am assuming is probably 4 on the UI ... errors at activating storage domain. I created a separate nfs share so that I can hopefully import my machines that are still limping along, since I havent rebooted the fileserver
2019-05-17 09:03:59,698-0400 DEBUG var changed: host "localhost" var "otopi_storage_domain_details" type "<type 'dict'>" value: "{ "changed": false, "exception": "Traceback (most recent call last):\n File \"/tmp/ansible_ovirt_storage_domain_payload_4b8QQJ/__main__.py\", line 664, in main\n storage_domains_module.post_create_check(sd_id)\n File \"/tmp/ansible_ovirt_storage_domain_payload_4b8QQJ/__main__.py\", line 526, in post_create_check\n id=storage_domain.id,\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/services.py\", line 3053, in add\n return self._internal_add(storage_domain, headers, query, wait)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 232, in _internal_add\n return future.wait() if wait else future\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 55, in wait\n return self._code(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 229, in callback\n self._check_fault(response)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 132, in _check_fault\n self._raise_error(response , body)\n File \"/usr/lib64/python2.7/site-packages/ovirtsdk4/service.py\", line 118, in _raise_error\n raise error\nError: Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400.\n", "failed": true, "msg": "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP response code is 400." }"
400 - Bad request?
I's suggest to check engine.log on the bootstrap engine VM; unfortunately engine error responses are not always that explicit.
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/2PW3IWTGGU6J33...
-- Simone Tiraboschi He / Him / His Principal Software Engineer Red Hat <https://www.redhat.com/> stirabos@redhat.com @redhatjobs <https://twitter.com/redhatjobs> redhatjobs <https://www.facebook.com/redhatjobs> @redhatjobs <https://instagram.com/redhatjobs> <https://red.ht/sig> <https://redhat.com/summit>
Simone, What would be causing the Cannot aquire host ID?? 2019-05-17 09:03:57,638-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9126b578-9eac-4517-9e56-960c4751c3d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:57,653-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9126b578-9eac-4517-9e56-960c4751c3d1] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: e85f74bd-5e43-4a8c-8158-eb4696e041bc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: adf59b7a-78a1-11e9-82af-00163e729513 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2019-05-17 09:03:57,678-04 INFO [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [66bed180] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: adf59b7a-78a1-11e9-82af-00163e729513 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN 2019-05-17 09:03:57,725-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-1) [2091c1ca] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2019-05-17 09:03:57,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [2091c1ca] START, ConnectStorageServerVDSCommand(HostName = host-93.home.local, StorageServerConnectionManagementVDSParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='7e8e82be-f033-487f-892c-16f9f12c31b7', connection='10.10 .32.211:/storage/Dixon', iqn='null', vfsType='null', mountOptions='', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4cbacffc 2019-05-17 09:03:57,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [2091c1ca] FINISH, ConnectStorageServerVDSCommand, return: {7e8e82be-f033-487f-892c-16f9f12c31b7=0}, log id: 4cbacffc 2019-05-17 09:03:57,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [2091c1ca] START, HSMGetStorageDomainInfoVDSCommand(HostName = host-93.home.local, HSMGetStorageDomainInfoVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storageDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}), log id: 5f095b14 2019-05-17 09:03:58,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [2091c1ca] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageDomainStatic:{name='hosted_storage', id='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}, null>, log id: 5f095b14 2019-05-17 09:03:58,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] START, CreateStoragePoolVDSCommand(HostName = host-93.home.local, CreateStoragePoolVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storagePoolName='Default', masterDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc', domainsIdList='[e85f74bd-5e43-4a8c-8158-eb4696e041bc]', masterVersion='2'}), log id: 58aabe3d 2019-05-17 09:03:59,128-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Failed in 'CreateStoragePoolVDS' method 2019-05-17 09:03:59,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host-93.home.local command CreateStoragePoolVDS failed: Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=661, message=Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))]]' 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] HostName = host-93.home.local 2019-05-17 09:03:59,134-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Command 'CreateStoragePoolVDSCommand(HostName = host-93.home.local, CreateStoragePoolVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storagePoolName='Default', masterDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc', domainsIdList='[e85f74bd-5e43-4a8c-8158-eb4696e041bc] ', masterVersion='2'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')), code = 661 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] FINISH, CreateStoragePoolVDSCommand, return: , log id: 58aabe3d 2019-05-17 09:03:59,135-04 ERROR [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [2091c1ca] Command 'org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'San lock lockspace add failure', 'No such device')), code = 661 (Failed with error AcquireHostIdFailure and code 661) 2019-05-17 09:03:59,137-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: id=adf59b7a-78a1-11e9-82af-00163e729513. 2019-05-17 09:03:59,139-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storageId='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}. 2019-05-17 09:03:59,142-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=e85f74bd-5e43-4a8c-8158-eb4696e041bc. 2019-05-17 09:03:59,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: USER_ATTACH_STORAGE_DOMAINS_TO_POOL_FAILED(1,003), Failed to attach Storage Domains to Data Center Default. (User: admin@internal-authz) 2019-05-17 09:03:59,179-04 INFO [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [2091c1ca] Lock freed to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:59,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain hosted_storage to Data Center Default. (User: admin@internal-authz) 2019-05-17 09:03:59,198-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-1) [2091c1ca] Trying to release exclusive lock which does not exist, lock key: 'e85f74bd-5e43-4a8c-8158-eb4696e041bcSTORAGE' 2019-05-17 09:03:59,198-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [2091c1ca] Lock freed to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:59,199-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: []
On Fri, May 17, 2019 at 4:54 PM <dan.midthun@gmail.com> wrote:
Simone, What would be causing the Cannot aquire host ID??
Was that NFS folder absolutely empty?
2019-05-17 09:03:57,638-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9126b578-9eac-4517-9e56-960c4751c3d1] Lock Acquired to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:57,653-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [9126b578-9eac-4517-9e56-960c4751c3d1] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: e85f74bd-5e43-4a8c-8158-eb4696e041bc Type: StorageAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN, ID: adf59b7a-78a1-11e9-82af-00163e729513 Type: StoragePoolAction group MANIPULATE_STORAGE_DOMAIN with role type ADMIN 2019-05-17 09:03:57,678-04 INFO [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [66bed180] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: adf59b7a-78a1-11e9-82af-00163e729513 Type: StoragePoolAction group CREATE_STORAGE_POOL with role type ADMIN 2019-05-17 09:03:57,725-04 INFO [org.ovirt.engine.core.bll.storage.connection.ConnectStorageToVdsCommand] (default task-1) [2091c1ca] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: SystemAction group CREATE_STORAGE_DOMAIN with role type ADMIN 2019-05-17 09:03:57,732-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [2091c1ca] START, ConnectStorageServerVDSCommand(HostName = host-93.home.local, StorageServerConnectionManagementVDSParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='00000000-0000-0000-0000-000000000000', storageType='NFS', connectionList='[StorageServerConnections:{id='7e8e82be-f033-487f-892c-16f9f12c31b7', connection='10.10 .32.211:/storage/Dixon', iqn='null', vfsType='null', mountOptions='', nfsVersion='AUTO', nfsRetrans='null', nfsTimeo='null', iface='null', netIfaceName='null'}]', sendNetworkEventOnFailure='true'}), log id: 4cbacffc 2019-05-17 09:03:57,756-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (default task-1) [2091c1ca] FINISH, ConnectStorageServerVDSCommand, return: {7e8e82be-f033-487f-892c-16f9f12c31b7=0}, log id: 4cbacffc 2019-05-17 09:03:57,768-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [2091c1ca] START, HSMGetStorageDomainInfoVDSCommand(HostName = host-93.home.local, HSMGetStorageDomainInfoVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storageDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}), log id: 5f095b14 2019-05-17 09:03:58,110-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (default task-1) [2091c1ca] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageDomainStatic:{name='hosted_storage', id='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}, null>, log id: 5f095b14 2019-05-17 09:03:58,113-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] START, CreateStoragePoolVDSCommand(HostName = host-93.home.local, CreateStoragePoolVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storagePoolName='Default', masterDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc', domainsIdList='[e85f74bd-5e43-4a8c-8158-eb4696e041bc]', masterVersion='2'}), log id: 58aabe3d 2019-05-17 09:03:59,128-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Failed in 'CreateStoragePoolVDS' method 2019-05-17 09:03:59,134-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), VDSM host-93.home.local command CreateStoragePoolVDS failed: Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')) 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Command 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand' return value 'StatusOnlyReturn [status=Status [code=661, message=Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device'))]]' 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] HostName = host-93.home.local 2019-05-17 09:03:59,134-04 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] Command 'CreateStoragePoolVDSCommand(HostName = host-93.home.local, CreateStoragePoolVDSCommandParameters:{hostId='7f7408f3-5558-4f9f-81f8-fa5c3f10c3f9', storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storagePoolName='Default', masterDomainId='e85f74bd-5e43-4a8c-8158-eb4696e041bc', domainsIdList='[e85f74bd-5e43-4a8c-8158-eb4696e041bc] ', masterVersion='2'})' execution failed: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'Sanlock lockspace add failure', 'No such device')), code = 661 2019-05-17 09:03:59,134-04 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (default task-1) [2091c1ca] FINISH, CreateStoragePoolVDSCommand, return: , log id: 58aabe3d 2019-05-17 09:03:59,135-04 ERROR [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [2091c1ca] Command 'org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand' failed: EngineException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire host id: (u'e85f74bd-5e43-4a8c-8158-eb4696e041bc', SanlockException(19, 'San lock lockspace add failure', 'No such device')), code = 661 (Failed with error AcquireHostIdFailure and code 661) 2019-05-17 09:03:59,137-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: id=adf59b7a-78a1-11e9-82af-00163e729513. 2019-05-17 09:03:59,139-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating NEW_ENTITY_ID of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: StoragePoolIsoMapId:{storagePoolId='adf59b7a-78a1-11e9-82af-00163e729513', storageId='e85f74bd-5e43-4a8c-8158-eb4696e041bc'}. 2019-05-17 09:03:59,142-04 INFO [org.ovirt.engine.core.bll.CommandCompensator] (default task-1) [2091c1ca] Command [id=fc08e1d1-9383-4353-b65b-b50121596fba]: Compensating DELETED_OR_UPDATED_ENTITY of org.ovirt.engine.core.common.businessentities.StorageDomainStatic; snapshot: id=e85f74bd-5e43-4a8c-8158-eb4696e041bc. 2019-05-17 09:03:59,173-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: USER_ATTACH_STORAGE_DOMAINS_TO_POOL_FAILED(1,003), Failed to attach Storage Domains to Data Center Default. (User: admin@internal-authz) 2019-05-17 09:03:59,179-04 INFO [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] (default task-1) [2091c1ca] Lock freed to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:59,192-04 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-1) [2091c1ca] EVENT_ID: USER_ATTACH_STORAGE_DOMAIN_TO_POOL_FAILED(963), Failed to attach Storage Domain hosted_storage to Data Center Default. (User: admin@internal-authz) 2019-05-17 09:03:59,198-04 WARN [org.ovirt.engine.core.bll.lock.InMemoryLockManager] (default task-1) [2091c1ca] Trying to release exclusive lock which does not exist, lock key: 'e85f74bd-5e43-4a8c-8158-eb4696e041bcSTORAGE' 2019-05-17 09:03:59,198-04 INFO [org.ovirt.engine.core.bll.storage.domain.AttachStorageDomainToPoolCommand] (default task-1) [2091c1ca] Lock freed to object 'EngineLock:{exclusiveLocks='[e85f74bd-5e43-4a8c-8158-eb4696e041bc=STORAGE]', sharedLocks=''}' 2019-05-17 09:03:59,199-04 ERROR [org.ovirt.engine.api.restapi.resource.AbstractBackendResource] (default task-1) [] Operation Failed: [] _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/F2WETYJGJHEUWU...
-- Simone Tiraboschi He / Him / His Principal Software Engineer Red Hat <https://www.redhat.com/> stirabos@redhat.com @redhatjobs <https://twitter.com/redhatjobs> redhatjobs <https://www.facebook.com/redhatjobs> @redhatjobs <https://instagram.com/redhatjobs> <https://red.ht/sig> <https://redhat.com/summit>
100% clean
I have a recording of the entire process, though its ~25 mins long that I can post. Also after the failure I was able to look at the engine log and have a recording of that as well.
participants (3)
-
dan.midthun＠gmail.com -
Ralf Schenk -
Simone Tiraboschi