Broker fails to start after upgrade 4.1 to 4.2 metadata_image_UUID can't be ''
Hi list! After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host... cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21 03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-21 03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last) cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21 03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2018-06-21 03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be '' cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID= Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID= This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3... But i can't resolution this bug correctly. Guru please help me!!! Thanx, Alex!
21.06.2018 20:15, reznikov_aa@soskol.com пишет:
Hi list!
After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host...
cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21 03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-21 03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last)
cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21 03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2018-06-21 03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be ''
cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID=
Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID=
This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3...
But i can't resolution this bug correctly.
Guru please help me!!!
Thanx, Alex!
Bump. I tried run workaround script from Simone Tiraboschi, but him not work properly for me. I not see volume ... hosted-engine.lockspace and hosted-engine.metada is null. [root@h4 ~]# ./workaround_1521011.sh + source /etc/ovirt-hosted-engine/hosted-engine.conf ++ fqdn=eng.lan ++ vm_disk_id=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ vm_disk_vol_id=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ vmid=ccdd675a-a58b-495a-9502-3e6a4b7e5228 ++ storage=ssd:/ovirt ++ mnt_options= ++ conf=/var/run/ovirt-hosted-engine-ha/vm.conf ++ host_id=4 ++ console=vnc ++ domainType=nfs3 ++ spUUID=00000000-0000-0000-0000-000000000000 ++ sdUUID=8905c9ac-d892-478d-8346-63b8fa1c5763 ++ connectionUUID=ce84071b-86a2-4e82-b4d9-06abf23dfbc4 ++ ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem ++ ca_subject='C=EN, L=Test, O=Test, CN=Test' ++ vdsm_use_ssl=true ++ gateway=10.245.183.1 ++ bridge=ovirtmgmt ++ lockspace_volume_UUID= ++ lockspace_image_UUID= ++ metadata_volume_UUID= ++ metadata_image_UUID= ++ conf_volume_UUID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ conf_image_UUID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ iqn= ++ portal= ++ user= ++ password= ++ port= ++ vdsm-client StorageDomain getImages storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 ++ grep - ++ tr -d , ++ xargs + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] How to create missing volume ? Is there any solution, or just reinstall HE?
On Mon, Jun 25, 2018 at 10:19 AM Reznikov Alexei <reznikov_aa@soskol.com> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com пишет:
Hi list!
After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host...
cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21
03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
Failed to start necessary monitors MainThread::ERROR::2018-06-21
03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
Traceback (most recent call last)
cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21
03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
Finished loading submonitors MainThread::WARNING::2018-06-21
03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__)
Can't connect vdsm storage: 'metadata_image_UUID can't be ''
cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID=
Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID=
This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3...
But i can't resolution this bug correctly.
Guru please help me!!!
Thanx, Alex!
Bump.
I tried run workaround script from Simone Tiraboschi, but him not work properly for me.
I not see volume ... hosted-engine.lockspace and hosted-engine.metada is null.
[root@h4 ~]# ./workaround_1521011.sh + source /etc/ovirt-hosted-engine/hosted-engine.conf ++ fqdn=eng.lan ++ vm_disk_id=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ vm_disk_vol_id=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ vmid=ccdd675a-a58b-495a-9502-3e6a4b7e5228 ++ storage=ssd:/ovirt ++ mnt_options= ++ conf=/var/run/ovirt-hosted-engine-ha/vm.conf ++ host_id=4 ++ console=vnc ++ domainType=nfs3 ++ spUUID=00000000-0000-0000-0000-000000000000 ++ sdUUID=8905c9ac-d892-478d-8346-63b8fa1c5763 ++ connectionUUID=ce84071b-86a2-4e82-b4d9-06abf23dfbc4 ++ ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem ++ ca_subject='C=EN, L=Test, O=Test, CN=Test' ++ vdsm_use_ssl=true ++ gateway=10.245.183.1 ++ bridge=ovirtmgmt ++ lockspace_volume_UUID= ++ lockspace_image_UUID= ++ metadata_volume_UUID= ++ metadata_image_UUID= ++ conf_volume_UUID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ conf_image_UUID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ iqn= ++ portal= ++ user= ++ password= ++ port= ++ vdsm-client StorageDomain getImages storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 ++ grep - ++ tr -d , ++ xargs + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]]
How to create missing volume ? Is there any solution, or just reinstall HE?
Hi, lockspace and metadata volume are supposed to be already there on the shared storage as result of past upgrades. That script was just designed to print out their uuid so that you can manually add them under /etc/ovirt-hosted-engine/hosted-engine.conf on all the hosts. From what I can see it's not detecting them on your host although I see you have many volumes there. Can you please try manually running vdsm-client Volume getInfo command for all the volumes and manually check volume descriptions?
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3Y7NKSBA3ZNNOT...
25.06.2018 13:05, Simone Tiraboschi пишет:
On Mon, Jun 25, 2018 at 10:19 AM Reznikov Alexei <reznikov_aa@soskol.com <mailto:reznikov_aa@soskol.com>> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com <mailto:reznikov_aa@soskol.com> пишет: > Hi list! > > After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't > start on host... > > cat /var/log/ovirt-hosted-engine-ha/agent.log > MainThread::ERROR::2018-06-21 > 03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
> Failed to start necessary monitors > MainThread::ERROR::2018-06-21 > 03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Traceback (most recent call last) > > cat /var/log/ovirt-hosted-engine-ha/broker.log > MainThread::INFO::2018-06-21 > 03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> Finished loading submonitors > MainThread::WARNING::2018-06-21 > 03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__)
> Can't connect vdsm storage: 'metadata_image_UUID can't be '' > > cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image > metadata_image_UUID= > > Also is: > cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock > lockspace_image_UUID= > lockspace_volume_UUID= > > This bug is very much like this > https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started > with version 3.3... > > But i can't resolution this bug correctly. > > Guru please help me!!! > > Thanx, Alex! > Bump.
I tried run workaround script from Simone Tiraboschi, but him not work properly for me.
I not see volume ... hosted-engine.lockspace and hosted-engine.metada is null.
[root@h4 ~]# ./workaround_1521011.sh + source /etc/ovirt-hosted-engine/hosted-engine.conf ++ fqdn=eng.lan ++ vm_disk_id=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ vm_disk_vol_id=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ vmid=ccdd675a-a58b-495a-9502-3e6a4b7e5228 ++ storage=ssd:/ovirt ++ mnt_options= ++ conf=/var/run/ovirt-hosted-engine-ha/vm.conf ++ host_id=4 ++ console=vnc ++ domainType=nfs3 ++ spUUID=00000000-0000-0000-0000-000000000000 ++ sdUUID=8905c9ac-d892-478d-8346-63b8fa1c5763 ++ connectionUUID=ce84071b-86a2-4e82-b4d9-06abf23dfbc4 ++ ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem ++ ca_subject='C=EN, L=Test, O=Test, CN=Test' ++ vdsm_use_ssl=true ++ gateway=10.245.183.1 ++ bridge=ovirtmgmt ++ lockspace_volume_UUID= ++ lockspace_image_UUID= ++ metadata_volume_UUID= ++ metadata_image_UUID= ++ conf_volume_UUID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ conf_image_UUID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ iqn= ++ portal= ++ user= ++ password= ++ port= ++ vdsm-client StorageDomain getImages storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 ++ grep - ++ tr -d , ++ xargs + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]]
How to create missing volume ? Is there any solution, or just reinstall HE?
Hi, lockspace and metadata volume are supposed to be already there on the shared storage as result of past upgrades. That script was just designed to print out their uuid so that you can manually add them under /etc/ovirt-hosted-engine/hosted-engine.conf on all the hosts.
From what I can see it's not detecting them on your host although I see you have many volumes there. Can you please try manually running vdsm-client Volume getInfo command for all the volumes and manually check volume descriptions?
_______________________________________________ Users mailing list -- users@ovirt.org <mailto:users@ovirt.org> To unsubscribe send an email to users-leave@ovirt.org <mailto:users-leave@ovirt.org> Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3Y7NKSBA3ZNNOT...
Simone thanks for answer! I'm not so good at using vdm-client, but I think this command get see descriptions on my hosted-engine storage ... [root@h4 etc]# sudo -u vdsm grep -rI --exclude-dir="\.lease" "DESCRIPTION" /rhev/data-center/mnt/ssd\:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/* /rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/3abe2f7b-02b9-40a3-8feb-f2809c22c0fb/83186aa6-aaa7-4ce3-9967-7cd3d97d777b.meta:DESCRIPTION={"Updated":true,"Size":20480,"Last Updated":"Thu Jun 21 01:40:54 MSK 2018","Storage Domains":[{"uuid":"8905c9ac-d892-478d-8346-63b8fa1c5763"}],"Disk Description":"OVF_STORE"} /rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/83e0550b-0fc3-40b1-955d-b07ebfbb3994/5a26be32-6c5b-4dcc-ac67-5c442f24df55.meta:DESCRIPTION={"Updated":true,"Size":20480,"Last Updated":"Thu Jun 21 01:40:54 MSK 2018","Storage Domains":[{"uuid":"8905c9ac-d892-478d-8346-63b8fa1c5763"}],"Disk Description":"OVF_STORE"} /rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/b5f353f5-9357-4aad-b1a3-751d411e6278/a20d9700-1b9a-41d8-bb4b-f2b7c168104f.meta:DESCRIPTION=HostedEngineConfigurationImage /rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/e9d7a377-e109-4b28-9a43-7a8c8b603749/cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b.meta:DESCRIPTION=Hosted Engine Image Simon, as seen in the descriptions there is no "hosted-engine.lockspace" and "hosted-engine.metadata". How do fix this ? -- С уважением, Резников Алексей инженер-программист ООО "ГорПТУс", группа компаний Осколтелеком web: fly.soskol.ru email: reznikov_aa@soskol.com тел. +7(4725)40-16-77
Hi, yes there is a solution described directly in the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1521011#c20 The provided script worked only for cases that had the necessary disks, but where the uuids were not written to the config files. You need to follow the procedure from comment 20 when no disks for lockspace and metadata exist at all. Best regards Martin Sivak On Mon, Jun 25, 2018 at 9:52 AM, Reznikov Alexei <reznikov_aa@soskol.com> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com пишет:
Hi list!
After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host...
cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21 03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-21 03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last)
cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21 03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2018-06-21 03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be ''
cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID=
Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID=
This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3...
But i can't resolution this bug correctly.
Guru please help me!!!
Thanx, Alex!
Bump.
I tried run workaround script from Simone Tiraboschi, but him not work properly for me.
I not see volume ... hosted-engine.lockspace and hosted-engine.metada is null.
[root@h4 ~]# ./workaround_1521011.sh + source /etc/ovirt-hosted-engine/hosted-engine.conf ++ fqdn=eng.lan ++ vm_disk_id=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ vm_disk_vol_id=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ vmid=ccdd675a-a58b-495a-9502-3e6a4b7e5228 ++ storage=ssd:/ovirt ++ mnt_options= ++ conf=/var/run/ovirt-hosted-engine-ha/vm.conf ++ host_id=4 ++ console=vnc ++ domainType=nfs3 ++ spUUID=00000000-0000-0000-0000-000000000000 ++ sdUUID=8905c9ac-d892-478d-8346-63b8fa1c5763 ++ connectionUUID=ce84071b-86a2-4e82-b4d9-06abf23dfbc4 ++ ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem ++ ca_subject='C=EN, L=Test, O=Test, CN=Test' ++ vdsm_use_ssl=true ++ gateway=10.245.183.1 ++ bridge=ovirtmgmt ++ lockspace_volume_UUID= ++ lockspace_image_UUID= ++ metadata_volume_UUID= ++ metadata_image_UUID= ++ conf_volume_UUID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ conf_image_UUID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ iqn= ++ portal= ++ user= ++ password= ++ port= ++ vdsm-client StorageDomain getImages storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 ++ grep - ++ tr -d , ++ xargs + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994 volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55 ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=b5f353f5-9357-4aad-b1a3-751d411e6278 volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]] + for i in '$(vdsm-client StorageDomain getImages storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client StorageDomain getVolumes storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 ++ grep - ++ tr -d , ++ xargs + for v in '$(vdsm-client StorageDomain getVolumes storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} | grep - | tr -d '\'','\'' | xargs)' ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid' ++ xargs + lockspace_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.lockspace") | .image' ++ xargs + lockspace_img_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid' ++ xargs + metadata_vol_uuid= + [[ ! -z '' ]] ++ vdsm-client Volume getInfo storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763 storagepoolID=00000000-0000-0000-0000-000000000000 imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749 volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b ++ jq '. | select(.description=="hosted-engine.metadata") | .image' ++ xargs + metadata_img_uuid= + [[ ! -z '' ]]
How to create missing volume ? Is there any solution, or just reinstall HE?
_______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/3Y7NKSBA3ZNNOT...
25.06.2018 15:12, Martin Sivak пишет:
Hi,
yes there is a solution described directly in the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1521011#c20
The provided script worked only for cases that had the necessary disks, but where the uuids were not written to the config files.
You need to follow the procedure from comment 20 when no disks for lockspace and metadata exist at all.
Best regards
Martin Sivak
On Mon, Jun 25, 2018 at 9:52 AM, Reznikov Alexei <reznikov_aa@soskol.com> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com пишет:
Hi list!
After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host...
cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21 03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-21 03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last)
cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21 03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2018-06-21 03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be ''
cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID=
Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID=
This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3...
But i can't resolution this bug correctly.
Guru please help me!!!
Thanx, Alex!
Thanks for answer Martin.
I get the problem in step 5, the procedure described in comment 20 [root @ h4 /] # sanlock direct init -s hosted-engine: 0: /rhev/data-center/mnt/ssd.lan \: _ ovirt / 8905c9ac-d892-478d-8346-63b8fa1c5763 / images / badd5883-ef71- 45bb-9073-a573f46a3b44 / e4408917-fe00-4567-8db0-bf464472ec01.lockspace init done -19 What does mean "init done -19", why do not I see any changes?
[root @ h4 /] # sanlock direct init -s hosted-engine: 0:
/rhev/data-center/mnt/ssd.lan \: _ ovirt / 8905c9ac-d892-478d-8346-63b8fa1c5763 / images / badd5883-ef71- 45bb-9073-a573f46a3b44 / e4408917-fe00-4567-8db0-bf464472ec01.lockspace init done -19
Do you really have all those spaces in the path? Also make sure the backslash in \:ovirt is doubled if you execute this from bash like you seem to be doing (\\:ovirt) Martin On Mon, Jun 25, 2018 at 3:38 PM, Reznikov Alexei <reznikov_aa@soskol.com> wrote:
25.06.2018 15:12, Martin Sivak пишет:
Hi,
yes there is a solution described directly in the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1521011#c20
The provided script worked only for cases that had the necessary disks, but where the uuids were not written to the config files.
You need to follow the procedure from comment 20 when no disks for lockspace and metadata exist at all.
Best regards
Martin Sivak
On Mon, Jun 25, 2018 at 9:52 AM, Reznikov Alexei <reznikov_aa@soskol.com> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com пишет:
Hi list!
After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker can't start on host...
cat /var/log/ovirt-hosted-engine-ha/agent.log MainThread::ERROR::2018-06-21
03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker) Failed to start necessary monitors MainThread::ERROR::2018-06-21
03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent) Traceback (most recent call last)
cat /var/log/ovirt-hosted-engine-ha/broker.log MainThread::INFO::2018-06-21
03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors) Finished loading submonitors MainThread::WARNING::2018-06-21
03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__) Can't connect vdsm storage: 'metadata_image_UUID can't be ''
cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep metadata_image metadata_image_UUID=
Also is: cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock lockspace_image_UUID= lockspace_volume_UUID=
This bug is very much like this https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster started with version 3.3...
But i can't resolution this bug correctly.
Guru please help me!!!
Thanx, Alex!
Thanks for answer Martin.
I get the problem in step 5, the procedure described in comment 20 [root @ h4 /] # sanlock direct init -s hosted-engine: 0: /rhev/data-center/mnt/ssd.lan \: _ ovirt / 8905c9ac-d892-478d-8346-63b8fa1c5763 / images / badd5883-ef71- 45bb-9073-a573f46a3b44 / e4408917-fe00-4567-8db0-bf464472ec01.lockspace init done -19
What does mean "init done -19", why do not I see any changes? _______________________________________________ Users mailing list -- users@ovirt.org To unsubscribe send an email to users-leave@ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/4KPMQ5KY7V2KK6...
26.06.2018 13:01, Martin Sivak пишет:
[root @ h4 /] # sanlock direct init -s hosted-engine: 0:
/rhev/data-center/mnt/ssd.lan \: _ ovirt / 8905c9ac-d892-478d-8346-63b8fa1c5763 / images / badd5883-ef71- 45bb-9073-a573f46a3b44 / e4408917-fe00-4567-8db0-bf464472ec01.lockspace init done -19 Do you really have all those spaces in the path? Also make sure the backslash in \:ovirt is doubled if you execute this from bash like you seem to be doing (\\:ovirt)
Martin
Martin, thank you very much! I successfully solved my problem! Alex
participants (4)
-
Martin Sivak -
Reznikov Alexei -
reznikov_aa＠soskol.com
-
Simone Tiraboschi