[ovirt-users] Re: Broker fails to start after upgrade 4.1 to 4.2 metadata_image_UUID can't be ''

25 Jun 2018

      25.06.2018 13:05, Simone Tiraboschi пишет:
...
On Mon, Jun 25, 2018 at 10:19 AM Reznikov Alexei 
<reznikov_aa@soskol.com <mailto:reznikov_aa@soskol.com>> wrote:
21.06.2018 20:15, reznikov_aa@soskol.com
    <mailto:reznikov_aa@soskol.com> пишет:
    > Hi list!
    >
    > After upgrade my cluster from 4.1.9 to 4.2.2, agent and broker
    can't
    > start on host...
    >
    > cat /var/log/ovirt-hosted-engine-ha/agent.log
    > MainThread::ERROR::2018-06-21
    >
    03:25:34,603::hosted_engine::538::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_broker)
> Failed to start necessary monitors
    > MainThread::ERROR::2018-06-21
    >
    03:25:34,604::agent::144::ovirt_hosted_engine_ha.agent.agent.Agent::(_run_agent)
> Traceback (most recent call last)
    >
    > cat /var/log/ovirt-hosted-engine-ha/broker.log
    > MainThread::INFO::2018-06-21
    >
    03:25:40,406::monitor::50::ovirt_hosted_engine_ha.broker.monitor.Monitor::(_discover_submonitors)
> Finished loading submonitors
    > MainThread::WARNING::2018-06-21
    >
    03:25:40,406::storage_broker::97::ovirt_hosted_engine_ha.broker.storage_broker.StorageBroker::(__init__)
> Can't connect vdsm storage: 'metadata_image_UUID can't be ''
    >
    > cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep
    metadata_image
    > metadata_image_UUID=
    >
    > Also is:
    > cat /etc/ovirt-hosted-engine/hosted-engine.conf | grep lock
    > lockspace_image_UUID=
    > lockspace_volume_UUID=
    >
    > This bug is very much like this
    > https://bugzilla.redhat.com/show_bug.cgi?id=1521011 My cluster
    started
    > with version 3.3...
    >
    > But i can't resolution this bug correctly.
    >
    > Guru please help me!!!
    >
    > Thanx, Alex!
    >
    Bump.
I tried run workaround script from Simone Tiraboschi, but him not
    work
    properly for me.
I not see volume ... hosted-engine.lockspace and
    hosted-engine.metada is
    null.
[root@h4 ~]# ./workaround_1521011.sh
    + source /etc/ovirt-hosted-engine/hosted-engine.conf
    ++ fqdn=eng.lan
    ++ vm_disk_id=e9d7a377-e109-4b28-9a43-7a8c8b603749
    ++ vm_disk_vol_id=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b
    ++ vmid=ccdd675a-a58b-495a-9502-3e6a4b7e5228
    ++ storage=ssd:/ovirt
    ++ mnt_options=
    ++ conf=/var/run/ovirt-hosted-engine-ha/vm.conf
    ++ host_id=4
    ++ console=vnc
    ++ domainType=nfs3
    ++ spUUID=00000000-0000-0000-0000-000000000000
    ++ sdUUID=8905c9ac-d892-478d-8346-63b8fa1c5763
    ++ connectionUUID=ce84071b-86a2-4e82-b4d9-06abf23dfbc4
    ++ ca_cert=/etc/pki/vdsm/libvirt-spice/ca-cert.pem
    ++ ca_subject='C=EN, L=Test, O=Test, CN=Test'
    ++ vdsm_use_ssl=true
    ++ gateway=10.245.183.1
    ++ bridge=ovirtmgmt
    ++ lockspace_volume_UUID=
    ++ lockspace_image_UUID=
    ++ metadata_volume_UUID=
    ++ metadata_image_UUID=
    ++ conf_volume_UUID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f
    ++ conf_image_UUID=b5f353f5-9357-4aad-b1a3-751d411e6278
    ++ iqn=
    ++ portal=
    ++ user=
    ++ password=
    ++ port=
    ++ vdsm-client StorageDomain getImages
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    ++ grep -
    ++ tr -d ,
    ++ xargs
    + for i in '$(vdsm-client StorageDomain getImages
    storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d
    '\'','\'' | xargs)'
    ++ vdsm-client StorageDomain getVolumes
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994
    ++ grep -
    ++ tr -d ,
    ++ xargs
    + for v in '$(vdsm-client StorageDomain getVolumes
    storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} |
    grep -
    | tr -d '\'','\'' | xargs)'
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994
    volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid'
    ++ xargs
    + lockspace_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994
    volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .image'
    ++ xargs
    + lockspace_img_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994
    volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55
    ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid'
    ++ xargs
    + metadata_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=83e0550b-0fc3-40b1-955d-b07ebfbb3994
    volumeID=5a26be32-6c5b-4dcc-ac67-5c442f24df55
    ++ jq '. | select(.description=="hosted-engine.metadata") | .image'
    ++ xargs
    + metadata_img_uuid=
    + [[ ! -z '' ]]
    + for i in '$(vdsm-client StorageDomain getImages
    storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d
    '\'','\'' | xargs)'
    ++ vdsm-client StorageDomain getVolumes
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb
    ++ grep -
    ++ tr -d ,
    ++ xargs
    + for v in '$(vdsm-client StorageDomain getVolumes
    storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} |
    grep -
    | tr -d '\'','\'' | xargs)'
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb
    volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid'
    ++ xargs
    + lockspace_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb
    volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .image'
    ++ xargs
    + lockspace_img_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb
    volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b
    ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid'
    ++ xargs
    + metadata_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=3abe2f7b-02b9-40a3-8feb-f2809c22c0fb
    volumeID=83186aa6-aaa7-4ce3-9967-7cd3d97d777b
    ++ jq '. | select(.description=="hosted-engine.metadata") | .image'
    ++ xargs
    + metadata_img_uuid=
    + [[ ! -z '' ]]
    + for i in '$(vdsm-client StorageDomain getImages
    storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d
    '\'','\'' | xargs)'
    ++ vdsm-client StorageDomain getVolumes
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=b5f353f5-9357-4aad-b1a3-751d411e6278
    ++ grep -
    ++ tr -d ,
    ++ xargs
    + for v in '$(vdsm-client StorageDomain getVolumes
    storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} |
    grep -
    | tr -d '\'','\'' | xargs)'
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=b5f353f5-9357-4aad-b1a3-751d411e6278
    volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid'
    ++ xargs
    + lockspace_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=b5f353f5-9357-4aad-b1a3-751d411e6278
    volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .image'
    ++ xargs
    + lockspace_img_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=b5f353f5-9357-4aad-b1a3-751d411e6278
    volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f
    ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid'
    ++ xargs
    + metadata_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=b5f353f5-9357-4aad-b1a3-751d411e6278
    volumeID=a20d9700-1b9a-41d8-bb4b-f2b7c168104f
    ++ jq '. | select(.description=="hosted-engine.metadata") | .image'
    ++ xargs
    + metadata_img_uuid=
    + [[ ! -z '' ]]
    + for i in '$(vdsm-client StorageDomain getImages
    storagedomainID=${sdUUID} storagepoolID=${spUUID} | grep - | tr -d
    '\'','\'' | xargs)'
    ++ vdsm-client StorageDomain getVolumes
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749
    ++ grep -
    ++ tr -d ,
    ++ xargs
    + for v in '$(vdsm-client StorageDomain getVolumes
    storagedomainID=${sdUUID} storagepoolID=${spUUID} imageID=${i} |
    grep -
    | tr -d '\'','\'' | xargs)'
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749
    volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .uuid'
    ++ xargs
    + lockspace_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749
    volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b
    ++ jq '. | select(.description=="hosted-engine.lockspace") | .image'
    ++ xargs
    + lockspace_img_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749
    volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b
    ++ jq '. | select(.description=="hosted-engine.metadata") | .uuid'
    ++ xargs
    + metadata_vol_uuid=
    + [[ ! -z '' ]]
    ++ vdsm-client Volume getInfo
    storagedomainID=8905c9ac-d892-478d-8346-63b8fa1c5763
    storagepoolID=00000000-0000-0000-0000-000000000000
    imageID=e9d7a377-e109-4b28-9a43-7a8c8b603749
    volumeID=cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b
    ++ jq '. | select(.description=="hosted-engine.metadata") | .image'
    ++ xargs
    + metadata_img_uuid=
    + [[ ! -z '' ]]
How to create missing volume ? Is there any solution, or just
    reinstall HE?
Hi,
lockspace and metadata volume are supposed to be already there on the 
shared storage as result of past upgrades.
That script was just designed to print out their uuid so that you can 
manually add them under /etc/ovirt-hosted-engine/hosted-engine.conf on 
all the hosts.
From what I can see it's not detecting them on your host although I 
see you have many volumes there.
Can you please try manually running vdsm-client Volume getInfo command 
for all the volumes and manually check volume descriptions?
_______________________________________________
    Users mailing list -- users@ovirt.org <mailto:users@ovirt.org>
    To unsubscribe send an email to users-leave@ovirt.org
    <mailto:users-leave@ovirt.org>
    Privacy Statement: https://www.ovirt.org/site/privacy-policy/
    oVirt Code of Conduct:
    https://www.ovirt.org/community/about/community-guidelines/
    List Archives:
    https://lists.ovirt.org/archives/list/users@ovirt.org/message/3Y7NKSBA3ZNNOT...
Simone thanks for answer!

I'm not so good at using vdm-client, but I think this command get see 
descriptions on my hosted-engine storage ...

[root@h4 etc]# sudo -u vdsm grep -rI --exclude-dir="\.lease" 
"DESCRIPTION" 
/rhev/data-center/mnt/ssd\:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/*
/rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/3abe2f7b-02b9-40a3-8feb-f2809c22c0fb/83186aa6-aaa7-4ce3-9967-7cd3d97d777b.meta:DESCRIPTION={"Updated":true,"Size":20480,"Last 
Updated":"Thu Jun 21 01:40:54 MSK 2018","Storage 
Domains":[{"uuid":"8905c9ac-d892-478d-8346-63b8fa1c5763"}],"Disk 
Description":"OVF_STORE"}
/rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/83e0550b-0fc3-40b1-955d-b07ebfbb3994/5a26be32-6c5b-4dcc-ac67-5c442f24df55.meta:DESCRIPTION={"Updated":true,"Size":20480,"Last 
Updated":"Thu Jun 21 01:40:54 MSK 2018","Storage 
Domains":[{"uuid":"8905c9ac-d892-478d-8346-63b8fa1c5763"}],"Disk 
Description":"OVF_STORE"}
/rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/b5f353f5-9357-4aad-b1a3-751d411e6278/a20d9700-1b9a-41d8-bb4b-f2b7c168104f.meta:DESCRIPTION=HostedEngineConfigurationImage
/rhev/data-center/mnt/ssd:_ovirt/8905c9ac-d892-478d-8346-63b8fa1c5763/images/e9d7a377-e109-4b28-9a43-7a8c8b603749/cd12a59e-7d84-4b4e-98c7-4c68e83ecd7b.meta:DESCRIPTION=Hosted 
Engine Image

Simon, as seen in the descriptions there is no "hosted-engine.lockspace" 
and "hosted-engine.metadata".

How do fix this ?

-- 

С уважением, Резников Алексей
инженер-программист ООО "ГорПТУс",
группа компаний Осколтелеком

web: fly.soskol.ru
email: reznikov_aa@soskol.com
тел.   +7(4725)40-16-77