[ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE

Tue Feb 9 08:07:20 UTC 2016

On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <paul at pazion.nl>
wrote:

>
> I found vdsClient and ran a few commands.
>
> [root at geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo
> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>
> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>
> version = 3
>
> role = Master
>
> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine
>
> type = NFS
>
> class = Data
>
> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']
>
> name = hostedengine_nfs
>
>
>
Can you please attach hosted-engine-setup logs? at the end of the setup
procedure the hosted-engine setup shouldn't b attached to any storagePool.

> [root at geisha-1 ~]# vdsClient -s 0 connectStoragePool
> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>
> Error using command: list index out of range
>
>
> ...
>
>
> [root at geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo
> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>
> Unknown pool id, pool not connected:
> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)
>
>
> [root at geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList
>
> 00000002-0002-0002-0002-000000000385
>
>
> Can I manually create the missing storage pool which is missing with:
>
> vdsClient  -s 0 createStoragePool 499b208c-9de9-4a2a-97de-30f410b4e6d4
> hostedengine_nfs 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>
> If this would be the solution, I still wonder why there is no existing
> storagePool for the storageDomain.
>
>
>
> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <paul at pazion.nl
> >:
>
>> I still keep getting
>>
>> The Hosted Engine Storage Domain doesn't exist. It should be imported
>> into the setup.
>>
>> I went digging a little further and found this in the engine.log
>>
>> 2016-02-08 22:30:25,083 INFO
>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand]
>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>> HSMGetStorageDomainInfoVDSCommand, return:
>> <StorageDomainStatic:{name='hostedengine_nfs',
>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'},
>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427
>> 2016-02-08 22:30:25,083 INFO
>>  [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery]
>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>> GetExistingStorageDomainListQuery, log id: 6ca437b1
>> 2016-02-08 22:30:25,084 INFO
>>  [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object
>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
>> 2016-02-08 22:30:25,102 WARN
>>  [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action
>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
>>
>> So it looks like it can't find the storage domain. When I check vdsm.log
>> I found an error:
>> Thread-711902::ERROR::2016-02-08
>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
>> {'message': "Unknown pool id, pool not connected:
>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}
>>
>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is also
>> trying to connect to in the engine.log.
>>
>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768  and
>> https://bugzilla.redhat.com/show_bug.cgi?id=1294457
>>
>> Can it be I have setup another DC with a master iscsi storage  which is
>> now offline?  ( 1 DC with Cluster,  VMs with hosted engine and master
>> storage FC all up and running.  )
>>
>> Not sure where to go from here. Can I somehow restart this storage domain
>> / do a new hosted-engine deploy with different settings?
>>
>>
>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <
>> paul at pazion.nl>:
>>
>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <
>>> stirabos at redhat.com>:
>>>
>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <
>>>> paul at pazion.nl> wrote:
>>>>
>>>>> Thanks for your response.
>>>>>
>>>>> I have reverted the commented entries and restarted ovirt-ha-agent.
>>>>> Now status on the daemon gives the error again:
>>>>> http://screencast.com/t/DCZo6QJM1
>>>>>
>>>>>
>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment with
>>>> 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?
>>>>
>>>
>>> I have  reinstalled host ( complete OS reinstall) and ran hosted-engine
>>> --deploy for redeployment on an existing oVirt ( hosted-engine ).
>>>
>>> As described in previous mail, during the deployment I got errors about
>>> a missing hostedengine.metadata symlink to /var/run/vdsd/storage/...
>>> I created these by hand and then I ran deployment again and it worked.
>>>
>>>
>>>>
>>>>> The hosted engine domain is not shown in the web storage overview...
>>>>> When I start import it shows a warning:
>>>>> http://screencast.com/t/7HfJ9qWFwb2L
>>>>>
>>>>> Can I manually import the hostedengine storage?
>>>>> ( Hosted engine is visible in web interface and I have running VMs on
>>>>> another storage domain ).
>>>>>
>>>>
>>>> It should auto-import it.
>>>> Adding Roy here.
>>>>
>>>>
>>> I believe this auto-import is failing...
>>> Where can I find logs about this auto-import?
>>>
>>>
>>>
>>>
>>>>
>>>>>
>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <
>>>>> stirabos at redhat.com>:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <
>>>>>> paul at pazion.nl> wrote:
>>>>>>
>>>>>>> Update:
>>>>>>>
>>>>>>
>>>>>> The OVF_STORE volume will contain the engine VM description in a
>>>>>> format that the engine can easily handle.
>>>>>> It will be created by the engine when:
>>>>>> - it imported the hosted-engine storage domain
>>>>>> - the engineVM is visible
>>>>>> - the datacenter is up (you need at least one additional regular
>>>>>> storage domain)
>>>>>>
>>>>>> Please make sure that all these conditions are met.
>>>>>>
>>>>>> If the OVF_STORE is not there or it's not accessible, ovirt-ha-agent
>>>>>> will fallback to the initial vm.conf from deployment time. You see a
>>>>>> warning since we are failing back to an initial status and this can
>>>>>> temporary override your latest customization to the engine VM configuration.
>>>>>>
>>>>>>
>>>>>>> 1. I was able to remove the warnirngs about the not found ovf /
>>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the
>>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were
>>>>>>> gone. ( I believe this is not a permanent solution ).
>>>>>>>
>>>>>>
>>>>>> Please revert it.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> 2. This morning I noticed the warning in the event log.
>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>> imported into the setup."
>>>>>>>
>>>>>>
>>>>>> It's just a side effect of your point 1, please revert it.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> Any help would be appreciated.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>>
>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <
>>>>>>> paul at pazion.nl>:
>>>>>>>
>>>>>>>>
>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run
>>>>>>>> version 3.6.
>>>>>>>> When deploying a new host have problems with the shared OVF storage
>>>>>>>> config part.
>>>>>>>>
>>>>>>>> 1. I get an error during deploy:
>>>>>>>>
>>>>>>>> [ INFO  ] Stage: Setup validation
>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed to
>>>>>>>> read metadata: [Errno 2] No such file or directory:
>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'
>>>>>>>> [ INFO  ] Stage: Clean up
>>>>>>>>
>>>>>>>> It seems the symlink of hosted-engine.metadata is missing.
>>>>>>>> So I added manually the /var/run/vdsm/storage folder with subfolder
>>>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.
>>>>>>>> Then I reran the hosted-engine deploy and this time all went well.
>>>>>>>>
>>>>>>>> 2. Now when the host is installed I notice the following in the logs
>>>>>>>>
>>>>>>>> ....
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing
>>>>>>>> images
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading
>>>>>>>> vm.conf from the shared storage domain
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying
>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find
>>>>>>>> OVF_STORE
>>>>>>>> Feb  5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable
>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>> Feb  5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume
>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>> trying to restart agent
>>>>>>>> Feb  5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume
>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>> trying to restart agent
>>>>>>>>
>>>>>>>> So at this point I was a bit stuck.
>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for the
>>>>>>>> hosted engine deploy.
>>>>>>>>
>>>>>>>> This time all went without any problem directly.
>>>>>>>> From there I retried upgrade and same problems again.
>>>>>>>>
>>>>>>>> I run ovirt hosted-engine on nfs3. I did found
>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314
>>>>>>>> I added no_subtree_check and tried deploy again, but again without
>>>>>>>> luck.
>>>>>>>>
>>>>>>>> I believe the hosted engine configs are not being migrated to the
>>>>>>>> shared storage. And my problem has todo with
>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage
>>>>>>>>
>>>>>>>> And found  /etc/ovirt-hosted-engine/hosted-engine.conf,
>>>>>>>> conf_volume_UUID, conf_image_UUID
>>>>>>>>
>>>>>>>>
>>>>>>>> Hosts run el7
>>>>>>>> Hosted engine runs el6
>>>>>>>> Hosted engine storage is through NFS3
>>>>>>>>
>>>>>>>> Can I import the hosted engine storage in to engine ( in web
>>>>>>>> interface )?
>>>>>>>> Or how can I create or trigger manually the OVF store the logs
>>>>>>>> complain about?
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Paul Groeneweg
>>>>>>>>
>>>>>>>> --
>>>>>>>> Met vriendelijke groeten,
>>>>>>>>
>>>>>>>> Paul Groeneweg
>>>>>>>> Pazion
>>>>>>>> Webdevelopment  -  Hosting  -  Apps
>>>>>>>>
>>>>>>>> T +31 26 3020038
>>>>>>>> M +31 614 277 577
>>>>>>>> E  paul at pazion.nl
>>>>>>>>
>>>>>>>>  ***disclaimer***
>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>> cooperation."
>>>>>>>>
>>>>>>> --
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>> Pazion
>>>>>>> Webdevelopment  -  Hosting  -  Apps
>>>>>>>
>>>>>>> T +31 26 3020038
>>>>>>> M +31 614 277 577
>>>>>>> E  paul at pazion.nl
>>>>>>>
>>>>>>>  ***disclaimer***
>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>> cooperation."
>>>>>>>
>>>>>> --
>>>>> Met vriendelijke groeten,
>>>>>
>>>>> Paul Groeneweg
>>>>> Pazion
>>>>> Webdevelopment  -  Hosting  -  Apps
>>>>>
>>>>> T +31 26 3020038
>>>>> M +31 614 277 577
>>>>> E  paul at pazion.nl
>>>>>
>>>>>  ***disclaimer***
>>>>> "This e-mail and any attachments thereto may contain information which
>>>>> is confidential and/or protected by intellectual property rights and are
>>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>>> information contained herein (including, but not limited to, total or
>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>> cooperation."
>>>>>
>>>> --
>>> Met vriendelijke groeten,
>>>
>>> Paul Groeneweg
>>> Pazion
>>> Webdevelopment  -  Hosting  -  Apps
>>>
>>> T +31 26 3020038
>>> M +31 614 277 577
>>> E  paul at pazion.nl
>>>
>>>  ***disclaimer***
>>> "This e-mail and any attachments thereto may contain information which
>>> is confidential and/or protected by intellectual property rights and are
>>> intended for the sole use of the recipient(s) named above. Any use of the
>>> information contained herein (including, but not limited to, total or
>>> partial reproduction, communication or distribution in any form) by persons
>>> other than the designated recipient(s) is prohibited. If you have received
>>> this e-mail in error, please notify the sender either by telephone or by
>>> e-mail and delete the material from any computer. Thank you for your
>>> cooperation."
>>>
>> --
>> Met vriendelijke groeten,
>>
>> Paul Groeneweg
>> Pazion
>> Webdevelopment  -  Hosting  -  Apps
>>
>> T +31 26 3020038
>> M +31 614 277 577
>> E  paul at pazion.nl
>>
>>  ***disclaimer***
>> "This e-mail and any attachments thereto may contain information which is
>> confidential and/or protected by intellectual property rights and are
>> intended for the sole use of the recipient(s) named above. Any use of the
>> information contained herein (including, but not limited to, total or
>> partial reproduction, communication or distribution in any form) by persons
>> other than the designated recipient(s) is prohibited. If you have received
>> this e-mail in error, please notify the sender either by telephone or by
>> e-mail and delete the material from any computer. Thank you for your
>> cooperation."
>>
> --
> Met vriendelijke groeten,
>
> Paul Groeneweg
> Pazion
> Webdevelopment  -  Hosting  -  Apps
>
> T +31 26 3020038
> M +31 614 277 577
> E  paul at pazion.nl
>
>  ***disclaimer***
> "This e-mail and any attachments thereto may contain information which is
> confidential and/or protected by intellectual property rights and are
> intended for the sole use of the recipient(s) named above. Any use of the
> information contained herein (including, but not limited to, total or
> partial reproduction, communication or distribution in any form) by persons
> other than the designated recipient(s) is prohibited. If you have received
> this e-mail in error, please notify the sender either by telephone or by
> e-mail and delete the material from any computer. Thank you for your
> cooperation."
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160209/febe0ed3/attachment-0001.html>