[ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE
Simone Tiraboschi
stirabos at redhat.com
Tue Feb 9 09:42:59 UTC 2016
On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
wrote:
> I have added 2 logs from 2 hosts
>
>
Both the hosts are just additional host:
2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
dialog.__logString:219 DIALOG:SEND The specified storage
location already contains a data domain. Is this an additional host setup
(Yes, No)[Yes]?
2016-02-07 20:32:37 INFO
otopi.plugins.ovirt_hosted_engine_setup.storage.storage
storage._handleHostId:249 Installing on additional host
2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
dialog.__logString:219 DIALOG:SEND The specified storage
location already contains a data domain. Is this an additional host setup
(Yes, No)[Yes]?
2016-02-05 23:47:07 INFO
otopi.plugins.ovirt_hosted_engine_setup.storage.storage
storage._handleHostId:123 Installing on additional host
Did you redeployed also the engine VM on a fresh storage domain or di you
just run 'hosted-engine --deploy' again on all the hosts?
If it's just an upgrade you don't have to run 'hosted-engine --deploy'
again.
Please follow this guide:
http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
>
> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi <stirabos at redhat.com
> >:
>
>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <paul at pazion.nl>
>> wrote:
>>
>>>
>>> I found vdsClient and ran a few commands.
>>>
>>> [root at geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo
>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>
>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>
>>> version = 3
>>>
>>> role = Master
>>>
>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine
>>>
>>> type = NFS
>>>
>>> class = Data
>>>
>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']
>>>
>>> name = hostedengine_nfs
>>>
>>>
>>>
>> Can you please attach hosted-engine-setup logs? at the end of the setup
>> procedure the hosted-engine setup shouldn't b attached to any storagePool.
>>
>>
>>> [root at geisha-1 ~]# vdsClient -s 0 connectStoragePool
>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>
>>> Error using command: list index out of range
>>>
>>>
>>> ...
>>>
>>>
>>> [root at geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo
>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>
>>> Unknown pool id, pool not connected:
>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)
>>>
>>>
>>> [root at geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList
>>>
>>> 00000002-0002-0002-0002-000000000385
>>>
>>>
>>> Can I manually create the missing storage pool which is missing with:
>>>
>>> vdsClient -s 0 createStoragePool 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>> hostedengine_nfs 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>
>>> If this would be the solution, I still wonder why there is no existing
>>> storagePool for the storageDomain.
>>>
>>>
>>>
>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <
>>> paul at pazion.nl>:
>>>
>>>> I still keep getting
>>>>
>>>> The Hosted Engine Storage Domain doesn't exist. It should be imported
>>>> into the setup.
>>>>
>>>> I went digging a little further and found this in the engine.log
>>>>
>>>> 2016-02-08 22:30:25,083 INFO
>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand]
>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>> HSMGetStorageDomainInfoVDSCommand, return:
>>>> <StorageDomainStatic:{name='hostedengine_nfs',
>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'},
>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427
>>>> 2016-02-08 22:30:25,083 INFO
>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery]
>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1
>>>> 2016-02-08 22:30:25,084 INFO
>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object
>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
>>>> 2016-02-08 22:30:25,102 WARN
>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action
>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
>>>>
>>>> So it looks like it can't find the storage domain. When I check
>>>> vdsm.log I found an error:
>>>> Thread-711902::ERROR::2016-02-08
>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
>>>> {'message': "Unknown pool id, pool not connected:
>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}
>>>>
>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is also
>>>> trying to connect to in the engine.log.
>>>>
>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768 and
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1294457
>>>>
>>>> Can it be I have setup another DC with a master iscsi storage which is
>>>> now offline? ( 1 DC with Cluster, VMs with hosted engine and master
>>>> storage FC all up and running. )
>>>>
>>>> Not sure where to go from here. Can I somehow restart this storage
>>>> domain / do a new hosted-engine deploy with different settings?
>>>>
>>>>
>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <
>>>> paul at pazion.nl>:
>>>>
>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <
>>>>> stirabos at redhat.com>:
>>>>>
>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <
>>>>>> paul at pazion.nl> wrote:
>>>>>>
>>>>>>> Thanks for your response.
>>>>>>>
>>>>>>> I have reverted the commented entries and restarted ovirt-ha-agent.
>>>>>>> Now status on the daemon gives the error again:
>>>>>>> http://screencast.com/t/DCZo6QJM1
>>>>>>>
>>>>>>>
>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment
>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?
>>>>>>
>>>>>
>>>>> I have reinstalled host ( complete OS reinstall) and ran
>>>>> hosted-engine --deploy for redeployment on an existing oVirt (
>>>>> hosted-engine ).
>>>>>
>>>>> As described in previous mail, during the deployment I got errors
>>>>> about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/...
>>>>> I created these by hand and then I ran deployment again and it worked.
>>>>>
>>>>>
>>>>>>
>>>>>>> The hosted engine domain is not shown in the web storage overview...
>>>>>>> When I start import it shows a warning:
>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L
>>>>>>>
>>>>>>> Can I manually import the hostedengine storage?
>>>>>>> ( Hosted engine is visible in web interface and I have running VMs
>>>>>>> on another storage domain ).
>>>>>>>
>>>>>>
>>>>>> It should auto-import it.
>>>>>> Adding Roy here.
>>>>>>
>>>>>>
>>>>> I believe this auto-import is failing...
>>>>> Where can I find logs about this auto-import?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>>>
>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <
>>>>>>> stirabos at redhat.com>:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <
>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>
>>>>>>>>> Update:
>>>>>>>>>
>>>>>>>>
>>>>>>>> The OVF_STORE volume will contain the engine VM description in a
>>>>>>>> format that the engine can easily handle.
>>>>>>>> It will be created by the engine when:
>>>>>>>> - it imported the hosted-engine storage domain
>>>>>>>> - the engineVM is visible
>>>>>>>> - the datacenter is up (you need at least one additional regular
>>>>>>>> storage domain)
>>>>>>>>
>>>>>>>> Please make sure that all these conditions are met.
>>>>>>>>
>>>>>>>> If the OVF_STORE is not there or it's not accessible,
>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from deployment time.
>>>>>>>> You see a warning since we are failing back to an initial status and this
>>>>>>>> can temporary override your latest customization to the engine VM
>>>>>>>> configuration.
>>>>>>>>
>>>>>>>>
>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf /
>>>>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the
>>>>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were
>>>>>>>>> gone. ( I believe this is not a permanent solution ).
>>>>>>>>>
>>>>>>>>
>>>>>>>> Please revert it.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2. This morning I noticed the warning in the event log.
>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>>>> imported into the setup."
>>>>>>>>>
>>>>>>>>
>>>>>>>> It's just a side effect of your point 1, please revert it.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Any help would be appreciated.
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>>
>>>>>>>>> Paul Groeneweg
>>>>>>>>>
>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <
>>>>>>>>> paul at pazion.nl>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run
>>>>>>>>>> version 3.6.
>>>>>>>>>> When deploying a new host have problems with the shared OVF
>>>>>>>>>> storage config part.
>>>>>>>>>>
>>>>>>>>>> 1. I get an error during deploy:
>>>>>>>>>>
>>>>>>>>>> [ INFO ] Stage: Setup validation
>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed to
>>>>>>>>>> read metadata: [Errno 2] No such file or directory:
>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'
>>>>>>>>>> [ INFO ] Stage: Clean up
>>>>>>>>>>
>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing.
>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with
>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.
>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went well.
>>>>>>>>>>
>>>>>>>>>> 2. Now when the host is installed I notice the following in the
>>>>>>>>>> logs
>>>>>>>>>>
>>>>>>>>>> ....
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing
>>>>>>>>>> images
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading
>>>>>>>>>> vm.conf from the shared storage domain
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying
>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find
>>>>>>>>>> OVF_STORE
>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable
>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume
>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>> trying to restart agent
>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume
>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>> trying to restart agent
>>>>>>>>>>
>>>>>>>>>> So at this point I was a bit stuck.
>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for the
>>>>>>>>>> hosted engine deploy.
>>>>>>>>>>
>>>>>>>>>> This time all went without any problem directly.
>>>>>>>>>> From there I retried upgrade and same problems again.
>>>>>>>>>>
>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found
>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314
>>>>>>>>>> I added no_subtree_check and tried deploy again, but again
>>>>>>>>>> without luck.
>>>>>>>>>>
>>>>>>>>>> I believe the hosted engine configs are not being migrated to the
>>>>>>>>>> shared storage. And my problem has todo with
>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage
>>>>>>>>>>
>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf,
>>>>>>>>>> conf_volume_UUID, conf_image_UUID
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hosts run el7
>>>>>>>>>> Hosted engine runs el6
>>>>>>>>>> Hosted engine storage is through NFS3
>>>>>>>>>>
>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web
>>>>>>>>>> interface )?
>>>>>>>>>> Or how can I create or trigger manually the OVF store the logs
>>>>>>>>>> complain about?
>>>>>>>>>>
>>>>>>>>>> Best Regards,
>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>
>>>>>>>>>> Paul Groeneweg
>>>>>>>>>> Pazion
>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>
>>>>>>>>>> T +31 26 3020038
>>>>>>>>>> M +31 614 277 577
>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>
>>>>>>>>>> ***disclaimer***
>>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>>> cooperation."
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>
>>>>>>>>> Paul Groeneweg
>>>>>>>>> Pazion
>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>
>>>>>>>>> T +31 26 3020038
>>>>>>>>> M +31 614 277 577
>>>>>>>>> E paul at pazion.nl
>>>>>>>>>
>>>>>>>>> ***disclaimer***
>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>> cooperation."
>>>>>>>>>
>>>>>>>> --
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>> Pazion
>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>
>>>>>>> T +31 26 3020038
>>>>>>> M +31 614 277 577
>>>>>>> E paul at pazion.nl
>>>>>>>
>>>>>>> ***disclaimer***
>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>> cooperation."
>>>>>>>
>>>>>> --
>>>>> Met vriendelijke groeten,
>>>>>
>>>>> Paul Groeneweg
>>>>> Pazion
>>>>> Webdevelopment - Hosting - Apps
>>>>>
>>>>> T +31 26 3020038
>>>>> M +31 614 277 577
>>>>> E paul at pazion.nl
>>>>>
>>>>> ***disclaimer***
>>>>> "This e-mail and any attachments thereto may contain information which
>>>>> is confidential and/or protected by intellectual property rights and are
>>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>>> information contained herein (including, but not limited to, total or
>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>> cooperation."
>>>>>
>>>> --
>>>> Met vriendelijke groeten,
>>>>
>>>> Paul Groeneweg
>>>> Pazion
>>>> Webdevelopment - Hosting - Apps
>>>>
>>>> T +31 26 3020038
>>>> M +31 614 277 577
>>>> E paul at pazion.nl
>>>>
>>>> ***disclaimer***
>>>> "This e-mail and any attachments thereto may contain information which
>>>> is confidential and/or protected by intellectual property rights and are
>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>> information contained herein (including, but not limited to, total or
>>>> partial reproduction, communication or distribution in any form) by persons
>>>> other than the designated recipient(s) is prohibited. If you have received
>>>> this e-mail in error, please notify the sender either by telephone or by
>>>> e-mail and delete the material from any computer. Thank you for your
>>>> cooperation."
>>>>
>>> --
>>> Met vriendelijke groeten,
>>>
>>> Paul Groeneweg
>>> Pazion
>>> Webdevelopment - Hosting - Apps
>>>
>>> T +31 26 3020038
>>> M +31 614 277 577
>>> E paul at pazion.nl
>>>
>>> ***disclaimer***
>>> "This e-mail and any attachments thereto may contain information which
>>> is confidential and/or protected by intellectual property rights and are
>>> intended for the sole use of the recipient(s) named above. Any use of the
>>> information contained herein (including, but not limited to, total or
>>> partial reproduction, communication or distribution in any form) by persons
>>> other than the designated recipient(s) is prohibited. If you have received
>>> this e-mail in error, please notify the sender either by telephone or by
>>> e-mail and delete the material from any computer. Thank you for your
>>> cooperation."
>>>
>> --
> Met vriendelijke groeten,
>
> Paul Groeneweg
> Pazion
> Webdevelopment - Hosting - Apps
>
> T +31 26 3020038
> M +31 614 277 577
> E paul at pazion.nl
>
> ***disclaimer***
> "This e-mail and any attachments thereto may contain information which is
> confidential and/or protected by intellectual property rights and are
> intended for the sole use of the recipient(s) named above. Any use of the
> information contained herein (including, but not limited to, total or
> partial reproduction, communication or distribution in any form) by persons
> other than the designated recipient(s) is prohibited. If you have received
> this e-mail in error, please notify the sender either by telephone or by
> e-mail and delete the material from any computer. Thank you for your
> cooperation."
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160209/6e20319d/attachment-0001.html>
More information about the Users
mailing list