Re: [ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE

On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <paul(a)pazion.nl&gt; wrote: > > I found vdsClient and ran a few commands. > > [root@geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo > 88b69eba-ef4f-4dbe-ba53-20dadd424d0e > > uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e > > version = 3 > > role = Master > > remotePath = hostedstorage.pazion.nl:/opt/hosted-engine > > type = NFS > > class = Data > > pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4'] > > name = hostedengine_nfs > > > Can you please attach hosted-engine-setup logs? at the end of the setup procedure the hosted-engine setup shouldn't b attached to any storagePool. > [root@geisha-1 ~]# vdsClient -s 0 connectStoragePool > 499b208c-9de9-4a2a-97de-30f410b4e6d4 > > Error using command: list index out of range > > > ... > > > [root@geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo > 499b208c-9de9-4a2a-97de-30f410b4e6d4 > > Unknown pool id, pool not connected: > ('499b208c-9de9-4a2a-97de-30f410b4e6d4',) > > > [root@geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList > > 00000002-0002-0002-0002-000000000385 > > > Can I manually create the missing storage pool which is missing with: > > vdsClient -s 0 createStoragePool 499b208c-9de9-4a2a-97de-30f410b4e6d4 > hostedengine_nfs 88b69eba-ef4f-4dbe-ba53-20dadd424d0e > > If this would be the solution, I still wonder why there is no existing > storagePool for the storageDomain. > > > > Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion < > paul(a)pazion.nl&gt;: > >> I still keep getting >> >> The Hosted Engine Storage Domain doesn't exist. It should be imported >> into the setup. >> >> I went digging a little further and found this in the engine.log >> >> 2016-02-08 22:30:25,083 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] >> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >> HSMGetStorageDomainInfoVDSCommand, return: >> <StorageDomainStatic:{name='hostedengine_nfs', >> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'}, >> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427 >> 2016-02-08 22:30:25,083 INFO >> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] >> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >> GetExistingStorageDomainListQuery, log id: 6ca437b1 >> 2016-02-08 22:30:25,084 INFO >> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object >> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}' >> 2016-02-08 22:30:25,102 WARN >> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action >> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons: >> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST >> >> So it looks like it can't find the storage domain. When I check vdsm.log >> I found an error: >> Thread-711902::ERROR::2016-02-08 >> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': >> {'message': "Unknown pool id, pool not connected: >> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}} >> >> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is also >> trying to connect to in the engine.log. >> >> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768 and >> https://bugzilla.redhat.com/show_bug.cgi?id=1294457 >> >> Can it be I have setup another DC with a master iscsi storage which is >> now offline? ( 1 DC with Cluster, VMs with hosted engine and master >> storage FC all up and running. ) >> >> Not sure where to go from here. Can I somehow restart this storage >> domain / do a new hosted-engine deploy with different settings? >> >> >> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion < >> paul(a)pazion.nl&gt;: >> >>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi < >>> stirabos(a)redhat.com&gt;: >>> >>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion < >>>> paul(a)pazion.nl&gt; wrote: >>>> >>>>> Thanks for your response. >>>>> >>>>> I have reverted the commented entries and restarted ovirt-ha-agent. >>>>> Now status on the daemon gives the error again: >>>>> http://screencast.com/t/DCZo6QJM1 >>>>> >>>>> >>>> Here I'm a bit confused: is this a fresh hosted-engine deployment with >>>> 3.6 or an upgrade from an existing 3.5 hosted-engine deployment? >>>> >>> >>> I have reinstalled host ( complete OS reinstall) and ran hosted-engine >>> --deploy for redeployment on an existing oVirt ( hosted-engine ). >>> >>> As described in previous mail, during the deployment I got errors about >>> a missing hostedengine.metadata symlink to /var/run/vdsd/storage/... >>> I created these by hand and then I ran deployment again and it worked. >>> >>> >>>> >>>>> The hosted engine domain is not shown in the web storage overview... >>>>> When I start import it shows a warning: >>>>> http://screencast.com/t/7HfJ9qWFwb2L >>>>> >>>>> Can I manually import the hostedengine storage? >>>>> ( Hosted engine is visible in web interface and I have running VMs on >>>>> another storage domain ). >>>>> >>>> >>>> It should auto-import it. >>>> Adding Roy here. >>>> >>>> >>> I believe this auto-import is failing... >>> Where can I find logs about this auto-import? >>> >>> >>> >>> >>>> >>>>> >>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi < >>>>> stirabos(a)redhat.com&gt;: >>>>> >>>>>> >>>>>> >>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion < >>>>>> paul(a)pazion.nl&gt; wrote: >>>>>> >>>>>>> Update: >>>>>>> >>>>>> >>>>>> The OVF_STORE volume will contain the engine VM description in a >>>>>> format that the engine can easily handle. >>>>>> It will be created by the engine when: >>>>>> - it imported the hosted-engine storage domain >>>>>> - the engineVM is visible >>>>>> - the datacenter is up (you need at least one additional regular >>>>>> storage domain) >>>>>> >>>>>> Please make sure that all these conditions are met. >>>>>> >>>>>> If the OVF_STORE is not there or it's not accessible, ovirt-ha-agent >>>>>> will fallback to the initial vm.conf from deployment time. You see a >>>>>> warning since we are failing back to an initial status and this can >>>>>> temporary override your latest customization to the engine VM configuration. >>>>>> >>>>>> >>>>>>> 1. I was able to remove the warnirngs about the not found ovf / >>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the >>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were >>>>>>> gone. ( I believe this is not a permanent solution ). >>>>>>> >>>>>> >>>>>> Please revert it. >>>>>> >>>>>> >>>>>>> >>>>>>> 2. This morning I noticed the warning in the event log. >>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be >>>>>>> imported into the setup." >>>>>>> >>>>>> >>>>>> It's just a side effect of your point 1, please revert it. >>>>>> >>>>>> >>>>>>> >>>>>>> Any help would be appreciated. >>>>>>> >>>>>>> Best Regards, >>>>>>> >>>>>>> Paul Groeneweg >>>>>>> >>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion < >>>>>>> paul(a)pazion.nl&gt;: >>>>>>> >>>>>>>> >>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run >>>>>>>> version 3.6. >>>>>>>> When deploying a new host have problems with the shared OVF >>>>>>>> storage config part. >>>>>>>> >>>>>>>> 1. I get an error during deploy: >>>>>>>> >>>>>>>> [ INFO ] Stage: Setup validation >>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed to >>>>>>>> read metadata: [Errno 2] No such file or directory: >>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata' >>>>>>>> [ INFO ] Stage: Clean up >>>>>>>> >>>>>>>> It seems the symlink of hosted-engine.metadata is missing. >>>>>>>> So I added manually the /var/run/vdsm/storage folder with >>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks. >>>>>>>> Then I reran the hosted-engine deploy and this time all went well. >>>>>>>> >>>>>>>> 2. Now when the host is installed I notice the following in the >>>>>>>> logs >>>>>>>> >>>>>>>> .... >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing >>>>>>>> images >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading >>>>>>>> vm.conf from the shared storage domain >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying >>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find >>>>>>>> OVF_STORE >>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable >>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable >>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume >>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - >>>>>>>> trying to restart agent >>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume >>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - >>>>>>>> trying to restart agent >>>>>>>> >>>>>>>> So at this point I was a bit stuck. >>>>>>>> I removed complete ovirt36 release and installed ovirt35 for the >>>>>>>> hosted engine deploy. >>>>>>>> >>>>>>>> This time all went without any problem directly. >>>>>>>> From there I retried upgrade and same problems again. >>>>>>>> >>>>>>>> I run ovirt hosted-engine on nfs3. I did found >>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314 >>>>>>>> I added no_subtree_check and tried deploy again, but again without >>>>>>>> luck. >>>>>>>> >>>>>>>> I believe the hosted engine configs are not being migrated to the >>>>>>>> shared storage. And my problem has todo with >>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage >>>>>>>> >>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf, >>>>>>>> conf_volume_UUID, conf_image_UUID >>>>>>>> >>>>>>>> >>>>>>>> Hosts run el7 >>>>>>>> Hosted engine runs el6 >>>>>>>> Hosted engine storage is through NFS3 >>>>>>>> >>>>>>>> Can I import the hosted engine storage in to engine ( in web >>>>>>>> interface )? >>>>>>>> Or how can I create or trigger manually the OVF store the logs >>>>>>>> complain about? >>>>>>>> >>>>>>>> Best Regards, >>>>>>>> Paul Groeneweg >>>>>>>> >>>>>>>> -- >>>>>>>> Met vriendelijke groeten, >>>>>>>> >>>>>>>> Paul Groeneweg >>>>>>>> Pazion >>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>> >>>>>>>> T +31 26 3020038 >>>>>>>> M +31 614 277 577 >>>>>>>> E paul(a)pazion.nl >>>>>>>> >>>>>>>> ***disclaimer*** >>>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>>> which is confidential and/or protected by intellectual property rights and >>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>>>>> the information contained herein (including, but not limited to, total or >>>>>>>> partial reproduction, communication or distribution in any form) by persons >>>>>>>> other than the designated recipient(s) is prohibited. If you have received >>>>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>>> cooperation." >>>>>>>> >>>>>>> -- >>>>>>> Met vriendelijke groeten, >>>>>>> >>>>>>> Paul Groeneweg >>>>>>> Pazion >>>>>>> Webdevelopment - Hosting - Apps >>>>>>> >>>>>>> T +31 26 3020038 >>>>>>> M +31 614 277 577 >>>>>>> E paul(a)pazion.nl >>>>>>> >>>>>>> ***disclaimer*** >>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>> which is confidential and/or protected by intellectual property rights and >>>>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>>>> the information contained herein (including, but not limited to, total or >>>>>>> partial reproduction, communication or distribution in any form) by persons >>>>>>> other than the designated recipient(s) is prohibited. If you have received >>>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>> cooperation." >>>>>>> >>>>>> -- >>>>> Met vriendelijke groeten, >>>>> >>>>> Paul Groeneweg >>>>> Pazion >>>>> Webdevelopment - Hosting - Apps >>>>> >>>>> T +31 26 3020038 >>>>> M +31 614 277 577 >>>>> E paul(a)pazion.nl >>>>> >>>>> ***disclaimer*** >>>>> "This e-mail and any attachments thereto may contain information >>>>> which is confidential and/or protected by intellectual property rights and >>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>> the information contained herein (including, but not limited to, total or >>>>> partial reproduction, communication or distribution in any form) by persons >>>>> other than the designated recipient(s) is prohibited. If you have received >>>>> this e-mail in error, please notify the sender either by telephone or by >>>>> e-mail and delete the material from any computer. Thank you for your >>>>> cooperation." >>>>> >>>> -- >>> Met vriendelijke groeten, >>> >>> Paul Groeneweg >>> Pazion >>> Webdevelopment - Hosting - Apps >>> >>> T +31 26 3020038 >>> M +31 614 277 577 >>> E paul(a)pazion.nl >>> >>> ***disclaimer*** >>> "This e-mail and any attachments thereto may contain information which >>> is confidential and/or protected by intellectual property rights and are >>> intended for the sole use of the recipient(s) named above. Any use of the >>> information contained herein (including, but not limited to, total or >>> partial reproduction, communication or distribution in any form) by persons >>> other than the designated recipient(s) is prohibited. If you have received >>> this e-mail in error, please notify the sender either by telephone or by >>> e-mail and delete the material from any computer. Thank you for your >>> cooperation." >>> >> -- >> Met vriendelijke groeten, >> >> Paul Groeneweg >> Pazion >> Webdevelopment - Hosting - Apps >> >> T +31 26 3020038 >> M +31 614 277 577 >> E paul(a)pazion.nl >> >> ***disclaimer*** >> "This e-mail and any attachments thereto may contain information which >> is confidential and/or protected by intellectual property rights and are >> intended for the sole use of the recipient(s) named above. Any use of the >> information contained herein (including, but not limited to, total or >> partial reproduction, communication or distribution in any form) by persons >> other than the designated recipient(s) is prohibited. If you have received >> this e-mail in error, please notify the sender either by telephone or by >> e-mail and delete the material from any computer. Thank you for your >> cooperation." >> > -- > Met vriendelijke groeten, > > Paul Groeneweg > Pazion > Webdevelopment - Hosting - Apps > > T +31 26 3020038 > M +31 614 277 577 > E paul(a)pazion.nl > > ***disclaimer*** > "This e-mail and any attachments thereto may contain information which is > confidential and/or protected by intellectual property rights and are > intended for the sole use of the recipient(s) named above. Any use of the > information contained herein (including, but not limited to, total or > partial reproduction, communication or distribution in any form) by persons > other than the designated recipient(s) is prohibited. If you have received > this e-mail in error, please notify the sender either by telephone or by > e-mail and delete the material from any computer. Thank you for your > cooperation." > --