Re: [ovirt-users] Failed to import the Hosted Engine Storage Domain

On 09/04/2017 06:32 PM, Simone Tiraboschi wrote: On Mon, Sep 4, 2017 at 6:24 PM, Arsène Gschwind <arsene.gschwind(a)unibas.ch > wrote: > > > On 09/04/2017 02:51 PM, Simone Tiraboschi wrote: > > > > On Mon, Sep 4, 2017 at 2:21 PM, Arsène Gschwind < > arsene.gschwind(a)unibas.ch&gt; wrote: > >> >> >> On 09/04/2017 02:01 PM, Simone Tiraboschi wrote: >> >> >> >> On Mon, Sep 4, 2017 at 1:55 PM, Arsène Gschwind < >> arsene.gschwind(a)unibas.ch&gt; wrote: >> >>> >>> >>> On 09/04/2017 01:52 PM, Simone Tiraboschi wrote: >>> >>> >>> >>> On Mon, Sep 4, 2017 at 12:23 PM, Arsène Gschwind < >>> arsene.gschwind(a)unibas.ch&gt; wrote: >>> >>>> Hi Simone, >>>> >>>> On 09/04/2017 11:14 AM, Simone Tiraboschi wrote: >>>> >>>> >>>> >>>> On Mon, Sep 4, 2017 at 10:56 AM, Arsène Gschwind < >>>> arsene.gschwind(a)unibas.ch&gt; wrote: >>>> >>>>> Hi Didi, >>>>> >>>>> On 09/04/2017 10:15 AM, Yedidyah Bar David wrote: >>>>> >>>>> On Mon, Sep 4, 2017 at 10:16 AM, Arsène Gschwind&lt;arsene.gschwind(a)unibas.ch&gt; <arsene.gschwind(a)unibas.ch&gt; wrote: >>>>> >>>>> Hi all, >>>>> >>>>> A while ago I had some problem with hosted-engine network which wasn't set >>>>> correctly at deploy time, so I finally decided to redeploy the hosted engine >>>>> in the hope the network will be set correctly this time. I've followed this >>>>> procedure: >>>>> >>>>> Stop all VMs >>>>> Full backup of HE DB and export to safe place >>>>> Cleanup HE storage following https://access.redhat.com/solutions/2121581 >>>>> Reboot Hosts >>>>> Re-deploy HE until DB recovery >>>>> Recover DB adding the following param: >>>>> --he-remove-storage-vm Removes the hosted-engine storage >>>>> domain, all its entities and the hosted-engine VM during restore. >>>>> --he-remove-hosts Removes all the hosted-engine hosts >>>>> during restore. >>>>> >>>>> Finalize HE deployment. >>>>> >>>>> Everything did run without errors and I'm able to access Web UI. >>>>> >>>>> But now I don't see my HE VM and its respective Storage Domain, the logs >>>>> says it isn't able to import it. I see all other SD and I'm able to manage >>>>> my VMs as before. >>>>> >>>>> Please find attached engine.log >>>>> >>>>> I think this is your problem: >>>>> >>>>> 2017-09-04 03:26:14,272+02 INFO >>>>> [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand] >>>>> (org.ovirt.thread.pool-6-thread-24) [2383eaa0] There are existing luns >>>>> in the system which are part of VG id >>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g' >>>>> >>>>> I don't see a VG with this ID, here the IDs I see on the hosts: >>>>> >>>>> VG #PV #LV #SN Attr VSize >>>>> VFree >>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1 10 0 wz--n- 99.62g >>>>> 14.50g >>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2 70 0 wz--n- 10.00t >>>>> 2.31t >>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2 47 0 wz--n- 5.27t >>>>> 2.50t >>>>> >>>> >>>> >>>> Could you please repeat the command on host adm-kvmh70 ? >>>> >>>> 2017-09-04 09:04:18,163+02 INFO [org.ovirt.engine.core.bll.st >>>> orage.domain.ImportHostedEngineStorageDomainCommand] >>>> (org.ovirt.thread.pool-6-thread-34) [247a3718] Running command: >>>> ImportHostedEngineStorageDomainCommand internal: true. >>>> 2017-09-04 09:04:18,189+02 INFO [org.ovirt.engine.core.vdsbro >>>> ker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) >>>> [7d2e6cb2] START, GetVGInfoVDSCommand(HostName = adm-kvmh70, >>>> GetVGInfoVDSCommandParameters:{runAsync='true', >>>> hostId='acbacabb-6c4a-43fd-a1e2-2d7ff2f6f98b', >>>> VGID='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'}), log id: 6693b98a >>>> 2017-09-04 09:04:18,232+02 INFO [org.ovirt.engine.core.vdsbro >>>> ker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) >>>> [7d2e6cb2] FINISH, GetVGInfoVDSCommand, return: >>>> [LUNs:{id='repl_HostedEngine', physicalVolumeId='kYN8Jj-FBDw-MhxI-XcoZ-w1zH-eQL8-IRIgzO', >>>> volumeGroupId='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g', >>>> serial='SHITACHI_OPEN-V_50488888', lunMapping='4', >>>> vendorId='HITACHI', productId='OPEN-V', lunConnections='[]', >>>> deviceSize='100', pvSize='0', peCount='797', peAllocatedCount='681', >>>> vendorName='HITACHI', pathsDictionary='[sdf=true, sdu=true, sdk=true, >>>> sdp=true]', pathsCapacity='[sdf=100, sdu=100, sdk=100, sdp=100]', >>>> lunType='FCP', status='null', diskId='null', diskAlias='null', >>>> storageDomainId='6b62cc06-fc44-4c38-af6d-bfd9cbe73246', >>>> storageDomainName='null', discardMaxSize='268435456', >>>> discardZeroesData='true'}], log id: 6693b98a >>>> 2017-09-04 09:04:18,245+02 INFO [org.ovirt.engine.core.bll.st >>>> orage.domain.AddExistingBlockStorageDomainCommand] >>>> (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] There are existing >>>> luns in the system which are part of VG id 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL >>>> 2z-wbTU8g' >>>> 2017-09-04 09:04:18,245+02 WARN [org.ovirt.engine.core.bll.st >>>> orage.domain.AddExistingBlockStorageDomainCommand] >>>> (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] Validation of action >>>> 'AddExistingBlockStorageDomain' failed for user SYSTEM. Reasons: >>>> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAIL >>>> ED_IMPORT_STORAGE_DOMAIN_EXTERNAL_LUN_DISK_EXIST >>>> >>>> I don't know which command you are talking about, I didn't run any >>>> command since it tries to import the SD automatically. >>>> >>> >>> Sorry, can you please run vgdisplay on adm-kvmh70 ? >>> >>> No problem! here the result: >>> >>> [root@adm-kvmh70 ~]# vgs >>> VG #PV #LV #SN Attr VSize >>> VFree >>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1 10 0 wz--n- 99.62g >>> 14.50g >>> b0414c06-d984-4001-a998-fd9a2e79fb83 2 70 0 wz--n- 10.00t >>> 2.31t >>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2 47 0 wz--n- 5.27t >>> 2.50t >>> vg_adm-kvmh70 1 3 0 wz--n- 277.90g >>> 218.68g >>> >>> >> OK, and >> vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >> >> [root@adm-kvmh70 ~]# vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >> --- Volume group --- >> VG Name 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >> System ID >> Format lvm2 >> Metadata Areas 2 >> Metadata Sequence No 23 >> VG Access read/write >> VG Status resizable >> MAX LV 0 >> Cur LV 10 >> Open LV 3 >> Max PV 0 >> Cur PV 1 >> Act PV 1 >> VG Size 99.62 GiB >> PE Size 128.00 MiB >> Total PE 797 >> Alloc PE / Size 681 / 85.12 GiB >> Free PE / Size 116 / 14.50 GiB >> VG UUID vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g >> > > OK, vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g is the uuid of the VG that > engine says that it includes the LUN used by the hosted-engine SD as > detected by the engine. > Its name is 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 which is also the uuid > of that storage domain. > > Now we need to understand what's there. > > Could you please check if an SD with uuid 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 > is visible in the engine? > > The visible SD in the engine > > [oVirt shell (connected)]# list storagedomains > > id : 072fbaa1-08f3-4a40-9f34-a5ca22dd1d74 > name : ovirt-image-repository > description: Public Glance repository for oVirt > > id : b2e30961-7cff-4cca-83d6-bee3a4f890ee > name : xxxxxxxxxxx > > id : b0414c06-d984-4001-a998-fd9a2e79fb83 > name : xxxxxxxxxxx > description: xxxxxxxxxxx > > id : 71fa1c61-4a68-469f-9a56-835190d32075 > name : xxxxxxxxxxx > > id : 9817e64b-fdd9-4d4c-91ee-f3482bb0cc56 > name : xxxxxxxxxxxx > description: xxxxxxxxxxxx > > Is adm-kvmh70 involved in hosted-engine? did you redeployed it? > > Yes adm-kvmh70 should be a host hosting HE and yes as written before I > did redeploy HE because of a network interface problem. I did cleanup the > lun using the RedHat guide, redeploy HE and restore the DB using the > mentioned parameter to remove HE information. > Could you please set adm-kvmh70 to maintenance mode from the engine and then reactivate it or better reboot it while in maintenance mode? I suspect that 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is just a left over of the previous hosted-engine storage domain kept open on the host. I did reboot adm-kvmh70 while in maintenance mode and also reboot the engine but it didn't change anything. The import failure is still there... I've noticed a ovirt-ha-agent error: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs Could that be related?

Thanks > > > In the past did you tried to manually import the hosted-engine SD or > something similar? > > No, before the "redeployment" the HE SD was auto imported successfully. > > Thanks a lot > > > > > thanks > > > > >> >> Thanks >> >> ? >> >> >>> >>> >>> >>>> >>>> Thanks, >>>> Arsene >>>> >>>> >>>> >>>>> >>>>> >>>>> Thanks for any help to resolve that issue. >>>>> >>>>> I guess you can try to remove this disk/lun from the engine and let it retry. >>>>> >>>>> Could let me know how to remove that lun from the engine? >>>>> >>>>> If the only disk is of the hosted-engine, I guess it should have been >>>>> removed by '--he-remove-storage-vm' - if so, please open a bug >>>>> describing your flow in detail. Thanks. >>>>> >>>>> It seems that this option didn't remove the he storage information >>>>> and that this VG is still the old one. >>>>> >>>>> Many thanks for your help >>>>> Rgds, >>>>> Arsene >>>>> >>>>> Best, >>>>> >>>>> >>>>> Arsène >>>>> >>>>> -- >>>>> >>>>> Arsène Gschwind >>>>> Fa. Sapify AG im Auftrag der Universität Basel >>>>> IT Services >>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch >>>>> ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 <+41%2061%20267%2014%2011> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing listUsers@ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>>> -- >>>>> >>>>> *Arsène Gschwind* >>>>> Fa. Sapify AG im Auftrag der Universität Basel >>>>> IT Services >>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch >>>>> >>>>> ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 >>>>> <+41%2061%20267%2014%2011> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>>> -- >>>> >>>> *Arsène Gschwind* >>>> Fa. Sapify AG im Auftrag der Universität Basel >>>> IT Services >>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch >>>> >>>> ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 >>>> <+41%2061%20267%2014%2011> >>>> >>> >>> >>> -- >>> >>> *Arsène Gschwind* >>> Fa. Sapify AG im Auftrag der Universität Basel >>> IT Services >>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch >>> >>> ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 >>> <+41%2061%20267%2014%2011> >>> >> >> >> -- >> >> *Arsène Gschwind* >> Fa. Sapify AG im Auftrag der Universität Basel >> IT Services >> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch >> >> ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 >> <+41%2061%20267%2014%2011> >> > > > -- > > *Arsène Gschwind* > Fa. Sapify AG im Auftrag der Universität Basel > IT Services > Klingelbergstr. 70 | CH-4056 Basel | Switzerland > Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch > > ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 > <+41%2061%20267%2014%2011> > -- *Arsène Gschwind* Fa. Sapify AG im Auftrag der Universität Basel IT Services Klingelbergstr. 70 | CH-4056 Basel | Switzerland Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch ITS-ServiceDesk: support-its(a)unibas.ch | +41 61 267 14 11 <+41%2061%20267%2014%2011>