Re: [ovirt-users] Failed to import the Hosted Engine Storage Domain

On Mon, Sep 4, 2017 at 7:32 PM, Arsène Gschwind <arsene.gschwind(a)unibas.ch <mailto:arsene.gschwind@unibas.ch>> wrote: On 09/04/2017 06:32 PM, Simone Tiraboschi wrote: > > > On Mon, Sep 4, 2017 at 6:24 PM, Arsène Gschwind > <arsene.gschwind(a)unibas.ch <mailto:arsene.gschwind@unibas.ch>> wrote: > > > > On 09/04/2017 02:51 PM, Simone Tiraboschi wrote: >> >> >> On Mon, Sep 4, 2017 at 2:21 PM, Arsène Gschwind >> <arsene.gschwind(a)unibas.ch >> <mailto:arsene.gschwind@unibas.ch>> wrote: >> >> >> >> On 09/04/2017 02:01 PM, Simone Tiraboschi wrote: >>> >>> >>> On Mon, Sep 4, 2017 at 1:55 PM, Arsène Gschwind >>> <arsene.gschwind(a)unibas.ch >>> <mailto:arsene.gschwind@unibas.ch>> wrote: >>> >>> >>> >>> On 09/04/2017 01:52 PM, Simone Tiraboschi wrote: >>>> >>>> >>>> On Mon, Sep 4, 2017 at 12:23 PM, Arsène Gschwind >>>> <arsene.gschwind(a)unibas.ch >>>> <mailto:arsene.gschwind@unibas.ch>> wrote: >>>> >>>> Hi Simone, >>>> >>>> >>>> On 09/04/2017 11:14 AM, Simone Tiraboschi wrote: >>>>> >>>>> >>>>> On Mon, Sep 4, 2017 at 10:56 AM, Arsène >>>>> Gschwind <arsene.gschwind(a)unibas.ch >>>>> <mailto:arsene.gschwind@unibas.ch>> wrote: >>>>> >>>>> Hi Didi, >>>>> >>>>> >>>>> On 09/04/2017 10:15 AM, Yedidyah Bar >>>>> David wrote: >>>>>> On Mon, Sep 4, 2017 at 10:16 AM, Arsène Gschwind >>>>>> <arsene.gschwind(a)unibas.ch&gt; >>>>>> <mailto:arsene.gschwind@unibas.ch> wrote: >>>>>>> Hi all, >>>>>>> >>>>>>> A while ago I had some problem with hosted-engine network which wasn't set >>>>>>> correctly at deploy time, so I finally decided to redeploy the hosted engine >>>>>>> in the hope the network will be set correctly this time. I've followed this >>>>>>> procedure: >>>>>>> >>>>>>> Stop all VMs >>>>>>> Full backup of HE DB and export to safe place >>>>>>> Cleanup HE storage followinghttps://access.redhat.com/solutions/2121581 >>>>>>> <https://access.redhat.com/solutions/2121581> >>>>>>> Reboot Hosts >>>>>>> Re-deploy HE until DB recovery >>>>>>> Recover DB adding the following param: >>>>>>> --he-remove-storage-vm Removes the hosted-engine storage >>>>>>> domain, all its entities and the hosted-engine VM during restore. >>>>>>> --he-remove-hosts Removes all the hosted-engine hosts >>>>>>> during restore. >>>>>>> >>>>>>> Finalize HE deployment. >>>>>>> >>>>>>> Everything did run without errors and I'm able to access Web UI. >>>>>>> >>>>>>> But now I don't see my HE VM and its respective Storage Domain, the logs >>>>>>> says it isn't able to import it. I see all other SD and I'm able to manage >>>>>>> my VMs as before. >>>>>>> >>>>>>> Please find attached engine.log >>>>>> I think this is your problem: >>>>>> >>>>>> 2017-09-04 03:26:14,272+02 INFO >>>>>> [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand] >>>>>> (org.ovirt.thread.pool-6-thread-24) [2383eaa0] There are existing luns >>>>>> in the system which are part of VG id >>>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g' >>>>> I don't see a VG with this ID, here the >>>>> IDs I see on the hosts: >>>>> >>>>> VG #PV #LV #SN Attr   VSize VFree >>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1  >>>>> 10   0 wz--n- 99.62g 14.50g >>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2  >>>>> 70   0 wz--n- 10.00t 2.31t >>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2  >>>>> 47   0 wz--n-  5.27t 2.50t >>>>> >>>>> >>>>> >>>>> Could you please repeat the command on host >>>>> adm-kvmh70 ? >>>>> >>>>> 2017-09-04 09:04:18,163+02 INFO >>>>>  [org.ovirt.engine.core.bll.st >>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.ImportHostedEngineSto...] >>>>> (org.ovirt.thread.pool-6-thread-34) >>>>> [247a3718] Running command: >>>>> ImportHostedEngineStorageDomainCommand >>>>> internal: true. >>>>> 2017-09-04 09:04:18,189+02 INFO >>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-34) >>>>> [7d2e6cb2] START, >>>>> GetVGInfoVDSCommand(HostName = adm-kvmh70, >>>>> GetVGInfoVDSCommandParameters:{runAsync='true', >>>>> hostId='acbacabb-6c4a-43fd-a1e2-2d7ff2f6f98b', >>>>> VGID='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'}), >>>>> log id: 6693b98a >>>>> 2017-09-04 09:04:18,232+02 INFO >>>>>  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] >>>>> (org.ovirt.thread.pool-6-thread-34) >>>>> [7d2e6cb2] FINISH, GetVGInfoVDSCommand, >>>>> return: [LUNs:{id='repl_HostedEngine', >>>>> physicalVolumeId='kYN8Jj-FBDw-MhxI-XcoZ-w1zH-eQL8-IRIgzO', >>>>> volumeGroupId='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g', >>>>> serial='SHITACHI_OPEN-V_50488888', >>>>> lunMapping='4', vendorId='HITACHI', >>>>> productId='OPEN-V', lunConnections='[]', >>>>> deviceSize='100', pvSize='0', peCount='797', >>>>> peAllocatedCount='681', vendorName='HITACHI', >>>>> pathsDictionary='[sdf=true, sdu=true, >>>>> sdk=true, sdp=true]', >>>>> pathsCapacity='[sdf=100, sdu=100, sdk=100, >>>>> sdp=100]', lunType='FCP', status='null', >>>>> diskId='null', diskAlias='null', >>>>> storageDomainId='6b62cc06-fc44-4c38-af6d-bfd9cbe73246', >>>>> storageDomainName='null', >>>>> discardMaxSize='268435456', >>>>> discardZeroesData='true'}], log id: 6693b98a >>>>> 2017-09-04 09:04:18,245+02 INFO >>>>>  [org.ovirt.engine.core.bll.st >>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.AddExistingBlockStora...] >>>>> (org.ovirt.thread.pool-6-thread-34) >>>>> [7d2e6cb2] There are existing luns in the >>>>> system which are part of VG id >>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g' >>>>> 2017-09-04 09:04:18,245+02 WARN >>>>>  [org.ovirt.engine.core.bll.st >>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.AddExistingBlockStora...] >>>>> (org.ovirt.thread.pool-6-thread-34) >>>>> [7d2e6cb2] Validation of action >>>>> 'AddExistingBlockStorageDomain' failed for >>>>> user SYSTEM. Reasons: >>>>> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_IMPORT_STORAGE_DOMAIN_EXTERNAL_LUN_DISK_EXIST >>>> I don't know which command you are talking >>>> about, I didn't run any command since it tries >>>> to import the SD automatically. >>>> >>>> >>>> Sorry, can you please run vgdisplay on adm-kvmh70 ? >>> No problem! here the result: >>> >>> [root@adm-kvmh70 ~]# vgs >>> VG #PV #LV #SN Attr   VSize VFree >>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1  10   0 >>> wz--n- 99.62g  14.50g >>> b0414c06-d984-4001-a998-fd9a2e79fb83 2  70   0 >>> wz--n- 10.00t   2.31t >>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2  47   0 >>> wz--n- 5.27t   2.50t >>> vg_adm-kvmh70          1   3   0 wz--n- 277.90g >>> 218.68g >>> >>> >>> OK, and >>>  vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >> [root@adm-kvmh70 ~]# vgdisplay >> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >>   --- Volume group --- >>   VG Name 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 >>   System ID >> Format lvm2 >>   Metadata Areas        2 >>   Metadata Sequence No  23 >>   VG Access read/write >>   VG Status resizable >>   MAX LV                0 >>   Cur LV                10 >>   Open LV               3 >>   Max PV                0 >>   Cur PV                1 >>   Act PV                1 >>   VG Size 99.62 GiB >>   PE Size 128.00 MiB >>   Total PE              797 >>   Alloc PE / Size       681 / 85.12 GiB >>   Free  PE / Size       116 / 14.50 GiB >>   VG UUID vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g >> >> >> OK, vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g is the uuid of >> the VG that engine says that it includes the LUN used by the >> hosted-engine SD as detected by the engine. >> Its name is 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 which is >> also the uuid of that storage domain. >> >> Now we need to understand what's there. >> >> Could you please check if an SD with uuid >> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is visible in the engine? > The visible SD in the engine > > [oVirt shell (connected)]# list storagedomains > > id         : 072fbaa1-08f3-4a40-9f34-a5ca22dd1d74 > name       : ovirt-image-repository > description: Public Glance repository for oVirt > > id         : b2e30961-7cff-4cca-83d6-bee3a4f890ee > name       : xxxxxxxxxxx > > id         : b0414c06-d984-4001-a998-fd9a2e79fb83 > name       : xxxxxxxxxxx > description: xxxxxxxxxxx > > id         : 71fa1c61-4a68-469f-9a56-835190d32075 > name       : xxxxxxxxxxx > > id         : 9817e64b-fdd9-4d4c-91ee-f3482bb0cc56 > name       : xxxxxxxxxxxx > description: xxxxxxxxxxxx > >> Is adm-kvmh70 involved in hosted-engine? did you redeployed it? > Yes adm-kvmh70 should be a host hosting HE and yes as written > before I did redeploy HE because of a network interface > problem. I did cleanup the lun using the RedHat guide, > redeploy HE and restore the DB using the mentioned parameter > to remove HE information. > > > Could you please set adm-kvmh70 to maintenance mode from the > engine and then reactivate it or better reboot it while in > maintenance mode? > I suspect that 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is just a > left over of the previous hosted-engine storage domain kept open > on the host. I did reboot adm-kvmh70 while in maintenance mode and also reboot the engine but it didn't change anything. The import failure is still there... I've noticed a ovirt-ha-agent error: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs Could that be related? This is just a side effects of that: the OVF_STORE disk is going to be created by the engine once it correctly imported the hosted-engine storage domain. Could you please share again your up-to-date engine.log? Thanks

> >> >> In the past did you tried to manually import the >> hosted-engine SD or something similar? > No, before the "redeployment" the HE SD was auto imported > successfully. > > Thanks a lot > > > >> >> thanks > >> >> Thanks >> >>> ? >>> >>> >>>> >>>> Thanks, >>>> Arsene >>>> >>>>> >>>>> >>>>>>> Thanks for any help to resolve that issue. >>>>>> I guess you can try to remove this disk/lun from the engine and let it retry. >>>>> Could let me know how to remove that lun >>>>> from the engine? >>>>>> If the only disk is of the hosted-engine, I guess it should have been >>>>>> removed by '--he-remove-storage-vm' - if so, please open a bug >>>>>> describing your flow in detail. Thanks. >>>>> It seems that this option didn't remove >>>>> the he storage information and that this >>>>> VG is still the old one. >>>>> >>>>> Many thanks for your help >>>>> Rgds, >>>>> Arsene >>>>>> Best, >>>>>> >>>>>>> Arsène >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Arsène Gschwind >>>>>>> Fa. Sapify AG im Auftrag der Universität Basel >>>>>>> IT Services >>>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland >>>>>>> Tel.+41 79 449 25 63 >>>>>>> <tel:+41%2079%20449%2025%2063> |http://its.unibas.ch >>>>>>> ITS-ServiceDesk:support-its@unibas.ch >>>>>>> <mailto:support-its@unibas.ch> |+41 61 267 14 11 >>>>>>> <tel:+41%2061%20267%2014%2011> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Users mailing list >>>>>>> Users(a)ovirt.org <mailto:Users@ovirt.org> >>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>> <http://lists.ovirt.org/mailman/listinfo/users> >>>>>>> >>>>> >>>>> -- >>>>> >>>>> *Arsène Gschwind* >>>>> Fa. Sapify AG im Auftrag der Universität >>>>> Basel >>>>> IT Services >>>>> Klingelbergstr. 70 |  CH-4056 Basel  |  >>>>> Switzerland >>>>> Tel. +41 79 449 25 63 >>>>> <tel:+41%2079%20449%2025%2063>  | >>>>> http://its.unibas.ch <http://its.unibas.ch/> >>>>> ITS-ServiceDesk: support-its(a)unibas.ch >>>>> <mailto:support-its@unibas.ch> | +41 61 >>>>> 267 14 11 <tel:+41%2061%20267%2014%2011> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org <mailto:Users@ovirt.org> >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> <http://lists.ovirt.org/mailman/listinfo/users> >>>>> >>>>> >>>> >>>> -- >>>> >>>> *Arsène Gschwind* >>>> Fa. Sapify AG im Auftrag der Universität Basel >>>> IT Services >>>> Klingelbergstr. 70 |  CH-4056 Basel  |  >>>> Switzerland >>>> Tel. +41 79 449 25 63 >>>> <tel:+41%2079%20449%2025%2063>  | >>>> http://its.unibas.ch <http://its.unibas.ch/> >>>> ITS-ServiceDesk: support-its(a)unibas.ch >>>> <mailto:support-its@unibas.ch> | +41 61 267 14 >>>> 11 <tel:+41%2061%20267%2014%2011> >>>> >>>> >>> >>> -- >>> >>> *Arsène Gschwind* >>> Fa. Sapify AG im Auftrag der Universität Basel >>> IT Services >>> Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland >>> Tel. +41 79 449 25 63 >>> <tel:+41%2079%20449%2025%2063>  | >>> http://its.unibas.ch <http://its.unibas.ch/> >>> ITS-ServiceDesk: support-its(a)unibas.ch >>> <mailto:support-its@unibas.ch> | +41 61 267 14 11 >>> <tel:+41%2061%20267%2014%2011> >>> >>> >> >> -- >> >> *Arsène Gschwind* >> Fa. Sapify AG im Auftrag der Universität Basel >> IT Services >> Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland >> Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063>  | >> http://its.unibas.ch <http://its.unibas.ch/> >> ITS-ServiceDesk: support-its(a)unibas.ch >> <mailto:support-its@unibas.ch> | +41 61 267 14 11 >> <tel:+41%2061%20267%2014%2011> >> >> > > -- > > *Arsène Gschwind* > Fa. Sapify AG im Auftrag der Universität Basel > IT Services > Klingelbergstr. 70 | CH-4056 Basel  | Switzerland > Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063>  | > http://its.unibas.ch <http://its.unibas.ch/> > ITS-ServiceDesk: support-its(a)unibas.ch > <mailto:support-its@unibas.ch> | +41 61 267 14 11 > <tel:+41%2061%20267%2014%2011> > > -- *Arsène Gschwind* Fa. Sapify AG im Auftrag der Universität Basel IT Services Klingelbergstr. 70 |  CH-4056 Basel  | Switzerland Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063>  | http://its.unibas.ch <http://its.unibas.ch/> ITS-ServiceDesk: support-its(a)unibas.ch <mailto:support-its@unibas.ch> | +41 61 267 14 11 <tel:+41%2061%20267%2014%2011>