[ovirt-users] Failed to import the Hosted Engine Storage Domain
Simone Tiraboschi
stirabos at redhat.com
Wed Sep 6 13:32:54 UTC 2017
On Wed, Sep 6, 2017 at 1:22 PM, Arsène Gschwind <arsene.gschwind at unibas.ch>
wrote:
>
>
> On 09/05/2017 09:02 AM, Arsène Gschwind wrote:
>
>
>
> On 09/04/2017 07:59 PM, Simone Tiraboschi wrote:
>
>
> On Mon, Sep 4, 2017 at 7:32 PM, Arsène Gschwind <arsene.gschwind at unibas.ch
> > wrote:
>
>>
>>
>> On 09/04/2017 06:32 PM, Simone Tiraboschi wrote:
>>
>>
>>
>> On Mon, Sep 4, 2017 at 6:24 PM, Arsène Gschwind <
>> arsene.gschwind at unibas.ch> wrote:
>>
>>>
>>>
>>> On 09/04/2017 02:51 PM, Simone Tiraboschi wrote:
>>>
>>>
>>>
>>> On Mon, Sep 4, 2017 at 2:21 PM, Arsène Gschwind <
>>> arsene.gschwind at unibas.ch> wrote:
>>>
>>>>
>>>>
>>>> On 09/04/2017 02:01 PM, Simone Tiraboschi wrote:
>>>>
>>>>
>>>>
>>>> On Mon, Sep 4, 2017 at 1:55 PM, Arsène Gschwind <
>>>> arsene.gschwind at unibas.ch> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 09/04/2017 01:52 PM, Simone Tiraboschi wrote:
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 4, 2017 at 12:23 PM, Arsène Gschwind <
>>>>> arsene.gschwind at unibas.ch> wrote:
>>>>>
>>>>>> Hi Simone,
>>>>>>
>>>>>> On 09/04/2017 11:14 AM, Simone Tiraboschi wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 4, 2017 at 10:56 AM, Arsène Gschwind <
>>>>>> arsene.gschwind at unibas.ch> wrote:
>>>>>>
>>>>>>> Hi Didi,
>>>>>>>
>>>>>>> On 09/04/2017 10:15 AM, Yedidyah Bar David wrote:
>>>>>>>
>>>>>>> On Mon, Sep 4, 2017 at 10:16 AM, Arsène Gschwind<arsene.gschwind at unibas.ch> <arsene.gschwind at unibas.ch> wrote:
>>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> A while ago I had some problem with hosted-engine network which wasn't set
>>>>>>> correctly at deploy time, so I finally decided to redeploy the hosted engine
>>>>>>> in the hope the network will be set correctly this time. I've followed this
>>>>>>> procedure:
>>>>>>>
>>>>>>> Stop all VMs
>>>>>>> Full backup of HE DB and export to safe place
>>>>>>> Cleanup HE storage following https://access.redhat.com/solutions/2121581
>>>>>>> Reboot Hosts
>>>>>>> Re-deploy HE until DB recovery
>>>>>>> Recover DB adding the following param:
>>>>>>> --he-remove-storage-vm Removes the hosted-engine storage
>>>>>>> domain, all its entities and the hosted-engine VM during restore.
>>>>>>> --he-remove-hosts Removes all the hosted-engine hosts
>>>>>>> during restore.
>>>>>>>
>>>>>>> Finalize HE deployment.
>>>>>>>
>>>>>>> Everything did run without errors and I'm able to access Web UI.
>>>>>>>
>>>>>>> But now I don't see my HE VM and its respective Storage Domain, the logs
>>>>>>> says it isn't able to import it. I see all other SD and I'm able to manage
>>>>>>> my VMs as before.
>>>>>>>
>>>>>>> Please find attached engine.log
>>>>>>>
>>>>>>> I think this is your problem:
>>>>>>>
>>>>>>> 2017-09-04 03:26:14,272+02 INFO
>>>>>>> [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand]
>>>>>>> (org.ovirt.thread.pool-6-thread-24) [2383eaa0] There are existing luns
>>>>>>> in the system which are part of VG id
>>>>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'
>>>>>>>
>>>>>>> I don't see a VG with this ID, here the IDs I see on the hosts:
>>>>>>>
>>>>>>> VG #PV #LV #SN Attr VSize
>>>>>>> VFree
>>>>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1 10 0 wz--n- 99.62g
>>>>>>> 14.50g
>>>>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2 70 0 wz--n- 10.00t
>>>>>>> 2.31t
>>>>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2 47 0 wz--n- 5.27t
>>>>>>> 2.50t
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Could you please repeat the command on host adm-kvmh70 ?
>>>>>>
>>>>>> 2017-09-04 09:04:18,163+02 INFO [org.ovirt.engine.core.bll.st
>>>>>> orage.domain.ImportHostedEngineStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34) [247a3718] Running command:
>>>>>> ImportHostedEngineStorageDomainCommand internal: true.
>>>>>> 2017-09-04 09:04:18,189+02 INFO [org.ovirt.engine.core.vdsbro
>>>>>> ker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] START, GetVGInfoVDSCommand(HostName = adm-kvmh70,
>>>>>> GetVGInfoVDSCommandParameters:{runAsync='true',
>>>>>> hostId='acbacabb-6c4a-43fd-a1e2-2d7ff2f6f98b',
>>>>>> VGID='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'}), log id: 6693b98a
>>>>>> 2017-09-04 09:04:18,232+02 INFO [org.ovirt.engine.core.vdsbro
>>>>>> ker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] FINISH, GetVGInfoVDSCommand, return:
>>>>>> [LUNs:{id='repl_HostedEngine', physicalVolumeId='kYN8Jj-FBDw-MhxI-XcoZ-w1zH-eQL8-IRIgzO',
>>>>>> volumeGroupId='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g',
>>>>>> serial='SHITACHI_OPEN-V_50488888', lunMapping='4',
>>>>>> vendorId='HITACHI', productId='OPEN-V', lunConnections='[]',
>>>>>> deviceSize='100', pvSize='0', peCount='797', peAllocatedCount='681',
>>>>>> vendorName='HITACHI', pathsDictionary='[sdf=true, sdu=true, sdk=true,
>>>>>> sdp=true]', pathsCapacity='[sdf=100, sdu=100, sdk=100, sdp=100]',
>>>>>> lunType='FCP', status='null', diskId='null', diskAlias='null',
>>>>>> storageDomainId='6b62cc06-fc44-4c38-af6d-bfd9cbe73246',
>>>>>> storageDomainName='null', discardMaxSize='268435456',
>>>>>> discardZeroesData='true'}], log id: 6693b98a
>>>>>> 2017-09-04 09:04:18,245+02 INFO [org.ovirt.engine.core.bll.st
>>>>>> orage.domain.AddExistingBlockStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] There are existing
>>>>>> luns in the system which are part of VG id 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL
>>>>>> 2z-wbTU8g'
>>>>>> 2017-09-04 09:04:18,245+02 WARN [org.ovirt.engine.core.bll.st
>>>>>> orage.domain.AddExistingBlockStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] Validation of action
>>>>>> 'AddExistingBlockStorageDomain' failed for user SYSTEM. Reasons:
>>>>>> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAIL
>>>>>> ED_IMPORT_STORAGE_DOMAIN_EXTERNAL_LUN_DISK_EXIST
>>>>>>
>>>>>> I don't know which command you are talking about, I didn't run any
>>>>>> command since it tries to import the SD automatically.
>>>>>>
>>>>>
>>>>> Sorry, can you please run vgdisplay on adm-kvmh70 ?
>>>>>
>>>>> No problem! here the result:
>>>>>
>>>>> [root at adm-kvmh70 ~]# vgs
>>>>> VG #PV #LV #SN Attr VSize
>>>>> VFree
>>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1 10 0 wz--n- 99.62g
>>>>> 14.50g
>>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2 70 0 wz--n- 10.00t
>>>>> 2.31t
>>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2 47 0 wz--n- 5.27t
>>>>> 2.50t
>>>>> vg_adm-kvmh70 1 3 0 wz--n- 277.90g
>>>>> 218.68g
>>>>>
>>>>>
>>>> OK, and
>>>> vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>>>
>>>> [root at adm-kvmh70 ~]# vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>>> --- Volume group ---
>>>> VG Name 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>>> System ID
>>>> Format lvm2
>>>> Metadata Areas 2
>>>> Metadata Sequence No 23
>>>> VG Access read/write
>>>> VG Status resizable
>>>> MAX LV 0
>>>> Cur LV 10
>>>> Open LV 3
>>>> Max PV 0
>>>> Cur PV 1
>>>> Act PV 1
>>>> VG Size 99.62 GiB
>>>> PE Size 128.00 MiB
>>>> Total PE 797
>>>> Alloc PE / Size 681 / 85.12 GiB
>>>> Free PE / Size 116 / 14.50 GiB
>>>> VG UUID vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g
>>>>
>>>
>>> OK, vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g is the uuid of the VG that
>>> engine says that it includes the LUN used by the hosted-engine SD as
>>> detected by the engine.
>>> Its name is 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 which is also the uuid
>>> of that storage domain.
>>>
>>> Now we need to understand what's there.
>>>
>>> Could you please check if an SD with uuid 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>> is visible in the engine?
>>>
>>> The visible SD in the engine
>>>
>>> [oVirt shell (connected)]# list storagedomains
>>>
>>> id : 072fbaa1-08f3-4a40-9f34-a5ca22dd1d74
>>> name : ovirt-image-repository
>>> description: Public Glance repository for oVirt
>>>
>>> id : b2e30961-7cff-4cca-83d6-bee3a4f890ee
>>> name : xxxxxxxxxxx
>>>
>>> id : b0414c06-d984-4001-a998-fd9a2e79fb83
>>> name : xxxxxxxxxxx
>>> description: xxxxxxxxxxx
>>>
>>> id : 71fa1c61-4a68-469f-9a56-835190d32075
>>> name : xxxxxxxxxxx
>>>
>>> id : 9817e64b-fdd9-4d4c-91ee-f3482bb0cc56
>>> name : xxxxxxxxxxxx
>>> description: xxxxxxxxxxxx
>>>
>>> Is adm-kvmh70 involved in hosted-engine? did you redeployed it?
>>>
>>> Yes adm-kvmh70 should be a host hosting HE and yes as written before I
>>> did redeploy HE because of a network interface problem. I did cleanup the
>>> lun using the RedHat guide, redeploy HE and restore the DB using the
>>> mentioned parameter to remove HE information.
>>>
>>
>> Could you please set adm-kvmh70 to maintenance mode from the engine and
>> then reactivate it or better reboot it while in maintenance mode?
>> I suspect that 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is just a left over
>> of the previous hosted-engine storage domain kept open on the host.
>>
>> I did reboot adm-kvmh70 while in maintenance mode and also reboot the
>> engine but it didn't change anything. The import failure is still there...
>> I've noticed a ovirt-ha-agent error:
>>
>> ovirt-ha-agent ovirt_hosted_engine_ha.agent.h
>> osted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE
>> volume, falling back to initial vm.conf. Please ensure you already added
>> your first data domain for regular VMs
>>
>> Could that be related?
>>
>
> This is just a side effects of that: the OVF_STORE disk is going to be
> created by the engine once it correctly imported the hosted-engine storage
> domain.
>
> Could you please share again your up-to-date engine.log?
>
>
>> Thanks
>>
>> Please find the latest engine log attached.
>
> Thanks
>
> When analyzing the log I do see some warning about existing LUNs which are
> not part of the VG, does this means the LUNs are existing in the DB?
>
That LUN is reported by VDSM to the engine.
Could you please share your /etc/ovirt-hosted-engine/hosted-engine.conf?
> If yes would it be possible to remove them from the engine DB?
>
> Thanks for your help,
> Arsene
>
>
>
>>
>>>
>>>
>>> In the past did you tried to manually import the hosted-engine SD or
>>> something similar?
>>>
>>> No, before the "redeployment" the HE SD was auto imported successfully.
>>>
>>> Thanks a lot
>>>
>>>
>>>
>>>
>>> thanks
>>>
>>>
>>>
>>>
>>>>
>>>> Thanks
>>>>
>>>> ?
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Arsene
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks for any help to resolve that issue.
>>>>>>>
>>>>>>> I guess you can try to remove this disk/lun from the engine and let it retry.
>>>>>>>
>>>>>>> Could let me know how to remove that lun from the engine?
>>>>>>>
>>>>>>> If the only disk is of the hosted-engine, I guess it should have been
>>>>>>> removed by '--he-remove-storage-vm' - if so, please open a bug
>>>>>>> describing your flow in detail. Thanks.
>>>>>>>
>>>>>>> It seems that this option didn't remove the he storage information
>>>>>>> and that this VG is still the old one.
>>>>>>>
>>>>>>> Many thanks for your help
>>>>>>> Rgds,
>>>>>>> Arsene
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>>
>>>>>>> Arsène
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Arsène Gschwind
>>>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>>>> IT Services
>>>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>>>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11 <+41%2061%20267%2014%2011>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> *Arsène Gschwind*
>>>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>>>> IT Services
>>>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>>>>>
>>>>>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>>>>>>> <+41%2061%20267%2014%2011>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list
>>>>>>> Users at ovirt.org
>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Arsène Gschwind*
>>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>>> IT Services
>>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>>>>
>>>>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>>>>>> <+41%2061%20267%2014%2011>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *Arsène Gschwind*
>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>> IT Services
>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>>>
>>>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>>>>> <+41%2061%20267%2014%2011>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> *Arsène Gschwind*
>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>> IT Services
>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>>
>>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>>>> <+41%2061%20267%2014%2011>
>>>>
>>>
>>>
>>> --
>>>
>>> *Arsène Gschwind*
>>> Fa. Sapify AG im Auftrag der Universität Basel
>>> IT Services
>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>>
>>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>>> <+41%2061%20267%2014%2011>
>>>
>>
>>
>> --
>>
>> *Arsène Gschwind*
>> Fa. Sapify AG im Auftrag der Universität Basel
>> IT Services
>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
>>
>> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
>> <+41%2061%20267%2014%2011>
>>
>
>
> --
>
> *Arsène Gschwind*
> Fa. Sapify AG im Auftrag der Universität Basel
> IT Services
> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
> <+41%2061%20267%2014%2011>
>
>
> _______________________________________________
> Users mailing listUsers at ovirt.orghttp://lists.ovirt.org/mailman/listinfo/users
>
>
> --
>
> *Arsène Gschwind*
> Fa. Sapify AG im Auftrag der Universität Basel
> IT Services
> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
> Tel. +41 79 449 25 63 <+41%2079%20449%2025%2063> | http://its.unibas.ch
> ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
> <+41%2061%20267%2014%2011>
>
> _______________________________________________
> Users mailing list
> Users at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170906/400c097c/attachment.html>
More information about the Users
mailing list