[ovirt-users] Failed to import the Hosted Engine Storage Domain
Arsène Gschwind
arsene.gschwind at unibas.ch
Tue Sep 5 07:02:57 UTC 2017
On 09/04/2017 07:59 PM, Simone Tiraboschi wrote:
>
> On Mon, Sep 4, 2017 at 7:32 PM, Arsène Gschwind
> <arsene.gschwind at unibas.ch <mailto:arsene.gschwind at unibas.ch>> wrote:
>
>
>
> On 09/04/2017 06:32 PM, Simone Tiraboschi wrote:
>>
>>
>> On Mon, Sep 4, 2017 at 6:24 PM, Arsène Gschwind
>> <arsene.gschwind at unibas.ch <mailto:arsene.gschwind at unibas.ch>> wrote:
>>
>>
>>
>> On 09/04/2017 02:51 PM, Simone Tiraboschi wrote:
>>>
>>>
>>> On Mon, Sep 4, 2017 at 2:21 PM, Arsène Gschwind
>>> <arsene.gschwind at unibas.ch
>>> <mailto:arsene.gschwind at unibas.ch>> wrote:
>>>
>>>
>>>
>>> On 09/04/2017 02:01 PM, Simone Tiraboschi wrote:
>>>>
>>>>
>>>> On Mon, Sep 4, 2017 at 1:55 PM, Arsène Gschwind
>>>> <arsene.gschwind at unibas.ch
>>>> <mailto:arsene.gschwind at unibas.ch>> wrote:
>>>>
>>>>
>>>>
>>>> On 09/04/2017 01:52 PM, Simone Tiraboschi wrote:
>>>>>
>>>>>
>>>>> On Mon, Sep 4, 2017 at 12:23 PM, Arsène Gschwind
>>>>> <arsene.gschwind at unibas.ch
>>>>> <mailto:arsene.gschwind at unibas.ch>> wrote:
>>>>>
>>>>> Hi Simone,
>>>>>
>>>>>
>>>>> On 09/04/2017 11:14 AM, Simone Tiraboschi wrote:
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 4, 2017 at 10:56 AM, Arsène
>>>>>> Gschwind <arsene.gschwind at unibas.ch
>>>>>> <mailto:arsene.gschwind at unibas.ch>> wrote:
>>>>>>
>>>>>> Hi Didi,
>>>>>>
>>>>>>
>>>>>> On 09/04/2017 10:15 AM, Yedidyah Bar
>>>>>> David wrote:
>>>>>>> On Mon, Sep 4, 2017 at 10:16 AM, Arsène Gschwind
>>>>>>> <arsene.gschwind at unibas.ch>
>>>>>>> <mailto:arsene.gschwind at unibas.ch> wrote:
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> A while ago I had some problem with hosted-engine network which wasn't set
>>>>>>>> correctly at deploy time, so I finally decided to redeploy the hosted engine
>>>>>>>> in the hope the network will be set correctly this time. I've followed this
>>>>>>>> procedure:
>>>>>>>>
>>>>>>>> Stop all VMs
>>>>>>>> Full backup of HE DB and export to safe place
>>>>>>>> Cleanup HE storage followinghttps://access.redhat.com/solutions/2121581
>>>>>>>> <https://access.redhat.com/solutions/2121581>
>>>>>>>> Reboot Hosts
>>>>>>>> Re-deploy HE until DB recovery
>>>>>>>> Recover DB adding the following param:
>>>>>>>> --he-remove-storage-vm Removes the hosted-engine storage
>>>>>>>> domain, all its entities and the hosted-engine VM during restore.
>>>>>>>> --he-remove-hosts Removes all the hosted-engine hosts
>>>>>>>> during restore.
>>>>>>>>
>>>>>>>> Finalize HE deployment.
>>>>>>>>
>>>>>>>> Everything did run without errors and I'm able to access Web UI.
>>>>>>>>
>>>>>>>> But now I don't see my HE VM and its respective Storage Domain, the logs
>>>>>>>> says it isn't able to import it. I see all other SD and I'm able to manage
>>>>>>>> my VMs as before.
>>>>>>>>
>>>>>>>> Please find attached engine.log
>>>>>>> I think this is your problem:
>>>>>>>
>>>>>>> 2017-09-04 03:26:14,272+02 INFO
>>>>>>> [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand]
>>>>>>> (org.ovirt.thread.pool-6-thread-24) [2383eaa0] There are existing luns
>>>>>>> in the system which are part of VG id
>>>>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'
>>>>>> I don't see a VG with this ID, here the
>>>>>> IDs I see on the hosts:
>>>>>>
>>>>>> VG #PV #LV #SN Attr VSize VFree
>>>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1
>>>>>> 10 0 wz--n- 99.62g 14.50g
>>>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2
>>>>>> 70 0 wz--n- 10.00t 2.31t
>>>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2
>>>>>> 47 0 wz--n- 5.27t 2.50t
>>>>>>
>>>>>>
>>>>>>
>>>>>> Could you please repeat the command on host
>>>>>> adm-kvmh70 ?
>>>>>>
>>>>>> 2017-09-04 09:04:18,163+02 INFO
>>>>>> [org.ovirt.engine.core.bll.st
>>>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.ImportHostedEngineStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34)
>>>>>> [247a3718] Running command:
>>>>>> ImportHostedEngineStorageDomainCommand
>>>>>> internal: true.
>>>>>> 2017-09-04 09:04:18,189+02 INFO
>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] START,
>>>>>> GetVGInfoVDSCommand(HostName = adm-kvmh70,
>>>>>> GetVGInfoVDSCommandParameters:{runAsync='true',
>>>>>> hostId='acbacabb-6c4a-43fd-a1e2-2d7ff2f6f98b',
>>>>>> VGID='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'}),
>>>>>> log id: 6693b98a
>>>>>> 2017-09-04 09:04:18,232+02 INFO
>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] FINISH, GetVGInfoVDSCommand,
>>>>>> return: [LUNs:{id='repl_HostedEngine',
>>>>>> physicalVolumeId='kYN8Jj-FBDw-MhxI-XcoZ-w1zH-eQL8-IRIgzO',
>>>>>> volumeGroupId='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g',
>>>>>> serial='SHITACHI_OPEN-V_50488888',
>>>>>> lunMapping='4', vendorId='HITACHI',
>>>>>> productId='OPEN-V', lunConnections='[]',
>>>>>> deviceSize='100', pvSize='0', peCount='797',
>>>>>> peAllocatedCount='681', vendorName='HITACHI',
>>>>>> pathsDictionary='[sdf=true, sdu=true,
>>>>>> sdk=true, sdp=true]',
>>>>>> pathsCapacity='[sdf=100, sdu=100, sdk=100,
>>>>>> sdp=100]', lunType='FCP', status='null',
>>>>>> diskId='null', diskAlias='null',
>>>>>> storageDomainId='6b62cc06-fc44-4c38-af6d-bfd9cbe73246',
>>>>>> storageDomainName='null',
>>>>>> discardMaxSize='268435456',
>>>>>> discardZeroesData='true'}], log id: 6693b98a
>>>>>> 2017-09-04 09:04:18,245+02 INFO
>>>>>> [org.ovirt.engine.core.bll.st
>>>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.AddExistingBlockStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] There are existing luns in the
>>>>>> system which are part of VG id
>>>>>> 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'
>>>>>> 2017-09-04 09:04:18,245+02 WARN
>>>>>> [org.ovirt.engine.core.bll.st
>>>>>> <http://org.ovirt.engine.core.bll.st>orage.domain.AddExistingBlockStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-6-thread-34)
>>>>>> [7d2e6cb2] Validation of action
>>>>>> 'AddExistingBlockStorageDomain' failed for
>>>>>> user SYSTEM. Reasons:
>>>>>> VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_IMPORT_STORAGE_DOMAIN_EXTERNAL_LUN_DISK_EXIST
>>>>> I don't know which command you are talking
>>>>> about, I didn't run any command since it tries
>>>>> to import the SD automatically.
>>>>>
>>>>>
>>>>> Sorry, can you please run vgdisplay on adm-kvmh70 ?
>>>> No problem! here the result:
>>>>
>>>> [root at adm-kvmh70 ~]# vgs
>>>> VG #PV #LV #SN Attr VSize VFree
>>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 1 10 0
>>>> wz--n- 99.62g 14.50g
>>>> b0414c06-d984-4001-a998-fd9a2e79fb83 2 70 0
>>>> wz--n- 10.00t 2.31t
>>>> b2e30961-7cff-4cca-83d6-bee3a4f890ee 2 47 0
>>>> wz--n- 5.27t 2.50t
>>>> vg_adm-kvmh70 1 3 0 wz--n- 277.90g
>>>> 218.68g
>>>>
>>>>
>>>> OK, and
>>>> vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>> [root at adm-kvmh70 ~]# vgdisplay
>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>> --- Volume group ---
>>> VG Name 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
>>> System ID
>>> Format lvm2
>>> Metadata Areas 2
>>> Metadata Sequence No 23
>>> VG Access read/write
>>> VG Status resizable
>>> MAX LV 0
>>> Cur LV 10
>>> Open LV 3
>>> Max PV 0
>>> Cur PV 1
>>> Act PV 1
>>> VG Size 99.62 GiB
>>> PE Size 128.00 MiB
>>> Total PE 797
>>> Alloc PE / Size 681 / 85.12 GiB
>>> Free PE / Size 116 / 14.50 GiB
>>> VG UUID vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g
>>>
>>>
>>> OK, vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g is the uuid of
>>> the VG that engine says that it includes the LUN used by the
>>> hosted-engine SD as detected by the engine.
>>> Its name is 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 which is
>>> also the uuid of that storage domain.
>>>
>>> Now we need to understand what's there.
>>>
>>> Could you please check if an SD with uuid
>>> 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is visible in the engine?
>> The visible SD in the engine
>>
>> [oVirt shell (connected)]# list storagedomains
>>
>> id : 072fbaa1-08f3-4a40-9f34-a5ca22dd1d74
>> name : ovirt-image-repository
>> description: Public Glance repository for oVirt
>>
>> id : b2e30961-7cff-4cca-83d6-bee3a4f890ee
>> name : xxxxxxxxxxx
>>
>> id : b0414c06-d984-4001-a998-fd9a2e79fb83
>> name : xxxxxxxxxxx
>> description: xxxxxxxxxxx
>>
>> id : 71fa1c61-4a68-469f-9a56-835190d32075
>> name : xxxxxxxxxxx
>>
>> id : 9817e64b-fdd9-4d4c-91ee-f3482bb0cc56
>> name : xxxxxxxxxxxx
>> description: xxxxxxxxxxxx
>>
>>> Is adm-kvmh70 involved in hosted-engine? did you redeployed it?
>> Yes adm-kvmh70 should be a host hosting HE and yes as written
>> before I did redeploy HE because of a network interface
>> problem. I did cleanup the lun using the RedHat guide,
>> redeploy HE and restore the DB using the mentioned parameter
>> to remove HE information.
>>
>>
>> Could you please set adm-kvmh70 to maintenance mode from the
>> engine and then reactivate it or better reboot it while in
>> maintenance mode?
>> I suspect that 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is just a
>> left over of the previous hosted-engine storage domain kept open
>> on the host.
> I did reboot adm-kvmh70 while in maintenance mode and also reboot
> the engine but it didn't change anything. The import failure is
> still there...
> I've noticed a ovirt-ha-agent error:
>
> ovirt-ha-agent
> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config
> ERROR Unable to identify the OVF_STORE volume, falling back to
> initial vm.conf. Please ensure you already added your first data
> domain for regular VMs
>
> Could that be related?
>
>
> This is just a side effects of that: the OVF_STORE disk is going to be
> created by the engine once it correctly imported the hosted-engine
> storage domain.
>
> Could you please share again your up-to-date engine.log?
>
> Thanks
>
Please find the latest engine log attached.
Thanks
>
>>
>>>
>>> In the past did you tried to manually import the
>>> hosted-engine SD or something similar?
>> No, before the "redeployment" the HE SD was auto imported
>> successfully.
>>
>> Thanks a lot
>>
>>
>>
>>>
>>> thanks
>>
>>>
>>> Thanks
>>>
>>>> ?
>>>>
>>>>
>>>>>
>>>>> Thanks,
>>>>> Arsene
>>>>>
>>>>>>
>>>>>>
>>>>>>>> Thanks for any help to resolve that issue.
>>>>>>> I guess you can try to remove this disk/lun from the engine and let it retry.
>>>>>> Could let me know how to remove that lun
>>>>>> from the engine?
>>>>>>> If the only disk is of the hosted-engine, I guess it should have been
>>>>>>> removed by '--he-remove-storage-vm' - if so, please open a bug
>>>>>>> describing your flow in detail. Thanks.
>>>>>> It seems that this option didn't remove
>>>>>> the he storage information and that this
>>>>>> VG is still the old one.
>>>>>>
>>>>>> Many thanks for your help
>>>>>> Rgds,
>>>>>> Arsene
>>>>>>> Best,
>>>>>>>
>>>>>>>> Arsène
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Arsène Gschwind
>>>>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>>>>> IT Services
>>>>>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>>>>>> Tel.+41 79 449 25 63
>>>>>>>> <tel:+41%2079%20449%2025%2063> |http://its.unibas.ch
>>>>>>>> ITS-ServiceDesk:support-its at unibas.ch
>>>>>>>> <mailto:support-its at unibas.ch> |+41 61 267 14 11
>>>>>>>> <tel:+41%2061%20267%2014%2011>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list
>>>>>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>>>> <http://lists.ovirt.org/mailman/listinfo/users>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> *Arsène Gschwind*
>>>>>> Fa. Sapify AG im Auftrag der Universität
>>>>>> Basel
>>>>>> IT Services
>>>>>> Klingelbergstr. 70 | CH-4056 Basel |
>>>>>> Switzerland
>>>>>> Tel. +41 79 449 25 63
>>>>>> <tel:+41%2079%20449%2025%2063> |
>>>>>> http://its.unibas.ch <http://its.unibas.ch/>
>>>>>> ITS-ServiceDesk: support-its at unibas.ch
>>>>>> <mailto:support-its at unibas.ch> | +41 61
>>>>>> 267 14 11 <tel:+41%2061%20267%2014%2011>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list
>>>>>> Users at ovirt.org <mailto:Users at ovirt.org>
>>>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>>> <http://lists.ovirt.org/mailman/listinfo/users>
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> *Arsène Gschwind*
>>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>>> IT Services
>>>>> Klingelbergstr. 70 | CH-4056 Basel |
>>>>> Switzerland
>>>>> Tel. +41 79 449 25 63
>>>>> <tel:+41%2079%20449%2025%2063> |
>>>>> http://its.unibas.ch <http://its.unibas.ch/>
>>>>> ITS-ServiceDesk: support-its at unibas.ch
>>>>> <mailto:support-its at unibas.ch> | +41 61 267 14
>>>>> 11 <tel:+41%2061%20267%2014%2011>
>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> *Arsène Gschwind*
>>>> Fa. Sapify AG im Auftrag der Universität Basel
>>>> IT Services
>>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>>> Tel. +41 79 449 25 63
>>>> <tel:+41%2079%20449%2025%2063> |
>>>> http://its.unibas.ch <http://its.unibas.ch/>
>>>> ITS-ServiceDesk: support-its at unibas.ch
>>>> <mailto:support-its at unibas.ch> | +41 61 267 14 11
>>>> <tel:+41%2061%20267%2014%2011>
>>>>
>>>>
>>>
>>> --
>>>
>>> *Arsène Gschwind*
>>> Fa. Sapify AG im Auftrag der Universität Basel
>>> IT Services
>>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>>> Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063> |
>>> http://its.unibas.ch <http://its.unibas.ch/>
>>> ITS-ServiceDesk: support-its at unibas.ch
>>> <mailto:support-its at unibas.ch> | +41 61 267 14 11
>>> <tel:+41%2061%20267%2014%2011>
>>>
>>>
>>
>> --
>>
>> *Arsène Gschwind*
>> Fa. Sapify AG im Auftrag der Universität Basel
>> IT Services
>> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
>> Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063> |
>> http://its.unibas.ch <http://its.unibas.ch/>
>> ITS-ServiceDesk: support-its at unibas.ch
>> <mailto:support-its at unibas.ch> | +41 61 267 14 11
>> <tel:+41%2061%20267%2014%2011>
>>
>>
>
> --
>
> *Arsène Gschwind*
> Fa. Sapify AG im Auftrag der Universität Basel
> IT Services
> Klingelbergstr. 70 | CH-4056 Basel | Switzerland
> Tel. +41 79 449 25 63 <tel:+41%2079%20449%2025%2063> |
> http://its.unibas.ch <http://its.unibas.ch/>
> ITS-ServiceDesk: support-its at unibas.ch
> <mailto:support-its at unibas.ch> | +41 61 267 14 11
> <tel:+41%2061%20267%2014%2011>
>
>
--
*Arsène Gschwind*
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 | CH-4056 Basel | Switzerland
Tel. +41 79 449 25 63 | http://its.unibas.ch <http://its.unibas.ch/>
ITS-ServiceDesk: support-its at unibas.ch | +41 61 267 14 11
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170905/f5e7512c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: engine.log.gz
Type: application/gzip
Size: 395563 bytes
Desc: not available
URL: <http://lists.ovirt.org/pipermail/users/attachments/20170905/f5e7512c/attachment.bin>
More information about the Users
mailing list