On Wed, Sep 6, 2017 at 1:22 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:



On 09/05/2017 09:02 AM, Arsène Gschwind wrote:



On 09/04/2017 07:59 PM, Simone Tiraboschi wrote:

On Mon, Sep 4, 2017 at 7:32 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:



On 09/04/2017 06:32 PM, Simone Tiraboschi wrote:


On Mon, Sep 4, 2017 at 6:24 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:



On 09/04/2017 02:51 PM, Simone Tiraboschi wrote:


On Mon, Sep 4, 2017 at 2:21 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:



On 09/04/2017 02:01 PM, Simone Tiraboschi wrote:


On Mon, Sep 4, 2017 at 1:55 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:



On 09/04/2017 01:52 PM, Simone Tiraboschi wrote:


On Mon, Sep 4, 2017 at 12:23 PM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:

Hi Simone,


On 09/04/2017 11:14 AM, Simone Tiraboschi wrote:


On Mon, Sep 4, 2017 at 10:56 AM, Arsène Gschwind <arsene.gschwind@unibas.ch> wrote:

Hi Didi,


On 09/04/2017 10:15 AM, Yedidyah Bar David wrote:
On Mon, Sep 4, 2017 at 10:16 AM, Arsène Gschwind
<arsene.gschwind@unibas.ch> wrote:

Hi all,

A while ago I had some problem with hosted-engine network which wasn't set
correctly at deploy time, so I finally decided to redeploy the hosted engine
in the hope the network will be set correctly this time. I've followed this
procedure:

Stop all VMs
Full backup of HE DB and export to safe place
Cleanup HE storage following https://access.redhat.com/solutions/2121581
Reboot Hosts
Re-deploy HE until DB recovery
Recover DB adding the following param:
 --he-remove-storage-vm             Removes the hosted-engine storage
domain, all its entities and the hosted-engine VM during restore.
 --he-remove-hosts                  Removes all the hosted-engine hosts
during restore.

Finalize HE deployment.

Everything did run without errors and I'm able to access Web UI.

But now I don't see my HE VM and its respective Storage Domain, the logs
says it isn't able to import it. I see all other SD and I'm able to manage
my VMs as before.

Please find attached engine.log

I think this is your problem:

2017-09-04 03:26:14,272+02 INFO
[org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand]
(org.ovirt.thread.pool-6-thread-24) [2383eaa0] There are existing luns
in the system which are part of VG id
'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'
I don't see a VG with this ID, here the IDs I see on the hosts:

  VG                                   #PV #LV #SN Attr   VSize  VFree
  6b62cc06-fc44-4c38-af6d-bfd9cbe73246   1  10   0 wz--n- 99.62g 14.50g
  b0414c06-d984-4001-a998-fd9a2e79fb83   2  70   0 wz--n- 10.00t  2.31t
  b2e30961-7cff-4cca-83d6-bee3a4f890ee   2  47   0 wz--n-  5.27t  2.50t


Could you please repeat the command on host adm-kvmh70 ?

2017-09-04 09:04:18,163+02 INFO  [org.ovirt.engine.core.bll.storage.domain.ImportHostedEngineStorageDomainCommand] (org.ovirt.thread.pool-6-thread-34) [247a3718] Running command: ImportHostedEngineStorageDomainCommand internal: true.
2017-09-04 09:04:18,189+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] START, GetVGInfoVDSCommand(HostName = adm-kvmh70, GetVGInfoVDSCommandParameters:{runAsync='true', hostId='acbacabb-6c4a-43fd-a1e2-2d7ff2f6f98b', VGID='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'}), log id: 6693b98a
2017-09-04 09:04:18,232+02 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.GetVGInfoVDSCommand] (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] FINISH, GetVGInfoVDSCommand, return: [LUNs:{id='repl_HostedEngine', physicalVolumeId='kYN8Jj-FBDw-MhxI-XcoZ-w1zH-eQL8-IRIgzO', volumeGroupId='vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g', serial='SHITACHI_OPEN-V_50488888', lunMapping='4', vendorId='HITACHI', productId='OPEN-V', lunConnections='[]', deviceSize='100', pvSize='0', peCount='797', peAllocatedCount='681', vendorName='HITACHI', pathsDictionary='[sdf=true, sdu=true, sdk=true, sdp=true]', pathsCapacity='[sdf=100, sdu=100, sdk=100, sdp=100]', lunType='FCP', status='null', diskId='null', diskAlias='null', storageDomainId='6b62cc06-fc44-4c38-af6d-bfd9cbe73246', storageDomainName='null', discardMaxSize='268435456', discardZeroesData='true'}], log id: 6693b98a
2017-09-04 09:04:18,245+02 INFO  [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand] (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] There are existing luns in the system which are part of VG id 'vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g'
2017-09-04 09:04:18,245+02 WARN  [org.ovirt.engine.core.bll.storage.domain.AddExistingBlockStorageDomainCommand] (org.ovirt.thread.pool-6-thread-34) [7d2e6cb2] Validation of action 'AddExistingBlockStorageDomain' failed for user SYSTEM. Reasons: VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__ADD,ACTION_TYPE_FAILED_IMPORT_STORAGE_DOMAIN_EXTERNAL_LUN_DISK_EXIST
I don't know which command you are talking about, I didn't run any command since it tries to import the SD automatically.

Sorry, can you please run vgdisplay on adm-kvmh70 ?
No problem! here the result:

[root@adm-kvmh70 ~]# vgs
  VG                                   #PV #LV #SN Attr   VSize   VFree 
  6b62cc06-fc44-4c38-af6d-bfd9cbe73246   1  10   0 wz--n-  99.62g  14.50g
  b0414c06-d984-4001-a998-fd9a2e79fb83   2  70   0 wz--n-  10.00t   2.31t
  b2e30961-7cff-4cca-83d6-bee3a4f890ee   2  47   0 wz--n-   5.27t   2.50t
  vg_adm-kvmh70                          1   3   0 wz--n- 277.90g 218.68g


OK, and 
   vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
[root@adm-kvmh70 ~]# vgdisplay 6b62cc06-fc44-4c38-af6d-bfd9cbe73246
  --- Volume group ---
  VG Name               6b62cc06-fc44-4c38-af6d-bfd9cbe73246
  System ID            
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  23
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                10
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               99.62 GiB
  PE Size               128.00 MiB
  Total PE              797
  Alloc PE / Size       681 / 85.12 GiB
  Free  PE / Size       116 / 14.50 GiB
  VG UUID               vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g

OK, vvIoS2-fZTZ-99Ox-Ltzq-pr8U-SL2z-wbTU8g is the uuid of the VG that engine says that it includes the LUN used by the hosted-engine SD as detected by the engine.
Its name is 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 which is also the uuid of that storage domain.

Now we need to understand what's there.

Could you please check if an SD with uuid 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is visible in the engine?
The visible SD in the engine

[oVirt shell (connected)]# list storagedomains

id         : 072fbaa1-08f3-4a40-9f34-a5ca22dd1d74
name       : ovirt-image-repository
description: Public Glance repository for oVirt

id         : b2e30961-7cff-4cca-83d6-bee3a4f890ee
name       : xxxxxxxxxxx

id         : b0414c06-d984-4001-a998-fd9a2e79fb83
name       : xxxxxxxxxxx
description: xxxxxxxxxxx

id         : 71fa1c61-4a68-469f-9a56-835190d32075
name       : xxxxxxxxxxx

id         : 9817e64b-fdd9-4d4c-91ee-f3482bb0cc56
name       : xxxxxxxxxxxx
description: xxxxxxxxxxxx

Is adm-kvmh70 involved in hosted-engine? did you redeployed it?
 Yes adm-kvmh70 should be a host hosting HE and yes as written before I did redeploy HE because of a network interface problem. I did cleanup the lun using the RedHat guide, redeploy HE and restore the DB using the mentioned parameter to remove HE information.

Could you please set adm-kvmh70 to maintenance mode from the engine and then reactivate it or better reboot it while in maintenance mode?
I suspect that 6b62cc06-fc44-4c38-af6d-bfd9cbe73246 is just a left over of the previous hosted-engine storage domain kept open on the host. 
I did reboot adm-kvmh70 while in maintenance mode and also reboot the engine but it didn't change anything. The import failure is still there...
I've noticed a ovirt-ha-agent error:

ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please ensure you already added your first data domain for regular VMs

Could that be related?

This is just a side effects of that: the OVF_STORE disk is going to be created by the engine once it correctly imported the hosted-engine storage domain.

Could you please share again your up-to-date engine.log?
 
Thanks

Please find the latest engine log attached.

Thanks
When analyzing the log I do see some warning about existing LUNs which are not part of the VG, does this means the LUNs are existing in the DB?

That LUN is reported by VDSM to the engine.
Could you please share your /etc/ovirt-hosted-engine/hosted-engine.conf?
 
If yes would it be possible to remove them from the engine DB?

Thanks for your help,
Arsene


 


In the past did you tried to manually import the hosted-engine SD or something similar?
 No, before the "redeployment" the HE SD was auto imported successfully.

Thanks a lot




thanks

 

Thanks

?
 

 

Thanks,
Arsene

 


Thanks for any help to resolve that issue.

I guess you can try to remove this disk/lun from the engine and let it retry.
 Could let me know how to remove that lun from the engine?
If the only disk is of the hosted-engine, I guess it should have been
removed by '--he-remove-storage-vm' - if so, please open a bug
describing your flow in detail. Thanks.
 It seems that this option didn't remove the he storage information and that this VG is still the old one.

Many thanks for your help
Rgds,
Arsene
Best,


Arsène

--

Arsène Gschwind
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11



--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11 



_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users

--

Arsène Gschwind    
Fa. Sapify AG im Auftrag der Universität Basel
IT Services
Klingelbergstr. 70 |  CH-4056 Basel  |  Switzerland
Tel. +41 79 449 25 63  |  http://its.unibas.ch
ITS-ServiceDesk: support-its@unibas.ch | +41 61 267 14 11


_______________________________________________
Users mailing list
Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users