11:25 a.m.
Hi list,
I'm working on a system running on oVirt 3.6 and the Engine is reporting
the warning "The Hosted Engine Storage Domain doesn't exist. It should
be imported into the setup." repeatedly in the Events tab into the Admin
Portal.
I've read into the list that Hosted Engine Storage Domain should be
imported automatically into the setup during the upgrade to 3.6
(original setup was on 3.5), but this not happened while the
HostedEngine is correctly visible into the VM tab after the upgrade.
The Hosted Engine Storage Domain is on a dedicated gluster volume but
considering that, if I remember correctly, oVirt 3.5 at that time did
not support gluster as a backend for the HostedEngine at that time I had
installed the engine using gluster's NFS server using
'localhost:/hosted-engine' as a mount point.
Currently on every nodes I can read into the log of the
ovirt-hosted-engine-ha agent the following lines:
MainThread::INFO::2017-03-17
14:04:17,773::hosted_engine::462::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Current state EngineUp (score: 3400)
MainThread::INFO::2017-03-17
14:04:17,774::hosted_engine::467::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(start_monitoring)
Best remote host virtnode-0-1 (id: 2
, score: 3400)
MainThread::INFO::2017-03-17
14:04:27,956::hosted_engine::613::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm)
Initializing VDSM
MainThread::INFO::2017-03-17
14:04:28,055::hosted_engine::658::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Connecting the storage
MainThread::INFO::2017-03-17
14:04:28,078::storage_server::218::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2017-03-17
14:04:28,278::storage_server::222::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Connecting storage server
MainThread::INFO::2017-03-17
14:04:28,398::storage_server::230::ovirt_hosted_engine_ha.lib.storage_server.StorageServer::(connect_storage_server)
Refreshing the storage domain
MainThread::INFO::2017-03-17
14:04:28,822::hosted_engine::685::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Preparing images
MainThread::INFO::2017-03-17
14:04:28,822::image::126::ovirt_hosted_engine_ha.lib.image.Image::(prepare_images)
Preparing images
MainThread::INFO::2017-03-17
14:04:29,308::hosted_engine::688::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_storage_images)
Reloading vm.conf from the
shared storage domain
MainThread::INFO::2017-03-17
14:04:29,309::config::206::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Trying to get a fresher copy
of vm configuration from the OVF_STORE
MainThread::WARNING::2017-03-17
14:04:29,567::ovf_store::104::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan)
Unable to find OVF_STORE
MainThread::ERROR::2017-03-17
14:04:29,691::config::235::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file)
Unable to get vm.conf from O
VF_STORE, falling back to initial vm.conf
...and the following lines into the logfile engine.log inside the Hosted
Engine:
2017-03-16 07:36:28,087 INFO
[org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
(org.ovirt.thread.pool-8-thread-38) [236d315c] Lock Acquired to object
'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
2017-03-16 07:36:28,115 WARN
[org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
(org.ovirt.thread.pool-8-thread-38) [236d315c] CanDoAction of action
'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
2017-03-16 07:36:28,116 INFO
[org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
(org.ovirt.thread.pool-8-thread-38) [236d315c] Lock freed to object
'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
How can I safely import the Hosted Engine Storage Domain into my setup?
In this situation is safe to upgrade to oVirt 4.0?
Greetings,
Paolo
11:55 a.m.
On Sat, Mar 18, 2017 at 12:25 PM, Paolo Margara <paolo.margara(a)polito.it> wrote:
Was the upgrade to 3.6 successful and clean?
This is normal at your current state.
That's the thing to debug. Did you check vdsm logs on the hosts, near
the time this happens?
I'd first try to solve this.
What OS do you have on your hosts? Are they all upgraded to 3.6?
See also:
https://www.ovirt.org/documentation/how-to/hosted-engine-host-OS-upgrade/
Best,
--
Didi
10:12 a.m.
Hi Yedidyah,
Il 19/03/2017 11:55, Yedidyah Bar David ha scritto:
The upgrade from 3.5 to 3.6 was
successful, as every subsequent minor
release upgrades. I rechecked the upgrade logs I haven't seen any
relevant error.
One addition information: I'm currently running on CentOS 7 and also the
original setup was on this release version.
Some moments before I saw the following lines into the
vdsm.log of the
host that execute the hosted engine and that is the SPM, but I see the
same lines also on the other nodes:
Thread-1746094::DEBUG::2017-03-16
07:36:00,412::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::moving from state init ->
state preparing
Thread-1746094::INFO::2017-03-16
07:36:00,413::logUtils::48::dispatcher::(wrapper) Run and protect:
getImagesList(sdUUID='3b5db584-5d21-41dc-8f8d-712ce9423a27', options=None)
Thread-1746094::DEBUG::2017-03-16
07:36:00,413::resourceManager::199::Storage.ResourceManager.Request::(__init__)
ResName=`Storage.3b5db584-5d21-41dc-8f8d-712ce9423a27`ReqID=`8ea3c7f3-8ccd-4127-96b1-ec97a3c7b8d4`::Request
was made in '/usr/share/vdsm/storage/hsm.py' line '3313' at
'getImagesList'
Thread-1746094::DEBUG::2017-03-16
07:36:00,413::resourceManager::545::Storage.ResourceManager::(registerResource)
Trying to register resource
'Storage.3b5db584-5d21-41dc-8f8d-712ce9423a27' for lock type 'shared'
Thread-1746094::DEBUG::2017-03-16
07:36:00,414::resourceManager::604::Storage.ResourceManager::(registerResource)
Resource 'Storage.3b5db584-5d21-41dc-8f8d-712ce9423a27' is free. Now
locking as 'shared' (1 active user)
Thread-1746094::DEBUG::2017-03-16
07:36:00,414::resourceManager::239::Storage.ResourceManager.Request::(grant)
ResName=`Storage.3b5db584-5d21-41dc-8f8d-712ce9423a27`ReqID=`8ea3c7f3-8ccd-4127-96b1-ec97a3c7b8d4`::Granted
request
Thread-1746094::DEBUG::2017-03-16
07:36:00,414::task::827::Storage.TaskManager.Task::(resourceAcquired)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::_resourcesAcquired:
Storage.3b5db584-5d21-41dc-8f8d-712ce9423a27 (shared)
Thread-1746094::DEBUG::2017-03-16
07:36:00,414::task::993::Storage.TaskManager.Task::(_decref)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::ref 1 aborting False
Thread-1746094::ERROR::2017-03-16
07:36:00,415::task::866::Storage.TaskManager.Task::(_setError)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::Unexpected error
Traceback (most recent call last):
File "/usr/share/vdsm/storage/task.py", line 873, in _run
return fn(*args, **kargs)
File "/usr/share/vdsm/logUtils.py", line 49, in wrapper
res = f(*args, **kwargs)
File "/usr/share/vdsm/storage/hsm.py", line 3315, in getImagesList
images = dom.getAllImages()
File "/usr/share/vdsm/storage/fileSD.py", line 373, in getAllImages
self.getPools()[0],
IndexError: list index out of range
Thread-1746094::DEBUG::2017-03-16
07:36:00,415::task::885::Storage.TaskManager.Task::(_run)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::Task._run:
ae5af1a1-207c-432d-acfa-f3e03e014ee6
('3b5db584-5d21-41dc-8f8d-712ce9423a27',) {} failed - stopping task
Thread-1746094::DEBUG::2017-03-16
07:36:00,415::task::1246::Storage.TaskManager.Task::(stop)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::stopping in state preparing
(force False)
Thread-1746094::DEBUG::2017-03-16
07:36:00,416::task::993::Storage.TaskManager.Task::(_decref)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::ref 1 aborting True
Thread-1746094::INFO::2017-03-16
07:36:00,416::task::1171::Storage.TaskManager.Task::(prepare)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::aborting: Task is aborted:
u'list index out of range' - code 100
Thread-1746094::DEBUG::2017-03-16
07:36:00,416::task::1176::Storage.TaskManager.Task::(prepare)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::Prepare: aborted: list
index out of range
Thread-1746094::DEBUG::2017-03-16
07:36:00,416::task::993::Storage.TaskManager.Task::(_decref)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::ref 0 aborting True
Thread-1746094::DEBUG::2017-03-16
07:36:00,416::task::928::Storage.TaskManager.Task::(_doAbort)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::Task._doAbort: force False
Thread-1746094::DEBUG::2017-03-16
07:36:00,416::resourceManager::980::Storage.ResourceManager.Owner::(cancelAll)
Owner.cancelAll requests {}
Thread-1746094::DEBUG::2017-03-16
07:36:00,417::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::moving from state preparing
-> state aborting
Thread-1746094::DEBUG::2017-03-16
07:36:00,417::task::550::Storage.TaskManager.Task::(__state_aborting)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::_aborting: recover policy none
Thread-1746094::DEBUG::2017-03-16
07:36:00,417::task::595::Storage.TaskManager.Task::(_updateState)
Task=`ae5af1a1-207c-432d-acfa-f3e03e014ee6`::moving from state aborting
-> state failed
After that I tried to execute a simple query on storage domains using
vdsClient and I got the following information:
# vdsClient -s 0 getStorageDomainsList
3b5db584-5d21-41dc-8f8d-712ce9423a27
0966f366-b5ae-49e8-b05e-bee1895c2d54
35223b83-e0bd-4c8d-91a9-8c6b85336e7d
2c3994e3-1f93-4f2a-8a0a-0b5d388a2be7
# vdsClient -s 0 getStorageDomainInfo 3b5db584-5d21-41dc-8f8d-712ce9423a27
uuid = 3b5db584-5d21-41dc-8f8d-712ce9423a27
version = 3
role = Regular
remotePath = localhost:/hosted-engine
type = NFS
class = Data
pool = []
name = default
# vdsClient -s 0 getImagesList 3b5db584-5d21-41dc-8f8d-712ce9423a27
list index out of range
All other storage domains have the pool attribute defined, could be this
the issue? How can I assign to a pool the Hosted Engine Storage Domain?
Greetings,
Paolo
11:15 a.m.
On Mon, Mar 20, 2017 at 10:12 AM, Paolo Margara <paolo.margara(a)polito.it>
wrote:
Your issue is probably here: by design all the hosts of a single datacenter
should be able to see all the storage domains including the hosted-engine
one but if try to mount it as localhost:/hosted-engine this will not be
possible.
This will be the result of the auto import process once feasible.
11:59 a.m.
On Mon, Mar 20, 2017 at 11:15 AM, Simone Tiraboschi <stirabos(a)redhat.com>
wrote:
This could be really tricky because I think that upgrading from an
hosted-engine env from 3.5 deployed on hyperconverged env but mounted on
NFS over a localhost loopback mount to 4.1 is something that is by far out
of the paths we tested so I think you can hit a few surprises there.
In 4.1 the expected configuration under
/etc/ovirt-hosted-engine/hosted-engine.conf
includes:
domainType=glusterfs
storage=<FIRST_HOST_ADDR>:/path
mnt_options=backup-volfile-servers=<SECOND_HOST_ADDR>:<THIRD_HOST_ADDR>
But these requires more recent vdsm and ovirt-hosted-engine-ha versions.
Then you also have to configure your engine to have both virt and gluster
on the same cluster.
Nothing is going to do them automatically for you on upgrades.
I see two options here:
1. easy, but with a substantial downtime: shutdown your whole DC, start
from scratch with gdeploy from 4.1 to configure a new gluster volume and
anew engine over there, once you have a new engine import your existing
storage domain and restart your VMs
2. a lot trickier, try to reach 4.1 status manually
editing /etc/ovirt-hosted-engine/hosted-engine.conf and so on and upgrading
everything to 4.1; this could be pretty risky because you are on a path we
never tested since hyperconverged hosted-engine at 3.5 wasn't released.
9:24 a.m.
This is a multi-part message in MIME format.
--------------17C4B33F916F521113721D9F
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
Hi Simone,
I'll respond inline
Il 20/03/2017 11:59, Simone Tiraboschi ha scritto:
I understood, definitely bad news for me. But currently
I'm running
oVirt 3.6.7 that, if I remember correctly, supports hyperconverged
setup, it's not possible to fix this issue with my current version? I've
installed vdsm 4.17.32-1.el7 with the vdsm-gluster package and
ovirt-hosted-engine-ha 1.3.5.7-1.el7.centos on CentOS 7.2.1511 and my
engine it's already configured to have both virt and gluster on the same
cluster. I cannot put the cluster in maintenance, stop the
hosted-engine, stop ovirt-hosted-engine-ha, edit hosted-engine.conf by
changing domainType, storage and mnt_options and restart
aovirt-hosted-engine-ha and the hosted-engine?
Greetings, Paolo
--------------17C4B33F916F521113721D9F
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta content="text/html; charset=utf-8"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi Simone,</p>
<p>I'll respond inline<br>
</p>
<br>
<div class="moz-cite-prefix">Il 20/03/2017 11:59, Simone Tiraboschi
ha scritto:<br>
</div>
<blockquote
cite="mid:CAN8-ONr2sC7nKPnDKh1j+rGRQ+kmKfODS6AJF9Wk3kYa8ujOcA@mail.gmail.com"
type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Mar 20, 2017 at 11:15 AM,
Simone Tiraboschi <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:stirabos@redhat.com"
target="_blank">stirabos(a)redhat.com</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div
class="gmail-m_-4888476689598940363gmail-h5">On
Mon, Mar 20, 2017 at 10:12 AM, Paolo Margara <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:paolo.margara@polito.it"
target="_blank">paolo.margara(a)polito.it</a>&gt;</span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">Hi
Yedidyah,<br>
<span
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-"><br>
Il 19/03/2017 11:55, Yedidyah Bar David ha
scritto:<br>
> On Sat, Mar 18, 2017 at 12:25 PM, Paolo
Margara <<a moz-do-not-send="true"
href="mailto:paolo.margara@polito.it"
target="_blank">paolo.margara(a)polito.it</a>&gt;
wrote:<br>
>> Hi list,<br>
>><br>
>> I'm working on a system running on
oVirt 3.6 and the Engine is reporting<br>
>> the warning "The Hosted Engine
Storage Domain doesn't exist. It should<br>
>> be imported into the setup."
repeatedly in the Events tab into the Admin<br>
>> Portal.<br>
>><br>
>> I've read into the list that Hosted
Engine Storage Domain should be<br>
>> imported automatically into the
setup during the upgrade to 3.6<br>
>> (original setup was on 3.5), but
this not happened while the<br>
>> HostedEngine is correctly visible
into the VM tab after the upgrade.<br>
> Was the upgrade to 3.6 successful and
clean?<br>
</span>The upgrade from 3.5 to 3.6 was
successful, as every subsequent minor<br>
release upgrades. I rechecked the upgrade logs
I haven't seen any<br>
relevant error.<br>
One addition information: I'm currently
running on CentOS 7 and also the<br>
original setup was on this release version.<br>
<div>
<div
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-h5">><br>
>> The Hosted Engine Storage Domain
is on a dedicated gluster volume but<br>
>> considering that, if I remember
correctly, oVirt 3.5 at that time did<br>
>> not support gluster as a backend
for the HostedEngine at that time I had<br>
>> installed the engine using
gluster's NFS server using<br>
>> 'localhost:/hosted-engine' as a
mount point.<br>
>><br>
>> Currently on every nodes I can
read into the log of the<br>
>> ovirt-hosted-engine-ha agent the
following lines:<br>
>><br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:17,773::hosted_engine::4<wbr>62::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring)<br>
>> Current state EngineUp (score:
3400)<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:17,774::hosted_engine::4<wbr>67::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(start_monitoring)<br>
>> Best remote host virtnode-0-1
(id: 2<br>
>> , score: 3400)<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:27,956::hosted_engine::6<wbr>13::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_vdsm)<br>
>> Initializing VDSM<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,055::hosted_engine::6<wbr>58::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_images)<br>
>> Connecting the storage<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,078::storage_server::<wbr>218::<a
moz-do-not-send="true"
href="http://ovirt_hosted_engine_ha.li"
target="_blank">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::(connect_storage_server)<br>
>> Connecting storage server<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,278::storage_server::<wbr>222::<a
moz-do-not-send="true"
href="http://ovirt_hosted_engine_ha.li"
target="_blank">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::(connect_storage_server)<br>
>> Connecting storage server<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,398::storage_server::<wbr>230::<a
moz-do-not-send="true"
href="http://ovirt_hosted_engine_ha.li"
target="_blank">ovirt_hosted_engine_ha.li</a><wbr>b.storage_server.StorageServer<wbr>::(connect_storage_server)<br>
>> Refreshing the storage domain<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,822::hosted_engine::6<wbr>85::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_images)<br>
>> Preparing images<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:28,822::image::126::ovir<wbr>t_hosted_engine_ha.lib.image.I<wbr>mage::(prepare_images)<br>
>> Preparing images<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:29,308::hosted_engine::6<wbr>88::ovirt_hosted_engine_ha.age<wbr>nt.hosted_engine.HostedEngine:<wbr>:(_initialize_storage_images)<br>
>> Reloading vm.conf from the<br>
>> shared storage domain<br>
>> MainThread::<a
class="moz-txt-link-freetext"
href="INFO::2017-03-17">INFO::2017-03-17</a><br>
>>
14:04:29,309::config::206::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(refresh_local_conf_file)<br>
>> Trying to get a fresher copy<br>
>> of vm configuration from the
OVF_STORE<br>
>>
MainThread::WARNING::2017-03-1<wbr>7<br>
>>
14:04:29,567::ovf_store::104::<wbr>ovirt_hosted_engine_ha.lib.ovf<wbr>.ovf_store.OVFStore::(scan)<br>
>> Unable to find OVF_STORE<br>
>> MainThread::ERROR::2017-03-17<br>
>>
14:04:29,691::config::235::ovi<wbr>rt_hosted_engine_ha.agent.host<wbr>ed_engine.HostedEngine.config:<wbr>:(refresh_local_conf_file)<br>
>> Unable to get vm.conf from O<br>
>> VF_STORE, falling back to initial
vm.conf<br>
> This is normal at your current state.<br>
><br>
>> ...and the following lines into
the logfile engine.log inside the Hosted<br>
>> Engine:<br>
>><br>
>> 2017-03-16 07:36:28,087 INFO<br>
>>
[org.ovirt.engine.core.bll.Imp<wbr>ortHostedEngineStorageDomainCo<wbr>mmand]<br>
>>
(org.ovirt.thread.pool-8-threa<wbr>d-38)
[236d315c] Lock Acquired to object<br>
>>
'EngineLock:{exclusiveLocks='[<wbr>]',
sharedLocks='null'}'<br>
>> 2017-03-16 07:36:28,115 WARN<br>
>>
[org.ovirt.engine.core.bll.Imp<wbr>ortHostedEngineStorageDomainCo<wbr>mmand]<br>
>>
(org.ovirt.thread.pool-8-threa<wbr>d-38)
[236d315c] CanDoAction of action<br>
>>
'ImportHostedEngineStorageDoma<wbr>in'
failed for user SYSTEM. Reasons:<br>
>>
VAR__ACTION__ADD,VAR__TYPE__ST<wbr>ORAGE__DOMAIN,ACTION_TYPE_FAIL<wbr>ED_STORAGE_DOMAIN_NOT_EXIST<br>
> That's the thing to debug. Did you
check vdsm logs on the hosts, near<br>
> the time this happens?<br>
</div>
</div>
Some moments before I saw the following lines
into the vdsm.log of the<br>
host that execute the hosted engine and that
is the SPM, but I see the<br>
same lines also on the other nodes:<br>
<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,412::task::595::Stora<wbr>ge.TaskManager.Task::(_updateS<wbr>tate)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::moving
from state init -><br>
state preparing<br>
Thread-1746094::<a class="moz-txt-link-freetext"
href="INFO::2017-03">INFO::2017-03</a>-<wbr>16<br>
07:36:00,413::logUtils::48::di<wbr>spatcher::(wrapper)
Run and protect:<br>
getImagesList(sdUUID='3b5db584<wbr>-5d21-41dc-8f8d-712ce9423a27',
options=None)<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,413::resourceManager:<wbr>:199::Storage.ResourceManager.<wbr>Request::(__init__)<br>
ResName=`Storage.3b5db584-5d21<wbr>-41dc-8f8d-712ce9423a27`ReqID=<wbr>`8ea3c7f3-8ccd-4127-96b1-ec97a<wbr>3c7b8d4`::Request<br>
was made in '/usr/share/vdsm/storage/hsm.p<wbr>y'
line '3313' at 'getImagesList'<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,413::resourceManager:<wbr>:545::Storage.ResourceManager:<wbr>:(registerResource)<br>
Trying to register resource<br>
'Storage.3b5db584-5d21-41dc-8f<wbr>8d-712ce9423a27'
for lock type 'shared'<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,414::resourceManager:<wbr>:604::Storage.ResourceManager:<wbr>:(registerResource)<br>
Resource
'Storage.3b5db584-5d21-41dc-8f<wbr>8d-712ce9423a27'
is free. Now<br>
locking as 'shared' (1 active user)<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,414::resourceManager:<wbr>:239::Storage.ResourceManager.<wbr>Request::(grant)<br>
ResName=`Storage.3b5db584-5d21<wbr>-41dc-8f8d-712ce9423a27`ReqID=<wbr>`8ea3c7f3-8ccd-4127-96b1-ec97a<wbr>3c7b8d4`::Granted<br>
request<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,414::task::827::Stora<wbr>ge.TaskManager.Task::(resource<wbr>Acquired)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::_resourcesAcqui<wbr>red:<br>
Storage.3b5db584-5d21-41dc-8f8<wbr>d-712ce9423a27
(shared)<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,414::task::993::Stora<wbr>ge.TaskManager.Task::(_decref)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::ref
1 aborting False<br>
Thread-1746094::ERROR::2017-03<wbr>-16<br>
07:36:00,415::task::866::Stora<wbr>ge.TaskManager.Task::(_setErro<wbr>r)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::Unexpected
error<br>
Traceback (most recent call last):<br>
File "/usr/share/vdsm/storage/task.<wbr>py",
line 873, in _run<br>
return fn(*args, **kargs)<br>
File "/usr/share/vdsm/logUtils.py", line 49,
in wrapper<br>
res = f(*args, **kwargs)<br>
File "/usr/share/vdsm/storage/hsm.p<wbr>y",
line 3315, in getImagesList<br>
images = dom.getAllImages()<br>
File
"/usr/share/vdsm/storage/fileS<wbr>D.py",
line 373, in getAllImages<br>
self.getPools()[0],<br>
IndexError: list index out of range<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,415::task::885::Stora<wbr>ge.TaskManager.Task::(_run)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::Task._run:<br>
ae5af1a1-207c-432d-acfa-f3e03e<wbr>014ee6<br>
('3b5db584-5d21-41dc-8f8d-712c<wbr>e9423a27',)
{} failed - stopping task<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,415::task::1246::Stor<wbr>age.TaskManager.Task::(stop)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::stopping
in state preparing<br>
(force False)<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,416::task::993::Stora<wbr>ge.TaskManager.Task::(_decref)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::ref
1 aborting True<br>
Thread-1746094::<a class="moz-txt-link-freetext"
href="INFO::2017-03">INFO::2017-03</a>-<wbr>16<br>
07:36:00,416::task::1171::Stor<wbr>age.TaskManager.Task::(prepare<wbr>)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::aborting:
Task is aborted:<br>
u'list index out of range' - code 100<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,416::task::1176::Stor<wbr>age.TaskManager.Task::(prepare<wbr>)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::Prepare:
aborted: list<br>
index out of range<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,416::task::993::Stora<wbr>ge.TaskManager.Task::(_decref)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::ref
0 aborting True<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,416::task::928::Stora<wbr>ge.TaskManager.Task::(_doAbort<wbr>)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::Task._doAbort:
force False<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,416::resourceManager:<wbr>:980::Storage.ResourceManager.<wbr>Owner::(cancelAll)<br>
Owner.cancelAll requests {}<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,417::task::595::Stora<wbr>ge.TaskManager.Task::(_updateS<wbr>tate)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::moving
from state preparing<br>
-> state aborting<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,417::task::550::Stora<wbr>ge.TaskManager.Task::(__state_<wbr>aborting)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::_aborting:
recover policy none<br>
Thread-1746094::DEBUG::2017-03<wbr>-16<br>
07:36:00,417::task::595::Stora<wbr>ge.TaskManager.Task::(_updateS<wbr>tate)<br>
Task=`ae5af1a1-207c-432d-acfa-<wbr>f3e03e014ee6`::moving
from state aborting<br>
-> state failed<br>
<br>
After that I tried to execute a simple query
on storage domains using<br>
vdsClient and I got the following information:<br>
<br>
# vdsClient -s 0 getStorageDomainsList<br>
3b5db584-5d21-41dc-8f8d-712ce9<wbr>423a27<br>
0966f366-b5ae-49e8-b05e-bee189<wbr>5c2d54<br>
35223b83-e0bd-4c8d-91a9-8c6b85<wbr>336e7d<br>
2c3994e3-1f93-4f2a-8a0a-0b5d38<wbr>8a2be7<br>
# vdsClient -s 0 getStorageDomainInfo
3b5db584-5d21-41dc-8f8d-712ce9<wbr>423a27<br>
uuid =
3b5db584-5d21-41dc-8f8d-712ce9<wbr>423a27<br>
version = 3<br>
role = Regular<br>
remotePath = localhost:/hosted-engine<br>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>Your issue is probably here: by design all the
hosts of a single datacenter should be able to see
all the storage domains including the
hosted-engine one but if try to mount it as
localhost:/hosted-engine this will not be
possible.</div>
<span class="gmail-m_-4888476689598940363gmail-">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
type = NFS<br>
class = Data<br>
pool = []<br>
name = default<br>
# vdsClient -s 0 getImagesList
3b5db584-5d21-41dc-8f8d-712ce9<wbr>423a27<br>
list index out of range<br>
<br>
All other storage domains have the pool
attribute defined, could be this<br>
the issue? How can I assign to a pool the Hosted
Engine Storage Domain?<br>
</blockquote>
<div><br>
</div>
</span>
<div>This will be the result of the auto import
process once feasible.</div>
<span class="gmail-m_-4888476689598940363gmail-">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px
0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-HOEnZb">
<div
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-h5">><br>
>> 2017-03-16 07:36:28,116 INFO<br>
>>
[org.ovirt.engine.core.bll.Imp<wbr>ortHostedEngineStorageDomainCo<wbr>mmand]<br>
>>
(org.ovirt.thread.pool-8-threa<wbr>d-38)
[236d315c] Lock freed to object<br>
>>
'EngineLock:{exclusiveLocks='[<wbr>]',
sharedLocks='null'}'<br>
>><br>
>> How can I safely import the Hosted
Engine Storage Domain into my setup?<br>
>> In this situation is safe to
upgrade to oVirt 4.0?<br>
</div>
</div>
</blockquote>
</span></div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>This could be really tricky because I think that
upgrading from an hosted-engine env from 3.5 deployed on
hyperconverged env but mounted on NFS over a localhost
loopback mount to 4.1 is something that is by far out of
the paths we tested so I think you can hit a few surprises
there.</div>
<div><br>
</div>
<div>In 4.1 the expected configuration under
/etc/ovirt-hosted-engine/<wbr>hosted-engine.conf
includes:</div>
<div>domainType=glusterfs<br>
</div>
<div>storage=<FIRST_HOST_ADDR>:/<wbr>path<br>
</div>
<div>mnt_options=<span class="gmail-il"
style="white-space:pre-wrap">backup</span><span
style="color:rgb(85,85,85);white-space:pre-wrap">-</span><span
class="gmail-il"
style="white-space:pre-wrap">volfile</span><span
style="color:rgb(85,85,85);white-space:pre-wrap">-</span><wbr
style="color:rgb(85,85,85);white-space:pre-wrap"><span
class="gmail-il"
style="white-space:pre-wrap">servers</span><span
style="color:rgb(85,85,85);white-space:pre-wrap">=</span><SECOND_HOST_ADDR><span
style="color:rgb(85,85,85);white-space:pre-wrap">:</span><THIRD_HOST_ADDR>
</div><div>
</div><div>But these requires more recent vdsm and ovirt-hosted-engine-ha
versions.</div><div>Then you also have to configure your engine to have both
virt and gluster on the same cluster.</div><div>Nothing is going to do them
automatically for you on upgrades.</div><div>
</div>I see two options here:
1. easy, but with a substantial downtime: shutdown your whole DC, start from scratch with
gdeploy from 4.1 to configure a new gluster volume and anew engine over there, once you
have a new engine import your existing storage domain and restart your
VMs</div><div class="gmail_quote">2. a lot trickier, try to reach
4.1 status manually editing /etc/ovirt-hosted-engine/hosted-engine.conf and so on and
upgrading everything to 4.1; this could be pretty risky because you are on a path we never
tested since hyperconverged hosted-engine at 3.5 wasn't
released.</div></div></div></blockquote>I understood, definitely
bad news for me. But currently I'm running oVirt 3.6.7 that, if I remember correctly,
supports hyperconverged setup, it's not possible to fix this issue with my current
version?
I've installed vdsm 4.17.32-1.el7 with the vdsm-gluster package and
ovirt-hosted-engine-ha 1.3.5.7-1.el7.centos on CentOS 7.2.1511 and my engine it's
already configured to have both virt and gluster on the same cluster.
I cannot put the cluster in maintenance, stop the hosted-engine, stop
ovirt-hosted-engine-ha, edit hosted-engine.conf by changing domainType, storage and
mnt_options and restart aovirt-hosted-engine-ha and the hosted-engine?
<blockquote
cite="mid:CAN8-ONr2sC7nKPnDKh1j+rGRQ+kmKfODS6AJF9Wk3kYa8ujOcA@mail.gmail.com"
type="cite"><div dir="ltr"><div
class="gmail_extra"><div class="gmail_quote"> <blockquote
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div
class="gmail_extra"><div class="gmail_quote"><span
class="gmail-m_-4888476689598940363gmail-"><blockquote
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex"><div
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-HOEnZb"><div
class="gmail-m_-4888476689598940363gmail-m_7393503524263847303gmail-h5">
> I'd first try to solve this.
>
> What OS do you have on your hosts? Are they all upgraded to 3.6?
>
> See also:
>
> <a moz-do-not-send="true"
href="https://www.ovirt.org/documentation/how-to/hosted-engine-host-...
rel="noreferrer"
target="_blank">https://www.ovirt.org/document<wbr>ati...
>
> Best,
>
>>
>> Greetings,
>> Paolo
>>
>> ______________________________<wbr>_________________
>> Users mailing list
>> <a moz-do-not-send="true"
href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a>
>> <a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
>
>
Greetings,
Paolo
______________________________<wbr>_________________
Users mailing list
<a moz-do-not-send="true" href="mailto:Users@ovirt.org"
target="_blank">Users(a)ovirt.org</a>
<a moz-do-not-send="true"
href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer"
target="_blank">http://lists.ovirt.org/mailman<wbr>/li...
</div></div></blockquote></span></div>
</div></div></blockquote></div></div></div></blockquote>Greetings,
Paolo
</body></html>
--------------17C4B33F916F521113721D9F--
11:59 a.m.
On Tue, Mar 21, 2017 at 9:24 AM, Paolo Margara <paolo.margara(a)polito.it>
wrote:
The support for custom mount options has been introduced here
https://gerrit.ovirt.org/#/c/57787/ so it should be available since
ovirt-hosted-engine-ha-1.3.5.6 and so you already have it on 3.6.7
In the mean time we had a lot of improvements for the hyperconverged
scenario, that's why I'm strongly suggesting you to upgrade to 4.1.
3002
days inactive
3006
days old
6 comments
3 participants
participants (3)
-
Paolo Margara -
Simone Tiraboschi -
Yedidyah Bar David