2:10 p.m.
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi,
At the beginning of february I successfully installed oVirt 3.6.2 (with hos=
ted engine) on 3 hosts, which are using 1 storage server with GlusterFS.
2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd h=
ost (AMD CPU) was added as host from oVirt web administration panel, withou=
t hosted engine deployment (I don't want the engine running on this host).
About 10 days ago I tried to reboot my oVirt environment (i.e. going to glo=
bal maintenance, shutting down the engine, turning off all the hosts, start=
ing them again, then setting maintenance mode to "none").
After the reboot, everything was fine with the Intel hosts and the hosted e=
ngine, but AMD host (the one without HA) was not operational.
I tryed to activate it, buti t failed with the following error:
"Host failed to attach one of Storage Domains attached to it."
If I log into my AMD host and I check the logs, I see that the storage doma=
in which is not mounted is the one of the hosted engine (but this could be =
correct, since this host won't run the hosted engine).
From /var/log/vdsm/vdsm.log:
Thread-29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(=
_produceDomain) Producing domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainC=
ache::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7=
e33c1125cd
Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainC=
ache::(_findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e=
33c1125cd
Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.OperationMutex=
::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::290::Storage.Misc.excCmd::(=
cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --=
config ' devices { preferred_names =3D ["^/dev/mapper/"]
ignore_suspended_d=
evices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 filter =3D [=
'\''r|.*|'\'' ] } global { locking_type=3D1
prioritise_write_locks=3D1 =
wait_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min =3D 50 retain=
_days =3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignores=
kippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_cou=
nt,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd=
4-9ff7-b7e33c1125cd (cwd None)
Thread-29::DEBUG::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(=
cmd) FAILED: <err> =3D ' WARNING: lvmetad is running but disabled. Restart=
lvmetad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33=
c1125cd" not found\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b=
7e33c1125cd\n'; <rc> =3D 5
Thread-29::WARNING::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloa=
dvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Re=
start lvmetad before enabling it!', ' Volume group "6fb10a49-5f1c-4bd4-9ff=
7-b7e33c1125cd" not found', ' Cannot process volume group 6fb10a49-5f1c-4b=
d4-9ff7-b7e33c1125cd']
Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.OperationMutex=
::(_reloadvgs) Operation 'lvm reload operation' released the operation mute=
x
Thread-29::ERROR::2016-02-25 11:44:01,245::sdc::145::Storage.StorageDomainC=
ache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-=
4bd4-9ff7-b7e33c1125cd',)
Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor::(=
_monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c=
d
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapp=
er
value =3D meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain =3D sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-=
4bd4-9ff7-b7e33c1125cd',)
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.Task=
Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:=
:moving from state init -> state preparing
jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::48::dispatcher=
::(wrapper) Run and protect: repoStats(options=3DNone)
jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::51::dispatcher=
::(wrapper) Run and protect: repoStats, Return response: {u'5f7991ba-fdf8-4=
b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'version':
3, 'acquired=
': True, 'delay': '0.00056349', 'lastCheck': '7.7',
'valid': True}, u'6fb10=
a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True,
'version': =
-1, 'acquired': False, 'delay': '0', 'lastCheck':
'2.0', 'valid': False}, u=
'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual':
True, 'versio=
n': 0, 'acquired': True, 'delay': '0.000561865',
'lastCheck': '8.4', 'valid=
': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0,
'actual': Tr=
ue, 'version': 3, 'acquired': True, 'delay':
'0.000227759', 'lastCheck': '8=
.7', 'valid': True}}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.Tas=
kManager.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::fin=
ished: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0,
'actual': True=
, 'version': 3, 'acquired': True, 'delay': '0.00056349',
'lastCheck': '7.7'=
, 'valid': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd':
{'code': 358, 'a=
ctual': True, 'version': -1, 'acquired': False, 'delay':
'0', 'lastCheck': =
'2.0', 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c':
{'code': 0=
, 'actual': True, 'version': 0, 'acquired': True, 'delay':
'0.000561865', '=
lastCheck': '8.4', 'valid': True},
u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a':=
{'code': 0, 'actual': True, 'version': 3, 'acquired':
True, 'delay': '0.00=
0227759', 'lastCheck': '8.7', 'valid': True}}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::595::Storage.Task=
Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:=
:moving from state preparing -> state finished
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager::940::S=
torage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} res=
ources {}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::resourceManager::977::S=
torage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::task::993::Storage.Task=
Manager.Task::(_decref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref =
0 aborting False
Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0=
b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=
=3D1 (cwd None)
Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n336 b=
ytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D 0
Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-=
060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=
=3D1 (cwd None)
Thread-31::DEBUG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 b=
ytes (339 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D 0
Thread-28::DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40=
-9974-c7adcd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 =
count=3D1 (cwd None)
Thread-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482 b=
ytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D 0
Other commands (executed on the host having the problem) which probably giv=
e useful information:
# vdsClient -s 0 getConnectedStoragePoolsList
00000001-0001-0001-0001-00000000020e
# vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020e
name =3D No Description
isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isod=
omain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1=
11111111111
pool_status =3D connected
lver =3D 6
spm_id =3D 2
master_uuid =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879
version =3D 3
domains =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f=
1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Activ=
e,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active
type =3D GLUSTERFS
master_ver =3D 1
5f7991ba-fdf8-4b40-9974-c7adcd4da879 =3D {'status': 'Active',
'disk=
free': '6374172262400', 'isoprefix': '', 'alerts': [],
'disktotal': '699543=
6371968', 'version': 3}
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd =3D {'status': 'Active',
'isop=
refix': '', 'alerts': [], 'version': -1}
e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active',
'disk=
free': '224145833984', 'isoprefix': '', 'alerts': [],
'disktotal': '2363171=
79904', 'version': 3}
5efea9c7-c4ec-44d4-a283-060d4c83303c =3D {'status': 'Active',
'disk=
free': '6374172262400', 'isoprefix':
'/rhev/data-center/mnt/glusterSD/srv-s=
tor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-=
1111-1111-1111-111111111111', 'alerts': [], 'disktotal':
'6995436371968', '=
version': 0}
# vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
Storage domain does not exist: ('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)
If I run this last command on one of the working hosts:
# vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
uuid =3D 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
version =3D 3
role =3D Regular
remotePath =3D srv-stor-01-ib0:/ovirtengine
type =3D GLUSTERFS
class =3D Data
pool =3D ['00000001-0001-0001-0001-00000000020e']
name =3D hosted_storage
(please note: this is the storage domain used for my hosted engine)
If I run "mount" on my AMD host (the one with the problem):
# mount
...
srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto=
r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou=
p_id=3D0,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i=
b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def=
ault_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib=
0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau=
lt_permissions,allow_other,max_read=3D131072)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,=
mode=3D700)
If I run "mount" on one of the Intel hosts (currently working):
# mount
...
srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01=
-ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0=
,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto=
r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou=
p_id=3D0,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i=
b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def=
ault_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib=
0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau=
lt_permissions,allow_other,max_read=3D131072)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D3272288k,m=
ode=3D700)
The only difference in "mount" is that the hosted-engine storage domain is =
not loaded on the host which should not run the engine. The other domains a=
re mounted correctly.
What could I do to solve this issue?
Best regards,
Giuseppe
--
Giuseppe Berellini
PTV SISTeMA
www.sistemaits.com<http://www.sistemaits.com/>
facebook.com/sistemaits<https://www.facebook.com/sistemaits>
linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzion...
-l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l->
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type
content=
=3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator
content=3D"Micros=
oft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.StileMessaggioDiPostaElettronica17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DIT
link=3D"#0563C1" v=
link=3D"#954F72"><div class=3DWordSection1><p
class=3DMsoNormal>Hi,<o:p></o=
:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>At th=
e beginning of february I successfully installed oVirt 3.6.2 (with hosted e=
ngine) on 3 hosts, which are using 1 storage server with GlusterFS.<o:p></o=
:p></p><p class=3DMsoNormal>2 hosts (with Intel CPU) are using HA and are
h=
osting the engine; the 3rd host (AMD CPU) was added as host from oVirt web =
administration panel, without hosted engine deployment (I don’t want =
the engine running on this host).<o:p></o:p></p><p
class=3DMsoNormal><o:p>&=
nbsp;</o:p></p><p class=3DMsoNormal>About 10 days ago I tried to reboot
my =
oVirt environment (i.e. going to global maintenance, shutting down the engi=
ne, turning off all the hosts, starting them again, then setting maintenanc=
e mode to “none”).<o:p></o:p></p><p
class=3DMsoNormal>After the=
reboot, everything was fine with the Intel hosts and the hosted engine, bu=
t AMD host (the one without HA) was not operational.<o:p></o:p></p><p
class=
=3DMsoNormal>I tryed to activate it, buti t failed with the following error=
:<o:p></o:p></p><p
class=3DMsoNormal> &n=
bsp; “Host failed to attach one of Storage Domains attached to it.=
221;<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3DMso=
Normal>If I log into my AMD host and I check the logs, I see that the stora=
ge domain which is not mounted is the one of the hosted engine (but this co=
uld be correct, since this host won’t run the hosted engine).<o:p></o=
:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>From =
/var/log/vdsm/vdsm.log:<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p=
</p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>Thread-=
29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(_produc=
eDomain) Producing domain
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>Thr=
ead-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainCach=
e::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7e33=
c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-famil=
y:"Courier New"'>Thread-29::ERROR::2016-02-25
11:44:01,158::sdc::156::Stora=
ge.StorageDomainCache::(_findUnfetchedDomain) looking for domain 6fb10a49-5=
f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoNormal><span =
style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25
11:44:01,1=
59::lvm::370::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload op=
eration' got the operation mutex<o:p></o:p></span></p><p
class=3DMsoNormal>=
<span style=3D'font-family:"Courier
New"'>Thread-29::DEBUG::2016-02-25 11:4=
4:01,159::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list =
0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_name=
s =3D ["^/dev/mapper/"] ignore_suspended_devices=3D1 write_cache_=
state=3D0 disable_after_error_count=3D3 filter =3D [ '\''r|.*|'\''
] } =
; global { locking_type=3D1 prioritise_write_locks=3D1 wa=
it_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min =
=3D 50 retain_days =3D 0 } ' --noheadings --units b --nosuffix --sepa=
rator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,ex=
tent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_nam=
e 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd (cwd
None)<o:p></o:p></span></p><p c=
lass=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Thread-29::DEBUG=
::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <=
;err> =3D ' WARNING: lvmetad is running but disabled. Restart lvme=
tad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-=
b7e33c1125cd" not found\n Cannot process volume group 6fb10a49-5=
f1c-4bd4-9ff7-b7e33c1125cd\n'; <rc> =3D
5<o:p></o:p></span></p><p cla=
ss=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Thread-29::WARNING=
::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloadvgs) lvm vgs fail=
ed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad=
before enabling it!', ' Volume group
"6fb10a49-5f1c-4bd4-9ff7-b=
7e33c1125cd" not found', ' Cannot process volume group
6fb10a49-=
5f1c-4bd4-9ff7-b7e33c1125cd']<o:p></o:p></span></p><p
class=3DMsoNormal><sp=
an style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25
11:44:0=
1,225::lvm::415::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload=
operation' released the operation
mutex<o:p></o:p></span></p><p class=3DMs=
oNormal><span style=3D'font-family:"Courier
New"'>Thread-29::ERROR::2016-02=
-25 11:44:01,245::sdc::145::Storage.StorageDomainCache::(_findDomain) domai=
n 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not
found<o:p></o:p></span></p><p cl=
ass=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Traceback (most r=
ecent call last):<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'=
font-family:"Courier New"'> File
"/usr/share/vdsm/storage/sdc.py=
", line 143, in _findDomain<o:p></o:p></span></p><p
class=3DMsoNormal>=
<span style=3D'font-family:"Courier
New"'> dom =3D findMe=
thod(sdUUID)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-=
family:"Courier New"'> File
"/usr/share/vdsm/storage/sdc.py"=
;, line 173, in _findUnfetchedDomain<o:p></o:p></span></p><p
class=3DMsoNor=
mal><span style=3D'font-family:"Courier
New"'> raise se.S=
torageDomainDoesNotExist(sdUUID)<o:p></o:p></span></p><p
class=3DMsoNormal>=
<span style=3D'font-family:"Courier
New"'>StorageDomainDoesNotExist: Storag=
e domain does not exist:
(u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o=
:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"=
'>Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor:=
:(_monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c112=
5cd<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'>Traceback (most recent call
last):<o:p></o:p></span></p><p cla=
ss=3DMsoNormal><span style=3D'font-family:"Courier
New"'> File "=
/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain<o:p><=
/o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier Ne=
w"'>
self._produceDomain()<o:p></o:p></span></p><p class=
=3DMsoNormal><span style=3D'font-family:"Courier
New"'> File "/u=
sr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapper<o:=
p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier=
New"'> value =3D meth(self, *a,
**kw)<o:p></o:p></span><=
/p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> Fi=
le "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDom=
ain<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'> self.domain =3D
sdCache.produce(self.sdUUID=
)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cou=
rier New"'> File "/usr/share/vdsm/storage/sdc.py",
line 100=
, in produce<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-=
family:"Courier New"'>
domain.getRealDomain()<o:p></o:p><=
/span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&n=
bsp; File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDo=
main<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"=
Courier New"'> return
self._cache._realProduce(self._sdUU=
ID)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'> File
"/usr/share/vdsm/storage/sdc.py", line 1=
24, in _realProduce<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier New"'> domain
=3D self._findDoma=
in(sdUUID)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-fa=
mily:"Courier New"'> File
"/usr/share/vdsm/storage/sdc.py",=
line 143, in _findDomain<o:p></o:p></span></p><p
class=3DMsoNormal><span s=
tyle=3D'font-family:"Courier New"'> dom
=3D findMethod(sd=
UUID)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:=
"Courier New"'> File
"/usr/share/vdsm/storage/sdc.py", line=
173, in _findUnfetchedDomain<o:p></o:p></span></p><p
class=3DMsoNormal><sp=
an style=3D'font-family:"Courier New"'>
raise se.StorageD=
omainDoesNotExist(sdUUID)<o:p></o:p></span></p><p
class=3DMsoNormal><span s=
tyle=3D'font-family:"Courier New"'>StorageDomainDoesNotExist: Storage
domai=
n does not exist:
(u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></sp=
an></p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>jsonr=
pc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.TaskManag=
er.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::movi=
ng from state init -> state
preparing<o:p></o:p></span></p><p class=3DMs=
oNormal><span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::INFO:=
:2016-02-25 11:44:03,293::logUtils::48::dispatcher::(wrapper) Run and prote=
ct: repoStats(options=3DNone)<o:p></o:p></span></p><p
class=3DMsoNormal><sp=
an style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::INFO::2016-02-25=
11:44:03,293::logUtils::51::dispatcher::(wrapper) Run and protect: repoSta=
ts, Return response: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0,
=
'actual': True, 'version': 3, 'acquired': True, 'delay':
'0.00056349', 'las=
tCheck': '7.7', 'valid': True},
u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'=
code': 358, 'actual': True, 'version': -1, 'acquired': False,
'delay': '0',=
'lastCheck': '2.0', 'valid': False},
u'5efea9c7-c4ec-44d4-a283-060d4c83303=
c': {'code': 0, 'actual': True, 'version': 0,
'acquired': True, 'delay': '0=
.000561865', 'lastCheck': '8.4', 'valid': True},
u'e84c6a1a-9f82-4fa6-9a3b-=
0b0bc0330d9a': {'code': 0, 'actual': True, 'version': 3,
'acquired': True, =
'delay': '0.000227759', 'lastCheck': '8.7',
'valid': True}}<o:p></o:p></spa=
n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>jsonrp=
c.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.TaskManag=
er.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::finished:=
{u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual':
True, 'ver=
sion': 3, 'acquired': True, 'delay': '0.00056349',
'lastCheck': '7.7', 'val=
id': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358,
'actual'=
: True, 'version': -1, 'acquired': False, 'delay': '0',
'lastCheck': '2.0',=
'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c':
{'code': 0, 'act=
ual': True, 'version': 0, 'acquired': True, 'delay':
'0.000561865', 'lastCh=
eck': '8.4', 'valid': True},
u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'cod=
e': 0, 'actual': True, 'version': 3, 'acquired': True,
'delay': '0.00022775=
9', 'lastCheck': '8.7', 'valid':
True}}<o:p></o:p></span></p><p class=3DMso=
Normal><span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::DEBUG:=
:2016-02-25 11:44:03,294::task::595::Storage.TaskManager.Task::(_updateStat=
e) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::moving from state prepari=
ng -> state finished<o:p></o:p></span></p><p
class=3DMsoNormal><span sty=
le=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:4=
4:03,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)=
Owner.releaseAll requests {} resources
{}<o:p></o:p></span></p><p class=3D=
MsoNormal><span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::DEB=
UG::2016-02-25 11:44:03,295::resourceManager::977::Storage.ResourceManager.=
Owner::(cancelAll) Owner.cancelAll requests
{}<o:p></o:p></span></p><p clas=
s=3DMsoNormal><span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0:=
:DEBUG::2016-02-25 11:44:03,295::task::993::Storage.TaskManager.Task::(_dec=
ref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref 0 aborting False<o:=
p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier=
New"'>Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc=
.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/=
rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6=
-9a3b-0b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 =
count=3D1 (cwd None)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier New"'>Thread-30::DEBUG::2016-02-25
11:44:04,630::f=
ileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =3D '0=
+1 records in\n0+1 records out\n336 bytes (336 B) copied, 0.000286148 s, 1.=
2 MB/s\n'; <rc> =3D 0<o:p></o:p></span></p><p
class=3DMsoNormal><span=
style=3D'font-family:"Courier New"'>Thread-31::DEBUG::2016-02-25
11:44:04,=
925::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cp=
u-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glusterSD/srv-stor-01-ib=
0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/dom_md/metadata iflag=3Dd=
irect of=3D/dev/null bs=3D4096 count=3D1 (cwd
None)<o:p></o:p></span></p><p=
class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Thread-31::DEB=
UG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd::(getReadDela=
y) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 bytes (33=
9 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D
0<o:p></o:p></span></=
p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Thread-28:=
:DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd::(getRead=
Delay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/=
mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40-9974-c7ad=
cd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 =
(cwd None)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-fa=
mily:"Courier New"'>Thread-28::DEBUG::2016-02-25
11:44:05,606::fileSD::173:=
:Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =3D '0+1 records =
in\n0+1 records out\n482 bytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; =
<rc> =3D 0<o:p></o:p></span></p><p
class=3DMsoNormal><o:p> </o:p=
</p><p
class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal><o:p>&n=
bsp;</o:p></p><p
class=3DMsoNormal>Other commands (executed on the host hav=
ing the problem) which probably give useful
information:<o:p></o:p></p><p c=
lass=3DMsoNormal><span style=3D'font-family:"Courier New"'>#
vdsClient -s 0=
getConnectedStoragePoolsList<o:p></o:p></span></p><p
class=3DMsoNormal><sp=
an style=3D'font-family:"Courier
New"'>00000001-0001-0001-0001-00000000020e=
<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cour=
ier New"'><o:p> </o:p></span></p><p
class=3DMsoNormal><span style=3D'f=
ont-family:"Courier New"'># vdsClient -s 0 getStoragePoolInfo
00000001-0001=
-0001-0001-00000000020e<o:p></o:p></span></p><p
class=3DMsoNormal><span sty=
le=3D'font-family:"Courier
New"'> =
name =3D No Description<o:p></o:p></span></p><p
class=3DMsoNormal><span st=
yle=3D'font-family:"Courier
New"'>  =
; isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/=
5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1111111=
11111<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:=
"Courier
New"'>
pool_status =3D c=
onnected<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-fami=
ly:"Courier
New"'>
lver =3D 6<o:p=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier =
New"'>
spm_id =3D 2<o:p></o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&nbs=
p; master_uuid =3D
5f7991ba-fdf8-4b40-9=
974-c7adcd4da879<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'f=
ont-family:"Courier
New"'>
versio=
n =3D 3<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-famil=
y:"Courier
New"'>
domains =3D 5f7=
991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f1c-4bd4-9ff7-b7e33c1125=
cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Active,e84c6a1a-9f82-4fa6-9a=
3b-0b0bc0330d9a:Active<o:p></o:p></span></p><p
class=3DMsoNormal><span styl=
e=3D'font-family:"Courier
New"'>
=
type =3D GLUSTERFS<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D=
'font-family:"Courier
New"'>
mast=
er_ver =3D 1<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-=
family:"Courier
New"'>
5f7991ba-f=
df8-4b40-9974-c7adcd4da879 =3D {'status': 'Active', 'diskfree':
'6374172262=
400', 'isoprefix': '', 'alerts': [], 'disktotal':
'6995436371968', 'version=
': 3}<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:=
"Courier
New"'>
6fb10a49-5f1c-4bd=
4-9ff7-b7e33c1125cd =3D {'status': 'Active', 'isoprefix':
'', 'alerts': [],=
'version': -1}<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'fo=
nt-family:"Courier
New"'>
e84c6a1=
a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active', 'diskfree':
'2241458=
33984', 'isoprefix': '', 'alerts': [], 'disktotal':
'236317179904', 'versio=
n': 3}<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-family=
:"Courier
New"'>
5efea9c7-c4ec-44=
d4-a283-060d4c83303c =3D {'status': 'Active', 'diskfree':
'6374172262400', =
'isoprefix': '/rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5e=
fea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-111111111=
111', 'alerts': [], 'disktotal': '6995436371968',
'version': 0}<o:p></o:p><=
/span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'><o=
:p> </o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"=
Courier New"'># vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7=
-b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font=
-family:"Courier New"'>Storage domain does not exist:
('6fb10a49-5f1c-4bd4-=
9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p
class=3DMsoNormal><o:p> <=
/o:p></p><p class=3DMsoNormal>If I run this last command on one of the
work=
ing hosts:<o:p></o:p></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'># vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-=
b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-=
family:"Courier
New"'>
uuid =3D 6=
fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoNorm=
al><span style=3D'font-family:"Courier
New"'> =
version =3D 3<o:p></o:p></span></p><p
class=3DMsoNormal><span =
style=3D'font-family:"Courier
New"'> &nb=
sp; role =3D Regular<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier
New"'>
r=
emotePath =3D
srv-stor-01-ib0:/ovirtengine<o:p></o:p></span></p><p class=3D=
MsoNormal><span style=3D'font-family:"Courier
New"'>  =
; type =3D
GLUSTERFS<o:p></o:p></span></p><p class=3DMsoN=
ormal><span style=3D'font-family:"Courier
New"'> &nb=
sp; class =3D Data<o:p></o:p></span></p><p
class=3DMsoNormal><s=
pan style=3D'font-family:"Courier
New"'>  =
; pool =3D
['00000001-0001-0001-0001-00000000020e']<o:p></o:p></span>=
</p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> &n=
bsp; name =3D
hosted_storage<o:p></o:p></span=
</p><p
class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>(please=
note: this is the storage domain used for my hosted
engine)<o:p></o:p></p>=
<p class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal><o:p> <=
/o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>If =
I run “mount” on my AMD host (the one with the
problem):<o:p></=
o:p></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'># mo=
unt<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'>...<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'f=
ont-family:"Courier New"'>srv-stor-01-ib0:/virtualmachines on
/rhev/data-ce=
nter/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines type fuse.glusterfs (rw=
,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_other,max_read=
=3D131072)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-fa=
mily:"Courier New"'>srv-stor-01-ib0:/isodomain on
/rhev/data-center/mnt/glu=
sterSD/srv-stor-01-ib0:_isodomain type fuse.glusterfs (rw,relatime,user_id=
=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></=
o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New=
"'>srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01=
-ib0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,de=
fault_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p
cl=
ass=3DMsoNormal><span style=3D'font-family:"Courier New"'>tmpfs
on /run/use=
r/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,mode=3D700)<o:p><=
/o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier Ne=
w"'><o:p> </o:p></span></p><p
class=3DMsoNormal>If I run “mount&=
#8221; on one of the Intel hosts (currently
working):<o:p></o:p></p><p clas=
s=3DMsoNormal><span style=3D'font-family:"Courier New"'>#
mount<o:p></o:p><=
/span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>..=
.<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cou=
rier New"'>srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/=
srv-stor-01-ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,g=
roup_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>srv=
-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-stor-0=
1-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,group_i=
d=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></span><=
/p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>srv-stor-=
01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodo=
main type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_perm=
issions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p
class=3DMsoN=
ormal><span style=3D'font-family:"Courier
New"'>srv-stor-01-ib0:/ssd-pcie o=
n /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie type fuse.glust=
erfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_other,=
max_read=3D131072)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D=
'font-family:"Courier New"'>tmpfs on /run/user/0 type tmpfs
(rw,nosuid,node=
v,relatime,size=3D3272288k,mode=3D700)<o:p></o:p></span></p><p
class=3DMsoN=
ormal><span style=3D'font-family:"Courier
New"'><o:p> </o:p></span></p=
<p class=3DMsoNormal>The only difference in
“mount” is that th=
e hosted-engine storage domain is not loaded
on the host which should not r=
un the engine. The other domains are mounted
correctly.<o:p></o:p></p><p cl=
ass=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>What could I do t=
o solve this issue?<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p=
<p class=3DMsoNormal>Best
regards,<o:p></o:p></p><p class=3DMsoNormal>&nbs=
p;
Giuseppe<o:p></o:p></p><p class=3DMs=
oNormal><o:p> </o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p c=
lass=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:aut=
o'><span
style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-farea=
st-language:IT'>--<br>Giuseppe
Berellini<o:p></o:p></span></p><p class=3DMs=
oNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
=
style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;color:red;mso-fare=
ast-language:IT'>PTV SISTeMA<o:p></o:p></span></p><p
class=3DMsoNormal styl=
e=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style=3D'fon=
t-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:IT'><a
hr=
ef=3D"http://www.sistemaits.com/"><span
style=3D'color:black;text-decoratio=
n:none'>www.sistemaits.com</span></a><o:p></o:p></span></p><p
class=3DMsoNo=
rmal><span
style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-far=
east-language:IT'><a
href=3D"https://www.facebook.com/sistemaits"><span sty=
le=3D'color:black;text-decoration:none'>facebook.com/sistemait...
br><a href=3D"https://www.linkedin.com/company/sistema-soluzioni-per-l-inge=
gneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-"><span
style=3D'colo=
r:black;text-decoration:none'>linkedin.com/SISTeMA</span><...
an><o:p></o:p></p></div></body></html>=
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_--
4:02 p.m.
New subject: R: One host failed to attach one of Storage Domains after reboot of all hosts
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi,
about one hour ago my AMD host came back up, after more than 10 days being =
down.
Apart from checking the logs (which I suppose didn't help in solving my pro=
blem), I enabled NFS share on my ISO domain.
I'm still not able to understand how that could help.
I would be really happy to better understand what happened! :-)
If anyone has ideas/explanations to share, you are welcome! :-)
Best regards,
Giuseppe
--
Giuseppe Berellini
PTV SISTeMA
www.sistemaits.com<http://www.sistemaits.com/>
facebook.com/sistemaits<https://www.facebook.com/sistemaits>
linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzion...
-l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l->
Da: users-bounces(a)ovirt.org [mailto:users-bounces@ovirt.org] Per conto di G=
iuseppe Berellini
Inviato: gioved=EC 25 febbraio 2016 12:10
A: users(a)ovirt.org
Oggetto: [ovirt-users] One host failed to attach one of Storage Domains aft=
er reboot of all hosts
Hi,
At the beginning of february I successfully installed oVirt 3.6.2 (with hos=
ted engine) on 3 hosts, which are using 1 storage server with GlusterFS.
2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd h=
ost (AMD CPU) was added as host from oVirt web administration panel, withou=
t hosted engine deployment (I don't want the engine running on this host).
About 10 days ago I tried to reboot my oVirt environment (i.e. going to glo=
bal maintenance, shutting down the engine, turning off all the hosts, start=
ing them again, then setting maintenance mode to "none").
After the reboot, everything was fine with the Intel hosts and the hosted e=
ngine, but AMD host (the one without HA) was not operational.
I tryed to activate it, buti t failed with the following error:
"Host failed to attach one of Storage Domains attached to it."
If I log into my AMD host and I check the logs, I see that the storage doma=
in which is not mounted is the one of the hosted engine (but this could be =
correct, since this host won't run the hosted engine).
From /var/log/vdsm/vdsm.log:
Thread-29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(=
_produceDomain) Producing domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainC=
ache::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7=
e33c1125cd
Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainC=
ache::(_findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e=
33c1125cd
Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.OperationMutex=
::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::290::Storage.Misc.excCmd::(=
cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --=
config ' devices { preferred_names =3D ["^/dev/mapper/"]
ignore_suspended_d=
evices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 filter =3D [=
'\''r|.*|'\'' ] } global { locking_type=3D1
prioritise_write_locks=3D1 =
wait_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min =3D 50 retain=
_days =3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignores=
kippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_cou=
nt,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd=
4-9ff7-b7e33c1125cd (cwd None)
Thread-29::DEBUG::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(=
cmd) FAILED: <err> =3D ' WARNING: lvmetad is running but disabled. Restart=
lvmetad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33=
c1125cd" not found\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b=
7e33c1125cd\n'; <rc> =3D 5
Thread-29::WARNING::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloa=
dvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Re=
start lvmetad before enabling it!', ' Volume group "6fb10a49-5f1c-4bd4-9ff=
7-b7e33c1125cd" not found', ' Cannot process volume group 6fb10a49-5f1c-4b=
d4-9ff7-b7e33c1125cd']
Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.OperationMutex=
::(_reloadvgs) Operation 'lvm reload operation' released the operation mute=
x
Thread-29::ERROR::2016-02-25 11:44:01,245::sdc::145::Storage.StorageDomainC=
ache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not found
Traceback (most recent call last):
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-=
4bd4-9ff7-b7e33c1125cd',)
Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor::(=
_monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c=
d
Traceback (most recent call last):
File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain
self._produceDomain()
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapp=
er
value =3D meth(self, *a, **kw)
File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain
self.domain =3D sdCache.produce(self.sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 100, in produce
domain.getRealDomain()
File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
return self._cache._realProduce(self._sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce
domain =3D self._findDomain(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain
dom =3D findMethod(sdUUID)
File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain
raise se.StorageDomainDoesNotExist(sdUUID)
StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-=
4bd4-9ff7-b7e33c1125cd',)
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.Task=
Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:=
:moving from state init -> state preparing
jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::48::dispatcher=
::(wrapper) Run and protect: repoStats(options=3DNone)
jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::51::dispatcher=
::(wrapper) Run and protect: repoStats, Return response: {u'5f7991ba-fdf8-4=
b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'version':
3, 'acquired=
': True, 'delay': '0.00056349', 'lastCheck': '7.7',
'valid': True}, u'6fb10=
a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True,
'version': =
-1, 'acquired': False, 'delay': '0', 'lastCheck':
'2.0', 'valid': False}, u=
'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual':
True, 'versio=
n': 0, 'acquired': True, 'delay': '0.000561865',
'lastCheck': '8.4', 'valid=
': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0,
'actual': Tr=
ue, 'version': 3, 'acquired': True, 'delay':
'0.000227759', 'lastCheck': '8=
.7', 'valid': True}}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.Tas=
kManager.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::fin=
ished: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0,
'actual': True=
, 'version': 3, 'acquired': True, 'delay': '0.00056349',
'lastCheck': '7.7'=
, 'valid': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd':
{'code': 358, 'a=
ctual': True, 'version': -1, 'acquired': False, 'delay':
'0', 'lastCheck': =
'2.0', 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c':
{'code': 0=
, 'actual': True, 'version': 0, 'acquired': True, 'delay':
'0.000561865', '=
lastCheck': '8.4', 'valid': True},
u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a':=
{'code': 0, 'actual': True, 'version': 3, 'acquired':
True, 'delay': '0.00=
0227759', 'lastCheck': '8.7', 'valid': True}}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::595::Storage.Task=
Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:=
:moving from state preparing -> state finished
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager::940::S=
torage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} res=
ources {}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::resourceManager::977::S=
torage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::task::993::Storage.Task=
Manager.Task::(_decref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref =
0 aborting False
Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0=
b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=
=3D1 (cwd None)
Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n336 b=
ytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D 0
Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-=
060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=
=3D1 (cwd None)
Thread-31::DEBUG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 b=
ytes (339 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D 0
Thread-28::DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da=
ta-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40=
-9974-c7adcd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 =
count=3D1 (cwd None)
Thread-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd=
::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482 b=
ytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D 0
Other commands (executed on the host having the problem) which probably giv=
e useful information:
# vdsClient -s 0 getConnectedStoragePoolsList
00000001-0001-0001-0001-00000000020e
# vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020e
name =3D No Description
isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isod=
omain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1=
11111111111
pool_status =3D connected
lver =3D 6
spm_id =3D 2
master_uuid =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879
version =3D 3
domains =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f=
1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Activ=
e,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active
type =3D GLUSTERFS
master_ver =3D 1
5f7991ba-fdf8-4b40-9974-c7adcd4da879 =3D {'status': 'Active',
'disk=
free': '6374172262400', 'isoprefix': '', 'alerts': [],
'disktotal': '699543=
6371968', 'version': 3}
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd =3D {'status': 'Active',
'isop=
refix': '', 'alerts': [], 'version': -1}
e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active',
'disk=
free': '224145833984', 'isoprefix': '', 'alerts': [],
'disktotal': '2363171=
79904', 'version': 3}
5efea9c7-c4ec-44d4-a283-060d4c83303c =3D {'status': 'Active',
'disk=
free': '6374172262400', 'isoprefix':
'/rhev/data-center/mnt/glusterSD/srv-s=
tor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-=
1111-1111-1111-111111111111', 'alerts': [], 'disktotal':
'6995436371968', '=
version': 0}
# vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
Storage domain does not exist: ('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)
If I run this last command on one of the working hosts:
# vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
uuid =3D 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd
version =3D 3
role =3D Regular
remotePath =3D srv-stor-01-ib0:/ovirtengine
type =3D GLUSTERFS
class =3D Data
pool =3D ['00000001-0001-0001-0001-00000000020e']
name =3D hosted_storage
(please note: this is the storage domain used for my hosted engine)
If I run "mount" on my AMD host (the one with the problem):
# mount
...
srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto=
r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou=
p_id=3D0,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i=
b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def=
ault_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib=
0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau=
lt_permissions,allow_other,max_read=3D131072)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,=
mode=3D700)
If I run "mount" on one of the Intel hosts (currently working):
# mount
...
srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01=
-ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0=
,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto=
r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou=
p_id=3D0,default_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i=
b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def=
ault_permissions,allow_other,max_read=3D131072)
srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib=
0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau=
lt_permissions,allow_other,max_read=3D131072)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D3272288k,m=
ode=3D700)
The only difference in "mount" is that the hosted-engine storage domain is =
not loaded on the host which should not run the engine. The other domains a=
re mounted correctly.
What could I do to solve this issue?
Best regards,
Giuseppe
--
Giuseppe Berellini
PTV SISTeMA
www.sistemaits.com<http://www.sistemaits.com/>
facebook.com/sistemaits<https://www.facebook.com/sistemaits>
linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzion...
-l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l->
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html;
charset=3Diso-8859-=
1">
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta name=3DGenerator
content=3D"Microso=
ft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.StileMessaggioDiPostaElettronica17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.StileMessaggioDiPostaElettronica18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DIT
link=3D"#0563C1" v=
link=3D"#954F72"><div class=3DWordSection1><p
class=3DMsoNormal><span style=
=3D'color:#1F497D'>Hi,<o:p></o:p></span></p><p
class=3DMsoNormal><span styl=
e=3D'color:#1F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal><span =
style=3D'color:#1F497D'>about one hour ago my AMD host came back up, after =
more than 10 days being down.<o:p></o:p></span></p><p
class=3DMsoNormal><sp=
an style=3D'color:#1F497D'>Apart from checking the logs (which I suppose di=
dn’t help in solving my problem), I enabled NFS share on my ISO domai=
n.<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>=
<o:p> </o:p></span></p><p
class=3DMsoNormal><span style=3D'color:#1F49=
7D'>I’m still not able to understand how that could
help.<o:p></o:p><=
/span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>I would be rea=
lly happy to better understand what happened!
:-)<o:p></o:p></span></p><p c=
lass=3DMsoNormal><span style=3D'color:#1F497D'>If anyone has
ideas/explanat=
ions to share, you are welcome! :-)<o:p></o:p></span></p><p
class=3DMsoNorm=
al><span
style=3D'color:#1F497D'><o:p> </o:p></span></p><p
class=3DMso=
Normal><span style=3D'color:#1F497D'>Best
regards,<o:p></o:p></span></p><p =
class=3DMsoNormal><span
style=3D'color:#1F497D'> &nb=
sp; Giuseppe</span><span
style=3D'color:#1F497D'><o:p></o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'><o:p> </o:p=
</span></p><div><p class=3DMsoNormal
style=3D'mso-margin-top-alt:auto;mso-=
margin-bottom-alt:auto'><span
style=3D'font-size:10.0pt;font-family:"Arial"=
,sans-serif;color:#1F497D;mso-fareast-language:IT'>--<br>Giuseppe Berellini=
<o:p></o:p></span></p><p class=3DMsoNormal
style=3D'mso-margin-top-alt:auto=
;mso-margin-bottom-alt:auto'><span
style=3D'font-size:10.0pt;font-family:"A=
rial",sans-serif;color:red;mso-fareast-language:IT'>PTV
SISTeMA<o:p></o:p><=
/span></p><p class=3DMsoNormal
style=3D'mso-margin-top-alt:auto;mso-margin-=
bottom-alt:auto'><span
style=3D'font-size:10.0pt;font-family:"Arial",sans-s=
erif;color:#1F497D;mso-fareast-language:IT'><a
href=3D"http://www.sistemait=
s.com/"><span
style=3D'color:black;text-decoration:none'>www.sistemaits.com=
</span></a><o:p></o:p></span></p></div><p
class=3DMsoNormal><span style=3D'=
font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-l=
anguage:IT'><a
href=3D"https://www.facebook.com/sistemaits"><span style=3D'=
color:black;text-decoration:none'>facebook.com/sistemaits</span...
=
href=3D"https://www.linkedin.com/company/sistema-soluzioni-per-l-ing...
-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-"><span
style=3D'color:blac=
k;text-decoration:none'>linkedin.com/SISTeMA</span></a>...
an
style=3D'color:#1F497D'><o:p></o:p></span></p><div><div
style=3D'border:=
none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DM=
soNormal><b><span
style=3D'mso-fareast-language:IT'>Da:</span></b><span sty=
le=3D'mso-fareast-language:IT'> users-bounces(a)ovirt.org [mailto:users-bounc=
es(a)ovirt.org] <b>Per conto di </b>Giuseppe
Berellini<br><b>Inviato:</b> gio=
ved=EC 25 febbraio 2016 12:10<br><b>A:</b>
users@ovirt.org<br><b>Oggetto:</=
b> [ovirt-users] One host failed to attach one of Storage Domains after reb=
oot of all
hosts<o:p></o:p></span></p></div></div><p
class=3DMsoNormal><o:p=
</o:p></p><p
class=3DMsoNormal>Hi,<o:p></o:p></p><p class=3DMsoNorma=
l><o:p> </o:p></p><p class=3DMsoNormal>At the
beginning of february I =
successfully installed oVirt 3.6.2 (with hosted engine) on 3 hosts, which a=
re using 1 storage server with GlusterFS.<o:p></o:p></p><p
class=3DMsoNorma=
l>2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd=
host (AMD CPU) was added as host from oVirt web administration panel, with=
out hosted engine deployment (I don’t want the engine running on this=
host).<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3D=
MsoNormal>About 10 days ago I tried to reboot my oVirt environment (i.e. go=
ing to global maintenance, shutting down the engine, turning off all the ho=
sts, starting them again, then setting maintenance mode to “none̶=
1;).<o:p></o:p></p><p class=3DMsoNormal>After the reboot,
everything was fi=
ne with the Intel hosts and the hosted engine, but AMD host (the one withou=
t HA) was not operational.<o:p></o:p></p><p class=3DMsoNormal>I
tryed to ac=
tivate it, buti t failed with the following error:<o:p></o:p></p><p
class=
=3DMsoNormal>
“Host failed =
to attach one of Storage Domains attached to
it.”<o:p></o:p></p><p cl=
ass=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal>If I log into my =
AMD host and I check the logs, I see that the storage domain which is not m=
ounted is the one of the hosted engine (but this could be correct, since th=
is host won’t run the hosted engine).<o:p></o:p></p><p
class=3DMsoNor=
mal><o:p> </o:p></p><p class=3DMsoNormal>From
/var/log/vdsm/vdsm.log:<=
o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma=
l><span style=3D'font-family:"Courier
New"'>Thread-29::DEBUG::2016-02-25 11=
:44:01,157::monitor::322::Storage.Monitor::(_produceDomain) Producing domai=
n 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoN=
ormal><span style=3D'font-family:"Courier
New"'>Thread-29::ERROR::2016-02-2=
5 11:44:01,158::sdc::139::Storage.StorageDomainCache::(_findDomain) looking=
for unfetched domain
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span=
</p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>Thread-=
29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainCache::(=
_findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125=
cd<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Co=
urier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.Op=
erationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operati=
on mutex<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-fami=
ly:"Courier New"'>Thread-29::DEBUG::2016-02-25
11:44:01,159::lvm::290::Stor=
age.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /u=
sr/sbin/lvm vgs --config ' devices { preferred_names =3D ["^/dev/mappe=
r/"] ignore_suspended_devices=3D1 write_cache_state=3D0 disable_after_=
error_count=3D3 filter =3D [ '\''r|.*|'\'' ] } global
{ locking=
_type=3D1 prioritise_write_locks=3D1 wait_for_locks=3D1 u=
se_lvmetad=3D0 } backup { retain_min =3D 50 retain_days =
=3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskipped=
cluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tag=
s,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd4-9ff7=
-b7e33c1125cd (cwd None)<o:p></o:p></span></p><p
class=3DMsoNormal><span st=
yle=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25
11:44:01,223=
::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <err> =3D '
WARN=
ING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n=
Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd" not fo=
und\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c=
d\n'; <rc> =3D 5<o:p></o:p></span></p><p
class=3DMsoNormal><span styl=
e=3D'font-family:"Courier New"'>Thread-29::WARNING::2016-02-25
11:44:01,225=
::lvm::375::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING=
: lvmetad is running but disabled. Restart lvmetad before enabling it!', '&=
nbsp; Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd" not fou=
nd', ' Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125=
cd']<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"=
Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.=
OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the =
operation mutex<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'fo=
nt-family:"Courier New"'>Thread-29::ERROR::2016-02-25
11:44:01,245::sdc::14=
5::Storage.StorageDomainCache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7=
-b7e33c1125cd not found<o:p></o:p></span></p><p
class=3DMsoNormal><span sty=
le=3D'font-family:"Courier New"'>Traceback (most recent call
last):<o:p></o=
:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"=
'> File "/usr/share/vdsm/storage/sdc.py", line 143, in
_fin=
dDomain<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-famil=
y:"Courier New"'> dom =3D
findMethod(sdUUID)<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&nb=
sp; File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfe=
tchedDomain<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-f=
amily:"Courier New"'> raise
se.StorageDomainDoesNotExist(=
sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-famil=
y:"Courier New"'>StorageDomainDoesNotExist: Storage domain does not
exist: =
(u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p
class=3D=
MsoNormal><span style=3D'font-family:"Courier
New"'>Thread-29::ERROR::2016-=
02-25 11:44:01,246::monitor::276::Storage.Monitor::(_monitorDomain) Error m=
onitoring domain
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p>=
<p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Traceback (m=
ost recent call last):<o:p></o:p></span></p><p
class=3DMsoNormal><span styl=
e=3D'font-family:"Courier New"'> File
"/usr/share/vdsm/storage/m=
onitor.py", line 264, in
_monitorDomain<o:p></o:p></span></p><p class=
=3DMsoNormal><span style=3D'font-family:"Courier
New"'> s=
elf._produceDomain()<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier New"'> File
"/usr/lib/python2.7/site-pa=
ckages/vdsm/utils.py", line 767, in
wrapper<o:p></o:p></span></p><p cl=
ass=3DMsoNormal><span style=3D'font-family:"Courier
New"'>  =
; value =3D meth(self, *a, **kw)<o:p></o:p></span></p><p
class=3DMsoNormal>=
<span style=3D'font-family:"Courier New"'> File
"/usr/share/vdsm=
/storage/monitor.py", line 323, in
_produceDomain<o:p></o:p></span></p=
<p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>  =
; self.domain =3D
sdCache.produce(self.sdUUID)<o:p></o:p></span></p><=
p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> File &=
quot;/usr/share/vdsm/storage/sdc.py", line 100, in
produce<o:p></o:p><=
/span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&n=
bsp;
domain.getRealDomain()<o:p></o:p></span></p><p class=3DMso=
Normal><span style=3D'font-family:"Courier New"'>
File "/usr/sha=
re/vdsm/storage/sdc.py", line 52, in
getRealDomain<o:p></o:p></span></=
p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> &nbs=
p; return
self._cache._realProduce(self._sdUUID)<o:p></o:p></span></p=
<p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'> File=
"/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier N=
ew"'> domain =3D
self._findDomain(sdUUID)<o:p></o:p></spa=
n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> =
File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain<=
o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Couri=
er New"'> dom =3D
findMethod(sdUUID)<o:p></o:p></span></p=
<p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'> File=
"/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDom=
ain<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'> raise
se.StorageDomainDoesNotExist(sdUUID)<=
o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Couri=
er New"'>StorageDomainDoesNotExist: Storage domain does not exist:
(u'6fb10=
a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p
class=3DMsoNorma=
l><span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::DEBUG::2016=
-02-25 11:44:03,292::task::595::Storage.TaskManager.Task::(_updateState) Ta=
sk=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::moving from state init -> s=
tate preparing<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'fon=
t-family:"Courier New"'>jsonrpc.Executor/0::INFO::2016-02-25
11:44:03,293::=
logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=3DNo=
ne)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier New"'>jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::5=
1::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'5f=
7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual': True,
'version':=
3, 'acquired': True, 'delay': '0.00056349', 'lastCheck':
'7.7', 'valid': T=
rue}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358,
'actual': True=
, 'version': -1, 'acquired': False, 'delay': '0',
'lastCheck': '2.0', 'vali=
d': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0,
'actual': =
True, 'version': 0, 'acquired': True, 'delay':
'0.000561865', 'lastCheck': =
'8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a':
{'code': 0,=
'actual': True, 'version': 3, 'acquired': True, 'delay':
'0.000227759', 'l=
astCheck': '8.7', 'valid':
True}}<o:p></o:p></span></p><p class=3DMsoNormal=
<span style=3D'font-family:"Courier
New"'>jsonrpc.Executor/0::DEBUG::2016-=
02-25
11:44:03,294::task::1191::Storage.TaskManager.Task::(prepare) Task=3D=
`2862ba96-8080-4e74-a55a-cdf93326631a`::finished: {u'5f7991ba-fdf8-4b40-997=
4-c7adcd4da879': {'code': 0, 'actual': True, 'version': 3,
'acquired': True=
, 'delay': '0.00056349', 'lastCheck': '7.7',
'valid': True}, u'6fb10a49-5f1=
c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True,
'version': -1, 'ac=
quired': False, 'delay': '0', 'lastCheck': '2.0',
'valid': False}, u'5efea9=
c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual': True,
'version': 0, =
'acquired': True, 'delay': '0.000561865', 'lastCheck':
'8.4', 'valid': True=
}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0, 'actual':
True, 've=
rsion': 3, 'acquired': True, 'delay': '0.000227759',
'lastCheck': '8.7', 'v=
alid': True}}<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font=
-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25
11:44:03,294::=
task::595::Storage.TaskManager.Task::(_updateState) Task=3D`2862ba96-8080-4=
e74-a55a-cdf93326631a`::moving from state preparing -> state finished<o:=
p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier=
New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager:=
:940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests=
{} resources {}<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'f=
ont-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25
11:44:03,29=
5::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.c=
ancelAll requests {}<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25
11:44:=
03,295::task::993::Storage.TaskManager.Task::(_decref) Task=3D`2862ba96-808=
0-4e74-a55a-cdf93326631a`::ref 0 aborting
False<o:p></o:p></span></p><p cla=
ss=3DMsoNormal><span style=3D'font-family:"Courier
New"'>Thread-30::DEBUG::=
2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /=
usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glus=
terSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a/dom_md=
/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier N=
ew"'>Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.e=
xcCmd::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records=
out\n336 bytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D =
0<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cou=
rier New"'>Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.=
Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=
=3D/rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec=
-44d4-a283-060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D=
4096 count=3D1 (cwd None)<o:p></o:p></span></p><p
class=3DMsoNormal><span s=
tyle=3D'font-family:"Courier New"'>Thread-31::DEBUG::2016-02-25
11:44:04,95=
0::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =
=3D '0+1 records in\n0+1 records out\n339 bytes (339 B) copied, 0.0005884 s=
, 576 kB/s\n'; <rc> =3D
0<o:p></o:p></span></p><p class=3DMsoNormal><=
span style=3D'font-family:"Courier New"'>Thread-28::DEBUG::2016-02-25
11:44=
:05,583::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset =
--cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glusterSD/srv-stor-0=
1-ib0:_virtualmachines/5f7991ba-fdf8-4b40-9974-c7adcd4da879/dom_md/metadata=
iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd
None)<o:p></o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>Thre=
ad-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd::(g=
etReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482=
bytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D
0<o:p></o=
:p></span></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma=
l><o:p> </o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal>Other commands (executed on the host having the problem) which=
probably give useful information:<o:p></o:p></p><p
class=3DMsoNormal><span=
style=3D'font-family:"Courier New"'># vdsClient -s 0
getConnectedStoragePo=
olsList<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-famil=
y:"Courier
New"'>00000001-0001-0001-0001-00000000020e<o:p></o:p></span></p>=
<p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'><o:p> <=
/o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier Ne=
w"'># vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020=
e<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cou=
rier
New"'>
name =3D No Descripti=
on<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Co=
urier
New"'>
isoprefix =3D /rhev/=
data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a28=
3-060d4c83303c/images/11111111-1111-1111-1111-111111111111<o:p></o:p></span=
</p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'> &=
nbsp; pool_status =3D
connected<o:p></o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&nbs=
p; lver =3D
6<o:p></o:p></span></p><p c=
lass=3DMsoNormal><span style=3D'font-family:"Courier
New"'> &nbs=
p; spm_id =3D
2<o:p></o:p></span></p><p class=3DMso=
Normal><span style=3D'font-family:"Courier
New"'> &n=
bsp; master_uuid =3D
5f7991ba-fdf8-4b40-9974-c7adcd4da879<o:p><=
/o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier Ne=
w"'>
version =3D 3<o:p></o:p></sp=
an></p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'> =
; domains =3D
5f7991ba-fdf8-4b40-9974-c=
7adcd4da879:Active,6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4e=
c-44d4-a283-060d4c83303c:Active,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active=
<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cour=
ier
New"'>
type =3D GLUSTERFS<o:p=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier =
New"'>
master_ver =3D 1<o:p></o:p=
</span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>=
5f7991ba-fdf8-4b40-9974-c7adcd4d=
a879 =3D {'status': 'Active', 'diskfree': '6374172262400',
'isoprefix': '',=
'alerts': [], 'disktotal': '6995436371968', 'version':
3}<o:p></o:p></span=
</p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'> &=
nbsp;
6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd =
=3D {'status': 'Active', 'isoprefix': '',
'alerts': [], 'version': -1}<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier N=
ew"'>
e84c6a1a-9f82-4fa6-9a3b-0b0=
bc0330d9a =3D {'status': 'Active', 'diskfree':
'224145833984', 'isoprefix':=
'', 'alerts': [], 'disktotal': '236317179904',
'version': 3}<o:p></o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>&nbs=
p;
5efea9c7-c4ec-44d4-a283-060d4c83303c=
=3D {'status': 'Active', 'diskfree': '6374172262400',
'isoprefix': '/rhev/=
data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a28=
3-060d4c83303c/images/11111111-1111-1111-1111-111111111111', 'alerts': [], =
'disktotal': '6995436371968', 'version':
0}<o:p></o:p></span></p><p class=
=3DMsoNormal><span style=3D'font-family:"Courier
New"'><o:p> </o:p></s=
pan></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'># vd=
sClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier N=
ew"'>Storage domain does not exist:
('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd'=
,)<o:p></o:p></span></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal>If I run this last command on one of the working hosts:<o:p></=
o:p></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'># vd=
sClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier N=
ew"'>
uuid =3D 6fb10a49-5f1c-4bd4=
-9ff7-b7e33c1125cd<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D=
'font-family:"Courier
New"'>
vers=
ion =3D 3<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-fam=
ily:"Courier
New"'>
role =3D Regu=
lar<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"C=
ourier
New"'>
remotePath =3D srv-=
stor-01-ib0:/ovirtengine<o:p></o:p></span></p><p
class=3DMsoNormal><span st=
yle=3D'font-family:"Courier
New"'>  =
; type =3D GLUSTERFS<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier
New"'>
c=
lass =3D Data<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font=
-family:"Courier
New"'>
pool =3D =
['00000001-0001-0001-0001-00000000020e']<o:p></o:p></span></p><p
class=3DMs=
oNormal><span style=3D'font-family:"Courier
New"'> &=
nbsp; name =3D
hosted_storage<o:p></o:p></span></p><p class=3DM=
soNormal><o:p> </o:p></p><p class=3DMsoNormal>(please
note: this is th=
e storage domain used for my hosted engine)<o:p></o:p></p><p
class=3DMsoNor=
mal><o:p> </o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>If I
run “moun=
t” on my AMD host (the one with the
problem):<o:p></o:p></p><p class=
=3DMsoNormal><span style=3D'font-family:"Courier New"'>#
mount<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier New"'>...=
<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cour=
ier New"'>srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/gluster=
SD/srv-stor-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_i=
d=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p><=
/o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Courier Ne=
w"'>srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-=
01-ib0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0=
,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p=
class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>srv-stor-01-ib=
0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie ty=
pe fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions=
,allow_other,max_read=3D131072)<o:p></o:p></span></p><p
class=3DMsoNormal><=
span style=3D'font-family:"Courier New"'>tmpfs on /run/user/0 type
tmpfs (r=
w,nosuid,nodev,relatime,size=3D13185796k,mode=3D700)<o:p></o:p></span></p><=
p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'><o:p> </=
o:p></span></p><p class=3DMsoNormal>If I run
“mount” on one of =
the Intel hosts (currently working):<o:p></o:p></p><p
class=3DMsoNormal><sp=
an style=3D'font-family:"Courier New"'>#
mount<o:p></o:p></span></p><p clas=
s=3DMsoNormal><span style=3D'font-family:"Courier
New"'>...<o:p></o:p></spa=
n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier
New"'>srv-st=
or-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_=
ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau=
lt_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p
class=
=3DMsoNormal><span style=3D'font-family:"Courier
New"'>srv-stor-01-ib0:/vir=
tualmachines on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmac=
hines type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_per=
missions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p
class=3DMso=
Normal><span style=3D'font-family:"Courier
New"'>srv-stor-01-ib0:/isodomain=
on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain type fuse.gl=
usterfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_oth=
er,max_read=3D131072)<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-family:"Courier New"'>srv-stor-01-ib0:/ssd-pcie on
/rhev/data-cent=
er/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie type fuse.glusterfs (rw,relatime=
,user_id=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072=
)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'font-family:"Cou=
rier New"'>tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=
=3D3272288k,mode=3D700)<o:p></o:p></span></p><p
class=3DMsoNormal><span sty=
le=3D'font-family:"Courier
New"'><o:p> </o:p></span></p><p
class=3DMso=
Normal>The only difference in “mount” is that the hosted-engine=
storage domain is not loaded on the host which should not run the engine. =
The other domains are mounted correctly.<o:p></o:p></p><p
class=3DMsoNormal=
<o:p> </o:p></p><p
class=3DMsoNormal>What could I do to solve this is=
sue?<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3DMso=
Normal>Best regards,<o:p></o:p></p><p
class=3DMsoNormal> &=
nbsp; Giuseppe<o:p></o:p></p><p
class=3DMsoNormal><o:p>&n=
bsp;</o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma=
l style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style=
=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:IT=
'>--<br>Giuseppe Berellini<o:p></o:p></span></p><p
class=3DMsoNormal style=
=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span
style=3D'font=
-size:10.0pt;font-family:"Arial",sans-serif;color:red;mso-fareast-language:=
IT'>PTV SISTeMA<o:p></o:p></span></p><p
class=3DMsoNormal style=3D'mso-marg=
in-top-alt:auto;mso-margin-bottom-alt:auto'><span
style=3D'font-size:10.0pt=
;font-family:"Arial",sans-serif;mso-fareast-language:IT'><a
href=3D"http://=
www.sistemaits.com/"><span
style=3D'color:black;text-decoration:none'>www.s=
istemaits.com</span></a><o:p></o:p></span></p><p
class=3DMsoNormal><span st=
yle=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language=
:IT'><a href=3D"https://www.facebook.com/sistemaits"><span
style=3D'color:b=
lack;text-decoration:none'>facebook.com/sistemaits</span><...
href=3D=
"https://www.linkedin.com/company/sistema-soluzioni-per-l-ingegneria-dei-si=
stemi-di-trasporto-e-l-infomobilit-s-r-l-"><span
style=3D'color:black;text-=
decoration:none'>linkedin.com/SISTeMA</span></a><br...
</p></div></body></html>=
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_--
3193
days inactive
3193
days old
1 comments
1 participants
participants (1)
-
Giuseppe Berellini