One host failed to attach one of Storage Domains after reboot of all hosts
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi, At the beginning of february I successfully installed oVirt 3.6.2 (with hos= ted engine) on 3 hosts, which are using 1 storage server with GlusterFS. 2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd h= ost (AMD CPU) was added as host from oVirt web administration panel, withou= t hosted engine deployment (I don't want the engine running on this host). About 10 days ago I tried to reboot my oVirt environment (i.e. going to glo= bal maintenance, shutting down the engine, turning off all the hosts, start= ing them again, then setting maintenance mode to "none"). After the reboot, everything was fine with the Intel hosts and the hosted e= ngine, but AMD host (the one without HA) was not operational. I tryed to activate it, buti t failed with the following error: "Host failed to attach one of Storage Domains attached to it." If I log into my AMD host and I check the logs, I see that the storage doma= in which is not mounted is the one of the hosted engine (but this could be = correct, since this host won't run the hosted engine).
From /var/log/vdsm/vdsm.log:
</p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-= 29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(_produc= eDomain) Producing domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></= span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thr= ead-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainCach= e::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7e33= c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-famil= y:"Courier New"'>Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::156::Stora= ge.StorageDomainCache::(_findUnfetchedDomain) looking for domain 6fb10a49-5= f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span =
</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p>&n= bsp;</o:p></p><p class=3DMsoNormal>Other commands (executed on the host hav= ing the problem) which probably give useful information:<o:p></o:p></p><p c= lass=3DMsoNormal><span style=3D'font-family:"Courier New"'># vdsClient -s 0= getConnectedStoragePoolsList<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-family:"Courier New"'>00000001-0001-0001-0001-00000000020e= <o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cour= ier New"'><o:p> </o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-family:"Courier New"'># vdsClient -s 0 getStoragePoolInfo 00000001-0001= -0001-0001-00000000020e<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-family:"Courier New"'> = name =3D No Description<o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-family:"Courier New"'> = ; isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/= 5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1111111= 11111<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:= "Courier New"'> pool_status =3D c= onnected<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fami= ly:"Courier New"'> lver =3D 6<o:p= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier = New"'> spm_id =3D 2<o:p></o:p></s=
</p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>(please= note: this is the storage domain used for my hosted engine)<o:p></o:p></p>= <p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p> <= /o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>If = I run “mount” on my AMD host (the one with the problem):<o:p></= o:p></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'># mo= unt<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'>...<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-family:"Courier New"'>srv-stor-01-ib0:/virtualmachines on /rhev/data-ce= nter/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines type fuse.glusterfs (rw= ,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_other,max_read= =3D131072)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fa= mily:"Courier New"'>srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glu= sterSD/srv-stor-01-ib0:_isodomain type fuse.glusterfs (rw,relatime,user_id= =3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></= o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New= "'>srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01= -ib0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,de= fault_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p cl= ass=3DMsoNormal><span style=3D'font-family:"Courier New"'>tmpfs on /run/use= r/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,mode=3D700)<o:p><= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier Ne= w"'><o:p> </o:p></span></p><p class=3DMsoNormal>If I run “mount&= #8221; on one of the Intel hosts (currently working):<o:p></o:p></p><p clas= s=3DMsoNormal><span style=3D'font-family:"Courier New"'># mount<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>..= .<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cou= rier New"'>srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/= srv-stor-01-ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,g= roup_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></= span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>srv= -stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-stor-0= 1-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,group_i= d=3D0,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></span><= /p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>srv-stor-= 01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodo=
<p class=3DMsoNormal>The only difference in “mount” is that th= e hosted-engine storage domain is not loaded on the host which should not r= un the engine. The other domains are mounted correctly.<o:p></o:p></p><p cl= ass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>What could I do t= o solve this issue?<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p= <p class=3DMsoNormal>Best regards,<o:p></o:p></p><p class=3DMsoNormal>&nbs=
Thread-29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(= _produceDomain) Producing domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainC= ache::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7= e33c1125cd Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainC= ache::(_findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e= 33c1125cd Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.OperationMutex= ::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::290::Storage.Misc.excCmd::(= cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --= config ' devices { preferred_names =3D ["^/dev/mapper/"] ignore_suspended_d= evices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 filter =3D [= '\''r|.*|'\'' ] } global { locking_type=3D1 prioritise_write_locks=3D1 = wait_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min =3D 50 retain= _days =3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignores= kippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_cou= nt,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd= 4-9ff7-b7e33c1125cd (cwd None) Thread-29::DEBUG::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(= cmd) FAILED: <err> =3D ' WARNING: lvmetad is running but disabled. Restart= lvmetad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33= c1125cd" not found\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b= 7e33c1125cd\n'; <rc> =3D 5 Thread-29::WARNING::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloa= dvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Re= start lvmetad before enabling it!', ' Volume group "6fb10a49-5f1c-4bd4-9ff= 7-b7e33c1125cd" not found', ' Cannot process volume group 6fb10a49-5f1c-4b= d4-9ff7-b7e33c1125cd'] Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.OperationMutex= ::(_reloadvgs) Operation 'lvm reload operation' released the operation mute= x Thread-29::ERROR::2016-02-25 11:44:01,245::sdc::145::Storage.StorageDomainC= ache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-= 4bd4-9ff7-b7e33c1125cd',) Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor::(= _monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c= d Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapp= er value =3D meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 100, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-= 4bd4-9ff7-b7e33c1125cd',) jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.Task= Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:= :moving from state init -> state preparing jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::48::dispatcher= ::(wrapper) Run and protect: repoStats(options=3DNone) jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::51::dispatcher= ::(wrapper) Run and protect: repoStats, Return response: {u'5f7991ba-fdf8-4= b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'version': 3, 'acquired= ': True, 'delay': '0.00056349', 'lastCheck': '7.7', 'valid': True}, u'6fb10= a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True, 'version': = -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.0', 'valid': False}, u= '5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual': True, 'versio= n': 0, 'acquired': True, 'delay': '0.000561865', 'lastCheck': '8.4', 'valid= ': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0, 'actual': Tr= ue, 'version': 3, 'acquired': True, 'delay': '0.000227759', 'lastCheck': '8= .7', 'valid': True}} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.Tas= kManager.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::fin= ished: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual': True= , 'version': 3, 'acquired': True, 'delay': '0.00056349', 'lastCheck': '7.7'= , 'valid': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'a= ctual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': = '2.0', 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000561865', '= lastCheck': '8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a':= {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.00= 0227759', 'lastCheck': '8.7', 'valid': True}} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::595::Storage.Task= Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:= :moving from state preparing -> state finished jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager::940::S= torage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} res= ources {} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::resourceManager::977::S= torage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::task::993::Storage.Task= Manager.Task::(_decref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref = 0 aborting False Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0= b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count= =3D1 (cwd None) Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n336 b= ytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D 0 Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-= 060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count= =3D1 (cwd None) Thread-31::DEBUG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 b= ytes (339 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D 0 Thread-28::DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40= -9974-c7adcd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 = count=3D1 (cwd None) Thread-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482 b= ytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D 0 Other commands (executed on the host having the problem) which probably giv= e useful information: # vdsClient -s 0 getConnectedStoragePoolsList 00000001-0001-0001-0001-00000000020e # vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020e name =3D No Description isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isod= omain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1= 11111111111 pool_status =3D connected lver =3D 6 spm_id =3D 2 master_uuid =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879 version =3D 3 domains =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f= 1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Activ= e,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active type =3D GLUSTERFS master_ver =3D 1 5f7991ba-fdf8-4b40-9974-c7adcd4da879 =3D {'status': 'Active', 'disk= free': '6374172262400', 'isoprefix': '', 'alerts': [], 'disktotal': '699543= 6371968', 'version': 3} 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd =3D {'status': 'Active', 'isop= refix': '', 'alerts': [], 'version': -1} e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active', 'disk= free': '224145833984', 'isoprefix': '', 'alerts': [], 'disktotal': '2363171= 79904', 'version': 3} 5efea9c7-c4ec-44d4-a283-060d4c83303c =3D {'status': 'Active', 'disk= free': '6374172262400', 'isoprefix': '/rhev/data-center/mnt/glusterSD/srv-s= tor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-= 1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '6995436371968', '= version': 0} # vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd Storage domain does not exist: ('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',) If I run this last command on one of the working hosts: # vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd uuid =3D 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd version =3D 3 role =3D Regular remotePath =3D srv-stor-01-ib0:/ovirtengine type =3D GLUSTERFS class =3D Data pool =3D ['00000001-0001-0001-0001-00000000020e'] name =3D hosted_storage (please note: this is the storage domain used for my hosted engine) If I run "mount" on my AMD host (the one with the problem): # mount ... srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto= r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou= p_id=3D0,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i= b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def= ault_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib= 0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau= lt_permissions,allow_other,max_read=3D131072) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,= mode=3D700) If I run "mount" on one of the Intel hosts (currently working): # mount ... srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01= -ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0= ,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto= r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou= p_id=3D0,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i= b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def= ault_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib= 0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau= lt_permissions,allow_other,max_read=3D131072) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D3272288k,m= ode=3D700) The only difference in "mount" is that the hosted-engine storage domain is = not loaded on the host which should not run the engine. The other domains a= re mounted correctly. What could I do to solve this issue? Best regards, Giuseppe -- Giuseppe Berellini PTV SISTeMA www.sistemaits.com<http://www.sistemaits.com/> facebook.com/sistemaits<https://www.facebook.com/sistemaits> linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzioni-per= -l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-> --_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type content= =3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros= oft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} span.StileMessaggioDiPostaElettronica17 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 2.0cm 2.0cm 2.0cm;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DIT link=3D"#0563C1" v= link=3D"#954F72"><div class=3DWordSection1><p class=3DMsoNormal>Hi,<o:p></o= :p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>At th= e beginning of february I successfully installed oVirt 3.6.2 (with hosted e= ngine) on 3 hosts, which are using 1 storage server with GlusterFS.<o:p></o= :p></p><p class=3DMsoNormal>2 hosts (with Intel CPU) are using HA and are h= osting the engine; the 3rd host (AMD CPU) was added as host from oVirt web = administration panel, without hosted engine deployment (I don’t want = the engine running on this host).<o:p></o:p></p><p class=3DMsoNormal><o:p>&= nbsp;</o:p></p><p class=3DMsoNormal>About 10 days ago I tried to reboot my = oVirt environment (i.e. going to global maintenance, shutting down the engi= ne, turning off all the hosts, starting them again, then setting maintenanc= e mode to “none”).<o:p></o:p></p><p class=3DMsoNormal>After the= reboot, everything was fine with the Intel hosts and the hosted engine, bu= t AMD host (the one without HA) was not operational.<o:p></o:p></p><p class= =3DMsoNormal>I tryed to activate it, buti t failed with the following error= :<o:p></o:p></p><p class=3DMsoNormal> &n= bsp; “Host failed to attach one of Storage Domains attached to it.= 221;<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMso= Normal>If I log into my AMD host and I check the logs, I see that the stora= ge domain which is not mounted is the one of the hosted engine (but this co= uld be correct, since this host won’t run the hosted engine).<o:p></o= :p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>From = /var/log/vdsm/vdsm.log:<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p= style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,1= 59::lvm::370::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload op= eration' got the operation mutex<o:p></o:p></span></p><p class=3DMsoNormal>= <span style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25 11:4= 4:01,159::lvm::290::Storage.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list = 0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --config ' devices { preferred_name= s =3D ["^/dev/mapper/"] ignore_suspended_devices=3D1 write_cache_= state=3D0 disable_after_error_count=3D3 filter =3D [ '\''r|.*|'\'' ] } = ; global { locking_type=3D1 prioritise_write_locks=3D1 wa= it_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min = =3D 50 retain_days =3D 0 } ' --noheadings --units b --nosuffix --sepa= rator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,ex= tent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_nam= e 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd (cwd None)<o:p></o:p></span></p><p c= lass=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-29::DEBUG= ::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <= ;err> =3D ' WARNING: lvmetad is running but disabled. Restart lvme= tad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-= b7e33c1125cd" not found\n Cannot process volume group 6fb10a49-5= f1c-4bd4-9ff7-b7e33c1125cd\n'; <rc> =3D 5<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-29::WARNING= ::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloadvgs) lvm vgs fail= ed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad= before enabling it!', ' Volume group "6fb10a49-5f1c-4bd4-9ff7-b= 7e33c1125cd" not found', ' Cannot process volume group 6fb10a49-= 5f1c-4bd4-9ff7-b7e33c1125cd']<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:0= 1,225::lvm::415::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload= operation' released the operation mutex<o:p></o:p></span></p><p class=3DMs= oNormal><span style=3D'font-family:"Courier New"'>Thread-29::ERROR::2016-02= -25 11:44:01,245::sdc::145::Storage.StorageDomainCache::(_findDomain) domai= n 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not found<o:p></o:p></span></p><p cl= ass=3DMsoNormal><span style=3D'font-family:"Courier New"'>Traceback (most r= ecent call last):<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'= font-family:"Courier New"'> File "/usr/share/vdsm/storage/sdc.py= ", line 143, in _findDomain<o:p></o:p></span></p><p class=3DMsoNormal>= <span style=3D'font-family:"Courier New"'> dom =3D findMe= thod(sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= family:"Courier New"'> File "/usr/share/vdsm/storage/sdc.py"= ;, line 173, in _findUnfetchedDomain<o:p></o:p></span></p><p class=3DMsoNor= mal><span style=3D'font-family:"Courier New"'> raise se.S= torageDomainDoesNotExist(sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal>= <span style=3D'font-family:"Courier New"'>StorageDomainDoesNotExist: Storag= e domain does not exist: (u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o= :p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"= '>Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor:= :(_monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c112= 5cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'>Traceback (most recent call last):<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-family:"Courier New"'> File "= /usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain<o:p><= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier Ne= w"'> self._produceDomain()<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-family:"Courier New"'> File "/u= sr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapper<o:= p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier= New"'> value =3D meth(self, *a, **kw)<o:p></o:p></span><= /p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> Fi= le "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDom= ain<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'> self.domain =3D sdCache.produce(self.sdUUID= )<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cou= rier New"'> File "/usr/share/vdsm/storage/sdc.py", line 100= , in produce<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= family:"Courier New"'> domain.getRealDomain()<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&n= bsp; File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDo= main<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"= Courier New"'> return self._cache._realProduce(self._sdUU= ID)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'> File "/usr/share/vdsm/storage/sdc.py", line 1= 24, in _realProduce<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'> domain =3D self._findDoma= in(sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fa= mily:"Courier New"'> File "/usr/share/vdsm/storage/sdc.py",= line 143, in _findDomain<o:p></o:p></span></p><p class=3DMsoNormal><span s= tyle=3D'font-family:"Courier New"'> dom =3D findMethod(sd= UUID)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:= "Courier New"'> File "/usr/share/vdsm/storage/sdc.py", line= 173, in _findUnfetchedDomain<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-family:"Courier New"'> raise se.StorageD= omainDoesNotExist(sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal><span s= tyle=3D'font-family:"Courier New"'>StorageDomainDoesNotExist: Storage domai= n does not exist: (u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></sp= an></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>jsonr= pc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.TaskManag= er.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::movi= ng from state init -> state preparing<o:p></o:p></span></p><p class=3DMs= oNormal><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::INFO:= :2016-02-25 11:44:03,293::logUtils::48::dispatcher::(wrapper) Run and prote= ct: repoStats(options=3DNone)<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::INFO::2016-02-25= 11:44:03,293::logUtils::51::dispatcher::(wrapper) Run and protect: repoSta= ts, Return response: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, = 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.00056349', 'las= tCheck': '7.7', 'valid': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'= code': 358, 'actual': True, 'version': -1, 'acquired': False, 'delay': '0',= 'lastCheck': '2.0', 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303= c': {'code': 0, 'actual': True, 'version': 0, 'acquired': True, 'delay': '0= .000561865', 'lastCheck': '8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-= 0b0bc0330d9a': {'code': 0, 'actual': True, 'version': 3, 'acquired': True, = 'delay': '0.000227759', 'lastCheck': '8.7', 'valid': True}}<o:p></o:p></spa= n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>jsonrp= c.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.TaskManag= er.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::finished:= {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'ver= sion': 3, 'acquired': True, 'delay': '0.00056349', 'lastCheck': '7.7', 'val= id': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual'= : True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.0',= 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'act= ual': True, 'version': 0, 'acquired': True, 'delay': '0.000561865', 'lastCh= eck': '8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'cod= e': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.00022775= 9', 'lastCheck': '8.7', 'valid': True}}<o:p></o:p></span></p><p class=3DMso= Normal><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG:= :2016-02-25 11:44:03,294::task::595::Storage.TaskManager.Task::(_updateStat= e) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::moving from state prepari= ng -> state finished<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:4= 4:03,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll)= Owner.releaseAll requests {} resources {}<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEB= UG::2016-02-25 11:44:03,295::resourceManager::977::Storage.ResourceManager.= Owner::(cancelAll) Owner.cancelAll requests {}<o:p></o:p></span></p><p clas= s=3DMsoNormal><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0:= :DEBUG::2016-02-25 11:44:03,295::task::993::Storage.TaskManager.Task::(_dec= ref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref 0 aborting False<o:= p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier= New"'>Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc= .excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/= rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6= -9a3b-0b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 = count=3D1 (cwd None)<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'>Thread-30::DEBUG::2016-02-25 11:44:04,630::f= ileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =3D '0= +1 records in\n0+1 records out\n336 bytes (336 B) copied, 0.000286148 s, 1.= 2 MB/s\n'; <rc> =3D 0<o:p></o:p></span></p><p class=3DMsoNormal><span= style=3D'font-family:"Courier New"'>Thread-31::DEBUG::2016-02-25 11:44:04,= 925::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset --cp= u-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glusterSD/srv-stor-01-ib= 0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/dom_md/metadata iflag=3Dd= irect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)<o:p></o:p></span></p><p= class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-31::DEB= UG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd::(getReadDela= y) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 bytes (33= 9 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D 0<o:p></o:p></span></= p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-28:= :DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd::(getRead= Delay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/= mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40-9974-c7ad= cd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 = (cwd None)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fa= mily:"Courier New"'>Thread-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173:= :Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> =3D '0+1 records = in\n0+1 records out\n482 bytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; = <rc> =3D 0<o:p></o:p></span></p><p class=3DMsoNormal><o:p> </o:p= pan></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&nbs= p; master_uuid =3D 5f7991ba-fdf8-4b40-9= 974-c7adcd4da879<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-family:"Courier New"'> versio= n =3D 3<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-famil= y:"Courier New"'> domains =3D 5f7= 991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f1c-4bd4-9ff7-b7e33c1125= cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Active,e84c6a1a-9f82-4fa6-9a= 3b-0b0bc0330d9a:Active<o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'font-family:"Courier New"'> = type =3D GLUSTERFS<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D= 'font-family:"Courier New"'> mast= er_ver =3D 1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= family:"Courier New"'> 5f7991ba-f= df8-4b40-9974-c7adcd4da879 =3D {'status': 'Active', 'diskfree': '6374172262= 400', 'isoprefix': '', 'alerts': [], 'disktotal': '6995436371968', 'version= ': 3}<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:= "Courier New"'> 6fb10a49-5f1c-4bd= 4-9ff7-b7e33c1125cd =3D {'status': 'Active', 'isoprefix': '', 'alerts': [],= 'version': -1}<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'fo= nt-family:"Courier New"'> e84c6a1= a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active', 'diskfree': '2241458= 33984', 'isoprefix': '', 'alerts': [], 'disktotal': '236317179904', 'versio= n': 3}<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family= :"Courier New"'> 5efea9c7-c4ec-44= d4-a283-060d4c83303c =3D {'status': 'Active', 'diskfree': '6374172262400', = 'isoprefix': '/rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5e= fea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-111111111= 111', 'alerts': [], 'disktotal': '6995436371968', 'version': 0}<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'><o= :p> </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"= Courier New"'># vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7= -b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font= -family:"Courier New"'>Storage domain does not exist: ('6fb10a49-5f1c-4bd4-= 9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p class=3DMsoNormal><o:p> <= /o:p></p><p class=3DMsoNormal>If I run this last command on one of the work= ing hosts:<o:p></o:p></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'># vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-= b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= family:"Courier New"'> uuid =3D 6= fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoNorm= al><span style=3D'font-family:"Courier New"'> = version =3D 3<o:p></o:p></span></p><p class=3DMsoNormal><span = style=3D'font-family:"Courier New"'> &nb= sp; role =3D Regular<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'> r= emotePath =3D srv-stor-01-ib0:/ovirtengine<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-family:"Courier New"'> = ; type =3D GLUSTERFS<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-family:"Courier New"'> &nb= sp; class =3D Data<o:p></o:p></span></p><p class=3DMsoNormal><s= pan style=3D'font-family:"Courier New"'> = ; pool =3D ['00000001-0001-0001-0001-00000000020e']<o:p></o:p></span>= </p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> &n= bsp; name =3D hosted_storage<o:p></o:p></span= main type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_perm= issions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-family:"Courier New"'>srv-stor-01-ib0:/ssd-pcie o= n /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie type fuse.glust= erfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_other,= max_read=3D131072)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D= 'font-family:"Courier New"'>tmpfs on /run/user/0 type tmpfs (rw,nosuid,node= v,relatime,size=3D3272288k,mode=3D700)<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-family:"Courier New"'><o:p> </o:p></span></p= p; Giuseppe<o:p></o:p></p><p class=3DMs= oNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p c= lass=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:aut= o'><span style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-farea= st-language:IT'>--<br>Giuseppe Berellini<o:p></o:p></span></p><p class=3DMs= oNormal style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span = style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;color:red;mso-fare= ast-language:IT'>PTV SISTeMA<o:p></o:p></span></p><p class=3DMsoNormal styl= e=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style=3D'fon= t-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:IT'><a hr= ef=3D"http://www.sistemaits.com/"><span style=3D'color:black;text-decoratio= n:none'>www.sistemaits.com</span></a><o:p></o:p></span></p><p class=3DMsoNo= rmal><span style=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-far= east-language:IT'><a href=3D"https://www.facebook.com/sistemaits"><span sty= le=3D'color:black;text-decoration:none'>facebook.com/sistemaits</span></a><= br><a href=3D"https://www.linkedin.com/company/sistema-soluzioni-per-l-inge= gneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-"><span style=3D'colo= r:black;text-decoration:none'>linkedin.com/SISTeMA</span></a><br> </sp= an><o:p></o:p></p></div></body></html>= --_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30B8CSKAWNE25CEXpt_--
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, about one hour ago my AMD host came back up, after more than 10 days being = down. Apart from checking the logs (which I suppose didn't help in solving my pro= blem), I enabled NFS share on my ISO domain. I'm still not able to understand how that could help. I would be really happy to better understand what happened! :-) If anyone has ideas/explanations to share, you are welcome! :-) Best regards, Giuseppe -- Giuseppe Berellini PTV SISTeMA www.sistemaits.com<http://www.sistemaits.com/> facebook.com/sistemaits<https://www.facebook.com/sistemaits> linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzioni-per= -l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-> Da: users-bounces@ovirt.org [mailto:users-bounces@ovirt.org] Per conto di G= iuseppe Berellini Inviato: gioved=EC 25 febbraio 2016 12:10 A: users@ovirt.org Oggetto: [ovirt-users] One host failed to attach one of Storage Domains aft= er reboot of all hosts Hi, At the beginning of february I successfully installed oVirt 3.6.2 (with hos= ted engine) on 3 hosts, which are using 1 storage server with GlusterFS. 2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd h= ost (AMD CPU) was added as host from oVirt web administration panel, withou= t hosted engine deployment (I don't want the engine running on this host). About 10 days ago I tried to reboot my oVirt environment (i.e. going to glo= bal maintenance, shutting down the engine, turning off all the hosts, start= ing them again, then setting maintenance mode to "none"). After the reboot, everything was fine with the Intel hosts and the hosted e= ngine, but AMD host (the one without HA) was not operational. I tryed to activate it, buti t failed with the following error: "Host failed to attach one of Storage Domains attached to it." If I log into my AMD host and I check the logs, I see that the storage doma= in which is not mounted is the one of the hosted engine (but this could be = correct, since this host won't run the hosted engine).
From /var/log/vdsm/vdsm.log:
Thread-29::DEBUG::2016-02-25 11:44:01,157::monitor::322::Storage.Monitor::(= _produceDomain) Producing domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::139::Storage.StorageDomainC= ache::(_findDomain) looking for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7= e33c1125cd Thread-29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainC= ache::(_findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e= 33c1125cd Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.OperationMutex= ::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::290::Storage.Misc.excCmd::(= cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /usr/sbin/lvm vgs --= config ' devices { preferred_names =3D ["^/dev/mapper/"] ignore_suspended_d= evices=3D1 write_cache_state=3D0 disable_after_error_count=3D3 filter =3D [= '\''r|.*|'\'' ] } global { locking_type=3D1 prioritise_write_locks=3D1 = wait_for_locks=3D1 use_lvmetad=3D0 } backup { retain_min =3D 50 retain= _days =3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignores= kippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_cou= nt,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd= 4-9ff7-b7e33c1125cd (cwd None) Thread-29::DEBUG::2016-02-25 11:44:01,223::lvm::290::Storage.Misc.excCmd::(= cmd) FAILED: <err> =3D ' WARNING: lvmetad is running but disabled. Restart= lvmetad before enabling it!\n Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33= c1125cd" not found\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b= 7e33c1125cd\n'; <rc> =3D 5 Thread-29::WARNING::2016-02-25 11:44:01,225::lvm::375::Storage.LVM::(_reloa= dvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Re= start lvmetad before enabling it!', ' Volume group "6fb10a49-5f1c-4bd4-9ff= 7-b7e33c1125cd" not found', ' Cannot process volume group 6fb10a49-5f1c-4b= d4-9ff7-b7e33c1125cd'] Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.OperationMutex= ::(_reloadvgs) Operation 'lvm reload operation' released the operation mute= x Thread-29::ERROR::2016-02-25 11:44:01,245::sdc::145::Storage.StorageDomainC= ache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd not found Traceback (most recent call last): File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-= 4bd4-9ff7-b7e33c1125cd',) Thread-29::ERROR::2016-02-25 11:44:01,246::monitor::276::Storage.Monitor::(= _monitorDomain) Error monitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c= d Traceback (most recent call last): File "/usr/share/vdsm/storage/monitor.py", line 264, in _monitorDomain self._produceDomain() File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 767, in wrapp= er value =3D meth(self, *a, **kw) File "/usr/share/vdsm/storage/monitor.py", line 323, in _produceDomain self.domain =3D sdCache.produce(self.sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 100, in produce domain.getRealDomain() File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain return self._cache._realProduce(self._sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce domain =3D self._findDomain(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain dom =3D findMethod(sdUUID) File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDomain raise se.StorageDomainDoesNotExist(sdUUID) StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10a49-5f1c-= 4bd4-9ff7-b7e33c1125cd',) jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,292::task::595::Storage.Task= Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:= :moving from state init -> state preparing jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::48::dispatcher= ::(wrapper) Run and protect: repoStats(options=3DNone) jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::51::dispatcher= ::(wrapper) Run and protect: repoStats, Return response: {u'5f7991ba-fdf8-4= b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'version': 3, 'acquired= ': True, 'delay': '0.00056349', 'lastCheck': '7.7', 'valid': True}, u'6fb10= a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True, 'version': = -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.0', 'valid': False}, u= '5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual': True, 'versio= n': 0, 'acquired': True, 'delay': '0.000561865', 'lastCheck': '8.4', 'valid= ': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0, 'actual': Tr= ue, 'version': 3, 'acquired': True, 'delay': '0.000227759', 'lastCheck': '8= .7', 'valid': True}} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::1191::Storage.Tas= kManager.Task::(prepare) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::fin= ished: {u'5f7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual': True= , 'version': 3, 'acquired': True, 'delay': '0.00056349', 'lastCheck': '7.7'= , 'valid': True}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'a= ctual': True, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': = '2.0', 'valid': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0= , 'actual': True, 'version': 0, 'acquired': True, 'delay': '0.000561865', '= lastCheck': '8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a':= {'code': 0, 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.00= 0227759', 'lastCheck': '8.7', 'valid': True}} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::task::595::Storage.Task= Manager.Task::(_updateState) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`:= :moving from state preparing -> state finished jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager::940::S= torage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} res= ources {} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::resourceManager::977::S= torage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,295::task::993::Storage.Task= Manager.Task::(_decref) Task=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::ref = 0 aborting False Thread-30::DEBUG::2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0= b0bc0330d9a/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count= =3D1 (cwd None) Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n336 b= ytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D 0 Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-= 060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count= =3D1 (cwd None) Thread-31::DEBUG::2016-02-25 11:44:04,950::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n339 b= ytes (339 B) copied, 0.0005884 s, 576 kB/s\n'; <rc> =3D 0 Thread-28::DEBUG::2016-02-25 11:44:05,583::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/da= ta-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmachines/5f7991ba-fdf8-4b40= -9974-c7adcd4da879/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 = count=3D1 (cwd None) Thread-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd= ::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482 b= ytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D 0 Other commands (executed on the host having the problem) which probably giv= e useful information: # vdsClient -s 0 getConnectedStoragePoolsList 00000001-0001-0001-0001-00000000020e # vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020e name =3D No Description isoprefix =3D /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isod= omain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-1111-1111-1111-1= 11111111111 pool_status =3D connected lver =3D 6 spm_id =3D 2 master_uuid =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879 version =3D 3 domains =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879:Active,6fb10a49-5f= 1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4ec-44d4-a283-060d4c83303c:Activ= e,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active type =3D GLUSTERFS master_ver =3D 1 5f7991ba-fdf8-4b40-9974-c7adcd4da879 =3D {'status': 'Active', 'disk= free': '6374172262400', 'isoprefix': '', 'alerts': [], 'disktotal': '699543= 6371968', 'version': 3} 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd =3D {'status': 'Active', 'isop= refix': '', 'alerts': [], 'version': -1} e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a =3D {'status': 'Active', 'disk= free': '224145833984', 'isoprefix': '', 'alerts': [], 'disktotal': '2363171= 79904', 'version': 3} 5efea9c7-c4ec-44d4-a283-060d4c83303c =3D {'status': 'Active', 'disk= free': '6374172262400', 'isoprefix': '/rhev/data-center/mnt/glusterSD/srv-s= tor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a283-060d4c83303c/images/11111111-= 1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '6995436371968', '= version': 0} # vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd Storage domain does not exist: ('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',) If I run this last command on one of the working hosts: # vdsClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd uuid =3D 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd version =3D 3 role =3D Regular remotePath =3D srv-stor-01-ib0:/ovirtengine type =3D GLUSTERFS class =3D Data pool =3D ['00000001-0001-0001-0001-00000000020e'] name =3D hosted_storage (please note: this is the storage domain used for my hosted engine) If I run "mount" on my AMD host (the one with the problem): # mount ... srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto= r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou= p_id=3D0,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i= b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def= ault_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib= 0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau= lt_permissions,allow_other,max_read=3D131072) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D13185796k,= mode=3D700) If I run "mount" on one of the Intel hosts (currently working): # mount ... srv-stor-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01= -ib0:_ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0= ,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/glusterSD/srv-sto= r-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_id=3D0,grou= p_id=3D0,default_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-01-i= b0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,def= ault_permissions,allow_other,max_read=3D131072) srv-stor-01-ib0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib= 0:_ssd-pcie type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau= lt_permissions,allow_other,max_read=3D131072) tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=3D3272288k,m= ode=3D700) The only difference in "mount" is that the hosted-engine storage domain is = not loaded on the host which should not run the engine. The other domains a= re mounted correctly. What could I do to solve this issue? Best regards, Giuseppe -- Giuseppe Berellini PTV SISTeMA www.sistemaits.com<http://www.sistemaits.com/> facebook.com/sistemaits<https://www.facebook.com/sistemaits> linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzioni-per= -l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-> --_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; charset=3Diso-8859-= 1"> <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"><head><meta name=3DGenerator content=3D"Microso= ft Word 15 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri",sans-serif; mso-fareast-language:EN-US;} a:link, span.MsoHyperlink {mso-style-priority:99; color:#0563C1; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:#954F72; text-decoration:underline;} span.StileMessaggioDiPostaElettronica17 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:windowtext;} span.StileMessaggioDiPostaElettronica18 {mso-style-type:personal-reply; font-family:"Calibri",sans-serif; color:#1F497D;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 2.0cm 2.0cm 2.0cm;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DIT link=3D"#0563C1" v= link=3D"#954F72"><div class=3DWordSection1><p class=3DMsoNormal><span style= =3D'color:#1F497D'>Hi,<o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'color:#1F497D'><o:p> </o:p></span></p><p class=3DMsoNormal><span = style=3D'color:#1F497D'>about one hour ago my AMD host came back up, after = more than 10 days being down.<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'color:#1F497D'>Apart from checking the logs (which I suppose di= dn’t help in solving my problem), I enabled NFS share on my ISO domai= n.<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'color:#1F497D'>= <o:p> </o:p></span></p><p class=3DMsoNormal><span style=3D'color:#1F49= 7D'>I’m still not able to understand how that could help.<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'color:#1F497D'>I would be rea= lly happy to better understand what happened! :-)<o:p></o:p></span></p><p c= lass=3DMsoNormal><span style=3D'color:#1F497D'>If anyone has ideas/explanat= ions to share, you are welcome! :-)<o:p></o:p></span></p><p class=3DMsoNorm= al><span style=3D'color:#1F497D'><o:p> </o:p></span></p><p class=3DMso= Normal><span style=3D'color:#1F497D'>Best regards,<o:p></o:p></span></p><p = class=3DMsoNormal><span style=3D'color:#1F497D'> &nb= sp; Giuseppe</span><span style=3D'color:#1F497D'><o:p></o:p></s= pan></p><p class=3DMsoNormal><span style=3D'color:#1F497D'><o:p> </o:p=
</span></p><div><p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-= margin-bottom-alt:auto'><span style=3D'font-size:10.0pt;font-family:"Arial"= ,sans-serif;color:#1F497D;mso-fareast-language:IT'>--<br>Giuseppe Berellini= <o:p></o:p></span></p><p class=3DMsoNormal style=3D'mso-margin-top-alt:auto= ;mso-margin-bottom-alt:auto'><span style=3D'font-size:10.0pt;font-family:"A= rial",sans-serif;color:red;mso-fareast-language:IT'>PTV SISTeMA<o:p></o:p><= /span></p><p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-margin-= bottom-alt:auto'><span style=3D'font-size:10.0pt;font-family:"Arial",sans-s= erif;color:#1F497D;mso-fareast-language:IT'><a href=3D"http://www.sistemait= s.com/"><span style=3D'color:black;text-decoration:none'>www.sistemaits.com= </span></a><o:p></o:p></span></p></div><p class=3DMsoNormal><span style=3D'= font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D;mso-fareast-l= anguage:IT'><a href=3D"https://www.facebook.com/sistemaits"><span style=3D'=
color:black;text-decoration:none'>facebook.com/sistemaits</span></a><br><a = href=3D"https://www.linkedin.com/company/sistema-soluzioni-per-l-ingegneria= -dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-"><span style=3D'color:blac= k;text-decoration:none'>linkedin.com/SISTeMA</span></a><br> </span><sp= an style=3D'color:#1F497D'><o:p></o:p></span></p><div><div style=3D'border:= none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=3DM= soNormal><b><span style=3D'mso-fareast-language:IT'>Da:</span></b><span sty= le=3D'mso-fareast-language:IT'> users-bounces@ovirt.org [mailto:users-bounc= es@ovirt.org] <b>Per conto di </b>Giuseppe Berellini<br><b>Inviato:</b> gio= ved=EC 25 febbraio 2016 12:10<br><b>A:</b> users@ovirt.org<br><b>Oggetto:</= b> [ovirt-users] One host failed to attach one of Storage Domains after reb= oot of all hosts<o:p></o:p></span></p></div></div><p class=3DMsoNormal><o:p= > </o:p></p><p class=3DMsoNormal>Hi,<o:p></o:p></p><p class=3DMsoNorma= l><o:p> </o:p></p><p class=3DMsoNormal>At the beginning of february I = successfully installed oVirt 3.6.2 (with hosted engine) on 3 hosts, which a= re using 1 storage server with GlusterFS.<o:p></o:p></p><p class=3DMsoNorma= l>2 hosts (with Intel CPU) are using HA and are hosting the engine; the 3rd= host (AMD CPU) was added as host from oVirt web administration panel, with= out hosted engine deployment (I don’t want the engine running on this= host).<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3D= MsoNormal>About 10 days ago I tried to reboot my oVirt environment (i.e. go= ing to global maintenance, shutting down the engine, turning off all the ho= sts, starting them again, then setting maintenance mode to “none̶= 1;).<o:p></o:p></p><p class=3DMsoNormal>After the reboot, everything was fi= ne with the Intel hosts and the hosted engine, but AMD host (the one withou= t HA) was not operational.<o:p></o:p></p><p class=3DMsoNormal>I tryed to ac= tivate it, buti t failed with the following error:<o:p></o:p></p><p class= =3DMsoNormal> “Host failed = to attach one of Storage Domains attached to it.”<o:p></o:p></p><p cl= ass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>If I log into my = AMD host and I check the logs, I see that the storage domain which is not m= ounted is the one of the hosted engine (but this could be correct, since th= is host won’t run the hosted engine).<o:p></o:p></p><p class=3DMsoNor= mal><o:p> </o:p></p><p class=3DMsoNormal>From /var/log/vdsm/vdsm.log:<= o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma= l><span style=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25 11= :44:01,157::monitor::322::Storage.Monitor::(_produceDomain) Producing domai= n 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-family:"Courier New"'>Thread-29::ERROR::2016-02-2= 5 11:44:01,158::sdc::139::Storage.StorageDomainCache::(_findDomain) looking= for unfetched domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span= ></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-= 29::ERROR::2016-02-25 11:44:01,158::sdc::156::Storage.StorageDomainCache::(= _findUnfetchedDomain) looking for domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125= cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Co= urier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::370::Storage.Op= erationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operati= on mutex<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fami= ly:"Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,159::lvm::290::Stor= age.Misc.excCmd::(cmd) /usr/bin/taskset --cpu-list 0-63 /usr/bin/sudo -n /u= sr/sbin/lvm vgs --config ' devices { preferred_names =3D ["^/dev/mappe= r/"] ignore_suspended_devices=3D1 write_cache_state=3D0 disable_after_= error_count=3D3 filter =3D [ '\''r|.*|'\'' ] } global { locking= _type=3D1 prioritise_write_locks=3D1 wait_for_locks=3D1 u= se_lvmetad=3D0 } backup { retain_min =3D 50 retain_days = =3D 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskipped= cluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tag= s,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 6fb10a49-5f1c-4bd4-9ff7= -b7e33c1125cd (cwd None)<o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-family:"Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,223= ::lvm::290::Storage.Misc.excCmd::(cmd) FAILED: <err> =3D ' WARN= ING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n= Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd" not fo= und\n Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125c= d\n'; <rc> =3D 5<o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'font-family:"Courier New"'>Thread-29::WARNING::2016-02-25 11:44:01,225= ::lvm::375::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING= : lvmetad is running but disabled. Restart lvmetad before enabling it!', '&= nbsp; Volume group "6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd" not fou= nd', ' Cannot process volume group 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125= cd']<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"= Courier New"'>Thread-29::DEBUG::2016-02-25 11:44:01,225::lvm::415::Storage.= OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the = operation mutex<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'fo= nt-family:"Courier New"'>Thread-29::ERROR::2016-02-25 11:44:01,245::sdc::14= 5::Storage.StorageDomainCache::(_findDomain) domain 6fb10a49-5f1c-4bd4-9ff7= -b7e33c1125cd not found<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-family:"Courier New"'>Traceback (most recent call last):<o:p></o= :p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"= '> File "/usr/share/vdsm/storage/sdc.py", line 143, in _fin= dDomain<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-famil= y:"Courier New"'> dom =3D findMethod(sdUUID)<o:p></o:p></= span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&nb= sp; File "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfe= tchedDomain<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-f= amily:"Courier New"'> raise se.StorageDomainDoesNotExist(= sdUUID)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-famil= y:"Courier New"'>StorageDomainDoesNotExist: Storage domain does not exist: = (u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-family:"Courier New"'>Thread-29::ERROR::2016-= 02-25 11:44:01,246::monitor::276::Storage.Monitor::(_monitorDomain) Error m= onitoring domain 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p></o:p></span></p>= <p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Traceback (m= ost recent call last):<o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'font-family:"Courier New"'> File "/usr/share/vdsm/storage/m= onitor.py", line 264, in _monitorDomain<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-family:"Courier New"'> s= elf._produceDomain()<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'> File "/usr/lib/python2.7/site-pa= ckages/vdsm/utils.py", line 767, in wrapper<o:p></o:p></span></p><p cl= ass=3DMsoNormal><span style=3D'font-family:"Courier New"'>  = ; value =3D meth(self, *a, **kw)<o:p></o:p></span></p><p class=3DMsoNormal>= <span style=3D'font-family:"Courier New"'> File "/usr/share/vdsm= /storage/monitor.py", line 323, in _produceDomain<o:p></o:p></span></p= ><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>  = ; self.domain =3D sdCache.produce(self.sdUUID)<o:p></o:p></span></p><= p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> File &= quot;/usr/share/vdsm/storage/sdc.py", line 100, in produce<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&n= bsp; domain.getRealDomain()<o:p></o:p></span></p><p class=3DMso= Normal><span style=3D'font-family:"Courier New"'> File "/usr/sha= re/vdsm/storage/sdc.py", line 52, in getRealDomain<o:p></o:p></span></= p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> &nbs= p; return self._cache._realProduce(self._sdUUID)<o:p></o:p></span></p= ><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> File= "/usr/share/vdsm/storage/sdc.py", line 124, in _realProduce<o:p>= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier N= ew"'> domain =3D self._findDomain(sdUUID)<o:p></o:p></spa= n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> = File "/usr/share/vdsm/storage/sdc.py", line 143, in _findDomain<= o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Couri= er New"'> dom =3D findMethod(sdUUID)<o:p></o:p></span></p= ><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> File= "/usr/share/vdsm/storage/sdc.py", line 173, in _findUnfetchedDom= ain<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'> raise se.StorageDomainDoesNotExist(sdUUID)<= o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Couri= er New"'>StorageDomainDoesNotExist: Storage domain does not exist: (u'6fb10= a49-5f1c-4bd4-9ff7-b7e33c1125cd',)<o:p></o:p></span></p><p class=3DMsoNorma= l><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016= -02-25 11:44:03,292::task::595::Storage.TaskManager.Task::(_updateState) Ta= sk=3D`2862ba96-8080-4e74-a55a-cdf93326631a`::moving from state init -> s= tate preparing<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'fon= t-family:"Courier New"'>jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::= logUtils::48::dispatcher::(wrapper) Run and protect: repoStats(options=3DNo= ne)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'>jsonrpc.Executor/0::INFO::2016-02-25 11:44:03,293::logUtils::5= 1::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'5f= 7991ba-fdf8-4b40-9974-c7adcd4da879': {'code': 0, 'actual': True, 'version':= 3, 'acquired': True, 'delay': '0.00056349', 'lastCheck': '7.7', 'valid': T= rue}, u'6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True= , 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.0', 'vali= d': False}, u'5efea9c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual': = True, 'version': 0, 'acquired': True, 'delay': '0.000561865', 'lastCheck': = '8.4', 'valid': True}, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0,= 'actual': True, 'version': 3, 'acquired': True, 'delay': '0.000227759', 'l= astCheck': '8.7', 'valid': True}}<o:p></o:p></span></p><p class=3DMsoNormal= ><span style=3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-= 02-25 11:44:03,294::task::1191::Storage.TaskManager.Task::(prepare) Task=3D= `2862ba96-8080-4e74-a55a-cdf93326631a`::finished: {u'5f7991ba-fdf8-4b40-997= 4-c7adcd4da879': {'code': 0, 'actual': True, 'version': 3, 'acquired': True= , 'delay': '0.00056349', 'lastCheck': '7.7', 'valid': True}, u'6fb10a49-5f1= c-4bd4-9ff7-b7e33c1125cd': {'code': 358, 'actual': True, 'version': -1, 'ac= quired': False, 'delay': '0', 'lastCheck': '2.0', 'valid': False}, u'5efea9= c7-c4ec-44d4-a283-060d4c83303c': {'code': 0, 'actual': True, 'version': 0, = 'acquired': True, 'delay': '0.000561865', 'lastCheck': '8.4', 'valid': True= }, u'e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a': {'code': 0, 'actual': True, 've= rsion': 3, 'acquired': True, 'delay': '0.000227759', 'lastCheck': '8.7', 'v= alid': True}}<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font= -family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::= task::595::Storage.TaskManager.Task::(_updateState) Task=3D`2862ba96-8080-4= e74-a55a-cdf93326631a`::moving from state preparing -> state finished<o:= p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier= New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,294::resourceManager:= :940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests= {} resources {}<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:03,29= 5::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.c= ancelAll requests {}<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'>jsonrpc.Executor/0::DEBUG::2016-02-25 11:44:= 03,295::task::993::Storage.TaskManager.Task::(_decref) Task=3D`2862ba96-808= 0-4e74-a55a-cdf93326631a`::ref 0 aborting False<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thread-30::DEBUG::= 2016-02-25 11:44:04,603::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /= usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glus= terSD/srv-stor-01-ib0:_ssd-pcie/e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a/dom_md= /metadata iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)<o:p>= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier N= ew"'>Thread-30::DEBUG::2016-02-25 11:44:04,630::fileSD::173::Storage.Misc.e= xcCmd::(getReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records= out\n336 bytes (336 B) copied, 0.000286148 s, 1.2 MB/s\n'; <rc> =3D = 0<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cou= rier New"'>Thread-31::DEBUG::2016-02-25 11:44:04,925::fileSD::173::Storage.= Misc.excCmd::(getReadDelay) /usr/bin/taskset --cpu-list 0-63 /usr/bin/dd if= =3D/rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec= -44d4-a283-060d4c83303c/dom_md/metadata iflag=3Ddirect of=3D/dev/null bs=3D= 4096 count=3D1 (cwd None)<o:p></o:p></span></p><p class=3DMsoNormal><span s= tyle=3D'font-family:"Courier New"'>Thread-31::DEBUG::2016-02-25 11:44:04,95= 0::fileSD::173::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = =3D '0+1 records in\n0+1 records out\n339 bytes (339 B) copied, 0.0005884 s= , 576 kB/s\n'; <rc> =3D 0<o:p></o:p></span></p><p class=3DMsoNormal><= span style=3D'font-family:"Courier New"'>Thread-28::DEBUG::2016-02-25 11:44= :05,583::fileSD::173::Storage.Misc.excCmd::(getReadDelay) /usr/bin/taskset = --cpu-list 0-63 /usr/bin/dd if=3D/rhev/data-center/mnt/glusterSD/srv-stor-0= 1-ib0:_virtualmachines/5f7991ba-fdf8-4b40-9974-c7adcd4da879/dom_md/metadata= iflag=3Ddirect of=3D/dev/null bs=3D4096 count=3D1 (cwd None)<o:p></o:p></s= pan></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>Thre= ad-28::DEBUG::2016-02-25 11:44:05,606::fileSD::173::Storage.Misc.excCmd::(g= etReadDelay) SUCCESS: <err> =3D '0+1 records in\n0+1 records out\n482= bytes (482 B) copied, 0.000637557 s, 756 kB/s\n'; <rc> =3D 0<o:p></o= :p></span></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma= l><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class= =3DMsoNormal>Other commands (executed on the host having the problem) which= probably give useful information:<o:p></o:p></p><p class=3DMsoNormal><span= style=3D'font-family:"Courier New"'># vdsClient -s 0 getConnectedStoragePo= olsList<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-famil= y:"Courier New"'>00000001-0001-0001-0001-00000000020e<o:p></o:p></span></p>= <p class=3DMsoNormal><span style=3D'font-family:"Courier New"'><o:p> <= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier Ne= w"'># vdsClient -s 0 getStoragePoolInfo 00000001-0001-0001-0001-00000000020= e<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cou= rier New"'> name =3D No Descripti= on<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Co= urier New"'> isoprefix =3D /rhev/= data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a28= 3-060d4c83303c/images/11111111-1111-1111-1111-111111111111<o:p></o:p></span= ></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> &= nbsp; pool_status =3D connected<o:p></o:p></s= pan></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&nbs= p; lver =3D 6<o:p></o:p></span></p><p c= lass=3DMsoNormal><span style=3D'font-family:"Courier New"'> &nbs= p; spm_id =3D 2<o:p></o:p></span></p><p class=3DMso= Normal><span style=3D'font-family:"Courier New"'> &n= bsp; master_uuid =3D 5f7991ba-fdf8-4b40-9974-c7adcd4da879<o:p><= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier Ne= w"'> version =3D 3<o:p></o:p></sp= an></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> = ; domains =3D 5f7991ba-fdf8-4b40-9974-c= 7adcd4da879:Active,6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd:Active,5efea9c7-c4e= c-44d4-a283-060d4c83303c:Active,e84c6a1a-9f82-4fa6-9a3b-0b0bc0330d9a:Active= <o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cour= ier New"'> type =3D GLUSTERFS<o:p= ></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier = New"'> master_ver =3D 1<o:p></o:p= ></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>= 5f7991ba-fdf8-4b40-9974-c7adcd4d= a879 =3D {'status': 'Active', 'diskfree': '6374172262400', 'isoprefix': '',= 'alerts': [], 'disktotal': '6995436371968', 'version': 3}<o:p></o:p></span= ></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'> &= nbsp; 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd = =3D {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}<o:p>= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier N= ew"'> e84c6a1a-9f82-4fa6-9a3b-0b0= bc0330d9a =3D {'status': 'Active', 'diskfree': '224145833984', 'isoprefix':= '', 'alerts': [], 'disktotal': '236317179904', 'version': 3}<o:p></o:p></s= pan></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>&nbs= p; 5efea9c7-c4ec-44d4-a283-060d4c83303c= =3D {'status': 'Active', 'diskfree': '6374172262400', 'isoprefix': '/rhev/= data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain/5efea9c7-c4ec-44d4-a28= 3-060d4c83303c/images/11111111-1111-1111-1111-111111111111', 'alerts': [], = 'disktotal': '6995436371968', 'version': 0}<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-family:"Courier New"'><o:p> </o:p></s= pan></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'># vd= sClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p>= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier N= ew"'>Storage domain does not exist: ('6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd'= ,)<o:p></o:p></span></p><p class=3DMsoNormal><o:p> </o:p></p><p class= =3DMsoNormal>If I run this last command on one of the working hosts:<o:p></= o:p></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'># vd= sClient -s 0 getStorageDomainInfo 6fb10a49-5f1c-4bd4-9ff7-b7e33c1125cd<o:p>= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier N= ew"'> uuid =3D 6fb10a49-5f1c-4bd4= -9ff7-b7e33c1125cd<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D= 'font-family:"Courier New"'> vers= ion =3D 3<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-fam= ily:"Courier New"'> role =3D Regu= lar<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"C= ourier New"'> remotePath =3D srv-= stor-01-ib0:/ovirtengine<o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-family:"Courier New"'>  = ; type =3D GLUSTERFS<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'> c= lass =3D Data<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font= -family:"Courier New"'> pool =3D = ['00000001-0001-0001-0001-00000000020e']<o:p></o:p></span></p><p class=3DMs= oNormal><span style=3D'font-family:"Courier New"'> &= nbsp; name =3D hosted_storage<o:p></o:p></span></p><p class=3DM= soNormal><o:p> </o:p></p><p class=3DMsoNormal>(please note: this is th= e storage domain used for my hosted engine)<o:p></o:p></p><p class=3DMsoNor= mal><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class= =3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>If I run “moun= t” on my AMD host (the one with the problem):<o:p></o:p></p><p class= =3DMsoNormal><span style=3D'font-family:"Courier New"'># mount<o:p></o:p></= span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>...= <o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cour= ier New"'>srv-stor-01-ib0:/virtualmachines on /rhev/data-center/mnt/gluster= SD/srv-stor-01-ib0:_virtualmachines type fuse.glusterfs (rw,relatime,user_i= d=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072)<o:p><= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Courier Ne= w"'>srv-stor-01-ib0:/isodomain on /rhev/data-center/mnt/glusterSD/srv-stor-= 01-ib0:_isodomain type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0= ,default_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p= class=3DMsoNormal><span style=3D'font-family:"Courier New"'>srv-stor-01-ib= 0:/ssd-pcie on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie ty= pe fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions= ,allow_other,max_read=3D131072)<o:p></o:p></span></p><p class=3DMsoNormal><= span style=3D'font-family:"Courier New"'>tmpfs on /run/user/0 type tmpfs (r= w,nosuid,nodev,relatime,size=3D13185796k,mode=3D700)<o:p></o:p></span></p><= p class=3DMsoNormal><span style=3D'font-family:"Courier New"'><o:p> </= o:p></span></p><p class=3DMsoNormal>If I run “mount” on one of = the Intel hosts (currently working):<o:p></o:p></p><p class=3DMsoNormal><sp= an style=3D'font-family:"Courier New"'># mount<o:p></o:p></span></p><p clas= s=3DMsoNormal><span style=3D'font-family:"Courier New"'>...<o:p></o:p></spa= n></p><p class=3DMsoNormal><span style=3D'font-family:"Courier New"'>srv-st= or-01-ib0:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_= ovirtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,defau= lt_permissions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-family:"Courier New"'>srv-stor-01-ib0:/vir= tualmachines on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_virtualmac= hines type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default_per= missions,allow_other,max_read=3D131072)<o:p></o:p></span></p><p class=3DMso= Normal><span style=3D'font-family:"Courier New"'>srv-stor-01-ib0:/isodomain= on /rhev/data-center/mnt/glusterSD/srv-stor-01-ib0:_isodomain type fuse.gl= usterfs (rw,relatime,user_id=3D0,group_id=3D0,default_permissions,allow_oth= er,max_read=3D131072)<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-family:"Courier New"'>srv-stor-01-ib0:/ssd-pcie on /rhev/data-cent= er/mnt/glusterSD/srv-stor-01-ib0:_ssd-pcie type fuse.glusterfs (rw,relatime= ,user_id=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072= )<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-family:"Cou= rier New"'>tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size= =3D3272288k,mode=3D700)<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-family:"Courier New"'><o:p> </o:p></span></p><p class=3DMso= Normal>The only difference in “mount” is that the hosted-engine= storage domain is not loaded on the host which should not run the engine. = The other domains are mounted correctly.<o:p></o:p></p><p class=3DMsoNormal= ><o:p> </o:p></p><p class=3DMsoNormal>What could I do to solve this is= sue?<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMso= Normal>Best regards,<o:p></o:p></p><p class=3DMsoNormal> &= nbsp; Giuseppe<o:p></o:p></p><p class=3DMsoNormal><o:p>&n= bsp;</o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma= l style=3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style= =3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language:IT= '>--<br>Giuseppe Berellini<o:p></o:p></span></p><p class=3DMsoNormal style= =3D'mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style=3D'font= -size:10.0pt;font-family:"Arial",sans-serif;color:red;mso-fareast-language:= IT'>PTV SISTeMA<o:p></o:p></span></p><p class=3DMsoNormal style=3D'mso-marg= in-top-alt:auto;mso-margin-bottom-alt:auto'><span style=3D'font-size:10.0pt= ;font-family:"Arial",sans-serif;mso-fareast-language:IT'><a href=3D"http://= www.sistemaits.com/"><span style=3D'color:black;text-decoration:none'>www.s= istemaits.com</span></a><o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-size:10.0pt;font-family:"Arial",sans-serif;mso-fareast-language= :IT'><a href=3D"https://www.facebook.com/sistemaits"><span style=3D'color:b= lack;text-decoration:none'>facebook.com/sistemaits</span></a><br><a href=3D= "https://www.linkedin.com/company/sistema-soluzioni-per-l-ingegneria-dei-si= stemi-di-trasporto-e-l-infomobilit-s-r-l-"><span style=3D'color:black;text-= decoration:none'>linkedin.com/SISTeMA</span></a><br> </span><o:p></o:p= ></p></div></body></html>=
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AD30C17SKAWNE25CEXpt_--
participants (1)
-
Giuseppe Berellini