New subject: R: Problem installing oVirt 3.6.3-rc with GlusterFS over InfiniBand

9 Feb 2016

      ...
        emulatedMachine =3D pc<o:p></o:=
...
</o:p></p><p class=3DMsoNormal>    return func(inst, *args,=
 **kwargs)<o:p></o:p></p><p class=3DMsoNormal>  File "/usr/lib64/=
...
</p><p class=3DMsoNormal>    if ret is None:raise libvirtEr=
ror('virDomainCreateXML() failed', conn=3Dself)<o:p></o:p></p><p class=3DMs=
oNormal>libvirtError: Failed to acquire lock: No space left on device<o:p><=
/o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>But=
 <o:p></o:p></p><p class=3DMsoNormal>[root@SRV-VMHOST-05 vdsm]# df -h<o:p><=
/o:p></p><p class=3DMsoNormal>Filesystem      =
;            &n=
bsp;            Size=
  Used Avail Use% Mounted on<o:p></o:p></p><p class=3DMsoNormal>/dev/m=
apper/centos_srv--vmhost--05-root   50G  2.8G   48=
G   6% /<o:p></o:p></p><p class=3DMsoNormal>devtmpfs  &=
nbsp;           &nbs=
...
<p class=3DMsoNormal>tmpfs        =
            &nb=
sp;            =
    16G     0   16G  =
; 0% /sys/fs/cgroup<o:p></o:p></p><p class=3DMsoNormal>/dev/mapper/centos_s=
rv--vmhost--05-home   84G   33M   84G &n=
bsp; 1% /home<o:p></o:p></p><p class=3DMsoNormal>/dev/sda1   =
;             &=
nbsp;           &nbs=
...
</o:p></p><p class=3DMsoNormal>srv-stor-01:/ovirtengine on /rhev/data-cent=
er/mnt/glusterSD/srv-stor-01:_ovirtengine type fuse.glusterfs (rw,relatime,=
user_id=3D0,group_id=3D0,default_permissions,allow_other,max_read=3D131072)=
<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorm=
al>(if I create a file in that folder, it appears on the gluster server).<o=
:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal=
<o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3D=
MsoNormal>On the engine VM in /var/log/ovirt-engine/engine.log I found the =
following:<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:41,165 INFO&=
nbsp; [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVDSCommand] (Defau=
ltQuartzScheduler_Worker-93) [] START, FullListVDSCommand(HostName =3D , Fu=
llListVDSCommandParameters:{runAsyn<o:p></o:p></p><p class=3DMsoNormal>c=3D=
'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', vds=3D'Host[,13ce3=
8e6-f4b6-42fa-bb8c-5ec84ad00ce0]', vmIds=3D'[429eec6e-2126-4740-9911-9c5ad4=
82e09f]'}), log id: 61eda464<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 =
11:55:42,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.FullList=
VDSCommand] (DefaultQuartzScheduler_Worker-93) [] FINISH, FullListVDSComman=
d, return: [{status=3DUp, nicModel=3Drtl8139,pv, emulat<o:p></o:p></p><p cl=
ass=3DMsoNormal>edMachine=3Dpc, guestDiskMapping=3D{}, vmId=3D429eec6e-2126=
-4740-9911-9c5ad482e09f, pid=3D11133, devices=3D[Ljava.lang.Object;@2099d01=
1, smp=3D4, vmType=3Dkvm, displayIp=3D0, display=3Dvnc, displaySecurePort=
=3D-1, memSize=3D8192,<o:p></o:p></p><p class=3DMsoNormal> displayPort=3D59=
00, cpuType=3DNehalem, spiceSecureChannels=3Dsmain,sdisplay,sinputs,scursor=
,splayback,srecord,ssmartcard,susbredir, statusTime=3D4364469020, vmName=3D=
HostedEngine, clientIp=3D, pauseCode=3DNOERR}], log id<o:p></o:p></p><p cla=
ss=3DMsoNormal>: 61eda464<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:=
55:42,173 INFO  [org.ovirt.engine.core.bll.storage.GetExistingStorageD=
omainListQuery] (org.ovirt.thread.pool-8-thread-35) [] START, GetExistingSt=
orageDomainListQuery(GetExistingStorageDomainLis<o:p></o:p></p><p class=3DM=
soNormal>tParameters:{refresh=3D'true', filtered=3D'false'}), log id: 5611a=
666<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:42,173 INFO  [=
org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSComman=
d] (org.ovirt.thread.pool-8-thread-35) [] START, HSMGetStorageDomainsListVD=
SCommand(HostName =3D srv-vm<o:p></o:p></p><p class=3DMsoNormal>host-05, HS=
MGetStorageDomainsListVDSCommandParameters:{runAsync=3D'true', hostId=3D'13=
ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', storagePoolId=3D'00000000-0000-0000-00=
00-000000000000', storageType=3D'null', storageDomainT<o:p></o:p></p><p cla=
ss=3DMsoNormal>ype=3D'Data', path=3D'null'}), log id: 63695be3<o:p></o:p></=
...
</p><p class=3DMsoNormal>        at org=
.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)<o=
:p></o:p></p><p class=3DMsoNormal>       =
; at org.jboss.invocation.InterceptorContext$Invocation.proceed(Interceptor=
Context.java:407)<o:p></o:p></p><p class=3DMsoNormal>   &nbs=
...
</o:p></p><p class=3DMsoNormal>        =
at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesComma=
ndBase.java:82) [bll.jar:]<o:p></o:p></p><p class=3DMsoNormal>  &=
nbsp;     at org.ovirt.engine.core.dal.VdcCommandBase.e=
xecute(VdcCommandBase.java:33) [dal.jar:]<o:p></o:p></p><p class=3DMsoNorma=
l>        at org.ovirt.engine.core.bll.B=
ackend.runQueryImpl(Backend.java:537) [bll.jar:]<o:p></o:p></p><p class=3DM=
soNormal>        at org.ovirt.engine.cor=
e.bll.Backend.runQuery(Backend.java:511) [bll.jar:]<o:p></o:p></p><p class=
=3DMsoNormal>        at sun.reflect.Gene=
ratedMethodAccessor98.invoke(Unknown Source) [:1.8.0_71]<o:p></o:p></p><p c=
lass=3DMsoNormal>        at sun.reflect.=
DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [=
rt.jar:1.8.0_71]<o:p></o:p></p><p class=3DMsoNormal>    =
;    at java.lang.reflect.Method.invoke(Method.java:497) [rt=
.jar:1.8.0_71]<o:p></o:p></p><p class=3DMsoNormal>    &=
nbsp;   at org.jboss.as.ee.component.ManagedReferenceMethodInterc=
eptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)<o:p></o:=
...
<p class=3DMsoNormal>Do you have any ideas about what I should do?<o:p></o=
:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Thank=
s,<o:p></o:p></p><p class=3DMsoNormal>      &=
nbsp; Giuseppe<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p c=
lass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal style=3D'mso-mar=
gin-top-alt:auto;mso-margin-bottom-alt:auto'><span style=3D'font-size:10.0p=
t;font-family:"Arial",sans-serif;mso-fareast-language:IT'>--<br>Giuseppe Be=
rellini<o:p></o:p></span></p><p class=3DMsoNormal style=3D'mso-margin-top-a=
lt:auto;mso-margin-bottom-alt:auto'><span style=3D'font-size:10.0pt;font-fa=
mily:"Arial",sans-serif;color:red;mso-fareast-language:IT'>PTV SISTeMA<o:p>=
</o:p></span></p><p class=3DMsoNormal style=3D'mso-margin-top-alt:auto;mso-=
margin-bottom-alt:auto'><span style=3D'font-size:10.0pt;font-family:"Arial"=
,sans-serif;mso-fareast-language:IT'>Phone +39 06 993 444 15<br>Mobile +39 =
349 3241969<br>Fax +39 06 993 348 72<br>Via Ruggero Bonghi, 11/B – 00=
184 Roma<o:p></o:p></span></p><p class=3DMsoNormal style=3D'mso-margin-top-=
alt:auto;mso-margin-bottom-alt:auto'><span style=3D'font-size:10.0pt;font-f=
amily:"Arial",sans-serif;mso-fareast-language:IT'>giuseppe.berellini@ptvgro=
up.com<br><a href=3D"http://www.sistemaits.com/"><span style=3D'color:black=
;text-decoration:none'>www.sistemaits.com</span></a><o:p></o:p></span></p><=
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AA9FA1ESKAWNE25CEXpt_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Hi,

I'm trying to setup oVirt 3.6.3 with self-hosted engine on 4 servers (vmhos=
t-03, vmhost-04, vmhost-05 for compute; stor-01 for storage). The storage s=
erver is GlusterFS 3.7.6, all the servers are in the same network and are a=
lso connected through InfiniBand DDR.

Network is OK, RDMA is working, IPoIB has been configured, it is possible t=
o manually mount GlusterFS volumes on each vmhost. firewalld and SELinux ar=
e disabled. Ovirtmgmt network is on ethernet.

The problem is that, after installing the hosted engine, I can connect to o=
Virt admin panel but:
- Datacenter is marked as down
- The only host is NOT recognized as an SPM
- In the storage tab there is no storage domain for the hosted engine (I on=
ly see a detached ISO domain and oVirt repo)
- when I try to create a storage domain, an error shows up (it's an "Uncaug=
ht exception")
- when I try to import a storage domain, an error shows up (it's about data=
center down and SPM not available)
- also, in Virtual Machines tab there are no VMs (neither the hosted engine=
, which is obviously up and reported as up by command "hosted-engine --vm-s=
tatus")

So basically it is not possible to do anything.
After setting the host in maintenance mode and rebooting, I cannot start th=
e engine VM anymore:

[root@SRV-VMHOST-05 ~]# hosted-engine --vm-start
VM exists and is down, destroying it
Machine destroyed

429eec6e-2126-4740-9911-9c5ad482e09f
        Status =3D WaitForLaunch
        nicModel =3D rtl8139,pv
        statusTime =3D 4300834920
        emulatedMachine =3D pc
        pid =3D 0
        vmName =3D HostedEngine
        devices =3D [{'index': '2', 'iface': 'ide', 'specParams': {}, 'read=
only': 'true', 'deviceId': '1c2205da-17c6-4ffe-9408-602a998d90dc', 'address=
': {'bus': '1', 'controller': '0', 'type': 'drive', 'target': '0', 'unit': =
'0'}, 'device': 'cdrom', 'shared': 'false', 'path': '', 'type': 'disk'}, {'=
index': '0', 'iface': 'virtio', 'format': 'raw', 'bootOrder': '1', 'poolID'=
: '00000000-0000-0000-0000-000000000000', 'volumeID': 'fe82ba21-942d-48cc-9=
bdb-f41c0f172dde', 'imageID': '131460bc-4599-4326-a026-e9e224e4bb5f', 'spec=
Params': {}, 'readonly': 'false', 'domainID': '162fc2e5-1897-46fb-b382-195c=
11ab8546', 'optional': 'false', 'deviceId': '131460bc-4599-4326-a026-e9e224=
e4bb5f', 'address': {'slot': '0x06', 'bus': '0x00', 'domain': '0x0000', 'ty=
pe': 'pci', 'function': '0x0'}, 'device': 'disk', 'shared': 'exclusive', 'p=
ropagateErrors': 'off', 'type': 'disk'}, {'device': 'scsi', 'model': 'virti=
o-scsi', 'type': 'controller'}, {'nicModel': 'pv', 'macAddr': '00:16:3e:30:=
a9:6e', 'linkActive': 'true', 'network': 'ovirtmgmt', 'filter': 'vdsm-no-ma=
c-spoofing', 'specParams': {}, 'deviceId': '3d3259a3-19a8-42c3-a50c-6724b47=
5c1ab', 'address': {'slot': '0x03', 'bus': '0x00', 'domain': '0x0000', 'typ=
e': 'pci', 'function': '0x0'}, 'device': 'bridge', 'type': 'interface'}, {'=
device': 'console', 'specParams': {}, 'type': 'console', 'deviceId': '885cc=
a16-2b59-42e4-a57c-0a89a0e823e8', 'alias': 'console0'}]
        guestDiskMapping =3D {}
        vmType =3D kvm
        clientIp =3D
        displaySecurePort =3D -1
        memSize =3D 8192
        displayPort =3D -1
        cpuType =3D Nehalem
        spiceSecureChannels =3D smain,sdisplay,sinputs,scursor,splayback,sr=
ecord,ssmartcard,susbredir
        smp =3D 4
        displayIp =3D 0
        display =3D vnc
but the status remains {"reason": "bad vm status", "health": "bad", "vm": "=
down", "detail": "down"}
We tried to use, for the engine volume, both rdma and tcp - nothing changed

In /var/log/ovirt-hosted-engine-ha/agent.log , these are the only error we =
found:

MainThread::WARNING::2016-02-08 18:17:23,160::ovf_store::105::ovirt_hosted_=
engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Unable to find OVF_STORE
MainThread::ERROR::2016-02-08 18:17:23,161::config::234::ovirt_hosted_engin=
e_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_conf_file) Una=
ble to get vm.conf from OVF_STORE, falling back to initial vm.conf

In vdsm.og I see
Thread-16399::INFO::2016-02-09 14:54:39,478::xmlrpc::84::vds.XMLRPCServer::=
(_process_requests) Request handler for 127.0.0.1:39823 started
Thread-16399::DEBUG::2016-02-09 14:54:39,478::bindingxmlrpc::1257::vds::(wr=
apper) client [127.0.0.1]::call vmGetStats with ('429eec6e-2126-4740-9911-9=
c5ad482e09f',) {}
Thread-16399::DEBUG::2016-02-09 14:54:39,479::bindingxmlrpc::1264::vds::(wr=
apper) return vmGetStats with {'status': {'message': 'Done', 'code': 0}, 's=
tatsList': [{'status': 'Down', 'exitMessage': 'Failed to acquire lock: No s=
pace left on device', 'statusTime': '4302636100', 'vmId': '429eec6e-2126-47=
40-9911-9c5ad482e09f', 'exitReason': 1, 'exitCode': 1}]}

When executing hosted-engine -vm-start, in vdsm.log appears this:
Thread-16977::ERROR::2016-02-09 14:59:12,146::vm::759::virt.vm::(_startUnde=
rlyingVm) vmId=3D`429eec6e-2126-4740-9911-9c5ad482e09f`::The vm start proce=
ss failed
Traceback (most recent call last):
  File "/usr/share/vdsm/virt/vm.py", line 703, in _startUnderlyingVm
    self._run()
  File "/usr/share/vdsm/virt/vm.py", line 1941, in _run
    self._connection.createXML(domxml, flags),
  File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 1=
24, in wrapper
    ret =3D f(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrap=
per
    return func(inst, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in creat=
eXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=
=3Dself)
libvirtError: Failed to acquire lock: No space left on device

But
[root@SRV-VMHOST-05 vdsm]# df -h
Filesystem                               Size  Used Avail Use% Mounted on
/dev/mapper/centos_srv--vmhost--05-root   50G  2.8G   48G   6% /
devtmpfs                                  16G     0   16G   0% /dev
tmpfs                                     16G     0   16G   0% /dev/shm
tmpfs                                     16G  105M   16G   1% /run
tmpfs                                     16G     0   16G   0% /sys/fs/cgro=
up
/dev/mapper/centos_srv--vmhost--05-home   84G   33M   84G   1% /home
/dev/sda1                                497M  178M  319M  36% /boot
srv-stor-01:/ovirtengine                 3.7T  3.0G  3.7T   1% /rhev/data-c=
enter/mnt/glusterSD/srv-stor-01:_ovirtengine
tmpfs                                    3.2G     0  3.2G   0% /run/user/0

I also verified that Gluster storage was correctly mounted:
[root@SRV-VMHOST-05 ~]# mount | grep gluster
srv-stor-01:/ovirtengine on /rhev/data-center/mnt/glusterSD/srv-stor-01:_ov=
irtengine type fuse.glusterfs (rw,relatime,user_id=3D0,group_id=3D0,default=
_permissions,allow_other,max_read=3D131072)

(if I create a file in that folder, it appears on the gluster server).

On the engine VM in /var/log/ovirt-engine/engine.log I found the following:
2016-02-09 11:55:41,165 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.Fu=
llListVDSCommand] (DefaultQuartzScheduler_Worker-93) [] START, FullListVDSC=
ommand(HostName =3D , FullListVDSCommandParameters:{runAsyn
c=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', vds=3D'Host[,1=
3ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0]', vmIds=3D'[429eec6e-2126-4740-9911-9c=
5ad482e09f]'}), log id: 61eda464
2016-02-09 11:55:42,169 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.Fu=
llListVDSCommand] (DefaultQuartzScheduler_Worker-93) [] FINISH, FullListVDS=
Command, return: [{status=3DUp, nicModel=3Drtl8139,pv, emulat
edMachine=3Dpc, guestDiskMapping=3D{}, vmId=3D429eec6e-2126-4740-9911-9c5ad=
482e09f, pid=3D11133, devices=3D[Ljava.lang.Object;@2099d011, smp=3D4, vmTy=
pe=3Dkvm, displayIp=3D0, display=3Dvnc, displaySecurePort=3D-1, memSize=3D8=
192,
displayPort=3D5900, cpuType=3DNehalem, spiceSecureChannels=3Dsmain,sdisplay=
,sinputs,scursor,splayback,srecord,ssmartcard,susbredir, statusTime=3D43644=
69020, vmName=3DHostedEngine, clientIp=3D, pauseCode=3DNOERR}], log id
: 61eda464
2016-02-09 11:55:42,173 INFO  [org.ovirt.engine.core.bll.storage.GetExistin=
gStorageDomainListQuery] (org.ovirt.thread.pool-8-thread-35) [] START, GetE=
xistingStorageDomainListQuery(GetExistingStorageDomainLis
tParameters:{refresh=3D'true', filtered=3D'false'}), log id: 5611a666
2016-02-09 11:55:42,173 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainsListVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] ST=
ART, HSMGetStorageDomainsListVDSCommand(HostName =3D srv-vm
host-05, HSMGetStorageDomainsListVDSCommandParameters:{runAsync=3D'true', h=
ostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', storagePoolId=3D'00000000-0=
000-0000-0000-000000000000', storageType=3D'null', storageDomainT
ype=3D'Data', path=3D'null'}), log id: 63695be3
2016-02-09 11:55:43,298 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainsListVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] FI=
NISH, HSMGetStorageDomainsListVDSCommand, return: [162fc2
e5-1897-46fb-b382-195c11ab8546], log id: 63695be3
2016-02-09 11:55:43,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] STA=
RT, HSMGetStorageDomainInfoVDSCommand(HostName =3D srv-vmho
st-05, HSMGetStorageDomainInfoVDSCommandParameters:{runAsync=3D'true', host=
Id=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', storageDomainId=3D'162fc2e5-18=
97-46fb-b382-195c11ab8546'}), log id: 7e520f35
2016-02-09 11:55:44,377 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] FIN=
ISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageD
omainStatic:{name=3D'EngineStorage', id=3D'162fc2e5-1897-46fb-b382-195c11ab=
8546'}, null>, log id: 7e520f35
2016-02-09 11:55:44,377 INFO  [org.ovirt.engine.core.bll.storage.GetExistin=
gStorageDomainListQuery] (org.ovirt.thread.pool-8-thread-35) [] FINISH, Get=
ExistingStorageDomainListQuery, log id: 5611a666
2016-02-09 11:55:44,378 INFO  [org.ovirt.engine.core.bll.ImportHostedEngine=
StorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [23427de7] Lock A=
cquired to object 'EngineLock:{exclusiveLocks=3D'[]', share
dLocks=3D'null'}'
2016-02-09 11:55:44,379 WARN  [org.ovirt.engine.core.bll.ImportHostedEngine=
StorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [23427de7] CanDoA=
ction of action 'ImportHostedEngineStorageDomain' failed
for user SYSTEM. Reasons: VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTIO=
N_TYPE_FAILED_MASTER_STORAGE_DOMAIN_NOT_ACTIVE
2016-02-09 11:55:44,379 INFO  [org.ovirt.engine.core.bll.ImportHostedEngine=
StorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [23427de7] Lock f=
reed to object 'EngineLock:{exclusiveLocks=3D'[]', sharedLo
cks=3D'null'}'
2016-02-09 11:55:46,625 INFO  [org.ovirt.engine.core.bll.UpdateVdsGroupComm=
and] (default task-26) [5118b768] Running command: UpdateVdsGroupCommand in=
ternal: false. Entities affected :  ID: 00000002-0002-000
2-0002-0000000000d9 Type: VdsGroupsAction group EDIT_CLUSTER_CONFIGURATION =
with role type ADMIN
2016-02-09 11:55:46,765 INFO  [org.ovirt.engine.core.dal.dbbroker.auditlogh=
andling.AuditLogDirector] (default task-26) [5118b768] Correlation ID: 5118=
b768, Call Stack: null, Custom Event ID: -1, Message: Hos
t cluster Default was updated by admin@internal
2016-02-09 11:55:46,932 INFO  [org.ovirt.engine.core.vdsbroker.gluster.Glus=
terServersListVDSCommand] (default task-6) [] START, GlusterServersListVDSC=
ommand(HostName =3D srv-vmhost-05, VdsIdVDSCommandParameter
sBase:{runAsync=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'})=
, log id: 559ab127
2016-02-09 11:55:47,503 INFO  [org.ovirt.engine.core.vdsbroker.gluster.Glus=
terServersListVDSCommand] (default task-13) [] START, GlusterServersListVDS=
Command(HostName =3D srv-vmhost-05, VdsIdVDSCommandParamete
rsBase:{runAsync=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'}=
), log id: 62d703e5
2016-02-09 11:55:47,510 INFO  [org.ovirt.engine.core.vdsbroker.gluster.Glus=
terServersListVDSCommand] (default task-6) [] FINISH, GlusterServersListVDS=
Command, log id: 559ab127
2016-02-09 11:55:47,511 ERROR [org.ovirt.engine.core.bll.gluster.GetAddedGl=
usterServersQuery] (default task-6) [] Query 'GetAddedGlusterServersQuery' =
failed: null
2016-02-09 11:55:47,511 ERROR [org.ovirt.engine.core.bll.gluster.GetAddedGl=
usterServersQuery] (default task-6) [] Exception: java.lang.NullPointerExce=
ption
        at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery.ge=
tAddedGlusterServers(GetAddedGlusterServersQuery.java:54) [bll.jar:]
        at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery.ex=
ecuteQueryCommand(GetAddedGlusterServersQuery.java:45) [bll.jar:]
        at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(Quer=
iesCommandBase.java:82) [bll.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.=
java:33) [dal.jar:]
        at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:537)=
 [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:511) [bl=
l.jar:]
        at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) [:1=
.8.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:43) [rt.jar:1.8.0_71]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_7=
1]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.proc=
essInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorConte=
xt.java:309)
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedI=
nterceptor.java:53)
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.=
processInvocation(UserInterceptorFactory.java:63)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorConte=
xt.java:309)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(Inter=
ceptorContext.java:407)
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delegateIntercep=
tion(Jsr299BindingsInterceptor.java:70) [wildfly-weld-8.2.1.Final.jar:8.2.1=
.Final]
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doMethodIntercep=
tion(Jsr299BindingsInterceptor.java:80) [wildfly-weld-8.2.1.Final.jar:8.2.1=
.Final]
        at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.processInvocatio=
n(Jsr299BindingsInterceptor.java:93) [wildfly-weld-8.2.1.Final.jar:8.2.1.Fi=
nal]
        at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.=
processInvocation(UserInterceptorFactory.java:63)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorConte=
xt.java:309)
        at org.jboss.invocation.InterceptorContext$Invocation.proceed(Inter=
ceptorContext.java:407)
        at org.ovirt.engine.core.bll.interceptors.CorrelationIdTrackerInter=
ceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [bll.jar:]
        at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source) [:1=
.8.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:43) [rt.jar:1.8.0_71]
                               ....
2016-02-09 11:55:47,985 INFO  [org.ovirt.engine.core.vdsbroker.gluster.Glus=
terServersListVDSCommand] (default task-14) [] START, GlusterServersListVDS=
Command(HostName =3D srv-vmhost-05, VdsIdVDSCommandParametersBase:{runAsync=
=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'}), log id: 61100=
c4d
2016-02-09 11:55:47,986 INFO  [org.ovirt.engine.core.vdsbroker.gluster.Glus=
terServersListVDSCommand] (default task-13) [] FINISH, GlusterServersListVD=
SCommand, log id: 62d703e5
2016-02-09 11:55:47,986 ERROR [org.ovirt.engine.core.bll.gluster.GetAddedGl=
usterServersQuery] (default task-13) [] Query 'GetAddedGlusterServersQuery'=
 failed: null
2016-02-09 11:55:47,987 ERROR [org.ovirt.engine.core.bll.gluster.GetAddedGl=
usterServersQuery] (default task-13) [] Exception: java.lang.NullPointerExc=
eption
        at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery.ge=
tAddedGlusterServers(GetAddedGlusterServersQuery.java:54) [bll.jar:]
        at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery.ex=
ecuteQueryCommand(GetAddedGlusterServersQuery.java:45) [bll.jar:]
        at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(Quer=
iesCommandBase.java:82) [bll.jar:]
        at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.=
java:33) [dal.jar:]
        at org.ovirt.engine.core.bll.Backend.runQueryImpl(Backend.java:537)=
 [bll.jar:]
        at org.ovirt.engine.core.bll.Backend.runQuery(Backend.java:511) [bl=
l.jar:]
        at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) [:1=
.8.0_71]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod=
AccessorImpl.java:43) [rt.jar:1.8.0_71]
        at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_7=
1]
        at org.jboss.as.ee.component.ManagedReferenceMethodInterceptor.proc=
essInvocation(ManagedReferenceMethodInterceptor.java:52)
        at org.jboss.invocation.InterceptorContext.proceed(InterceptorConte=
xt.java:309)
        at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedI=
nterceptor.java:53)
                               ....

Do you have any ideas about what I should do?

Thanks,
        Giuseppe

--
Giuseppe Berellini
PTV SISTeMA
Phone +39 06 993 444 15
Mobile +39 349 3241969
Fax +39 06 993 348 72
Via Ruggero Bonghi, 11/B - 00184 Roma
giuseppe.berellini@ptvgroup.com
www.sistemaits.com<http://www.sistemaits.com/>
facebook.com/sistemaits<https://www.facebook.com/sistemaits>
linkedin.com/SISTeMA<https://www.linkedin.com/company/sistema-soluzioni-per=
-l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l->

--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AA9FA1ESKAWNE25CEXpt_
Content-Type: text/html; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

<html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" =
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type content=
=3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros=
oft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:Wingdings;
	panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
	{font-family:"Cambria Math";
	panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:#0563C1;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:#954F72;
	text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
	{mso-style-priority:34;
	margin-top:0cm;
	margin-right:0cm;
	margin-bottom:0cm;
	margin-left:36.0pt;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
span.StileMessaggioDiPostaElettronica17
	{mso-style-type:personal-compose;
	font-family:"Calibri",sans-serif;
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri",sans-serif;
	mso-fareast-language:EN-US;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 2.0cm 2.0cm 2.0cm;}
div.WordSection1
	{page:WordSection1;}
/* List Definitions */
@list l0
	{mso-list-id:1938980631;
	mso-list-type:hybrid;
	mso-list-template-ids:-1062313228 68157441 68157443 68157445 68157441 6815=
7443 68157445 68157441 68157443 68157445;}
@list l0:level1
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Symbol;}
@list l0:level2
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:"Courier New";}
@list l0:level3
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Wingdings;}
@list l0:level4
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Symbol;}
@list l0:level5
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:"Courier New";}
@list l0:level6
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Wingdings;}
@list l0:level7
	{mso-level-number-format:bullet;
	mso-level-text:\F0B7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Symbol;}
@list l0:level8
	{mso-level-number-format:bullet;
	mso-level-text:o;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:"Courier New";}
@list l0:level9
	{mso-level-number-format:bullet;
	mso-level-text:\F0A7;
	mso-level-tab-stop:none;
	mso-level-number-position:left;
	text-indent:-18.0pt;
	font-family:Wingdings;}
ol
	{margin-bottom:0cm;}
ul
	{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DIT link=3D"#0563C1" v=
link=3D"#954F72"><div class=3DWordSection1><p class=3DMsoNormal>Hi,<o:p></o=
:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>IR=
17;m trying to setup oVirt 3.6.3 with self-hosted engine on 4 servers (vmho=
st-03, vmhost-04, vmhost-05 for compute; stor-01 for storage). The storage =
server is GlusterFS 3.7.6, all the servers are in the same network and are =
also connected through InfiniBand DDR.<o:p></o:p></p><p class=3DMsoNormal><=
o:p> </o:p></p><p class=3DMsoNormal>Network is OK, RDMA is working, IP=
oIB has been configured, it is possible to manually mount GlusterFS volumes=
 on each vmhost. firewalld and SELinux are disabled. Ovirtmgmt network is o=
n ethernet.<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p clas=
s=3DMsoNormal>The problem is that, after installing the hosted engine, I ca=
n connect to oVirt admin panel but:<o:p></o:p></p><p class=3DMsoNormal>- Da=
tacenter is marked as down<o:p></o:p></p><p class=3DMsoNormal>- The only ho=
st is NOT recognized as an SPM<o:p></o:p></p><p class=3DMsoNormal>- In the =
storage tab there is no storage domain for the hosted engine (I only see a =
detached ISO domain and oVirt repo)<o:p></o:p></p><p class=3DMsoNormal>- wh=
en I try to create a storage domain, an error shows up (it’s an ̶=
0;Uncaught exception”)<o:p></o:p></p><p class=3DMsoNormal>- when I tr=
y to import a storage domain, an error shows up (it’s about datacente=
r down and SPM not available)<o:p></o:p></p><p class=3DMsoNormal>- also, in=
 Virtual Machines tab there are no VMs (neither the hosted engine, which is=
 obviously up and reported as up by command “hosted-engine --vm-statu=
s”)<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal>So basically it is not possible to do anything.<o:p></o:p></p>=
<p class=3DMsoNormal>After setting the host in maintenance mode and rebooti=
ng, I cannot start the engine VM anymore:<o:p></o:p></p><p class=3DMsoNorma=
l><o:p> </o:p></p><p class=3DMsoNormal>[root@SRV-VMHOST-05 ~]# hosted-=
engine --vm-start<o:p></o:p></p><p class=3DMsoNormal>VM exists and is down,=
 destroying it<o:p></o:p></p><p class=3DMsoNormal>Machine destroyed<o:p></o=
:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>429ee=
c6e-2126-4740-9911-9c5ad482e09f<o:p></o:p></p><p class=3DMsoNormal> &n=
bsp;      Status =3D WaitForLaunch<o:p></o:p></p><=
p class=3DMsoNormal>        nicModel =3D=
 rtl8139,pv<o:p></o:p></p><p class=3DMsoNormal>    &nbs=
p;   statusTime =3D 4300834920<o:p></o:p></p><p class=3DMsoNormal=
p></p><p class=3DMsoNormal>        pid =
=3D 0<o:p></o:p></p><p class=3DMsoNormal>     &nbs=
p;  vmName =3D HostedEngine<o:p></o:p></p><p class=3DMsoNormal> &=
nbsp;      devices =3D [{'index': '2', 'iface': 'i=
de', 'specParams': {}, 'readonly': 'true', 'deviceId': '1c2205da-17c6-4ffe-=
9408-602a998d90dc', 'address': {'bus': '1', 'controller': '0', 'type': 'dri=
ve', 'target': '0', 'unit': '0'}, 'device': 'cdrom', 'shared': 'false', 'pa=
th': '', 'type': 'disk'}, {'index': '0', 'iface': 'virtio', 'format': 'raw'=
, 'bootOrder': '1', 'poolID': '00000000-0000-0000-0000-000000000000', 'volu=
meID': 'fe82ba21-942d-48cc-9bdb-f41c0f172dde', 'imageID': '131460bc-4599-43=
26-a026-e9e224e4bb5f', 'specParams': {}, 'readonly': 'false', 'domainID': '=
162fc2e5-1897-46fb-b382-195c11ab8546', 'optional': 'false', 'deviceId': '13=
1460bc-4599-4326-a026-e9e224e4bb5f', 'address': {'slot': '0x06', 'bus': '0x=
00', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'disk=
', 'shared': 'exclusive', 'propagateErrors': 'off', 'type': 'disk'}, {'devi=
ce': 'scsi', 'model': 'virtio-scsi', 'type': 'controller'}, {'nicModel': 'p=
v', 'macAddr': '00:16:3e:30:a9:6e', 'linkActive': 'true', 'network': 'ovirt=
mgmt', 'filter': 'vdsm-no-mac-spoofing', 'specParams': {}, 'deviceId': '3d3=
259a3-19a8-42c3-a50c-6724b475c1ab', 'address': {'slot': '0x03', 'bus': '0x0=
0', 'domain': '0x0000', 'type': 'pci', 'function': '0x0'}, 'device': 'bridg=
e', 'type': 'interface'}, {'device': 'console', 'specParams': {}, 'type': '=
console', 'deviceId': '885cca16-2b59-42e4-a57c-0a89a0e823e8', 'alias': 'con=
sole0'}]<o:p></o:p></p><p class=3DMsoNormal>     &=
nbsp;  guestDiskMapping =3D {}<o:p></o:p></p><p class=3DMsoNormal>&nbs=
p;       vmType =3D kvm<o:p></o:p></p><p clas=
s=3DMsoNormal>        clientIp =3D<o:p><=
/o:p></p><p class=3DMsoNormal>        di=
splaySecurePort =3D -1<o:p></o:p></p><p class=3DMsoNormal>   =
;     memSize =3D 8192<o:p></o:p></p><p class=3DMsoNorm=
al>        displayPort =3D -1<o:p></o:p>=
</p><p class=3DMsoNormal>        cpuType=
 =3D Nehalem<o:p></o:p></p><p class=3DMsoNormal>    &nb=
sp;   spiceSecureChannels =3D smain,sdisplay,sinputs,scursor,spla=
yback,srecord,ssmartcard,susbredir<o:p></o:p></p><p class=3DMsoNormal> =
;       smp =3D 4<o:p></o:p></p><p class=3DMs=
oNormal>        displayIp =3D 0<o:p></o:=
p></p><p class=3DMsoNormal>        displ=
ay =3D vnc<o:p></o:p></p><p class=3DMsoNormal>but the status remains {"=
;reason": "bad vm status", "health": "bad&quo=
t;, "vm": "down", "detail": "down"}=
<o:p></o:p></p><p class=3DMsoNormal>We tried to use, for the engine volume,=
 both rdma and tcp – nothing changed<o:p></o:p></p><p class=3DMsoNorm=
al><o:p> </o:p></p><p class=3DMsoNormal>In /var/log/ovirt-hosted-engin=
e-ha/agent.log , these are the only error we found:<o:p></o:p></p><p class=
=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal> MainThread::WARNING=
::2016-02-08 18:17:23,160::ovf_store::105::ovirt_hosted_engine_ha.lib.ovf.o=
vf_store.OVFStore::(scan) Unable to find OVF_STORE<o:p></o:p></p><p class=
=3DMsoNormal>MainThread::ERROR::2016-02-08 18:17:23,161::config::234::ovirt=
_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(refresh_local_c=
onf_file) Unable to get vm.conf from OVF_STORE, falling back to initial vm.=
conf<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMso=
Normal>In vdsm.og I see <o:p></o:p></p><p class=3DMsoNormal>Thread-16399::I=
NFO::2016-02-09 14:54:39,478::xmlrpc::84::vds.XMLRPCServer::(_process_reque=
sts) Request handler for 127.0.0.1:39823 started<o:p></o:p></p><p class=3DM=
soNormal>Thread-16399::DEBUG::2016-02-09 14:54:39,478::bindingxmlrpc::1257:=
:vds::(wrapper) client [127.0.0.1]::call vmGetStats with ('429eec6e-2126-47=
40-9911-9c5ad482e09f',) {}<o:p></o:p></p><p class=3DMsoNormal>Thread-16399:=
:DEBUG::2016-02-09 14:54:39,479::bindingxmlrpc::1264::vds::(wrapper) return=
 vmGetStats with {'status': {'message': 'Done', 'code': 0}, 'statsList': [{=
'status': 'Down', 'exitMessage': 'Failed to acquire lock: No space left on =
device', 'statusTime': '4302636100', 'vmId': '429eec6e-2126-4740-9911-9c5ad=
482e09f', 'exitReason': 1, 'exitCode': 1}]}<o:p></o:p></p><p class=3DMsoNor=
mal><o:p> </o:p></p><p class=3DMsoNormal>When executing hosted-engine =
–vm-start, in vdsm.log appears this:<o:p></o:p></p><p class=3DMsoNorm=
al>Thread-16977::ERROR::2016-02-09 14:59:12,146::vm::759::virt.vm::(_startU=
nderlyingVm) vmId=3D`429eec6e-2126-4740-9911-9c5ad482e09f`::The vm start pr=
ocess failed<o:p></o:p></p><p class=3DMsoNormal>Traceback (most recent call=
 last):<o:p></o:p></p><p class=3DMsoNormal>  File "/usr/share/vds=
m/virt/vm.py", line 703, in _startUnderlyingVm<o:p></o:p></p><p class=
=3DMsoNormal>    self._run()<o:p></o:p></p><p class=3DMsoNor=
mal>  File "/usr/share/vdsm/virt/vm.py", line 1941, in _run<=
o:p></o:p></p><p class=3DMsoNormal>    self._connection.crea=
teXML(domxml, flags),<o:p></o:p></p><p class=3DMsoNormal>  File "=
/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124,=
 in wrapper<o:p></o:p></p><p class=3DMsoNormal>    ret =3D f=
(*args, **kwargs)<o:p></o:p></p><p class=3DMsoNormal>  File "/usr=
/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper<o:p=
python2.7/site-packages/libvirt.py", line 3611, in createXML<o:p></o:p=
p;            &=
nbsp;      16G     0 &nbs=
p; 16G   0% /dev<o:p></o:p></p><p class=3DMsoNormal>tmpfs &n=
bsp;            =
;            &n=
bsp;          16G  &=
nbsp;  0   16G   0% /dev/shm<o:p></o:p></p><p clas=
s=3DMsoNormal>tmpfs         &n=
bsp;            =
;            &n=
bsp;  16G  105M   16G   1% /run<o:p></o:p></p=
p;   497M  178M  319M  36% /boot<o:p></o:p></=
p><p class=3DMsoNormal>srv-stor-01:/ovirtengine    &nbs=
p;            3.7T&n=
bsp; 3.0G  3.7T   1% /rhev/data-center/mnt/glusterSD/srv-sto=
r-01:_ovirtengine<o:p></o:p></p><p class=3DMsoNormal>tmpfs   =
;            &n=
bsp;            =
;        3.2G     0&=
nbsp; 3.2G   0% /run/user/0<o:p></o:p></p><p class=3DMsoNormal><o=
:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMso=
Normal>I also verified that Gluster storage was correctly mounted:<o:p></o:=
p></p><p class=3DMsoNormal>[root@SRV-VMHOST-05 ~]# mount | grep gluster<o:p=
p><p class=3DMsoNormal>2016-02-09 11:55:43,298 INFO  [org.ovirt.engine=
.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (org.ovirt.th=
read.pool-8-thread-35) [] FINISH, HSMGetStorageDomainsListVDSCommand, retur=
n: [162fc2<o:p></o:p></p><p class=3DMsoNormal>e5-1897-46fb-b382-195c11ab854=
6], log id: 63695be3<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:43=
,365 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDom=
ainInfoVDSCommand] (org.ovirt.thread.pool-8-thread-35) [] START, HSMGetStor=
ageDomainInfoVDSCommand(HostName =3D srv-vmho<o:p></o:p></p><p class=3DMsoN=
ormal>st-05, HSMGetStorageDomainInfoVDSCommandParameters:{runAsync=3D'true'=
, hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0', storageDomainId=3D'162fc=
2e5-1897-46fb-b382-195c11ab8546'}), log id: 7e520f35<o:p></o:p></p><p class=
=3DMsoNormal>2016-02-09 11:55:44,377 INFO  [org.ovirt.engine.core.vdsb=
roker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (org.ovirt.thread.pool-8=
-thread-35) [] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <Stora=
geD<o:p></o:p></p><p class=3DMsoNormal>omainStatic:{name=3D'EngineStorage',=
 id=3D'162fc2e5-1897-46fb-b382-195c11ab8546'}, null>, log id: 7e520f35<o=
:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:44,377 INFO  [org.o=
virt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (org.ovirt.=
thread.pool-8-thread-35) [] FINISH, GetExistingStorageDomainListQuery, log =
id: 5611a666<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:44,378 INF=
O  [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] =
(org.ovirt.thread.pool-8-thread-35) [23427de7] Lock Acquired to object 'Eng=
ineLock:{exclusiveLocks=3D'[]', share<o:p></o:p></p><p class=3DMsoNormal>dL=
ocks=3D'null'}'<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:44,379 =
WARN  [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainComman=
d] (org.ovirt.thread.pool-8-thread-35) [23427de7] CanDoAction of action 'Im=
portHostedEngineStorageDomain' failed<o:p></o:p></p><p class=3DMsoNormal>fo=
r user SYSTEM. Reasons: VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_=
TYPE_FAILED_MASTER_STORAGE_DOMAIN_NOT_ACTIVE<o:p></o:p></p><p class=3DMsoNo=
rmal>2016-02-09 11:55:44,379 INFO  [org.ovirt.engine.core.bll.ImportHo=
stedEngineStorageDomainCommand] (org.ovirt.thread.pool-8-thread-35) [23427d=
e7] Lock freed to object 'EngineLock:{exclusiveLocks=3D'[]', sharedLo<o:p><=
/o:p></p><p class=3DMsoNormal>cks=3D'null'}'<o:p></o:p></p><p class=3DMsoNo=
rmal>2016-02-09 11:55:46,625 INFO  [org.ovirt.engine.core.bll.UpdateVd=
sGroupCommand] (default task-26) [5118b768] Running command: UpdateVdsGroup=
Command internal: false. Entities affected :  ID: 00000002-0002-000<o:=
p></o:p></p><p class=3DMsoNormal>2-0002-0000000000d9 Type: VdsGroupsAction =
group EDIT_CLUSTER_CONFIGURATION with role type ADMIN<o:p></o:p></p><p clas=
s=3DMsoNormal>2016-02-09 11:55:46,765 INFO  [org.ovirt.engine.core.dal=
.dbbroker.auditloghandling.AuditLogDirector] (default task-26) [5118b768] C=
orrelation ID: 5118b768, Call Stack: null, Custom Event ID: -1, Message: Ho=
s<o:p></o:p></p><p class=3DMsoNormal>t cluster Default was updated by admin=
@internal<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:46,932 INFO&n=
bsp; [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]=
 (default task-6) [] START, GlusterServersListVDSCommand(HostName =3D srv-v=
mhost-05, VdsIdVDSCommandParameter<o:p></o:p></p><p class=3DMsoNormal>sBase=
:{runAsync=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'}), log=
 id: 559ab127<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:47,503 IN=
FO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSComm=
and] (default task-13) [] START, GlusterServersListVDSCommand(HostName =3D =
srv-vmhost-05, VdsIdVDSCommandParamete<o:p></o:p></p><p class=3DMsoNormal>r=
sBase:{runAsync=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'})=
, log id: 62d703e5<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:47,5=
10 INFO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVD=
SCommand] (default task-6) [] FINISH, GlusterServersListVDSCommand, log id:=
 559ab127<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:47,511 ERROR =
[org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery] (default ta=
sk-6) [] Query 'GetAddedGlusterServersQuery' failed: null<o:p></o:p></p><p =
class=3DMsoNormal>2016-02-09 11:55:47,511 ERROR [org.ovirt.engine.core.bll.=
gluster.GetAddedGlusterServersQuery] (default task-6) [] Exception: java.la=
ng.NullPointerException<o:p></o:p></p><p class=3DMsoNormal>  &nbs=
p;     at org.ovirt.engine.core.bll.gluster.GetAddedGlu=
sterServersQuery.getAddedGlusterServers(GetAddedGlusterServersQuery.java:54=
) [bll.jar:]<o:p></o:p></p><p class=3DMsoNormal>    &nb=
sp;   at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServers=
Query.executeQueryCommand(GetAddedGlusterServersQuery.java:45) [bll.jar:]<o=
:p></o:p></p><p class=3DMsoNormal>       =
; at org.ovirt.engine.core.bll.QueriesCommandBase.executeCommand(QueriesCom=
mandBase.java:82) [bll.jar:]<o:p></o:p></p><p class=3DMsoNormal>  =
;      at org.ovirt.engine.core.dal.VdcCommandBase=
.execute(VdcCommandBase.java:33) [dal.jar:]<o:p></o:p></p><p class=3DMsoNor=
mal>        at org.ovirt.engine.core.bll=
.Backend.runQueryImpl(Backend.java:537) [bll.jar:]<o:p></o:p></p><p class=
=3DMsoNormal>        at org.ovirt.engine=
.core.bll.Backend.runQuery(Backend.java:511) [bll.jar:]<o:p></o:p></p><p cl=
ass=3DMsoNormal>        at sun.reflect.G=
eneratedMethodAccessor98.invoke(Unknown Source) [:1.8.0_71]<o:p></o:p></p><=
p class=3DMsoNormal>        at sun.refle=
ct.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43=
) [rt.jar:1.8.0_71]<o:p></o:p></p><p class=3DMsoNormal>   &n=
bsp;    at java.lang.reflect.Method.invoke(Method.java:497) =
[rt.jar:1.8.0_71]<o:p></o:p></p><p class=3DMsoNormal>   &nbs=
p;    at org.jboss.as.ee.component.ManagedReferenceMethodInt=
erceptor.processInvocation(ManagedReferenceMethodInterceptor.java:52)<o:p><=
/o:p></p><p class=3DMsoNormal>        at=
 org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:30=
9)<o:p></o:p></p><p class=3DMsoNormal>      &=
nbsp; at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInt=
erceptor.java:53)<o:p></o:p></p><p class=3DMsoNormal>   &nbs=
p;    at org.jboss.as.ee.component.interceptors.UserIntercep=
torFactory$1.processInvocation(UserInterceptorFactory.java:63)<o:p></o:p></=
p><p class=3DMsoNormal>        at org.jb=
oss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)<o:p>=
</o:p></p><p class=3DMsoNormal>        a=
t org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorCon=
text.java:407)<o:p></o:p></p><p class=3DMsoNormal>    &=
nbsp;   at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.delega=
teInterception(Jsr299BindingsInterceptor.java:70) [wildfly-weld-8.2.1.Final=
.jar:8.2.1.Final]<o:p></o:p></p><p class=3DMsoNormal>   &nbs=
p;    at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.doM=
ethodInterception(Jsr299BindingsInterceptor.java:80) [wildfly-weld-8.2.1.Fi=
nal.jar:8.2.1.Final]<o:p></o:p></p><p class=3DMsoNormal>   &=
nbsp;    at org.jboss.as.weld.ejb.Jsr299BindingsInterceptor.=
processInvocation(Jsr299BindingsInterceptor.java:93) [wildfly-weld-8.2.1.Fi=
nal.jar:8.2.1.Final]<o:p></o:p></p><p class=3DMsoNormal>   &=
nbsp;    at org.jboss.as.ee.component.interceptors.UserInter=
ceptorFactory$1.processInvocation(UserInterceptorFactory.java:63)<o:p></o:p=
p;    at org.ovirt.engine.core.bll.interceptors.CorrelationI=
dTrackerInterceptor.aroundInvoke(CorrelationIdTrackerInterceptor.java:13) [=
bll.jar:]<o:p></o:p></p><p class=3DMsoNormal>     =
   at sun.reflect.GeneratedMethodAccessor74.invoke(Unknown Source=
) [:1.8.0_71]<o:p></o:p></p><p class=3DMsoNormal>    &n=
bsp;   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Delegat=
ingMethodAccessorImpl.java:43) [rt.jar:1.8.0_71]<o:p></o:p></p><p class=3DM=
soNormal>           =
            &nb=
sp;       ....<o:p></o:p></p><p class=3DMsoNo=
rmal>2016-02-09 11:55:47,985 INFO  [org.ovirt.engine.core.vdsbroker.gl=
uster.GlusterServersListVDSCommand] (default task-14) [] START, GlusterServ=
ersListVDSCommand(HostName =3D srv-vmhost-05, VdsIdVDSCommandParametersBase=
:{runAsync=3D'true', hostId=3D'13ce38e6-f4b6-42fa-bb8c-5ec84ad00ce0'}), log=
 id: 61100c4d<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:47,986 IN=
FO  [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSComm=
and] (default task-13) [] FINISH, GlusterServersListVDSCommand, log id: 62d=
703e5<o:p></o:p></p><p class=3DMsoNormal>2016-02-09 11:55:47,986 ERROR [org=
.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQuery] (default task-1=
3) [] Query 'GetAddedGlusterServersQuery' failed: null<o:p></o:p></p><p cla=
ss=3DMsoNormal>2016-02-09 11:55:47,987 ERROR [org.ovirt.engine.core.bll.glu=
ster.GetAddedGlusterServersQuery] (default task-13) [] Exception: java.lang=
.NullPointerException<o:p></o:p></p><p class=3DMsoNormal>   =
     at org.ovirt.engine.core.bll.gluster.GetAddedGlust=
erServersQuery.getAddedGlusterServers(GetAddedGlusterServersQuery.java:54) =
[bll.jar:]<o:p></o:p></p><p class=3DMsoNormal>     =
;   at org.ovirt.engine.core.bll.gluster.GetAddedGlusterServersQu=
ery.executeQueryCommand(GetAddedGlusterServersQuery.java:45) [bll.jar:]<o:p=
p></p><p class=3DMsoNormal>        at or=
g.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:309)<=
o:p></o:p></p><p class=3DMsoNormal>      &nbs=
p; at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterc=
eptor.java:53)<o:p></o:p></p><p class=3DMsoNormal>    &=
nbsp;           &nbs=
p;            &=
nbsp; ....<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><o:p> </o:p></p=
p class=3DMsoNormal><span style=3D'font-size:10.0pt;font-family:"Arial",san=
s-serif;mso-fareast-language:IT'><a href=3D"https://www.facebook.com/sistem=
aits"><span style=3D'color:black;text-decoration:none'>facebook.com/sistema=
its</span></a><br><a href=3D"https://www.linkedin.com/company/sistema-soluz=
ioni-per-l-ingegneria-dei-sistemi-di-trasporto-e-l-infomobilit-s-r-l-"><spa=
n style=3D'color:black;text-decoration:none'>linkedin.com/SISTeMA</span></a=
...
<br> </span><o:p></o:p></p></div></body></html>=
--_000_03FD1DFE37AC0E47A173AC5A0835B1C1090AA9FA1ESKAWNE25CEXpt_--

Problem installing oVirt 3.6.3-rc with GlusterFS over InfiniBand

Giuseppe Berellini

Elad Ben Aharon

Giuseppe Berellini

Elad Ben Aharon

Giuseppe Berellini

Elad Ben Aharon

Giuseppe Berellini

Giuseppe Berellini

Giuseppe Berellini

Elad Ben Aharon

Giuseppe Berellini

Elad Ben Aharon

tags

participants (2)