[ovirt-users] Re: ovirt node ng 4.3.0 rc1 and HCI single host problems

On Tue, Jan 15, 2019 at 4:32 PM Gianluca Cecchi <gianluca.cecchi(a)gmail.com&gt; wrote: > The mail was partly scrambled in its contents so I put some clarification > here: > > On Tue, Jan 15, 2019 at 2:38 PM Gianluca Cecchi < > gianluca.cecchi(a)gmail.com&gt; wrote: > >> >>> >> So after starting from scratch and using also the info as detailed on >> thread: >> https://www.mail-archive.com/users@ovirt.org/msg52879.html >> >> the steps now have been: >> >> - install from ovirt-node-ng-installer-4.3.0-2019011010.el7.iso and >> reboot >> >> - connect to cockpit and open terminal >> > > This step is related to ssh daemon > cd /etc/ssh > chmod 600 *key > systemctl restart sshd > > The step below is related to ovirt-imageio-daemon > > >> mkdir /var/run/vdsm >> chmod 755 /var/run/vdsm >> chown vdsm.kvm /var/run/vdsm >> mkdir /var/run/vdsm/dhclientmon >> chmod 755 /var/run/vdsm/dhclientmon/ >> chown vdsm.kvm /var/run/vdsm/dhclientmon/ >> mkdir /var/run/vdsm/trackedInterfaces >> chmod 755 /var/run/vdsm/trackedInterfaces/ >> chown vdsm.kvm /var/run/vdsm/trackedInterfaces/ >> mkdir /var/run/vdsm/v2v >> chmod 700 /var/run/vdsm/v2v >> chown vdsm.kvm /var/run/vdsm/v2v/ >> mkdir /var/run/vdsm/vhostuser >> chmod 755 /var/run/vdsm/vhostuser/ >> chown vdsm.kvm /var/run/vdsm/vhostuser/ >> mkdir /var/run/vdsm/payload >> chmod 755 /var/run/vdsm/payload/ >> chown vdsm.kvm /var/run/vdsm/payload/ >> >> systemctl restart sshd >> > > Actually: > > systemctl restart ovirt-imageio-daemon > > >> >> - put in the newer version of vdsm-api.pickle >> from vdsm-api-4.30.5-2.gitf824ec2.el7.noarch.rpm >> in /usr/lib/python2.7/site-packages/vdsm/rpc/vdsm-api.pickle >> > > download of vdsm-api.pickle can be directly done here eventually: > > https://drive.google.com/file/d/1AhakKhm_dzx-Gxt-Y1OojzRUwHs75kot/view?us... > > >> >> - run the wizard for the gluster+he setup (the right positioned option) >> inside the gdeploy text window click edit and add >> " >> [diskcount] >> 1 >> >> " >> under the section >> " >> [disktype] >> jbod >> " >> >> In my case with single disk I choose JBOD options > > >> - first 2 steps ok >> >> - last step fails in finish part >> >> [ INFO ] TASK [oVirt.hosted-engine-setup : Fetch Datacenter name] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Add NFS storage domain] >> [ INFO ] skipping: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Add glusterfs storage domain] >> [ INFO ] changed: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Add iSCSI storage domain] >> [ INFO ] skipping: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Add Fibre Channel storage >> domain] >> [ INFO ] skipping: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Get storage domain details] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : debug] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Find the appliance OVF] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : debug] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Parse OVF] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Get required size] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : debug] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Remove unsuitable storage >> domain] >> [ INFO ] skipping: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : debug] >> [ INFO ] ok: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Check storage domain free >> space] >> [ INFO ] skipping: [localhost] >> [ INFO ] TASK [oVirt.hosted-engine-setup : Activate storage domain] >> [ ERROR ] Error: Fault reason is "Operation Failed". Fault detail is >> "[]". HTTP response code is 400. >> [ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": >> "Fault reason is \"Operation Failed\". Fault detail is \"[]\". HTTP >> response code is 400."} >> >> On engine.log I see >> >> 2019-01-15 13:50:35,317+01 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] START, CreateStoragePoolVDSCommand(HostName = >> ov4301.localdomain.lo >> cal, >> CreateStoragePoolVDSCommandParameters:{hostId='e8f105f1-37ed-4ac4-bfc3-b1e55ed3027f', >> storagePoolId='96a31a7e-18bb-11e9-9a34-00163e6196f3', >> storagePoolName='Default', masterDomainId='14ec2fc7-8c2 >> b-487c-8f4f-428644650928', >> domainsIdList='[14ec2fc7-8c2b-487c-8f4f-428644650928]', >> masterVersion='1'}), log id: 4baccd53 >> 2019-01-15 13:50:36,345+01 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] Failed in 'CreateStoragePoolVDS' method >> 2019-01-15 13:50:36,354+01 ERROR >> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] >> (default task-2) [51725212] EVENT_ID: VDS_BROKER_COMMAND_FAILURE(10,802), >> VDSM ov4301.localdomain.local command CreateStoragePoolVDS failed: Cannot >> acquire host id: (u'14ec2fc7-8c2b-487c-8f4f-428644650928', >> SanlockException(-203, 'Sanlock lockspace add failure', 'Watchdog device >> error')) >> 2019-01-15 13:50:36,354+01 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] Command >> 'org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand' >> return value 'StatusOnlyReturn [status=Status [code=661, message=Cannot >> acquire host id: (u'14ec2fc7-8c2b-487c-8f4f-428644650928', >> SanlockException(-203, 'Sanlock lockspace add failure', 'Watchdog device >> error'))]]' >> 2019-01-15 13:50:36,354+01 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] HostName = ov4301.localdomain.local >> 2019-01-15 13:50:36,355+01 ERROR >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] Command 'CreateStoragePoolVDSCommand(HostName = >> ov4301.localdomain.local, >> CreateStoragePoolVDSCommandParameters:{hostId='e8f105f1-37ed-4ac4-bfc3-b1e55ed3027f', >> storagePoolId='96a31a7e-18bb-11e9-9a34-00163e6196f3', >> storagePoolName='Default', >> masterDomainId='14ec2fc7-8c2b-487c-8f4f-428644650928', >> domainsIdList='[14ec2fc7-8c2b-487c-8f4f-428644650928]', >> masterVersion='1'})' execution failed: VDSGenericException: >> VDSErrorException: Failed to CreateStoragePoolVDS, error = Cannot acquire >> host id: (u'14ec2fc7-8c2b-487c-8f4f-428644650928', SanlockException(-203, >> 'Sanlock lockspace add failure', 'Watchdog device error')), code = 661 >> 2019-01-15 13:50:36,355+01 INFO >> [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] >> (default task-2) [51725212] FINISH, CreateStoragePoolVDSCommand, return: , >> log id: 4baccd53 >> 2019-01-15 13:50:36,355+01 ERROR >> [org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand] >> (default task-2) [51725212] Command >> 'org.ovirt.engine.core.bll.storage.pool.AddStoragePoolWithStoragesCommand' >> failed: EngineException: >> org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: >> VDSGenericException: VDSErrorException: Failed to CreateStoragePoolVDS, >> error = Cannot acquire host id: (u'14ec2fc7-8c2b-487c-8f4f-428644650928', >> SanlockException(-203, 'Sanlock lockspace add failure', 'Watchdog device >> error')), code = 661 (Failed with error AcquireHostIdFailure and code 661) >> 2019-01-15 13:50:36,379+01 INFO >> [org.ovirt.engine.core.bll.CommandCompensator] (default task-2) [51725212] >> Command [id=c55d9962-368e-4e0c-8fee-bd06e7570062]: Compensating >> DELETED_OR_UPDATED_ENTITY of >> org.ovirt.engine.core.common.businessentities.StoragePool; snapshot: >> id=96a31a7e-18bb-11e9-9a34-00163e6196f3. >> >> On host: >> >> [root@ov4301 log]# cat /etc/hosts >> 192.168.124.50 ov43eng.localdomain.local # temporary entry added by >> hosted-engine-setup for the bootstrap VM >> 127.0.0.1 localhost localhost.localdomain localhost4 >> localhost4.localdomain4 >> ::1 localhost localhost.localdomain localhost6 >> localhost6.localdomain6 >> 192.168.122.210 ov43eng.localdomain.local ov43eng >> 192.168.122.211 ov4301.localdomain.local ov4301 >> [root@ov4301 log]# >> >> [root@ov4301 log]# df -h | grep gluster >> /dev/mapper/gluster_vg_sdb-gluster_lv_engine 64G >> 36M 64G 1% /gluster_bricks/engine >> /dev/mapper/gluster_vg_sdb-gluster_lv_data 30G >> 34M 30G 1% /gluster_bricks/data >> /dev/mapper/gluster_vg_sdb-gluster_lv_vmstore 20G >> 34M 20G 1% /gluster_bricks/vmstore >> 192.168.123.211:/engine 64G >> 691M 64G 2% /rhev/data-center/mnt/glusterSD/192.168.123.211:_engine >> [root@ov4301 log]# >> >> >> and in its messages: >> >> Jan 15 13:35:49 ov4301 dnsmasq-dhcp[22934]: DHCPREQUEST(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 >> Jan 15 13:35:49 ov4301 dnsmasq-dhcp[22934]: DHCPACK(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 ov43eng >> Jan 15 13:35:49 ov4301 dnsmasq-dhcp[22934]: not giving name ov43eng to >> the DHCP lease of 192.168.124.50 because the name exists in /etc/hosts with >> address 192.168.122.210 >> Jan 15 13:40:01 ov4301 systemd: Started Session 38 of user root. >> Jan 15 13:47:12 ov4301 vdsm[29591]: WARN MOM not available. >> Jan 15 13:47:12 ov4301 vdsm[29591]: WARN MOM not available, KSM stats >> will be missing. >> Jan 15 13:49:05 ov4301 python: ansible-setup Invoked with filter=* >> gather_subset=['all'] fact_path=/etc/ansible/facts.d gather_timeout=10 >> Jan 15 13:49:19 ov4301 python: ansible-stat Invoked with >> checksum_algorithm=sha1 get_checksum=True follow=False >> path=/var/tmp/localvmOIXI_W get_md5=None get_mime=True get_attributes=True >> Jan 15 13:49:24 ov4301 python: ansible-ovirt_auth Invoked with >> username=None kerberos=False timeout=0 url=None insecure=True hostname=None >> compress=True state=present headers=None token=None ovirt_auth=None >> ca_file=None password=NOT_LOGGING_PARAMETER >> Jan 15 13:49:29 ov4301 python: ansible-ovirt_host_facts Invoked with >> all_content=False pattern=name=ov4301.localdomain.local fetch_nested=False >> nested_attributes=[] auth={'timeout': 0, 'url': ' >> https://ov43eng.localdomain.local/ovirt-engine/api';, 'insecure': True, >> 'kerberos': False, 'compress': True, 'headers': None, 'token': >> 'Q8qt0Z9DmHJRdg3wk7YxNOAs0JPpBMxxstVx3I8skbulwRWp1SsVXuZYq4DUuPWeEnUZ2bD8TAuwCzJ3qlFYlw', >> 'ca_file': None} >> Jan 15 13:49:35 ov4301 python: ansible-ovirt_cluster_facts Invoked with >> pattern= fetch_nested=False nested_attributes=[] auth={'timeout': 0, 'url': >> 'https://ov43eng.localdomain.local/ovirt-engine/api', 'insecure': True, >> 'kerberos': False, 'compress': True, 'headers': None, 'token': >> 'Q8qt0Z9DmHJRdg3wk7YxNOAs0JPpBMxxstVx3I8skbulwRWp1SsVXuZYq4DUuPWeEnUZ2bD8TAuwCzJ3qlFYlw', >> 'ca_file': None} >> Jan 15 13:49:43 ov4301 python: ansible-ovirt_datacenter_facts Invoked >> with pattern= fetch_nested=False nested_attributes=[] auth={'timeout': 0, >> 'url': 'https://ov43eng.localdomain.local/ovirt-engine/api', >> 'insecure': True, 'kerberos': False, 'compress': True, 'headers': None, >> 'token': >> 'Q8qt0Z9DmHJRdg3wk7YxNOAs0JPpBMxxstVx3I8skbulwRWp1SsVXuZYq4bD8TAuwCzJ3qlFYlw', >> 'ca_file': None} host=ov4301.localdomain.local nested_attributes=[] >> wait=True domain_function=data name=hosted_storage >> critical_space_action_blocker=None posixfs=None poll_interval=3 >> glusterfs=None nfs=None timeout=180 backup=None discard_after_delete=None >> Jan 15 13:50:34 ov4301 systemd: Started Session c24 of user root. >> Jan 15 13:50:34 ov4301 sanlock[26802]: 2019-01-15 13:50:34 11549 >> [21290]: s1 wdmd_connect failed -13 >> Jan 15 13:50:34 ov4301 sanlock[26802]: 2019-01-15 13:50:34 11549 >> [21290]: s1 connect_watchdog failed -1 >> Jan 15 13:50:35 ov4301 sanlock[26802]: 2019-01-15 13:50:35 11550 >> [26810]: s1 add_lockspace fail result -203 >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: DHCPREQUEST(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: DHCPACK(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 ov43eng >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: not giving name ov43eng to >> the DHCP lease of 192.168.124.50 because the name exists in /etc/hosts with >> address 192.168.122.210 >> Jan 15 13:59:34 ov4301 chronyd[26447]: Source 212.45.144.206 replaced >> with 80.211.52.109 >> Jan 15 13:59:44 ov4301 vdsm[29591]: WARN MOM not available. >> Jan 15 13:59:44 ov4301 vdsm[29591]: WARN MOM not available, KSM stats >> will be missing. >> DUuPWeEnUZ2bD8TAuwCzJ3qlFYlw', 'ca_file': None} >> Jan 15 13:49:54 ov4301 python: ansible-ovirt_storage_domain Invoked with >> comment=None warning_low_space=None fetch_nested=False localfs=None >> data_center=Default id=None iscsi=None state=unattached >> wipe_after_delete=None destroy=None fcp=None description=None format=None >> auth={'username':********@internal', 'url': ' >> https://ov43eng.localdomain.local/ovirt-engine/api';, 'insecure': True, >> 'password': 'passw0rd'} host=ov4301.localdomain.local nested_attributes=[] >> wait=True domain_function=data name=hosted_storage >> critical_space_action_blocker=None posixfs=None poll_interval=3 >> glusterfs={'path': '/engine', 'mount_options': None, 'address': >> '192.168.123.211'} nfs=None timeout=180 backup=None >> discard_after_delete=None >> Jan 15 13:50:01 ov4301 systemd: Started Session 39 of user root. >> Jan 15 13:50:01 ov4301 systemd: Created slice vdsm-glusterfs.slice. >> Jan 15 13:50:01 ov4301 systemd: Started /usr/bin/mount -t glusterfs >> 192.168.123.211:/engine /rhev/data-center/mnt/glusterSD/192.168.123.211: >> _engine. >> Jan 15 13:50:01 ov4301 kernel: fuse init (API version 7.22) >> Jan 15 13:50:01 ov4301 systemd: Mounting FUSE Control File System... >> Jan 15 13:50:02 ov4301 systemd: Mounted FUSE Control File System. >> Jan 15 13:50:02 ov4301 systemd: Started Session c20 of user root. >> Jan 15 13:50:02 ov4301 systemd: Started Session c21 of user root. >> Jan 15 13:50:03 ov4301 systemd: Started Session c22 of user root. >> Jan 15 13:50:03 ov4301 systemd: Started Session c23 of user root. >> Jan 15 13:50:12 ov4301 python: ansible-ovirt_storage_domain_facts >> Invoked with pattern=name=hosted_storage fetch_nested=False >> nested_attributes=[] auth={'timeout': 0, 'url': ' >> https://ov43eng.localdomain.local/ovirt-engine/api';, 'insecure': True, >> 'kerberos': False, 'compress': True, 'headers': None, 'token': >> 'Q8qt0Z9DmHJRdg3wk7YxNOAs0JPpBMxxstVx3I8skbulwRWp1SsVXuZYq4DUuPWeEnUZ2bD8TAuwCzJ3qlFYlw', >> 'ca_file': None} >> Jan 15 13:50:18 ov4301 python: ansible-find Invoked with excludes=None >> paths=['/var/tmp/localvmOIXI_W/master'] file_type=file age=None >> contains=None recurse=True age_stamp=mtime patterns=['^.*.(?<!meta).ovf$'] >> depth=None get_checksum=False use_regex=True follow=False hidden=False >> size=None >> Jan 15 13:50:21 ov4301 python: ansible-xml Invoked with >> xpath=/ovf:Envelope/Section/Disk count=False set_children=None >> xmlstring=None strip_cdata_tags=False attribute=size pretty_print=False >> add_children=None value=None content=attribute state=present >> namespaces={'vssd': ' >> http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_VirtualSystemSettin...;, >> 'rasd': ' >> http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_ResourceAllocationS...;, >> 'xsi': 'http://www.w3.org/2001/XMLSchema-instance', 'ovf': ' >> http://schemas.dmtf.org/ovf/envelope/1/'} input_type=yaml >> print_match=False >> path=/var/tmp/localvmOIXI_W/master/vms/c99e3e6b-db14-446f-aaee-48a056d3dd93/c99e3e6b-db14-446f-aaee-48a056d3dd93.ovf >> backup=False >> Jan 15 13:50:30 ov4301 python: ansible-ovirt_storage_domain Invoked with >> comment=None warning_low_space=None fetch_nested=False localfs=None >> data_center=Default id=None iscsi=None state=present wipe_after_delete=None >> destroy=None fcp=None description=None format=None auth={'timeout': 0, >> 'url': 'https://ov43eng.localdomain.local/ovirt-engine/api', >> 'insecure': True, 'kerberos': False, 'compress': True, 'headers': None, >> 'token': >> 'Q8qt0Z9DmHJRdg3wk7YxNOAs0JPpBMxxstVx3I8skbulwRWp1SsVXuZYq4DUuPWeEnUZ2bD8TAuwCzJ3qlFYlw', >> 'ca_file': None} host=ov4301.localdomain.local nested_attributes=[] >> wait=True domain_function=data name=hosted_storage >> critical_space_action_blocker=None posixfs=None poll_interval=3 >> glusterfs=None nfs=None timeout=180 backup=None discard_after_delete=None >> Jan 15 13:50:34 ov4301 systemd: Started Session c24 of user root. >> Jan 15 13:50:34 ov4301 sanlock[26802]: 2019-01-15 13:50:34 11549 >> [21290]: s1 wdmd_connect failed -13 >> Jan 15 13:50:34 ov4301 sanlock[26802]: 2019-01-15 13:50:34 11549 >> [21290]: s1 connect_watchdog failed -1 >> Jan 15 13:50:35 ov4301 sanlock[26802]: 2019-01-15 13:50:35 11550 >> [26810]: s1 add_lockspace fail result -203 >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: DHCPREQUEST(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: DHCPACK(virbr0) >> 192.168.124.50 00:16:3e:61:96:f3 ov43eng >> Jan 15 13:56:48 ov4301 dnsmasq-dhcp[22934]: not giving name ov43eng to >> the DHCP lease of 192.168.124.50 because the name exists in /etc/hosts with >> address 192.168.122.210 >> Jan 15 13:59:34 ov4301 chronyd[26447]: Source 212.45.144.206 replaced >> with 80.211.52.109 >> Jan 15 13:59:44 ov4301 vdsm[29591]: WARN MOM not available. >> Jan 15 13:59:44 ov4301 vdsm[29591]: WARN MOM not available, KSM stats >> will be missing. >> >> In vdsm.log : >> >> 2019-01-15 13:50:34,980+0100 INFO (jsonrpc/2) [storage.StorageDomain] >> sdUUID=14ec2fc7-8c2b-487c-8f4f-428644650928 (fileSD:533) >> 2019-01-15 13:50:34,984+0100 INFO (jsonrpc/2) [storage.StoragePool] >> Creating pool directory >> '/rhev/data-center/96a31a7e-18bb-11e9-9a34-00163e6196f3' (sp:634) >> 2019-01-15 13:50:34,984+0100 INFO (jsonrpc/2) [storage.fileUtils] >> Creating directory: /rhev/data-center/96a31a7e-18bb-11e9-9a34-00163e6196f3 >> mode: None (fileUtils:199) >> 2019-01-15 13:50:34,985+0100 INFO (jsonrpc/2) [storage.SANLock] >> Acquiring host id for domain 14ec2fc7-8c2b-487c-8f4f-428644650928 (id=250, >> async=False) (clusterlock:294) >> 2019-01-15 13:50:35,987+0100 INFO (jsonrpc/2) [vdsm.api] FINISH >> createStoragePool error=Cannot acquire host id: >> (u'14ec2fc7-8c2b-487c-8f4f-428644650928', SanlockException(-203, 'Sanlock >> lockspace add failure', 'Watchdog device error')) >> from=::ffff:192.168.124.50,42356, flow_id=51725212, >> task_id=7cbd7c09-e934-4396-bd9d-61e9f0e00bd3 (api:52) >> 2019-01-15 13:50:35,988+0100 ERROR (jsonrpc/2) >> [storage.TaskManager.Task] (Task='7cbd7c09-e934-4396-bd9d-61e9f0e00bd3') >> Unexpected error (task:875) >> Traceback (most recent call last): >> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line >> 882, in _run >> return fn(*args, **kargs) >> File "<string>", line 2, in createStoragePool >> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50, >> in method >> ret = func(*args, **kwargs) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line >> 1003, in createStoragePool >> leaseParams) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 636, >> in create >> self._acquireTemporaryClusterLock(msdUUID, leaseParams) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 567, >> in _acquireTemporaryClusterLock >> msd.acquireHostId(self.id) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 860, >> in acquireHostId >> self._manifest.acquireHostId(hostId, async) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/sd.py", line 467, >> in acquireHostId >> self._domainLock.acquireHostId(hostId, async) >> File "/usr/lib/python2.7/site-packages/vdsm/storage/clusterlock.py", >> line 325, in acquireHostId >> raise se.AcquireHostIdFailure(self._sdUUID, e) >> AcquireHostIdFailure: Cannot acquire host id: >> (u'14ec2fc7-8c2b-487c-8f4f-428644650928', SanlockException(-203, 'Sanlock >> lockspace add failure', 'Watchdog device error')) >> 2019-01-15 13:50:35,988+0100 INFO (jsonrpc/2) >> [storage.TaskManager.Task] (Task='7cbd7c09-e934-4396-bd9d-61e9f0e00bd3') >> aborting: Task is aborted: "Cannot acquire host id: >> (u'14ec2fc7-8c2b-487c-8f4f-428644650928', SanlockException(-203, 'Sanlock >> lockspace add failure', 'Watchdog device error'))" - code 661 (task:1181) >> 2019-01-15 13:50:35,989+0100 ERROR (jsonrpc/2) [storage.Dispatcher] >> FINISH createStoragePool error=Cannot acquire host id: >> (u'14ec2fc7-8c2b-487c-8f4f-428644650928', SanlockException(-203, 'Sanlock >> lockspace add failure', 'Watchdog device error')) (dispatcher:81) >> 2019-01-15 13:50:35,990+0100 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] >> RPC call StoragePool.create failed (error 661) in 1.01 seconds >> (__init__:312) >> 2019-01-15 13:50:38,109+0100 INFO (vmrecovery) [vdsm.api] START >> getConnectedStoragePoolsList(options=None) from=internal, >> task_id=e69af7e1-c456-4822-ae06-7b309263257d (api:48) >> 2019-01-15 13:50:38,109+0100 INFO (vmrecovery) [vdsm.api] FINISH >> getConnectedStoragePoolsList return={'poollist': []} from=internal, >> task_id=e69af7e1-c456-4822-ae06-7b309263257d (api:54) >> 2019-01-15 13:50:38,110+0100 INFO (vmrecovery) [vds] recovery: waiting >> for storage pool to go up (clientIF:705) >> 2019-01-15 13:50:39,802+0100 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer] >> RPC call GlusterHost.list succeeded in 0.33 seconds (__init__:312) >> 2019-01-15 13:50:39,996+0100 INFO (jsonrpc/7) [jsonrpc.JsonRpcServer] >> RPC call GlusterVolume.list succeeded in 0.18 seconds (__init__:312) >> 2019-01-15 13:50:43,115+0100 INFO (vmrecovery) [vdsm.api] START >> getConnectedStoragePoolsList(options=None) from=internal, >> task_id=729c054e-021a-4d2e-b36c-edfb186a1210 (api:48) >> 2019-01-15 13:50:43,116+0100 INFO (vmrecovery) [vdsm.api] FINISH >> getConnectedStoragePoolsList return={'poollist': []} from=internal, >> task_id=729c054e-021a-4d2e-b36c-edfb186a1210 (api:54) >> 2019-01-15 13:50:43,116+0100 INFO (vmrecovery) [vds] recovery: waiting >> for storage pool to go up (clientIF:705) >> 2019-01-15 13:50:43,611+0100 INFO (jsonrpc/5) [api.host] START >> getStats() from=::ffff:192.168.124.50,42356 (api:48) >> 2019-01-15 13:50:43,628+0100 INFO (jsonrpc/5) [vdsm.api] START >> repoStats(domains=()) from=::ffff:192.168.124.50,42356, >> task_id=868d05f4-5535-4cb3-b283-92d3c1595bb3 (api:48) >> 2019-01-15 13:50:43,628+0100 INFO (jsonrpc/5) [vdsm.api] FINISH >> repoStats return={} from=::ffff:192.168.124.50,42356, >> task_id=868d05f4-5535-4cb3-b283-92d3c1595bb3 (api:54) >> 2019-01-15 13:50:43,628+0100 INFO (jsonrpc/5) [vdsm.api] START >> multipath_health() from=::ffff:192.168.124.50,42356, >> task_id=7f4fdad0-2b2e-4dcf-88b8-b7cf2689d4d9 (api:48) >> >> >> Gianluca >> >> > > regarding sanlock daemon and watchdog multiple daemon, the latter seems > to have no log file but I can only see status of service: > > [root@ov4301 ~]# systemctl status wdmd > ● wdmd.service - Watchdog Multiplexing Daemon > Loaded: loaded (/usr/lib/systemd/system/wdmd.service; disabled; vendor > preset: disabled) > Active: active (running) since Tue 2019-01-15 10:38:13 CET; 4h 32min > ago > Main PID: 3763 (wdmd) > Tasks: 1 > CGroup: /system.slice/wdmd.service > └─3763 /usr/sbin/wdmd > > Jan 15 10:38:12 ov4301.localdomain.local systemd[1]: Starting Watchdog > Multiplexing Daemon... > Jan 15 10:38:13 ov4301.localdomain.local systemd-wdmd[3707]: Loading the > softdog kernel module: ...] > Jan 15 10:38:13 ov4301.localdomain.local wdmd[3759]: group 'sanlock' not > found, using socket gid: 0 > Looks like the host is not configured properly. Running vdsm-tool configure --force should fix this, and must be part of the install process.

> Jan 15 10:38:13 ov4301.localdomain.local wdmd[3763]: wdmd started S0 H1 G0 > Jan 15 10:38:13 ov4301.localdomain.local wdmd[3763]: /dev/watchdog0 armed > with fire_timeout 60 > Jan 15 10:38:13 ov4301.localdomain.local systemd[1]: Started Watchdog > Multiplexing Daemon. > Hint: Some lines were ellipsized, use -l to show in full. > [root@ov4301 ~]# > > while for sanlock: > > [root@ov4301 ~]# cat /var/log/sanlock.log > 2019-01-15 10:38:13 6 [3721]: sanlock daemon started 3.6.0 host > 8fd6d41c-99e8-4c3a-8212-68dd1856927c.ov4301.loc > 2019-01-15 10:38:18 11 [3721]: helper pid 3725 dead wait 0 > 2019-01-15 12:54:56 8211 [26802]: sanlock daemon started 3.6.0 host > 4dc13694-53a6-41dc-93b2-8ce371e903f5.ov4301.loc > 2019-01-15 12:54:57 8211 [26802]: set scheduler RR|RESET_ON_FORK priority > 99 failed: Operation not permitted > 2019-01-15 13:50:34 11549 [26810]: s1 lockspace > 14ec2fc7-8c2b-487c-8f4f-428644650928:250:/rhev/data-center/mnt/glusterSD/192.168.123.211: > _engine/14ec2fc7-8c2b-487c-8f4f-428644650928/dom_md/ids:0 > 2019-01-15 13:50:34 11549 [21290]: s1 wdmd_connect failed -13 > 2019-01-15 13:50:34 11549 [21290]: s1 connect_watchdog failed -1 > 2019-01-15 13:50:35 11550 [26810]: s1 add_lockspace fail result -203 > [root@ov4301 ~]# > > The question is: why these daemons start (with error) upon the node boot > after install? > Or is it a bug that the groups are not in place yet and I can restart in > the inital step (as for ovirt-imageio-daemon) other daemons too? > > > > _______________________________________________ > Users mailing list -- users(a)ovirt.org > To unsubscribe send an email to users-leave(a)ovirt.org > Privacy Statement: https://www.ovirt.org/site/privacy-policy/ > oVirt Code of Conduct: > https://www.ovirt.org/community/about/community-guidelines/ > List Archives: > https://lists.ovirt.org/archives/list/users@ovirt.org/message/ZHLDIPDFQ5P... > _______________________________________________ Users mailing list -- users(a)ovirt.org To unsubscribe send an email to users-leave(a)ovirt.org Privacy Statement: https://www.ovirt.org/site/privacy-policy/ oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/ List Archives: https://lists.ovirt.org/archives/list/users@ovirt.org/message/RDYYVHJ6EZC...