7:52 a.m.
Hi Alexei,
In order to debug it check the following:
1. Check gluster:1.1 All bricks up ?1.2 All bricks healed (gluster volume heal data info
summary) and no split-brain
2. Go to the problematic host and check the mount point is there2.1. Check permissions
(should be vdsm:kvm) and fix with chown -R if needed2.2. Check the OVF_STORE from the logs
that it exists2.3. Check that vdsm can extract the file:sudo -u vdsm tar -tvf
/rhev/data-center/mnt/glusterSD/msk-gluster-facility.xxxx:_data/DOMAIN-UUID/Volume-UUID/Image-ID
3 Configure virsh alias, as it's quite helpful:alias virsh='virsh -c
qemu:///system?authfile=/etc/ovirt-hosted-engine/virsh_auth.conf'
4. If VM is running - go to the host and get the xml:virsh dumpxml HostedEngine >
/root/HostedEngine.xml4.1. Get the Network:virsh net-dumpxml vdsm-ovirtmgmt >
/root/vdsm-ovirtmgmt.xml4.2 If not , Here is mine:[root@ovirt1 ~]# virsh net-dumpxml
vdsm-ovirtmgmt
<network>
<name>vdsm-ovirtmgmt</name>
<uuid>7ae538ce-d419-4dae-93b8-3a4d27700227</uuid>
<forward mode='bridge'/>
<bridge name='ovirtmgmt'/>
</network>
UUID is not important, as my first recovery was with different one.
5. If you Hosted Engine is down:5.1 Remove the VM (if exists anywhere)on all nodes:virsh
undefine HostedEngine5.2 Verify that the nodes are in global maintenance:hosted-engine
--vm-status5.3 Define the Engine on only 1 machinevirsh define HostedEngine.xmlvirsh
net-define vdsm-ovirtmgmt.xml
virsh start HostedEngine
Note: if it complains about the storage - there is no link in
/var/run/vdsm/storage/DOMAIN-UUID/Volume-UUID to your Volume-UUIDHere is how it looks
mine:[root@ovirt1 808423f9-8a5c-40cd-bc9f-2568c85b8c74]# ll
/var/run/vdsm/storage/808423f9-8a5c-40cd-bc9f-2568c85b8c74
total 24
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 07:42 2c74697a-8bd9-4472-8a98-bf624f3462d5 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/2c74697a-8bd9-4472-8a98-bf624f3462d5
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 07:45 3ec27d6d-921c-4348-b799-f50543b6f919 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/3ec27d6d-921c-4348-b799-f50543b6f919
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 08:28 441abdc8-6cb1-49a4-903f-a1ec0ed88429 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/441abdc8-6cb1-49a4-903f-a1ec0ed88429
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 21:15 8ec7a465-151e-4ac3-92a7-965ecf854501 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/8ec7a465-151e-4ac3-92a7-965ecf854501
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 08:28 94ade632-6ecc-4901-8cec-8e39f3d69cb0 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/94ade632-6ecc-4901-8cec-8e39f3d69cb0
lrwxrwxrwx. 1 vdsm kvm 139 Mar 17 07:42 fe62a281-51e9-4b23-87b3-2deb52357304 ->
/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_engine/808423f9-8a5c-40cd-bc9f-2568c85b8c74/images/fe62a281-51e9-4b23-87b3-2deb52357304
Once you create your link , start it again.
6. Wait till OVF is fixed (takes more than the settings in the engine :) )
Good Luck!
Best Regards,Strahil Nikolov
В понеделник, 18 март 2019 г., 12:57:30 ч. Гринуич+2, Николаев Алексей
<alexeynikolaev.post(a)yandex.ru> написа:
Hi all! I have a very similar problem after update one of the two nodes to version 4.3.1.
This node77-02 lost connection to gluster volume named DATA, but not to volume with hosted
engine. node77-02 /var/log/messages Mar 18 13:40:00 node77-02 journal: ovirt-ha-agent
ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm ERROR Failed scanning
for OVF_STORE due to Command Volume.getInfo with args {'storagepoolID':
'00000000-0000-0000-0000-000000000000', 'storagedomainID':
'2ee71105-1810-46eb-9388-cc6caccf9fac', 'volumeID':
u'224e4b80-2744-4d7f-bd9f-43eb8fe6cf11', 'imageID':
u'43b75b50-cad4-411f-8f51-2e99e52f4c77'} failed:#012(code=201, message=Volume does
not exist: (u'224e4b80-2744-4d7f-bd9f-43eb8fe6cf11',))Mar 18 13:40:00 node77-02
journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config.vm
ERROR Unable to identify the OVF_STORE volume, falling back to initial vm.conf. Please
ensure you already added your first data domain for regular VMs HostedEngine VM works fine
on all nodes. But node77-02 failed witherror in webUI: ConnectStoragePoolVDS failed:
Cannot find master domain: u'spUUID=5a5cca91-01f8-01af-0297-00000000025f,
msdUUID=7d5de684-58ff-4fbc-905d-3048fc55b2b1' node77-02 vdsm.log 2019-03-18
13:51:46,287+0300 WARN (jsonrpc/7) [storage.StorageServer.MountConnection] gluster server
u'msk-gluster-facility.xxxx' is not in bricks ['node-msk-gluster203',
'node-msk-gluster205', 'node-msk-gluster201'], possibly mounting duplicate
servers (storageServer:317)2019-03-18 13:51:46,287+0300 INFO (jsonrpc/7) [storage.Mount]
mounting msk-gluster-facility.ipt.fsin.uis:/data at
/rhev/data-center/mnt/glusterSD/msk-gluster-facility.xxxx:_data (mount:204)2019-03-18
13:51:46,474+0300 ERROR (jsonrpc/7) [storage.HSM] Could not connect to storageServer
(hsm:2415)Traceback (most recent call last): File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 2412, in
connectStorageServer conObj.connect() File
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 179, in
connect six.reraise(t, v, tb) File
"/usr/lib/python2.7/site-packages/vdsm/storage/storageServer.py", line 171, in
connect self._mount.mount(self.options, self._vfsType, cgroup=self.CGROUP) File
"/usr/lib/python2.7/site-packages/vdsm/storage/mount.py", line 207, in mount
cgroup=cgroup) File
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 55, in
__call__ return callMethod() File
"/usr/lib/python2.7/site-packages/vdsm/common/supervdsm.py", line 53, in
<lambda> **kwargs) File "<string>", line 2, in mount File
"/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
raise convert_to_error(kind, result)MountError: (1, ';Running scope as unit
run-10121.scope.\nMount failed. Please check the log file for more
details.\n') ------------------------------ 2019-03-18 13:51:46,830+0300 ERROR
(jsonrpc/4) [storage.TaskManager.Task]
(Task='fe81642e-2421-4169-a08b-51467e8f01fe') Unexpected error (task:875)Traceback
(most recent call last): File
"/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line 882, in _run
return fn(*args, **kargs) File "<string>", line 2, in connectStoragePool
File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 48, in method
ret = func(*args, **kwargs) File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1035, in
connectStoragePool spUUID, hostID, msdUUID, masterVersion, domainsMap) File
"/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 1097, in
_connectStoragePool res = pool.connect(hostID, msdUUID, masterVersion) File
"/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 700, in connect
self.__rebuild(msdUUID=msdUUID, masterVersion=masterVersion) File
"/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1274, in __rebuild
self.setMasterDomain(msdUUID, masterVersion) File
"/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1495, in
setMasterDomain raise se.StoragePoolMasterNotFound(self.spUUID,
msdUUID)StoragePoolMasterNotFound: Cannot find master domain:
u'spUUID=5a5cca91-01f8-01af-0297-00000000025f,
msdUUID=7d5de684-58ff-4fbc-905d-3048fc55b2b1' What the bestpractice to recovery this
problem? 15.03.2019, 13:47, "Strahil Nikolov" <hunter86_bg(a)yahoo.com>:
On Fri, Mar 15, 2019 at 8:12 AM Strahil Nikolov <hunter86_bg(a)yahoo.com> wrote:
Ok, I have managed to recover again and no issues are detected this time.I guess this case
is quite rare and nobody has experienced that.
>Hi,>can you please explain how you fixed it? I have set again to global
maintenance, defined the HostedEngine from the old xml (taken from old vdsm log) , defined
the network and powered it off.Set the OVF update period to 5 min , but it took several
hours until the OVF_STORE were updated. Once this happened I restarted the ovirt-ha-agent
ovirt-ha-broker on both nodes.Then I powered off the HostedEngine and undefined it from
ovirt1. then I set the maintenance to 'none' and the VM powered on ovirt1.In order
to test a failure, I removed the global maintenance and powered off the HostedEngine from
itself (via ssh). It was brought back to the other node. In order to test failure of
ovirt2, I set ovirt1 in local maintenance and removed it (mode 'none') and again
shutdown the VM via ssh and it started again to ovirt1. It seems to be working, as I have
later shut down the Engine several times and it managed to start without issues. I'm
not sure this is related, but I had detected that ovirt2 was out-of-sync of the
vdsm-ovirtmgmt network , but it got fixed easily via the UI. Best Regards,Strahil
Nikolov ,
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3B7OQUA733E...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EMPIGC7JHHW...