7:53 p.m.
Hi,
I deployed a oVirt (4.3.10) cluster with HostedEngine and GlusterFS volumes (engine,
vmstore, data), the glusterfs cluster on node1/node2/node3, and the engine vm can be
running on those 3 nodes.
Then I added a 4th nodes into cluster.
But, when I operates on Eninge Web Portal, it's always reports 503 error, then I
checked `hsoted-engine --vm-status`, see below:
```
[root@vhost1 ~]# hosted-engine –vm-status
–== Host vhost1.yhmk.lan (id: 1) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost1.<span style=”background-color: rgb(255, 255, 255); color: rgb(51, 51,
51);”>alatest</span>.lan
Host ID : 1
Engine status : {“reason”: “bad vm status”, “health”: “bad”, “vm”: “down_unexpected”,
“detail”: “Down”}
Score : 0
stopped : False
Local maintenance : False
crc32 : 1f25baff
local_conf_timestamp : 1253650
Host timestamp : 1253649
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1253649 (Thu Apr 8 08:05:48 2021)
host-id=1
score=0
vm_conf_refresh_time=1253650 (Thu Apr 8 08:05:48 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Thu Jan 15 20:23:29 1970
–== Host vhost2.yhmk.lan (id: 2) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost2.<span style=”background-color: rgb(255, 255, 255); color: rgb(51, 51,
51);”>alatest</span>.lan
Host ID : 2
Engine status : {“reason”: “vm not running on this host”, “health”: “bad”, “vm”:
“down_unexpected”, “detail”: “unknown”}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 539fc30c
local_conf_timestamp : 1253343
Host timestamp : 1253343
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1253343 (Thu Apr 8 08:05:46 2021)
host-id=2
score=3400
vm_conf_refresh_time=1253343 (Thu Apr 8 08:05:46 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
–== Host vhost3.yhmk.lan (id: 3) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost3.alatest.lan
Host ID : 3
Engine status : {“reason”: “bad vm status”, “health”: “bad”, “vm”: “up”, “detail”:
“Powering up”}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 4072e0b8
local_conf_timestamp : 1252345
Host timestamp : 1252345
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1252345 (Thu Apr 8 08:05:42 2021)
host-id=3
score=3400
vm_conf_refresh_time=1252345 (Thu Apr 8 08:05:42 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
```
Then, wait a moment, can access web portal again, and check the hosts status, alway
reports one or more hosts with label `unavaiable as HA score`, but it will dispear later.
And I found, sometimes the engine vm will migrated to another nodes whill this problem
occur.
So, seems the HostedEngine is not stable, always occur this problem, could you please help
me with this ? Thanks!
12:45 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
On Thu, Apr 8, 2021 at 8:26 AM <mengz.you(a)outlook.com> wrote:
Hi,
I deployed a oVirt (4.3.10) cluster with HostedEngine and GlusterFS volumes (engine,
vmstore, data), the glusterfs cluster on node1/node2/node3, and the engine vm can be
running on those 3 nodes.
Then I added a 4th nodes into cluster.
How exactly did you add it?
What is "cluster"?
- the cluster inside the engine's DB (what you see in the admin UI)
- gluster cluster
- hosted-engine cluster (to do that, you should choose that option
when you add the host)
But, when I operates on Eninge Web Portal, it's always reports 503 error,
What do you mean in "always"? You can't use the web portal anymore at all?
then I checked `hsoted-engine --vm-status`, see below:
```
[root@vhost1 ~]# hosted-engine –vm-status
–== Host vhost1.yhmk.lan (id: 1) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost1.<span style=”background-color: rgb(255, 255, 255); color: rgb(51,
51, 51);”>alatest</span>.lan
Host ID : 1
Engine status : {“reason”: “bad vm status”, “health”: “bad”, “vm”: “down_unexpected”,
“detail”: “Down”}
Score : 0
stopped : False
Local maintenance : False
crc32 : 1f25baff
local_conf_timestamp : 1253650
Host timestamp : 1253649
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1253649 (Thu Apr 8 08:05:48 2021)
host-id=1
score=0
vm_conf_refresh_time=1253650 (Thu Apr 8 08:05:48 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineUnexpectedlyDown
stopped=False
timeout=Thu Jan 15 20:23:29 1970
–== Host vhost2.yhmk.lan (id: 2) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost2.<span style=”background-color: rgb(255, 255, 255); color: rgb(51,
51, 51);”>alatest</span>.lan
Host ID : 2
Engine status : {“reason”: “vm not running on this host”, “health”: “bad”, “vm”:
“down_unexpected”, “detail”: “unknown”}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 539fc30c
local_conf_timestamp : 1253343
Host timestamp : 1253343
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1253343 (Thu Apr 8 08:05:46 2021)
host-id=2
score=3400
vm_conf_refresh_time=1253343 (Thu Apr 8 08:05:46 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineDown
stopped=False
–== Host vhost3.yhmk.lan (id: 3) status ==–
conf_on_shared_storage : True
Status up-to-date : True
Hostname : vhost3.alatest.lan
Host ID : 3
Engine status : {“reason”: “bad vm status”, “health”: “bad”, “vm”: “up”, “detail”:
“Powering up”}
Score : 3400
stopped : False
Local maintenance : False
crc32 : 4072e0b8
local_conf_timestamp : 1252345
Host timestamp : 1252345
Extra metadata (valid at timestamp):
metadata_parse_version=1
metadata_feature_version=1
timestamp=1252345 (Thu Apr 8 08:05:42 2021)
host-id=3
score=3400
vm_conf_refresh_time=1252345 (Thu Apr 8 08:05:42 2021)
conf_on_shared_storage=True
maintenance=False
state=EngineStarting
stopped=False
```
Then, wait a moment, can access web portal again, and check the hosts status, alway
reports one or more hosts with label `unavaiable as HA score`, but it will dispear later.
And I found, sometimes the engine vm will migrated to another nodes whill this problem
occur.
So, seems the HostedEngine is not stable, always occur this problem, could you please
help me with this ? Thanks!
Please check/share /var/log/ovirt-hosted-engine-ha/* on all hosts. Thanks.
Best regards,
--
Didi
6:39 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
Hi, Didi
The "aways" means, when I operates on Admin Portal normally, somtimes the
operation reports a dialog box with 503 error.
And wait some times, it reback to login page, then I can login to do some operations
again.
I do not add the 4th nodes to glusterfs, just add it to oVirt cluster as computer host.
7:30 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
On Thu, Apr 8, 2021 at 2:40 PM <mengz.you(a)outlook.com> wrote:
Hi, Didi
The "aways" means, when I operates on Admin Portal normally, somtimes the
operation reports a dialog box with 503 error.
And wait some times, it reback to login page, then I can login to do some operations
again.
This indeed sounds like the engine or engine vm are/were sometimes
rebooted/restarted, which might be due to misconfiguration somewhere.
I do not add the 4th nodes to glusterfs, just add it to oVirt cluster as computer host.
OK, and did you choose to add it to the hosted-engine cluster?
Did you check the -ha logs?
Best regards,
--
Didi
10:01 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
Hi,
Sorry, what's your meanings for "choose to add it to the hosted-engine
cluster"? I just add the 4th without choose to hosted-engine run (means the
hosted-egnine vm will not mirgrated the 4th host, just in 1, 2, 3 hosts".
Sorry, where the -ha logs? on hosts or engine vm?
And, whether this related to DNS server? seems lots of queries to DNS server
12:15 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
On Thu, Apr 8, 2021 at 6:02 PM <mengz.you(a)outlook.com> wrote:
Hi,
Sorry, what's your meanings for "choose to add it to the hosted-engine
cluster"?
Compute -> Hosts -> New -> "Hosted Engine" -> set "Choose
hosted
engine deployment action" to "Deploy"
I just add the 4th without choose to hosted-engine run (means the
hosted-egnine vm will not mirgrated the 4th host, just in 1, 2, 3 hosts".
If indeed so, it means it's not part of your hosted-engine cluster,
and if so, it means that your problems are likely not related to it.
Sorry, where the -ha logs? on hosts or engine vm?
Hosts
And, whether this related to DNS server? seems lots of queries to DNS server
Might be, yes. Perhaps you misconfigured something? Set the wrong name
in the engine, or hosts, or DNS?
Best regards,
--
Didi
11:16 a.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
Hi, Didi
Another question about glusterfs, I only used host1, host2 and host3 as gluster cluster,
see:
```
[root@vhost2 ~]# gluster pool list
UUID Hostname State
b30d556d-73f9-4d63-8bc6-8ee541684b9c gs3.yhmk.sg Connected
2eb2f9fe-0237-4825-bf89-facc9d99534a gs1.yhmk.sg Connected
c6e85a13-5186-4433-a3c8-0284f826361d localhost Connected
```
But, why I run `gluster pool list` on host4, the result is:
```
[root@vhost4 ~]# gluster pool list
UUID Hostname State
c6e85a13-5186-4433-a3c8-0284f826361d gs2.yhmk.sg Disconnected
2eb2f9fe-0237-4825-bf89-facc9d99534a gs1.yhmk.sg Connected
b30d556d-73f9-4d63-8bc6-8ee541684b9c gs3.yhmk.sg Connected
4259036b-f4db-4d89-bb88-4dcf3c5f882c localhost Connected
[root@vhost4 ~]# gluster volume list
data
engine
vmstore
```
It's strange, why the host4 in the cluster, and can check the volume info?
2:26 p.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
Hi, Didi
And I checked `/var/log/ovirt-engine/engine.log`, it reports lots of WARN about
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn], see below
```
2021-04-12 03:15:58,840+08 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler7) [41356b43] EVENT_ID: GLUSTER_SERVER_STATUS_DISCONNECTED(4,163),
Gluster server vhost4 set to DISCONNECTED on cluster RQ940Cluster.
2021-04-12 03:15:58,841+08 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler7) [41356b43] START, GlusterVolumesListVDSCommand(HostName =
vhost2,
GlusterVolumesListVDSParameters:{hostId='044dc235-e067-4d39-9c4f-e32b1be6f2df'}),
log id: 57eb6144
2021-04-12 03:15:59,120+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs1.yhmk.sg:/gluster_bricks/engine/engine' of volume
'5b9caa26-9e8e-42a5-b39f-99a623eb9f1c' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,125+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs2.yhmk.sg:/gluster_bricks/engine/engine' of volume
'5b9caa26-9e8e-42a5-b39f-99a623eb9f1c' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,128+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs3.yhmk.sg:/gluster_bricks/engine/engine' of volume
'5b9caa26-9e8e-42a5-b39f-99a623eb9f1c' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,131+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs1.yhmk.sg:/gluster_bricks/data/data' of volume
'28f02a31-3a37-4c53-bd10-b85339860984' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,134+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs2.yhmk.sg:/gluster_bricks/data/data' of volume
'28f02a31-3a37-4c53-bd10-b85339860984' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,138+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs3.yhmk.sg:/gluster_bricks/data/data' of volume
'28f02a31-3a37-4c53-bd10-b85339860984' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,141+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs1.yhmk.sg:/gluster_bricks/vmstore/vmstore' of volume
'3b2d9e50-1645-4471-aa79-5778d5cc5d04' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,144+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs2.yhmk.sg:/gluster_bricks/vmstore/vmstore' of volume
'3b2d9e50-1645-4471-aa79-5778d5cc5d04' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
2021-04-12 03:15:59,147+08 WARN
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListReturn]
(DefaultQuartzScheduler7) [41356b43] Could not associate brick
'gs3.yhmk.sg:/gluster_bricks/vmstore/vmstore' of volume
'3b2d9e50-1645-4471-aa79-5778d5cc5d04' with correct network as no gluster network
found in cluster 'ee5c7870-8ca2-11eb-b58f-00163e2d1b80'
```
What's those logs means? And from admin web portal, I see all bricks are up sttatus.
8:16 p.m.
New subject: [Help] The engine is not stable on HostedEngine with GlusterFS Herperverged deployment
It's an effect that also had me puzzled for a long time: To my understanding gluster
volume command should only ever show peers that contribute bricks to a volume, not peers
in general.
Now perhaps an exception needs to be made for hosts that have been enabled to run the
management engine, as that might require Gluster insights.
But I have also seen that with hosts that I only added temporarily to oVirt as a compute
node and I believe I have even seen really ill effects:
Given G1+G2+G3 holding 3R or 2R+1A bricks for the typical oVirt volumes
engine/data/vmstore, I had been adding C1+C2+C3 as mere hosts.
When I then rebooted G3, while C1+C2 were also down, all of a sudden G1+G2 would shut down
their bricks (and refuse to bring them back up) for lack of quota... I had to bring C1+C2
back up to regain quota and then delete C1+C2 from oVirt to take them out of the quorum
for a volume to which they contributed no bricks.
And then often enough, I had to actually detach them as peers via the gluster CLI, because
the GUI didn't finish the job.
Now of course, that only works when G1+G2+G3 are actually all up, too, because otherwise
peers can't be detached....
I've just posted a query on this issue yesterday: Hopefully someone from the
development team will shed some insights into the logic, so we can test better and
potentially open an issue to fix.
1108
days inactive
1120
days old
8 comments
3 participants
participants (3)
-
mengz.you＠outlook.com -
Thomas Hoberg -
Yedidyah Bar David