4:36 a.m.
Hi Folks,
I did a minor upgrade on the first host in my cluster and now it is reporting "Non
Operational"
This is what yum showed as updatable. However, I did the update through the ovirt-engine
web interface.
ovirt-node-ng-image-update.noarch
4.4.9-1.el8 ovirt-4.4
Obsoleting Packages
ovirt-node-ng-image-update.noarch
4.4.9-1.el8 ovirt-4.4
ovirt-node-ng-image-update.noarch
4.4.8.3-1.el8 @System
ovirt-node-ng-image-update.noarch
4.4.9-1.el8 ovirt-4.4
ovirt-node-ng-image-update-placeholder.noarch
4.4.8.3-1.el8 @System
How do I start to debug this issue?
Also, it looks like the vmstore brick is not mounting on that host. I only see the engine
mounted.
Broken server:
root(a)ovirt1.dgi log]# mount | grep storage
ovirt1-storage.dgi:/engine on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine
type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
Working server:
[root(a)ovirt2.dgi ~]# mount | grep storage
ovirt1-storage.dgi:/engine on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine
type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
ovirt1-storage.dgi:/vmstore on /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore
type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
I tried putting the server into maintenance mode and running a reinstall on it. No change.
I'de really appreciate some help sorting this our.
Cheers,
Gervais
9:37 a.m.
On Tuesday, 23 November 2021 03:36:07 CET Gervais de Montbrun wrote:
Hi Folks,
I did a minor upgrade on the first host in my cluster and now it is
reporting "Non Operational"
This is what yum showed as updatable. However, I did the update through the
ovirt-engine web interface.
ovirt-node-ng-image-update.noarch
4.4.9-1.el8
ovirt-4.4 Obsoleting Packages
ovirt-node-ng-image-update.noarch
4.4.9-1.el8
ovirt-4.4 ovirt-node-ng-image-update.noarch
4.4.8.3-1.el8
@System ovirt-node-ng-image-update.noarch
4.4.9-1.el8
ovirt-4.4
ovirt-node-ng-image-update-placeholder.noarch
4.4.8.3-1.el8
@System
How do I start to debug this issue?
Check engine log in /var/log/ovirt-engine/engine.log on the machine where
engine runs
Also, it looks like the vmstore brick is not mounting on that host. I only
see the engine mounted.
Could you also attach relevant part of vdsm log (/var/log/vdsm/vdsm.log) from
the machine where mount failed? You should see some mount related error there.
This could be also a reason why hosts become non-operational.
Thanks
Vojta
Broken server:
root(a)ovirt1.dgi log]# mount | grep storage
ovirt1-storage.dgi:/engine on
/rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
131072) Working server:
[root(a)ovirt2.dgi ~]# mount | grep storage
ovirt1-storage.dgi:/engine on
/rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
131072) ovirt1-storage.dgi:/vmstore on
/rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore type
fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
131072)
I tried putting the server into maintenance mode and running a reinstall on
it. No change. I'de really appreciate some help sorting this our.
Cheers,
Gervais
3:42 p.m.
Hi Vojta,
Thanks for the help.
I tried to activate my server this morning and captured the logs from vdsm.log and
engine.log. They are attached.
Something went awry with my gluster (I think) as it is showing that the bricks on the
affected server (ovirt1) are not mounted:
The networking looks fine.
Cheers,
Gervais
On Nov 23, 2021, at 3:37 AM, Vojtech Juranek
<vjuranek(a)redhat.com> wrote:
On Tuesday, 23 November 2021 03:36:07 CET Gervais de Montbrun wrote:
> Hi Folks,
>
> I did a minor upgrade on the first host in my cluster and now it is
> reporting "Non Operational"
>
> This is what yum showed as updatable. However, I did the update through the
> ovirt-engine web interface.
>
> ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4 Obsoleting Packages
> ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4 ovirt-node-ng-image-update.noarch
> 4.4.8.3-1.el8
> @System ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4
> ovirt-node-ng-image-update-placeholder.noarch
> 4.4.8.3-1.el8
> @System
>
> How do I start to debug this issue?
Check engine log in /var/log/ovirt-engine/engine.log on the machine where
engine runs
>
>
> Also, it looks like the vmstore brick is not mounting on that host. I only
> see the engine mounted.
Could you also attach relevant part of vdsm log (/var/log/vdsm/vdsm.log) from
the machine where mount failed? You should see some mount related error there.
This could be also a reason why hosts become non-operational.
Thanks
Vojta
> Broken server:
> root(a)ovirt1.dgi log]# mount | grep storage
> ovirt1-storage.dgi:/engine on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072) Working server:
> [root(a)ovirt2.dgi ~]# mount | grep storage
> ovirt1-storage.dgi:/engine on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072) ovirt1-storage.dgi:/vmstore on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072)
>
>
> I tried putting the server into maintenance mode and running a reinstall on
> it. No change. I'de really appreciate some help sorting this our.
>
> Cheers,
> Gervais
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S6C7R6LUTJX...
6:27 p.m.
On Tuesday, 23 November 2021 14:42:31 CET Gervais de Montbrun wrote:
Hi Vojta,
Thanks for the help.
I tried to activate my server this morning and captured the logs from
vdsm.log and engine.log. They are attached.
Something went awry with my gluster (I think) as it is showing that the
bricks on the affected server (ovirt1) are not mounted:
It seems not to be available, therefore vdsm fails with "OSError: [Errno 116]
Stale file handle" and therefore fails to mount it. I'd suggest to investigate
what's happening with you Gluster storage, eventually try to mount it manually
from affected machine - if you are able to mount it manually, vdsm should be
able to mount it as well.
Given lots of warning in engine log "Could not associate brick 'ovirt1-
storage.dgi:/gluster_bricks/vmstore/vmstore' of volume '2670ff29-8d43-4610-
a437-c6ec2c235753' with correct network as no gluster network found in cluster
'404c8d14-73c1-11eb-8755-00163e5907f6'", I'd probably first take a look
on the
network.
Vojta
6:57 p.m.
Hello Gervais,
is the brick mounted on ovirt1 ? can you mount it using the settings in
/etc/fstab ?
The hostname is not using a FQDN for ovirt1
assuming you have a storage network for the gluster nodes the engine needs to resolve be
able to resolve the host addresses
ovirt1-storage.dgi
ovirt2-storage.dgi
ovirt3-storage.dgi
So that it can assign them to the correct network.
When the volume is showing yellow you can force restart them again from the GUI.
Regards,
Paul S.
________________________________
From: Gervais de Montbrun <gervais(a)demontbrun.com>
Sent: 23 November 2021 13:42
To: Vojtech Juranek <vjuranek(a)redhat.com>
Cc: users(a)ovirt.org <users(a)ovirt.org>
Subject: [ovirt-users] Re: How to debug "Non Operational" host
Caution External Mail: Do not click any links or open any attachments unless you trust the
sender and know that the content is safe.
Hi Vojta,
Thanks for the help.
I tried to activate my server this morning and captured the logs from vdsm.log and
engine.log. They are attached.
Something went awry with my gluster (I think) as it is showing that the bricks on the
affected server (ovirt1) are not mounted:
[cid:2a29d29c-b652-4af8-acf0-1270cb8864bc@eurprd03.prod.outlook.com]
[cid:b0dd6964-58c9-453f-8a6b-fdda6641bde7@eurprd03.prod.outlook.com]
[cid:fb25f398-906f-4a72-9927-b0fdf45e8a23@eurprd03.prod.outlook.com]
The networking looks fine.
Cheers,
Gervais
On Nov 23, 2021, at 3:37 AM, Vojtech Juranek
<vjuranek(a)redhat.com> wrote:
On Tuesday, 23 November 2021 03:36:07 CET Gervais de Montbrun wrote:
> Hi Folks,
>
> I did a minor upgrade on the first host in my cluster and now it is
> reporting "Non Operational"
>
> This is what yum showed as updatable. However, I did the update through the
> ovirt-engine web interface.
>
> ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4 Obsoleting Packages
> ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4 ovirt-node-ng-image-update.noarch
> 4.4.8.3-1.el8
> @System ovirt-node-ng-image-update.noarch
> 4.4.9-1.el8
> ovirt-4.4
> ovirt-node-ng-image-update-placeholder.noarch
> 4.4.8.3-1.el8
> @System
>
> How do I start to debug this issue?
Check engine log in /var/log/ovirt-engine/engine.log on the machine where
engine runs
>
>
> Also, it looks like the vmstore brick is not mounting on that host. I only
> see the engine mounted.
Could you also attach relevant part of vdsm log (/var/log/vdsm/vdsm.log) from
the machine where mount failed? You should see some mount related error there.
This could be also a reason why hosts become non-operational.
Thanks
Vojta
> Broken server:
> root(a)ovirt1.dgi log]# mount | grep storage
> ovirt1-storage.dgi:/engine on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072) Working server:
> [root(a)ovirt2.dgi ~]# mount | grep storage
> ovirt1-storage.dgi:/engine on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_engine type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072) ovirt1-storage.dgi:/vmstore on
> /rhev/data-center/mnt/glusterSD/ovirt1-storage.dgi:_vmstore type
> fuse.glusterfs
> (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=
> 131072)
>
>
> I tried putting the server into maintenance mode and running a reinstall on
> it. No change. I'de really appreciate some help sorting this our.
>
> Cheers,
> Gervais
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement:
https://www.ovirt.org/privacy-policy.html<https://eur02.safelinks.prot...
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/<https://eu...
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S6C7R6LUTJX...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement:
https://www.ovirt.org/privacy-policy.html<https://eur02.safelinks.prot...
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/<https://eu...
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/AWSWTXS6CEA...
To view the terms under which this email is distributed, please go to:-
https://leedsbeckett.ac.uk/disclaimer/email
1091
days inactive
1091
days old
4 comments
3 participants
participants (3)
-
Gervais de Montbrun -
Staniforth, Paul -
Vojtech Juranek