5:25 p.m.
I'm abandoning my production ovirt cluster due to instability. I have a 7 host
cluster running about 300 vms and have been for over a year. It has become unstable over
the past three days. I have random hosts both, compute and storage disconnecting. AND
many vms disconnecting and becoming unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts running 3.12.5.
I submitted a bugzilla bug and they immediately assigned it to the storage people but have
not responded with any meaningful information. I have submitted several logs.
I have found some discussion on problems with instability with gluster 3.12.5. I would be
willing to upgrade my gluster to a more stable version if that's the culprit. I
installed gluster using the ovirt gui and this is the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be looking to get a resolution
the problems I'm facing.
10:25 p.m.
Hi,
I would have a look at engine.log, it might provide usefull informations.
Also, i would test i different storage type (maybe a quick nfs data domain
) and see if problem persist with that one too.
On Thu, Feb 14, 2019, 01:26 <dscott(a)umbctraining.com wrote:
I'm abandoning my production ovirt cluster due to instability.
I have a
7 host cluster running about 300 vms and have been for over a year. It has
become unstable over the past three days. I have random hosts both,
compute and storage disconnecting. AND many vms disconnecting and becoming
unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts
running 3.12.5. I submitted a bugzilla bug and they immediately assigned
it to the storage people but have not responded with any meaningful
information. I have submitted several logs.
I have found some discussion on problems with instability with gluster
3.12.5. I would be willing to upgrade my gluster to a more stable version
if that's the culprit. I installed gluster using the ovirt gui and this is
the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be looking to
get a resolution the problems I'm facing.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
12:52 a.m.
I have a three node HCI gluster which was previously running 4.2 with zero
problems. I just upgraded it yesterday. I ran in to a few bugs right away
with the upgrade process, but aside from that I also discovered other users
with severe GlusterFS problems since the upgrade to new GlusterFS version.
It is less than 24 hours since I upgrade my cluster and I just got a notice
that one of my GlusterFS bricks is offline. There does appear to be a very
real and serious issue here with the latest updates.
On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
I'm abandoning my production ovirt cluster due to instability.
I have a
7 host cluster running about 300 vms and have been for over a year. It has
become unstable over the past three days. I have random hosts both,
compute and storage disconnecting. AND many vms disconnecting and becoming
unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts
running 3.12.5. I submitted a bugzilla bug and they immediately assigned
it to the storage people but have not responded with any meaningful
information. I have submitted several logs.
I have found some discussion on problems with instability with gluster
3.12.5. I would be willing to upgrade my gluster to a more stable version
if that's the culprit. I installed gluster using the ovirt gui and this is
the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be looking to
get a resolution the problems I'm facing.
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
5:12 a.m.
I do believe something went wrong after fully updating everything last Friday. I updated
all the ovirt compute nodes on Friday and gluster/engine on Saturday. I have been
experiencing these issues every since. I have pour over engine.log and seems to be
connection to storage issue.
________________________________
From: Jayme <jaymef(a)gmail.com>
Sent: Thursday, February 14, 2019 1:52:59 AM
To: Darryl Scott
Cc: users
Subject: Re: [ovirt-users] Ovirt Cluster completely unstable
I have a three node HCI gluster which was previously running 4.2 with zero problems. I
just upgraded it yesterday. I ran in to a few bugs right away with the upgrade process,
but aside from that I also discovered other users with severe GlusterFS problems since the
upgrade to new GlusterFS version. It is less than 24 hours since I upgrade my cluster and
I just got a notice that one of my GlusterFS bricks is offline. There does appear to be a
very real and serious issue here with the latest updates.
On Wed, Feb 13, 2019 at 7:26 PM
<dscott@umbctraining.com<mailto:dscott@umbctraining.com>> wrote:
I'm abandoning my production ovirt cluster due to instability. I have a 7 host
cluster running about 300 vms and have been for over a year. It has become unstable over
the past three days. I have random hosts both, compute and storage disconnecting. AND
many vms disconnecting and becoming unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts running 3.12.5.
I submitted a bugzilla bug and they immediately assigned it to the storage people but have
not responded with any meaningful information. I have submitted several logs.
I have found some discussion on problems with instability with gluster 3.12.5. I would be
willing to upgrade my gluster to a more stable version if that's the culprit. I
installed gluster using the ovirt gui and this is the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be looking to get a resolution
the problems I'm facing.
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
5:57 a.m.
Hello,
my problems on gluster started with 4.2.6 or 4.2.7. around end of
September. I still have VM's paused the one or other day an they are
reactivated either by HA oder manually. So i want to testify your
experiences. Even while I'm using bonded network connections there are
communication problems without heavy load or other tasks running.
My new Cluster on EPYC Hardware running on NFS 4.2 storage volumes based
on ZFS runs rock solid and VM's are much faster regarding I/O. Gluster
3.12.5 sucks !
Bye
Am 14.02.2019 um 07:52 schrieb Jayme:
I have a three node HCI gluster which was previously running 4.2
with
zero problems. I just upgraded it yesterday. I ran in to a few bugs
right away with the upgrade process, but aside from that I also
discovered other users with severe GlusterFS problems since the
upgrade to new GlusterFS version. It is less than 24 hours since I
upgrade my cluster and I just got a notice that one of my GlusterFS
bricks is offline. There does appear to be a very real and serious
issue here with the latest updates.
On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com
<mailto:dscott@umbctraining.com>> wrote:
I'm abandoning my production ovirt cluster due to instability. I
have a 7 host cluster running about 300 vms and have been for over
a year. It has become unstable over the past three days. I have
random hosts both, compute and storage disconnecting. AND many
vms disconnecting and becoming unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs
hosts running 3.12.5. I submitted a bugzilla bug and they
immediately assigned it to the storage people but have not
responded with any meaningful information. I have submitted
several logs.
I have found some discussion on problems with instability with
gluster 3.12.5. I would be willing to upgrade my gluster to a
more stable version if that's the culprit. I installed gluster
using the ovirt gui and this is the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be
looking to get a resolution the problems I'm facing.
_______________________________________________
Users mailing list -- users(a)ovirt.org <mailto:users@ovirt.org>
To unsubscribe send an email to users-leave(a)ovirt.org
<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
--
*Ralf Schenk*
fon +49 (0) 24 05 / 40 83 70
fax +49 (0) 24 05 / 40 83 759
mail *rs(a)databay.de* <mailto:rs@databay.de>
*Databay AG*
Jens-Otto-Krag-Straße 11
D-52146 Würselen
*www.databay.de* <http://www.databay.de>
Sitz/Amtsgericht Aachen • HRB:8437 • USt-IdNr.: DE 210844202
Vorstand: Ralf Schenk, Dipl.-Ing. Jens Conze, Aresch Yavari, Dipl.-Kfm.
Philipp Hermanns
Aufsichtsratsvorsitzender: Wilhelm Dohmen
------------------------------------------------------------------------
7 a.m.
On Thu, Feb 14, 2019 at 4:56 AM <dscott(a)umbctraining.com> wrote:
I'm abandoning my production ovirt cluster due to instability. I have a 7 host
cluster running about 300 vms and have been for over a year. It has become unstable over
the past three days. I have random hosts both, compute and storage disconnecting. AND
many vms disconnecting and becoming unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts running 3.12.5.
I submitted a bugzilla bug and they immediately assigned it to the storage people but have
not responded with any meaningful information. I have submitted several logs.
Can you point to the bug filed?
+Krutika Dhananjay to look at it
>
> I have found some discussion on problems with instability with gluster 3.12.5. I
would be willing to upgrade my gluster to a more stable version if that's the culprit.
I installed gluster using the ovirt gui and this is the version the ovirt gui installed.
>
> Is there an ovirt health monitor available? Where should I be looking to get a
resolution the problems I'm facing.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
8:16 a.m.
Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jaymef(a)gmail.com> ha
scritto:
I have a three node HCI gluster which was previously running 4.2 with
zero
problems. I just upgraded it yesterday. I ran in to a few bugs right away
with the upgrade process, but aside from that I also discovered other users
with severe GlusterFS problems since the upgrade to new GlusterFS version.
It is less than 24 hours since I upgrade my cluster and I just got a notice
that one of my GlusterFS bricks is offline. There does appear to be a very
real and serious issue here with the latest updates.
tracking the issue on Gluster side on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1677160
If you can help Gluster community providing requested logs it would be
great.
On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
> I'm abandoning my production ovirt cluster due to instability. I have a
> 7 host cluster running about 300 vms and have been for over a year. It has
> become unstable over the past three days. I have random hosts both,
> compute and storage disconnecting. AND many vms disconnecting and becoming
> unusable.
>
> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts
> running 3.12.5. I submitted a bugzilla bug and they immediately assigned
> it to the storage people but have not responded with any meaningful
> information. I have submitted several logs.
>
> I have found some discussion on problems with instability with gluster
> 3.12.5. I would be willing to upgrade my gluster to a more stable version
> if that's the culprit. I installed gluster using the ovirt gui and this is
> the version the ovirt gui installed.
>
> Is there an ovirt health monitor available? Where should I be looking to
> get a resolution the problems I'm facing.
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
>
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://red.ht/sig>
8:28 a.m.
https://bugzilla.redhat.com/show_bug.cgi?id=1677160 doesn't seem relevant
to me? Is that the correct link?
Like I mentioned in a previous email I'm also having problems with Gluster
bricks going offline since upgrading to oVirt 4.3 yesterday (previously
I've never had a single issue with gluster nor have had a brick ever go
down). I suspect this will continue to happen daily as some other users on
this group have suggested. I was able to pull some logs from engine and
gluster from around the time the brick dropped. My setup is 3 node HCI and
I was previously running the latest 4.2 updates (before upgrading to 4.3).
My hardware is has a lot of overhead and I'm on 10Gbe gluster backend (the
servers were certainly not under any significant amount of load when the
brick went offline). To recover I had to place the host in maintenance
mode and reboot (although I suspect I could have simply unmounted and
remounted gluster mounts).
grep "2019-02-14" engine.log-20190214 | grep
"GLUSTER_BRICK_STATUS_CHANGED"
2019-02-14 02:41:48,018-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 03:20:11,189-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine of
cluster Default from DOWN to UP via cli.
2019-02-14 03:20:14,819-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b of
cluster Default from DOWN to UP via cli.
2019-02-14 03:20:19,692-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of
cluster Default from DOWN to UP via cli.
2019-02-14 03:20:25,022-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a of
cluster Default from DOWN to UP via cli.
2019-02-14 03:20:29,088-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:34,099-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of volume
non_prod_a of cluster Default from DOWN to UP via cli
glusterd.log
# grep -B20 -A20 "2019-02-14 02:41" glusterd.log
[2019-02-14 02:36:49.585034] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:36:49.597788] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:36:49.597788] and
[2019-02-14 02:36:49.900505]
[2019-02-14 02:36:53.437539] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.452816] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:53.864153] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.875835] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:30.958649] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:36:35.322129] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:36:39.639645] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:36:45.301275] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:36:53.875835] and
[2019-02-14 02:36:54.180780]
[2019-02-14 02:37:59.193409] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:38:44.065560] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:38:44.072680] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:38:44.077841] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:38:44.082798] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:38:44.088237] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:38:44.093518] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:37:59.193409] and
[2019-02-14 02:38:44.100494]
[2019-02-14 02:41:58.649683] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 6 times between [2019-02-14 02:41:58.649683] and
[2019-02-14 02:43:00.286999]
[2019-02-14 02:43:46.366743] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:43:46.373587] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:43:46.378997] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:43:46.384324] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:43:46.390310] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:43:46.397031] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:43:46.404083] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:45:47.302884] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:45:47.309697] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:45:47.315149] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:45:47.320806] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:45:47.326865] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:45:47.332192] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:45:47.338991] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:46:47.789575] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:46:47.795276] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:46:47.800584] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:46:47.770601] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:46:47.778161] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:46:47.784020] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
engine.log
# grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214
2019-02-14 02:41:43,495-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 172c9ee8
2019-02-14 02:41:43,609-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89],
log id: 172c9ee8
2019-02-14 02:41:43,610-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 3a0e9d63
2019-02-14 02:41:43,703-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14],
log id: 3a0e9d63
2019-02-14 02:41:43,704-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 49966b05
2019-02-14 02:41:44,213-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05
2019-02-14 02:41:44,214-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 30db0ce2
2019-02-14 02:41:44,311-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3],
log id: 30db0ce2
2019-02-14 02:41:44,312-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 6671d0d7
2019-02-14 02:41:44,329-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,345-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,374-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,405-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da],
log id: 6671d0d7
2019-02-14 02:41:44,406-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 6d2bc6d3
2019-02-14 02:41:44,908-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3
2019-02-14 02:41:44,909-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
volumeName='non_prod_b'}), log id: 36ae23c6
2019-02-14 02:41:47,336-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,351-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,379-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,979-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVolumeAdvancedDetailsVDSCommand, return:
org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b,
log id: 36ae23c6
2019-02-14 02:41:48,018-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 02:41:48,046-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b
of volume non_prod_b on cluster Default is down.
2019-02-14 02:41:48,139-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:48,140-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] START,
GlusterServersListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: e1fb23
2019-02-14 02:41:48,911-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand,
return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED,
host2.replaced.domain.com:CONNECTED], log id: e1fb23
2019-02-14 02:41:48,930-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:48,931-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] START,
GlusterVolumesListVDSCommand(HostName = Host0,
GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 68f1aecc
2019-02-14 02:41:49,366-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand,
return:
{6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85,
3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae,
71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb,
752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c,
c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac,
79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392},
log id: 68f1aecc
2019-02-14 02:41:49,489-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 38debe74
2019-02-14 02:41:49,581-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478],
log id: 38debe74
2019-02-14 02:41:49,582-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 7ec02237
2019-02-14 02:41:49,660-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126],
log id: 7ec02237
2019-02-14 02:41:49,661-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 42cdad27
2019-02-14 02:41:50,142-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27
2019-02-14 02:41:50,143-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 12f5fdf2
2019-02-14 02:41:50,248-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127],
log id: 12f5fdf2
2019-02-14 02:41:50,249-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 1256aa5e
2019-02-14 02:41:50,338-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe],
log id: 1256aa5e
2019-02-14 02:41:50,339-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 3dd752e4
2019-02-14 02:41:50,847-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4
2019-02-14 02:41:50,848-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 29a6272c
2019-02-14 02:41:50,954-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c],
log id: 29a6272c
2019-02-14 02:41:50,955-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 501814db
2019-02-14 02:41:51,044-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4],
log id: 501814db
2019-02-14 02:41:51,045-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 7acf4cbf
2019-02-14 02:41:51,546-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf
2019-02-14 02:41:51,547-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
volumeName='non_prod_a'}), log id: 11c42649
On Thu, Feb 14, 2019 at 10:16 AM Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jaymef(a)gmail.com> ha
scritto:
> I have a three node HCI gluster which was previously running 4.2 with
> zero problems. I just upgraded it yesterday. I ran in to a few bugs right
> away with the upgrade process, but aside from that I also discovered other
> users with severe GlusterFS problems since the upgrade to new GlusterFS
> version. It is less than 24 hours since I upgrade my cluster and I just
> got a notice that one of my GlusterFS bricks is offline. There does appear
> to be a very real and serious issue here with the latest updates.
>
tracking the issue on Gluster side on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1677160
If you can help Gluster community providing requested logs it would be
great.
>
>
> On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
>
>> I'm abandoning my production ovirt cluster due to instability. I have
>> a 7 host cluster running about 300 vms and have been for over a year. It
>> has become unstable over the past three days. I have random hosts both,
>> compute and storage disconnecting. AND many vms disconnecting and becoming
>> unusable.
>>
>> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts
>> running 3.12.5. I submitted a bugzilla bug and they immediately assigned
>> it to the storage people but have not responded with any meaningful
>> information. I have submitted several logs.
>>
>> I have found some discussion on problems with instability with gluster
>> 3.12.5. I would be willing to upgrade my gluster to a more stable version
>> if that's the culprit. I installed gluster using the ovirt gui and this is
>> the version the ovirt gui installed.
>>
>> Is there an ovirt health monitor available? Where should I be looking
>> to get a resolution the problems I'm facing.
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
>>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
>
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo(a)redhat.com
<https://red.ht/sig>
9:45 a.m.
Hi Jayme,
btw. in the past there was a long hunting for gluster problems on this list.
as resolution, there was a failed single disk drive on one gluster host.
the drive was direct connected without controller and smart checks,
so no alert was generated, only gluster problems over days.
please check you *physical* existents and online status of your gluster drives.
my two cents
Oliver
9:49 a.m.
Oliver,
Thanks for the input, I do recall reading that thread before, I'm 99.9%
sure it's not the problem here but I will double check, if anything to rule
it out. These bricks are new enterprise SSDs that are less than 3 months
old with almost 0 wear on them and the issues I'm experiencing only started
within hours after upgrading my environment to oVirt 4.3 (and is the same
issue other users are complaining about regarding gluster bricks going
offline after upgrading to 4.3). I think it's fairly clear that there is a
gluster problem in play.
On Thu, Feb 14, 2019 at 11:45 AM Oliver Riesener <
Oliver.Riesener(a)hs-bremen.de> wrote:
Hi Jayme,
btw. in the past there was a long hunting for gluster problems on this
list.
as resolution, there was a failed single disk drive on one gluster host.
the drive was direct connected without controller and smart checks,
so no alert was generated, only gluster problems over days.
please check you *physical* existents and online status of your gluster
drives.
my two cents
Oliver
9:52 a.m.
On Thu, Feb 14, 2019 at 8:24 PM Jayme <jaymef(a)gmail.com> wrote:
https://bugzilla.redhat.com/show_bug.cgi?id=1677160 doesn't seem
relevant
to me? Is that the correct link?
Like I mentioned in a previous email I'm also having problems with Gluster
bricks going offline since upgrading to oVirt 4.3 yesterday (previously
I've never had a single issue with gluster nor have had a brick ever go
down). I suspect this will continue to happen daily as some other users on
this group have suggested. I was able to pull some logs from engine and
gluster from around the time the brick dropped. My setup is 3 node HCI and
I was previously running the latest 4.2 updates (before upgrading to 4.3).
My hardware is has a lot of overhead and I'm on 10Gbe gluster backend (the
servers were certainly not under any significant amount of load when the
brick went offline). To recover I had to place the host in maintenance
mode and reboot (although I suspect I could have simply unmounted and
remounted gluster mounts).
Anything in the brick logs..the below logs only indicate that engine
detected that brick was down. To get to why the brick was marked down, the
bricks logs would help
grep "2019-02-14" engine.log-20190214 | grep
"GLUSTER_BRICK_STATUS_CHANGED"
2019-02-14 02:41:48,018-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 03:20:11,189-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine
of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:14,819-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b
of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:19,692-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of
cluster Default from DOWN to UP via cli.
2019-02-14 03:20:25,022-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a
of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:29,088-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from DOWN to UP via cli.
2019-02-14 03:20:34,099-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler3) [760f7851] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of volume
non_prod_a of cluster Default from DOWN to UP via cli
glusterd.log
# grep -B20 -A20 "2019-02-14 02:41" glusterd.log
[2019-02-14 02:36:49.585034] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:36:49.597788] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:36:49.597788] and
[2019-02-14 02:36:49.900505]
[2019-02-14 02:36:53.437539] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.452816] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:53.864153] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:36:53.875835] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:30.958649] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:36:35.322129] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:36:39.639645] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:36:45.301275] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:36:53.875835] and
[2019-02-14 02:36:54.180780]
[2019-02-14 02:37:59.193409] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:38:44.065560] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:38:44.072680] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:38:44.077841] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:38:44.082798] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:38:44.088237] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:38:44.093518] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2 times between [2019-02-14 02:37:59.193409] and
[2019-02-14 02:38:44.100494]
[2019-02-14 02:41:58.649683] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 6 times between [2019-02-14 02:41:58.649683] and
[2019-02-14 02:43:00.286999]
[2019-02-14 02:43:46.366743] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:43:46.373587] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:43:46.378997] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:43:46.384324] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:43:46.390310] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:43:46.397031] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:43:46.404083] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:45:47.302884] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:45:47.309697] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:45:47.315149] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
[2019-02-14 02:45:47.320806] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:45:47.326865] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:45:47.332192] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:45:47.338991] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:46:47.789575] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_b
[2019-02-14 02:46:47.795276] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_a
[2019-02-14 02:46:47.800584] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume prod_b
[2019-02-14 02:46:47.770601] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume engine
[2019-02-14 02:46:47.778161] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume isos
[2019-02-14 02:46:47.784020] I [MSGID: 106499]
[glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume non_prod_a
engine.log
# grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214
2019-02-14 02:41:43,495-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 172c9ee8
2019-02-14 02:41:43,609-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89],
log id: 172c9ee8
2019-02-14 02:41:43,610-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 3a0e9d63
2019-02-14 02:41:43,703-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14],
log id: 3a0e9d63
2019-02-14 02:41:43,704-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 49966b05
2019-02-14 02:41:44,213-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05
2019-02-14 02:41:44,214-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 30db0ce2
2019-02-14 02:41:44,311-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3],
log id: 30db0ce2
2019-02-14 02:41:44,312-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 6671d0d7
2019-02-14 02:41:44,329-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,345-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,374-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:44,405-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da],
log id: 6671d0d7
2019-02-14 02:41:44,406-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 6d2bc6d3
2019-02-14 02:41:44,908-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3
2019-02-14 02:41:44,909-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
volumeName='non_prod_b'}), log id: 36ae23c6
2019-02-14 02:41:47,336-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,351-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,379-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:47,979-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVolumeAdvancedDetailsVDSCommand, return:
org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b,
log id: 36ae23c6
2019-02-14 02:41:48,018-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of volume
non_prod_b of cluster Default from UP to DOWN via cli.
2019-02-14 02:41:48,046-04 WARN
[org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
(DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b
of volume non_prod_b on cluster Default is down.
2019-02-14 02:41:48,139-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:48,140-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] START,
GlusterServersListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: e1fb23
2019-02-14 02:41:48,911-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand,
return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED,
host2.replaced.domain.com:CONNECTED], log id: e1fb23
2019-02-14 02:41:48,930-04 INFO
[org.ovirt.engine.core.bll.lock.InMemoryLockManager]
(DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
sharedLocks=''}'
2019-02-14 02:41:48,931-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] START,
GlusterVolumesListVDSCommand(HostName = Host0,
GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 68f1aecc
2019-02-14 02:41:49,366-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
(DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand,
return:
{6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85,
3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae,
71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb,
752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c,
c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac,
79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392},
log id: 68f1aecc
2019-02-14 02:41:49,489-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 38debe74
2019-02-14 02:41:49,581-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478],
log id: 38debe74
2019-02-14 02:41:49,582-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 7ec02237
2019-02-14 02:41:49,660-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126],
log id: 7ec02237
2019-02-14 02:41:49,661-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host0,
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
log id: 42cdad27
2019-02-14 02:41:50,142-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27
2019-02-14 02:41:50,143-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 12f5fdf2
2019-02-14 02:41:50,248-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127],
log id: 12f5fdf2
2019-02-14 02:41:50,249-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 1256aa5e
2019-02-14 02:41:50,338-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe],
log id: 1256aa5e
2019-02-14 02:41:50,339-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
log id: 3dd752e4
2019-02-14 02:41:50,847-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4
2019-02-14 02:41:50,848-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 29a6272c
2019-02-14 02:41:50,954-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalLogicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c],
log id: 29a6272c
2019-02-14 02:41:50,955-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 501814db
2019-02-14 02:41:51,044-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterLocalPhysicalVolumeListVDSCommand, return:
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2,
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4],
log id: 501814db
2019-02-14 02:41:51,045-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
log id: 7acf4cbf
2019-02-14 02:41:51,546-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] FINISH,
GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf
2019-02-14 02:41:51,547-04 INFO
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
(DefaultQuartzScheduler1) [5ff5b093] START,
GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
volumeName='non_prod_a'}), log id: 11c42649
On Thu, Feb 14, 2019 at 10:16 AM Sandro Bonazzola <sbonazzo(a)redhat.com>
wrote:
>
>
> Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jaymef(a)gmail.com> ha
> scritto:
>
>> I have a three node HCI gluster which was previously running 4.2 with
>> zero problems. I just upgraded it yesterday. I ran in to a few bugs right
>> away with the upgrade process, but aside from that I also discovered other
>> users with severe GlusterFS problems since the upgrade to new GlusterFS
>> version. It is less than 24 hours since I upgrade my cluster and I just
>> got a notice that one of my GlusterFS bricks is offline. There does appear
>> to be a very real and serious issue here with the latest updates.
>>
>
> tracking the issue on Gluster side on this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1677160
> If you can help Gluster community providing requested logs it would be
> great.
>
>
>
>
>>
>>
>> On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
>>
>>> I'm abandoning my production ovirt cluster due to instability. I have
>>> a 7 host cluster running about 300 vms and have been for over a year. It
>>> has become unstable over the past three days. I have random hosts both,
>>> compute and storage disconnecting. AND many vms disconnecting and becoming
>>> unusable.
>>>
>>> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs
>>> hosts running 3.12.5. I submitted a bugzilla bug and they immediately
>>> assigned it to the storage people but have not responded with any
>>> meaningful information. I have submitted several logs.
>>>
>>> I have found some discussion on problems with instability with gluster
>>> 3.12.5. I would be willing to upgrade my gluster to a more stable version
>>> if that's the culprit. I installed gluster using the ovirt gui and this
is
>>> the version the ovirt gui installed.
>>>
>>> Is there an ovirt health monitor available? Where should I be looking
>>> to get a resolution the problems I'm facing.
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
>>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
>>
>
>
> --
>
> SANDRO BONAZZOLA
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo(a)redhat.com
> <https://red.ht/sig>
>
_______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YV6ERVJ4OY...
10:15 a.m.
[2019-02-14 02:20:29.611099] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:20:29.611131] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:20:29.619521] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:20:29.619867] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 19988 times between [2019-02-14 02:19:31.377315] and
[2019-02-14 02:21:14.033991]
[2019-02-14 02:21:30.303440] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 10 times between [2019-02-14 02:21:30.303440] and
[2019-02-14 02:23:20.421140]
[2019-02-14 02:23:33.142281] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 34 times between [2019-02-14 02:23:33.142281] and
[2019-02-14 02:25:29.115156]
[2019-02-14 02:25:30.326469] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:25:53.973830] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:25:53.973896] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:53.973928] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:25:54.627728] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:54.628149] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.396855] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:25:56.396926] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:56.396957] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:25:56.404566] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.404866] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 25 times between [2019-02-14 02:25:30.326469] and
[2019-02-14 02:27:25.965601]
[2019-02-14 02:28:10.538374] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 38 times between [2019-02-14 02:28:10.538374] and
[2019-02-14 02:29:22.622679]
[2019-02-14 02:29:48.891040] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:29:56.026002] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:31:22.494824] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 02:31:22.494914] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:31:22.494946] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:31:22.503920] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:31:22.504205] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15301 times between [2019-02-14 02:29:48.891040] and
[2019-02-14 02:31:28.963054]
[2019-02-14 02:31:31.016063] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 628 times between [2019-02-14 02:31:31.016063] and
[2019-02-14 02:33:27.936726]
[2019-02-14 02:33:35.878655] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 1776 times between [2019-02-14 02:33:35.878655] and
[2019-02-14 02:35:28.143659]
[2019-02-14 02:35:29.607848] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:00.483996] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:36:00.484066] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:00.484098] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:36:00.779203] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:00.779662] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.892372] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:36:49.892451] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:49.892481] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:36:49.900530] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.900863] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15772 times between [2019-02-14 02:35:29.607848] and
[2019-02-14 02:37:24.862502]
[2019-02-14 02:38:11.704945] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 23 times between [2019-02-14 02:38:11.704945] and
[2019-02-14 02:39:24.469485]
[2019-02-14 02:39:47.550186] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 34 times between [2019-02-14 02:39:47.550186] and
[2019-02-14 02:41:28.642587]
[2019-02-14 02:41:29.746302] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:42:15.811205] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:42:15.811284] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:42:15.811317] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:42:15.819841] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:42:15.820156] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 5147 times between [2019-02-14 02:41:29.746302] and
[2019-02-14 02:43:28.777825]
[2019-02-14 02:43:35.252750] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 17909 times between [2019-02-14 02:43:35.252750] and
[2019-02-14 02:45:29.331082]
[2019-02-14 02:45:29.497786] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:46:08.243167] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:46:08.243230] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:46:08.243259] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:46:09.512844] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:46:09.513302] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2892 times between [2019-02-14 02:45:29.497786] and
[2019-02-14 02:47:23.957871]
[2019-02-14 02:47:30.087716] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:47:42.722682] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 02:47:42.722779] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:47:42.722810] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:47:42.731133] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:47:42.731447] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 564 times between [2019-02-14 02:47:30.087716] and
[2019-02-14 02:49:28.480491]
[2019-02-14 02:49:32.129061] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15058 times between [2019-02-14 02:49:32.129061] and
[2019-02-14 02:51:26.667738]
[2019-02-14 02:52:35.362594] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:53:08.127443] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:53:08.127528] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:53:08.127562] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:53:08.135384] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:53:08.135682] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 11 times between [2019-02-14 02:52:35.362594] and
[2019-02-14 02:53:24.927967]
[2019-02-14 02:53:56.793284] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 38 times between [2019-02-14 02:53:56.793284] and
[2019-02-14 02:55:29.101365]
[2019-02-14 02:55:30.445077] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:56:14.259883] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:56:14.259948] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:56:14.259980] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:56:14.336000] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:56:14.336327] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 19 times between [2019-02-14 02:55:30.445077] and
[2019-02-14 02:57:28.589205]
[2019-02-14 02:57:35.071899] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:58:33.855724] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:58:33.855805] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:58:33.855838] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:58:33.864814] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:58:33.865124] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 35 times between [2019-02-14 02:57:35.071899] and
[2019-02-14 02:59:29.398638]
[2019-02-14 02:59:33.501166] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 03:00:01.622537] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 21104 times between [2019-02-14 02:59:33.501166] and
[2019-02-14 03:01:29.325701]
[2019-02-14 03:01:30.635130] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 640 times between [2019-02-14 03:01:30.635130] and
[2019-02-14 03:03:29.020755]
[2019-02-14 03:03:31.238543] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 03:04:00.722382] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 03:04:00.722466] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 03:04:00.722499] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:ca4e5703-7129-4242-9fbf-9b119de9e0eb-GRAPH_ID:0-PID:35692-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
On Thu, Feb 14, 2019 at 11:52 AM Sahina Bose <sabose(a)redhat.com> wrote:
On Thu, Feb 14, 2019 at 8:24 PM Jayme <jaymef(a)gmail.com> wrote:
> https://bugzilla.redhat.com/show_bug.cgi?id=1677160 doesn't seem
> relevant to me? Is that the correct link?
>
> Like I mentioned in a previous email I'm also having problems with
> Gluster bricks going offline since upgrading to oVirt 4.3 yesterday
> (previously I've never had a single issue with gluster nor have had a brick
> ever go down). I suspect this will continue to happen daily as some other
> users on this group have suggested. I was able to pull some logs from
> engine and gluster from around the time the brick dropped. My setup is 3
> node HCI and I was previously running the latest 4.2 updates (before
> upgrading to 4.3). My hardware is has a lot of overhead and I'm on 10Gbe
> gluster backend (the servers were certainly not under any significant
> amount of load when the brick went offline). To recover I had to place the
> host in maintenance mode and reboot (although I suspect I could have simply
> unmounted and remounted gluster mounts).
>
Anything in the brick logs..the below logs only indicate that engine
detected that brick was down. To get to why the brick was marked down, the
bricks logs would help
> grep "2019-02-14" engine.log-20190214 | grep
> "GLUSTER_BRICK_STATUS_CHANGED"
> 2019-02-14 02:41:48,018-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
> volume non_prod_b of cluster Default from UP to DOWN via cli.
> 2019-02-14 03:20:11,189-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/engine/engine of volume engine
> of cluster Default from DOWN to UP via cli.
> 2019-02-14 03:20:14,819-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume prod_b
> of cluster Default from DOWN to UP via cli.
> 2019-02-14 03:20:19,692-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of
> cluster Default from DOWN to UP via cli.
> 2019-02-14 03:20:25,022-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume prod_a
> of cluster Default from DOWN to UP via cli.
> 2019-02-14 03:20:29,088-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
> volume non_prod_b of cluster Default from DOWN to UP via cli.
> 2019-02-14 03:20:34,099-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of
> volume non_prod_a of cluster Default from DOWN to UP via cli
>
> glusterd.log
>
> # grep -B20 -A20 "2019-02-14 02:41" glusterd.log
> [2019-02-14 02:36:49.585034] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_b
> [2019-02-14 02:36:49.597788] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> The message "E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler" repeated 2 times between [2019-02-14 02:36:49.597788] and
> [2019-02-14 02:36:49.900505]
> [2019-02-14 02:36:53.437539] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
> [2019-02-14 02:36:53.452816] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> [2019-02-14 02:36:53.864153] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
> [2019-02-14 02:36:53.875835] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> [2019-02-14 02:36:30.958649] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume engine
> [2019-02-14 02:36:35.322129] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_b
> [2019-02-14 02:36:39.639645] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume isos
> [2019-02-14 02:36:45.301275] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_a
> The message "E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler" repeated 2 times between [2019-02-14 02:36:53.875835] and
> [2019-02-14 02:36:54.180780]
> [2019-02-14 02:37:59.193409] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> [2019-02-14 02:38:44.065560] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume engine
> [2019-02-14 02:38:44.072680] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume isos
> [2019-02-14 02:38:44.077841] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
> [2019-02-14 02:38:44.082798] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_b
> [2019-02-14 02:38:44.088237] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_a
> [2019-02-14 02:38:44.093518] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_b
> The message "E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler" repeated 2 times between [2019-02-14 02:37:59.193409] and
> [2019-02-14 02:38:44.100494]
> [2019-02-14 02:41:58.649683] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> The message "E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler" repeated 6 times between [2019-02-14 02:41:58.649683] and
> [2019-02-14 02:43:00.286999]
> [2019-02-14 02:43:46.366743] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume engine
> [2019-02-14 02:43:46.373587] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume isos
> [2019-02-14 02:43:46.378997] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
> [2019-02-14 02:43:46.384324] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_b
> [2019-02-14 02:43:46.390310] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_a
> [2019-02-14 02:43:46.397031] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_b
> [2019-02-14 02:43:46.404083] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> [2019-02-14 02:45:47.302884] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume engine
> [2019-02-14 02:45:47.309697] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume isos
> [2019-02-14 02:45:47.315149] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
> [2019-02-14 02:45:47.320806] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_b
> [2019-02-14 02:45:47.326865] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_a
> [2019-02-14 02:45:47.332192] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_b
> [2019-02-14 02:45:47.338991] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
> handler
> [2019-02-14 02:46:47.789575] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_b
> [2019-02-14 02:46:47.795276] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_a
> [2019-02-14 02:46:47.800584] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume prod_b
> [2019-02-14 02:46:47.770601] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume engine
> [2019-02-14 02:46:47.778161] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume isos
> [2019-02-14 02:46:47.784020] I [MSGID: 106499]
> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume non_prod_a
>
> engine.log
>
> # grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214
> 2019-02-14 02:41:43,495-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 172c9ee8
> 2019-02-14 02:41:43,609-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89],
> log id: 172c9ee8
> 2019-02-14 02:41:43,610-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 3a0e9d63
> 2019-02-14 02:41:43,703-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14],
> log id: 3a0e9d63
> 2019-02-14 02:41:43,704-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 49966b05
> 2019-02-14 02:41:44,213-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05
> 2019-02-14 02:41:44,214-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 30db0ce2
> 2019-02-14 02:41:44,311-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3],
> log id: 30db0ce2
> 2019-02-14 02:41:44,312-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 6671d0d7
> 2019-02-14 02:41:44,329-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:44,345-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:44,374-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:44,405-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da],
> log id: 6671d0d7
> 2019-02-14 02:41:44,406-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 6d2bc6d3
> 2019-02-14 02:41:44,908-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3
> 2019-02-14 02:41:44,909-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
>
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
> volumeName='non_prod_b'}), log id: 36ae23c6
> 2019-02-14 02:41:47,336-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:47,351-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:47,379-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:47,979-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVolumeAdvancedDetailsVDSCommand, return:
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b,
> log id: 36ae23c6
> 2019-02-14 02:41:48,018-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
> volume non_prod_b of cluster Default from UP to DOWN via cli.
> 2019-02-14 02:41:48,046-04 WARN
> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
> GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b
> of volume non_prod_b on cluster Default is down.
> 2019-02-14 02:41:48,139-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:48,140-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [7b9bd2d] START,
> GlusterServersListVDSCommand(HostName = Host0,
>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: e1fb23
> 2019-02-14 02:41:48,911-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand,
> return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED,
> host2.replaced.domain.com:CONNECTED], log id: e1fb23
> 2019-02-14 02:41:48,930-04 INFO
> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
> sharedLocks=''}'
> 2019-02-14 02:41:48,931-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler3) [7b9bd2d] START,
> GlusterVolumesListVDSCommand(HostName = Host0,
>
GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 68f1aecc
> 2019-02-14 02:41:49,366-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand,
> return:
>
{6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85,
>
3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae,
>
71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb,
>
752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c,
>
c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac,
>
79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392},
> log id: 68f1aecc
> 2019-02-14 02:41:49,489-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0,
>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 38debe74
> 2019-02-14 02:41:49,581-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478],
> log id: 38debe74
> 2019-02-14 02:41:49,582-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0,
>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 7ec02237
> 2019-02-14 02:41:49,660-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126],
> log id: 7ec02237
> 2019-02-14 02:41:49,661-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVDOVolumeListVDSCommand(HostName = Host0,
>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
> log id: 42cdad27
> 2019-02-14 02:41:50,142-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27
> 2019-02-14 02:41:50,143-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 12f5fdf2
> 2019-02-14 02:41:50,248-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127],
> log id: 12f5fdf2
> 2019-02-14 02:41:50,249-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 1256aa5e
> 2019-02-14 02:41:50,338-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe],
> log id: 1256aa5e
> 2019-02-14 02:41:50,339-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
> log id: 3dd752e4
> 2019-02-14 02:41:50,847-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4
> 2019-02-14 02:41:50,848-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 29a6272c
> 2019-02-14 02:41:50,954-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c],
> log id: 29a6272c
> 2019-02-14 02:41:50,955-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 501814db
> 2019-02-14 02:41:51,044-04 INFO
>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2,
>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4],
> log id: 501814db
> 2019-02-14 02:41:51,045-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
> log id: 7acf4cbf
> 2019-02-14 02:41:51,546-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf
> 2019-02-14 02:41:51,547-04 INFO
> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
> (DefaultQuartzScheduler1) [5ff5b093] START,
> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
>
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
> volumeName='non_prod_a'}), log id: 11c42649
>
> On Thu, Feb 14, 2019 at 10:16 AM Sandro Bonazzola <sbonazzo(a)redhat.com>
> wrote:
>
>>
>>
>> Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jaymef(a)gmail.com> ha
>> scritto:
>>
>>> I have a three node HCI gluster which was previously running 4.2 with
>>> zero problems. I just upgraded it yesterday. I ran in to a few bugs right
>>> away with the upgrade process, but aside from that I also discovered other
>>> users with severe GlusterFS problems since the upgrade to new GlusterFS
>>> version. It is less than 24 hours since I upgrade my cluster and I just
>>> got a notice that one of my GlusterFS bricks is offline. There does appear
>>> to be a very real and serious issue here with the latest updates.
>>>
>>
>> tracking the issue on Gluster side on this bug:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1677160
>> If you can help Gluster community providing requested logs it would be
>> great.
>>
>>
>>
>>
>>>
>>>
>>> On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
>>>
>>>> I'm abandoning my production ovirt cluster due to instability. I
>>>> have a 7 host cluster running about 300 vms and have been for over a
year.
>>>> It has become unstable over the past three days. I have random hosts
both,
>>>> compute and storage disconnecting. AND many vms disconnecting and
becoming
>>>> unusable.
>>>>
>>>> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs
>>>> hosts running 3.12.5. I submitted a bugzilla bug and they immediately
>>>> assigned it to the storage people but have not responded with any
>>>> meaningful information. I have submitted several logs.
>>>>
>>>> I have found some discussion on problems with instability with gluster
>>>> 3.12.5. I would be willing to upgrade my gluster to a more stable
version
>>>> if that's the culprit. I installed gluster using the ovirt gui and
this is
>>>> the version the ovirt gui installed.
>>>>
>>>> Is there an ovirt health monitor available? Where should I be looking
>>>> to get a resolution the problems I'm facing.
>>>> _______________________________________________
>>>> Users mailing list -- users(a)ovirt.org
>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>> oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> List Archives:
>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
>>>>
>>> _______________________________________________
>>> Users mailing list -- users(a)ovirt.org
>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>> oVirt Code of Conduct:
>>> https://www.ovirt.org/community/about/community-guidelines/
>>> List Archives:
>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
>>>
>>
>>
>> --
>>
>> SANDRO BONAZZOLA
>>
>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> sbonazzo(a)redhat.com
>> <https://red.ht/sig>
>>
> _______________________________________________
> Users mailing list -- users(a)ovirt.org
> To unsubscribe send an email to users-leave(a)ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YV6ERVJ4OY...
>
12:29 p.m.
Is it this bug
https://bugzilla.redhat.com/show_bug.cgi?id=1651246
On Thu, Feb 14, 2019 at 11:50 AM Jayme <jaymef(a)gmail.com> wrote:
[2019-02-14 02:20:29.611099] I [login.c:110:gf_auth] 0-auth/login:
allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:20:29.611131] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:20:29.619521] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:20:29.619867] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:ee716e24-e187-4b57-a371-cab544f41162-GRAPH_ID:0-PID:30671-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 19988 times between [2019-02-14 02:19:31.377315] and
[2019-02-14 02:21:14.033991]
[2019-02-14 02:21:30.303440] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 10 times between [2019-02-14 02:21:30.303440] and
[2019-02-14 02:23:20.421140]
[2019-02-14 02:23:33.142281] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 34 times between [2019-02-14 02:23:33.142281] and
[2019-02-14 02:25:29.115156]
[2019-02-14 02:25:30.326469] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:25:53.973830] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:25:53.973896] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:53.973928] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:25:54.627728] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:54.628149] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:4a6b8860-8274-4b3b-b400-d66cbfb97349-GRAPH_ID:0-PID:33522-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.396855] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:25:56.396926] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:25:56.396957] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:25:56.404566] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:25:56.404866] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:963c2196-108c-485d-aca6-a236906d2acf-GRAPH_ID:0-PID:33635-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 25 times between [2019-02-14 02:25:30.326469] and
[2019-02-14 02:27:25.965601]
[2019-02-14 02:28:10.538374] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 38 times between [2019-02-14 02:28:10.538374] and
[2019-02-14 02:29:22.622679]
[2019-02-14 02:29:48.891040] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:29:56.026002] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:31:22.494824] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 02:31:22.494914] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:31:22.494946] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:31:22.503920] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:31:22.504205] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:210b837a-1c98-4d88-aab8-5ce895398d81-GRAPH_ID:0-PID:18765-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15301 times between [2019-02-14 02:29:48.891040] and
[2019-02-14 02:31:28.963054]
[2019-02-14 02:31:31.016063] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 628 times between [2019-02-14 02:31:31.016063] and
[2019-02-14 02:33:27.936726]
[2019-02-14 02:33:35.878655] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 1776 times between [2019-02-14 02:33:35.878655] and
[2019-02-14 02:35:28.143659]
[2019-02-14 02:35:29.607848] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:36:00.483996] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:36:00.484066] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:00.484098] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:36:00.779203] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:00.779662] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:9f2cd861-23c9-412b-be2c-8c45658bb155-GRAPH_ID:0-PID:39820-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.892372] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:36:49.892451] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:36:49.892481] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:36:49.900530] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:36:49.900863] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:91cc19de-30ae-4639-a407-c6fce4682fa4-GRAPH_ID:0-PID:39353-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15772 times between [2019-02-14 02:35:29.607848] and
[2019-02-14 02:37:24.862502]
[2019-02-14 02:38:11.704945] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 23 times between [2019-02-14 02:38:11.704945] and
[2019-02-14 02:39:24.469485]
[2019-02-14 02:39:47.550186] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 34 times between [2019-02-14 02:39:47.550186] and
[2019-02-14 02:41:28.642587]
[2019-02-14 02:41:29.746302] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:42:15.811205] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:42:15.811284] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:42:15.811317] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:42:15.819841] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:42:15.820156] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:6de46e97-ef84-43c8-a472-4e942471a8f6-GRAPH_ID:0-PID:43752-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 5147 times between [2019-02-14 02:41:29.746302] and
[2019-02-14 02:43:28.777825]
[2019-02-14 02:43:35.252750] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 17909 times between [2019-02-14 02:43:35.252750] and
[2019-02-14 02:45:29.331082]
[2019-02-14 02:45:29.497786] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:46:08.243167] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:46:08.243230] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:46:08.243259] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:46:09.512844] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:46:09.513302] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:56fb6f47-3e05-4f07-abdf-cb49213cf5c0-GRAPH_ID:0-PID:44348-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 2892 times between [2019-02-14 02:45:29.497786] and
[2019-02-14 02:47:23.957871]
[2019-02-14 02:47:30.087716] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:47:42.722682] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 02:47:42.722779] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:47:42.722810] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:47:42.731133] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:47:42.731447] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:34bb8b68-3483-4744-bd33-7a47cd4214b2-GRAPH_ID:0-PID:27248-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 564 times between [2019-02-14 02:47:30.087716] and
[2019-02-14 02:49:28.480491]
[2019-02-14 02:49:32.129061] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 15058 times between [2019-02-14 02:49:32.129061] and
[2019-02-14 02:51:26.667738]
[2019-02-14 02:52:35.362594] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:53:08.127443] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:53:08.127528] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:53:08.127562] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:53:08.135384] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:53:08.135682] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:3fbc7931-7ac7-4dfd-9169-6271a18fc0f9-GRAPH_ID:0-PID:48068-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 11 times between [2019-02-14 02:52:35.362594] and
[2019-02-14 02:53:24.927967]
[2019-02-14 02:53:56.793284] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 38 times between [2019-02-14 02:53:56.793284] and
[2019-02-14 02:55:29.101365]
[2019-02-14 02:55:30.445077] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:56:14.259883] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.222"
[2019-02-14 02:56:14.259948] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:56:14.259980] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:56:14.336000] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:56:14.336327] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:d7f3996c-4c68-4e15-b0e2-0197bc358ddd-GRAPH_ID:0-PID:885-HOST:host2.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 19 times between [2019-02-14 02:55:30.445077] and
[2019-02-14 02:57:28.589205]
[2019-02-14 02:57:35.071899] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 02:58:33.855724] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.220"
[2019-02-14 02:58:33.855805] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 02:58:33.855838] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
[2019-02-14 02:58:33.864814] I [MSGID: 115036]
[server.c:469:server_rpc_notify] 0-non_prod_b-server: disconnecting
connection from
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
[2019-02-14 02:58:33.865124] I [MSGID: 101055]
[client_t.c:435:gf_client_unref] 0-non_prod_b-server: Shutting down
connection
CTX_ID:868382a8-7e38-481c-b5a3-80c1bdd21f28-GRAPH_ID:0-PID:4631-HOST:host0.replaced.domain.com-PC_NAME:non_prod_b-client-2-RECON_NO:-0
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 35 times between [2019-02-14 02:57:35.071899] and
[2019-02-14 02:59:29.398638]
[2019-02-14 02:59:33.501166] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 03:00:01.622537] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 21104 times between [2019-02-14 02:59:33.501166] and
[2019-02-14 03:01:29.325701]
[2019-02-14 03:01:30.635130] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
The message "E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler" repeated 640 times between [2019-02-14 03:01:30.635130] and
[2019-02-14 03:03:29.020755]
[2019-02-14 03:03:31.238543] E [MSGID: 101191]
[event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
handler
[2019-02-14 03:04:00.722382] I [addr.c:54:compare_addr_and_update]
0-/gluster_bricks/non_prod_b/non_prod_b: allowed = "*", received addr =
"10.11.0.221"
[2019-02-14 03:04:00.722466] I [login.c:110:gf_auth] 0-auth/login: allowed
user names: 7b741fe4-72ca-41ba-8efb-7add1e4fe6f3
[2019-02-14 03:04:00.722499] I [MSGID: 115029]
[server-handshake.c:537:server_setvolume] 0-non_prod_b-server: accepted
client from
CTX_ID:ca4e5703-7129-4242-9fbf-9b119de9e0eb-GRAPH_ID:0-PID:35692-HOST:host1.replaced.domain.comm-PC_NAME:non_prod_b-client-2-RECON_NO:-0
(version: 5.3)
On Thu, Feb 14, 2019 at 11:52 AM Sahina Bose <sabose(a)redhat.com> wrote:
>
>
> On Thu, Feb 14, 2019 at 8:24 PM Jayme <jaymef(a)gmail.com> wrote:
>
>> https://bugzilla.redhat.com/show_bug.cgi?id=1677160 doesn't seem
>> relevant to me? Is that the correct link?
>>
>> Like I mentioned in a previous email I'm also having problems with
>> Gluster bricks going offline since upgrading to oVirt 4.3 yesterday
>> (previously I've never had a single issue with gluster nor have had a brick
>> ever go down). I suspect this will continue to happen daily as some other
>> users on this group have suggested. I was able to pull some logs from
>> engine and gluster from around the time the brick dropped. My setup is 3
>> node HCI and I was previously running the latest 4.2 updates (before
>> upgrading to 4.3). My hardware is has a lot of overhead and I'm on 10Gbe
>> gluster backend (the servers were certainly not under any significant
>> amount of load when the brick went offline). To recover I had to place the
>> host in maintenance mode and reboot (although I suspect I could have simply
>> unmounted and remounted gluster mounts).
>>
>
> Anything in the brick logs..the below logs only indicate that engine
> detected that brick was down. To get to why the brick was marked down, the
> bricks logs would help
>
>
>> grep "2019-02-14" engine.log-20190214 | grep
>> "GLUSTER_BRICK_STATUS_CHANGED"
>> 2019-02-14 02:41:48,018-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
>> volume non_prod_b of cluster Default from UP to DOWN via cli.
>> 2019-02-14 03:20:11,189-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/engine/engine of volume
>> engine of cluster Default from DOWN to UP via cli.
>> 2019-02-14 03:20:14,819-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/prod_b/prod_b of volume
>> prod_b of cluster Default from DOWN to UP via cli.
>> 2019-02-14 03:20:19,692-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/isos/isos of volume isos of
>> cluster Default from DOWN to UP via cli.
>> 2019-02-14 03:20:25,022-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/prod_a/prod_a of volume
>> prod_a of cluster Default from DOWN to UP via cli.
>> 2019-02-14 03:20:29,088-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
>> volume non_prod_b of cluster Default from DOWN to UP via cli.
>> 2019-02-14 03:20:34,099-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler3) [760f7851] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/non_prod_a/non_prod_a of
>> volume non_prod_a of cluster Default from DOWN to UP via cli
>>
>> glusterd.log
>>
>> # grep -B20 -A20 "2019-02-14 02:41" glusterd.log
>> [2019-02-14 02:36:49.585034] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_b
>> [2019-02-14 02:36:49.597788] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler" repeated 2 times between [2019-02-14 02:36:49.597788] and
>> [2019-02-14 02:36:49.900505]
>> [2019-02-14 02:36:53.437539] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>> [2019-02-14 02:36:53.452816] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> [2019-02-14 02:36:53.864153] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>> [2019-02-14 02:36:53.875835] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> [2019-02-14 02:36:30.958649] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume engine
>> [2019-02-14 02:36:35.322129] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_b
>> [2019-02-14 02:36:39.639645] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume isos
>> [2019-02-14 02:36:45.301275] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_a
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler" repeated 2 times between [2019-02-14 02:36:53.875835] and
>> [2019-02-14 02:36:54.180780]
>> [2019-02-14 02:37:59.193409] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> [2019-02-14 02:38:44.065560] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume engine
>> [2019-02-14 02:38:44.072680] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume isos
>> [2019-02-14 02:38:44.077841] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>> [2019-02-14 02:38:44.082798] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_b
>> [2019-02-14 02:38:44.088237] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_a
>> [2019-02-14 02:38:44.093518] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_b
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler" repeated 2 times between [2019-02-14 02:37:59.193409] and
>> [2019-02-14 02:38:44.100494]
>> [2019-02-14 02:41:58.649683] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> The message "E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler" repeated 6 times between [2019-02-14 02:41:58.649683] and
>> [2019-02-14 02:43:00.286999]
>> [2019-02-14 02:43:46.366743] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume engine
>> [2019-02-14 02:43:46.373587] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume isos
>> [2019-02-14 02:43:46.378997] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>> [2019-02-14 02:43:46.384324] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_b
>> [2019-02-14 02:43:46.390310] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_a
>> [2019-02-14 02:43:46.397031] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_b
>> [2019-02-14 02:43:46.404083] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> [2019-02-14 02:45:47.302884] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume engine
>> [2019-02-14 02:45:47.309697] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume isos
>> [2019-02-14 02:45:47.315149] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>> [2019-02-14 02:45:47.320806] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_b
>> [2019-02-14 02:45:47.326865] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_a
>> [2019-02-14 02:45:47.332192] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_b
>> [2019-02-14 02:45:47.338991] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch
>> handler
>> [2019-02-14 02:46:47.789575] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_b
>> [2019-02-14 02:46:47.795276] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_a
>> [2019-02-14 02:46:47.800584] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume prod_b
>> [2019-02-14 02:46:47.770601] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume engine
>> [2019-02-14 02:46:47.778161] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume isos
>> [2019-02-14 02:46:47.784020] I [MSGID: 106499]
>> [glusterd-handler.c:4389:__glusterd_handle_status_volume] 0-management:
>> Received status volume req for volume non_prod_a
>>
>> engine.log
>>
>> # grep -B20 -A20 "2019-02-14 02:41:48" engine.log-20190214
>> 2019-02-14 02:41:43,495-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 172c9ee8
>> 2019-02-14 02:41:43,609-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@479fcb69,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6443e68f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2b4cf035,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5864f06a,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6119ac8c,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1a9549be,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5614cf81,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@290c9289,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5dd26e8,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@35355754,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@452deeb4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8f8b442,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@647e29d3,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7bee4dff,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@511c4478,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c0bb0bd,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@92e325e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@260731,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33aaacc9,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@72657c59,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@aa10c89],
>> log id: 172c9ee8
>> 2019-02-14 02:41:43,610-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 3a0e9d63
>> 2019-02-14 02:41:43,703-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5ca4a20f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@57a8a76,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7bd1b14],
>> log id: 3a0e9d63
>> 2019-02-14 02:41:43,704-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 49966b05
>> 2019-02-14 02:41:44,213-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 49966b05
>> 2019-02-14 02:41:44,214-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 30db0ce2
>> 2019-02-14 02:41:44,311-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@61a309b5,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@ea9cb2e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@749d57bd,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c49f9d0,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@655eb54d,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@256ee273,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3bd079dc,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@6804900f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@78e0a49f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2acfbc8a,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12e92e96,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5ea1502c,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2398c33b,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7464102e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2f221daa,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7b561852,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1eb29d18,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4a030b80,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@75739027,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3eac8253,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@34fc82c3],
>> log id: 30db0ce2
>> 2019-02-14 02:41:44,312-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 6671d0d7
>> 2019-02-14 02:41:44,329-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:44,345-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:44,374-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:44,405-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@f6a9696,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@558e3332,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@5b449da],
>> log id: 6671d0d7
>> 2019-02-14 02:41:44,406-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 6d2bc6d3
>> 2019-02-14 02:41:44,908-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 6d2bc6d3
>> 2019-02-14 02:41:44,909-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
>>
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
>> volumeName='non_prod_b'}), log id: 36ae23c6
>> 2019-02-14 02:41:47,336-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:47,351-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:47,379-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler3) [7b9bd2d] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:47,979-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVolumeAdvancedDetailsVDSCommand, return:
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeAdvancedDetails@7a4a787b,
>> log id: 36ae23c6
>> 2019-02-14 02:41:48,018-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
>> GLUSTER_BRICK_STATUS_CHANGED(4,086), Detected change in status of brick
>> host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b of
>> volume non_prod_b of cluster Default from UP to DOWN via cli.
>> 2019-02-14 02:41:48,046-04 WARN
>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector]
>> (DefaultQuartzScheduler1) [5ff5b093] EVENT_ID:
>> GLUSTER_BRICK_STATUS_DOWN(4,151), Status of brick
host2.replaced.domain.com:/gluster_bricks/non_prod_b/non_prod_b
>> of volume non_prod_b on cluster Default is down.
>> 2019-02-14 02:41:48,139-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:48,140-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>> (DefaultQuartzScheduler3) [7b9bd2d] START,
>> GlusterServersListVDSCommand(HostName = Host0,
>>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
>> log id: e1fb23
>> 2019-02-14 02:41:48,911-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand]
>> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterServersListVDSCommand,
>> return: [10.12.0.220/24:CONNECTED, host1.replaced.domain.com:CONNECTED,
>> host2.replaced.domain.com:CONNECTED], log id: e1fb23
>> 2019-02-14 02:41:48,930-04 INFO
>> [org.ovirt.engine.core.bll.lock.InMemoryLockManager]
>> (DefaultQuartzScheduler1) [5ff5b093] Failed to acquire lock and wait lock
>>
'EngineLock:{exclusiveLocks='[a45fe964-9989-11e8-b3f7-00163e4bf18a=GLUSTER]',
>> sharedLocks=''}'
>> 2019-02-14 02:41:48,931-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> (DefaultQuartzScheduler3) [7b9bd2d] START,
>> GlusterVolumesListVDSCommand(HostName = Host0,
>>
GlusterVolumesListVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
>> log id: 68f1aecc
>> 2019-02-14 02:41:49,366-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand]
>> (DefaultQuartzScheduler3) [7b9bd2d] FINISH, GlusterVolumesListVDSCommand,
>> return:
>>
{6c05dfc6-4dc0-41e3-a12f-55b4767f1d35=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1952a85,
>>
3f8f6a0f-aed4-48e3-9129-18a2a3f64eef=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@2f6688ae,
>>
71ff56d9-79b8-445d-b637-72ffc974f109=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@730210fb,
>>
752a9438-cd11-426c-b384-bc3c5f86ed07=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@c3be510c,
>>
c3e7447e-8514-4e4a-9ff5-a648fe6aa537=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@450befac,
>>
79e8e93c-57c8-4541-a360-726cec3790cf=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@1926e392},
>> log id: 68f1aecc
>> 2019-02-14 02:41:49,489-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host0,
>>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
>> log id: 38debe74
>> 2019-02-14 02:41:49,581-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e5a7925,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cdf5c9e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443cb62,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@49a3e880,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@443d23c0,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1250bc75,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8d27d86,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5e6363f4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@73ed78db,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@64c9d1c7,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7fecbe95,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3a551e5f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2266926e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@88b380c,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1209279e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3c6466,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@16df63ed,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@47456262,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c2b88c3,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f57c074,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12fa0478],
>> log id: 38debe74
>> 2019-02-14 02:41:49,582-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host0,
>>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
>> log id: 7ec02237
>> 2019-02-14 02:41:49,660-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3eedd0bc,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@7f78e375,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@3d63e126],
>> log id: 7ec02237
>> 2019-02-14 02:41:49,661-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVDOVolumeListVDSCommand(HostName = Host0,
>>
VdsIdVDSCommandParametersBase:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5'}),
>> log id: 42cdad27
>> 2019-02-14 02:41:50,142-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 42cdad27
>> 2019-02-14 02:41:50,143-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 12f5fdf2
>> 2019-02-14 02:41:50,248-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2aaed792,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@8e66930,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@276d599e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1aca2aec,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@46846c60,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7d103269,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@30fc25fc,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7baae445,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1ea8603c,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62578afa,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@33d58089,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1f71d27a,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4205e828,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1c5bbac8,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@395a002,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@12664008,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7f4faec4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@3e03d61f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1038e46d,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@307e8062,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@32453127],
>> log id: 12f5fdf2
>> 2019-02-14 02:41:50,249-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 1256aa5e
>> 2019-02-14 02:41:50,338-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@459a2ff5,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@123cab4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1af41fbe],
>> log id: 1256aa5e
>> 2019-02-14 02:41:50,339-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVDOVolumeListVDSCommand(HostName = Host1,
>>
VdsIdVDSCommandParametersBase:{hostId='fb1e62d5-1dc1-4ccc-8b2b-cf48f7077d0d'}),
>> log id: 3dd752e4
>> 2019-02-14 02:41:50,847-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 3dd752e4
>> 2019-02-14 02:41:50,848-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalLogicalVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 29a6272c
>> 2019-02-14 02:41:50,954-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalLogicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalLogicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@364f3ec6,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@c7cce5e,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@b3bed47,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@13bc244b,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5cca81f4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@36aeba0d,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@62ab384a,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@1047d628,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@188a30f5,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@5bb79f3b,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60e5956f,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@4e3df9cd,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@7796567,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@60d06cf4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@2cd2d36c,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@d80a4aa,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@411eaa20,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@22cac93b,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@18b927bd,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@101465f4,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalLogicalVolume@246f927c],
>> log id: 29a6272c
>> 2019-02-14 02:41:50,955-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterLocalPhysicalVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 501814db
>> 2019-02-14 02:41:51,044-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterLocalPhysicalVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterLocalPhysicalVolumeListVDSCommand, return:
>>
[org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@1cd55aa,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@32c5aba2,
>>
org.ovirt.engine.core.common.businessentities.gluster.GlusterLocalPhysicalVolume@6ae123f4],
>> log id: 501814db
>> 2019-02-14 02:41:51,045-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVDOVolumeListVDSCommand(HostName = Host2,
>>
VdsIdVDSCommandParametersBase:{hostId='fd0752d8-2d41-45b0-887a-0ffacbb8a237'}),
>> log id: 7acf4cbf
>> 2019-02-14 02:41:51,546-04 INFO
>> [org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVDOVolumeListVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] FINISH,
>> GetGlusterVDOVolumeListVDSCommand, return: [], log id: 7acf4cbf
>> 2019-02-14 02:41:51,547-04 INFO
>>
[org.ovirt.engine.core.vdsbroker.gluster.GetGlusterVolumeAdvancedDetailsVDSCommand]
>> (DefaultQuartzScheduler1) [5ff5b093] START,
>> GetGlusterVolumeAdvancedDetailsVDSCommand(HostName = Host0,
>>
GlusterVolumeAdvancedDetailsVDSParameters:{hostId='771c67eb-56e6-4736-8c67-668502d4ecf5',
>> volumeName='non_prod_a'}), log id: 11c42649
>>
>> On Thu, Feb 14, 2019 at 10:16 AM Sandro Bonazzola <sbonazzo(a)redhat.com>
>> wrote:
>>
>>>
>>>
>>> Il giorno gio 14 feb 2019 alle ore 07:54 Jayme <jaymef(a)gmail.com> ha
>>> scritto:
>>>
>>>> I have a three node HCI gluster which was previously running 4.2 with
>>>> zero problems. I just upgraded it yesterday. I ran in to a few bugs
right
>>>> away with the upgrade process, but aside from that I also discovered
other
>>>> users with severe GlusterFS problems since the upgrade to new GlusterFS
>>>> version. It is less than 24 hours since I upgrade my cluster and I just
>>>> got a notice that one of my GlusterFS bricks is offline. There does
appear
>>>> to be a very real and serious issue here with the latest updates.
>>>>
>>>
>>> tracking the issue on Gluster side on this bug:
>>> https://bugzilla.redhat.com/show_bug.cgi?id=1677160
>>> If you can help Gluster community providing requested logs it would be
>>> great.
>>>
>>>
>>>
>>>
>>>>
>>>>
>>>> On Wed, Feb 13, 2019 at 7:26 PM <dscott(a)umbctraining.com> wrote:
>>>>
>>>>> I'm abandoning my production ovirt cluster due to instability.
I
>>>>> have a 7 host cluster running about 300 vms and have been for over a
year.
>>>>> It has become unstable over the past three days. I have random hosts
both,
>>>>> compute and storage disconnecting. AND many vms disconnecting and
becoming
>>>>> unusable.
>>>>>
>>>>> 7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs
>>>>> hosts running 3.12.5. I submitted a bugzilla bug and they
immediately
>>>>> assigned it to the storage people but have not responded with any
>>>>> meaningful information. I have submitted several logs.
>>>>>
>>>>> I have found some discussion on problems with instability with
>>>>> gluster 3.12.5. I would be willing to upgrade my gluster to a more
stable
>>>>> version if that's the culprit. I installed gluster using the
ovirt gui and
>>>>> this is the version the ovirt gui installed.
>>>>>
>>>>> Is there an ovirt health monitor available? Where should I be
>>>>> looking to get a resolution the problems I'm facing.
>>>>> _______________________________________________
>>>>> Users mailing list -- users(a)ovirt.org
>>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>>> oVirt Code of Conduct:
>>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>>> List Archives:
>>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
>>>>>
>>>> _______________________________________________
>>>> Users mailing list -- users(a)ovirt.org
>>>> To unsubscribe send an email to users-leave(a)ovirt.org
>>>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>>>> oVirt Code of Conduct:
>>>> https://www.ovirt.org/community/about/community-guidelines/
>>>> List Archives:
>>>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
>>>>
>>>
>>>
>>> --
>>>
>>> SANDRO BONAZZOLA
>>>
>>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>>
>>> Red Hat EMEA <https://www.redhat.com/>
>>>
>>> sbonazzo(a)redhat.com
>>> <https://red.ht/sig>
>>>
>> _______________________________________________
>> Users mailing list -- users(a)ovirt.org
>> To unsubscribe send an email to users-leave(a)ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>>
https://lists.ovirt.org/archives/list/users@ovirt.org/message/4YV6ERVJ4OY...
>>
> _______________________________________________
Users mailing list -- users(a)ovirt.org
To unsubscribe send an email to users-leave(a)ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/U3GL3SYP6BW...
5:24 a.m.
Sandro
I don't have ovirt-log-collector on my ovirt engine. How can obtain? I see a github
repo to make file, I do not want to be making files on my ovirt-engine, just not yet, I
could possible on weekend.
Where can I obtain the ovirt-log-collector?
________________________________
From: Sandro Bonazzola <sbonazzo(a)redhat.com>
Sent: Thursday, February 14, 2019 9:16:05 AM
To: Jayme
Cc: Darryl Scott; users
Subject: Re: [ovirt-users] Re: Ovirt Cluster completely unstable
Il giorno gio 14 feb 2019 alle ore 07:54 Jayme
<jaymef@gmail.com<mailto:jaymef@gmail.com>> ha scritto:
I have a three node HCI gluster which was previously running 4.2 with zero problems. I
just upgraded it yesterday. I ran in to a few bugs right away with the upgrade process,
but aside from that I also discovered other users with severe GlusterFS problems since the
upgrade to new GlusterFS version. It is less than 24 hours since I upgrade my cluster and
I just got a notice that one of my GlusterFS bricks is offline. There does appear to be a
very real and serious issue here with the latest updates.
tracking the issue on Gluster side on this bug:
https://bugzilla.redhat.com/show_bug.cgi?id=1677160
If you can help Gluster community providing requested logs it would be great.
On Wed, Feb 13, 2019 at 7:26 PM
<dscott@umbctraining.com<mailto:dscott@umbctraining.com>> wrote:
I'm abandoning my production ovirt cluster due to instability. I have a 7 host
cluster running about 300 vms and have been for over a year. It has become unstable over
the past three days. I have random hosts both, compute and storage disconnecting. AND
many vms disconnecting and becoming unusable.
7 host are 4 compute hosts running Ovirt 4.2.8 and three glusterfs hosts running 3.12.5.
I submitted a bugzilla bug and they immediately assigned it to the storage people but have
not responded with any meaningful information. I have submitted several logs.
I have found some discussion on problems with instability with gluster 3.12.5. I would be
willing to upgrade my gluster to a more stable version if that's the culprit. I
installed gluster using the ovirt gui and this is the version the ovirt gui installed.
Is there an ovirt health monitor available? Where should I be looking to get a resolution
the problems I'm facing.
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BL4M3JQA3IE...
_______________________________________________
Users mailing list -- users@ovirt.org<mailto:users@ovirt.org>
To unsubscribe send an email to users-leave@ovirt.org<mailto:users-leave@ovirt.org>
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/QULCBXHTKSC...
--
SANDRO BONAZZOLA
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA<https://www.redhat.com/>
sbonazzo@redhat.com<mailto:sbonazzo@redhat.com>
[https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...
![https://www.redhat.com/files/brand/email/sig-redhat.png]<https://red.h...](https://www.redhat.com/files/brand/email/sig-redhat.png]%3Chttps://red.ht/sig%3E){kind=link}
1904
days inactive
1907
days old
13 comments
9 participants
participants (9)
-
Darryl Scott -
Donny Davis -
dscott＠umbctraining.com
-
Jayme -
Leo David -
Oliver Riesener -
Ralf Schenk -
Sahina Bose -
Sandro Bonazzola