8:43 p.m.
This is a multi-part message in MIME format.
--------------F560D7B5EE4FCE4640FF5B04
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
Hi,
Thanks a lot for having a look.
HA VMs were migrated, non HA vms were turned off, syslogs were not
saying anything useful, dmesg reported graceful reboot.
What errors are indicating? may be there is a useful hint to proceed in
investigation?
Thanks in advance again
Cheers
Erekle
On 02/23/2018 06:15 PM, Mahdi Adnan wrote:
Hi,
The log does't indicate HV reboot, and i see lots of errors in the logs.
During the reboot, what happened to the VM inside of the HV ? migrated
? paused ? what about the system's logs ? does it indicate a graceful
shutdown ?
--
Respectfully*
**Mahdi A. Mahdi*
------------------------------------------------------------------------
*From:* Erekle Magradze <erekle.magradze(a)recogizer.de>
*Sent:* Friday, February 23, 2018 2:48 PM
*To:* Mahdi Adnan; users(a)ovirt.org
*Subject:* Re: [ovirt-users] rebooting hypervisors from time to time
Thanks for the reply,
I've attached all the logs from yesterday, reboot has happened during
the day but this is not the first time and this is not the only one
hypervisor.
Kind Regards
Erekle
On 02/23/2018 09:00 AM, Mahdi Adnan wrote:
> Hi,
>
> Can you post the VDSM and Engine logs ?
>
>
> --
>
> Respectfully*
> **Mahdi A. Mahdi*
>
> ------------------------------------------------------------------------
> *From:* users-bounces(a)ovirt.org <mailto:users-bounces@ovirt.org>
> <users-bounces(a)ovirt.org> <mailto:users-bounces@ovirt.org> on behalf
> of Erekle Magradze <erekle.magradze(a)recogizer.de>
> <mailto:erekle.magradze@recogizer.de>
> *Sent:* Thursday, February 22, 2018 11:48 PM
> *To:* users(a)ovirt.org <mailto:users@ovirt.org>
> *Subject:* Re: [ovirt-users] rebooting hypervisors from time to time
> Dear all,
>
> It would be great if someone will share any experience regarding the
> similar case, would be great to have a hint where to start investigation.
>
> Thanks again
>
> Cheers
>
> Erekle
>
>
> On 02/22/2018 05:05 PM, Erekle Magradze wrote:
> > Hello there,
> >
> > I am facing the following problem from time to time one of the
> > hypervisor (there are 3 of them)s is rebooting, I am using
> > ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage
> > backend (glusterfs-3.12.5-2.el7.x86_64).
> >
> > I am suspecting gluster because of the e.g. message bellow from one of
> > the volumes,
> >
> > Could you please help and suggest to which direction should
> > investigation go?
> >
> > Thanks in advance
> >
> > Cheers
> >
> > Erekle
> >
> >
> > [2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]
> > [2018-02-22 15:41:10.198701] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:41:10.198704] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:42:11.293608] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:53:16.245720] I [MSGID: 100030]
> > [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running
> > /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs
> > --volfile-server=10.0.0.21 --volfi
> > le-server=10.0.0.22 --volfile-server=10.0.0.23
> > --volfile-id=/virtimages
> > /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)
> > [2018-02-22 15:53:16.263712] W [MSGID: 101002]
> > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family'
> > is deprecated, preferred is 'transport.address-family', continuing
> > with correction
> > [2018-02-22 15:53:16.269595] I [MSGID: 101190]
> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
> > thread with index 1
> > [2018-02-22 15:53:16.273483] I [MSGID: 101190]
> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
> > thread with index 2
> > [2018-02-22 15:53:16.273594] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead:
> > option 'parallel-readdir' is not recognized
> > [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-0: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-1: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
> > 0-virtimages-client-0: changing port to 49152 (from 0)
> > [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-2: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.282126] I [MSGID: 114057]
> > [client-handshake.c:1478:select_server_supported_programs]
> > 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437),
> > Version (330)
> > [2018-02-22 15:53:16.282573] I [MSGID: 114046]
> > [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0:
> > Connected to virtimages-client-0, attached to remote volume
> > '/mnt/virtimages/virtimgs'.
> > [2018-02-22 15:53:16.282584] I [MSGID: 114047]
> > [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0:
> > Server and Client lk-version numbers are not same, reopening the fds
> > [2018-02-22 15:53:16.282665] I [MSGID: 108005]
> > [afr-common.c:4929:__afr_handle_child_up_event]
> > 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back
> > up; going online.
> > [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
> > 0-virtimages-client-1: changing port to 49152 (from 0)
> > [2018-02-22 15:53:16.282934] I [MSGID: 114035]
> > [client-handshake.c:202:client_set_lk_version_cbk]
> > 0-virtimages-client-0: Server lk version = 1
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org <mailto:Users@ovirt.org>
> > http://lists.ovirt.org/mailman/listinfo/users
>
> --
> Recogizer Group GmbH
>
> Dr.rer.nat. Erekle Magradze
> Lead Big Data Engineering & DevOps
> Rheinwerkallee 2, 53227 Bonn
> Tel: +49 228 29974555
>
> E-Mail erekle.magradze(a)recogizer.de <mailto:erekle.magradze@recogizer.de>
> recogizer.com
>
> -----------------------------------------------------------------
>
> Recogizer Group GmbH
> Geschäftsführer: Oliver Habisch, Carsten Kreutze
> Handelsregister: Amtsgericht Bonn HRB 20724
> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
> E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den
> Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie
> die unbefugte Weitergabe dieser Mail und der darin enthaltenen
> Informationen ist nicht gestattet.
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--------------F560D7B5EE4FCE4640FF5B04
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Thanks a lot for having a look.<br>
</p>
<p>HA VMs were migrated, non HA vms were turned off, syslogs were
not saying anything useful, dmesg reported graceful reboot.</p>
<p>What errors are indicating? may be there is a useful hint to
proceed in investigation?</p>
<p>Thanks in advance again</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 02/23/2018 06:15 PM, Mahdi Adnan
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0@DM5PR01MB2506.prod.exchangelabs.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P
{margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
The log does't indicate HV reboot, and i see lots of errors in
the logs.</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
During the reboot, what happened to the VM inside of the HV ?
migrated ? paused ? what about the system's logs ? does it
indicate a graceful shutdown ?</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font
color="#000000">Respectfully<b><br>
</b><b>Mahdi A.
Mahdi</b></font></font><font
color="#3366ff"><br>
<br>
</font></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
face="Calibri, sans-serif"
color="#000000"><b>From:</b> Erekle
Magradze <a class="moz-txt-link-rfc2396E"
href="mailto:erekle.magradze@recogizer.de"><erekle.magradze@recogizer.de></a><br>
<b>Sent:</b> Friday, February 23, 2018 2:48 PM<br>
<b>To:</b> Mahdi Adnan; <a
class="moz-txt-link-abbreviated"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from
time to time</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">
<p>Thanks for the reply,</p>
<p>I've attached all the logs from yesterday, reboot has
happened during the day but this is not the first time and
this is not the only one hypervisor.</p>
<p>Kind Regards</p>
<p>Erekle</p>
<br>
<div class="x_moz-cite-prefix">On 02/23/2018 09:00 AM, Mahdi
Adnan wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
Can you post the VDSM and Engine logs ?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_signature"><br>
<div class="x_ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font
color="#000000">Respectfully<b><br>
</b><b>Mahdi A.
Mahdi</b></font></font><font
color="#3366ff"><br>
<br>
</font></div>
</div>
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b>
<a class="x_moz-txt-link-abbreviated"
href="mailto:users-bounces@ovirt.org"
moz-do-not-send="true">users-bounces(a)ovirt.org</a>
<a class="x_moz-txt-link-rfc2396E"
href="mailto:users-bounces@ovirt.org"
moz-do-not-send="true">&lt;users-bounces(a)ovirt.org&gt;</a>
on behalf of Erekle Magradze
<a class="x_moz-txt-link-rfc2396E"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">&lt;erekle.magradze(a)recogizer.de&gt;</a><br>
<b>Sent:</b> Thursday, February 22, 2018 11:48 PM<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated"
href="mailto:users@ovirt.org"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] rebooting hypervisors
from time to time</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font
size="2"><span
style="font-size:11pt">
<div class="x_PlainText">Dear all,<br>
<br>
It would be great if someone will share any experience
regarding the <br>
similar case, would be great to have a hint where to
start investigation.<br>
<br>
Thanks again<br>
<br>
Cheers<br>
<br>
Erekle<br>
<br>
<br>
On 02/22/2018 05:05 PM, Erekle Magradze wrote:<br>
> Hello there,<br>
><br>
> I am facing the following problem from time to
time one of the <br>
> hypervisor (there are 3 of them)s is rebooting, I
am using <br>
> ovirt-release42-4.2.1-1.el7.centos.noarch and
glsuter as a storage <br>
> backend (glusterfs-3.12.5-2.el7.x86_64).<br>
><br>
> I am suspecting gluster because of the e.g.
message bellow from one of <br>
> the volumes,<br>
><br>
> Could you please help and suggest to which
direction should <br>
> investigation go?<br>
><br>
> Thanks in advance<br>
><br>
> Cheers<br>
><br>
> Erekle<br>
><br>
><br>
> [2018-02-22 15:36:10.011687] and [2018-02-22
15:37:10.955013]<br>
> [2018-02-22 15:41:10.198701] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:41:10.198704] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:42:11.293608] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:53:16.245720] I [MSGID: 100030] <br>
> [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs:
Started running <br>
> /usr/sbin/glusterfs version 3.12.5 (args:
/usr/sbin/glusterfs <br>
> --volfile-server=10.0.0.21 --volfi<br>
> le-server=10.0.0.22 --volfile-server=10.0.0.23 <br>
> --volfile-id=/virtimages <br>
>
/rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)<br>
> [2018-02-22 15:53:16.263712] W [MSGID: 101002] <br>
> [options.c:995:xl_opt_validate] 0-glusterfs:
option 'address-family' <br>
> is deprecated, preferred is
'transport.address-family', continuing <br>
> with correction<br>
> [2018-02-22 15:53:16.269595] I [MSGID: 101190] <br>
> [event-epoll.c:613:event_dispatch_epoll_worker]
0-epoll: Started <br>
> thread with index 1<br>
> [2018-02-22 15:53:16.273483] I [MSGID: 101190] <br>
> [event-epoll.c:613:event_dispatch_epoll_worker]
0-epoll: Started <br>
> thread with index 2<br>
> [2018-02-22 15:53:16.273594] W [MSGID: 101174] <br>
> [graph.c:363:_log_if_unknown_option]
0-virtimages-readdir-ahead: <br>
> option 'parallel-readdir' is not recognized<br>
> [2018-02-22 15:53:16.273703] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-0: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.276455] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-1: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.276683] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] <br>
> 0-virtimages-client-0: changing port to 49152
(from 0)<br>
> [2018-02-22 15:53:16.279191] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-2: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.282126] I [MSGID: 114057] <br>
>
[client-handshake.c:1478:select_server_supported_programs]
<br>
> 0-virtimages-client-0: Using Program GlusterFS
3.3, Num (1298437), <br>
> Version (330)<br>
> [2018-02-22 15:53:16.282573] I [MSGID: 114046] <br>
> [client-handshake.c:1231:client_setvolume_cbk]
0-virtimages-client-0: <br>
> Connected to virtimages-client-0, attached to
remote volume <br>
> '/mnt/virtimages/virtimgs'.<br>
> [2018-02-22 15:53:16.282584] I [MSGID: 114047] <br>
> [client-handshake.c:1242:client_setvolume_cbk]
0-virtimages-client-0: <br>
> Server and Client lk-version numbers are not
same, reopening the fds<br>
> [2018-02-22 15:53:16.282665] I [MSGID: 108005] <br>
> [afr-common.c:4929:__afr_handle_child_up_event] <br>
> 0-virtimages-replicate-0: Subvolume
'virtimages-client-0' came back <br>
> up; going online.<br>
> [2018-02-22 15:53:16.282877] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] <br>
> 0-virtimages-client-1: changing port to 49152
(from 0)<br>
> [2018-02-22 15:53:16.282934] I [MSGID: 114035] <br>
>
[client-handshake.c:202:client_set_lk_version_cbk] <br>
> 0-virtimages-client-0: Server lk version = 1<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a class="x_moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
> <a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
-- <br>
Recogizer Group GmbH<br>
<br>
Dr.rer.nat. Erekle Magradze<br>
Lead Big Data Engineering & DevOps<br>
Rheinwerkallee 2, 53227 Bonn<br>
Tel: +49 228 29974555<br>
<br>
E-Mail <a class="x_moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">
erekle.magradze(a)recogizer.de</a><br>
recogizer.com<br>
<br>
-----------------------------------------------------------------<br>
<br>
Recogizer Group GmbH<br>
Geschäftsführer: Oliver Habisch, Carsten Kreutze<br>
Handelsregister: Amtsgericht Bonn HRB 20724<br>
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993<br>
Diese E-Mail enthält vertrauliche und/oder rechtlich
geschützte Informationen. Wenn Sie nicht der richtige
Adressat sind oder diese E-Mail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und
löschen Sie diese Mail. Das unerlaubte Kopieren sowie
die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a class="x_moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</div>
</span></font></div>
</blockquote>
<br>
</div>
</blockquote>
<br>
</body>
</html>
--------------F560D7B5EE4FCE4640FF5B04--