9:15 a.m.
--_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi,
The log does't indicate HV reboot, and i see lots of errors in the logs.
During the reboot, what happened to the VM inside of the HV ? migrated ? pa=
used ? what about the system's logs ? does it indicate a graceful shutdown =
?
--
Respectfully
Mahdi A. Mahdi
________________________________
From: Erekle Magradze <erekle.magradze(a)recogizer.de Sent:
Friday, February 23, 2018 2:48 PM
To: Mahdi Adnan; users(a)ovirt.org
Subject: Re: [ovirt-users] rebooting hypervisors from time to time
Thanks for the reply,
I've attached all the logs from yesterday, reboot has happened during the d=
ay but this is not the first time and this is not the only one hypervisor.
Kind Regards
Erekle
On 02/23/2018 09:00 AM, Mahdi Adnan wrote:
Hi,
Can you post the VDSM and Engine logs ?
--
Respectfully
Mahdi A. Mahdi
________________________________
From: users-bounces@ovirt.org<mailto:users-bounces@ovirt.org> <users-bounce=
s@ovirt.org><mailto:users-bounces@ovirt.org> on behalf of Erekle Magradze <=
erekle.magradze@recogizer.de><mailto:erekle.magradze@recogizer.de Sent: Thursday, February 22, 2018 11:48 PM
To: users@ovirt.org<mailto:users@ovirt.org Subject:
Re: [ovirt-users] rebooting hypervisors from time to time
Dear all,
It would be great if someone will share any experience regarding the
similar case, would be great to have a hint where to start investigation.
Thanks again
Cheers
Erekle
On 02/22/2018 05:05 PM, Erekle Magradze wrote:
recogizer.com
-----------------------------------------------------------------
Recogizer Group GmbH
Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa=
tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC=
mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch=
en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d=
ieser Mail und der darin enthaltenen Informationen ist nicht gestattet.
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail erekle.magradze@recogizer.de<mailto:erekle.magradze@recogizer.de recogizer.com
-----------------------------------------------------------------
Recogizer Group GmbH
Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa=
tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC=
mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch=
en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d=
ieser Mail und der darin enthaltenen Informationen ist nicht gestattet.
--_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html <head <meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=3Diso-8859-=
1" <style type=3D"text/css"
style=3D"display:none;"> P {margin-top:0;margin-bo=
ttom:0;} </style </head <body dir=3D"ltr" <div
style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);" Hi,</div <div
style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);" <br </div <div style=3D"font-family: Calibri, Helvetica,
sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);" The log does't indicate HV reboot, and i see lots of errors
in the logs.</d=
iv <div style=3D"font-family: Calibri, Helvetica,
sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);" During the reboot, what happened to the VM inside of the HV ?
migrated ? pa=
used ? what about the system's logs ? does it indicate a graceful shutdown =
?</div <div style=3D"font-family:
Calibri, Helvetica, sans-serif; font-size: 12pt;=
color: rgb(0, 0, 0);" <br
</div <div
id=3D"signature"><br <div
class=3D"ecxmoz-signature">-- <br
<br <font
color=3D"#3366ff"><font
color=3D"#000000">Respectfully<b><br
</b><b>Mahdi A. Mahdi</b></font></font><font
color=3D"#3366ff"><br
<br </font><font
color=3D"#3366ff"></font></div
</div <hr
style=3D"display:inline-block;width:98%" tabindex=3D"-1" <div id=3D"divRplyFwdMsg"
dir=3D"ltr"><font face=3D"Calibri, sans-serif" st=
yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b>
Erekle Magradze <e=
rekle.magradze(a)recogizer.de&gt;<br
<b>Sent:</b> Friday, February 23, 2018 2:48 PM<br <b>To:</b> Mahdi Adnan; users(a)ovirt.org<br <b>Subject:</b> Re: [ovirt-users] rebooting
hypervisors from time to time</=
font <div> </div
</div <div
style=3D"background-color:#FFFFFF"
<p>Thanks for the reply,</p
<p>I've attached all the logs from yesterday, reboot has happened during
th=
e day but this is not the first time and this is not the only one hyperviso=
r.</p <p>Kind Regards</p <p>Erekle</p
<br <div
class=3D"x_moz-cite-prefix">On 02/23/2018 09:00 AM, Mahdi Adnan wrote:=
<br </div
<blockquote type=3D"cite"><style type=3D"text/css"
style=3D"display:none" <!--
p
{margin-top:0;
margin-bottom:0}
-- </style <div
style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0); background-color:rgba(0,0,0,0)"
Hi,</div <div
style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0); background-color:rgba(0,0,0,0)"
<br </div <div style=3D"font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; col=
or:rgb(0,0,0); background-color:rgba(0,0,0,0)" Can you
post the VDSM and Engine logs ?</div <div
style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col=
or:rgb(0,0,0)" <br
</div <div
id=3D"x_signature"><br <div
class=3D"x_ecxmoz-signature">-- <br
<br <font
color=3D"#3366ff"><font
color=3D"#000000">Respectfully<b><br
</b><b>Mahdi A. Mahdi</b></font></font><font
color=3D"#3366ff"><br
<br </font></div </div <hr
tabindex=3D"-1" style=3D"display:inline-block; width:98%" <div id=3D"x_divRplyFwdMsg"
dir=3D"ltr"><font face=3D"Calibri, sans-serif" =
color=3D"#000000" style=3D"font-size:11pt"><b>From:</b <a class=3D"x_moz-txt-link-abbreviated"
href=3D"mailto:users-bounces@ovirt.=
org">users-bounces(a)ovirt.org</a <a
class=3D"x_moz-txt-link-rfc2396E" href=3D"mailto:users-bounces@ovirt.org=
">&lt;users-bounces(a)ovirt.org&gt;</a> on behalf of Erekle Magradze
<a class=3D"x_moz-txt-link-rfc2396E"
href=3D"mailto:erekle.magradze@recogiz=
er.de">&lt;erekle.magradze(a)recogizer.de&gt;</a><br <b>Sent:</b> Thursday, February 22, 2018 11:48
PM<br <b>To:</b> <a
class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:users@ovi=
rt.org">users(a)ovirt.org</a><br
<b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from time to
time</=
font <div> </div
</div <div
class=3D"x_BodyFragment"><font size=3D"2"><span
style=3D"font-size:11p=
t" <div class=3D"x_PlainText">Dear all,<br <br It would be great if someone will
share any experience regarding the <br similar
case, would be great to have a hint where to start investigation.<b=
r <br Thanks again<br <br Cheers<br <br Erekle<br <br <br On
02/22/2018 05:05 PM, Erekle Magradze wrote:<br >
Hello there,<br ><br > I am facing the following problem from time to time
one of the <br > hypervisor (there are 3 of
them)s is rebooting, I am using <br >
ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage <br=
> backend (glusterfs-3.12.5-2.el7.x86_64).<br ><br >
I am suspecting gluster because of the e.g. message bellow from one of=
<br > the volumes,<br
><br > Could you please help and
suggest to which direction should <br >
investigation go?<br ><br > Thanks in advance<br
><br > Cheers<br ><br >
Erekle<br ><br ><br >
[2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]<br > [2018-02-22 15:41:10.198701] I [MSGID: 109063]
<br > [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br > anomalies in (null) (gfid
=3D 00000000-0000-0000-0000-000000000000). <=
br > Holes=3D1 overlaps=3D0<br > [2018-02-22 15:41:10.198704] I [MSGID: 109063]
<br > [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br > anomalies in (null) (gfid
=3D 00000000-0000-0000-0000-000000000000). <=
br > Holes=3D1 overlaps=3D0<br > [2018-02-22 15:42:11.293608] I [MSGID: 109063]
<br > [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br > anomalies in (null) (gfid
=3D 00000000-0000-0000-0000-000000000000). <=
br > Holes=3D1 overlaps=3D0<br > [2018-02-22 15:53:16.245720] I [MSGID: 100030]
<br > [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs:
Started running <br > /usr/sbin/glusterfs version
3.12.5 (args: /usr/sbin/glusterfs <br >
--volfile-server=3D10.0.0.21 --volfi<br >
le-server=3D10.0.0.22 --volfile-server=3D10.0.0.23 <br >
--volfile-id=3D/virtimages <br >
/rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)<br >
[2018-02-22 15:53:16.263712] W [MSGID: 101002] <br >
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' <=
br > is deprecated, preferred is
'transport.address-family', continuing <br=
> with correction<br >
[2018-02-22 15:53:16.269595] I [MSGID: 101190] <br >
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br > thread with index 1<br >
[2018-02-22 15:53:16.273483] I [MSGID: 101190] <br >
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br > thread with index 2<br >
[2018-02-22 15:53:16.273594] W [MSGID: 101174] <br >
[graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead: <br > option 'parallel-readdir' is not
recognized<br > [2018-02-22
15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify] =
<br > 0-virtimages-client-0: parent translators are ready,
attempting <br > connect on
transport<br > [2018-02-22
15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify] =
<br > 0-virtimages-client-1: parent translators are ready,
attempting <br > connect on
transport<br > [2018-02-22
15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br=
> 0-virtimages-client-0: changing port to 49152 (from
0)<br > [2018-02-22
15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify] =
<br > 0-virtimages-client-2: parent translators are ready,
attempting <br > connect on
transport<br > [2018-02-22
15:53:16.282126] I [MSGID: 114057] <br >
[client-handshake.c:1478:select_server_supported_programs] <br > 0-virtimages-client-0: Using Program GlusterFS 3.3,
Num (1298437), <br=
> Version (330)<br >
[2018-02-22 15:53:16.282573] I [MSGID: 114046] <br >
[client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0: =
<br > Connected to virtimages-client-0, attached to remote
volume <br >
'/mnt/virtimages/virtimgs'.<br >
[2018-02-22 15:53:16.282584] I [MSGID: 114047] <br >
[client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0: =
<br > Server and Client lk-version numbers are not same,
reopening the fds<b=
r > [2018-02-22 15:53:16.282665] I [MSGID: 108005]
<br > [afr-common.c:4929:__afr_handle_child_up_event]
<br > 0-virtimages-replicate-0: Subvolume
'virtimages-client-0' came back <b=
r > up; going online.<br >
[2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br=
> 0-virtimages-client-1: changing port to 49152 (from
0)<br > [2018-02-22
15:53:16.282934] I [MSGID: 114035] <br >
[client-handshake.c:202:client_set_lk_version_cbk] <br >
0-virtimages-client-0: Server lk version =3D 1<br
><br >
_______________________________________________<br >
Users mailing list<br > <a
class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org=
">Users(a)ovirt.org</a><br >
<a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
.ovirt.org/mailman/listinfo/users</a><br
<br -- <br Recogizer Group GmbH<br
<br Dr.rer.nat. Erekle
Magradze<br Lead Big Data Engineering &
DevOps<br Rheinwerkallee 2, 53227
Bonn<br Tel: +49 228
29974555<br <br E-Mail
<a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:erekle.magrad=
ze(a)recogizer.de"
erekle.magradze(a)recogizer.de</a><br
recogizer.com<br <br
-----------------------------------------------------------------<br <br Recogizer Group GmbH<br Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze<br Handelsregister: Amtsgericht Bonn HRB 20724<br Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993<br Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich
gesch=FCtzte Informa=
tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC=
mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch=
en Sie diese Mail. Das unerlaubte Kopieren
sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Infor=
mationen ist nicht gestattet.<br
<br
_______________________________________________<br Users
mailing list<br <a
class=3D"x_moz-txt-link-abbreviated"
href=3D"mailto:Users@ovirt.org">Use=
rs(a)ovirt.org</a><br <a
href=3D"http://lists.ovirt.org/mailman/listinfo/users">http:...
t.org/mailman/listinfo/users</a><br
</div
</span></font></div
</blockquote <br <pre
class=3D"x_moz-signature" cols=3D"72">--=20
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail <a class=3D"x_moz-txt-link-abbreviated"
href=3D"mailto:erekle.magrad=
ze@recogizer.de">erekle.magradze(a)recogizer.de</a
recogizer.com
-----------------------------------------------------------------
Recogizer Group GmbH
Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze
Handelsregister: Amtsgericht Bonn HRB 20724
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa=
tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC=
mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch=
en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d=
ieser Mail und der darin enthaltenen Informationen ist nicht gestattet.</pr=
e </div
</body </html
--_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_--
Hello there,
I am facing the following problem from time to time one of the
hypervisor (there are 3 of them)s is rebooting, I am using
ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage
backend (glusterfs-3.12.5-2.el7.x86_64).
I am suspecting gluster because of the e.g. message bellow from one of
the volumes,
Could you please help and suggest to which direction should
investigation go?
Thanks in advance
Cheers
Erekle
[2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]
[2018-02-22 15:41:10.198701] I [MSGID: 109063]
[dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000).
Holes=3D1 overlaps=3D0
[2018-02-22 15:41:10.198704] I [MSGID: 109063]
[dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000).
Holes=3D1 overlaps=3D0
[2018-02-22 15:42:11.293608] I [MSGID: 109063]
[dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000).
Holes=3D1 overlaps=3D0
[2018-02-22 15:53:16.245720] I [MSGID: 100030]
[glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running
/usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs
--volfile-server=3D10.0.0.21 --volfi
le-server=3D10.0.0.22 --volfile-server=3D10.0.0.23
--volfile-id=3D/virtimages
/rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)
[2018-02-22 15:53:16.263712] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family'
is deprecated, preferred is 'transport.address-family', continuing
with correction
[2018-02-22 15:53:16.269595] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1
[2018-02-22 15:53:16.273483] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 2
[2018-02-22 15:53:16.273594] W [MSGID: 101174]
[graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead:
option 'parallel-readdir' is not recognized
[2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify]
0-virtimages-client-0: parent translators are ready, attempting
connect on transport
[2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify]
0-virtimages-client-1: parent translators are ready, attempting
connect on transport
[2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-virtimages-client-0: changing port to 49152 (from 0)
[2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify]
0-virtimages-client-2: parent translators are ready, attempting
connect on transport
[2018-02-22 15:53:16.282126] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437),
Version (330)
[2018-02-22 15:53:16.282573] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0:
Connected to virtimages-client-0, attached to remote volume
'/mnt/virtimages/virtimgs'.
[2018-02-22 15:53:16.282584] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0:
Server and Client lk-version numbers are not same, reopening the fds
[2018-02-22 15:53:16.282665] I [MSGID: 108005]
[afr-common.c:4929:__afr_handle_child_up_event]
0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back
up; going online.
[2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
0-virtimages-client-1: changing port to 49152 (from 0)
[2018-02-22 15:53:16.282934] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-virtimages-client-0: Server lk version =3D 1
_______________________________________________
Users mailing list
Users@ovirt.org<mailto:Users@ovirt.org
http://lists.ovirt.org/mailman/listinfo/users
--
Recogizer Group GmbH
Dr.rer.nat. Erekle Magradze
Lead Big Data Engineering & DevOps
Rheinwerkallee 2, 53227 Bonn
Tel: +49 228 29974555
E-Mail erekle.magradze@recogizer.de<mailto:erekle.magradze@recogizer.de
9:43 a.m.
New subject: rebooting hypervisors from time to time
This is a multi-part message in MIME format.
--------------F560D7B5EE4FCE4640FF5B04
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 8bit
Hi,
Thanks a lot for having a look.
HA VMs were migrated, non HA vms were turned off, syslogs were not
saying anything useful, dmesg reported graceful reboot.
What errors are indicating? may be there is a useful hint to proceed in
investigation?
Thanks in advance again
Cheers
Erekle
On 02/23/2018 06:15 PM, Mahdi Adnan wrote:
Hi,
The log does't indicate HV reboot, and i see lots of errors in the logs.
During the reboot, what happened to the VM inside of the HV ? migrated
? paused ? what about the system's logs ? does it indicate a graceful
shutdown ?
--
Respectfully*
**Mahdi A. Mahdi*
------------------------------------------------------------------------
*From:* Erekle Magradze <erekle.magradze(a)recogizer.de>
*Sent:* Friday, February 23, 2018 2:48 PM
*To:* Mahdi Adnan; users(a)ovirt.org
*Subject:* Re: [ovirt-users] rebooting hypervisors from time to time
Thanks for the reply,
I've attached all the logs from yesterday, reboot has happened during
the day but this is not the first time and this is not the only one
hypervisor.
Kind Regards
Erekle
On 02/23/2018 09:00 AM, Mahdi Adnan wrote:
> Hi,
>
> Can you post the VDSM and Engine logs ?
>
>
> --
>
> Respectfully*
> **Mahdi A. Mahdi*
>
> ------------------------------------------------------------------------
> *From:* users-bounces(a)ovirt.org <mailto:users-bounces@ovirt.org>
> <users-bounces(a)ovirt.org> <mailto:users-bounces@ovirt.org> on behalf
> of Erekle Magradze <erekle.magradze(a)recogizer.de>
> <mailto:erekle.magradze@recogizer.de>
> *Sent:* Thursday, February 22, 2018 11:48 PM
> *To:* users(a)ovirt.org <mailto:users@ovirt.org>
> *Subject:* Re: [ovirt-users] rebooting hypervisors from time to time
> Dear all,
>
> It would be great if someone will share any experience regarding the
> similar case, would be great to have a hint where to start investigation.
>
> Thanks again
>
> Cheers
>
> Erekle
>
>
> On 02/22/2018 05:05 PM, Erekle Magradze wrote:
> > Hello there,
> >
> > I am facing the following problem from time to time one of the
> > hypervisor (there are 3 of them)s is rebooting, I am using
> > ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage
> > backend (glusterfs-3.12.5-2.el7.x86_64).
> >
> > I am suspecting gluster because of the e.g. message bellow from one of
> > the volumes,
> >
> > Could you please help and suggest to which direction should
> > investigation go?
> >
> > Thanks in advance
> >
> > Cheers
> >
> > Erekle
> >
> >
> > [2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]
> > [2018-02-22 15:41:10.198701] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:41:10.198704] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:42:11.293608] I [MSGID: 109063]
> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found
> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000).
> > Holes=1 overlaps=0
> > [2018-02-22 15:53:16.245720] I [MSGID: 100030]
> > [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running
> > /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs
> > --volfile-server=10.0.0.21 --volfi
> > le-server=10.0.0.22 --volfile-server=10.0.0.23
> > --volfile-id=/virtimages
> > /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)
> > [2018-02-22 15:53:16.263712] W [MSGID: 101002]
> > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family'
> > is deprecated, preferred is 'transport.address-family', continuing
> > with correction
> > [2018-02-22 15:53:16.269595] I [MSGID: 101190]
> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
> > thread with index 1
> > [2018-02-22 15:53:16.273483] I [MSGID: 101190]
> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started
> > thread with index 2
> > [2018-02-22 15:53:16.273594] W [MSGID: 101174]
> > [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead:
> > option 'parallel-readdir' is not recognized
> > [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-0: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-1: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
> > 0-virtimages-client-0: changing port to 49152 (from 0)
> > [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify]
> > 0-virtimages-client-2: parent translators are ready, attempting
> > connect on transport
> > [2018-02-22 15:53:16.282126] I [MSGID: 114057]
> > [client-handshake.c:1478:select_server_supported_programs]
> > 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437),
> > Version (330)
> > [2018-02-22 15:53:16.282573] I [MSGID: 114046]
> > [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0:
> > Connected to virtimages-client-0, attached to remote volume
> > '/mnt/virtimages/virtimgs'.
> > [2018-02-22 15:53:16.282584] I [MSGID: 114047]
> > [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0:
> > Server and Client lk-version numbers are not same, reopening the fds
> > [2018-02-22 15:53:16.282665] I [MSGID: 108005]
> > [afr-common.c:4929:__afr_handle_child_up_event]
> > 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back
> > up; going online.
> > [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig]
> > 0-virtimages-client-1: changing port to 49152 (from 0)
> > [2018-02-22 15:53:16.282934] I [MSGID: 114035]
> > [client-handshake.c:202:client_set_lk_version_cbk]
> > 0-virtimages-client-0: Server lk version = 1
> >
> > _______________________________________________
> > Users mailing list
> > Users(a)ovirt.org <mailto:Users@ovirt.org>
> > http://lists.ovirt.org/mailman/listinfo/users
>
> --
> Recogizer Group GmbH
>
> Dr.rer.nat. Erekle Magradze
> Lead Big Data Engineering & DevOps
> Rheinwerkallee 2, 53227 Bonn
> Tel: +49 228 29974555
>
> E-Mail erekle.magradze(a)recogizer.de <mailto:erekle.magradze@recogizer.de>
> recogizer.com
>
> -----------------------------------------------------------------
>
> Recogizer Group GmbH
> Geschäftsführer: Oliver Habisch, Carsten Kreutze
> Handelsregister: Amtsgericht Bonn HRB 20724
> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993
> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
> Informationen. Wenn Sie nicht der richtige Adressat sind oder diese
> E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den
> Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie
> die unbefugte Weitergabe dieser Mail und der darin enthaltenen
> Informationen ist nicht gestattet.
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org <mailto:Users@ovirt.org>
> http://lists.ovirt.org/mailman/listinfo/users
--------------F560D7B5EE4FCE4640FF5B04
Content-Type: text/html; charset=windows-1252
Content-Transfer-Encoding: 8bit
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi,</p>
<p>Thanks a lot for having a look.<br>
</p>
<p>HA VMs were migrated, non HA vms were turned off, syslogs were
not saying anything useful, dmesg reported graceful reboot.</p>
<p>What errors are indicating? may be there is a useful hint to
proceed in investigation?</p>
<p>Thanks in advance again</p>
<p>Cheers</p>
<p>Erekle<br>
</p>
<br>
<div class="moz-cite-prefix">On 02/23/2018 06:15 PM, Mahdi Adnan
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0@DM5PR01MB2506.prod.exchangelabs.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"> P
{margin-top:0;margin-bottom:0;} </style>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
The log does't indicate HV reboot, and i see lots of errors in
the logs.</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0,
0, 0, 0);">
During the reboot, what happened to the VM inside of the HV ?
migrated ? paused ? what about the system's logs ? does it
indicate a graceful shutdown ?</div>
<div style="font-family: Calibri, Helvetica, sans-serif;
font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font
color="#000000">Respectfully<b><br>
</b><b>Mahdi A.
Mahdi</b></font></font><font
color="#3366ff"><br>
<br>
</font></div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt"
face="Calibri, sans-serif"
color="#000000"><b>From:</b> Erekle
Magradze <a class="moz-txt-link-rfc2396E"
href="mailto:erekle.magradze@recogizer.de"><erekle.magradze@recogizer.de></a><br>
<b>Sent:</b> Friday, February 23, 2018 2:48 PM<br>
<b>To:</b> Mahdi Adnan; <a
class="moz-txt-link-abbreviated"
href="mailto:users@ovirt.org">users@ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from
time to time</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">
<p>Thanks for the reply,</p>
<p>I've attached all the logs from yesterday, reboot has
happened during the day but this is not the first time and
this is not the only one hypervisor.</p>
<p>Kind Regards</p>
<p>Erekle</p>
<br>
<div class="x_moz-cite-prefix">On 02/23/2018 09:00 AM, Mahdi
Adnan wrote:<br>
</div>
<blockquote type="cite">
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
Hi,</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0);
background-color:rgba(0,0,0,0)">
Can you post the VDSM and Engine logs ?</div>
<div style="font-family:Calibri,Helvetica,sans-serif;
font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div id="x_signature"><br>
<div class="x_ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font
color="#000000">Respectfully<b><br>
</b><b>Mahdi A.
Mahdi</b></font></font><font
color="#3366ff"><br>
<br>
</font></div>
</div>
<hr tabindex="-1" style="display:inline-block;
width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font
style="font-size:11pt" face="Calibri, sans-serif"
color="#000000"><b>From:</b>
<a class="x_moz-txt-link-abbreviated"
href="mailto:users-bounces@ovirt.org"
moz-do-not-send="true">users-bounces(a)ovirt.org</a>
<a class="x_moz-txt-link-rfc2396E"
href="mailto:users-bounces@ovirt.org"
moz-do-not-send="true">&lt;users-bounces(a)ovirt.org&gt;</a>
on behalf of Erekle Magradze
<a class="x_moz-txt-link-rfc2396E"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">&lt;erekle.magradze(a)recogizer.de&gt;</a><br>
<b>Sent:</b> Thursday, February 22, 2018 11:48 PM<br>
<b>To:</b> <a class="x_moz-txt-link-abbreviated"
href="mailto:users@ovirt.org"
moz-do-not-send="true">users(a)ovirt.org</a><br>
<b>Subject:</b> Re: [ovirt-users] rebooting hypervisors
from time to time</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font
size="2"><span
style="font-size:11pt">
<div class="x_PlainText">Dear all,<br>
<br>
It would be great if someone will share any experience
regarding the <br>
similar case, would be great to have a hint where to
start investigation.<br>
<br>
Thanks again<br>
<br>
Cheers<br>
<br>
Erekle<br>
<br>
<br>
On 02/22/2018 05:05 PM, Erekle Magradze wrote:<br>
> Hello there,<br>
><br>
> I am facing the following problem from time to
time one of the <br>
> hypervisor (there are 3 of them)s is rebooting, I
am using <br>
> ovirt-release42-4.2.1-1.el7.centos.noarch and
glsuter as a storage <br>
> backend (glusterfs-3.12.5-2.el7.x86_64).<br>
><br>
> I am suspecting gluster because of the e.g.
message bellow from one of <br>
> the volumes,<br>
><br>
> Could you please help and suggest to which
direction should <br>
> investigation go?<br>
><br>
> Thanks in advance<br>
><br>
> Cheers<br>
><br>
> Erekle<br>
><br>
><br>
> [2018-02-22 15:36:10.011687] and [2018-02-22
15:37:10.955013]<br>
> [2018-02-22 15:41:10.198701] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:41:10.198704] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:42:11.293608] I [MSGID: 109063] <br>
> [dht-layout.c:716:dht_layout_normalize]
0-virtimages-dht: Found <br>
> anomalies in (null) (gfid =
00000000-0000-0000-0000-000000000000). <br>
> Holes=1 overlaps=0<br>
> [2018-02-22 15:53:16.245720] I [MSGID: 100030] <br>
> [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs:
Started running <br>
> /usr/sbin/glusterfs version 3.12.5 (args:
/usr/sbin/glusterfs <br>
> --volfile-server=10.0.0.21 --volfi<br>
> le-server=10.0.0.22 --volfile-server=10.0.0.23 <br>
> --volfile-id=/virtimages <br>
>
/rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)<br>
> [2018-02-22 15:53:16.263712] W [MSGID: 101002] <br>
> [options.c:995:xl_opt_validate] 0-glusterfs:
option 'address-family' <br>
> is deprecated, preferred is
'transport.address-family', continuing <br>
> with correction<br>
> [2018-02-22 15:53:16.269595] I [MSGID: 101190] <br>
> [event-epoll.c:613:event_dispatch_epoll_worker]
0-epoll: Started <br>
> thread with index 1<br>
> [2018-02-22 15:53:16.273483] I [MSGID: 101190] <br>
> [event-epoll.c:613:event_dispatch_epoll_worker]
0-epoll: Started <br>
> thread with index 2<br>
> [2018-02-22 15:53:16.273594] W [MSGID: 101174] <br>
> [graph.c:363:_log_if_unknown_option]
0-virtimages-readdir-ahead: <br>
> option 'parallel-readdir' is not recognized<br>
> [2018-02-22 15:53:16.273703] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-0: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.276455] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-1: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.276683] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] <br>
> 0-virtimages-client-0: changing port to 49152
(from 0)<br>
> [2018-02-22 15:53:16.279191] I [MSGID: 114020]
[client.c:2360:notify] <br>
> 0-virtimages-client-2: parent translators are
ready, attempting <br>
> connect on transport<br>
> [2018-02-22 15:53:16.282126] I [MSGID: 114057] <br>
>
[client-handshake.c:1478:select_server_supported_programs]
<br>
> 0-virtimages-client-0: Using Program GlusterFS
3.3, Num (1298437), <br>
> Version (330)<br>
> [2018-02-22 15:53:16.282573] I [MSGID: 114046] <br>
> [client-handshake.c:1231:client_setvolume_cbk]
0-virtimages-client-0: <br>
> Connected to virtimages-client-0, attached to
remote volume <br>
> '/mnt/virtimages/virtimgs'.<br>
> [2018-02-22 15:53:16.282584] I [MSGID: 114047] <br>
> [client-handshake.c:1242:client_setvolume_cbk]
0-virtimages-client-0: <br>
> Server and Client lk-version numbers are not
same, reopening the fds<br>
> [2018-02-22 15:53:16.282665] I [MSGID: 108005] <br>
> [afr-common.c:4929:__afr_handle_child_up_event] <br>
> 0-virtimages-replicate-0: Subvolume
'virtimages-client-0' came back <br>
> up; going online.<br>
> [2018-02-22 15:53:16.282877] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] <br>
> 0-virtimages-client-1: changing port to 49152
(from 0)<br>
> [2018-02-22 15:53:16.282934] I [MSGID: 114035] <br>
>
[client-handshake.c:202:client_set_lk_version_cbk] <br>
> 0-virtimages-client-0: Server lk version = 1<br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a class="x_moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
> <a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
<br>
-- <br>
Recogizer Group GmbH<br>
<br>
Dr.rer.nat. Erekle Magradze<br>
Lead Big Data Engineering & DevOps<br>
Rheinwerkallee 2, 53227 Bonn<br>
Tel: +49 228 29974555<br>
<br>
E-Mail <a class="x_moz-txt-link-abbreviated"
href="mailto:erekle.magradze@recogizer.de"
moz-do-not-send="true">
erekle.magradze(a)recogizer.de</a><br>
recogizer.com<br>
<br>
-----------------------------------------------------------------<br>
<br>
Recogizer Group GmbH<br>
Geschäftsführer: Oliver Habisch, Carsten Kreutze<br>
Handelsregister: Amtsgericht Bonn HRB 20724<br>
Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993<br>
Diese E-Mail enthält vertrauliche und/oder rechtlich
geschützte Informationen. Wenn Sie nicht der richtige
Adressat sind oder diese E-Mail irrtümlich erhalten
haben, informieren Sie bitte sofort den Absender und
löschen Sie diese Mail. Das unerlaubte Kopieren sowie
die unbefugte Weitergabe dieser Mail und der darin
enthaltenen Informationen ist nicht gestattet.<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a class="x_moz-txt-link-abbreviated"
href="mailto:Users@ovirt.org"
moz-do-not-send="true">Users(a)ovirt.org</a><br>
<a
href="http://lists.ovirt.org/mailman/listinfo/users"
moz-do-not-send="true">http://lists.ovirt.org/mailman/listin...
</div>
</span></font></div>
</blockquote>
<br>
</div>
</blockquote>
<br>
</body>
</html>
--------------F560D7B5EE4FCE4640FF5B04--
2255
days inactive
2255
days old
1 comments
2 participants
participants (2)
-
Erekle Magradze -
Mahdi Adnan