--_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hi, The log does't indicate HV reboot, and i see lots of errors in the logs. During the reboot, what happened to the VM inside of the HV ? migrated ? pa= used ? what about the system's logs ? does it indicate a graceful shutdown = ? -- Respectfully Mahdi A. Mahdi ________________________________ From: Erekle Magradze <erekle.magradze@recogizer.de> Sent: Friday, February 23, 2018 2:48 PM To: Mahdi Adnan; users@ovirt.org Subject: Re: [ovirt-users] rebooting hypervisors from time to time Thanks for the reply, I've attached all the logs from yesterday, reboot has happened during the d= ay but this is not the first time and this is not the only one hypervisor. Kind Regards Erekle On 02/23/2018 09:00 AM, Mahdi Adnan wrote: Hi, Can you post the VDSM and Engine logs ? -- Respectfully Mahdi A. Mahdi ________________________________ From: users-bounces@ovirt.org<mailto:users-bounces@ovirt.org> <users-bounce= s@ovirt.org><mailto:users-bounces@ovirt.org> on behalf of Erekle Magradze <= erekle.magradze@recogizer.de><mailto:erekle.magradze@recogizer.de> Sent: Thursday, February 22, 2018 11:48 PM To: users@ovirt.org<mailto:users@ovirt.org> Subject: Re: [ovirt-users] rebooting hypervisors from time to time Dear all, It would be great if someone will share any experience regarding the similar case, would be great to have a hint where to start investigation. Thanks again Cheers Erekle On 02/22/2018 05:05 PM, Erekle Magradze wrote:
Hello there,
I am facing the following problem from time to time one of the hypervisor (there are 3 of them)s is rebooting, I am using ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage backend (glusterfs-3.12.5-2.el7.x86_64).
I am suspecting gluster because of the e.g. message bellow from one of the volumes,
Could you please help and suggest to which direction should investigation go?
Thanks in advance
Cheers
Erekle
[2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013] [2018-02-22 15:41:10.198701] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). Holes=3D1 overlaps=3D0 [2018-02-22 15:41:10.198704] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). Holes=3D1 overlaps=3D0 [2018-02-22 15:42:11.293608] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). Holes=3D1 overlaps=3D0 [2018-02-22 15:53:16.245720] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs --volfile-server=3D10.0.0.21 --volfi le-server=3D10.0.0.22 --volfile-server=3D10.0.0.23 --volfile-id=3D/virtimages /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages) [2018-02-22 15:53:16.263712] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2018-02-22 15:53:16.269595] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-02-22 15:53:16.273483] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2018-02-22 15:53:16.273594] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead: option 'parallel-readdir' is not recognized [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-0: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-1: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-virtimages-client-0: changing port to 49152 (from 0) [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-2: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.282126] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 15:53:16.282573] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0: Connected to virtimages-client-0, attached to remote volume '/mnt/virtimages/virtimgs'. [2018-02-22 15:53:16.282584] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 15:53:16.282665] I [MSGID: 108005] [afr-common.c:4929:__afr_handle_child_up_event] 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back up; going online. [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-virtimages-client-1: changing port to 49152 (from 0) [2018-02-22 15:53:16.282934] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-virtimages-client-0: Server lk version =3D 1
_______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users
-- Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze@recogizer.de<mailto:erekle.magradze@recogizer.de> recogizer.com ----------------------------------------------------------------- Recogizer Group GmbH Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa= tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC= mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch= en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d= ieser Mail und der darin enthaltenen Informationen ist nicht gestattet. _______________________________________________ Users mailing list Users@ovirt.org<mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users -- Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail erekle.magradze@recogizer.de<mailto:erekle.magradze@recogizer.de> recogizer.com ----------------------------------------------------------------- Recogizer Group GmbH Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa= tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC= mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch= en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d= ieser Mail und der darin enthaltenen Informationen ist nicht gestattet. --_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"> P {margin-top:0;margin-bo= ttom:0;} </style> </head> <body dir=3D"ltr"> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> Hi,</div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> <br> </div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> The log does't indicate HV reboot, and i see lots of errors in the logs.</d= iv> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> During the reboot, what happened to the VM inside of the HV ? migrated ? pa= used ? what about the system's logs ? does it indicate a graceful shutdown = ?</div> <div style=3D"font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;= color: rgb(0, 0, 0);"> <br> </div> <div id=3D"signature"><br> <div class=3D"ecxmoz-signature">-- <br> <br> <font color=3D"#3366ff"><font color=3D"#000000">Respectfully<b><br> </b><b>Mahdi A. Mahdi</b></font></font><font color=3D"#3366ff"><br> <br> </font><font color=3D"#3366ff"></font></div> </div> <hr style=3D"display:inline-block;width:98%" tabindex=3D"-1"> <div id=3D"divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" st= yle=3D"font-size:11pt" color=3D"#000000"><b>From:</b> Erekle Magradze <e= rekle.magradze@recogizer.de><br> <b>Sent:</b> Friday, February 23, 2018 2:48 PM<br> <b>To:</b> Mahdi Adnan; users@ovirt.org<br> <b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from time to time</= font> <div> </div> </div> <div style=3D"background-color:#FFFFFF"> <p>Thanks for the reply,</p> <p>I've attached all the logs from yesterday, reboot has happened during th= e day but this is not the first time and this is not the only one hyperviso= r.</p> <p>Kind Regards</p> <p>Erekle</p> <br> <div class=3D"x_moz-cite-prefix">On 02/23/2018 09:00 AM, Mahdi Adnan wrote:= <br> </div> <blockquote type=3D"cite"><style type=3D"text/css" style=3D"display:none"> <!-- p {margin-top:0; margin-bottom:0} --> </style> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0); background-color:rgba(0,0,0,0)"> Hi,</div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0); background-color:rgba(0,0,0,0)"> <br> </div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0); background-color:rgba(0,0,0,0)"> Can you post the VDSM and Engine logs ?</div> <div style=3D"font-family:Calibri,Helvetica,sans-serif; font-size:12pt; col= or:rgb(0,0,0)"> <br> </div> <div id=3D"x_signature"><br> <div class=3D"x_ecxmoz-signature">-- <br> <br> <font color=3D"#3366ff"><font color=3D"#000000">Respectfully<b><br> </b><b>Mahdi A. Mahdi</b></font></font><font color=3D"#3366ff"><br> <br> </font></div> </div> <hr tabindex=3D"-1" style=3D"display:inline-block; width:98%"> <div id=3D"x_divRplyFwdMsg" dir=3D"ltr"><font face=3D"Calibri, sans-serif" = color=3D"#000000" style=3D"font-size:11pt"><b>From:</b> <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:users-bounces@ovirt.= org">users-bounces@ovirt.org</a> <a class=3D"x_moz-txt-link-rfc2396E" href=3D"mailto:users-bounces@ovirt.org= "><users-bounces@ovirt.org></a> on behalf of Erekle Magradze <a class=3D"x_moz-txt-link-rfc2396E" href=3D"mailto:erekle.magradze@recogiz= er.de"><erekle.magradze@recogizer.de></a><br> <b>Sent:</b> Thursday, February 22, 2018 11:48 PM<br> <b>To:</b> <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:users@ovi= rt.org">users@ovirt.org</a><br> <b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from time to time</= font> <div> </div> </div> <div class=3D"x_BodyFragment"><font size=3D"2"><span style=3D"font-size:11p= t"> <div class=3D"x_PlainText">Dear all,<br> <br> It would be great if someone will share any experience regarding the <br> similar case, would be great to have a hint where to start investigation.<b= r> <br> Thanks again<br> <br> Cheers<br> <br> Erekle<br> <br> <br> On 02/22/2018 05:05 PM, Erekle Magradze wrote:<br> > Hello there,<br> ><br> > I am facing the following problem from time to time one of the <br> > hypervisor (there are 3 of them)s is rebooting, I am using <br> > ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage <br=
> backend (glusterfs-3.12.5-2.el7.x86_64).<br> ><br> > I am suspecting gluster because of the e.g. message bellow from one of= <br> > the volumes,<br> ><br> > Could you please help and suggest to which direction should <br> > investigation go?<br> ><br> > Thanks in advance<br> ><br> > Cheers<br> ><br> > Erekle<br> ><br> ><br> > [2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]<br> > [2018-02-22 15:41:10.198701] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). <= br> > Holes=3D1 overlaps=3D0<br> > [2018-02-22 15:41:10.198704] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). <= br> > Holes=3D1 overlaps=3D0<br> > [2018-02-22 15:42:11.293608] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid =3D 00000000-0000-0000-0000-000000000000). <= br> > Holes=3D1 overlaps=3D0<br> > [2018-02-22 15:53:16.245720] I [MSGID: 100030] <br> > [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running <br> > /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs <br> > --volfile-server=3D10.0.0.21 --volfi<br> > le-server=3D10.0.0.22 --volfile-server=3D10.0.0.23 <br> > --volfile-id=3D/virtimages <br> > /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)<br> > [2018-02-22 15:53:16.263712] W [MSGID: 101002] <br> > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' <= br> > is deprecated, preferred is 'transport.address-family', continuing <br=
> with correction<br> > [2018-02-22 15:53:16.269595] I [MSGID: 101190] <br> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br> > thread with index 1<br> > [2018-02-22 15:53:16.273483] I [MSGID: 101190] <br> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br> > thread with index 2<br> > [2018-02-22 15:53:16.273594] W [MSGID: 101174] <br> > [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead: <br> > option 'parallel-readdir' is not recognized<br> > [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify] = <br> > 0-virtimages-client-0: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify] = <br> > 0-virtimages-client-1: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br=
> 0-virtimages-client-0: changing port to 49152 (from 0)<br> > [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify] = <br> > 0-virtimages-client-2: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.282126] I [MSGID: 114057] <br> > [client-handshake.c:1478:select_server_supported_programs] <br> > 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437), <br=
> Version (330)<br> > [2018-02-22 15:53:16.282573] I [MSGID: 114046] <br> > [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0: = <br> > Connected to virtimages-client-0, attached to remote volume <br> > '/mnt/virtimages/virtimgs'.<br> > [2018-02-22 15:53:16.282584] I [MSGID: 114047] <br> > [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0: = <br> > Server and Client lk-version numbers are not same, reopening the fds<b= r> > [2018-02-22 15:53:16.282665] I [MSGID: 108005] <br> > [afr-common.c:4929:__afr_handle_child_up_event] <br> > 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back <b= r> > up; going online.<br> > [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br=
> 0-virtimages-client-1: changing port to 49152 (from 0)<br> > [2018-02-22 15:53:16.282934] I [MSGID: 114035] <br> > [client-handshake.c:202:client_set_lk_version_cbk] <br> > 0-virtimages-client-0: Server lk version =3D 1<br> ><br> > _______________________________________________<br> > Users mailing list<br> > <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org= ">Users@ovirt.org</a><br> > <a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http://lists= .ovirt.org/mailman/listinfo/users</a><br> <br> -- <br> Recogizer Group GmbH<br> <br> Dr.rer.nat. Erekle Magradze<br> Lead Big Data Engineering & DevOps<br> Rheinwerkallee 2, 53227 Bonn<br> Tel: +49 228 29974555<br> <br> E-Mail <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:erekle.magrad= ze@recogizer.de"> erekle.magradze@recogizer.de</a><br> recogizer.com<br> <br> -----------------------------------------------------------------<br> <br> Recogizer Group GmbH<br> Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze<br> Handelsregister: Amtsgericht Bonn HRB 20724<br> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993<br> Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa= tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC= mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch= en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Infor= mationen ist nicht gestattet.<br> <br> _______________________________________________<br> Users mailing list<br> <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:Users@ovirt.org">Use= rs@ovirt.org</a><br> <a href=3D"http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovir= t.org/mailman/listinfo/users</a><br> </div> </span></font></div> </blockquote> <br> <pre class=3D"x_moz-signature" cols=3D"72">--=20 Recogizer Group GmbH Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555 E-Mail <a class=3D"x_moz-txt-link-abbreviated" href=3D"mailto:erekle.magrad= ze@recogizer.de">erekle.magradze@recogizer.de</a> recogizer.com ----------------------------------------------------------------- Recogizer Group GmbH Gesch=E4ftsf=FChrer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 Diese E-Mail enth=E4lt vertrauliche und/oder rechtlich gesch=FCtzte Informa= tionen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt=FC= mlich erhalten haben, informieren Sie bitte sofort den Absender und l=F6sch= en Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe d= ieser Mail und der darin enthaltenen Informationen ist nicht gestattet.</pr= e> </div> </body> </html> --_000_DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0DM5PR01MB2506prod_--