This is a multi-part message in MIME format. --------------F560D7B5EE4FCE4640FF5B04 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Hi, Thanks a lot for having a look. HA VMs were migrated, non HA vms were turned off, syslogs were not saying anything useful, dmesg reported graceful reboot. What errors are indicating? may be there is a useful hint to proceed in investigation? Thanks in advance again Cheers Erekle On 02/23/2018 06:15 PM, Mahdi Adnan wrote:

Hi,

The log does't indicate HV reboot, and i see lots of errors in the logs. During the reboot, what happened to the VM inside of the HV ? migrated ? paused ? what about the system's logs ? does it indicate a graceful shutdown ?

--

Respectfully* **Mahdi A. Mahdi*

------------------------------------------------------------------------ *From:* Erekle Magradze <erekle.magradze@recogizer.de> *Sent:* Friday, February 23, 2018 2:48 PM *To:* Mahdi Adnan; users@ovirt.org *Subject:* Re: [ovirt-users] rebooting hypervisors from time to time

Thanks for the reply,

I've attached all the logs from yesterday, reboot has happened during the day but this is not the first time and this is not the only one hypervisor.

Kind Regards

Erekle

On 02/23/2018 09:00 AM, Mahdi Adnan wrote:

...

Hi,

Can you post the VDSM and Engine logs ?

--

Respectfully* **Mahdi A. Mahdi*

------------------------------------------------------------------------ *From:* users-bounces@ovirt.org <mailto:users-bounces@ovirt.org> <users-bounces@ovirt.org> <mailto:users-bounces@ovirt.org> on behalf of Erekle Magradze <erekle.magradze@recogizer.de> <mailto:erekle.magradze@recogizer.de> *Sent:* Thursday, February 22, 2018 11:48 PM *To:* users@ovirt.org <mailto:users@ovirt.org> *Subject:* Re: [ovirt-users] rebooting hypervisors from time to time Dear all,

It would be great if someone will share any experience regarding the similar case, would be great to have a hint where to start investigation.

Thanks again

Cheers

Erekle

On 02/22/2018 05:05 PM, Erekle Magradze wrote:

...

Hello there,

I am facing the following problem from time to time one of the hypervisor (there are 3 of them)s is rebooting, I am using ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage backend (glusterfs-3.12.5-2.el7.x86_64).

I am suspecting gluster because of the e.g. message bellow from one of the volumes,

Could you please help and suggest to which direction should investigation go?

Thanks in advance

Cheers

Erekle

[2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013] [2018-02-22 15:41:10.198701] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 [2018-02-22 15:41:10.198704] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 [2018-02-22 15:42:11.293608] I [MSGID: 109063] [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0 [2018-02-22 15:53:16.245720] I [MSGID: 100030] [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs --volfile-server=10.0.0.21 --volfi le-server=10.0.0.22 --volfile-server=10.0.0.23 --volfile-id=/virtimages /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages) [2018-02-22 15:53:16.263712] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction [2018-02-22 15:53:16.269595] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2018-02-22 15:53:16.273483] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2 [2018-02-22 15:53:16.273594] W [MSGID: 101174] [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead: option 'parallel-readdir' is not recognized [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-0: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-1: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-virtimages-client-0: changing port to 49152 (from 0) [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify] 0-virtimages-client-2: parent translators are ready, attempting connect on transport [2018-02-22 15:53:16.282126] I [MSGID: 114057] [client-handshake.c:1478:select_server_supported_programs] 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330) [2018-02-22 15:53:16.282573] I [MSGID: 114046] [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0: Connected to virtimages-client-0, attached to remote volume '/mnt/virtimages/virtimgs'. [2018-02-22 15:53:16.282584] I [MSGID: 114047] [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0: Server and Client lk-version numbers are not same, reopening the fds [2018-02-22 15:53:16.282665] I [MSGID: 108005] [afr-common.c:4929:__afr_handle_child_up_event] 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back up; going online. [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig] 0-virtimages-client-1: changing port to 49152 (from 0) [2018-02-22 15:53:16.282934] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-virtimages-client-0: Server lk version = 1

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users

-- Recogizer Group GmbH

Dr.rer.nat. Erekle Magradze Lead Big Data Engineering & DevOps Rheinwerkallee 2, 53227 Bonn Tel: +49 228 29974555

E-Mail erekle.magradze@recogizer.de <mailto:erekle.magradze@recogizer.de> recogizer.com

-----------------------------------------------------------------

Recogizer Group GmbH Geschäftsführer: Oliver Habisch, Carsten Kreutze Handelsregister: Amtsgericht Bonn HRB 20724 Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993 Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Informationen ist nicht gestattet.

_______________________________________________ Users mailing list Users@ovirt.org <mailto:Users@ovirt.org> http://lists.ovirt.org/mailman/listinfo/users

--------------F560D7B5EE4FCE4640FF5B04 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> </head> <body text="#000000" bgcolor="#FFFFFF"> <p>Hi,</p> <p>Thanks a lot for having a look.<br> </p> <p>HA VMs were migrated, non HA vms were turned off, syslogs were not saying anything useful, dmesg reported graceful reboot.</p> <p>What errors are indicating? may be there is a useful hint to proceed in investigation?</p> <p>Thanks in advance again</p> <p>Cheers</p> <p>Erekle<br> </p> <br> <div class="moz-cite-prefix">On 02/23/2018 06:15 PM, Mahdi Adnan wrote:<br> </div> <blockquote type="cite" cite="mid:DM5PR01MB2506CA22D55C58A5210C6EA6FFCC0@DM5PR01MB2506.prod.exchangelabs.com"> <meta http-equiv="Content-Type" content="text/html; charset=windows-1252"> <style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style> <div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> Hi,</div> <div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> <br> </div> <div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> The log does't indicate HV reboot, and i see lots of errors in the logs.</div> <div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0);"> During the reboot, what happened to the VM inside of the HV ? migrated ? paused ? what about the system's logs ? does it indicate a graceful shutdown ?</div> <div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"> <br> </div> <div id="signature"><br> <div class="ecxmoz-signature">-- <br> <br> <font color="#3366ff"><font color="#000000">Respectfully<b><br> </b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br> <br> </font></div> </div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Erekle Magradze <a class="moz-txt-link-rfc2396E" href="mailto:erekle.magradze@recogizer.de"><erekle.magradze@recogizer.de></a><br> <b>Sent:</b> Friday, February 23, 2018 2:48 PM<br> <b>To:</b> Mahdi Adnan; <a class="moz-txt-link-abbreviated" href="mailto:users@ovirt.org">users@ovirt.org</a><br> <b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from time to time</font> <div> </div> </div> <div style="background-color:#FFFFFF"> <p>Thanks for the reply,</p> <p>I've attached all the logs from yesterday, reboot has happened during the day but this is not the first time and this is not the only one hypervisor.</p> <p>Kind Regards</p> <p>Erekle</p> <br> <div class="x_moz-cite-prefix">On 02/23/2018 09:00 AM, Mahdi Adnan wrote:<br> </div> <blockquote type="cite"> <style type="text/css" style="display:none"> <!-- p {margin-top:0; margin-bottom:0} --> </style> <div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"> Hi,</div> <div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"> <br> </div> <div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0); background-color:rgba(0,0,0,0)"> Can you post the VDSM and Engine logs ?</div> <div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"> <br> </div> <div id="x_signature"><br> <div class="x_ecxmoz-signature">-- <br> <br> <font color="#3366ff"><font color="#000000">Respectfully<b><br> </b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br> <br> </font></div> </div> <hr tabindex="-1" style="display:inline-block; width:98%"> <div id="x_divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:users-bounces@ovirt.org" moz-do-not-send="true">users-bounces@ovirt.org</a> <a class="x_moz-txt-link-rfc2396E" href="mailto:users-bounces@ovirt.org" moz-do-not-send="true"><users-bounces@ovirt.org></a> on behalf of Erekle Magradze <a class="x_moz-txt-link-rfc2396E" href="mailto:erekle.magradze@recogizer.de" moz-do-not-send="true"><erekle.magradze@recogizer.de></a><br> <b>Sent:</b> Thursday, February 22, 2018 11:48 PM<br> <b>To:</b> <a class="x_moz-txt-link-abbreviated" href="mailto:users@ovirt.org" moz-do-not-send="true">users@ovirt.org</a><br> <b>Subject:</b> Re: [ovirt-users] rebooting hypervisors from time to time</font> <div> </div> </div> <div class="x_BodyFragment"><font size="2"><span style="font-size:11pt"> <div class="x_PlainText">Dear all,<br> <br> It would be great if someone will share any experience regarding the <br> similar case, would be great to have a hint where to start investigation.<br> <br> Thanks again<br> <br> Cheers<br> <br> Erekle<br> <br> <br> On 02/22/2018 05:05 PM, Erekle Magradze wrote:<br> > Hello there,<br> ><br> > I am facing the following problem from time to time one of the <br> > hypervisor (there are 3 of them)s is rebooting, I am using <br> > ovirt-release42-4.2.1-1.el7.centos.noarch and glsuter as a storage <br> > backend (glusterfs-3.12.5-2.el7.x86_64).<br> ><br> > I am suspecting gluster because of the e.g. message bellow from one of <br> > the volumes,<br> ><br> > Could you please help and suggest to which direction should <br> > investigation go?<br> ><br> > Thanks in advance<br> ><br> > Cheers<br> ><br> > Erekle<br> ><br> ><br> > [2018-02-22 15:36:10.011687] and [2018-02-22 15:37:10.955013]<br> > [2018-02-22 15:41:10.198701] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). <br> > Holes=1 overlaps=0<br> > [2018-02-22 15:41:10.198704] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). <br> > Holes=1 overlaps=0<br> > [2018-02-22 15:42:11.293608] I [MSGID: 109063] <br> > [dht-layout.c:716:dht_layout_normalize] 0-virtimages-dht: Found <br> > anomalies in (null) (gfid = 00000000-0000-0000-0000-000000000000). <br> > Holes=1 overlaps=0<br> > [2018-02-22 15:53:16.245720] I [MSGID: 100030] <br> > [glusterfsd.c:2524:main] 0-/usr/sbin/glusterfs: Started running <br> > /usr/sbin/glusterfs version 3.12.5 (args: /usr/sbin/glusterfs <br> > --volfile-server=10.0.0.21 --volfi<br> > le-server=10.0.0.22 --volfile-server=10.0.0.23 <br> > --volfile-id=/virtimages <br> > /rhev/data-center/mnt/glusterSD/10.0.0.21:_virtimages)<br> > [2018-02-22 15:53:16.263712] W [MSGID: 101002] <br> > [options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' <br> > is deprecated, preferred is 'transport.address-family', continuing <br> > with correction<br> > [2018-02-22 15:53:16.269595] I [MSGID: 101190] <br> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br> > thread with index 1<br> > [2018-02-22 15:53:16.273483] I [MSGID: 101190] <br> > [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started <br> > thread with index 2<br> > [2018-02-22 15:53:16.273594] W [MSGID: 101174] <br> > [graph.c:363:_log_if_unknown_option] 0-virtimages-readdir-ahead: <br> > option 'parallel-readdir' is not recognized<br> > [2018-02-22 15:53:16.273703] I [MSGID: 114020] [client.c:2360:notify] <br> > 0-virtimages-client-0: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.276455] I [MSGID: 114020] [client.c:2360:notify] <br> > 0-virtimages-client-1: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.276683] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br> > 0-virtimages-client-0: changing port to 49152 (from 0)<br> > [2018-02-22 15:53:16.279191] I [MSGID: 114020] [client.c:2360:notify] <br> > 0-virtimages-client-2: parent translators are ready, attempting <br> > connect on transport<br> > [2018-02-22 15:53:16.282126] I [MSGID: 114057] <br> > [client-handshake.c:1478:select_server_supported_programs] <br> > 0-virtimages-client-0: Using Program GlusterFS 3.3, Num (1298437), <br> > Version (330)<br> > [2018-02-22 15:53:16.282573] I [MSGID: 114046] <br> > [client-handshake.c:1231:client_setvolume_cbk] 0-virtimages-client-0: <br> > Connected to virtimages-client-0, attached to remote volume <br> > '/mnt/virtimages/virtimgs'.<br> > [2018-02-22 15:53:16.282584] I [MSGID: 114047] <br> > [client-handshake.c:1242:client_setvolume_cbk] 0-virtimages-client-0: <br> > Server and Client lk-version numbers are not same, reopening the fds<br> > [2018-02-22 15:53:16.282665] I [MSGID: 108005] <br> > [afr-common.c:4929:__afr_handle_child_up_event] <br> > 0-virtimages-replicate-0: Subvolume 'virtimages-client-0' came back <br> > up; going online.<br> > [2018-02-22 15:53:16.282877] I [rpc-clnt.c:1986:rpc_clnt_reconfig] <br> > 0-virtimages-client-1: changing port to 49152 (from 0)<br> > [2018-02-22 15:53:16.282934] I [MSGID: 114035] <br> > [client-handshake.c:202:client_set_lk_version_cbk] <br> > 0-virtimages-client-0: Server lk version = 1<br> ><br> > _______________________________________________<br> > Users mailing list<br> > <a class="x_moz-txt-link-abbreviated" href="mailto:Users@ovirt.org" moz-do-not-send="true">Users@ovirt.org</a><br> > <a href="http://lists.ovirt.org/mailman/listinfo/users" moz-do-not-send="true">http://lists.ovirt.org/mailman/listinfo/users</a><br> <br> -- <br> Recogizer Group GmbH<br> <br> Dr.rer.nat. Erekle Magradze<br> Lead Big Data Engineering & DevOps<br> Rheinwerkallee 2, 53227 Bonn<br> Tel: +49 228 29974555<br> <br> E-Mail <a class="x_moz-txt-link-abbreviated" href="mailto:erekle.magradze@recogizer.de" moz-do-not-send="true"> erekle.magradze@recogizer.de</a><br> recogizer.com<br> <br> -----------------------------------------------------------------<br> <br> Recogizer Group GmbH<br> Geschäftsführer: Oliver Habisch, Carsten Kreutze<br> Handelsregister: Amtsgericht Bonn HRB 20724<br> Sitz der Gesellschaft: Bonn; USt-ID-Nr.: DE294195993<br> Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und löschen Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail und der darin enthaltenen Informationen ist nicht gestattet.<br> <br> _______________________________________________<br> Users mailing list<br> <a class="x_moz-txt-link-abbreviated" href="mailto:Users@ovirt.org" moz-do-not-send="true">Users@ovirt.org</a><br> <a href="http://lists.ovirt.org/mailman/listinfo/users" moz-do-not-send="true">http://lists.ovirt.org/mailman/listinfo/users</a><br> </div> </span></font></div> </blockquote> <br> </div> </blockquote> <br> </body> </html> --------------F560D7B5EE4FCE4640FF5B04--