<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.E-MailFormatvorlage18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.E-MailFormatvorlage19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=DE link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Hi Markus,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>gdb is available on CentOS 7, but what do you mean by qemu-debug? I Installed qemu-kvm-tools, maybe this is the pendant for CentOS?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'>Regards, Christian<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'>Von:</span></b><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Markus Stockhausen [mailto:stockhausen@collogia.de] <br><b>Gesend</b></span><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>et:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Dienstag, 15. September 2015 20:40<br><b>An:</b> Daniel Helgenberger <daniel.helgenberger@m-box.de><br><b>Cc:</b> Christian Hailer <christian@hailer.eu>; ydary@redhat.com; users@ovirt.org<br><b>Betreff:</b> Re: [ovirt-users] Some VMs in status "not responding" in oVirt interface<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p>Do you have a chance to install qemu-debug? If yes I would try a backtrace.<o:p></o:p></p><p>gdb -p <qemu-pid><br># bt<o:p></o:p></p><p>Markus<o:p></o:p></p><div><p class=MsoNormal>Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger <<a href="mailto:daniel.helgenberger@m-box.de">daniel.helgenberger@m-box.de</a>>:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal>Hello,<br><br>I do not want to hijack the thread but maybe my issue is related?<br><br>It might have started with ovirt 3.5.3; but I cannot tell for sure.<br><br>For me, one vm (foreman) is affected; the second time in 14 days. I can confirm this as I also loose any network connection to the VM and<br>the ability to connect a console.<br>Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid of qemu-kvm process>'<br><br>As far as I can tell the VM became unresponsive at around Sep 15 12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs (see<br>attached).<br><br>Below the engine.log part.<br><br>Versions:<br>ovirt-engine-3.5.4.2-1.el7.centos.noarch<br><br>vdsm-4.16.26-0.el7.centos<br>libvirt-1.2.8-16.el7_1.3<br><br>engine.log (1200 - 1300:<br>2015-09-15 12:03:47,949 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-56) [264d502a] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:08:02,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update<br>VMs/Templates Ovf.<br>2015-09-15 12:08:02,709 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected : ID:<br>00000002-0002-0002-0002-000000000088 Type: l<br>2015-09-15 12:08:02,780 INFO [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] (DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks= key: 00000002-0002-0002-0002-000000000088 value: OVF_UPDATE<br>2015-09-15 12:08:47,997 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:13:06,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] START, GetFileStatsVDSCommand( storagePoolId = 00000002-0002-0002-0002-000000000088, ignoreFailoverLimit = false), log id: 1503968<br>2015-09-15 12:13:07,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] (org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] FINISH, GetFileStatsVDSCommand, return: {pfSense-2.0-RELEASE-i386.iso={status=0, ctime=1432286887.0, size=115709952},<br>Fedora-15-i686-Live8<br>2015-09-15 12:13:07,178 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished<br>automatic refresh process for ISO file type with success, for storage domain id 84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.<br>2015-09-15 12:13:48,043 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:18:48,088 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-44) [6345e698] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:23:48,137 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-13) HA reservation<br>status for cluster Default is OK<br>2015-09-15 12:28:48,183 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-76) [154c91d5] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:33:48,229 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:34:49,432 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM<br>foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> NotResponding<br>2015-09-15 12:34:49,578 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41)<br>[5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM foreman is not responding.<br>2015-09-15 12:38:48,273 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-10) [7a800766] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:43:48,320 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-42) [440f1c40] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:48:48,366 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-70) HA reservation<br>status for cluster Default is OK<br>2015-09-15 12:53:48,412 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-12) [50221cdc] HA<br>reservation status for cluster Default is OK<br>2015-09-15 12:58:48,459 INFO [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] (DefaultQuartzScheduler_Worker-3) HA reservation<br>status for cluster Default is OK<br><br><br><br>On 29.08.2015 22:48, Christian Hailer wrote:<br>> Hello,<br>> <br>> last Wednesday I wanted to update my oVirt 3.5 hypervisor. It is a single Centos <br>> 7 server, so I started by suspending the VMs in order to set the oVirt engine <br>> host to maintenance mode. During the process of suspending the VMs the server <br>> crashed, kernel panic…<br>> <br>> After restarting the server I installed the updates via yum an restarted the <br>> server again. Afterwards, all the VMs could be started again. Some hours later <br>> my monitoring system registered some unresponsive hosts, I had a look in the <br>> oVirt interface, 3 of the VMs were in the state “not responding”, marked by a <br>> question mark.<br>> <br>> I tried to shut down the VMs, but oVirt wasn’t able to do so. I tried to reset <br>> the status in the database with the sql statement<br>> <br>> update vm_dynamic set status = 0 where vm_guid = (select vm_guid from vm_static <br>> where vm_name = 'MYVMNAME');<br>> <br>> but that didn’t help, either. Only rebooting the whole hypervisor helped… <br>> afterwards everything worked again. But only for a few hours, then one of the <br>> VMs entered the “not responding” state again… again only a reboot helped. <br>> Yesterday it happened again:<br>> <br>> 2015-08-28 17:44:22,664 INFO <br>> [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] <br>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC <br>> 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --> NotResponding<br>> <br>> 2015-08-28 17:44:22,692 WARN <br>> [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] <br>> (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: null, Call Stack: <br>> null, Custom Event ID: -1, Message: VM DC is not responding.<br>> <br>> Does anybody know what I can do? Where should I have a look? Hints are greatly <br>> appreciated!<br>> <br>> Thanks,<br>> <br>> Christian<br>> <br><br>-- <br>Daniel Helgenberger<br>m box bewegtbild GmbH<br><br>P: +49/30/2408781-22<br>F: +49/30/2408781-10<br><br>ACKERSTR. 19<br>D-10115 BERLIN<br><br><br><a href="http://www.m-box.de">www.m-box.de</a> <a href="http://www.monkeymen.tv">www.monkeymen.tv</a><br><br>Geschäftsführer: Martin Retschitzegger / Michaela Göllner<br>Handeslregister: Amtsgericht Charlottenburg / HRB 112767<o:p></o:p></p></div></div></blockquote></div></div></body></html>