Re: [ovirt-users] Some VMs in status "not responding" in oVirt interface

This is a multipart message in MIME format. ------=_NextPart_000_0479_01D0EFFC.E11214C0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Hi Markus, =20 gdb is available on CentOS 7, but what do you mean by qemu-debug? I = Installed qemu-kvm-tools, maybe this is the pendant for CentOS? =20 qemu-kvm-tools.x86_64 : KVM debugging and diagnostics tools qemu-kvm-tools-ev.x86_64 : KVM debugging and diagnostics tools qemu-kvm-tools-rhev.x86_64 : KVM debugging and diagnostics tools =20 Regards, Christian =20 Von: Markus Stockhausen [mailto:stockhausen@collogia.de]=20 Gesendet: Dienstag, 15. September 2015 20:40 An: Daniel Helgenberger <daniel.helgenberger(a)m-box.de&gt; Cc: Christian Hailer <christian(a)hailer.eu>; ydary(a)redhat.com; = users(a)ovirt.org Betreff: Re: [ovirt-users] Some VMs in status "not responding" in oVirt = interface =20 Do you have a chance to install qemu-debug? If yes I would try a = backtrace. gdb -p <qemu-pid> # bt Markus Am 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger = <daniel.helgenberger(a)m-box.de <mailto:daniel.helgenberger@m-box.de> >: Hello, I do not want to hijack the thread but maybe my issue is related? It might have started with ovirt 3.5.3; but I cannot tell for sure. For me, one vm (foreman) is affected; the second time in 14 days. I can = confirm this as I also loose any network connection to the VM and the ability to connect a console. Also, the only thing witch 'fixes' the issue is right now 'kill -9 <pid = of qemu-kvm process>' As far as I can tell the VM became unresponsive at around Sep 15 = 12:30:01; engine logged this at 12:34. Nothing obvious in VDSM logs (see attached). Below the engine.log part. Versions: ovirt-engine-3.5.4.2-1.el7.centos.noarch vdsm-4.16.26-0.el7.centos libvirt-1.2.8-16.el7_1.3 engine.log (1200 - 1300: 2015-09-15 12:03:47,949 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-56) [264d502a] HA reservation status for cluster Default is OK 2015-09-15 12:08:02,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] = (DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to update VMs/Templates Ovf. 2015-09-15 12:08:02,709 INFO = [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] = (DefaultQuartzScheduler_Worker-89) [5e9f4ba6] Running command: ProcessOvfUpdateForStoragePoolCommand = internal: true. Entities affected : ID: 00000002-0002-0002-0002-000000000088 Type: l 2015-09-15 12:08:02,780 INFO = [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] = (DefaultQuartzScheduler_Worker-89) [5e9f4ba6] Lock freed to object EngineLock [exclusiveLocks=3D key: = 00000002-0002-0002-0002-000000000088 value: OVF_UPDATE 2015-09-15 12:08:47,997 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA reservation status for cluster Default is OK 2015-09-15 12:13:06,998 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] = (org.ovirt.thread.pool-8-thread-48) [50221cdc] START, GetFileStatsVDSCommand( storagePoolId =3D = 00000002-0002-0002-0002-000000000088, ignoreFailoverLimit =3D false), = log id: 1503968 2015-09-15 12:13:07,137 INFO = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] = (org.ovirt.thread.pool-8-thread-48) [50221cdc] FINISH, GetFileStatsVDSCommand, return: = {pfSense-2.0-RELEASE-i386.iso=3D{status=3D0, ctime=3D1432286887.0, = size=3D115709952}, Fedora-15-i686-Live8 2015-09-15 12:13:07,178 INFO = [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] = (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished automatic refresh process for ISO file type with success, for storage = domain id 84dcb2fc-fb63-442f-aa77-3e84dc7d5a72. 2015-09-15 12:13:48,043 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA reservation status for cluster Default is OK 2015-09-15 12:18:48,088 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-44) [6345e698] HA reservation status for cluster Default is OK 2015-09-15 12:23:48,137 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-13) HA reservation status for cluster Default is OK 2015-09-15 12:28:48,183 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-76) [154c91d5] HA reservation status for cluster Default is OK 2015-09-15 12:33:48,229 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA reservation status for cluster Default is OK 2015-09-15 12:34:49,432 INFO = [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] = (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM foreman 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --> = NotResponding 2015-09-15 12:34:49,578 WARN = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler_Worker-41) [5f2a4b68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, = Message: VM foreman is not responding. 2015-09-15 12:38:48,273 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-10) [7a800766] HA reservation status for cluster Default is OK 2015-09-15 12:43:48,320 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-42) [440f1c40] HA reservation status for cluster Default is OK 2015-09-15 12:48:48,366 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-70) HA reservation status for cluster Default is OK 2015-09-15 12:53:48,412 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-12) [50221cdc] HA reservation status for cluster Default is OK 2015-09-15 12:58:48,459 INFO = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-3) HA reservation status for cluster Default is OK On 29.08.2015 22:48, Christian Hailer wrote: single Centos=20 engine=20 server=20 restarted the=20 hours later=20 in the=20 responding=E2=80=9D, marked by a=20 I tried to reset=20 from vm_static=20 hypervisor helped=E2=80=A6=20 of the=20 again only a reboot helped.=20 Call Stack:=20 greatly=20 --=20 Daniel Helgenberger m box bewegtbild GmbH P: +49/30/2408781-22 F: +49/30/2408781-10 ACKERSTR. 19 D-10115 BERLIN www.m-box.de <http://www.m-box.de> www.monkeymen.tv = <http://www.monkeymen.tv>=20 Gesch=C3=A4ftsf=C3=BChrer: Martin Retschitzegger / Michaela G=C3=B6llner Handeslregister: Amtsgericht Charlottenburg / HRB 112767 ------=_NextPart_000_0479_01D0EFFC.E11214C0 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" = xmlns:o=3D"urn:schemas-microsoft-com:office:office" = xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" = xmlns=3D"http://www.w3.org/TR/REC-html40"><head><meta = http-equiv=3DContent-Type content=3D"text/html; charset=3Dutf-8"><meta = name=3DGenerator content=3D"Microsoft Word 15 (filtered = medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0cm; margin-bottom:.0001pt; font-size:12.0pt; font-family:"Times New Roman",serif;} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} p {mso-style-priority:99; mso-margin-top-alt:auto; margin-right:0cm; mso-margin-bottom-alt:auto; margin-left:0cm; font-size:12.0pt; font-family:"Times New Roman",serif;} span.E-MailFormatvorlage18 {mso-style-type:personal; font-family:"Calibri",sans-serif; color:#1F497D;} span.E-MailFormatvorlage19 {mso-style-type:personal-compose; font-family:"Calibri",sans-serif; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-size:10.0pt;} @page WordSection1 {size:612.0pt 792.0pt; margin:70.85pt 70.85pt 2.0cm 70.85pt;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DDE link=3Dblue = vlink=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>Hi Markus,<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>gdb is available on CentOS 7, but what do = you mean by qemu-debug? I Installed qemu-kvm-tools, maybe this is the = pendant for CentOS?<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>qemu-kvm-tools.x86_64 : KVM debugging and = diagnostics tools<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>qemu-kvm-tools-ev.x86_64 : KVM debugging and = diagnostics tools<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>qemu-kvm-tools-rhev.x86_64 : KVM debugging = and diagnostics tools<o:p></o:p></span></p><p class=3DMsoNormal><span = lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'>Regards, Christian<o:p></o:p></span></p><p = class=3DMsoNormal><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;= mso-fareast-language:EN-US'><o:p>&nbsp;</o:p></span></p><div><div = style=3D'border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm = 0cm 0cm'><p class=3DMsoNormal><b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>Von:</span></= b><span lang=3DEN-US = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'> Markus = Stockhausen [mailto:stockhausen@collogia.de] = <br><b>Gesend</b></span><b><span = style=3D'font-size:11.0pt;font-family:"Calibri",sans-serif'>et:</span></b= Dienstag, 15. September 2015 20:40<br><b>An:</b> Daniel Helgenberger = &lt;daniel.helgenberger@m-box.de&gt;<br><b>Cc:</b> Christian Hailer = &amp;lt;christian(a)hailer.eu&amp;gt;; ydary(a)redhat.com; = users@ovirt.org<br><b>Betreff:</b> Re: [ovirt-users] Some VMs in status = &quot;not responding&quot; in oVirt = interface<o:p></o:p></span></p></div></div><p = class=3DMsoNormal><o:p>&nbsp;</o:p></p><p>Do you have a chance to = install qemu-debug? If yes I would try a backtrace.<o:p></o:p></p><p>gdb = -p &lt;qemu-pid&gt;<br># = bt<o:p></o:p></p><p>Markus<o:p></o:p></p><div><p class=3DMsoNormal>Am = 15.09.2015 4:15 nachm. schrieb Daniel Helgenberger &lt;<a = href=3D"mailto:daniel.helgenberger@m-box.de">daniel.helgenberger@m-box.de= </a>&gt;:<o:p></o:p></p><blockquote = style=3D'border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm = 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p = class=3DMsoNormal>Hello,<br><br>I do not want to hijack the thread but = maybe my issue is related?<br><br>It might have started with ovirt = 3.5.3; but I cannot tell for sure.<br><br>For me, one vm (foreman) is = affected; the second time in 14 days. I can confirm this as I also loose = any network connection to the VM and<br>the ability to connect a = console.<br>Also, the only thing witch 'fixes' the issue is right now = 'kill -9 &lt;pid of qemu-kvm process&gt;'<br><br>As far as I can tell = the VM became unresponsive at around Sep 15 12:30:01; engine logged this = at 12:34. Nothing obvious in VDSM logs (see<br>attached).<br><br>Below = the engine.log = part.<br><br>Versions:<br>ovirt-engine-3.5.4.2-1.el7.centos.noarch<br><br= (1200 - 1300:<br>2015-09-15 12:03:47,949 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-56) [264d502a] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:08:02,708 INFO&nbsp; = [org.ovirt.engine.core.bll.OvfDataUpdater] = (DefaultQuartzScheduler_Worker-89) [2e7bf56e] Attempting to = update<br>VMs/Templates Ovf.<br>2015-09-15 12:08:02,709 INFO&nbsp; = [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] = (DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Running command: = ProcessOvfUpdateForStoragePoolCommand internal: true. Entities affected = :&nbsp; ID:<br>00000002-0002-0002-0002-000000000088 Type: = l<br>2015-09-15 12:08:02,780 INFO&nbsp; = [org.ovirt.engine.core.bll.ProcessOvfUpdateForStoragePoolCommand] = (DefaultQuartzScheduler_Worker-89)<br>[5e9f4ba6] Lock freed to object = EngineLock [exclusiveLocks=3D key: 00000002-0002-0002-0002-000000000088 = value: OVF_UPDATE<br>2015-09-15 12:08:47,997 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-21) [3fc854a2] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:13:06,998 INFO&nbsp; = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] = (org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] START, = GetFileStatsVDSCommand( storagePoolId =3D = 00000002-0002-0002-0002-000000000088, ignoreFailoverLimit =3D false), = log id: 1503968<br>2015-09-15 12:13:07,137 INFO&nbsp; = [org.ovirt.engine.core.vdsbroker.vdsbroker.GetFileStatsVDSCommand] = (org.ovirt.thread.pool-8-thread-48)<br>[50221cdc] FINISH, = GetFileStatsVDSCommand, return: = {pfSense-2.0-RELEASE-i386.iso=3D{status=3D0, ctime=3D1432286887.0, = size=3D115709952},<br>Fedora-15-i686-Live8<br>2015-09-15 12:13:07,178 = INFO&nbsp; [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] = (org.ovirt.thread.pool-8-thread-48) [50221cdc] Finished<br>automatic = refresh process for ISO file type with success, for storage domain id = 84dcb2fc-fb63-442f-aa77-3e84dc7d5a72.<br>2015-09-15 12:13:48,043 = INFO&nbsp; [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-87) [4fa1bb16] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:18:48,088 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-44) [6345e698] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:23:48,137 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-13) HA reservation<br>status for cluster = Default is OK<br>2015-09-15 12:28:48,183 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-76) [154c91d5] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:33:48,229 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-36) [27c73ac6] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:34:49,432 INFO&nbsp; = [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] = (DefaultQuartzScheduler_Worker-41) [5f2a4b68] VM<br>foreman = 8b57ff1d-2800-48ad-b267-fd8e9e2f6fb2 moved from Up --&gt; = NotResponding<br>2015-09-15 12:34:49,578 WARN&nbsp; = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = (DefaultQuartzScheduler_Worker-41)<br>[5f2a4b68] Correlation ID: null, = Call Stack: null, Custom Event ID: -1, Message: VM foreman is not = responding.<br>2015-09-15 12:38:48,273 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-10) [7a800766] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:43:48,320 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-42) [440f1c40] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:48:48,366 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-70) HA reservation<br>status for cluster = Default is OK<br>2015-09-15 12:53:48,412 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-12) [50221cdc] HA<br>reservation status = for cluster Default is OK<br>2015-09-15 12:58:48,459 INFO&nbsp; = [org.ovirt.engine.core.bll.scheduling.HaReservationHandling] = (DefaultQuartzScheduler_Worker-3) HA reservation<br>status for cluster = Default is OK<br><br><br><br>On 29.08.2015 22:48, Christian Hailer = wrote:<br>&gt; Hello,<br>&gt; <br>&gt; last Wednesday I wanted to update = my oVirt 3.5 hypervisor. It is a single Centos <br>&gt; 7 server, so I = started by suspending the VMs in order to set the oVirt engine <br>&gt; = host to maintenance mode. During the process of suspending the VMs the = server <br>&gt; crashed, kernel panic=E2=80=A6<br>&gt; <br>&gt; After = restarting the server I installed the updates via yum an restarted the = <br>&gt; server again. Afterwards, all the VMs could be started again. = Some hours later <br>&gt; my monitoring system registered some = unresponsive hosts, I had a look in the <br>&gt; oVirt interface, 3 of = the VMs were in the state =E2=80=9Cnot responding=E2=80=9D, marked by a = <br>&gt; question mark.<br>&gt; <br>&gt; I tried to shut down the VMs, = but oVirt wasn=E2=80=99t able to do so. I tried to reset <br>&gt; the = status in the database with the sql statement<br>&gt; <br>&gt; update = vm_dynamic set status =3D 0 where vm_guid =3D (select vm_guid from = vm_static <br>&gt; where vm_name =3D 'MYVMNAME');<br>&gt; <br>&gt; but = that didn=E2=80=99t help, either. Only rebooting the whole hypervisor = helped=E2=80=A6 <br>&gt; afterwards everything worked again. But only = for a few hours, then one of the <br>&gt; VMs entered the =E2=80=9Cnot = responding=E2=80=9D state again=E2=80=A6 again only a reboot helped. = <br>&gt; Yesterday it happened again:<br>&gt; <br>&gt; 2015-08-28 = 17:44:22,664 INFO&nbsp; <br>&gt; = [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] <br>&gt; = (DefaultQuartzScheduler_Worker-60) [4ef90b12] VM DC <br>&gt; = 0f3d1f06-e516-48ce-aa6f-7273c33d3491 moved from Up --&gt; = NotResponding<br>&gt; <br>&gt; 2015-08-28 17:44:22,692 WARN&nbsp; = <br>&gt; = [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] = <br>&gt; (DefaultQuartzScheduler_Worker-60) [4ef90b12] Correlation ID: = null, Call Stack: <br>&gt; null, Custom Event ID: -1, Message: VM DC is = not responding.<br>&gt; <br>&gt; Does anybody know what I can do? Where = should I have a look? Hints are greatly <br>&gt; appreciated!<br>&gt; = <br>&gt; Thanks,<br>&gt; <br>&gt; Christian<br>&gt; <br><br>-- = <br>Daniel Helgenberger<br>m box bewegtbild GmbH<br><br>P: = +49/30/2408781-22<br>F: +49/30/2408781-10<br><br>ACKERSTR. 19<br>D-10115 = BERLIN<br><br><br><a href=3D"http://www.m-box.de">www.m-box.de</a>&nbsp; = <a = href=3D"http://www.monkeymen.tv">www.monkeymen.tv</a><br><br>Gesch=C3=A4f= tsf=C3=BChrer: Martin Retschitzegger / Michaela = G=C3=B6llner<br>Handeslregister: Amtsgericht Charlottenburg / HRB = 112767<o:p></o:p></p></div></div></blockquote></div></div></body></html> ------=_NextPart_000_0479_01D0EFFC.E11214C0--