<br></div> <div class=3D"qtdSeparateBR"><br><br></div><div class=3D"yahoo_= quoted" style=3D"display: block;"> <div style=3D"font-family: Helvetica Neu= e, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 13px;"> <div sty= le=3D"font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida = Grande, sans-serif; font-size: 16px;"> <div dir=3D"ltr"><font size=3D"2" fa= ce=3D"Arial"> On Monday, July 3, 2017 3:01 PM, Yaniv Kaul <ykaul@redhat.= com> wrote:<br></font></div> <br><br> <div class=3D"y_msg_container"><d= iv id=3D"yiv5808088584"><div><div dir=3D"ltr"><br clear=3D"none"><div class= =3D"yiv5808088584gmail_extra"><br clear=3D"none"><div class=3D"yiv580808858= 4gmail_quote">On Mon, Jul 3, 2017 at 6:49 AM, M Mahboubian <span dir=3D"ltr= "><<a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:m_mahboubian@yah= oo.com" target=3D"_blank" href=3D"mailto:m_mahboubian@yahoo.com">m_mahboubi= an@yahoo.com</a>></span> wrote:<br clear=3D"none"><blockquote class=3D"y= iv5808088584gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc so=
Kernel 4.1? From where?</div><div> </div><blockquote class=3D"yiv5808= 088584gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa=
<div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725= 992_6548"><span id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_= 1499051725992_7076">So for example during shutdown also it froze and shows = something like this in event section:</span></div><div id=3D"yiv5808088584m= _-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6548"><br clear=3D"non= e"></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14= 99051725992_6548"><b id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym= 19_1_1499051725992_9195">VM ILMU_WEB has been paused due to storage I/O pro= blem.</b></div></div></div></div></div></div></blockquote><div><br clear=3D= "none"></div><div>We might need to get libvirt debug logs (and perhaps jour= nal output of the host).</div><div class=3D"yiv5808088584yqt9794303116" id= =3D"yiv5808088584yqtfd75759"><div>Y.</div><div> </div><blockquote clas= s=3D"yiv5808088584gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #= ccc solid;padding-left:1ex;"><div><div style=3D"color:#000;background-color= :#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-ser= if;font-size:13px;"><div id=3D"yiv5808088584m_-6379656960604602434yiv663901= 9017"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14990= 51725992_6544"><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960604602434yu= i_3_16_0_ym19_1_1499051725992_6546"><div id=3D"yiv5808088584m_-637965696060= 4602434yui_3_16_0_ym19_1_1499051725992_6548"><br clear=3D"none"></div><div = id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_65= 48"><br clear=3D"none"></div>More information:</div><div dir=3D"ltr" id=3D"= yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6546"><b= r clear=3D"none"></div><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960604= 602434yui_3_16_0_ym19_1_1499051725992_6546">VDSM log at the time of this is= sue (The issue happened at Jul 3, 2017 9:50:43 AM):</div><div dir=3D"ltr" i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_654= 6"><br clear=3D"none"></div><div dir=3D"ltr" id=3D"yiv5808088584m_-63796569= 60604602434yui_3_16_0_ym19_1_1499051725992_6546"><div dir=3D"ltr" id=3D"yiv= 5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_8436"><pre = id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_88= 75" style=3D"">2017-07-03 09:50:37,113+0800 INFO (jsonrpc/0) [jsonrpc.Json= RpcServer] RPC call Host.getAllVmStats succeeded in 0.00 seconds (__init__:=
------=_Part_3181214_2007753566.1499068839683 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi Yanis, Thank you for your reply.=C2=A0 | Interesting - what interface are they using?| Is that raw or raw sparse? = How did you perform the conversion? (or no conversion - just copied the dis= ks over?) The VM disks are in the SAN storage in order to use oVirt we just pointed t= hem to the oVirt VMs. This is how we did it precisely:First we created the = VMs in oVirt with disks which are the same size as the existing disks. then= we deleted these disks which was generated by oVirt and renamed our existi= ng disks to match the deleted ones naming-wise. Finally we started the oVir= t VMs and they were able to run and these VMs are always ok without any iss= ue. The new VMs which have problem are from scratch (no template). One thing th= ough, all these new VMs are created based on an CentOS 7 ISO. We have not t= ried any other flavor of Linux. The kernel 4.1 is actually from Oracle Linux repository since we needed to = have OCFS2 support. So after installing oVirt we updated the kernel to Orac= le Linux kernel 4.1 since that kernel supports OCFS2. | We might need to get libvirt debug logs (and perhaps journal output of th= e host). I'll get this information and post here. Regards =20 On Monday, July 3, 2017 3:01 PM, Yaniv Kaul <ykaul@redhat.com> wrote: =20 =20 On Mon, Jul 3, 2017 at 6:49 AM, M Mahboubian <m_mahboubian@yahoo.com> wrote= : Hi Yaniv, Thanks for your reply. Apologies for my late reply we had a long holiday he= re.=C2=A0 To answer you: Yes the =C2=A0guest VM become completely frozen and non responsive as soon = as its disk has any activity for example when we shutdown or do a yum updat= e.=C2=A0 Versions of all the components involved - guest OS, host OS (qemu-kvm versi= on), how do you run the VM (vdsm log would be helpful here), exact storage = specification (1Gb or 10Gb link? What is the NFS version? What is it hosted= on? etc.)=C2=A0Y. Some facts about our environment: 1) Previously, this environment was using XEN using raw disk and we change = it to Ovirt (Ovirt were able to read the VMs's disks without any conversion= .)=C2=A0 Interesting - what interface are they using?Is that raw or raw sparse? How = did you perform the conversion? (or no conversion - just copied the disks o= ver?)=C2=A0 2) The issue we are facing is not happening for any of the existing VMs.=C2= =A03) This issue only happens for new VMs. New VMs from blank, or from a template (as a snapshot over the previous VMs= ) ?=C2=A0 4) Guest (kernel v3.10) and host(kernel v4.1) OSes are both CentOS 7 minima= l installation.=C2=A0 Kernel 4.1? From where?=C2=A0 5) NFS version 4 and Using Ovirt 4.16)=C2=A0The network speed is 1 GB. That might be very slow (but should not cause such an issue, unless severel= y overloaded.=C2=A0 7)=C2=A0The output for rpm -qa | grep qemu-kvm shows:=C2=A0 =C2=A0 =C2=A0qe= mu-kvm-common-ev-2.6.0-28. e17_3.6.1.x86_64 =C2=A0 =C2=A0 =C2=A0qemu-kvm-tools-ev-2.6.0-28. e17_3.6.1.x86_64=C2=A0 =C2= =A0 =C2=A0qemu-kvm-ev-2.6.0-28.e17_3.6. 1.x86_64 That's good - that's almost the latest-greatest.=C2=A0 8)=C2=A0The storage is from a SAN device which is connected to the NFS serv= er using fiber channel. So for example during shutdown also it froze and shows something like this = in event section: VM ILMU_WEB has been paused due to storage I/O problem. We might need to get libvirt debug logs (and perhaps journal output of the = host).Y.=C2=A0 More information: VDSM log at the time of this issue (The issue happened at Jul 3, 2017 9:50:= 43 AM): 2017-07-03 09:50:37,113+0800 INFO (jsonrpc/0) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:515) 2017-07-03 09:50:37,897+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmIoTunePolicies succeeded in 0.02 seconds (__init__:515) 2017-07-03 09:50:42,510+0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:515) 2017-07-03 09:50:43,548+0800 INFO (jsonrpc/3) [dispatcher] Run and protect= : repoStats(options=3DNone) (logUtils:51) 2017-07-03 09:50:43,548+0800 INFO (jsonrpc/3) [dispatcher] Run and protect= : repoStats, Return response: {u'e01186c1-7e44-4808-b551- 4722f0f8e84b': {'= code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00014= 4822', 'lastCheck': '8.9', 'valid': True}, u'721b5233-b0ba-4722-8a7d- ba2a3= 72190a0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'dela= y': '0.000327909', 'lastCheck': '8.9', 'valid': True}, u'94775bd3-3244-45b4= -8a06- 37eff8856afa': {'code': 0, 'actual': True, 'version': 4, 'acquired':= True, 'delay': '0.000256425', 'lastCheck': '8.9', 'valid': True}, u'731bb7= 71-5b73-4b5c-ac46- 56499df97721': {'code': 0, 'actual': True, 'version': 0,= 'acquired': True, 'delay': '0.000238159', 'lastCheck': '8.9', 'valid': Tru= e}, u'f620781f-93d4-4410-8697- eb41045cacd6': {'code': 0, 'actual': True, '= version': 4, 'acquired': True, 'delay': '0.00022004', 'lastCheck': '8.9', '= valid': True}, u'a1a7d0a4-e3b6-4bd5-862b- 96e70dae3f29': {'code': 0, 'actua= l': True, 'version': 0, 'acquired': True, 'delay': '0.000298581', 'lastChec= k': '8.8', 'valid': True}} (logUtils:54) 2017-07-03 09:50:43,563+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] RPC = call Host.getStats succeeded in 0.01 seconds (__init__:515) 2017-07-03 09:50:46,737+0800 INFO (periodic/3) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' 721b5233-b0ba-4722-8a7d- ba2a372190a0', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'3c26476e-1dae-44d7= - 9208-531b91ae5ae1', volUUID=3Du'a7e789fb-6646-4d0a- 9b51-f5ab8242c8d5', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,738+0800 INFO (periodic/0) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' f620781f-93d4-4410-8697- eb41045cacd6', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'2158fdae-54e1-413d= - a844-73da5d1bb4ca', volUUID=3Du'6ee0b0eb-0bba-4e18- 9c00-c1539b632e8a', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,740+0800 INFO (periodic/2) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' f620781f-93d4-4410-8697- eb41045cacd6', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'a967016d-a56b-41e8= - b7a2-57903cbd2825', volUUID=3Du'784514cb-2b33-431c- b193-045f23c596d8', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,741+0800 INFO (periodic/1) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' 721b5233-b0ba-4722-8a7d- ba2a372190a0', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'bb35c163-f068-4f08= - a1c2-28c4cb1b76d9', volUUID=3Du'fce7e0a0-7411-4d8c- b72c-2f46c4b4db1e', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,743+0800 INFO (periodic/0) [dispatcher] Run and protec= t: getVolumeSize, Return response: {'truesize': '6361276416', 'apparentsize= ': '107374182400'} (logUtils:54) ............ 2017-07-03 09:52:16,941+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop device scsi0-0-0-0 err= or eio (vm:4112) 2017-07-03 09:52:16,941+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError (vm:4997) 2017-07-03 09:52:16,942+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onSuspend (vm:4997) 2017-07-03 09:52:16,942+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop device scsi0-0-0-0 err= or eio (vm:4112) 2017-07-03 09:52:16,943+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError (vm:4997) 2017-07-03 09:52:16,943+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop device scsi0-0-0-0 err= or eio (vm:4112) 2017-07-03 09:52:16,944+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError On Thursday, June 22, 2017, 2:48 PM, Yaniv Kaul <ykaul@redhat.com> wrote: On Thu, Jun 22, 2017 at 5:07 AM, M Mahboubian <m_mahboubian@yahoo.com> wrot= e: Dear all,I appreciate if anybody could possibly help with the issue I am fa= cing. In our environment we have 2 hosts 1 NFS server and 1 ovirt engine server. = The NFS server provides storage to the VMs in the hosts. I can create new VMs and install os but once i do something like yum update= the VM freezes. I can reproduce this every single time I do yum update. Is it paused, or completely frozen?=C2=A0 what information/log files should I provide you to trubleshoot this? Versions of all the components involved - guest OS, host OS (qemu-kvm versi= on), how do you run the VM (vdsm log would be helpful here), exact storage = specification (1Gb or 10Gb link? What is the NFS version? What is it hosted= on? etc.)=C2=A0Y. =C2=A0Regards ______________________________ _________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/ mailman/listinfo/users =20 ------=_Part_3181214_2007753566.1499068839683 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <html><head></head><body><div style=3D"color:#000; background-color:#fff; f= ont-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font= -size:13px"><div id=3D"yui_3_16_0_ym19_1_1499067397139_3359"><span>Hi Yanis= ,</span></div><div id=3D"yui_3_16_0_ym19_1_1499067397139_3360"><span><br></= span></div><div id=3D"yui_3_16_0_ym19_1_1499067397139_3361" dir=3D"ltr"><sp= an>Thank you for your reply. </span></div><div id=3D"yui_3_16_0_ym19_1= _1499067397139_3361" dir=3D"ltr"><span><br></span></div><div style=3D"font-= family: "Helvetica Neue", "Segoe UI", Helvetica, Arial,= "Lucida Grande", sans-serif;" id=3D"yui_3_16_0_ym19_1_1499067397= 139_3420">| Interesting - what interface are they using?</div><div style=3D= "font-family: "Helvetica Neue", "Segoe UI", Helvetica, = Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0= _ym19_1_1499067397139_3421">| Is that raw or raw sparse? How did you perfor= m the conversion? (or no conversion - just copied the disks over?)</div><di= v style=3D"font-family: "Helvetica Neue", "Segoe UI", H= elvetica, Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"= yui_3_16_0_ym19_1_1499067397139_3421"><br></div><div style=3D"font-family: = "Helvetica Neue", "Segoe UI", Helvetica, Arial, "L= ucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_149906= 7397139_3421">The VM disks are in the SAN storage in order to use oVirt we = just pointed them to the oVirt VMs. This is how we did it precisely:</div><= div style=3D"font-family: "Helvetica Neue", "Segoe UI",= Helvetica, Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" id= =3D"yui_3_16_0_ym19_1_1499067397139_3421">First we created the VMs in oVirt= with disks which are the same size as the existing disks. then we deleted = these disks which was generated by oVirt and renamed our existing disks to = match the deleted ones naming-wise. Finally we started the oVirt VMs and th= ey were able to run and these VMs are always ok without any issue.</div><di= v style=3D"font-family: "Helvetica Neue", "Segoe UI", H= elvetica, Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"= yui_3_16_0_ym19_1_1499067397139_3421"><br></div><div style=3D"font-family: = "Helvetica Neue", "Segoe UI", Helvetica, Arial, "L= ucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_149906= 7397139_3421">The new VMs which have problem are from scratch (no template)= . One thing though, all these new VMs are created based on an CentOS 7 ISO.= We have not tried any other flavor of Linux.</div><div style=3D"font-famil= y: "Helvetica Neue", "Segoe UI", Helvetica, Arial, &quo= t;Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_149= 9067397139_3421"><br></div><div style=3D"font-family: "Helvetica Neue&= quot;, "Segoe UI", Helvetica, Arial, "Lucida Grande", s= ans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421">The ker= nel 4.1 is actually from Oracle Linux repository since we needed to have OC= FS2 support. So after installing oVirt we updated the kernel to Oracle Linu= x kernel 4.1 since that kernel supports OCFS2.</div><div style=3D"font-fami= ly: "Helvetica Neue", "Segoe UI", Helvetica, Arial, &qu= ot;Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_14= 99067397139_3421"><br></div><div style=3D"font-family: "Helvetica Neue= ", "Segoe UI", Helvetica, Arial, "Lucida Grande", = sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421">| We m= ight need to get libvirt debug logs (and perhaps journal output of the host= ).<br></div><div style=3D"font-family: "Helvetica Neue", "Se= goe UI", Helvetica, Arial, "Lucida Grande", sans-serif;" dir= =3D"ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421"><br></div><div style= =3D"font-family: "Helvetica Neue", "Segoe UI", Helvetic= a, Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_1= 6_0_ym19_1_1499067397139_3421">I'll get this information and post here.</di= v><div style=3D"font-family: "Helvetica Neue", "Segoe UI&quo= t;, Helvetica, Arial, "Lucida Grande", sans-serif;" dir=3D"ltr" i= d=3D"yui_3_16_0_ym19_1_1499067397139_3421"><br></div><div style=3D"font-fam= ily: "Helvetica Neue", "Segoe UI", Helvetica, Arial, &q= uot;Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1= 499067397139_3421">Regards</div><div style=3D"font-family: "Helvetica = Neue", "Segoe UI", Helvetica, Arial, "Lucida Grande&quo= t;, sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421"><b= r></div><div style=3D"font-family: "Helvetica Neue", "Segoe = UI", Helvetica, Arial, "Lucida Grande", sans-serif;" dir=3D"= ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421"><br></div><div style=3D"fo= nt-family: "Helvetica Neue", "Segoe UI", Helvetica, Ari= al, "Lucida Grande", sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym= 19_1_1499067397139_3421"><br></div><div style=3D"font-family: "Helveti= ca Neue", "Segoe UI", Helvetica, Arial, "Lucida Grande&= quot;, sans-serif;" dir=3D"ltr" id=3D"yui_3_16_0_ym19_1_1499067397139_3421"= lid;padding-left:1ex;"><div><div style=3D"color:#000;background-color:#fff;= font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;fon= t-size:13px;"><div id=3D"yiv5808088584m_-6379656960604602434yiv6639019017">= <div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14990517259= 92_6544"> Hi Yaniv,<div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14= 99051725992_6543"><br clear=3D"none"></div><div id=3D"yiv5808088584m_-63796= 56960604602434yui_3_16_0_ym19_1_1499051725992_6566">Thanks for your reply. = Apologies for my late reply we had a long holiday here. </div><div id= =3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6565= "><br clear=3D"none"></div><div id=3D"yiv5808088584m_-6379656960604602434yu= i_3_16_0_ym19_1_1499051725992_6564">To answer you:</div><div id=3D"yiv58080= 88584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6563"><br clear= =3D"none"></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym= 19_1_1499051725992_6557">Yes the guest VM become completely frozen an= d non responsive as soon as its disk has any activity for example when we s= hutdown or do a yum update. </div><div id=3D"yiv5808088584m_-637965696= 0604602434yui_3_16_0_ym19_1_1499051725992_6556"><br clear=3D"none"></div><d= iv id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992= _6555"><br clear=3D"none"></div><div dir=3D"ltr" id=3D"yiv5808088584m_-6379= 656960604602434yui_3_16_0_ym19_1_1499051725992_6546"><div id=3D"yiv58080885= 84m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6554"><span id=3D"y= iv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6553">Ver= sions of all the components involved - guest OS, host OS (qemu-kvm version)= , how do you run the VM (vdsm log would be helpful here), exact storage spe= cification (1Gb or 10Gb link? What is the NFS version? What is it hosted on= ? etc.)</span></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_= 0_ym19_1_1499051725992_6552"><span> Y.</span></div><div id=3D"yiv58080= 88584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6551"><span><br = clear=3D"none"></span></div><div id=3D"yiv5808088584m_-6379656960604602434y= ui_3_16_0_ym19_1_1499051725992_6551"><span id=3D"yiv5808088584m_-6379656960= 604602434yui_3_16_0_ym19_1_1499051725992_9556">Some facts about our environ= ment:</span></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_= ym19_1_1499051725992_6551"><span><br clear=3D"none"></span></div><div id=3D= "yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6551"><= span id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14990517259= 92_9703">1) Previously, this environment was using XEN using raw disk and w= e change it to Ovirt (Ovirt were able to read the VMs's disks without any c= onversion.) </span></div></div></div></div></div></div></blockquote><d= iv><br clear=3D"none"></div><div>Interesting - what interface are they usin= g?</div><div>Is that raw or raw sparse? How did you perform the conversion?= (or no conversion - just copied the disks over?)</div><div> </div><bl= ockquote class=3D"yiv5808088584gmail_quote" style=3D"margin:0 0 0 .8ex;bord= er-left:1px #ccc solid;padding-left:1ex;"><div><div style=3D"color:#000;bac= kground-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Gra= nde, sans-serif;font-size:13px;"><div id=3D"yiv5808088584m_-637965696060460= 2434yiv6639019017"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0= _ym19_1_1499051725992_6544"><div dir=3D"ltr" id=3D"yiv5808088584m_-63796569= 60604602434yui_3_16_0_ym19_1_1499051725992_6546"><div id=3D"yiv5808088584m_= -6379656960604602434yui_3_16_0_ym19_1_1499051725992_6551"><span id=3D"yiv58= 08088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_9800">2) The = issue we are facing is not happening for any of the existing VMs. </sp= an></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14= 99051725992_6551"><span id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0= _ym19_1_1499051725992_9686"><b id=3D"yiv5808088584m_-6379656960604602434yui= _3_16_0_ym19_1_1499051725992_9685">3) This issue only happens for new VMs.<= /b></span></div></div></div></div></div></div></blockquote><div><br clear= =3D"none"></div><div>New VMs from blank, or from a template (as a snapshot = over the previous VMs) ?</div><div> </div><blockquote class=3D"yiv5808= 088584gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;pa= dding-left:1ex;"><div><div style=3D"color:#000;background-color:#fff;font-f= amily:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size= :13px;"><div id=3D"yiv5808088584m_-6379656960604602434yiv6639019017"><div i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_654= 4"><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym1= 9_1_1499051725992_6546"><div id=3D"yiv5808088584m_-6379656960604602434yui_3= _16_0_ym19_1_1499051725992_6551"><span>4) G</span>uest (kernel v3.10) and h= ost(kernel v4.1) OSes are both CentOS 7 minimal installation. </div></= div></div></div></div></div></blockquote><div><br clear=3D"none"></div><div= dding-left:1ex;"><div><div style=3D"color:#000;background-color:#fff;font-f= amily:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size= :13px;"><div id=3D"yiv5808088584m_-6379656960604602434yiv6639019017"><div i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_654= 4"><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym1= 9_1_1499051725992_6546"><div id=3D"yiv5808088584m_-6379656960604602434yui_3= _16_0_ym19_1_1499051725992_6551"><b id=3D"yiv5808088584m_-63796569606046024= 34yui_3_16_0_ym19_1_1499051725992_9591">5) NFS version 4</b> and Using Ovir= t 4.1</div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_= 1499051725992_6550"><span>6) </span><b>The network speed is 1 GB.</b><= /div></div></div></div></div></div></blockquote><div><br clear=3D"none"></d= iv><div>That might be very slow (but should not cause such an issue, unless= severely overloaded. </div><div><br clear=3D"none"></div><blockquote = class=3D"yiv5808088584gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1= px #ccc solid;padding-left:1ex;"><div><div style=3D"color:#000;background-c= olor:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grande, sans= -serif;font-size:13px;"><div id=3D"yiv5808088584m_-6379656960604602434yiv66= 39019017"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1= 499051725992_6544"><div dir=3D"ltr" id=3D"yiv5808088584m_-63796569606046024= 34yui_3_16_0_ym19_1_1499051725992_6546"><div id=3D"yiv5808088584m_-63796569= 60604602434yui_3_16_0_ym19_1_1499051725992_6550"><span>7) </span>The o= utput for rpm -qa | grep qemu-kvm shows:</div><div id=3D"yiv5808088584m_-63= 79656960604602434yui_3_16_0_ym19_1_1499051725992_6550"><b id=3D"yiv58080885= 84m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_9222"> = qemu-kvm-common-ev-2.6.0-28. e17_3.6.1.x86_64</b><br clear=3D"none">= </div><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_= ym19_1_1499051725992_6550"><b id=3D"yiv5808088584m_-6379656960604602434yui_= 3_16_0_ym19_1_1499051725992_9360"><span id=3D"yiv5808088584m_-6379656960604= 602434yui_3_16_0_ym19_1_1499051725992_8120"> qemu-kvm-to= ols-ev-</span>2.6.0-28. e17_3.6.1.x86_64</b></div><div dir=3D"ltr" id=3D"yi= v5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6550"><b i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_935= 9"> qemu-kvm-ev-2.6.0-28.e17_3.6. 1.x86_64</b></div></di= v></div></div></div></div></blockquote><div><br clear=3D"none"></div><div>T= hat's good - that's almost the latest-greatest.</div><div> </div><bloc= kquote class=3D"yiv5808088584gmail_quote" style=3D"margin:0 0 0 .8ex;border= -left:1px #ccc solid;padding-left:1ex;"><div><div style=3D"color:#000;backg= round-color:#fff;font-family:Helvetica Neue, Helvetica, Arial, Lucida Grand= e, sans-serif;font-size:13px;"><div id=3D"yiv5808088584m_-63796569606046024= 34yiv6639019017"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_y= m19_1_1499051725992_6544"><div dir=3D"ltr" id=3D"yiv5808088584m_-6379656960= 604602434yui_3_16_0_ym19_1_1499051725992_6546"><div dir=3D"ltr" id=3D"yiv58= 08088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6550"><b>8)&n= bsp;</b>The storage is from a SAN device which is connected to the NFS serv= er using fiber channel.</div><div id=3D"yiv5808088584m_-6379656960604602434= yui_3_16_0_ym19_1_1499051725992_6548"><span><br clear=3D"none"></span></div= 515) 2017-07-03 09:50:37,897+0800 INFO (jsonrpc/1) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmIoTunePolicies succeeded in 0.02 seconds (__init__:515) 2017-07-03 09:50:42,510+0800 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer] RPC = call Host.getAllVmStats succeeded in 0.00 seconds (__init__:515) <b id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992= _9241">2017-07-03 09:50:43,548+0800 INFO (jsonrpc/3) [dispatcher] Run and = protect: repoStats(options=3DNone) (logUtils:51) 2017-07-03 09:50:43,548+0800 INFO (jsonrpc/3) [dispatcher] Run and protect= : repoStats, Return response: {u'e01186c1-7e44-4808-b551- 4722f0f8e84b': {'= code': 0, 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.00014= 4822', 'lastCheck': '8.9', 'valid': True}, u'721b5233-b0ba-4722-8a7d- ba2a3= 72190a0': {'code': 0, 'actual': True, 'version': 4, 'acquired': True, 'dela= y': '0.000327909', 'lastCheck': '8.9', 'valid': True}, u'94775bd3-3244-45b4= -8a06- 37eff8856afa': {'code': 0, 'actual': True, 'version': 4, 'acquired':= True, 'delay': '0.000256425', 'lastCheck': '8.9', 'valid': True}, u'731bb7= 71-5b73-4b5c-ac46- 56499df97721': {'code': 0, 'actual': True, 'version': 0,= 'acquired': True, 'delay': '0.000238159', 'lastCheck': '8.9', 'valid': Tru= e}, u'f620781f-93d4-4410-8697- eb41045cacd6': {'code': 0, 'actual': True, '= version': 4, 'acquired': True, 'delay': '0.00022004', 'lastCheck': '8.9', '= valid': True}, u'a1a7d0a4-e3b6-4bd5-862b- 96e70dae3f29': {'code': 0, 'actua= l': True, 'version': 0, 'acquired': True, 'delay': '0.000298581', 'lastChec= k': '8.8', 'valid': True}} (logUtils:54) </b>2017-07-03 09:50:43,563+0800 INFO (jsonrpc/3) [jsonrpc.JsonRpcServer] = RPC call Host.getStats succeeded in 0.01 seconds (__init__:515) 2017-07-03 09:50:46,737+0800 INFO (periodic/3) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' 721b5233-b0ba-4722-8a7d- ba2a372190a0', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'3c26476e-1dae-44d7= - 9208-531b91ae5ae1', volUUID=3Du'a7e789fb-6646-4d0a- 9b51-f5ab8242c8d5', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,738+0800 INFO (periodic/0) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' f620781f-93d4-4410-8697- eb41045cacd6', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'2158fdae-54e1-413d= - a844-73da5d1bb4ca', volUUID=3Du'6ee0b0eb-0bba-4e18- 9c00-c1539b632e8a', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,740+0800 INFO (periodic/2) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' f620781f-93d4-4410-8697- eb41045cacd6', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'a967016d-a56b-41e8= - b7a2-57903cbd2825', volUUID=3Du'784514cb-2b33-431c- b193-045f23c596d8', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,741+0800 INFO (periodic/1) [dispatcher] Run and protec= t: getVolumeSize(sdUUID=3Du' 721b5233-b0ba-4722-8a7d- ba2a372190a0', spUUID= =3Du'b04ca6e4-2660-4eaa- acdb-c1dae4e21f2d', imgUUID=3Du'bb35c163-f068-4f08= - a1c2-28c4cb1b76d9', volUUID=3Du'fce7e0a0-7411-4d8c- b72c-2f46c4b4db1e', o= ptions=3DNone) (logUtils:51) 2017-07-03 09:50:46,743+0800 INFO (periodic/0) [dispatcher] Run and protec= t: getVolumeSize, Return response: {'truesize': '6361276416', 'apparentsize= ': '107374182400'} (logUtils:54)</pre><pre id=3D"yiv5808088584m_-6379656960= 604602434yui_3_16_0_ym19_1_1499051725992_8875" style=3D""><br clear=3D"none= "></pre><pre id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_149= 9051725992_8875" style=3D"">......</pre><pre id=3D"yiv5808088584m_-63796569= 60604602434yui_3_16_0_ym19_1_1499051725992_8875" style=3D"">......</pre><pr= e id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_= 8875" style=3D""><br clear=3D"none"></pre><pre id=3D"yiv5808088584m_-637965= 6960604602434yui_3_16_0_ym19_1_1499051725992_8875" style=3D""></pre><pre id= =3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_9311= " style=3D""><b id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_= 1499051725992_9341">2017-07-03 09:52:16,941+0800 INFO (libvirt/events) [vi= rt.vm] (vmId=3D'c84f519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop de= vice scsi0-0-0-0 error eio (vm:4112) 2017-07-03 09:52:16,941+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError (vm:4997) 2017-07-03 09:52:16,942+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onSuspend (vm:4997) 2017-07-03 09:52:16,942+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop device scsi0-0-0-0 err= or eio (vm:4112) 2017-07-03 09:52:16,943+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError (vm:4997) 2017-07-03 09:52:16,943+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') abnormal vm stop device scsi0-0-0-0 err= or eio (vm:4112) 2017-07-03 09:52:16,944+0800 INFO (libvirt/events) [virt.vm] (vmId=3D'c84f= 519e-398d-40a3- 85b2-b7e53f3d7f67') CPU stopped: onIOError</b></pre><pre id= =3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_9311= " style=3D""><b><br clear=3D"none"></b></pre><pre id=3D"yiv5808088584m_-637= 9656960604602434yui_3_16_0_ym19_1_1499051725992_9311" style=3D""><b><br cle= ar=3D"none"></b></pre><pre id=3D"yiv5808088584m_-6379656960604602434yui_3_1= 6_0_ym19_1_1499051725992_9311" style=3D""><b><br clear=3D"none"></b></pre><= /div><div class=3D"yiv5808088584m_-6379656960604602434qtdSeparateBR"><br cl= ear=3D"none"><br clear=3D"none"></div><div class=3D"yiv5808088584m_-6379656= 960604602434yiv6639019017yqt2677119843" id=3D"yiv5808088584m_-6379656960604= 602434yiv6639019017yqtfd82856"><div class=3D"yiv5808088584m_-63796569606046= 02434yiv6639019017yahoo-quoted-begin" id=3D"yiv5808088584m_-637965696060460= 2434yui_3_16_0_ym19_1_1499051725992_6545" style=3D"font-size:15px;color:#71= 5ffa;padding-top:15px;margin-top:0;"><b id=3D"yiv5808088584m_-6379656960604= 602434yui_3_16_0_ym19_1_1499051725992_9344">On Thursday, June 22, 2</b>017,= 2:48 PM, Yaniv Kaul <<a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mail= to:ykaul@redhat.com" target=3D"_blank" href=3D"mailto:ykaul@redhat.com">yka= ul@redhat.com</a>> wrote:</div><blockquote class=3D"yiv5808088584m_-6379= 656960604602434yiv6639019017iosymail" id=3D"yiv5808088584m_-637965696060460= 2434yui_3_16_0_ym19_1_1499051725992_6642"><div id=3D"yiv5808088584m_-637965= 6960604602434yiv6639019017"><div id=3D"yiv5808088584m_-6379656960604602434y= ui_3_16_0_ym19_1_1499051725992_6641"><div dir=3D"ltr" id=3D"yiv5808088584m_= -6379656960604602434yui_3_16_0_ym19_1_1499051725992_6640"><br clear=3D"none= "><div class=3D"yiv5808088584m_-6379656960604602434yiv6639019017gmail_extra= " id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_= 6643"><br clear=3D"none"><div class=3D"yiv5808088584m_-6379656960604602434y= iv6639019017gmail_quote" id=3D"yiv5808088584m_-6379656960604602434yui_3_16_= 0_ym19_1_1499051725992_6644">On Thu, Jun 22, 2017 at 5:07 AM, M Mahboubian = <span dir=3D"ltr"><<a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:= m_mahboubian@yahoo.com" target=3D"_blank" href=3D"mailto:m_mahboubian@yahoo= .com">m_mahboubian@yahoo.com</a>></span> wrote:<br clear=3D"none"><block= quote class=3D"yiv5808088584m_-6379656960604602434yiv6639019017gmail_quote"= id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6= 646" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,20= 4);padding-left:1ex;"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_1= 6_0_ym19_1_1499051725992_6645"> Dear all,<div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14= 99051725992_6647">I appreciate if anybody could possibly help with the issu= e I am facing.</div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_= 0_ym19_1_1499051725992_6648"><br clear=3D"none"></div><div id=3D"yiv5808088= 584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6649">In our envir= onment we have 2 hosts 1 NFS server and 1 ovirt engine server. The NFS serv= er provides storage to the VMs in the hosts.</div><div id=3D"yiv5808088584m= _-6379656960604602434yui_3_16_0_ym19_1_1499051725992_6650"><br clear=3D"non= e"></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_14= 99051725992_6651">I can create new VMs and install os but once i do somethi= ng like yum update the VM freezes. I can reproduce this every single time I= do yum update.</div></div></blockquote><div id=3D"yiv5808088584m_-63796569= 60604602434yui_3_16_0_ym19_1_1499051725992_6652"><br clear=3D"none"></div><= div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_149905172599= 2_6653">Is it paused, or completely frozen?</div><div id=3D"yiv5808088584m_= -6379656960604602434yui_3_16_0_ym19_1_1499051725992_6654"> <br clear= =3D"none"></div><blockquote class=3D"yiv5808088584m_-6379656960604602434yiv= 6639019017gmail_quote" id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_= ym19_1_1499051725992_9059" style=3D"margin:0px 0px 0px 0.8ex;border-left:1p= x solid rgb(204,204,204);padding-left:1ex;"><div id=3D"yiv5808088584m_-6379= 656960604602434yui_3_16_0_ym19_1_1499051725992_9058"><div id=3D"yiv58080885= 84m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_9057"><br clear=3D"= none"></div><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1= _1499051725992_9791">what information/log files should I provide you to tru= bleshoot this?</div></div></blockquote><div><br clear=3D"none"></div><div i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_983= 4">Versions of all the components involved - guest OS, host OS (qemu-kvm ve= rsion), how do you run the VM (vdsm log would be helpful here), exact stora= ge specification (1Gb or 10Gb link? What is the NFS version? What is it hos= ted on? etc.)</div><div> Y.</div><div class=3D"yiv5808088584m_-6379656= 960604602434yiv6639019017yqt6606837936" id=3D"yiv5808088584m_-6379656960604= 602434yiv6639019017yqtfd75418"><div><br clear=3D"none"></div></div><blockqu= ote class=3D"yiv5808088584m_-6379656960604602434yiv6639019017gmail_quote" i= d=3D"yiv5808088584m_-6379656960604602434yui_3_16_0_ym19_1_1499051725992_979= 4" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)= ;padding-left:1ex;"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_16_= 0_ym19_1_1499051725992_9793"><div class=3D"yiv5808088584m_-6379656960604602= 434yiv6639019017yqt6606837936" id=3D"yiv5808088584m_-6379656960604602434yiv= 6639019017yqtfd33806"><div id=3D"yiv5808088584m_-6379656960604602434yui_3_1= 6_0_ym19_1_1499051725992_9792"><br clear=3D"none"></div><div> Regards<= /div></div> </div><br clear=3D"none">______________________________ _________________<b= r clear=3D"none"> Users mailing list<br clear=3D"none"> <a rel=3D"nofollow" shape=3D"rect" ymailto=3D"mailto:Users@ovirt.org" targe= t=3D"_blank" href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br clear= =3D"none"> <a rel=3D"nofollow" shape=3D"rect" target=3D"_blank" href=3D"http://lists.o= virt.org/mailman/listinfo/users">http://lists.ovirt.org/ mailman/listinfo/u= sers</a><div class=3D"yiv5808088584m_-6379656960604602434yiv6639019017yqt66= 06837936" id=3D"yiv5808088584m_-6379656960604602434yiv6639019017yqtfd65242"=
<br clear=3D"none"> <br clear=3D"none"></div></blockquote></div><div class=3D"yiv5808088584m_-6= 379656960604602434yiv6639019017yqt6606837936" id=3D"yiv5808088584m_-6379656= 960604602434yiv6639019017yqtfd02968"><br clear=3D"none"></div></div></div><= /div></div><blockquote></blockquote></blockquote></div></div><div class=3D"= yiv5808088584m_-6379656960604602434yiv6639019017yqt2677119843" id=3D"yiv580= 8088584m_-6379656960604602434yiv6639019017yqtfd23176"> </div></div></div></div></div></blockquote></div></div><div class=3D"yiv580= 8088584yqt9794303116" id=3D"yiv5808088584yqtfd09260"><br clear=3D"none"></d= iv></div></div></div></div><br><br></div> </div> </div> </div></div></bod= y></html> ------=_Part_3181214_2007753566.1499068839683--