Re: [ovirt-users] vm has been paused due to unknown storage

------=_Part_4068378_74557332.1496222176801 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi, I found the cause of this problem. I had to turn off sharding. De: suporte(a)logicworks.pt Para: "Sahina Bose" <sabose(a)redhat.com&gt; Cc: "ovirt users" <users(a)ovirt.org&gt; Enviadas: Sexta-feira, 26 De Maio de 2017 12:27:43 Assunto: Re: [ovirt-users] vm has been paused due to unknown storage Hi, I updated glusterfs: glusterfs-client-xlators-3.8.12-1.el7.x86_64 glusterfs-cli-3.8.12-1.el7.x86_64 glusterfs-api-3.8.12-1.el7.x86_64 glusterfs-fuse-3.8.12-1.el7.x86_64 glusterfs-server-3.8.12-1.el7.x86_64 glusterfs-libs-3.8.12-1.el7.x86_64 glusterfs-3.8.12-1.el7.x86_64 Now I cannot add a volume disk preallocated, after a while it breaks. message log: May 26 11:18:16 node journal: vdsm root ERROR VM metrics collection failed#012Traceback (most recent call last):#012 File "/usr/lib/python2.7/site-packages/vdsm/virt/vmstats.py", line 221, in send_metrics#012 diskinfo['readOps']#012KeyError: 'readOps' vdsm.log 2017-05-26 11:18:16,715+0100 ERROR (periodic/3) [root] VM metrics collection failed (vmstats:264) 2017-05-26 11:19:39,369+0100 ERROR (tasks/5) [storage.Volume] Unexpected error (fileVolume:456) 2017-05-26 11:19:39,373+0100 ERROR (tasks/5) [storage.Volume] Unexpected error (volume:1107) 2017-05-26 11:19:39,374+0100 ERROR (tasks/5) [storage.TaskManager.Task] (Task='5b2adb9a-e24e-48fa-9f01-f21c23588aef') Unexpected error (task:870) glusterfs [2017-05-26 10:53:08.247219] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.164 (00000000-0000-0000-0000-000000000000) [No data available] [2017-05-26 10:53:14.899499] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.167 (00000000-0000-0000-0000-000000000000) [No data available] [2017-05-26 10:53:14.899526] E [MSGID: 133010] [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard 167 failed. Base file gfid = 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available] [2017-05-26 10:53:19.712567] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.169 (00000000-0000-0000-0000-000000000000) [No data available] [2017-05-26 10:53:19.712614] E [MSGID: 133010] [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard 169 failed. Base file gfid = 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available] [2017-05-26 10:53:29.419317] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.173 (00000000-0000-0000-0000-000000000000) [No data available] [2017-05-26 10:53:29.419369] E [MSGID: 133010] [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard 173 failed. Base file gfid = 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available] thanks De: "Sahina Bose" <sabose(a)redhat.com&gt; Para: suporte(a)logicworks.pt, "Krutika Dhananjay" <kdhananj(a)redhat.com&gt; Cc: "ovirt users" <users(a)ovirt.org&gt; Enviadas: Quinta-feira, 25 De Maio de 2017 7:12:40 Assunto: Re: [ovirt-users] vm has been paused due to unknown storage The glusterfs logs contain below errors: [2017-05-22 18:12:50.941883] E [MSGID: 133010] [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard 50 failed. Base file gfid = 33f1fe3e-c626-49f2-861e-2259c972931d [No data available] [2017-05-22 18:12:50.945085] W [fuse-bridge.c:1291:fuse_err_cbk] 0-glusterfs-fuse: 61306713: FSYNC() ERR => -1 (No data available) Krutika, could you take a look? On Thu, May 25, 2017 at 1:02 AM, < suporte(a)logicworks.pt > wrote: Hi, I setup an ovirt hosted enine, in only one server with local gluster bricks. When running a MS SQL 2012 process to rebuild a data base, which take around 4 hours, after a while the VM is paused with the error: vm has been paused due to unknown storage The VM disk is in Thin provision Ovirt and gluter versions: Version 4.1.1.8-1.el7.centos glusterfs-cli-3.8.11-1.el7.x86_64 glusterfs-libs-3.8.11-1.el7.x86_64 glusterfs-3.8.11-1.el7.x86_64 glusterfs-client-xlators-3.8.11-1.el7.x86_64 glusterfs-fuse-3.8.11-1.el7.x86_64 glusterfs-api-3.8.11-1.el7.x86_64 glusterfs-server-3.8.11-1.el7.x86_64 I can find the reason why The logs are attached. Any idea? Thanks -- Jose Ferradeira http://www.logicworks.pt _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users ------=_Part_4068378_74557332.1496222176801 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: Times New Roman; font-size: 10pt; co= lor: #000000"><div>Hi,<br></div><div><br data-mce-bogus=3D"1"></div><div>I = found the cause of this problem. I had to turn off sharding.<br data-mce-bo= gus=3D"1"></div><div><br></div><hr id=3D"zwchr" data-marker=3D"__DIVIDER__"= ra: </b>"Sahina Bose" &amp;lt;sabose(a)redhat.com&amp;gt;&lt;br&gt;&lt;b&gt;Cc: </b>"ovirt users"= &amp;lt;users(a)ovirt.org&amp;gt;&lt;br&gt;&lt;b&gt;Enviadas: </b>Sexta-feira, 26 De Maio de 201= 7 12:27:43<br><b>Assunto: </b>Re: [ovirt-users] vm has been paused due to u= nknown storage<br></div><div><br></div><div data-marker=3D"__QUOTED_TEXT__"= 0;" data-mce-style=3D"font-family: Times New Roman; font-size: 10pt; color:= #000000;"><div>Hi,</div><br><div>I updated glusterfs:</div><div>glusterfs-= client-xlators-3.8.12-1.el7.x86_64<br>glusterfs-cli-3.8.12-1.el7.x86_64<br>= glusterfs-api-3.8.12-1.el7.x86_64<br>glusterfs-fuse-3.8.12-1.el7.x86_64<br>= glusterfs-server-3.8.12-1.el7.x86_64<br>glusterfs-libs-3.8.12-1.el7.x86_64<= br>glusterfs-3.8.12-1.el7.x86_64</div><br><div>Now I cannot add a volume di= sk preallocated, after a while it breaks.</div><br><div>message log:</div><= div>May 26 11:18:16 node journal: vdsm root ERROR VM metrics collection fai= led#012Traceback (most recent call last):#012 &nbsp;File "/usr/lib/python2.= 7/site-packages/vdsm/virt/vmstats.py", line 221, in send_metrics#012 &nbsp;= &nbsp;diskinfo['readOps']#012KeyError: 'readOps'</div><br><div>vdsm.log<br= on failed (vmstats:264)<br>2017-05-26 11:19:39,369+0100 ERROR (tasks/5) [st= orage.Volume] Unexpected error (fileVolume:456)<br>2017-05-26 11:19:39,373+= 0100 ERROR (tasks/5) [storage.Volume] Unexpected error (volume:1107)<br>201= 7-05-26 11:19:39,374+0100 ERROR (tasks/5) [storage.TaskManager.Task] (Task= =3D'5b2adb9a-e24e-48fa-9f01-f21c23588aef') Unexpected error (task:870)</div= nt-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-client-0: remote operation f= ailed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e251ac6f5e.164 (00000000-000= 0-0000-0000-000000000000) [No data available]<br>[2017-05-26 10:53:14.89949= 9] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gv2-cl= ient-0: remote operation failed. Path: /.shard/55b94942-dee5-4f69-8b0f-52e2= 51ac6f5e.167 (00000000-0000-0000-0000-000000000000) [No data available]<br>= [2017-05-26 10:53:14.899526] E [MSGID: 133010] [shard.c:1725:shard_common_l= ookup_shards_cbk] 0-gv2-shard: Lookup on shard 167 failed. Base file gfid = =3D 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available]<br>[2017-05-26= 10:53:19.712567] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_looku= p_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard/55b94942-dee5= -4f69-8b0f-52e251ac6f5e.169 (00000000-0000-0000-0000-000000000000) [No data= available]<br>[2017-05-26 10:53:19.712614] E [MSGID: 133010] [shard.c:1725= :shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard 169 failed. B= ase file gfid =3D 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No data available]<= br>[2017-05-26 10:53:29.419317] W [MSGID: 114031] [client-rpc-fops.c:2933:c= lient3_3_lookup_cbk] 0-gv2-client-0: remote operation failed. Path: /.shard= /55b94942-dee5-4f69-8b0f-52e251ac6f5e.173 (00000000-0000-0000-0000-00000000= 0000) [No data available]<br>[2017-05-26 10:53:29.419369] E [MSGID: 133010]= [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup on shard= 173 failed. Base file gfid =3D 55b94942-dee5-4f69-8b0f-52e251ac6f5e [No da= ta available]<br></div><br><br><div>thanks</div><br><hr id=3D"zwchr"><div><= b>De: </b>"Sahina Bose" &amp;lt;sabose(a)redhat.com&amp;gt;&lt;br&gt;&lt;b&gt;Para: </b>suporte@l= ogicworks.pt, "Krutika Dhananjay" &amp;lt;kdhananj(a)redhat.com&amp;gt;&lt;br&gt;&lt;b&gt;Cc: </b= De Maio de 2017 7:12:40<br><b>Assunto: </b>Re: [ovirt-users] vm has been p= aused due to unknown storage<br></div><br><div><div dir=3D"ltr"><div>The gl= usterfs logs contain below errors:<br>[2017-05-22 18:12:50.941883] E [MSGID= : 133010] [shard.c:1725:shard_common_lookup_shards_cbk] 0-gv2-shard: Lookup= on shard 50 failed. Base file gfid =3D 33f1fe3e-c626-49f2-861e-2259c972931= d [No data available]<br>[2017-05-22 18:12:50.945085] W [fuse-bridge.c:1291= :fuse_err_cbk] 0-glusterfs-fuse: 61306713: FSYNC() ERR =3D&gt; -1 (No data = available)<br><br></div>Krutika, could you take a look?<br></div><div class= =3D"gmail_extra"><br><div class=3D"gmail_quote">On Thu, May 25, 2017 at 1:0= 2 AM, <span dir=3D"ltr">&lt;<a href=3D"mailto:suporte@logicworks.pt" target= =3D"_blank" data-mce-href=3D"mailto:suporte@logicworks.pt">suporte@logicwor= ks.pt</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"m= argin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;" data-mc= e-style=3D"margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1= ex;"><div><div style=3D"font-family: Times New Roman; font-size: 10pt; colo= r: #000000;" data-mce-style=3D"font-family: Times New Roman; font-size: 10p= t; color: #000000;"><div>Hi,<br></div><br><div>I setup an ovirt hosted enin= e, in only one server with local gluster bricks.<br></div><br><div>When run= ning a MS SQL 2012 process to rebuild a data base, which take around 4 hour= s, after a while the VM is paused with the error:<br></div><div><pre>vm has= been paused due to unknown storage<br><br>The VM disk is in Thin provision Ovirt and gluter versions: Version 4.1.1.8-1.el7.centos glusterfs-cli-3.8.11-1.el7.x86_64 glusterfs-libs-3.8.11-1.el7.x86_64 glusterfs-3.8.11-1.el7.x86_64 glusterfs-client-xlators-3.8.11-1.el7.x86_64 glusterfs-fuse-3.8.11-1.el7.x86_64 glusterfs-api-3.8.11-1.el7.x86_64 glusterfs-server-3.8.11-1.el7.x86_64</pre></div><br><div>I can find the rea= son why<br></div><div>The logs are attached.<br></div><br><div>Any idea?<br= " data-mce-style=3D"color: #888888;"><br></span></span></div><span class=3D= "HOEnZb"><span class=3D"HOEnZb"><span style=3D"color: #888888;" data-mce-st= yle=3D"color: #888888;"><br></span></span></span><div>-- <br></div><div><hr= style=3D"width: 100%; height: 2px;" data-mce-style=3D"width: 100%; height:= 2px;">Jose Ferradeira<br><a href=3D"http://www.logicworks.pt" target=3D"_b= lank" data-mce-href=3D"http://www.logicworks.pt">http://www.logicworks.pt</= a><br></div></div></div><br>_______________________________________________= <br> Users mailing list<br> <a href=3D"mailto:Users@ovirt.org" target=3D"_b= lank" data-mce-href=3D"mailto:Users@ovirt.org">Users@ovirt.org</a><br> <a h= ref=3D"http://lists.ovirt.org/mailman/listinfo/users" rel=3D"noreferrer" ta= rget=3D"_blank" data-mce-href=3D"http://lists.ovirt.org/mailman/listinfo/us= ers">http://lists.ovirt.org/mailman/listinfo/users</a><b... <br></blockquote= _____<br>Users mailing list<br>Users@ovirt.org<br>http://lists.ovirt.org/ma= ilman/listinfo/users<br></div></div></body></html> ------=_Part_4068378_74557332.1496222176801--