<div dir="ltr">Actually I was seeing something similar yesterday and I have seen very sporadic failures in this area in the past. It seems that in some cases the VG is not properly set up before we try to use it (perhaps a race with devicemapper). I wonder if adding some sleeps in BlockStorageDomain.create() would mask the issue. If so, we need to figure out where the race is happening and might need to add some polling to make sure the underlying devices are ready.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Apr 6, 2017 at 2:35 PM, Liron Aravot <span dir="ltr"><<a href="mailto:laravot@redhat.com" target="_blank">laravot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks <span style="font-size:12.8px">Alexey,</span><div><span style="font-size:12.8px">Nir/Adam/Raz - any initial thoughts about that? have you encountered that issue before?</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">Regards,</span></div><div><span style="font-size:12.8px">Liron</span></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Apr 5, 2017 at 12:21 PM, Николаев Алексей <span dir="ltr"><<a href="mailto:alexeynikolaev.post@yandex.ru" target="_blank">alexeynikolaev.post@yandex.ru</a><wbr>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi Liron,</div><div> </div><div>Well, this error appear on freshly installed oVirt hosted engine <span>4.1.1.6-1 with oVirt Node NG 4.1.</span></div><div><span>There is no engine major version upgrade.</span></div><div><span>Node NG was updated from 4.1.0 to 4.1.1 by the </span>ovirt-node-ng-image* pkgs.</div><div> </div><div><span>vdsm version is:</span></div><div><div> </div><div>Installed Packages</div><div>Name : vdsm</div><div>Arch : x86_64</div><div>Version : 4.19.10</div><div>Release : 1.el7.centos</div><div>------------------------------<wbr>------------------------</div><div> </div><div>I also install oVirt 3.6 hosted engine on another server (CentOS 7.2 minimal, not updated to 7.3) and add the some ISCSI target as the data domain. This work OK.</div><div>After I update engine from 3.6 to 4.1. ISCSI data domain is OK after update. Another ISCSI targets was added as data domain without errors.</div><div> </div><div><div><span>vdsm version is:</span></div><div> </div><div>Installed Packages</div><div>Name : vdsm</div><div>Arch : noarch</div><div>Version : 4.17.32</div><div>Release : 0.el7.centos</div></div></div><div> </div><div>04.04.2017, 16:48, "Liron Aravot" <<a href="mailto:laravot@redhat.com" target="_blank">laravot@redhat.com</a>>:</div><div class="m_-7738959148593280048HOEnZb"><div class="m_-7738959148593280048h5"><blockquote type="cite"><div>Hi Alexey,<div> </div><div>Looking in your logs - it indeed look weird, the vg is created seconds later we fail to get it when attempting to create the domain.</div><div> </div><div>Did you upgrade/downgraded only the engine version or the vdsm version as well?</div><div> </div><div><div><span>2017-04-03 11</span>:16:39,595+0300 INFO (jsonrpc/4) [dispatcher] Run and protect: createVG(vgname=u'3050e121-83b<wbr>5-474e-943f-6dc52af38a17', devlist=[u'<span>3600140505</span>d11d22e61<wbr>f46a6ba4a</div><div>7bdc7'], force=False, options=None) (logUtils:51)</div><div> </div><div> </div><div>output error', ' /dev/mapper/<span>36001405</span>b7dd7b800<wbr>e7049ecbf<span>6830637</span>: read failed after 0 of 4096 at <span>4294967230464</span>: Input/output error', ' /dev/mapper/<span>36001405</span>b7dd7b800<wbr>e7049ecbf</div><div><span>6830637</span>: read failed after 0 of 4096 at <span>4294967287808</span>: Input/output error', ' WARNING: Error counts reached a limit of 3. Device /dev/mapper/<span>36001405</span>b7dd7b800e<wbr>7049ecbf68306</div><div>37 was disabled', ' Failed to find physical volume "/dev/mapper/<span>3600140505</span>d11d22e<wbr>61f46a6ba4a7bdc7".'] (lvm:323)</div><div><span>2017-04-03 11</span>:16:38,832+0300 WARN (jsonrpc/5) [storage.HSM] getPV failed for guid: <span>3600140505</span>d11d22e61f46a6ba4a7b<wbr>dc7 (hsm:1973)</div><div>Traceback (most recent call last):</div><div> File "/usr/share/vdsm/storage/hsm.p<wbr>y", line 1970, in _getDeviceList</div><div> pv = lvm.getPV(guid)</div><div> File "/usr/share/vdsm/storage/lvm.p<wbr>y", line 853, in getPV</div><div> raise se.((pvName,))</div></div><div> </div></div><div> <div>On Tue, Apr 4, 2017 at 3:33 PM, Николаев Алексей <span><<a href="mailto:alexeynikolaev.post@yandex.ru" target="_blank">alexeynikolaev.post@yandex.ru</a><wbr>></span> wrote:<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div>The last I was update oVirt oVirt <span>3.6.7.5-1 to </span>oVirt <span>4.1.1.6-1. Main ISCSI datadomain is OK. Another one ISCSI datadomain was added</span> successfully.</div><div> </div><div>The problem are only with freshly installed oVirt <span>4.1.1.6-1. </span></div><div> </div><div>Added logs after update ovirt and add new <span>ISCSI datadomains.</span></div><div> </div><div>03.04.2017, 17:26, "Николаев Алексей" <<a href="mailto:alexeynikolaev.post@yandex.ru" target="_blank">alexeynikolaev.post@yandex.ru</a><wbr>>:</div><div><div><blockquote type="cite"><div>Also I test again ISCSI TARGET with oVirt <span>3.6.7.5-1 and CentOS 7.2.1511 host. ISCSI data domain was added succesfully. "the truth is out there".</span></div><div><span>Add logs.</span></div><div> </div><div>03.04.2017, 11:24, "Николаев Алексей" <<a href="mailto:alexeynikolaev.post@yandex.ru" target="_blank">alexeynikolaev.post@yandex.ru</a><wbr>>:</div><blockquote type="cite"><div>Add logs. Thx!</div><div> </div><div> </div><div>02.04.2017, 16:19, "Liron Aravot" <<a href="mailto:laravot@redhat.com" target="_blank">laravot@redhat.com</a>>:</div><blockquote type="cite"><div>Hi Alexey,<div>can you please attach the engine/vdsm logs?</div><div> </div><div>thanks.</div></div><div> <div>On Fri, Mar 31, 2017 at 11:08 AM, Николаев Алексей <span><<a href="mailto:alexeynikolaev.post@yandex.ru" target="_blank">alexeynikolaev.post@yandex.ru</a><wbr>></span> wrote:<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Hi, community!</div><div> </div><div>I'm try using ISCSI DATA DOMAIN with oVirt <span>4.1.1.6-1 and </span>oVirt Node 4.1.1<span>. But get this error while adding data domain.</span></div><div> </div><div>Error while executing action New SAN Storage Domain: Cannot deactivate Logical Volume</div><div> </div><div><blockquote><div><span><span><span><span>2017-03-31 10</span></span></span></span>:47:45,099+03 ERROR [org.ovirt.engine.core.vdsbrok<wbr>er.vdsbroker.CreateStorageDoma<wbr>inVDSCommand] (default task-41) [717af17f] Command 'CreateStorageDomainVDSCommand<wbr>(HostName = node169-07., CreateStorageDomainVDSCommandP<wbr>arameters:{runAsync='true', hostId='ca177a14-56ef-4736-a0a<wbr>9-ab9dc2a8eb90', storageDomain='StorageDomainSt<wbr>atic:{name='data1', id='<span><span><span><span>56038842-2</span></span></span></span>fbe-4ada-96f1-c4<wbr>b4e66fd0b7'}', args='sUG5rL-zqvF-DMox-uMK3-Xl<wbr>91-ZiYw-k27C3B'})' execution failed: VDSGenericException: VDSErrorException: Failed in vdscommand to CreateStorageDomainVDS, error = Cannot deactivate Logical Volume: ('General Storage Exception: (\'5 [] [\\\' /dev/mapper/<span><span><span><span>36001405</span></span></span></span>b7dd7b800e<wbr>7049ecbf<span><span><span>6830637</span></span></span>: read failed after 0 of 4096 at 0: Input/output error\\\', \\\' /dev/mapper/<span><span><span><span>36001405</span></span></span></span>b7dd7b800e<wbr>7049ecbf<span><span><span>6830637</span></span></span>: read failed after 0 of 4096 at <span><span><span><span>4294967230464</span></span></span></span>: Input/output error\\\', \\\' /dev/mapper/<span><span><span><span>36001405</span></span></span></span>b7dd7b800e<wbr>7049ecbf<span><span><span>6830637</span></span></span>: read failed after 0 of 4096 at <span><span><span><span>4294967287808</span></span></span></span>: Input/output error\\\', \\\' WARNING: Error counts reached a limit of 3. Device /dev/mapper/<span><span><span><span>36001405</span></span></span></span>b7dd7b800e<wbr>7049ecbf<span><span><span>6830637</span></span></span> was disabled\\\', \\\' WARNING: Error counts reached a limit of 3. Device /dev/mapper/<span><span><span><span>36001405</span></span></span></span>b7dd7b800e<wbr>7049ecbf<span><span><span>6830637</span></span></span> was disabled\\\', \\\' Volume group "<span><span><span><span>56038842-2</span></span></span></span>fbe-4ada-96f1-c4b4e<wbr>66fd0b7" not found\\\', \\\' Cannot process volume group <span><span><span><span>56038842-2</span></span></span></span>fbe-4ada-96f1-c4b4e6<wbr>6fd0b7\\\']\\n<span><span><span><span>56038842</span></span></span></span>-2fbe-<wbr>4ada-96f1-c4b4e66fd0b7/[\\\'<wbr>master\\\']\',)',)</div></blockquote><div> </div><div>Previously, this ISCSI DATA DOMAIN works well with oVirt 3.6 and CentOS 7.2 host.</div><div>How I can debug this error?</div></div><br>______________________________<wbr>_________________<br>Users mailing list<br><a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a><br> </blockquote></div></div></blockquote>,<p>______________________________<wbr>_________________<br>Users mailing list<br><a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a></p></blockquote>,<p>______________________________<wbr>_________________<br>Users mailing list<br><a href="mailto:Users@ovirt.org" target="_blank">Users@ovirt.org</a><br><a href="http://lists.ovirt.org/mailman/listinfo/users" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/users</a></p></blockquote></div></div></blockquote></div></div></blockquote></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Adam Litke</div></div>
</div>