<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Apr 10, 2017 at 3:06 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div class="gmail-h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 10, 2017 at 2:44 PM, Ondrej Svoboda <span dir="ltr"><<a href="mailto:osvoboda@redhat.com" target="_blank">osvoboda@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes, this is what struck me about your situation. Will you be able to find relevant logs regarding multipath configuration, in which we would see when (or even why) the third connection was created on the first node, and only one connection on the second?<br></div><div class="gmail-m_806424937102480534gmail-HOEnZb"><div class="gmail-m_806424937102480534gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 10, 2017 at 2:17 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span>On Mon, Apr 10, 2017 at 2:12 PM, Ondrej Svoboda <span dir="ltr"><<a href="mailto:osvoboda@redhat.com" target="_blank">osvoboda@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div>Gianluca,<br><br></div>I can see that the workaround you describe here (to complete multipath configuration in CLI) fixes an inconsistency in observed iSCSI sessions. I think it is a shortcoming in oVirt that you had to resort to manual configuration. Could you file a bug about this? Ideally, following the bug template presented to you by Bugzilla, i.e. "Expected: two iSCSI sessions", "Got: one the first node ... one the second node".<br><br></div>Edy, Martin, do you think you could help out here?<br><br></div>Thanks,<br></div>Ondra</div></blockquote><div><br></div></span><div>Ok, this evening I'm going to open a bugzilla for that.</div><div>Please keep in mind that on the already configured node (where before node addition there were two connections in place with multipath), actually the node addition generates a third connection, added to the existing two, using "default" as iSCSI interface (clearly seen if I run "iscsiadm -m session -P1") ....</div><span class="gmail-m_806424937102480534gmail-m_-2554033091313694766HOEnZb"><font color="#888888"><div><br></div><div>Gianluca</div><div> </div></font></span></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div><div class="gmail_extra">vdsm log of the already configured host is here for that day:</div><div class="gmail_extra"><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvQzdCUmtIT1NOT2c/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/<wbr>0BwoPbcrMv8mvQzdCUmtIT1NOT2c/<wbr>view?usp=sharing</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Installation / configuration of the second node happened between 11:30 AM and 01:30 PM of 6th of April.</div><div class="gmail_extra"><br></div><div class="gmail_extra">Aound 12:29 you will find:</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">2017-04-06 12:29:05,832+0200 INFO (jsonrpc/7) [dispatcher] Run and protect: getVGInfo, Return response: {'info': {'state': 'OK', 'vgsize': '1099108974592', 'name': '5ed04196-87f1-480e-9fee-<wbr>9dd450a3b53b', 'vgfree': '182536110080', 'vgUUID': 'rIENae-3NLj-o4t8-GVuJ-ZKKb-<wbr>ksTk-qBkMrE', 'pvlist': [{'vendorID': 'EQLOGIC', 'capacity': '1099108974592', 'fwrev': '0000', 'pe_alloc_count': '6829', 'vgUUID': 'rIENae-3NLj-o4t8-GVuJ-ZKKb-<wbr>ksTk-qBkMrE', 'pathlist': [{'connection': '10.10.100.9', 'iqn': 'iqn.2001-05.com.equallogic:4-<wbr>771816-e5d0dfb59-<wbr>1c9b240297958d53-ovsd3910', 'portal': '1', 'port': '3260', 'initiatorname': 'p1p1.100'}, {'connection': '10.10.100.9', 'iqn': 'iqn.2001-05.com.equallogic:4-<wbr>771816-e5d0dfb59-<wbr>1c9b240297958d53-ovsd3910', 'portal': '1', 'port': '3260', 'initiatorname': 'p1p2'}, {'connection': '10.10.100.9', 'iqn': 'iqn.2001-05.com.equallogic:4-<wbr>771816-e5d0dfb59-<wbr>1c9b240297958d53-ovsd3910', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'pe_count': '8189', 'discard_max_bytes': 15728640, 'pathstatus': [{'type': 'iSCSI', 'physdev': 'sde', 'capacity': '1099526307840', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdf', 'capacity': '1099526307840', 'state': 'active', 'lun': '0'}, {'type': 'iSCSI', 'physdev': 'sdg', 'capacity': '1099526307840', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'discard_zeroes_data': 1, 'pvUUID': 'g9pjI0-oifQ-kz2O-0Afy-xdnx-<wbr>THYD-eTWgqB', 'serial': 'SEQLOGIC_100E-00_<wbr>64817197B5DFD0E5538D959702249B<wbr>1C', 'GUID': '<wbr>364817197b5dfd0e5538d959702249<wbr>b1c', 'devcapacity': '1099526307840', 'productID': '100E-00'}], 'type': 3, 'attr': {'allocation': 'n', 'partial': '-', 'exported': '-', 'permission': 'w', 'clustered': '-', 'resizeable': 'z'}}} (logUtils:54)</div><div><br></div><div>and around 12:39 you will find </div><div><br></div><div><div>2017-04-06 12:39:11,003+0200 ERROR (check/loop) [storage.Monitor] Error checking path /dev/5ed04196-87f1-480e-9fee-<wbr>9dd450a3b53b/metadata (monitor:485)</div><div>Traceback (most recent call last):</div><div> File "/usr/share/vdsm/storage/<wbr>monitor.py", line 483, in _pathChecked</div><div> delay = result.delay()</div><div> File "/usr/lib/python2.7/site-<wbr>packages/vdsm/storage/check.<wbr>py", line 368, in delay</div><div> raise exception.<wbr>MiscFileReadException(self.<wbr>path, self.rc, self.err)</div><div>MiscFileReadException: Internal file read failure: ('/dev/5ed04196-87f1-480e-<wbr>9fee-9dd450a3b53b/metadata', 1, bytearray(b"/usr/bin/dd: error reading \'/dev/5ed04196-87f1-480e-<wbr>9fee-9dd450a3b53b/metadata\': Input/output error\n0+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.000234164 s, 0.0 kB/s\n"))</div><div>2017-04-06 12:39:11,020+0200 INFO (check/loop) [storage.Monitor] Domain 5ed04196-87f1-480e-9fee-<wbr>9dd450a3b53b became INVALID (monitor:456)</div></div><div><br></div><div>that I think corresponds to the moment when I executed "iscsiadm -m session -u" and had the automaic remediation of the correctly defined paths</div><span class="gmail-HOEnZb"><font color="#888888"><div><br></div><div>Gianluca</div></font></span></div></div>
</blockquote></div><br></div><div class="gmail_extra">So I come back here because I have an "orthogonal" action with the same effect.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I have already in place the same 2 oVirt hosts using one 4Tb iSCSI lun.</div><div class="gmail_extra">With the configuration detailed at beginning of thread, taht I resend:</div><div class="gmail_extra"><a href="http://lists.ovirt.org/pipermail/users/2017-March/080992.html" target="_blank" style="font-size:12.8px">http://lists.ovirt.org/<wbr>pipermail/users/2017-March/<wbr>080992.html</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">On them I have multipath access defined from inside oVirt like this at OS level</div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">[root@ov300 ~]# iscsiadm -m session -P 1</div><div class="gmail_extra">Target: iqn.2001-05.com.equallogic:4-771816-e5d0dfb59-1c9b240297958d53-ovsd3910 (non-flash)</div><div class="gmail_extra"><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.41:3260">10.10.100.41:3260</a>,1</div><div class="gmail_extra"><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.9:3260">10.10.100.9:3260</a>,1</div><div class="gmail_extra"><span style="white-space:pre"> </span>**********</div><div class="gmail_extra"><span style="white-space:pre"> </span>Interface:</div><div class="gmail_extra"><span style="white-space:pre"> </span>**********</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Name: p1p2</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Transport: tcp</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.88</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Netdev: p1p2</div><div class="gmail_extra"><span style="white-space:pre"> </span>SID: 1</div><div class="gmail_extra"><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div class="gmail_extra"><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div class="gmail_extra"><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div class="gmail_extra"><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.42:3260">10.10.100.42:3260</a>,1</div><div class="gmail_extra"><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.9:3260">10.10.100.9:3260</a>,1</div><div class="gmail_extra"><span style="white-space:pre"> </span>**********</div><div class="gmail_extra"><span style="white-space:pre"> </span>Interface:</div><div class="gmail_extra"><span style="white-space:pre"> </span>**********</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Name: p1p1.100</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Transport: tcp</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.87</div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div class="gmail_extra"><span style="white-space:pre"> </span>Iface Netdev: p1p1.100</div><div class="gmail_extra"><span style="white-space:pre"> </span>SID: 2</div><div class="gmail_extra"><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div class="gmail_extra"><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div class="gmail_extra"><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div class="gmail_extra"><br></div><div class="gmail_extra">So the network adapter used in the multipath config are p1p2 and p1p1.100</div><div class="gmail_extra"><br></div><div class="gmail_extra">Now I go and add a new storage domain: a 5Tb lun on another DELL EQLOGIC storage with ip address 10.10.100.7, so on the same lan as the existing one.</div><div class="gmail_extra"><br></div><div class="gmail_extra">I go through</div><div>Storage -> New Domain</div><div>discover targets with the ip</div><div>only one path is detected</div><div>I connect and the storage domain is activated</div><div><br></div><div>But on both oVirt hosts I have only one path for this LUN and at portal level the generic "default" Iface Name is used:</div><div><br></div><div>iscsiadm -m session -P1 gives the 2 connections above, plus</div><div><br></div><div> Target: iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 (non-flash)</div><div><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.38:3260">10.10.100.38:3260</a>,1</div><div><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Interface:</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Iface Name: default</div><div><span style="white-space:pre"> </span>Iface Transport: tcp</div><div><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.88</div><div><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div><span style="white-space:pre"> </span>Iface Netdev: <empty></div><div><span style="white-space:pre"> </span>SID: 3</div><div><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div><br></div><div><br></div><div>I can now retry to put host in maintenance and then activate and see if then it uses iscsi1 and iscsi2, but I proceed manually</div><div><br></div><div><div>From web gui</div><div>Datacenter --> MyDC</div><div>iSCSI Multipathing</div><div><br></div><div>iscsi1 -> edit</div><div>I see that only the 10.10.100.9 4Tb storage target is selected</div><div>--> I select the second one too (10.10.100.7) --> OK</div><div><br></div><div>No tasks seen in web gui but at os side I see added:</div><div><br></div><div><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.37:3260">10.10.100.37:3260</a>,1</div><div><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Interface:</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Iface Name: p1p1.100</div><div><span style="white-space:pre"> </span>Iface Transport: tcp</div><div><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.87</div><div><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div><span style="white-space:pre"> </span>Iface Netdev: p1p1.100</div><div><span style="white-space:pre"> </span>SID: 4</div><div><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div><br></div><div><br></div><div>The same for iscsi2 --> edit</div><div>only the 10.10.100.9 4Tb storage target is selected</div><div>--> I select the second one too (10.10.100.7) --> OK</div><div><br></div><div>Now I see for this lun a total of 3 connections (the initial "default" one and the iscsi1 and iscsi2 ones):</div><div><br></div><div>Target: iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 (non-flash)</div><div><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.38:3260">10.10.100.38:3260</a>,1</div><div><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Interface:</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Iface Name: default</div><div><span style="white-space:pre"> </span>Iface Transport: tcp</div><div><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.88</div><div><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div><span style="white-space:pre"> </span>Iface Netdev: <empty></div><div><span style="white-space:pre"> </span>SID: 3</div><div><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div><br></div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Interface:</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Iface Name: p1p2</div><div><span style="white-space:pre"> </span>Iface Transport: tcp</div><div><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.88</div><div><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div><span style="white-space:pre"> </span>Iface Netdev: p1p2</div><div><span style="white-space:pre"> </span>SID: 5</div><div><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div><span style="white-space:pre"> </span>Current Portal: <a href="http://10.10.100.37:3260">10.10.100.37:3260</a>,1</div><div><span style="white-space:pre"> </span>Persistent Portal: <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Interface:</div><div><span style="white-space:pre"> </span>**********</div><div><span style="white-space:pre"> </span>Iface Name: p1p1.100</div><div><span style="white-space:pre"> </span>Iface Transport: tcp</div><div><span style="white-space:pre"> </span>Iface Initiatorname: iqn.1994-05.com.redhat:f2d7fc1e2fc</div><div><span style="white-space:pre"> </span>Iface IPaddress: 10.10.100.87</div><div><span style="white-space:pre"> </span>Iface HWaddress: <empty></div><div><span style="white-space:pre"> </span>Iface Netdev: p1p1.100</div><div><span style="white-space:pre"> </span>SID: 4</div><div><span style="white-space:pre"> </span>iSCSI Connection State: LOGGED IN</div><div><span style="white-space:pre"> </span>iSCSI Session State: LOGGED_IN</div><div><span style="white-space:pre"> </span>Internal iscsid Session State: NO CHANGE</div><div><br></div><div><br></div><div>[root@ov300 ~]# iscsiadm -m session</div><div>tcp: [1] <a href="http://10.10.100.9:3260">10.10.100.9:3260</a>,1 iqn.2001-05.com.equallogic:4-771816-e5d0dfb59-1c9b240297958d53-ovsd3910 (non-flash)</div><div>tcp: [2] <a href="http://10.10.100.9:3260">10.10.100.9:3260</a>,1 iqn.2001-05.com.equallogic:4-771816-e5d0dfb59-1c9b240297958d53-ovsd3910 (non-flash)</div><div>tcp: [3] <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1 iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 (non-flash)</div><div>tcp: [4] <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1 iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 (non-flash)</div><div>tcp: [5] <a href="http://10.10.100.7:3260">10.10.100.7:3260</a>,1 iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750 (non-flash)</div><div>[root@ov300 ~]# </div><div><br></div><div>At the end I manually remove the wrong "default" one created by the web gui on both nodes:</div><div><br></div><div>[root@ov300 ~]# iscsiadm -m session -r 3 -u</div><div>Logging out of session [sid: 3, target: iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750, portal: 10.10.100.7,3260]</div><div>Logout of [sid: 3, target: iqn.2001-05.com.equallogic:0-8a0906-9d1c8500d-28e3c937b8d59521-ovsd3750, portal: 10.10.100.7,3260] successful.</div><div>[root@ov300 ~]# </div></div><div><br></div><div>Is this the expected workflow, or in case of multipathing configured at oVirt level, oVirt should have directly configured the new LUN in multipathed way?</div><div><br></div><div>I have still to verify if I put one host in maintenance and then I activate it that all goes as expected for both LUNs.</div><div><br></div><div>Thanks,</div><div>Gianluca</div><div><br></div><div><br></div></div></div>