<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Nov 4, 2014 at 6:34 PM, Arman Khalatyan <span dir="ltr"><<a href="mailto:arm2arm@gmail.com" target="_blank">arm2arm@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><p>I will he I.teresting to see your iscsi setup with drbd. Did you got splitbrain before failure?<br>
Did you check if your target went to readonly mode? <br>
Thanks <br><span class=""><font color="#888888">
Arman.</font></span></p><div class=""><div class="h5">
<div class="gmail_quote"><br></div></div></div></blockquote><div><br></div><div> I used some information provided here, even if it is with CentOS 5.7 and lvm on top of drbd, while in my setup I have CentOS 6.5 and drbd on top of lvm:</div><div><br></div><div><div><a href="http://blogs.mindspew-age.com/2012/04/05/adventures-in-high-availability-ha-iscsi-with-drbd-iscsi-and-pacemaker/">http://blogs.mindspew-age.com/2012/04/05/adventures-in-high-availability-ha-iscsi-with-drbd-iscsi-and-pacemaker/</a></div></div><div><br></div><div>- my drbd resource definition for iSCSI HA:</div><div><div>[root@srvmgmt01 ~]# cat iscsiha.res</div><div>resource iscsiha {</div><div> disk {</div><div> disk-flushes no;</div><div> md-flushes no;</div><div> fencing resource-and-stonith;</div><div> }</div><div> device minor 2;</div><div> disk /dev/iscsihavg/iscsihalv;</div><div> syncer {</div><div> rate 30M;</div><div> verify-alg md5;</div><div> }</div><div> handlers {</div><div> fence-peer "/usr/lib/drbd/crm-fence-peer.sh";</div><div> after-resync-target "/usr/lib/drbd/crm-unfence-peer.sh";</div><div> }</div><div> on srvmgmt01.localdomain.local {</div><div> address <a href="http://192.168.230.51:7790">192.168.230.51:7790</a>;</div><div> meta-disk internal;</div><div> }</div><div> on srvmgmt02.localdomain.local {</div><div> address <a href="http://192.168.230.52:7790">192.168.230.52:7790</a>;</div><div> meta-disk internal;</div><div> }</div><div>}</div></div><div><br></div><div><br></div><div>- tgtd is setup to start on both nodes at startup</div><div>iscsi and iscsid services configured to off</div><div><br></div><div><div>- Put the agents iSCSILogicalUnit e iSCSITarget under </div><div> /usr/lib/ocf/resource.d/heartbeat/ on both nodes</div><div><br></div><div>downloaded from here, as they are not provided in plain CentOS:</div><div><a href="http://linux-ha.org/doc/man-pages/re-ra-iSCSITarget.html">http://linux-ha.org/doc/man-pages/re-ra-iSCSITarget.html</a><br></div></div><div><br></div><div>- Here below the pcs steps to create the group:</div><div><br></div><div><div>pcs cluster cib iscsiha_cfg</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_drbd_iscsiha ocf:linbit:drbd drbd_resource=iscsiha \</div><div>op monitor interval="29s" role="Master" timeout="30" op monitor interval="31s" \</div><div>role="Slave" timeout="30" op start interval="0" timeout="240" op stop interval="0" timeout="100"</div><div><br></div><div>pcs -f iscsiha_cfg resource master ms_drbd_iscsiha p_drbd_iscsiha \</div><div>master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_iscsi_store1 ocf:heartbeat:iSCSITarget \</div><div>params implementation="tgt" iqn="iqn.2014-07.local.localdomain:store1" tid="1" \</div><div>allowed_initiators="10.10.1.61 10.10.1.62 10.10.1.63" incoming_username="iscsiuser" incoming_password="iscsipwd" \</div><div>op start interval="0" timeout="60" \</div><div>op stop interval="0" timeout="60" \</div><div>op monitor interval="30" timeout="60"</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_iscsi_store1_lun1 ocf:heartbeat:iSCSILogicalUnit \</div><div>params implementation="tgt" target_iqn="iqn.2014-07.local.localdomain:store1" lun="1" \</div><div>path="/dev/drbd/by-res/iscsiha" \</div><div>op start interval="0" timeout="60" \</div><div>op stop interval="0" timeout="60" \</div><div>op monitor interval="30" timeout="60"</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_ip_iscsi ocf:heartbeat:IPaddr2 \</div><div>params ip="10.10.1.71" \</div><div>op start interval="0" timeout="20" \</div><div>op stop interval="0" timeout="20" \</div><div>op monitor interval="30" timeout="20"</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_portblock-store1-block ocf:heartbeat:portblock \<br></div><div>params ip="10.10.1.71" portno="3260" protocol="tcp" action="block"</div><div><br></div><div>pcs -f iscsiha_cfg resource create p_portblock-store1-unblock ocf:heartbeat:portblock \</div><div>params ip="10.10.1.71" portno="3260" protocol="tcp" action="unblock" \</div><div>op monitor interval="30s"</div><div><br></div><div>pcs -f iscsiha_cfg resource group add g_iscsiha p_portblock-store1-block p_ip_iscsi p_iscsi_store1 \</div><div>p_iscsi_store1_lun1 p_portblock-store1-unblock</div><div><br></div><div>pcs -f iscsiha_cfg constraint colocation add Started g_iscsiha with Master ms_drbd_iscsiha INFINITY</div><div><br></div><div>pcs -f iscsiha_cfg constraint order promote ms_drbd_iscsiha then start g_iscsiha</div><div><br></div><div>pcs cluster cib-push iscsiha_cfg</div></div><div><br></div><div><br></div><div>- output of "crm_mon -1"</div><div><br></div><div><div> Resource Group: g_iscsiha</div><div> p_portblock-store1-block<span class="" style="white-space:pre"> </span>(ocf::heartbeat:portblock):<span class="" style="white-space:pre"> </span>Started srvmgmt01.localdomain.local </div><div> p_ip_iscsi<span class="" style="white-space:pre"> </span>(ocf::heartbeat:IPaddr2):<span class="" style="white-space:pre"> </span>Started srvmgmt01.localdomain.local </div><div> p_iscsi_store1<span class="" style="white-space:pre"> </span>(ocf::heartbeat:iSCSITarget):<span class="" style="white-space:pre"> </span>Started srvmgmt01.localdomain.local </div><div> p_iscsi_store1_lun1<span class="" style="white-space:pre"> </span>(ocf::heartbeat:iSCSILogicalUnit):<span class="" style="white-space:pre"> </span>Started srvmgmt01.localdomain.local </div><div> p_portblock-store1-unblock<span class="" style="white-space:pre"> </span>(ocf::heartbeat:portblock):<span class="" style="white-space:pre"> </span>Started srvmgmt01.localdomain.local </div></div><div><br></div><div><div>- output of tgtadm on both nodes while srvmgmt01 is active for the group</div><div><br></div><div>[root@srvmgmt01 ~]# tgtadm --mode target --op show </div><div>Target 1: iqn.2014-07.local.localdomain:store1</div><div> System information:</div><div> Driver: iscsi</div><div> State: ready</div><div> I_T nexus information:</div><div> LUN information:</div><div> LUN: 0</div><div> Type: controller</div><div> SCSI ID: IET 00010000</div><div> SCSI SN: beaf10</div><div> Size: 0 MB, Block size: 1</div><div> Online: Yes</div><div> Removable media: No</div><div> Prevent removal: No</div><div> Readonly: No</div><div> Backing store type: null</div><div> Backing store path: None</div><div> Backing store flags: </div><div> LUN: 1</div><div> Type: disk</div><div> SCSI ID: p_iscsi_store1_l</div><div> SCSI SN: 66666a41</div><div> Size: 214738 MB, Block size: 512</div><div> Online: Yes</div><div> Removable media: No</div><div> Prevent removal: No</div><div> Readonly: No</div><div> Backing store type: rdwr</div><div> Backing store path: /dev/drbd/by-res/iscsiha</div><div> Backing store flags: </div><div> Account information:</div><div> iscsiuser</div><div> ACL information:</div><div> 10.10.1.61</div><div> 10.10.1.62</div><div> 10.10.1.63</div><div><br></div><div>on the passive node:<br></div><div>[root@srvmgmt02 heartbeat]# tgtadm --mode target --op show </div><div>[root@srvmgmt02 heartbeat]# </div></div><div><br></div><div>TBV performance and tuning values taken from here:</div><div><a href="http://www.dbarticles.com/centos-6-iscsi-tgtd-setup-and-performance-adjustments/">http://www.dbarticles.com/centos-6-iscsi-tgtd-setup-and-performance-adjustments/</a><br></div><div><br></div><div>my cluster is basic for testing so not critical for my environment...</div><div>at the momento only 1Gbit/s network and one adapter for drbd replica and one for iSCSI traffic</div><div>Tested with some I/O basic benchmarks on VM insisting on the SD and I got about 90-95MB/s on both drbd and iSCSI networks. Also relocation of iSCSI service while benchmark active seemed not to cause problems with SD and VM.</div><div><br></div><div>- I also enabled iptables on cluster nodes so that the initiators (oVirt hosts) could connect to the ip alias dedicated to iSCSI servicing:</div><div>in /etc/sysconfig/iptables:</div><div># iSCSI</div><div>-A INPUT -p tcp -m tcp -d 10.10.1.71 --dport 3260 -j ACCEPT</div><div><br></div><div>I have to recheck the logs to give exact scenario of what happened causing the problem.... not being a critical system is not so well monitored at the moment...</div><div><br></div><div>comments welcome</div><div>Gianluca</div><div><br></div></div></div></div>