Re: [ovirt-users] can not use iscsi storage type onovirtandGlusterfshyper-convergedenvironment

Monday, 5 December 2016

On Tue, Dec 6, 2016 at 3:15 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote:

...
  Hi Nir ：
      before change code ,supervdsm report error log like those :
 ===================================  ------>

 MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
 17:12:16,372::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper)
 call getPathsStatus with () {}

 MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
 17:12:16,373::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus)
 /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status (cwd None)

 MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
 17:12:16,377::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus)
 SUCCESS: <err> = ''; <rc> = 0

 MainProcess|jsonrpc.Executor/2::ERROR::2016-12-02
 17:12:16,378::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) Error
 in getPathsStatus
 =======================================  <----------------
     my environment have other dm device , dmsetup status will show like
 this as follow, so will occure above error !
 ==================================================== -------->
 flash_sdd: 0 976771072 flashcache stats:
         reads(5723182), writes(47879823)
         read hits(2938125), read hit percent(51)
         write hits(12617126) write hit percent(26)
         dirty write hits(4592227) dirty write hit percent(9)
         replacement(701168), write replacement(3189334)
         write invalidates(0), read invalidates(1)
 ----
 ====================================================   <--------------
       use your patch to test , add or import iscsi type storage domain
 have no problem  , log like follow :
 ====================================================== --------->

 MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06
 08:58:07,439::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper)
 call getPathsStatus with () {}

 MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06
 08:58:07,439::devicemapper::155::Storage.Misc.excCmd::(_getPathsStatus)
 /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status --target
 multipath (cwd None)
 MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06
 08:58:07,443::devicemapper::155::Storage.Misc.excCmd::(_getPathsStatus)
 SUCCESS: <err> = ''; <rc> = 0
 ============================================================
 =====================

   Regards,
     humaorong

Thanks for testing this.

Sahina, can you test if this solve the issue with hyperconverge setup?
Maybe we can enable multipath with this patch.

Cheers,
Nir

...

 ------------------ Original ------------------
 *From: * "Nir Soffer&quot;&lt;nsoffer(a)redhat.com&gt;;
 *Date: * Tue, Dec 6, 2016 01:41 AM
 *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
 *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;; "Jeff
Nelson&quot;&lt;jenelson(a)redhat.com&gt;;
 &quot;胡晓宇&quot;&lt;samuel.xhu(a)horebdata.cn&gt;;
 *Subject: * Re: [ovirt-users] can not use iscsi storage type
 onovirtandGlusterfshyper-convergedenvironment

 Hi 胡茂荣,

 Can you test this patch?
 https://gerrit.ovirt.org/67844

 I also need more information on your setup, to add more
 details to the commit message.

 Thanks for reporting this,
 Nir

 On Mon, Dec 5, 2016 at 10:28 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote:

>
>   Thanks for Yaniv Kaul ,  change code need build vdsm source code , and
> if only  change /usr/share/vdsm/storage/devicemapper.py will not really
> take effect .
>
>    could this problem as a bug , and correct it to ovirt vdsm source code
> ?
>
> ------------------ Original ------------------
> *From: * "Yaniv Kaul&quot;&lt;ykaul(a)redhat.com&gt;;
> *Date: * Sun, Dec 4, 2016 07:07 PM
> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
> *Cc: * &quot;胡晓宇&quot;&lt;samuel.xhu(a)horebdata.cn&gt;;
&quot;users&quot;&lt;users(a)ovirt.org&gt;; "Sahina
> Bose&quot;&lt;sabose(a)redhat.com&gt;; "Jeff
Nelson&quot;&lt;jenelson(a)redhat.com&gt;;
> *Subject: * Re: [ovirt-users] can not use iscsi storage type
> onovirtandGlusterfshyper-converged environment
>
>
>
> On Dec 2, 2016 11:53 AM, "胡茂荣" <maorong.hu(a)horebdata.cn&gt; wrote:
>
>    I find supervdsm  used " /usr/sbin/dmsetup status" :
>
> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
> 17:12:16,372::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper)
> call getPathsStatus with () {}
> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
> 17:12:16,373::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus)
> /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status (cwd None)
> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02
> 17:12:16,377::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus)
> SUCCESS: <err> = ''; <rc> = 0
> MainProcess|jsonrpc.Executor/2::ERROR::2016-12-02
> 17:12:16,378::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) Error
> in getPathsStatus
>
> problem :
> how can I change  Storage.Misc.excCmd _getPathsStatus  "
> /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status"  to :
>
>    /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status  --target
> multipah
>
>   I think add iscsi type storage ,  if supervdsm scan mulitpah will solve
> my problem .(my environment have other dm devices, use "dmsetup status"
> will show them, and vdsm get dm path status will occur error )
>
> ============================================================
> ====================
> so I changed some as follow :
>  （1）
>    I  define EXT_DMSETUP_STATUS   in 
/usr/lib/python2.7/site-packages/vdsm/constants.py
> :
>
> /usr/lib/python2.7/site-packages/vdsm/constants.py:EXT_DMSETUP =
> '/usr/sbin/dmsetup'
> /usr/lib/python2.7/site-packages/vdsm/constants.py:EXT_DMSETUP_STATUS =
> "/usr/sbin/dmsetup status --target multipath"
>
>  (2)
>  /usr/share/vdsm/storage/devicemapper.py     add :
> from vdsm.constants import EXT_DMSETUP_STATUS
>
> and changed  getPathsStatus cmd  to " EXT_DMSETUP_STATUS" :
>
> def _getPathsStatus():
>     cmd = [EXT_DMSETUP_STATUS]                ##### before : cmd=[
> EXT_DMSETUP,"status"]
>
>
> Why not change this to:
> cmd = [EXT_DMSETUP,  "status", "--target",
"multipath"]
>
> Y.
>
>     rc, out, err = misc.execCmd(cmd)
> ============================================================
> ===========================
>
>  but log in supervdsm log also not change . Please help me ，how to change
> code to let supervdsm exec "/usr/sbin/dmsetup status --target multipath"
>   in function  getPathsStatus（） 。
>
>
>
>
> ------------------ Original ------------------
> *From: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
> *Date: * Fri, Nov 25, 2016 05:44 PM
> *To: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;;
> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff
Nelson"<
> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;;
> *Subject: * Re: [ovirt-users] can not use iscsi storage type on
> ovirtandGlusterfshyper-converged environment
>
>
> ===================================---
>
>    ###vdsm or supervdsm log  report :
>
>     MainProcess|jsonrpc.Executor/7::ERROR::2016-11-01
> 11:07:00,178::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper)
> Error in getPathsStatus
>
> MainProcess|jsonrpc.Executor/4::ERROR::2016-11-01
> 11:07:20,964::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper)
> Error in getPathsStatus
>  ============================  some code info ------------>
> [root@horeba storage]# pwd
> /usr/share/vdsm/storage
>
> [root@horeba storage]# grep "getPathsStatus" -R ./
> ./devicemapper.py:def _getPathsStatus():
> ./devicemapper.py:def getPathsStatus():
> ./devicemapper.py:    return getProxy().getPathsStatus()
> ./multipath.py:    pathStatuses = devicemapper.getPathsStatus()
>
> def _getPathsStatus():
>     cmd = [EXT_DMSETUP, "status"]
>     rc, out, err = misc.execCmd(cmd)
>     if rc != 0:
>         raise Exception("Could not get device statuses")
>
>     res = {}
>     for statusLine in out:
>         try:
>             devName, statusLine = statusLine.split(":", 1)
>         except ValueError:
>             if len(out) == 1:
>                 # return an empty dict when status output is: No devices
> found
>                 return res
>             else:
>                 raise
>
>         for m in PATH_STATUS_RE.finditer(statusLine):
>             devNum, status = m.groups()
>             physdevName = findDev(*[int(i) for i in devNum.split(":")])
>             res[physdevName] = {"A": "active", "F":
"failed"}[status]
>
>     return res
> def getPathsStatus():
>     return getProxy().getPathsStatus()
> =============================================
>   and flashcache dm device will error  when use getPathsStatus()
> function .  could change code not check flashcache dm device ?
> ========================================dmsetup info ----------->
> [root@horebc ~]# dmsetup status
> flash_sdb: 0 976771072 flashcache stats:
>         reads(1388761), writes(15548965)
>         read hits(1235671), read hit percent(88)
>         write hits(6539144) write hit percent(42)
>         dirty write hits(21372) dirty write hit percent(0)
>         replacement(147711), write replacement(524881)
>         write invalidates(0), read invalidates(1)
>         pending enqueues(810), pending inval(810)
>         metadata dirties(15196370), metadata cleans(15196322)
>         metadata batch(30087377) metadata ssd writes(305315)
>         cleanings(15196322) fallow cleanings(48187)
>         no room(337139) front merge(716153) back merge(14391395)
>         force_clean_block(0)
>         disk reads(153093), disk writes(15530535) ssd reads(16431974) ssd
> writes(15672221)
>         uncached reads(3714), uncached writes(334235), uncached IO
> requeue(0)
>         disk read errors(0), disk write errors(0) ssd read errors(0) ssd
> write errors(0)
>         uncached sequential reads(0), uncached sequential writes(0)
>         pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
>         lru hot blocks(12158976), lru warm blocks(12158976)
>         lru promotions(0), lru demotions(0)
> dedbd337--ca66--43ff--b78c--4e9347682a9c-metadata: 0 1048576 linear
> dedbd337--ca66--43ff--b78c--4e9347682a9c-leases: 0 4194304 linear
> 23137643634356633: 0 2147483648 multipath 2 0 0 0 1 1 A 0 1 2 8:128 A 0
> 0 1
> dedbd337--ca66--43ff--b78c--4e9347682a9c-master: 0 2097152 linear
> dedbd337--ca66--43ff--b78c--4e9347682a9c-inbox: 0 262144 linear
> dedbd337--ca66--43ff--b78c--4e9347682a9c-outbox: 0 262144 linear
>
> [root@horebc ~]# dmsetup info -C
> Name                                              Maj Min Stat Open Targ
> Event  UUID
> flash_sdb                                         253   0 L--w    1    1
>      0
> dedbd337--ca66--43ff--b78c--4e9347682a9c-metadata 253   4 L--w    0    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIGwifAPyYj9GmjFzCmJkIf9vF
> FFHn9n7V
> dedbd337--ca66--43ff--b78c--4e9347682a9c-leases   253   6 L--w    0    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIVSCllWEYYKziY1bSeiTL0dAK
> Ad27JqDT
> 23137643634356633                                 253   3 L--w    6    1
>      0 mpath-23137643634356633
> dedbd337--ca66--43ff--b78c--4e9347682a9c-master   253   9 L--w    0    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIiEnFZklRhZfFZ4YRdYWFImKW
> sUGr5pHg
> dedbd337--ca66--43ff--b78c--4e9347682a9c-inbox    253   8 L--w    0    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGILobNK8KRD4SzDWyg50aG7jGd
> cNAi3KNw
> dedbd337--ca66--43ff--b78c--4e9347682a9c-outbox   253   5 L--w    0    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIAvbT8CLegbVL802bG3QgLtH7
> I7llmS6R
> flash_sdf                                         253   2 L--w    1    1
>      0
> dedbd337--ca66--43ff--b78c--4e9347682a9c-ids      253   7 L--w    1    1
>      0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIIkeaFaWvxa9wbHy7WrfiXNsP
> 4F2J3gg0
> flash_sdd                                         253   1 L--w    1    1
>      0
>
> ====================================================================
> and flashcache dm have no UUID  ，I think it can export it  before
> checking .
>
>
> humaorong
>   2016-11-25
>
>
>
> ------------------ Original ------------------
> *From: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
> *Date: * Fri, Nov 25, 2016 01:18 PM
> *To: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;;
> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff
Nelson"<
> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;;
> *Subject: * Re: [ovirt-users] can not use iscsi storage type on
> ovirtandGlusterfshyper-converged environment
>
>
>     I find this problem more info :
>       I use flashcache  on ovirt hosts , have dm device in /dev/mapper/ :
>
> [root@horeba init.d]# dmsetup info -C     （and I set them in multipath
> blacklist ）
> Name             Maj Min Stat Open Targ Event  UUID
>
> flash_sdb        253   0 L--w    0    1      0
>
> flash_sdf        253   2 L--w    0    1      0
>
> flash_sdd        253   1 L--w    0    1      0
> [root@horeba init.d]# multipath -l
> [root@horeba init.d]#
>
> [root@horeba init.d]# ll /dev/mapper/
> total 0
> crw------- 1 root root 10, 236 Nov 25 10:09 control
> lrwxrwxrwx 1 root root       7 Nov 25 12:51 flash_sdb -> ../dm-0
> lrwxrwxrwx 1 root root       7 Nov 25 12:51 flash_sdd -> ../dm-1
> lrwxrwxrwx 1 root root       7 Nov 25 12:51 flash_sdf -> ../dm-2
>
>   on this condition , ovirt UI add  iscsi type storage  will fail .
>
>    If I delete the flashcache device  /dev/mapper/flash_*  , ovirt UI
>  add iscsi type storage  have no problem .
>
>    I need flashcache for using ssd cache on my environment ,  how can I
> use iscsi type storage on this environment , please help me , Thanks！
>
>
> ------------------ Original ------------------
> *From: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;;
> *Date: * Thu, Nov 3, 2016 05:54 PM
> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff
Nelson"<
> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;;
> *Subject: * Re: [ovirt-users] can not use iscsi storage type on
> ovirtandGlusterfshyper-converged environment
>
> A wild guess, not sure if it is related - can you check if multipathd
> service is enabled. If you set up your oVirt-Gluster hyperconverged
> environment via gdeploy, multipathd service is disabled and the
> /etc/multipath.conf is edited to blacklist all devices - this was to fix
> https://bugzilla.redhat.com/show_bug.cgi?id=1303940
>
> Since you mention you're unable to add iscsi storage only in this
> environment, thought it's worth checking.
>
> On Thu, Nov 3, 2016 at 6:40 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote:
>
>>
>>      my   environment rpm  are :
>>  [root@horeba ~]# rpm -q vdsm
>> vdsm-4.18.13-1.el7.centos.x86_64
>>
>> [root@horeba ~]# rpm -aq | grep ovirt
>> ovirt-vmconsole-host-1.0.4-1.el7.centos.noarch
>> ovirt-imageio-common-0.4.0-1.el7.noarch
>> ovirt-hosted-engine-setup-2.0.2.2-1.el7.centos.noarch
>> ovirt-imageio-daemon-0.4.0-1.el7.noarch
>> ovirt-engine-appliance-4.0-20160928.1.el7.centos.noarch
>> ovirt-vmconsole-1.0.4-1.el7.centos.noarch
>> ovirt-host-deploy-1.5.2-1.el7.centos.noarch
>> ovirt-hosted-engine-ha-2.0.4-1.el7.centos.noarch
>> ovirt-release40-4.0.4-1.noarch
>> ovirt-setup-lib-1.0.2-1.el7.centos.noarch
>> ovirt-engine-sdk-python-3.6.9.1-1.el7.centos.noarch
>>
>>   and I test if not on ' ovirt and Glusterfs hyper-converged
>> environment ' ,  ovirt WEB UI  add iscsi storage  work ok .
>>
>>
>> ------------------ Original ------------------
>> *From: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;;
>> *Date: * Wed, Nov 2, 2016 07:37 PM
>> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
>> *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;; "Jeff
Nelson&quot;&lt;jenelson(a)redhat.com&gt;;
>> "Nir Soffer&quot;&lt;nsoffer(a)redhat.com&gt;;
>> *Subject: * Re: [ovirt-users] can not use iscsi storage type on
>> ovirtandGlusterfs hyper-converged environment
>>
>> Thanks for the logs,
>>
>> What kind of VDSM version are you using?
>>     "rpm -q vdsm"
>> There seems to be a similar issue which was reported recently in the
>> VDSM area
>> (see https://bugzilla.redhat.com/show_bug.cgi?id=1197292)
>> It should be fixed in later versions of VDSM vdsm-4.16.12-2.el7ev.x86_64
>> Adding also Nir and Jeff to the thread, if they have any insights
>>
>> Regards,
>> Maor
>>
>> On Wed, Nov 2, 2016 at 4:11 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote:
>>
>>>
>>>  Hi Maor:
>>>       vdsm/supervdsm/engine log on attachment .  I mkfs.xfs the lun
>>> block device and mount to /mnt , dd write  ,dmesg not report error ,dd
>>> result is ok :
>>>
>>> /dev/sdi                      50G   33M   50G   1% /mnt
>>>
>>> [root@horebc mnt]# for i in `seq 3`; do dd if=/dev/zero of=./file
>>> bs=1G count=1 oflag=direct ; done
>>> 1+0 records in
>>> 1+0 records out
>>> 1073741824 bytes (1.1 GB) copied, 13.3232 s, 80.6 MB/s
>>> 1+0 records in
>>> 1+0 records out
>>> 1073741824 bytes (1.1 GB) copied, 9.89988 s, 108 MB/s
>>> 1+0 records in
>>> 1+0 records out
>>> 1073741824 bytes (1.1 GB) copied, 14.0143 s, 76.6 MB/s
>>>
>>>    my envirnment  have three  network segments (hosts have 3 network
>>> segments ) :
>>>        engine  and glusterfs mount : 192.168.11.X/24
>>>         glusterfs brick : 192.168.10.x/24
>>>         iscsi : 192.168.1.0/24
>>>
>>>     and I add 192.168.1.0/24 to engine vm ,  ovirt web UI report the
>>> same error .
>>>
>>>  humaorong
>>>   2016-11-2
>>>
>>> ------------------ Original ------------------
>>> *From: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;;
>>> *Date: * Tue, Nov 1, 2016 08:14 PM
>>> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;;
>>> *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;;
>>> *Subject: * Re: [ovirt-users] can not use iscsi storage type on ovirt
>>> andGlusterfs hyper-converged environment
>>>
>>> Hi 胡茂荣Can u please also add the VDSM and engine logs.
>>> If you try discover and connect to those luns directly from your Host
>>> does it work?
>>>
>>> Regards,
>>> Maor
>>>
>>>
>>> On Tue, Nov 1, 2016 at 6:12 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote:
>>>
>>>>
>>>>
>>>>     on ovirt and Glusterfs hyper-converged environment , can not use
>>>> iscsi storage type , UI report error: "Could not retrieve LUNs,
>>>> please check your storage." , vdsm report :"VDSM
hosted_engine_3
>>>> command failed: Error block device action: ()" .
>>>>     but this block device alse login on centos 7 host :
>>>> =============================================================
>>>>
>>>> ## lsscsi
>>>>
>>>> [7:0:0:0]   disk    SCST_BIO DEVFOR_OVIRT_rbd  221  /dev/sdi
>>>>
>>>>   ## dmesg :
>>>>
>>>> [684521.131186] sd 7:0:0:0: [sdi] Attached SCSI disk
>>>>
>>>> ===================================---
>>>>
>>>>    ###vdsm or supervdsm log  report :
>>>>
>>>>     MainProcess|jsonrpc.Executor/7::ERROR::2016-11-01
>>>> 11:07:00,178::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper)
>>>> Error in getPathsStatus
>>>>
>>>> MainProcess|jsonrpc.Executor/4::ERROR::2016-11-01
>>>> 11:07:20,964::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper)
>>>> Error in getPathsStatus
>>>>
>>>>    jsonrpc.Executor/4::DEBUG::2016-11-01
>>>> 11:07:04,251::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI
>>>> scan, this will take up to 30 seconds
>>>>
>>>> jsonrpc.Executor/5::INFO::2016-11-01 11:07:19,413::iscsi::567::Stor
>>>> age.ISCSI::(setRpFilterIfNeeded) iSCSI iface.net_ifacename not
>>>> provided. Skipping.
>>>>
>>>> 11:09:15,753::iscsiadm::119::Storage.Misc.excCmd::(_runCmd)
>>>> /usr/bin/taskset --cpu-list 0-7 /usr/bin/sudo -n /usr/sbin/iscsiadm -m
>>>> session -R (cwd None)
>>>>
>>>> ======================================
>>>>
>>>>      the other info please the attachment "bug-info.doc".
>>>>
>>>>      this prolem on ovirt3.6 and 4.X  ovirt and Glusterfs
>>>> hyper-converged environment . how can I use iscsi storage type on ovirt
>>>> and Glusterfs hyper-converged environment .Please help me !
>>>>
>>>>     humaorong
>>>>
>>>>    2016-11-1
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users(a)ovirt.org
>>>> http://lists.ovirt.org/mailman/listinfo/users
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> Users mailing list
>> Users(a)ovirt.org
>> http://lists.ovirt.org/mailman/listinfo/users
>>
>>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>
>
> _______________________________________________
> Users mailing list
> Users(a)ovirt.org
> http://lists.ovirt.org/mailman/listinfo/users
>
>

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

Re: [ovirt-users] can not use iscsi storage type onovirtandGlusterfshyper-convergedenvironment