Re: [ovirt-users] can not use iscsi storage type onovirtandGlusterfshyper-convergedenvironment

On Tue, Dec 6, 2016 at 3:15 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote: > Hi Nir ： > before change code ,supervdsm report error log like those : > =================================== ------> > > MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 > 17:12:16,372::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper) > call getPathsStatus with () {} > > MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 > 17:12:16,373::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus) > /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status (cwd None) > > MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 > 17:12:16,377::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus) > SUCCESS: <err> = ''; <rc> = 0 > > MainProcess|jsonrpc.Executor/2::ERROR::2016-12-02 > 17:12:16,378::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) Error > in getPathsStatus > ======================================= <---------------- > my environment have other dm device , dmsetup status will show like > this as follow, so will occure above error ! > ==================================================== --------> > flash_sdd: 0 976771072 flashcache stats: > reads(5723182), writes(47879823) > read hits(2938125), read hit percent(51) > write hits(12617126) write hit percent(26) > dirty write hits(4592227) dirty write hit percent(9) > replacement(701168), write replacement(3189334) > write invalidates(0), read invalidates(1) > ---- > ==================================================== <-------------- > use your patch to test , add or import iscsi type storage domain > have no problem , log like follow : > ====================================================== ---------> > > MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06 > 08:58:07,439::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper) > call getPathsStatus with () {} > > MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06 > 08:58:07,439::devicemapper::155::Storage.Misc.excCmd::(_getPathsStatus) > /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status --target > multipath (cwd None) > MainProcess|jsonrpc.Executor/1::DEBUG::2016-12-06 > 08:58:07,443::devicemapper::155::Storage.Misc.excCmd::(_getPathsStatus) > SUCCESS: <err> = ''; <rc> = 0 > ============================================================ > ===================== > > Regards, > humaorong > Thanks for testing this. Sahina, can you test if this solve the issue with hyperconverge setup? Maybe we can enable multipath with this patch.

Cheers, Nir > > > ------------------ Original ------------------ > *From: * "Nir Soffer&quot;&lt;nsoffer(a)redhat.com&gt;; > *Date: * Tue, Dec 6, 2016 01:41 AM > *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; > *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;; "Jeff Nelson&quot;&lt;jenelson(a)redhat.com&gt;; > &quot;胡晓宇&quot;&lt;samuel.xhu(a)horebdata.cn&gt;; > *Subject: * Re: [ovirt-users] can not use iscsi storage type > onovirtandGlusterfshyper-convergedenvironment > > Hi 胡茂荣, > > Can you test this patch? > https://gerrit.ovirt.org/67844 > > I also need more information on your setup, to add more > details to the commit message. > > Thanks for reporting this, > Nir > > On Mon, Dec 5, 2016 at 10:28 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote: > >> >> Thanks for Yaniv Kaul , change code need build vdsm source code , and >> if only change /usr/share/vdsm/storage/devicemapper.py will not really >> take effect . >> >> could this problem as a bug , and correct it to ovirt vdsm source >> code ? >> >> ------------------ Original ------------------ >> *From: * "Yaniv Kaul&quot;&lt;ykaul(a)redhat.com&gt;; >> *Date: * Sun, Dec 4, 2016 07:07 PM >> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >> *Cc: * &quot;胡晓宇&quot;&lt;samuel.xhu(a)horebdata.cn&gt;; &quot;users&quot;&lt;users(a)ovirt.org&gt;; >> "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;; "Jeff Nelson&quot;&lt;jenelson(a)redhat.com&gt;; >> *Subject: * Re: [ovirt-users] can not use iscsi storage type >> onovirtandGlusterfshyper-converged environment >> >> >> >> On Dec 2, 2016 11:53 AM, "胡茂荣" <maorong.hu(a)horebdata.cn&gt; wrote: >> >> I find supervdsm used " /usr/sbin/dmsetup status" : >> >> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 >> 17:12:16,372::supervdsmServer::92::SuperVdsm.ServerCallback::(wrapper) >> call getPathsStatus with () {} >> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 >> 17:12:16,373::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus) >> /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status (cwd None) >> MainProcess|jsonrpc.Executor/2::DEBUG::2016-12-02 >> 17:12:16,377::devicemapper::154::Storage.Misc.excCmd::(_getPathsStatus) >> SUCCESS: <err> = ''; <rc> = 0 >> MainProcess|jsonrpc.Executor/2::ERROR::2016-12-02 >> 17:12:16,378::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) Error >> in getPathsStatus >> >> problem : >> how can I change Storage.Misc.excCmd _getPathsStatus " >> /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status" to : >> >> /usr/bin/taskset --cpu-list 0-7 /usr/sbin/dmsetup status --target >> multipah >> >> I think add iscsi type storage , if supervdsm scan mulitpah will >> solve my problem .(my environment have other dm devices, use "dmsetup >> status" will show them, and vdsm get dm path status will occur error ) >> >> ============================================================ >> ==================== >> so I changed some as follow : >> （1） >> I define EXT_DMSETUP_STATUS in /usr/lib/python2.7/site-packages/vdsm/constants.py >> : >> >> /usr/lib/python2.7/site-packages/vdsm/constants.py:EXT_DMSETUP = >> '/usr/sbin/dmsetup' >> /usr/lib/python2.7/site-packages/vdsm/constants.py:EXT_DMSETUP_STATUS = >> "/usr/sbin/dmsetup status --target multipath" >> >> (2) >> /usr/share/vdsm/storage/devicemapper.py add : >> from vdsm.constants import EXT_DMSETUP_STATUS >> >> and changed getPathsStatus cmd to " EXT_DMSETUP_STATUS" : >> >> def _getPathsStatus(): >> cmd = [EXT_DMSETUP_STATUS] ##### before : cmd=[ >> EXT_DMSETUP,"status"] >> >> >> Why not change this to: >> cmd = [EXT_DMSETUP, "status", "--target", "multipath"] >> >> Y. >> >> rc, out, err = misc.execCmd(cmd) >> ============================================================ >> =========================== >> >> but log in supervdsm log also not change . Please help me ，how to >> change code to let supervdsm exec "/usr/sbin/dmsetup status --target >> multipath" in function getPathsStatus（） 。 >> >> >> >> >> ------------------ Original ------------------ >> *From: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >> *Date: * Fri, Nov 25, 2016 05:44 PM >> *To: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;; >> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff Nelson"< >> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;; >> *Subject: * Re: [ovirt-users] can not use iscsi storage type on >> ovirtandGlusterfshyper-converged environment >> >> >> ===================================--- >> >> ###vdsm or supervdsm log report : >> >> MainProcess|jsonrpc.Executor/7::ERROR::2016-11-01 >> 11:07:00,178::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) >> Error in getPathsStatus >> >> MainProcess|jsonrpc.Executor/4::ERROR::2016-11-01 >> 11:07:20,964::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) >> Error in getPathsStatus >> ============================ some code info ------------> >> [root@horeba storage]# pwd >> /usr/share/vdsm/storage >> >> [root@horeba storage]# grep "getPathsStatus" -R ./ >> ./devicemapper.py:def _getPathsStatus(): >> ./devicemapper.py:def getPathsStatus(): >> ./devicemapper.py: return getProxy().getPathsStatus() >> ./multipath.py: pathStatuses = devicemapper.getPathsStatus() >> >> def _getPathsStatus(): >> cmd = [EXT_DMSETUP, "status"] >> rc, out, err = misc.execCmd(cmd) >> if rc != 0: >> raise Exception("Could not get device statuses") >> >> res = {} >> for statusLine in out: >> try: >> devName, statusLine = statusLine.split(":", 1) >> except ValueError: >> if len(out) == 1: >> # return an empty dict when status output is: No devices >> found >> return res >> else: >> raise >> >> for m in PATH_STATUS_RE.finditer(statusLine): >> devNum, status = m.groups() >> physdevName = findDev(*[int(i) for i in devNum.split(":")]) >> res[physdevName] = {"A": "active", "F": "failed"}[status] >> >> return res >> def getPathsStatus(): >> return getProxy().getPathsStatus() >> ============================================= >> and flashcache dm device will error when use getPathsStatus() >> function . could change code not check flashcache dm device ? >> ========================================dmsetup info -----------> >> [root@horebc ~]# dmsetup status >> flash_sdb: 0 976771072 flashcache stats: >> reads(1388761), writes(15548965) >> read hits(1235671), read hit percent(88) >> write hits(6539144) write hit percent(42) >> dirty write hits(21372) dirty write hit percent(0) >> replacement(147711), write replacement(524881) >> write invalidates(0), read invalidates(1) >> pending enqueues(810), pending inval(810) >> metadata dirties(15196370), metadata cleans(15196322) >> metadata batch(30087377) metadata ssd writes(305315) >> cleanings(15196322) fallow cleanings(48187) >> no room(337139) front merge(716153) back merge(14391395) >> force_clean_block(0) >> disk reads(153093), disk writes(15530535) ssd reads(16431974) >> ssd writes(15672221) >> uncached reads(3714), uncached writes(334235), uncached IO >> requeue(0) >> disk read errors(0), disk write errors(0) ssd read errors(0) ssd >> write errors(0) >> uncached sequential reads(0), uncached sequential writes(0) >> pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0) >> lru hot blocks(12158976), lru warm blocks(12158976) >> lru promotions(0), lru demotions(0) >> dedbd337--ca66--43ff--b78c--4e9347682a9c-metadata: 0 1048576 linear >> dedbd337--ca66--43ff--b78c--4e9347682a9c-leases: 0 4194304 linear >> 23137643634356633: 0 2147483648 multipath 2 0 0 0 1 1 A 0 1 2 8:128 A 0 >> 0 1 >> dedbd337--ca66--43ff--b78c--4e9347682a9c-master: 0 2097152 linear >> dedbd337--ca66--43ff--b78c--4e9347682a9c-inbox: 0 262144 linear >> dedbd337--ca66--43ff--b78c--4e9347682a9c-outbox: 0 262144 linear >> >> [root@horebc ~]# dmsetup info -C >> Name Maj Min Stat Open Targ >> Event UUID >> flash_sdb 253 0 L--w 1 1 >> 0 >> dedbd337--ca66--43ff--b78c--4e9347682a9c-metadata 253 4 L--w 0 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIGwifAPyYj9GmjFzCmJkIf9vF >> FFHn9n7V >> dedbd337--ca66--43ff--b78c--4e9347682a9c-leases 253 6 L--w 0 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIVSCllWEYYKziY1bSeiTL0dAK >> Ad27JqDT >> 23137643634356633 253 3 L--w 6 1 >> 0 mpath-23137643634356633 >> dedbd337--ca66--43ff--b78c--4e9347682a9c-master 253 9 L--w 0 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIiEnFZklRhZfFZ4YRdYWFImKW >> sUGr5pHg >> dedbd337--ca66--43ff--b78c--4e9347682a9c-inbox 253 8 L--w 0 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGILobNK8KRD4SzDWyg50aG7jGd >> cNAi3KNw >> dedbd337--ca66--43ff--b78c--4e9347682a9c-outbox 253 5 L--w 0 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIAvbT8CLegbVL802bG3QgLtH7 >> I7llmS6R >> flash_sdf 253 2 L--w 1 1 >> 0 >> dedbd337--ca66--43ff--b78c--4e9347682a9c-ids 253 7 L--w 1 >> 1 0 LVM-zSZKhQXbs2Ms5ceM2Wlt3p46BgsS8gGIIkeaFaWvxa9wbHy7WrfiXNsP >> 4F2J3gg0 >> flash_sdd 253 1 L--w 1 1 >> 0 >> >> ==================================================================== >> and flashcache dm have no UUID ，I think it can export it before >> checking . >> >> >> humaorong >> 2016-11-25 >> >> >> >> ------------------ Original ------------------ >> *From: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >> *Date: * Fri, Nov 25, 2016 01:18 PM >> *To: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;; >> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff Nelson"< >> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;; >> *Subject: * Re: [ovirt-users] can not use iscsi storage type on >> ovirtandGlusterfshyper-converged environment >> >> >> I find this problem more info : >> I use flashcache on ovirt hosts , have dm device in /dev/mapper/ >> : >> >> [root@horeba init.d]# dmsetup info -C （and I set them in multipath >> blacklist ） >> Name Maj Min Stat Open Targ Event UUID >> >> flash_sdb 253 0 L--w 0 1 0 >> >> flash_sdf 253 2 L--w 0 1 0 >> >> flash_sdd 253 1 L--w 0 1 0 >> [root@horeba init.d]# multipath -l >> [root@horeba init.d]# >> >> [root@horeba init.d]# ll /dev/mapper/ >> total 0 >> crw------- 1 root root 10, 236 Nov 25 10:09 control >> lrwxrwxrwx 1 root root 7 Nov 25 12:51 flash_sdb -> ../dm-0 >> lrwxrwxrwx 1 root root 7 Nov 25 12:51 flash_sdd -> ../dm-1 >> lrwxrwxrwx 1 root root 7 Nov 25 12:51 flash_sdf -> ../dm-2 >> >> on this condition , ovirt UI add iscsi type storage will fail . >> >> If I delete the flashcache device /dev/mapper/flash_* , ovirt UI >> add iscsi type storage have no problem . >> >> I need flashcache for using ssd cache on my environment , how can I >> use iscsi type storage on this environment , please help me , Thanks！ >> >> >> ------------------ Original ------------------ >> *From: * "Sahina Bose&quot;&lt;sabose(a)redhat.com&gt;; >> *Date: * Thu, Nov 3, 2016 05:54 PM >> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >> *Cc: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; "Jeff Nelson"< >> jenelson(a)redhat.com>; &quot;users&quot;&lt;users(a)ovirt.org&gt;; >> *Subject: * Re: [ovirt-users] can not use iscsi storage type on >> ovirtandGlusterfshyper-converged environment >> >> A wild guess, not sure if it is related - can you check if multipathd >> service is enabled. If you set up your oVirt-Gluster hyperconverged >> environment via gdeploy, multipathd service is disabled and the >> /etc/multipath.conf is edited to blacklist all devices - this was to fix >> https://bugzilla.redhat.com/show_bug.cgi?id=1303940 >> >> Since you mention you're unable to add iscsi storage only in this >> environment, thought it's worth checking. >> >> On Thu, Nov 3, 2016 at 6:40 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote: >> >>> >>> my environment rpm are : >>> [root@horeba ~]# rpm -q vdsm >>> vdsm-4.18.13-1.el7.centos.x86_64 >>> >>> [root@horeba ~]# rpm -aq | grep ovirt >>> ovirt-vmconsole-host-1.0.4-1.el7.centos.noarch >>> ovirt-imageio-common-0.4.0-1.el7.noarch >>> ovirt-hosted-engine-setup-2.0.2.2-1.el7.centos.noarch >>> ovirt-imageio-daemon-0.4.0-1.el7.noarch >>> ovirt-engine-appliance-4.0-20160928.1.el7.centos.noarch >>> ovirt-vmconsole-1.0.4-1.el7.centos.noarch >>> ovirt-host-deploy-1.5.2-1.el7.centos.noarch >>> ovirt-hosted-engine-ha-2.0.4-1.el7.centos.noarch >>> ovirt-release40-4.0.4-1.noarch >>> ovirt-setup-lib-1.0.2-1.el7.centos.noarch >>> ovirt-engine-sdk-python-3.6.9.1-1.el7.centos.noarch >>> >>> and I test if not on ' ovirt and Glusterfs hyper-converged >>> environment ' , ovirt WEB UI add iscsi storage work ok . >>> >>> >>> ------------------ Original ------------------ >>> *From: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; >>> *Date: * Wed, Nov 2, 2016 07:37 PM >>> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >>> *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;; "Jeff Nelson&quot;&lt;jenelson(a)redhat.com&gt;; >>> "Nir Soffer&quot;&lt;nsoffer(a)redhat.com&gt;; >>> *Subject: * Re: [ovirt-users] can not use iscsi storage type on >>> ovirtandGlusterfs hyper-converged environment >>> >>> Thanks for the logs, >>> >>> What kind of VDSM version are you using? >>> "rpm -q vdsm" >>> There seems to be a similar issue which was reported recently in the >>> VDSM area >>> (see https://bugzilla.redhat.com/show_bug.cgi?id=1197292) >>> It should be fixed in later versions of VDSM >>> vdsm-4.16.12-2.el7ev.x86_64 >>> Adding also Nir and Jeff to the thread, if they have any insights >>> >>> Regards, >>> Maor >>> >>> On Wed, Nov 2, 2016 at 4:11 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote: >>> >>>> >>>> Hi Maor: >>>> vdsm/supervdsm/engine log on attachment . I mkfs.xfs the lun >>>> block device and mount to /mnt , dd write ,dmesg not report error ,dd >>>> result is ok : >>>> >>>> /dev/sdi 50G 33M 50G 1% /mnt >>>> >>>> [root@horebc mnt]# for i in `seq 3`; do dd if=/dev/zero of=./file >>>> bs=1G count=1 oflag=direct ; done >>>> 1+0 records in >>>> 1+0 records out >>>> 1073741824 bytes (1.1 GB) copied, 13.3232 s, 80.6 MB/s >>>> 1+0 records in >>>> 1+0 records out >>>> 1073741824 bytes (1.1 GB) copied, 9.89988 s, 108 MB/s >>>> 1+0 records in >>>> 1+0 records out >>>> 1073741824 bytes (1.1 GB) copied, 14.0143 s, 76.6 MB/s >>>> >>>> my envirnment have three network segments (hosts have 3 network >>>> segments ) : >>>> engine and glusterfs mount : 192.168.11.X/24 >>>> glusterfs brick : 192.168.10.x/24 >>>> iscsi : 192.168.1.0/24 >>>> >>>> and I add 192.168.1.0/24 to engine vm , ovirt web UI report the >>>> same error . >>>> >>>> humaorong >>>> 2016-11-2 >>>> >>>> ------------------ Original ------------------ >>>> *From: * "Maor Lipchuk&quot;&lt;mlipchuk(a)redhat.com&gt;; >>>> *Date: * Tue, Nov 1, 2016 08:14 PM >>>> *To: * &quot;胡茂荣&quot;&lt;maorong.hu(a)horebdata.cn&gt;; >>>> *Cc: * &quot;users&quot;&lt;users(a)ovirt.org&gt;; >>>> *Subject: * Re: [ovirt-users] can not use iscsi storage type on ovirt >>>> andGlusterfs hyper-converged environment >>>> >>>> Hi 胡茂荣Can u please also add the VDSM and engine logs. >>>> If you try discover and connect to those luns directly from your Host >>>> does it work? >>>> >>>> Regards, >>>> Maor >>>> >>>> >>>> On Tue, Nov 1, 2016 at 6:12 AM, 胡茂荣 <maorong.hu(a)horebdata.cn&gt; wrote: >>>> >>>>> >>>>> >>>>> on ovirt and Glusterfs hyper-converged environment , can not use >>>>> iscsi storage type , UI report error: "Could not retrieve LUNs, >>>>> please check your storage." , vdsm report :"VDSM hosted_engine_3 >>>>> command failed: Error block device action: ()" . >>>>> but this block device alse login on centos 7 host : >>>>> ============================================================= >>>>> >>>>> ## lsscsi >>>>> >>>>> [7:0:0:0] disk SCST_BIO DEVFOR_OVIRT_rbd 221 /dev/sdi >>>>> >>>>> ## dmesg : >>>>> >>>>> [684521.131186] sd 7:0:0:0: [sdi] Attached SCSI disk >>>>> >>>>> ===================================--- >>>>> >>>>> ###vdsm or supervdsm log report : >>>>> >>>>> MainProcess|jsonrpc.Executor/7::ERROR::2016-11-01 >>>>> 11:07:00,178::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) >>>>> Error in getPathsStatus >>>>> >>>>> MainProcess|jsonrpc.Executor/4::ERROR::2016-11-01 >>>>> 11:07:20,964::supervdsmServer::96::SuperVdsm.ServerCallback::(wrapper) >>>>> Error in getPathsStatus >>>>> >>>>> jsonrpc.Executor/4::DEBUG::2016-11-01 >>>>> 11:07:04,251::iscsi::434::Storage.ISCSI::(rescan) Performing SCSI >>>>> scan, this will take up to 30 seconds >>>>> >>>>> jsonrpc.Executor/5::INFO::2016-11-01 11:07:19,413::iscsi::567::Stor >>>>> age.ISCSI::(setRpFilterIfNeeded) iSCSI iface.net_ifacename not >>>>> provided. Skipping. >>>>> >>>>> 11:09:15,753::iscsiadm::119::Storage.Misc.excCmd::(_runCmd) >>>>> /usr/bin/taskset --cpu-list 0-7 /usr/bin/sudo -n /usr/sbin/iscsiadm -m >>>>> session -R (cwd None) >>>>> >>>>> ====================================== >>>>> >>>>> the other info please the attachment "bug-info.doc". >>>>> >>>>> this prolem on ovirt3.6 and 4.X ovirt and Glusterfs >>>>> hyper-converged environment . how can I use iscsi storage type on ovirt >>>>> and Glusterfs hyper-converged environment .Please help me ! >>>>> >>>>> humaorong >>>>> >>>>> 2016-11-1 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Users mailing list >>>>> Users(a)ovirt.org >>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>> >>>>> >>>> >>> >>> _______________________________________________ >>> Users mailing list >>> Users(a)ovirt.org >>> http://lists.ovirt.org/mailman/listinfo/users >>> >>> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org >> http://lists.ovirt.org/mailman/listinfo/users >> >> >