- Users - oVirt List Archives

Re: [Users] Users Digest, Vol 28, Issue 299
by Ertan ERBEK 28 Jan '14

28 Jan '14

This is a multi-part message in MIME format. --------------070306080206060704030900 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Fnaly I fix my problem with following step 1) Check your maximum shared memory : $ |sudo cat /proc/sys/kernel/shmmax 33554432| You need to increase the shared memory to fix this. 2) To fix it temporarily, run this command : $ |sudo sysctl -w kernel.shmmax=134217728| 3) To fix this permenantly, set this in your configuration: $ |sudo vi /etc/sysctl.conf kernel.shmmax = 134217728 | On 28-01-2014 15:02, users-request(a)ovirt.org wrote: > Send Users mailing list submissions to > users(a)ovirt.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.ovirt.org/mailman/listinfo/users > or, via email, send a message with subject or body 'help' to > users-request(a)ovirt.org > > You can reach the person managing the list at > users-owner(a)ovirt.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of Users digest..." > > > Today's Topics: > > 1. Re: Multi-Host Network Configuration (Piotr Kliczewski) > 2. ovirt-3.3.3 release postponed due to blockers (Sandro Bonazzola) > 3. Re: Vm's being paused (Neil) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Tue, 28 Jan 2014 13:28:43 +0100 > From: Piotr Kliczewski <piotr.kliczewski(a)gmail.com> > To: Dan Kenigsberg <danken(a)redhat.com> > Cc: users(a)ovirt.org > Subject: Re: [Users] Multi-Host Network Configuration > Message-ID: > <CAKU0_rnOGobq35prWuaC+RVXEOQd00z3cjDAf4NEZG+3BSZKzg(a)mail.gmail.com> > Content-Type: text/plain; charset=ISO-8859-1 > > On Mon, Jan 27, 2014 at 7:20 PM, Dan Kenigsberg <danken(a)redhat.com> wrote: >> On Mon, Jan 27, 2014 at 04:23:47PM +0100, Piotr Kliczewski wrote: >>> Hi, >>> >>> I played with multi host network config and used two boxes to test it. >>> One el6 (vdsm) and f19 (engine, vdsm). During the test I noticed that >>> vdsm on f19 haven't joined to cluster (known issue). I performed >>> modification of vlan and MTU. Both boxes were modified but I noticed >>> that by accident I modified ovirtmgmt network and I noticed that I >>> lost connectivity. el6 box recovered whereas f19 haven't. I think it >>> was because the vdsm was local to the engine. >> That may be so. To make sure, I'd love to see your vdsm.log and >> supervdsm.log of the time of the modification. >> > I do not have the logs anymore but will try to recreate it and send > you the logs. > >>> I spend sometime trying >>> to recover network configuration. >> Thanks for testing this feature. > > ------------------------------ > > Message: 2 > Date: Tue, 28 Jan 2014 13:55:43 +0100 > From: Sandro Bonazzola <sbonazzo(a)redhat.com> > To: arch <arch(a)ovirt.org>, engine-devel <engine-devel(a)ovirt.org>, > "Users(a)ovirt.org" <Users(a)ovirt.org>, VDSM Project Development > <vdsm-devel(a)lists.fedorahosted.org> > Subject: [Users] ovirt-3.3.3 release postponed due to blockers > Message-ID: <52E7A8CF.6040301(a)redhat.com> > Content-Type: text/plain; charset=ISO-8859-15 > > Hi, > oVirt 3.3.3 release need to be postponed. > A recent release of python-cpopen-1.3 is breaking dependency resolution on EL6 and F19 so vdsm can't be installed and node can't be composed. > A recent change in vdsm-python-cpopen added the needed conflict against python-cpopen and removed Provides / Obsoletes on python-cpopen. > But no new releases of python-cpopen is available fixing the correct Provides / Obsoletes pair. > > oVirt 3.3.2 / stable is affected too. > > Please fix > Bug 1056470 - python-cpopen-1.3 is not providing obsoleted package. > Bug 1056464 - python-cpopen-1.3 is not providing obsoleted package. > ASAP. > > --------------070306080206060704030900 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit <html> <head> <meta content="text/html; charset=UTF-8" http-equiv="Content-Type"> </head> <body bgcolor="#FFFFFF" text="#000000"> Fnaly I fix my problem with following step 1) Check your maximum shared memory : $ <code>sudo cat /proc/sys/kernel/shmmax 33554432</code> You need to increase the shared memory to fix this. 2) To fix it temporarily, run this command : $ <code>sudo sysctl -w kernel.shmmax=134217728</code> 3) To fix this permenantly, set this in your configuration: $ <code>sudo vi /etc/sysctl.conf kernel.shmmax = 134217728 </code> <div class="moz-cite-prefix">On 28-01-2014 15:02, <a class="moz-txt-link-abbreviated" href="mailto:users-request@ovirt.org">users-request(a)ovirt.org</a> wrote: </div> <blockquote cite="mid:mailman.198.1390914150.1976.users@ovirt.org" type="cite"> <pre wrap="">Send Users mailing list submissions to <a class="moz-txt-link-abbreviated" href="mailto:users@ovirt.org">users(a)ovirt.org</a> To subscribe or unsubscribe via the World Wide Web, visit <a class="moz-txt-link-freetext" href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a> or, via email, send a message with subject or body 'help' to <a class="moz-txt-link-abbreviated" href="mailto:users-request@ovirt.org">users-request(a)ovirt.org</a> You can reach the person managing the list at <a class="moz-txt-link-abbreviated" href="mailto:users-owner@ovirt.org">users-owner(a)ovirt.org</a> When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..." Today's Topics: 1. Re: Multi-Host Network Configuration (Piotr Kliczewski) 2. ovirt-3.3.3 release postponed due to blockers (Sandro Bonazzola) 3. Re: Vm's being paused (Neil) ---------------------------------------------------------------------- Message: 1 Date: Tue, 28 Jan 2014 13:28:43 +0100 From: Piotr Kliczewski <a class="moz-txt-link-rfc2396E" href="mailto:piotr.kliczewski@gmail.com"><piotr.kliczewski(a)gmail.com></a> To: Dan Kenigsberg <a class="moz-txt-link-rfc2396E" href="mailto:danken@redhat.com"><danken(a)redhat.com></a> Cc: <a class="moz-txt-link-abbreviated" href="mailto:users@ovirt.org">users(a)ovirt.org</a> Subject: Re: [Users] Multi-Host Network Configuration Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:CAKU0_rnOGobq35prWuaC+RVXEOQd00z3cjDAf4NEZG+3BSZKzg@mail.gmail.com"><CAKU0_rnOGobq35prWuaC+RVXEOQd00z3cjDAf4NEZG+3BSZKzg(a)mail.gmail.com></a> Content-Type: text/plain; charset=ISO-8859-1 On Mon, Jan 27, 2014 at 7:20 PM, Dan Kenigsberg <a class="moz-txt-link-rfc2396E" href="mailto:danken@redhat.com"><danken(a)redhat.com></a> wrote: </pre> <blockquote type="cite"> <pre wrap="">On Mon, Jan 27, 2014 at 04:23:47PM +0100, Piotr Kliczewski wrote: </pre> <blockquote type="cite"> <pre wrap="">Hi, I played with multi host network config and used two boxes to test it. One el6 (vdsm) and f19 (engine, vdsm). During the test I noticed that vdsm on f19 haven't joined to cluster (known issue). I performed modification of vlan and MTU. Both boxes were modified but I noticed that by accident I modified ovirtmgmt network and I noticed that I lost connectivity. el6 box recovered whereas f19 haven't. I think it was because the vdsm was local to the engine. </pre> </blockquote> <pre wrap=""> That may be so. To make sure, I'd love to see your vdsm.log and supervdsm.log of the time of the modification. </pre> </blockquote> <pre wrap=""> I do not have the logs anymore but will try to recreate it and send you the logs. </pre> <blockquote type="cite"> <blockquote type="cite"> <pre wrap="">I spend sometime trying to recover network configuration. </pre> </blockquote> <pre wrap=""> Thanks for testing this feature. </pre> </blockquote> <pre wrap=""> ------------------------------ Message: 2 Date: Tue, 28 Jan 2014 13:55:43 +0100 From: Sandro Bonazzola <a class="moz-txt-link-rfc2396E" href="mailto:sbonazzo@redhat.com"><sbonazzo(a)redhat.com></a> To: arch <a class="moz-txt-link-rfc2396E" href="mailto:arch@ovirt.org"><arch(a)ovirt.org></a>, engine-devel <a class="moz-txt-link-rfc2396E" href="mailto:engine-devel@ovirt.org"><engine-devel(a)ovirt.org></a>, <a class="moz-txt-link-rfc2396E" href="mailto:Users@ovirt.org">"Users(a)ovirt.org"</a> <a class="moz-txt-link-rfc2396E" href="mailto:Users@ovirt.org"><Users(a)ovirt.org></a>, VDSM Project Development <a class="moz-txt-link-rfc2396E" href="mailto:vdsm-devel@lists.fedorahosted.org"><vdsm-devel(a)lists.fedorahosted.org></a> Subject: [Users] ovirt-3.3.3 release postponed due to blockers Message-ID: <a class="moz-txt-link-rfc2396E" href="mailto:52E7A8CF.6040301@redhat.com"><52E7A8CF.6040301(a)redhat.com></a> Content-Type: text/plain; charset=ISO-8859-15 Hi, oVirt 3.3.3 release need to be postponed. A recent release of python-cpopen-1.3 is breaking dependency resolution on EL6 and F19 so vdsm can't be installed and node can't be composed. A recent change in vdsm-python-cpopen added the needed conflict against python-cpopen and removed Provides / Obsoletes on python-cpopen. But no new releases of python-cpopen is available fixing the correct Provides / Obsoletes pair. oVirt 3.3.2 / stable is affected too. Please fix Bug 1056470 - python-cpopen-1.3 is not providing obsoleted package. Bug 1056464 - python-cpopen-1.3 is not providing obsoleted package. ASAP. </pre> </blockquote> </body> </html> --------------070306080206060704030900--

1 0

[Users] Multi-Host Network Configuration
by Piotr Kliczewski 28 Jan '14

28 Jan '14

Hi, I played with multi host network config and used two boxes to test it. One el6 (vdsm) and f19 (engine, vdsm). During the test I noticed that vdsm on f19 haven't joined to cluster (known issue). I performed modification of vlan and MTU. Both boxes were modified but I noticed that by accident I modified ovirtmgmt network and I noticed that I lost connectivity. el6 box recovered whereas f19 haven't. I think it was because the vdsm was local to the engine. I spend sometime trying to recover network configuration. Thanks, Piotr

2 2

[Users] Update request to support "Classless IN-ADDR.ARPA delegation"
by Michelle&Judie 28 Jan '14

28 Jan '14

Hi guys, I found a problem on installing server, which has address from small ip zone, less than a class C. "engine-setup" checks FQDN resolvability in "./plugins/ovirt-engine-setup/config/hostname.py". This program requests forward & reverse query to DNS, and see if it comes back to host domain name. But it does not support RFC2317("Classless IN-ADDR.ARPA delegation") type domain, and fails on FQDN check. See following URLs for RFC2317 explanation. http://support.simpledns.com/KB/a146/how-to-sub-delegate-a-reverse-zone.aspx http://www.rfc-editor.org/rfc/rfc2317.txt I made patch to "hostname.py" to clear this problem. Please check following patch, and consider upgrading on the next release. --- hostname.py.org 2013-12-13 17:22:11.000000000 +0900 +++ hostname.py 2014-01-28 18:01:09.514791351 +0900 @@ -96,15 +96,16 @@ flags=re.VERBOSE, pattern=r""" ^ - (?P<query>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).in-addr.arpa. + (?P<query>[\w/.-]+) + \. \s+ \d+ \s+ IN \s+ - PTR + (?P<type>(PTR|CNAME)) \s+ - (?P<answer>[\w.-]+) + (?P<answer>[\w/.-]+) \. $ """ @@ -171,11 +172,32 @@ args=args, raiseOnError=False ) + ad=addr.split('.') + query="{3}.{2}.{1}.{0}.in-addr.arpa".format(*ad) if rc == 0: for line in stdout: found = self._DIG_REVLOOKUP_RE.search(line) - if found: - names.add(found.group('answer')) + if not found: + continue + if query != found.group('query'): + continue + answer=found.group('answer') + if found.group('type') == 'CNAME': + self.logger.debug( + '{query} CNAME to: {answer}'.format( + query=query, + answer=answer, + ) + ) + query=answer + continue + self.logger.debug( + '{query} resolves to: {answer}'.format( + query=query, + answer=answer, + ) + ) + names.add(answer) return names def _validateFQDNresolvability(self, fqdn): Upgrading to a better patch is greatly appreciated. Thank you. Regards, Mark Kachi

2 2

[Users] oVirt 3.4 testing day report [configuration of max allowed downtime during live migration]
by Tal Nisan 28 Jan '14

28 Jan '14

This is a multi-part message in MIME format. --------------050907020607010606010307 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, During test day I've tested Martik Betak's feature "Enable configuration of maximum allowed downtime during live migration per guest" ( https://bugzilla.redhat.com/1055434 <https://bugzilla.redhat.com/show_bug.cgi?id=1055434>). For the tests I've created new VMs and tested the following: _Webadmin_ - All newly added components to the Virtual Machine popup under the Host tab appear correctly and well aligned with the rest of the components - OK - The checkbox component of "Use custom migration downtime" works correctly and grays out the input box when unchecked - OK - The data entered in the input box is validated for input that does not consist of numbers, empty value, a number not in the range or a negative number - OK - The valid number entered in the input is updated correctly in the database - OK - An existing migration downtime value is displayed correctly when editing the VM - OK - When unchecking the checkbox and saving the property in the database is nullified - OK _ __REST API_ - The valid number entered in the " migration_downtime" field is updated correctly in the database - OK - An existing migration downtime value is displayed correctly when GETting the VM - OK - A non existing migration downtime value of a VM is displayed as "-1" - OK - Updating an existing migration downtime of a VM to "-1" nullifies the value in the database - OK _VDSM_ The correct migration downtime value set in the VM is transferred correctly to "vmMigrate" command "downtime" parameter - OK Summary: All the tests went ok without failures but during the test I've found a bug regardless to the tests scope, when I tested a change of the migration downtime in the cluster level it turns our that the value set for the cluster custom properties via webadmin is not updated in the database, a bug was opened for that issue - https://bugzilla.redhat.com/1058703 <https://bugzilla.redhat.com/show_bug.cgi?id=1058703> --------------050907020607010606010307 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi, During test day I've tested Martik Betak's feature "Enable configuration of maximum allowed downtime during live migration per guest" ( <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1055434">https://bugzilla.redhat.com/1055434</a>). For the tests I've created new VMs and tested the following: Webadmin - All newly added components to the Virtual Machine popup under the Host tab appear correctly and well aligned with the rest of the components - OK - The checkbox component of "Use custom migration downtime" works correctly and grays out the input box when unchecked - OK - The data entered in the input box is validated for input that does not consist of numbers, empty value, a number not in the range or a negative number - OK - The valid number entered in the input is updated correctly in the database - OK - An  existing migration downtime value is displayed correctly when editing the VM - OK - When unchecking the checkbox and saving the property in the database is nullified - OK REST API - The valid number entered in the " <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> migration_downtime" field is updated correctly in the database - OK - An  existing migration downtime value is displayed correctly when GETting the VM - OK - A non existing migration downtime value of a VM is displayed as "-1" - OK - Updating an existing migration downtime of a VM to "-1" nullifies the value in the database - OK VDSM The correct migration downtime value set in the VM is transferred correctly to "vmMigrate" command "downtime" parameter - OK Summary: All the tests went ok without failures but during the test I've found a bug regardless to the tests scope, when I tested a change of the migration downtime in the cluster level it turns our that the value set for the cluster custom properties via webadmin is not updated in the database, a bug was opened for that issue - <meta http-equiv="content-type" content="text/html; charset=ISO-8859-1"> <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1058703">https://bugzilla.redhat.com/1058703</a> </body> </html> --------------050907020607010606010307--

1 0

[Users] oVirt 3.4 test day summary
by Doron Fediuck 28 Jan '14

28 Jan '14

Hi all, thanks for joining us to the first 3.4 test day. We all struggled with network bandwidth, and hopefully next testday will be better, so join us again for the next oVirt test day. Here are some statistics: Total of bugs opened: 108 Broken into areas: Unnamed 16 gluster 1 infra 21 integration 8 network 10 node 2 sla 3 storage 24 ux 1 virt 22 Top 3 reporters: amureini(a)redhat.com 20 emesika(a)redhat.com 7 oourfali(a)redhat.com 7 IRC participants, top 3: 115 sbonazzo 100 didi 81 OaaSvc Special contributors: Andrew Lau (andrew(a)andrewklau.com) for his work and reports on hosted engine. For their special contribution, Andrew Lau and OaaSvc will get special T-shirts of the oVirt community (please contact me offline). Thanks again and see you soon in the 2nd test day! Doron

2 2

[Users] Setup Networks: Unexpected exception
by Frank Wall 28 Jan '14

28 Jan '14

Hi, still testing 3.4 and I'm unable to save a node's network configuration in webadmin: Error while executing action Setup Networks: Unexpected exception My configuration on the ovirt node: - manually added net2 bridge, attached to eth1 My configuration in ovirt-engine webadmin: - added new network net2 - noticed that ovirt failed to find this network on node - tried to add net2 to node with "Setup Host Networks" Error in engine.log [1]. Error in vdsm.log [2]. I think it could be related to BZ 1054195: https://bugzilla.redhat.com/show_bug.cgi?id=1054195 ([NetworkLabels] Attaching two labeled networks to a cluster result in failure of the latter) I'm not sure, because I only wanted to add *one* new network. Please note that this is a self-hosted engine setup. Just in case this makes a difference... ovirt-engine: ovirt-engine-3.4.0-0.5.beta1.el6.noarch ovirt node: vdsm-4.14.1-17.gitcf59a55.el6.x86_64 ovirt-hosted-engine-setup-1.2.0-0.0.master.20140117.gitfaf77a5.el6.noarch Thanks - Frank [1] 2014-01-27 23:41:08,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--127.0.0.1-8702-10) [1 783e132] START, SetupNetworksVDSCommand(HostName = mgt03rn.example.com, HostId = a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, for ce=false, checkConnectivity=true, conectivityTimeout=120, networks=[net2 {id=db98fa95-e922-4060-8d39-f9ac0cb2f16f, description=Jumphost Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-000000000002, mtu=0, vmNetwork=true, cl uster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false} , providedBy=null, label=null, qosId=null}], bonds=[], interfaces=[bond001 {id=c5b50ccf-5b74-4737-b7cd-980c9c8acf51, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=bond001, macAddress=2c:44:fd:82:f5:5f, networkName=null, bondOptions=mode=802.3ad, bootProtocol=STATIC_IP, address=10.0.0.103, subnet=255.255.255.0, gateway=null, mtu=1500, bridged=false, type=0, networkImplementationDetails=null}, eth3 {id=7aaf1ac1-944a-4fe6-9d22-7dc41c6e275c, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth3, macAddress=2C:44:FD:82:F5:5F, networkName=null, bondName=bond001, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth4 {id=0c23834d-97ae-462a-9701-e89b3dc6a83a, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth4, macAddress=D8:9D:67:22:B6:4C, networkName=null, bondName=bond001, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth1 {id=54cb3cf6-c4bd-4907-bf28-9020022965d5, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth1, macAddress=2c:44:fd:82:f5:5d, networkName=net2, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=0, bridged=true, speed=1000, type=0, networkImplementationDetails=null}, eth2 {id=a53c448f-8061-460f-9c24-3081a2376de7, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth2, macAddress=2c:44:fd:82:f5:5e, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth5 {id=e9f15827-bb15-41d9-8ccc-49d812cde8a6, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth5, macAddress=d8:9d:67:22:b6:4d, networkName=null, bondName=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null}, eth0 {id=b4aea8bc-bdde-4e1e-a206-46ee853220c0, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth0, macAddress=2c:44:fd:82:f5:5c, networkName=ovirtmgmt, bondName=null, bootProtocol=STATIC_IP, address=10.0.0.103, subnet=255.255.0.0, gateway=10.0.0.1, mtu=1500, bridged=true, speed=1000, type=2, networkImplementationDetails={inSync=true, managed=true}}], removedNetworks=[], removedBonds=[]), log id: 78e823a3 2014-01-27 23:41:08,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--127.0.0.1-8702-10) [1783e132] FINISH, SetupNetworksVDSCommand, log id: 78e823a3 2014-01-27 23:41:09,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--127.0.0.1-8702-10) [1783e132] Failed in SetupNetworksVDS method 2014-01-27 23:41:09,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--127.0.0.1-8702-10) [1783e132] org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Unexpected exception, code = 16 2014-01-27 23:41:09,324 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (ajp--127.0.0.1-8702-10) [1783e132] Command SetupNetworksVDSCommand(HostName = mgt03rn.example.com, HostId = a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, force=false, checkConnectivity=true, conectivityTimeout=120, networks=[net2 {id=db98fa95-e922-4060-8d39-f9ac0cb2f16f, description=Jumphost Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=00000002-0002-0002-0002-000000000002, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false}, providedBy=null, label=null, qosId=null}], bonds=[], interfaces=[bond001 {id=c5b50ccf-5b74-4737-b7cd-980c9c8acf51, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=bond001, macAddress=2c:44:fd:82:f5:5f, networkName=null, bondOptions=mode=802.3ad, bootProtocol=STATIC_IP, address=10.0.0.103, subnet=255.255.255.0, gateway=null, mtu=1500, bridged=false, type=0, networkImplementationDetails=null}, eth3 {id=7aaf1ac1-944a-4fe6-9d22-7dc41c6e275c, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth3, macAddress=2C:44:FD:82:F5:5F, networkName=null, bondName=bond001, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth4 {id=0c23834d-97ae-462a-9701-e89b3dc6a83a, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth4, macAddress=D8:9D:67:22:B6:4C, networkName=null, bondName=bond001, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth1 {id=54cb3cf6-c4bd-4907-bf28-9020022965d5, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth1, macAddress=2c:44:fd:82:f5:5d, networkName=net2, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=0, bridged=true, speed=1000, type=0, networkImplementationDetails=null}, eth2 {id=a53c448f-8061-460f-9c24-3081a2376de7, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth2, macAddress=2c:44:fd:82:f5:5e, networkName=null, bondName=null, bootProtocol=NONE, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, eth5 {id=e9f15827-bb15-41d9-8ccc-49d812cde8a6, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth5, macAddress=d8:9d:67:22:b6:4d, networkName=null, bondName=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null}, eth0 {id=b4aea8bc-bdde-4e1e-a206-46ee853220c0, vdsId=a11f5383-b8f7-4bed-b1f3-3c7c46ecbe7a, name=eth0, macAddress=2c:44:fd:82:f5:5c, networkName=ovirtmgmt, bondName=null, bootProtocol=STATIC_IP, address=10.0.0.103, subnet=255.255.0.0, gateway=10.0.0.1, mtu=1500, bridged=true, speed=1000, type=2, networkImplementationDetails={inSync=true, managed=true}}], removedNetworks=[], removedBonds=[]) execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Unexpected exception, code = 16 2014-01-27 23:41:09,328 ERROR [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (ajp--127.0.0.1-8702-10) [1783e132] Command org.ovirt.engine.core.bll.network.host.SetupNetworksCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to SetupNetworksVDS, error = Unexpected exception, code = 16 (Failed with error unexpected and code 16) [2] Thread-119::DEBUG::2014-01-27 23:41:09,769::BindingXMLRPC::970::vds::(wrapper) client [10.0.0.12]::call setupNetworks with ({'net2': {'nic': 'eth1', 'STP': 'no', 'bridged': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {} flowID [1783e132] Thread-120::DEBUG::2014-01-27 23:41:09,770::BindingXMLRPC::970::vds::(wrapper) client [10.0.0.12]::call ping with () {} flowID [1783e132] Thread-120::DEBUG::2014-01-27 23:41:09,770::BindingXMLRPC::977::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}} Thread-95::DEBUG::2014-01-27 23:41:09,867::fileSD::225::Storage.Misc.excCmd::(getReadDelay) '/bin/dd iflag=direct if=/rhev/data-center/mnt/10.0.0.10:_vol__svm0__data__nfs__orgmgt/7ea2a278-7659-41d7-92c4-5d0f41a75bf8/dom_md/metadata bs=4096 count=1' (cwd None) Thread-95::DEBUG::2014-01-27 23:41:09,871::fileSD::225::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n512 bytes (512 B) copied, 0.000273105 s, 1.9 MB/s\n'; <rc> = 0 Thread-119::ERROR::2014-01-27 23:41:10,119::BindingXMLRPC::989::vds::(wrapper) unexpected error Traceback (most recent call last): File "/usr/share/vdsm/BindingXMLRPC.py", line 973, in wrapper res = f(*args, **kwargs) File "/usr/share/vdsm/BindingXMLRPC.py", line 402, in setupNetworks return api.setupNetworks(networks, bondings, options) File "/usr/share/vdsm/API.py", line 1294, in setupNetworks supervdsm.getProxy().setupNetworks(networks, bondings, options) File "/usr/share/vdsm/supervdsm.py", line 50, in __call__ return callMethod() File "/usr/share/vdsm/supervdsm.py", line 48, in <lambda> **kwargs) File "<string>", line 2, in setupNetworks File "/usr/lib64/python2.6/multiprocessing/managers.py", line 740, in _callmethod raise convert_to_error(kind, result) AttributeError: 'ConfigWriter' object has no attribute 'getEthtoolOpts'

5 7

[Users] oVirt 3.4 - testing days report [read only disks]
by Martin Perina 28 Jan '14

28 Jan '14

Hi, I tested read only disks feature [1] and it worked fine. I used this configuration: 1) RHEL 6 VM rhel1 with shared disk mounted as RW, 2) RHEL 6 VM rhel2 with shared disk mounted as RO 3) Windows 7 VM win71 with shared disk mounted as RO rhel2 and win71 VMs sees all files on shared disk, but the disk was read only for them. Martin [1] http://www.ovirt.org/Features/Read_Only_Disk

1 0

[Users] Strange UI bug: tab "VM" unusable / oVirt 3.3.1+
by Frank Wall 28 Jan '14

28 Jan '14

Hi, I'm experiencing a strange UI bug in the Amdin Portal. The "Virtual Machines" tab is unusable in serval ways: - when selecting a VM, all buttons are greyed-out, except "New VM" - you cannot power on, power off, open console or edit a VM - when selecting a VM, the VM-specific tabs do not appear on the bottom All other tabs, especially the "Hosts" tab are working as expected. Initially I suspected an incompatible browser on my side, so I tried with Firefox 25, Chrome and IE 11. No success. This UI bug hit me after upgrading from oVirt 3.3.0 to 3.3.1. And it is still reproducable in oVirt 3.3.2 BETA1. Any hint on this? Or any advice how to provide some useful debug output? I have to mention that I am able to workaround this bug: - select a VM - right click, choose "New VM", click "Cancel" - now most VM actions are no longer greyed-out - power on the VM - still the "console" button remains greyed-out Thanks - Frank

5 19

[Users] OVirt 3.3.2 Snapshot Pane empty in Firefox
by Markus Stockhausen 28 Jan '14

28 Jan '14

------=_NextPartTM-000-7b46d2aa-6d60-401b-93f2-4073fbfdcf92 Content-Type: multipart/alternative; boundary="_000_12EF8D94C6F8734FB2FF37B9FBEDD173585DB468EXCHANGEcollogi_" --_000_12EF8D94C6F8734FB2FF37B9FBEDD173585DB468EXCHANGEcollogi_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello, I had a mysterios behaviour in the webinterface twice this week. The Snapshot list of a VM remained empty in Firefox although I know that snapshots exist. The "Create Snapshot" button was still working and issued the right commands. Nevertheless the list remained empty. Opening an IE session prooved that everything was ok. The pane was populated with the list of snapshots. After restarting Firefox everything was fine again. Has anybody experienced simialr issues and if yes is there already an opn BZ for that? Markus --_000_12EF8D94C6F8734FB2FF37B9FBEDD173585DB468EXCHANGEcollogi_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html dir=3D"ltr"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style id=3D"owaParaStyle" type=3D"text/css">P {margin-top:0;margin-bottom:= 0;}</style> </head> <body ocsi=3D"0" fpstyle=3D"1"> <div style=3D"direction: ltr;font-family: Tahoma;color: #000000;font-size: = 10pt;">Hello, I had a mysterios behaviour in the webinterface twice this week. The Snapshot list of a VM remained empty in Firefox although I know that snapshots exist. The "Create Snapshot" button was still working and issued the right commands. Nevertheless the list remained empty. Opening an IE session prooved that everything was ok. The pane was populated with the list of snapshots. After restarting Firefox everything was fine again. Has anybody experienced simialr issues and if yes is there already an opn BZ for that? Markus </div> </body> </html> --_000_12EF8D94C6F8734FB2FF37B9FBEDD173585DB468EXCHANGEcollogi_-- ------=_NextPartTM-000-7b46d2aa-6d60-401b-93f2-4073fbfdcf92 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-7b46d2aa-6d60-401b-93f2-4073fbfdcf92--

3 4

Re: [Users] Data Center stuck between "Non Responsive" and "Contending"
by Ted Miller 28 Jan '14

28 Jan '14

On 1/26/2014 4:00 PM, Itamar Heim wrote: > On 01/26/2014 10:51 PM, Ted Miller wrote: >> >> On 1/26/2014 3:10 PM, Itamar Heim wrote: >>> On 01/26/2014 10:08 PM, Ted Miller wrote: >>>> My Data Center is down, and won't come back up. >>>> >>>> Data Center Status on the GUI flips between "Non Responsive" and >>>> "Contending" >>>> >>>> Also noted: >>>> Host sometimes seen flipping between "Low" and "Contending" in SPM >>>> column. >>>> Storage VM2 "Data (Master)" is in "Cross Data-Center Status" = Unknown >>>> VM2 is "up" under "Volumes" tab >>>> >>>> Created another volume for VM storage. It shows up in "volumes" tab, >>>> but when I try to add "New Domain" in storage tab, says that "There are >>>> No Data Centers to which the Storage Domain can be attached" >>>> >>>> Setup: >>>> 2 hosts w/ glusterfs storage >>>> 1 engine >>>> all 3 computers Centos 6.5, just updated >>>> ovirt-engine 3.3.0.1-1.el6 >>>> ovirt-engine-lib 3.3.2-1.el6 >>>> ovirt-host-deploy.noarch 1.1.3-1.el6 >>>> glusterfs.x86_64 3.4.2-1.el6 >>>> >>>> This loop seems to repeat in the ovirt-engine log (grep of log showing >>>> only DefaultQuartzScheduler_Worker-79 thread: >>>> >>>> 2014-01-26 14:44:58,416 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] >>>> (DefaultQuartzScheduler_Worker-79) Irs placed on server >>>> 9a591103-83be-4ca9-b207-06929223b541 failed. Proceed Failover >>>> 2014-01-26 14:44:58,511 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] >>>> (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - office4a, >>>> spmStatus Free, storage pool mill >>>> 2014-01-26 14:44:58,550 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] >>>> (DefaultQuartzScheduler_Worker-79) SpmStatus on vds >>>> 127ed939-34af-41a8-87a0-e2f6174b1877: Free >>>> 2014-01-26 14:44:58,571 INFO >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] >>>> (DefaultQuartzScheduler_Worker-79) starting spm on vds office4a, storage >>>> pool mill, prevId 2, LVER 15 >>>> 2014-01-26 14:44:58,579 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName = >>>> office4a, HostId = 127ed939-34af-41a8-87a0-e2f6174b1877, storagePoolId = >>>> 536a864d-83aa-473a-a675-e38aafdd9071, prevId=2, prevLVER=15, >>>> storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log >>>> id: 74c38eb7 >>>> 2014-01-26 14:44:58,617 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) spmStart polling started: taskId = >>>> e8986753-fc80-4b11-a11d-6d3470b1728c >>>> 2014-01-26 14:45:00,662 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method >>>> 2014-01-26 14:45:00,664 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) Error code AcquireHostIdFailure and >>>> error message VDSGenericException: VDSErrorException: Failed to >>>> HSMGetTaskStatusVDS, error = Cannot acquire host id >>>> 2014-01-26 14:45:00,665 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId = >>>> e8986753-fc80-4b11-a11d-6d3470b1728c task status = finished >>>> 2014-01-26 14:45:00,666 ERROR >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result: >>>> cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to >>>> HSMGetTaskStatusVDS, error = Cannot acquire host id >>>> 2014-01-26 14:45:00,695 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm >>>> status: Free >>>> 2014-01-26 14:45:00,702 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) START, >>>> HSMClearTaskVDSCommand(HostName = office4a, HostId = >>>> 127ed939-34af-41a8-87a0-e2f6174b1877, >>>> taskId=e8986753-fc80-4b11-a11d-6d3470b1728c), log id: 336ec5a6 >>>> 2014-01-26 14:45:00,722 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log >>>> id: 336ec5a6 >>>> 2014-01-26 14:45:00,724 INFO >>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] >>>> (DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return: >>>> org.ovirt.engine.core.common.businessentities.SpmStatusResult@13652652, >>>> log id: 74c38eb7 >>>> 2014-01-26 14:45:00,733 INFO >>>> [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] >>>> (DefaultQuartzScheduler_Worker-79) Running command: >>>> SetStoragePoolStatusCommand internal: true. Entities affected : ID: >>>> 536a864d-83aa-473a-a675-e38aafdd9071 Type: StoragePool >>>> 2014-01-26 14:45:00,778 ERROR >>>> [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] >>>> (DefaultQuartzScheduler_Worker-79) >>>> IrsBroker::Failed::GetStoragePoolInfoVDS due to: >>>> IrsSpmStartFailedException: IRSGenericException: IRSErrorException: >>>> SpmStart failed >>>> >>>> Ted Miller >>>> Elkhart, IN, USA >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> >>> is this gluster storage (guessing sunce you mentioned a 'volume') >> yes (mentioned under "setup" above) >>> does it have a quorum? >> Volume Name: VM2 >> Type: Replicate >> Volume ID: 7bea8d3b-ec2a-4939-8da8-a82e6bda841e >> Status: Started >> Number of Bricks: 1 x 3 = 3 >> Transport-type: tcp >> Bricks: >> Brick1: 10.41.65.2:/bricks/01/VM2 >> Brick2: 10.41.65.4:/bricks/01/VM2 >> Brick3: 10.41.65.4:/bricks/101/VM2 >> Options Reconfigured: >> cluster.server-quorum-type: server >> storage.owner-gid: 36 >> storage.owner-uid: 36 >> auth.allow: * >> user.cifs: off >> nfs.disa >>> (there were reports of split brain on the domain metadata before when >>> no quorum exist for gluster) >> after full heal: >> >> [root@office4a ~]$ gluster volume heal VM2 info >> Gathering Heal info on volume VM2 has been successful >> >> Brick 10.41.65.2:/bricks/01/VM2 >> Number of entries: 0 >> >> Brick 10.41.65.4:/bricks/01/VM2 >> Number of entries: 0 >> >> Brick 10.41.65.4:/bricks/101/VM2 >> Number of entries: 0 >> [root@office4a ~]$ gluster volume heal VM2 info split-brain >> Gathering Heal info on volume VM2 has been successful >> >> Brick 10.41.65.2:/bricks/01/VM2 >> Number of entries: 0 >> >> Brick 10.41.65.4:/bricks/01/VM2 >> Number of entries: 0 >> >> Brick 10.41.65.4:/bricks/101/VM2 >> Number of entries: 0 >> >> noticed this in host /var/log/messages (while looking for something >> else). Loop seems to repeat over and over. >> >> Jan 26 15:35:52 office4a sanlock[3763]: 2014-01-26 15:35:52-0500 14678 >> [30419]: read_sectors delta_leader offset 512 rv -90 >> /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids >> >> Jan 26 15:35:53 office4a sanlock[3763]: 2014-01-26 15:35:53-0500 14679 >> [3771]: s1997 add_lockspace fail result -90 >> Jan 26 15:35:58 office4a vdsm TaskManager.Task ERROR >> Task=`89885661-88eb-4ea3-8793-00438735e4ab`::Unexpected error#012Traceback >> (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line >> 857, in _run#012 return fn(*args, **kargs)#012 File >> "/usr/share/vdsm/logUtils.py", line 45, in wrapper#012 res = f(*args, >> **kwargs)#012 File "/usr/share/vdsm/storage/hsm.py", line 2111, in >> getAllTasksStatuses#012 allTasksStatus = sp.getAllTasksStatuses()#012 >> File "/usr/share/vdsm/storage/securable.py", line 66, in wrapper#012 raise >> SecureError()#012SecureError >> Jan 26 15:35:59 office4a sanlock[3763]: 2014-01-26 15:35:59-0500 14686 >> [30495]: read_sectors delta_leader offset 512 rv -90 >> /rhev/data-center/mnt/glusterSD/10.41.65.2:VM2/0322a407-2b16-40dc-ac67-13d387c6eb4c/dom_md/ids >> >> Jan 26 15:36:00 office4a sanlock[3763]: 2014-01-26 15:36:00-0500 14687 >> [3772]: s1998 add_lockspace fail result -90 >> Jan 26 15:36:00 office4a vdsm TaskManager.Task ERROR >> Task=`8db9ff1a-2894-407a-915a-279f6a7eb205`::Unexpected error#012Traceback >> (most recent call last):#012 File "/usr/share/vdsm/storage/task.py", line >> 857, in _run#012 return fn(*args, **kargs)#012 File >> "/usr/share/vdsm/storage/task.py", line 318, in run#012 return >> self.cmd(*self.argslist, **self.argsdict)#012 File >> "/usr/share/vdsm/storage/sp.py", line 273, in startSpm#012 >> self.masterDomain.acquireHostId(self.id)#012 File >> "/usr/share/vdsm/storage/sd.py", line 458, in acquireHostId#012 >> self._clusterLock.acquireHostId(hostId, async)#012 File >> "/usr/share/vdsm/storage/clusterlock.py", line 189, in >> acquireHostId#012 raise se.AcquireHostIdFailure(self._sdUUID, >> e)#012AcquireHostIdFailure: Cannot acquire host id: >> ('0322a407-2b16-40dc-ac67-13d387c6eb4c', SanlockException(90, 'Sanlock >> lockspace add failure', 'Message too long')) >> >> Ted Miller >> Elkhart, IN, USA >> > > this is the new storage domain? what about the previous volume for the > first SD? The default/default data center/cluster had to be abandoned because of a split-brain that could not be healed. Can't remove old storage from database, can't get data center up due to corrupt storage, ends up a circular argument. I started over with same hosts, totally new storage in new data center. This mill/one data center/cluster was working fine with VM2 storage, then died. Ted Miller

4 8