Re: [ovirt-users] Unable to reinstall hosts after network removal.

From: "Arman Khalatyan" <arm2arm(a)gmail.com&gt; To: "Moti Asayag" <masayag(a)redhat.com&gt; Cc: "users" <users(a)ovirt.org&gt; Sent: Tuesday, December 30, 2014 2:21:26 PM Subject: Re: [ovirt-users] Unable to reinstall hosts after network removal. Ok I found the conflict. The ip rule list does not have any record on IB0 interface, all other networks are updated correctly but not IB one. Probably this is not a good idea to have an ISCSI storage devices with the same interface as VM network. the network update operation trying to put interface up and down in that point all iscsi devices going to IO/failure this stops to update IB interface data. Maybe one need to put whole cluster to maintenance mode in order to change Storage dependent network setup?

this is the main trouble which prevents to fix/change any network setup on IB0 interface. tail -f /var/log/vdsm/vdsm.log Traceback (most recent call last): File "/usr/share/vdsm/storage/hsm.py", line 2430, in connectStorageServer conObj.connect() File "/usr/share/vdsm/storage/storageServer.py", line 367, in connect iscsi.addIscsiNode(self._iface, self._target, self._cred) File "/usr/share/vdsm/storage/iscsi.py", line 164, in addIscsiNode iscsiadm.node_login(iface.name, portalStr, target.iqn) File "/usr/share/vdsm/storage/iscsiadm.py", line 295, in node_login raise IscsiNodeError(rc, out, err) IscsiNodeError: (8, ['Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260] (multiple)'], ['iscsiadm: Could not login to [iface: default, target: iqn.2003-01.org.linux-iscsi.data01.x8664:sn.f5c9b017e103, portal: 10.10.10.31,3260].', 'iscsiadm: initiator reported error (8 - connection timed out)', 'iscsiadm: Could not log into all portals']) PS "save network configuration" is always checked. *********************************************************** Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany *********************************************************** On Tue, Dec 30, 2014 at 7:05 AM, Moti Asayag <masayag(a)redhat.com&gt; wrote: > > > ----- Original Message ----- > > From: "Arman Khalatyan" <arm2arm(a)gmail.com&gt; > > To: "Moti Asayag" <masayag(a)redhat.com&gt; > > Cc: "users" <users(a)ovirt.org&gt; > > Sent: Monday, December 29, 2014 3:59:46 PM > > Subject: Re: [ovirt-users] Unable to reinstall hosts after network > removal. > > > > My setup has 3 networks: 1xIB,1x10Gbit+1Gbit for ovirt-management. > > The ovirt network does not have any trouble, it is always there. > > I was trying to rename or remove my IB network which was used for VM > > migrations. > > I was using web-GUI, which was removing IB0 network w/o problem. After > > removal the hosts where ok. > > Have you checked the option of "save network configuration" in the setup > networks > dialog (iirc should have been checked by default) ? > > Could you also attach the /var/log/ovirt-engine/engine.log from the engine > server > and /var/log/vdsm/vdsm.log and /var/log/vdsm/supervdsm.log from the node so > we can see which request was sent to vdsm and its result ? > > > Then I put them to maintenance mode. To refresh > > iptables rules I did reinstall. then reinstall was failing with message > > that IB0 not attached to any interface. But IB0 interface is not possible > > to attach it is already deleted and not visible in any network dialog. > > After creating interface with the same name everything is online now. > > My current interface list is following: > > virsh -r net-list > > Name State Autostart Persistent > > -------------------------------------------------- > > ;vdsmdummy; active no no > > vdsm-cls10G active yes yes > > vdsm-IB0 active yes yes > > vdsm-ovirtmgmt active yes yes > > > > On this host the IB0 is not atached to any interface, I wondered, if it > > should show up in the net list? > > > > I think the GUI does not rename/remove the interface from the DB. Some > > constrain keeps still IB0 in DB. > > > > > > > > *********************************************************** > > > > Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für > > Astrophysik Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany > > > > *********************************************************** > > > > > > On Sun, Dec 28, 2014 at 10:31 AM, Moti Asayag <masayag(a)redhat.com&gt; > wrote: > > > > > > > > > > > ----- Original Message ----- > > > > From: "Arman Khalatyan" <arm2arm(a)gmail.com&gt; > > > > To: "users" <users(a)ovirt.org&gt; > > > > Sent: Wednesday, December 24, 2014 1:22:43 PM > > > > Subject: [ovirt-users] Unable to reinstall hosts after network > removal. > > > > > > > > Hello, > > > > I have a little trouble with ovirt 3.5 on CentOS6.6: > > > > I was removing all networks from all hosts. > > > > > > Did you use the setup networks dialog from the UI in order to remove > those > > > networks ? > > > Or have you removed those networks from the host directly (where you > > > should used the: > > > 1. virsh net-destroy 'the-network-name' > > > 2. virsh net-undefine 'the-network-name' > > > ) > > > > > > can you report the output of 'virsh -r net-list' ? > > > > > > > Then after removing network from data center the hosts went to > unusable. > > > > > > What was the host's status prior to removing its networks ? Was it up ? > > > > > > > Every time after reinstall the host claims that the network is not > > > > configured, but it s already removed from network tab in DC. > > > > > > What is the missing network name ? Is it 'ovirtmgmt' ? > > > > > > > Where from it gets the old configuration? the old interfaces also > > > restored > > > > every time on the reinstalled hosts. > > > > > > The hosts via vdsm reports their network configuration via the > > > 'getCapabilities' verb > > > of vdsm. You can try running it on the host: > > > > > > vdsClient -s 0 getVdsCaps > > > > > > and examine the nics / neworks / bridges / vlans / bonds elements. > > > > > > > Which DB table is in charge of dc-networks? > > > > > > > > > > The retrieved information from vdsm is reported to 'vds_interace' > table. > > > The dc networks are stored in 'networks' table and networks attached to > > > clusters are stored in network_cluster table. > > > > > > I wouldn't recommend on deleting entries from the tables directly. > There > > > are > > > certain constraints which shouldn't be violated, i.e. the management > > > network > > > 'ovirtmgnt' is blocked for removal from the engine. > > > > > > > Thanks, > > > > Arman. > > > > > > > > > > > > *********************************************************** > > > > Dr. Arman Khalatyan eScience -SuperComputing Leibniz-Institut für > > > Astrophysik > > > > Potsdam (AIP) An der Sternwarte 16, 14482 Potsdam, Germany > > > > *********************************************************** > > > > > > > > _______________________________________________ > > > > Users mailing list > > > > Users(a)ovirt.org > > > > http://lists.ovirt.org/mailman/listinfo/users > > > > > > > > > >