March 2014 - Users - oVirt List Archives

Re: [Users] method "glusterHostsList" is not supported
by Itamar Heim 04 Mar '14

04 Mar '14

On 03/04/2014 11:15 AM, René Koch wrote: > On 03/04/2014 09:40 AM, Itamar Heim wrote: >> On 03/04/2014 10:16 AM, René Koch wrote: >>> On 03/04/2014 05:14 AM, Sahina Bose wrote: >>>> >>>> On 03/04/2014 01:59 AM, Itamar Heim wrote: >>>>> On 03/03/2014 07:26 PM, René Koch wrote: >>>>>> Hi list, >>>>>> >>>>>> My hosted engine is running again, so I want to start a new thread >>>>>> for >>>>>> another issue with my setup. >>>>>> >>>>>> I have a GlusterFS storage domain, which can be mounted from CLI >>>>>> without >>>>>> problems. oVirt is 3.4 from ovirt-3.4.0-prerelease repository >>>>>> running on >>>>>> CentOS 6.5 with latest updates (both OS and oVirt). >>>>>> >>>>>> Both hosts, which act as hypervisors and GlusterFS nodes are in state >>>>>> "Non Operational" in oVirt because "Gluster command [Non interactive >>>>>> user] failed on server ovirt-host02.dmz.linuxland.at." >>>>>> >>>>>> In engine.log I see the entry "glusterHostList is not supported" >>>>>> (attached are the log entries when activating one of the hosts): >>>>>> >>>>>> 2014-03-03 18:17:11,764 ERROR >>>>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>>>>> >>>>>> (org.ovirt.thread.pool-6-thread-21) [6eee3cbd] Command >>>>>> GlusterServersListVDSCommand(HostName = >>>>>> ovirt-host02.dmz.linuxland.at, >>>>>> HostId = dd399eeb-f623-457a-9986-a7efc69010b2) execution failed. >>>>>> Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: >>>>>> <type >>>>>> 'exceptions.Exception'>:method "glusterHostsList" is not supported >>>>>> >>>>>> Can you give me a hint what this means and how I can activate my >>>>>> hosts >>>>>> and storage again? >>>>>> Thanks a lot! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>> >>>>> sahina ? >>>> >>>> Do you have vdsm-gluster on the node? >>> >>> No, I didn't had it on (both) nodes. >>> >>> After installing vdsm-gluster the storage works fine again. Thanks a >>> lot! >>> >>> But there's one strange thing. According to oVirt logs and yum.log >>> GlusterFS storage worked fine until yesterdays "yum update", where the >>> following packages where updated: >>> >>> Mar 03 10:01:09 Updated: ovirt-hosted-engine-ha-1.1.0-1.el6.noarch >>> Mar 03 10:01:10 Updated: otopi-1.2.0-0.5.rc.el6.noarch >>> Mar 03 10:01:11 Updated: ovirt-engine-sdk-python-3.4.0.6-1.el6.noarch >>> Mar 03 10:01:12 Updated: ovirt-hosted-engine-setup-1.1.0-1.el6.noarch >>> Mar 03 10:01:13 Updated: libtiff-3.9.4-10.el6_5.x86_64 >>> >>> According to yum.log vdsm-gluster was never installed on these hosts, >>> but storage did work. >>> >>> Shouldn't vdsm-gluster be a requirement for hosts and therefor be >>> installed during host setup? >>> >>> Do you have any clue why it storage did work until the update of these >>> packages? >> >> vdsm-gluster is to manage gluster storage, not consume gluster storage. >> did you enable the gluster mode post installing the hosts? >> (if you enabled it before installing them, or re-install[1] post >> enabling gluster mode, it should have deployed vdsm-gluster as well) >> >> [1] after moving hos to maintenance. > > Yes, I did it after installing the hosts. > > Would it be possible to add a check before activating this option? > > I'm thinking of the following: > - Edit Cluster > - Enable Gluster Service > - OK checks if all hosts in the cluster have vdsm-gluster installed - if > not an error message occurs that this package is required in order to > enable the gluster service. makes sense - please open a bug to track. check would probably warn if unreachable hosts exists to warn user.

2 1

Re: [Users] method "glusterHostsList" is not supported
by Itamar Heim 04 Mar '14

04 Mar '14

On 03/04/2014 10:16 AM, René Koch wrote: > On 03/04/2014 05:14 AM, Sahina Bose wrote: >> >> On 03/04/2014 01:59 AM, Itamar Heim wrote: >>> On 03/03/2014 07:26 PM, René Koch wrote: >>>> Hi list, >>>> >>>> My hosted engine is running again, so I want to start a new thread for >>>> another issue with my setup. >>>> >>>> I have a GlusterFS storage domain, which can be mounted from CLI >>>> without >>>> problems. oVirt is 3.4 from ovirt-3.4.0-prerelease repository >>>> running on >>>> CentOS 6.5 with latest updates (both OS and oVirt). >>>> >>>> Both hosts, which act as hypervisors and GlusterFS nodes are in state >>>> "Non Operational" in oVirt because "Gluster command [Non interactive >>>> user] failed on server ovirt-host02.dmz.linuxland.at." >>>> >>>> In engine.log I see the entry "glusterHostList is not supported" >>>> (attached are the log entries when activating one of the hosts): >>>> >>>> 2014-03-03 18:17:11,764 ERROR >>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>>> (org.ovirt.thread.pool-6-thread-21) [6eee3cbd] Command >>>> GlusterServersListVDSCommand(HostName = ovirt-host02.dmz.linuxland.at, >>>> HostId = dd399eeb-f623-457a-9986-a7efc69010b2) execution failed. >>>> Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: >>>> <type >>>> 'exceptions.Exception'>:method "glusterHostsList" is not supported >>>> >>>> Can you give me a hint what this means and how I can activate my hosts >>>> and storage again? >>>> Thanks a lot! >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> >>> sahina ? >> >> Do you have vdsm-gluster on the node? > > No, I didn't had it on (both) nodes. > > After installing vdsm-gluster the storage works fine again. Thanks a lot! > > But there's one strange thing. According to oVirt logs and yum.log > GlusterFS storage worked fine until yesterdays "yum update", where the > following packages where updated: > > Mar 03 10:01:09 Updated: ovirt-hosted-engine-ha-1.1.0-1.el6.noarch > Mar 03 10:01:10 Updated: otopi-1.2.0-0.5.rc.el6.noarch > Mar 03 10:01:11 Updated: ovirt-engine-sdk-python-3.4.0.6-1.el6.noarch > Mar 03 10:01:12 Updated: ovirt-hosted-engine-setup-1.1.0-1.el6.noarch > Mar 03 10:01:13 Updated: libtiff-3.9.4-10.el6_5.x86_64 > > According to yum.log vdsm-gluster was never installed on these hosts, > but storage did work. > > Shouldn't vdsm-gluster be a requirement for hosts and therefor be > installed during host setup? > > Do you have any clue why it storage did work until the update of these > packages? vdsm-gluster is to manage gluster storage, not consume gluster storage. did you enable the gluster mode post installing the hosts? (if you enabled it before installing them, or re-install[1] post enabling gluster mode, it should have deployed vdsm-gluster as well) [1] after moving hos to maintenance.

2 1

Re: [Users] method "glusterHostsList" is not supported
by Sahina Bose 04 Mar '14

04 Mar '14

On 03/04/2014 01:46 PM, René Koch wrote: > On 03/04/2014 05:14 AM, Sahina Bose wrote: >> >> On 03/04/2014 01:59 AM, Itamar Heim wrote: >>> On 03/03/2014 07:26 PM, René Koch wrote: >>>> Hi list, >>>> >>>> My hosted engine is running again, so I want to start a new thread for >>>> another issue with my setup. >>>> >>>> I have a GlusterFS storage domain, which can be mounted from CLI >>>> without >>>> problems. oVirt is 3.4 from ovirt-3.4.0-prerelease repository >>>> running on >>>> CentOS 6.5 with latest updates (both OS and oVirt). >>>> >>>> Both hosts, which act as hypervisors and GlusterFS nodes are in state >>>> "Non Operational" in oVirt because "Gluster command [Non interactive >>>> user] failed on server ovirt-host02.dmz.linuxland.at." >>>> >>>> In engine.log I see the entry "glusterHostList is not supported" >>>> (attached are the log entries when activating one of the hosts): >>>> >>>> 2014-03-03 18:17:11,764 ERROR >>>> [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] >>>> (org.ovirt.thread.pool-6-thread-21) [6eee3cbd] Command >>>> GlusterServersListVDSCommand(HostName = ovirt-host02.dmz.linuxland.at, >>>> HostId = dd399eeb-f623-457a-9986-a7efc69010b2) execution failed. >>>> Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: >>>> <type >>>> 'exceptions.Exception'>:method "glusterHostsList" is not supported >>>> >>>> Can you give me a hint what this means and how I can activate my hosts >>>> and storage again? >>>> Thanks a lot! >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Users mailing list >>>> Users(a)ovirt.org >>>> http://lists.ovirt.org/mailman/listinfo/users >>>> >>> >>> sahina ? >> >> Do you have vdsm-gluster on the node? > > No, I didn't had it on (both) nodes. > > After installing vdsm-gluster the storage works fine again. Thanks a lot! > > But there's one strange thing. According to oVirt logs and yum.log > GlusterFS storage worked fine until yesterdays "yum update", where the > following packages where updated: > > Mar 03 10:01:09 Updated: ovirt-hosted-engine-ha-1.1.0-1.el6.noarch > Mar 03 10:01:10 Updated: otopi-1.2.0-0.5.rc.el6.noarch > Mar 03 10:01:11 Updated: ovirt-engine-sdk-python-3.4.0.6-1.el6.noarch > Mar 03 10:01:12 Updated: ovirt-hosted-engine-setup-1.1.0-1.el6.noarch > Mar 03 10:01:13 Updated: libtiff-3.9.4-10.el6_5.x86_64 > > According to yum.log vdsm-gluster was never installed on these hosts, > but storage did work. > > Shouldn't vdsm-gluster be a requirement for hosts and therefor be > installed during host setup? > > Do you have any clue why it storage did work until the update of these > packages? The host moving to Non-Operational state with error "Gluster command failed..." is dependent on whether "Enable gluster service" is checked on your cluster. This check indicates that you also want to manage gluster storage provisioning on the nodes. A recent change now checks that vdsm-gluster support is available for such clusters. That's probably why you are seeing this error after update. > > > Regards, > René >

2 1

[Users] method "glusterHostsList" is not supported
by René Koch 04 Mar '14

04 Mar '14

Hi list, My hosted engine is running again, so I want to start a new thread for another issue with my setup. I have a GlusterFS storage domain, which can be mounted from CLI without problems. oVirt is 3.4 from ovirt-3.4.0-prerelease repository running on CentOS 6.5 with latest updates (both OS and oVirt). Both hosts, which act as hypervisors and GlusterFS nodes are in state "Non Operational" in oVirt because "Gluster command [Non interactive user] failed on server ovirt-host02.dmz.linuxland.at." In engine.log I see the entry "glusterHostList is not supported" (attached are the log entries when activating one of the hosts): 2014-03-03 18:17:11,764 ERROR [org.ovirt.engine.core.vdsbroker.gluster.GlusterServersListVDSCommand] (org.ovirt.thread.pool-6-thread-21) [6eee3cbd] Command GlusterServersListVDSCommand(HostName = ovirt-host02.dmz.linuxland.at, HostId = dd399eeb-f623-457a-9986-a7efc69010b2) execution failed. Exception: VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterHostsList" is not supported Can you give me a hint what this means and how I can activate my hosts and storage again? Thanks a lot! -- Best Regards René Koch Senior Solution Architect ============================================ LIS-Linuxland GmbH Brünner Straße 163, A-1210 Vienna Phone: +43 1 236 91 60 Mobile: +43 660 / 512 21 31 E-Mail: rkoch(a)linuxland.at ============================================

3 3

[Users] Creating Logical Network issue - API
by Tejesh M 04 Mar '14

04 Mar '14

Hi , I'm trying to create Logical Network and assign it to cluster & want to bond this to one of the Network Interface on Host. so that i can isolate my VM from other networks. The issue is even if i set Cluster while creating Logical Network, it is not setting the cluster. In Web Admin it is in unset state. Also, let me know how do i bind this Logical network on Host Interface. *Java code:* Network nw1=new Network(); VLAN vlan = new VLAN(); vlan.setId(2000); nw1.setVlan(vlan); IP ip = new IP(); ip.setAddress("192.168.1.151"); ip.setGateway("192.168.1.1"); ip.setNetmask("255.255.255.0"); nw1.setIp(ip); nw1.setName("apiNetwork"); nw1.setDataCenter(api.getDataCenters().get("testDC")); nw1.setCluster(api.getClusters().get("testCluster")); org.ovirt.engine.sdk.decorators.Network nw2 = api.getNetworks().add(nw1); -- Thanks & Regards Tejesh

2 15

[Users] help requested
by aditya mamidwar 03 Mar '14

03 Mar '14

hey, i want to study how ovirt implements the ISCSI protocol for disk access in SANs, i want to study the source code files. from where can i get them? am thinking of implementing the aoe protocol. also can i get help on how such plugins are made for ovirt, which files need to be changed, and procedure to test the plugin by building the engine. -- -Aditya Mamidwar

3 2

[Users] hosted engine setup (network)
by René Koch 03 Mar '14

03 Mar '14

Hi, During hosted engine setup I faced issues with network configuration on my CentOS 6.5 hosts: My network setup was the following which doesn't work with hosted engine setup: - eth0.200 (VLAN tag 200) should become ovirtmgmt interface - GATEWAY is in /etc/sysconfig/network and not in ifcfg-eth0.200 The following errors occured: 014-02-25 15:42:31 INFO otopi.plugins.ovirt_hosted_engine_setup.network.bridge bridge._misc:189 Configuring the management bridge 2014-02-25 15:42:31 DEBUG otopi.plugins.ovirt_hosted_engine_setup.network.bridge bridge._misc:198 getVdsCaps for eth0: {'addr': '', 'cfg': {'DEVICE': 'eth0', 'PEERDNS': 'no', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'none', 'HWADDR': '00:25:9 0:75:87:C2', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::225:90ff:fe75:87c2/64'], 'mtu': '1500', 'netmask': '', 'hwaddr': '00:25:90:75:87:c2', 'speed': 1000} 2014-02-25 15:42:31 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/network/bridge.py", line 231, in _misc 'gateway=%s' % caps['cfg']['GATEWAY'], KeyError: 'GATEWAY' This can be solved by setting GATEWAY in ifcfg-eth* file instead of /etc/sysconfig/network Next, VLAN tagged interfaces don't work: 2014-02-25 15:35:20 INFO otopi.plugins.ovirt_hosted_engine_setup.network.bridge bridge._misc:189 Configuring the management bridge 2014-02-25 15:35:20 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/network/bridge.py", line 194, in _misc ].s.getVdsCapabilities()['info']['nics'][nics] KeyError: 'eth0.200' So I had to do the following: - Set switch port to untagged for this VLAN - Configure eth0 instead of eth0.200 - Run hosted engine setup - Configure VLAN for ovirtmgmt in oVirt (which surprisingly worked for an active interface) - Configure VLAN tagging on hosts - Reconfigure switch ports - Sync networks in oVirt network setup tab As this was a real pain to do I would love to see support for VLAN tagging in hosted engine setup :) Shall I open bug reports for these 2 issues or are they already known? -- Best Regards René Koch Senior Solution Architect ============================================ LIS-Linuxland GmbH Brünner Straße 163, A-1210 Vienna Phone: +43 1 236 91 60 Mobile: +43 660 / 512 21 31 E-Mail: rkoch(a)linuxland.at ============================================

2 2

Re: [Users] hosted engine issues
by Martin Sivak 03 Mar '14

03 Mar '14

The agent is required. In fact it contains all the logic. -- Martin Sivák msivak(a)redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message ----- > On 03/03/2014 02:05 PM, Martin Sivak wrote: > > Hi René, > > > >> # python --version > >> Python 2.6.6 > > > > Then I guess the traceback is my fault... > > > > See http://gerrit.ovirt.org/#/c/25269/ for the fix. I will try to get it > > into the soonest release possible. > > > Thanks. Do I have to patch the files manually or is ovirt-ha-agent not > strictly required for hosted engine? Some features like restarting > engine on 2nd node want work if ovirt-ha-agent isn't working, I guess. > > > > >> I can't see a full filesystem here: > >> > > > > Me neither. Is everything Read-Write? Read-Only FS might report no space > > left as well in some cases. Other than that, I do not know. > > No, I can write to all disks. > Btw, the same error message occurs on both nodes... > > > Regards, > René > > > > > > Regards > > -- > > Martin Sivák > > msivak(a)redhat.com > > Red Hat Czech > > RHEV-M SLA / Brno, CZ > > > > ----- Original Message ----- > >> On 03/03/2014 12:05 PM, Martin Sivak wrote: > >>> Hi René, > >>> > >>> thanks for the report. > >>> > >>>>> TypeError: super() argument 1 must be type, not classobj > >>> What Python version are you using? > >> > >> # python --version > >> Python 2.6.6 > >> > >>> > >>> You can debug a crash of this version of ha-agent using: > >>> > >>> /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon --pdb > >> > >> This gives me the same information as in vdsm.log > >> > >>> > >>> But this exception is trying to tell you that > >>> FSMLoggerAdapter(logging.LoggerAdapter) does not have object in the > >>> ancestor list. And that is very weird. > >>> > >>> It can be related to the disk space issues. > >>> > >>>>> libvirtError: Failed to acquire lock: No space left on device > >>> > >>> Check the free space on all your devices, including /tmp and /var. Or > >>> post > >>> the output of "df -h" command here > >> > >> I can't see a full filesystem here: > >> > >> # df -h > >> Filesystem Size Used Avail Use% Mounted on > >> /dev/mapper/vg0-lv_root 5.0G 1.1G 3.6G 24% / > >> tmpfs 16G 0 16G 0% /dev/shm > >> /dev/sda1 243M 45M 185M 20% /boot > >> /dev/mapper/vg0-lv_data 281G 21G 261G 8% /data > >> /dev/mapper/vg0-lv_tmp 2.0G 69M 1.9G 4% /tmp > >> /dev/mapper/vg0-lv_var 5.0G 384M 4.3G 9% /var > >> ovirt-host01:/engine 281G 21G 261G 8% > >> /rhev/data-center/mnt/ovirt-host01:_engine > >> > >> > >> Thanks, > >> René > >> > >> > >>> > >>> Regards > >>> > >>> -- > >>> Martin Sivák > >>> msivak(a)redhat.com > >>> Red Hat Czech > >>> RHEV-M SLA / Brno, CZ > >>> > >>> ----- Original Message ----- > >>>> Il 03/03/2014 11:33, René Koch ha scritto: > >>>>> Hi, > >>>>> > >>>>> I have some issues with hosted engine (oVirt 3.4 prerelease repo on > >>>>> CentOS > >>>>> 6.5). > >>>>> My setups is the following: > >>>>> 2 hosts (will be 4 in the future) with 4 GlusterFS shares: > >>>>> - engine (for hosted engine) > >>>>> - iso (for ISO domain) > >>>>> - ovirt (oVirt storage domain) > >>>>> > >>>>> I had a split-brain situation today (after rebooting both nodes) on > >>>>> hosted-engine.lockspace file on engine GlusterFS volume which I > >>>>> resolved. > >>>> > >>>> How did you solved it? By switching to NFS only? > >>>> > >>>> > >>>>> hosted engine used engine share via NFS (TCP) as glusterfs isn't > >>>>> supported > >>>>> for oVirt hosted engine, yet. I'll switch to GlusterFS as soon as oVirt > >>>>> will support it (I hope this will be soon as RHEV 3.3 is already > >>>>> supporting > >>>>> GlusterFS for hosted engine). > >>>>> > >>>>> > >>>>> First of all ovirt-ha-agent fails to start on both nodes: > >>>>> > >>>>> # service ovirt-ha-agent start > >>>>> Starting ovirt-ha-agent: [ OK ] > >>>>> # service ovirt-ha-agent status > >>>>> ovirt-ha-agent dead but subsys locked > >>>>> > >>>>> > >>>>> MainThread::INFO::2014-03-03 > >>>>> 11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>>>> ovirt-hosted-engine-ha agent 1.1.0 started > >>>>> MainThread::INFO::2014-03-03 > >>>>> 11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > >>>>> Found > >>>>> certificate common name: 10.0.200.101 > >>>>> MainThread::CRITICAL::2014-03-03 > >>>>> 11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>>>> Could not start ha-agent > >>>>> Traceback (most recent call last): > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>>>> line 97, in run > >>>>> self._run_agent() > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>>>> line 154, in _run_agent > >>>>> hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > >>>>> line 152, in __init__ > >>>>> "STOP_VM": self._stop_engine_vm > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", > >>>>> line 56, in __init__ > >>>>> logger, actions) > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>>>> line 93, in __init__ > >>>>> self._logger = FSMLoggerAdapter(logger, self) > >>>>> File > >>>>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>>>> line 16, in __init__ > >>>>> super(FSMLoggerAdapter, self).__init__(logger, None) > >>>>> TypeError: super() argument 1 must be type, not classobj > >>>>> > >>>>> > >>>>> > >>>>> If I want to start my hosted engine, I receive the following error in > >>>>> vdsm > >>>>> logs, which makes absolutly no sense to me, as there is plenty of disk > >>>>> space available: > >>>>> > >>>>> Thread-62::DEBUG::2014-03-03 > >>>>> 11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown > >>>>> libvirterror: ecode: 38 edom: 42 level: 2 message: Failed > >>>>> to acquire lock: No space left on device > >>>> > >>>> seems like a vdsm failure in starting monitor the hosted engine storage > >>>> domain. > >>>> Can you attach vdsm logs? > >>>> > >>>> > >>>> > >>>>> Thread-62::DEBUG::2014-03-03 > >>>>> 11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm) > >>>>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released > >>>>> Thread-62::ERROR::2014-03-03 > >>>>> 11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm) > >>>>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process > >>>>> failed > >>>>> Traceback (most recent call last): > >>>>> File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm > >>>>> self._run() > >>>>> File "/usr/share/vdsm/vm.py", line 3159, in _run > >>>>> self._connection.createXML(domxml, flags), > >>>>> File > >>>>> "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", > >>>>> line > >>>>> 92, in wrapper > >>>>> ret = f(*args, **kwargs) > >>>>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in > >>>>> createXML > >>>>> if ret is None:raise libvirtError('virDomainCreateXML() failed', > >>>>> conn=self) > >>>>> libvirtError: Failed to acquire lock: No space left on device > >>>>> Thread-62::DEBUG::2014-03-03 > >>>>> 11:24:46,286::vm::2720::vm.Vm::(setDownStatus) > >>>>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: > >>>>> Failed > >>>>> to acquire lock: No space left on device > >>>>> > >>>>> # df -h | grep engine > >>>>> ovirt-host01:/engine 281G 21G 261G 8% > >>>>> /rhev/data-center/mnt/ovirt-host01:_engine > >>>>> > >>>>> # sudo -u vdsm dd if=/dev/zero > >>>>> of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test > >>>>> bs=512 count=100 > >>>>> 100+0 records in > >>>>> 100+0 records out > >>>>> 51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s > >>>>> > >>>>> > >>>>> Could you give me some information on how to fix the ovirt-ha-agent and > >>>>> then hosted-engine storage issue? Thanks a lot. > >>>>> > >>>>> Btw, I had some issues during installation which I will explain in > >>>>> separate > >>>>> emails. > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Sandro Bonazzola > >>>> Better technology. Faster innovation. Powered by community > >>>> collaboration. > >>>> See how it works at redhat.com > >>>> > >> >

2 1

Re: [Users] hosted engine issues
by Yedidyah Bar David 03 Mar '14

03 Mar '14

----- Original Message ----- > From: "René Koch" <rkoch(a)linuxland.at> > To: "Yedidyah Bar David" <didi(a)redhat.com>, "Martin Sivak" <msivak(a)redhat.com> > Cc: users(a)ovirt.org > Sent: Monday, March 3, 2014 4:10:51 PM > Subject: Re: [Users] hosted engine issues > > On 03/03/2014 02:13 PM, Yedidyah Bar David wrote: > >> Me neither. Is everything Read-Write? Read-Only FS might report no space > >> left > >> as well in some cases. Other than that, I do not know. > > > > Perhaps some ipc resource? semaphores? > > > > Please check: > > > > ipcs > > > > cat /proc/sys/kernel/sem > > > > I know nothing about libvirt, that's just a wild guess. > > # ipcs > > ------ Shared Memory Segments -------- > key shmid owner perms bytes nattch status > > 0x00000000 0 root 644 80 2 > > 0x00000000 32769 root 644 16384 2 > > 0x00000000 65538 root 644 280 2 > > > ------ Semaphore Arrays -------- > key semid owner perms nsems > 0x00000000 0 root 600 1 > 0x00000000 65537 root 600 1 > 0x000000a7 163842 root 600 1 This means you have 3 semaphore sets, of one semaphore each. > > ------ Message Queues -------- > key msqid owner perms used-bytes messages > Also the rest is moderate usage. > # cat /proc/sys/kernel/sem > 250 32000 32 128 So you are far from the maxima (250 per set, 32000 total, 128 sets). > > > Do you see anything in this output? > I have no clue how to interpret this... See e.g. http://man7.org/linux/man-pages/man5/proc.5.html Is the above on a node? engine? both nodes are similar? If so, that's not the reason for the "no space left on device". If this error is reproducible, you can try to find the process that this happens to (perhaps libvirtd, vdsmd, or the hosted-engine ha daemon) and do: strace -f -o /tmp/trace1 -tt -s 512 -p PID where PID is the pid of that process, then search /tmp/trace1 for 'no space left on device' and see the exact call that failed. -- Didi

2 1

Re: [Users] hosted engine issues
by Martin Sivak 03 Mar '14

03 Mar '14

Hi René, > # python --version > Python 2.6.6 Then I guess the traceback is my fault... See http://gerrit.ovirt.org/#/c/25269/ for the fix. I will try to get it into the soonest release possible. > I can't see a full filesystem here: > Me neither. Is everything Read-Write? Read-Only FS might report no space left as well in some cases. Other than that, I do not know. Regards -- Martin Sivák msivak(a)redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message ----- > On 03/03/2014 12:05 PM, Martin Sivak wrote: > > Hi René, > > > > thanks for the report. > > > >>> TypeError: super() argument 1 must be type, not classobj > > What Python version are you using? > > # python --version > Python 2.6.6 > > > > > You can debug a crash of this version of ha-agent using: > > > > /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon --pdb > > This gives me the same information as in vdsm.log > > > > > But this exception is trying to tell you that > > FSMLoggerAdapter(logging.LoggerAdapter) does not have object in the > > ancestor list. And that is very weird. > > > > It can be related to the disk space issues. > > > >>> libvirtError: Failed to acquire lock: No space left on device > > > > Check the free space on all your devices, including /tmp and /var. Or post > > the output of "df -h" command here > > I can't see a full filesystem here: > > # df -h > Filesystem Size Used Avail Use% Mounted on > /dev/mapper/vg0-lv_root 5.0G 1.1G 3.6G 24% / > tmpfs 16G 0 16G 0% /dev/shm > /dev/sda1 243M 45M 185M 20% /boot > /dev/mapper/vg0-lv_data 281G 21G 261G 8% /data > /dev/mapper/vg0-lv_tmp 2.0G 69M 1.9G 4% /tmp > /dev/mapper/vg0-lv_var 5.0G 384M 4.3G 9% /var > ovirt-host01:/engine 281G 21G 261G 8% > /rhev/data-center/mnt/ovirt-host01:_engine > > > Thanks, > René > > > > > > Regards > > > > -- > > Martin Sivák > > msivak(a)redhat.com > > Red Hat Czech > > RHEV-M SLA / Brno, CZ > > > > ----- Original Message ----- > >> Il 03/03/2014 11:33, René Koch ha scritto: > >>> Hi, > >>> > >>> I have some issues with hosted engine (oVirt 3.4 prerelease repo on > >>> CentOS > >>> 6.5). > >>> My setups is the following: > >>> 2 hosts (will be 4 in the future) with 4 GlusterFS shares: > >>> - engine (for hosted engine) > >>> - iso (for ISO domain) > >>> - ovirt (oVirt storage domain) > >>> > >>> I had a split-brain situation today (after rebooting both nodes) on > >>> hosted-engine.lockspace file on engine GlusterFS volume which I resolved. > >> > >> How did you solved it? By switching to NFS only? > >> > >> > >>> hosted engine used engine share via NFS (TCP) as glusterfs isn't > >>> supported > >>> for oVirt hosted engine, yet. I'll switch to GlusterFS as soon as oVirt > >>> will support it (I hope this will be soon as RHEV 3.3 is already > >>> supporting > >>> GlusterFS for hosted engine). > >>> > >>> > >>> First of all ovirt-ha-agent fails to start on both nodes: > >>> > >>> # service ovirt-ha-agent start > >>> Starting ovirt-ha-agent: [ OK ] > >>> # service ovirt-ha-agent status > >>> ovirt-ha-agent dead but subsys locked > >>> > >>> > >>> MainThread::INFO::2014-03-03 > >>> 11:20:39,539::agent::52::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> ovirt-hosted-engine-ha agent 1.1.0 started > >>> MainThread::INFO::2014-03-03 > >>> 11:20:39,590::hosted_engine::223::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_get_hostname) > >>> Found > >>> certificate common name: 10.0.200.101 > >>> MainThread::CRITICAL::2014-03-03 > >>> 11:20:39,590::agent::103::ovirt_hosted_engine_ha.agent.agent.Agent::(run) > >>> Could not start ha-agent > >>> Traceback (most recent call last): > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> line 97, in run > >>> self._run_agent() > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/agent.py", > >>> line 154, in _run_agent > >>> hosted_engine.HostedEngine(self.shutdown_requested).start_monitoring() > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/hosted_engine.py", > >>> line 152, in __init__ > >>> "STOP_VM": self._stop_engine_vm > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/agent/state_machine.py", > >>> line 56, in __init__ > >>> logger, actions) > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>> line 93, in __init__ > >>> self._logger = FSMLoggerAdapter(logger, self) > >>> File > >>> "/usr/lib/python2.6/site-packages/ovirt_hosted_engine_ha/lib/fsm/machine.py", > >>> line 16, in __init__ > >>> super(FSMLoggerAdapter, self).__init__(logger, None) > >>> TypeError: super() argument 1 must be type, not classobj > >>> > >>> > >>> > >>> If I want to start my hosted engine, I receive the following error in > >>> vdsm > >>> logs, which makes absolutly no sense to me, as there is plenty of disk > >>> space available: > >>> > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,282::libvirtconnection::124::root::(wrapper) Unknown > >>> libvirterror: ecode: 38 edom: 42 level: 2 message: Failed > >>> to acquire lock: No space left on device > >> > >> seems like a vdsm failure in starting monitor the hosted engine storage > >> domain. > >> Can you attach vdsm logs? > >> > >> > >> > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,282::vm::2252::vm.Vm::(_startUnderlyingVm) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::_ongoingCreations released > >>> Thread-62::ERROR::2014-03-03 > >>> 11:24:46,283::vm::2278::vm.Vm::(_startUnderlyingVm) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::The vm start process failed > >>> Traceback (most recent call last): > >>> File "/usr/share/vdsm/vm.py", line 2238, in _startUnderlyingVm > >>> self._run() > >>> File "/usr/share/vdsm/vm.py", line 3159, in _run > >>> self._connection.createXML(domxml, flags), > >>> File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", > >>> line > >>> 92, in wrapper > >>> ret = f(*args, **kwargs) > >>> File "/usr/lib64/python2.6/site-packages/libvirt.py", line 2665, in > >>> createXML > >>> if ret is None:raise libvirtError('virDomainCreateXML() failed', > >>> conn=self) > >>> libvirtError: Failed to acquire lock: No space left on device > >>> Thread-62::DEBUG::2014-03-03 > >>> 11:24:46,286::vm::2720::vm.Vm::(setDownStatus) > >>> vmId=`f26dd37e-13b5-430c-b2f2-ecd098b82a91`::Changed state to Down: > >>> Failed > >>> to acquire lock: No space left on device > >>> > >>> # df -h | grep engine > >>> ovirt-host01:/engine 281G 21G 261G 8% > >>> /rhev/data-center/mnt/ovirt-host01:_engine > >>> > >>> # sudo -u vdsm dd if=/dev/zero > >>> of=/rhev/data-center/mnt/ovirt-host01:_engine/2851af27-8744-445d-9fb1-a0d083c8dc82/images/0e4d270f-2f7e-4b2b-847f-f114a4ba9bdc/test > >>> bs=512 count=100 > >>> 100+0 records in > >>> 100+0 records out > >>> 51200 bytes (51 kB) copied, 0.0230566 s, 2.2 MB/s > >>> > >>> > >>> Could you give me some information on how to fix the ovirt-ha-agent and > >>> then hosted-engine storage issue? Thanks a lot. > >>> > >>> Btw, I had some issues during installation which I will explain in > >>> separate > >>> emails. > >>> > >>> > >> > >> > >> -- > >> Sandro Bonazzola > >> Better technology. Faster innovation. Powered by community collaboration. > >> See how it works at redhat.com > >> >

3 3