hosted-engine --deploy fails
Hi All! Just tried to install the hosted engine on a fresh CentOS 6.6: beside setting up a gluster cluster, I just added the repo and installed hosted-engine-setup package. Otherwise it's just a very minimal installation. hosted-engine deploy failed immediately. The issue seems to be the same as described here http://lists.ovirt.org/pipermail/users/2014-October/028461.html but that conversation didn't continue after the reporting user was asked for additional details. Failed installation: [root@vhost1 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141223224236-y9ttk4.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Hardware supports virtualization [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ ERROR ] Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination Packages: [root@vhost1 ~]# rpm -qa|grep hosted-engine ovirt-hosted-engine-setup-1.2.1-1.el6.noarch ovirt-hosted-engine-ha-1.2.4-1.el6.noarch [root@vhost1 ~]# rpm -qa|grep vdsm vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch vdsm-python-4.16.7-1.gitdb83943.el6.noarch vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-4.16.7-1.gitdb83943.el6.x86_64 vdsm-cli-4.16.7-1.gitdb83943.el6.noarch vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch So here's the output that was requested in the other thread. Hope someone can help me here. Thanks! [root@vhost1 ~]# find /var/lib/vdsm/persistence /var/lib/vdsm/persistence [root@vhost1 ~]# find /var/run/vdsm/netconf find: `/var/run/vdsm/netconf': No such file or directory [root@vhost1 ~]# ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:25:90:d8:0a:b0 brd ff:ff:ff:ff:ff:ff 4: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN link/ether 62:e5:28:13:9d:ba brd ff:ff:ff:ff:ff:ff 5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff virsh -r net [root@vhost1 ~]# virsh -r net error: unknown command: 'net' [root@vhost1 ~]# virsh -r net-list Name State Autostart Persistent -------------------------------------------------- ;vdsmdummy; active no no [root@vhost1 ~]# vdsm-tool restore-nets Traceback (most recent call last): File "/usr/share/vdsm/vdsm-restore-net-config", line 137, in <module> restore() File "/usr/share/vdsm/vdsm-restore-net-config", line 123, in restore unified_restoration() File "/usr/share/vdsm/vdsm-restore-net-config", line 57, in unified_restoration _inRollback=True) File "/usr/share/vdsm/network/api.py", line 616, in setupNetworks netinfo._libvirtNets2vdsm(libvirt_nets))) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 822, in get d['nics'][dev.name] = _nicinfo(dev, paddr, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 653, in _nicinfo info = _devinfo(link, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 681, in _devinfo ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo(link.name, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 370, in getIpInfo ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 209, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 36, in restore_command restore() File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 45, in restore raise EnvironmentError('Failed to restore the persisted networks') EnvironmentError: Failed to restore the persisted networks following is mentioned in the original thread, but should help other googlers: ovirt-hosted-engine-setup.log (first attempt): 2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: Starting multipathd daemon: [ OK ] Starting rpcbind: [ OK ] Starting ntpd: [ OK ] Loading the softdog kernel module: [ OK ] Starting wdmd: [ OK ] Starting sanlock: [ OK ] supervdsm start[ OK ] Starting iscsid: [ OK ] [ OK ] vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED] 2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing' 2014-12-23 22:04:38 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:04:38 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute ovirt-hosted-engine-setup.log (further attempts): 2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED] 2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd 2014-12-23 22:42:40 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:42:40 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute ---- Andreas
Hi All! Just did more research on this and it seems as if the reason was related to my interface configuration. Disclaimer upfront: I've a public IP configured on this server (since it's a hosted root server), but changed the IP addr here to 192.168.0.99 I started with the output from ´vdsm-tool restore-nets': ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack So I dumped the addr dictionary: {'address': '192.168.0.99', 'family': 'inet', 'flags': frozenset(['permanent']), 'index': 2, 'label': 'eth0', 'prefixlen': 32, 'scope': 'universe'} I've no clue why there's no "/32" at the end, but that's what my netmask actually is due to the special configuration I got from my hosting provider: [root@vhost1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:52:9F:A8:AA:BB IPADDR=192.168.0.99 NETMASK=255.255.255.255 SCOPE="peer 192.168.0.1" (again, public IPs changed to private one, if that matters. And I skipped the IPv6 config above...) So what I did next, was to patch the netinfo.py: [root@vhost1 vdsm]# diff -u netinfo_orig.py netinfo.py --- netinfo_orig.py 2014-12-24 22:16:23.362198715 +0100 +++ netinfo.py 2014-12-24 22:16:02.567625247 +0100 @@ -368,7 +368,12 @@ if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') + """Assume /32 if no prefix was found""" + if addr['address'].find('/') == -1: + ipv4addr = addr['address'] + prefix = "32" + else: + ipv4addr, prefix = addr['address'].split('/') ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address']) and recompiled it: [root@vhost1 vdsm]# python -m py_compile netinfo.py Et voilà: vdsm-tool ran fine: `hosted-engine --deploy' passed the previous failing stage! Hope this helps to find the root cause.... Regards, Andreas On Tue, Dec 23, 2014 at 10:58 PM, Andreas Mather <andreas@allaboutapps.at> wrote:
Hi All!
Just tried to install the hosted engine on a fresh CentOS 6.6: beside setting up a gluster cluster, I just added the repo and installed hosted-engine-setup package. Otherwise it's just a very minimal installation. hosted-engine deploy failed immediately. The issue seems to be the same as described here http://lists.ovirt.org/pipermail/users/2014-October/028461.html but that conversation didn't continue after the reporting user was asked for additional details.
Failed installation:
[root@vhost1 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20141223224236-y9ttk4.log Version: otopi-1.3.0 (otopi-1.3.0-1.el6) [ INFO ] Hardware supports virtualization [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ ERROR ] Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/etc/ovirt-hosted-engine/answers.conf' [ INFO ] Answer file '/etc/ovirt-hosted-engine/answers.conf' has been updated [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination
Packages:
[root@vhost1 ~]# rpm -qa|grep hosted-engine ovirt-hosted-engine-setup-1.2.1-1.el6.noarch ovirt-hosted-engine-ha-1.2.4-1.el6.noarch
[root@vhost1 ~]# rpm -qa|grep vdsm vdsm-yajsonrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-python-zombiereaper-4.16.7-1.gitdb83943.el6.noarch vdsm-python-4.16.7-1.gitdb83943.el6.noarch vdsm-xmlrpc-4.16.7-1.gitdb83943.el6.noarch vdsm-4.16.7-1.gitdb83943.el6.x86_64 vdsm-cli-4.16.7-1.gitdb83943.el6.noarch vdsm-jsonrpc-4.16.7-1.gitdb83943.el6.noarch
So here's the output that was requested in the other thread. Hope someone can help me here. Thanks!
[root@vhost1 ~]# find /var/lib/vdsm/persistence /var/lib/vdsm/persistence
[root@vhost1 ~]# find /var/run/vdsm/netconf find: `/var/run/vdsm/netconf': No such file or directory
[root@vhost1 ~]# ip l 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:25:90:d8:0a:b0 brd ff:ff:ff:ff:ff:ff 4: ;vdsmdummy;: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN link/ether 62:e5:28:13:9d:ba brd ff:ff:ff:ff:ff:ff 5: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noop state DOWN link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff virsh -r net
[root@vhost1 ~]# virsh -r net error: unknown command: 'net'
[root@vhost1 ~]# virsh -r net-list Name State Autostart Persistent -------------------------------------------------- ;vdsmdummy; active no no
[root@vhost1 ~]# vdsm-tool restore-nets Traceback (most recent call last): File "/usr/share/vdsm/vdsm-restore-net-config", line 137, in <module> restore() File "/usr/share/vdsm/vdsm-restore-net-config", line 123, in restore unified_restoration() File "/usr/share/vdsm/vdsm-restore-net-config", line 57, in unified_restoration _inRollback=True) File "/usr/share/vdsm/network/api.py", line 616, in setupNetworks netinfo._libvirtNets2vdsm(libvirt_nets))) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 822, in get d['nics'][dev.name] = _nicinfo(dev, paddr, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 653, in _nicinfo info = _devinfo(link, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 681, in _devinfo ipv4addr, ipv4netmask, ipv4addrs, ipv6addrs = getIpInfo(link.name, ipaddrs) File "/usr/lib/python2.6/site-packages/vdsm/netinfo.py", line 370, in getIpInfo ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack Traceback (most recent call last): File "/usr/bin/vdsm-tool", line 209, in main return tool_command[cmd]["command"](*args) File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 36, in restore_command restore() File "/usr/lib/python2.6/site-packages/vdsm/tool/restore_nets.py", line 45, in restore raise EnvironmentError('Failed to restore the persisted networks') EnvironmentError: Failed to restore the persisted networks
following is mentioned in the original thread, but should help other googlers:
ovirt-hosted-engine-setup.log (first attempt):
2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: Starting multipathd daemon: [ OK ] Starting rpcbind: [ OK ] Starting ntpd: [ OK ] Loading the softdog kernel module: [ OK ] Starting wdmd: [ OK ] Starting sanlock: [ OK ] supervdsm start[ OK ] Starting iscsid: [ OK ] [ OK ] vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED]
2014-12-23 22:04:38 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd libvirt: Network Filter Driver error : Network filter not found: no nwfilter with matching name 'vdsm-no-mac-spoofing'
2014-12-23 22:04:38 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:04:38 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute
ovirt-hosted-engine-setup.log (further attempts):
2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:861 execute-output: ('/sbin/service', 'vdsmd', 'start') stdout: vdsm: Running mkdirs vdsm: Running configure_coredump vdsm: Running configure_vdsm_logs vdsm: Running run_init_hooks vdsm: Running check_is_configured libvirt is already configured for vdsm vdsm: Running validate_configuration SUCCESS: ssl configured to true. No conflicts vdsm: Running prepare_transient_repository vdsm: Running syslog_available vdsm: Running nwfilter vdsm: Running dummybr vdsm: Running load_needed_modules vdsm: Running tune_system vdsm: Running test_space vdsm: Running test_lo vdsm: Running unified_network_persistence_upgrade vdsm: stopped during execute unified_network_persistence_upgrade task (task returned with error code 1). vdsm start[FAILED]
2014-12-23 22:42:40 DEBUG otopi.plugins.otopi.services.rhel plugin.execute:866 execute-output: ('/sbin/service', 'vdsmd', 'start') stderr: initctl: Job is already running: libvirtd
2014-12-23 22:42:40 DEBUG otopi.context context._executeMethod:152 method exception Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/otopi/context.py", line 142, in _executeMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/system/vdsmenv.py", line 155, in _late_setup state=True File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 188, in state 'start' if state else 'stop' File "/usr/share/otopi/plugins/otopi/services/rhel.py", line 96, in _executeServiceCommand raiseOnError=raiseOnError File "/usr/lib/python2.6/site-packages/otopi/plugin.py", line 871, in execute command=args[0], RuntimeError: Command '/sbin/service' failed to execute 2014-12-23 22:42:40 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment setup': Command '/sbin/service' failed to execute
----
Andreas
----- Original Message -----
From: "Andreas Mather" <andreas@allaboutapps.at> To: users@ovirt.org Sent: Wednesday, December 24, 2014 11:29:58 PM Subject: Re: [ovirt-users] hosted-engine --deploy fails
Hi All!
Just did more research on this and it seems as if the reason was related to my interface configuration. Disclaimer upfront: I've a public IP configured on this server (since it's a hosted root server), but changed the IP addr here to 192.168.0.99
I started with the output from ´vdsm-tool restore-nets': ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack
So I dumped the addr dictionary: {'address': '192.168.0.99', 'family': 'inet', 'flags': frozenset(['permanent']), 'index': 2, 'label': 'eth0', 'prefixlen': 32, 'scope': 'universe'}
I've no clue why there's no "/32" at the end, but that's what my netmask actually is due to the special configuration I got from my hosting provider:
[root@vhost1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:52:9F:A8:AA:BB IPADDR=192.168.0.99 NETMASK=255.255.255.255 SCOPE="peer 192.168.0.1"
(again, public IPs changed to private one, if that matters. And I skipped the IPv6 config above...)
So what I did next, was to patch the netinfo.py: [root@vhost1 vdsm]# diff -u netinfo_orig.py netinfo.py --- netinfo_orig.py 2014-12-24 22:16:23.362198715 +0100 +++ netinfo.py 2014-12-24 22:16:02.567625247 +0100 @@ -368,7 +368,12 @@ if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') + """Assume /32 if no prefix was found""" + if addr['address'].find('/') == -1: + ipv4addr = addr['address'] + prefix = "32" + else: + ipv4addr, prefix = addr['address'].split('/') ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address'])
and recompiled it: [root@vhost1 vdsm]# python -m py_compile netinfo.py
Et voilà: vdsm-tool ran fine: `hosted-engine --deploy' passed the previous failing stage!
Thanks for great analysis, report and patch! Would you like to push it to gerrit? See [1] and [2] Adding Dan in case you do not want to, so that your patch isn't lost...
Hope this helps to find the root cause....
Not sure what you mean - did you have any other problem after applying your patch? Seems to me that the root cause is some code (the part you patched or something earlier) did not expect a prefix of /32, which is indeed quite rare. Not even certain how it works - did you also get a default gateway? How can you access it, if it's not in your subnet? [1] http://www.ovirt.org/Develop [2] http://www.ovirt.org/Working_with_oVirt_Gerrit Best regards, -- Didi
Hi!
Hope this helps to find the root cause....
Not sure what you mean - did you have any other problem after applying your patch? Seems to me that the root cause is some code (the part you patched or something earlier) did not expect a prefix of /32, which is indeed quite rare. Not even certain how it works - did you also get a default gateway? How can you access it, if it's not in your subnet?
Well, since I don't really know the code I wasn't sure if it's ok to assume /32 in this case. Actually, I hit a couple of other issues afterwards (e.g. the "SCOPE" line wasn't copied into the new bridge interface configuration). So from my experience, I'd be better to tell users with configurations like mine that the configuration is just not supported. Since I got this config from my provider, I'm also not 100% sure what the SCOPE does, but it seems as if it would make a PtP interface out of a regular ethernet interface, i.e. "hey kernel, with this interface/address I can contact exactly one other host - my gateway". So summed up, I'd suggest to warn users with this configuration and tell them to reconfigure first (without SCOPE and a regular subnet mask). At least that's what I did and what helped after I hit other issues after this fix. Regards, Andreas --- *Andreas Mather* On Wed, Dec 31, 2014 at 11:40 AM, Yedidyah Bar David <didi@redhat.com> wrote:
From: "Andreas Mather" <andreas@allaboutapps.at> To: users@ovirt.org Sent: Wednesday, December 24, 2014 11:29:58 PM Subject: Re: [ovirt-users] hosted-engine --deploy fails
Hi All!
Just did more research on this and it seems as if the reason was related to my interface configuration. Disclaimer upfront: I've a public IP configured on this server (since it's a hosted root server), but changed the IP addr here to 192.168.0.99
I started with the output from ´vdsm-tool restore-nets': ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack
So I dumped the addr dictionary: {'address': '192.168.0.99', 'family': 'inet', 'flags': frozenset(['permanent']), 'index': 2, 'label': 'eth0', 'prefixlen': 32, 'scope': 'universe'}
I've no clue why there's no "/32" at the end, but that's what my netmask actually is due to the special configuration I got from my hosting
----- Original Message ----- provider:
[root@vhost1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:52:9F:A8:AA:BB IPADDR=192.168.0.99 NETMASK=255.255.255.255 SCOPE="peer 192.168.0.1"
(again, public IPs changed to private one, if that matters. And I
skipped the
IPv6 config above...)
So what I did next, was to patch the netinfo.py: [root@vhost1 vdsm]# diff -u netinfo_orig.py netinfo.py --- netinfo_orig.py 2014-12-24 22:16:23.362198715 +0100 +++ netinfo.py 2014-12-24 22:16:02.567625247 +0100 @@ -368,7 +368,12 @@ if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') + """Assume /32 if no prefix was found""" + if addr['address'].find('/') == -1: + ipv4addr = addr['address'] + prefix = "32" + else: + ipv4addr, prefix = addr['address'].split('/') ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address'])
and recompiled it: [root@vhost1 vdsm]# python -m py_compile netinfo.py
Et voilà: vdsm-tool ran fine: `hosted-engine --deploy' passed the previous failing stage!
Thanks for great analysis, report and patch! Would you like to push it to gerrit? See [1] and [2]
Adding Dan in case you do not want to, so that your patch isn't lost...
Hope this helps to find the root cause....
Not sure what you mean - did you have any other problem after applying your patch? Seems to me that the root cause is some code (the part you patched or something earlier) did not expect a prefix of /32, which is indeed quite rare. Not even certain how it works - did you also get a default gateway? How can you access it, if it's not in your subnet?
[1] http://www.ovirt.org/Develop [2] http://www.ovirt.org/Working_with_oVirt_Gerrit
Best regards, -- Didi
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02.01.2015 11:22, Andreas Mather wrote:
Well, since I don't really know the code I wasn't sure if it's ok to assume /32 in this case.
Well in general the detection should be correct. I'm not quite sure why it does not work in all cases? The code should not assume any "standard" mask when it can't find the right one. kind regards Sven -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQGcBAEBAgAGBQJUppTeAAoJEAq0kGAWDrqlDBUL/1ed1dE3gK3kxNXDtnc/RkNp mgrYYEaVio4e06/9Kg1rHAqFeWHCJqZ4BXUOTIKZjpful3kF5cxnJSVpMNy/Zjpx T7BSBrA60AW6bt2nVpkrjBQeIznCsvDArAsupvBQwqYe9giTecwhTuy7jHJyggjP FIm76Nl1Y1AZjivMSdPE83H1kru7dzQjP9dEBhte0tLtwOPt2C3Nk8ljbCXX6KFh j4KXHmukXxaFzm7+6T0CFmdAOLlqWZuZ+vxKn0Tr4kulLL07ataT9tvz5XTDNveI uJ7bx+S0pzmMjyxjU4KpGy2pO6WopcMnGiy1eariRUgeM7RhOTHneSbwEMbz2M4b 4aLjbrGa1kP8+COQD355e/KmwmoOLZxH40rnzdPC8Uhqs4SAR+euGW0jncG1tSLX EJJlXxn+THTQbjtB6BDZz00dlcig96UC02iEORIWjOWjc0dx5hT24op71+HxP5XY tfDZNkCM3y2i7Hu3+b8NGcHLw69TeMlT25sk8wJwOA== =/axC -----END PGP SIGNATURE-----
On Wed, Dec 31, 2014 at 05:40:29AM -0500, Yedidyah Bar David wrote:
----- Original Message -----
From: "Andreas Mather" <andreas@allaboutapps.at> To: users@ovirt.org Sent: Wednesday, December 24, 2014 11:29:58 PM Subject: Re: [ovirt-users] hosted-engine --deploy fails
Hi All!
Just did more research on this and it seems as if the reason was related to my interface configuration. Disclaimer upfront: I've a public IP configured on this server (since it's a hosted root server), but changed the IP addr here to 192.168.0.99
I started with the output from ´vdsm-tool restore-nets': ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack
Thanks for this report. I believe that this was fixed by http://gerrit.ovirt.org/35317 which would be in the long-delayed ovirt-3.5.1. Unfortunately, the official build of vdsm-4.16.10 for el6 is being delayed http://lists.ovirt.org/pipermail/devel/2014-December/009566.html but for other platforms it is already testable. I'd apreciate if you try it out!
So I dumped the addr dictionary: {'address': '192.168.0.99', 'family': 'inet', 'flags': frozenset(['permanent']), 'index': 2, 'label': 'eth0', 'prefixlen': 32, 'scope': 'universe'}
I've no clue why there's no "/32" at the end, but that's what my netmask actually is due to the special configuration I got from my hosting provider:
[root@vhost1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:52:9F:A8:AA:BB IPADDR=192.168.0.99 NETMASK=255.255.255.255 SCOPE="peer 192.168.0.1"
(again, public IPs changed to private one, if that matters. And I skipped the IPv6 config above...)
So what I did next, was to patch the netinfo.py: [root@vhost1 vdsm]# diff -u netinfo_orig.py netinfo.py --- netinfo_orig.py 2014-12-24 22:16:23.362198715 +0100 +++ netinfo.py 2014-12-24 22:16:02.567625247 +0100 @@ -368,7 +368,12 @@ if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') + """Assume /32 if no prefix was found""" + if addr['address'].find('/') == -1: + ipv4addr = addr['address'] + prefix = "32" + else: + ipv4addr, prefix = addr['address'].split('/') ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address'])
and recompiled it: [root@vhost1 vdsm]# python -m py_compile netinfo.py
Et voilà: vdsm-tool ran fine: `hosted-engine --deploy' passed the previous failing stage!
Thanks for great analysis, report and patch! Would you like to push it to gerrit? See [1] and [2]
Adding Dan in case you do not want to, so that your patch isn't lost...
Hope this helps to find the root cause....
Not sure what you mean - did you have any other problem after applying your patch? Seems to me that the root cause is some code (the part you patched or something earlier) did not expect a prefix of /32, which is indeed quite rare. Not even certain how it works - did you also get a default gateway? How can you access it, if it's not in your subnet?
[1] http://www.ovirt.org/Develop [2] http://www.ovirt.org/Working_with_oVirt_Gerrit
Best regards, -- Didi
Hi!
I believe that this was fixed by http://gerrit.ovirt.org/35317 which would be in the long-delayed ovirt-3.5.1.
I'd apreciate if you try it out!
I can't really try it out, since I had to move forward and can't do the tests now :-/ But according to the diff, this should do the same as what I did (prefix seems to be ignored in this function anyway). Nevertheless, people shouldn't use a SCOPE="peer ..." configuration (as mine was before), but should stick to a regular addr/subnet/gateway config. At least this fixed my issues.... Regards, Andreas On Fri, Jan 2, 2015 at 7:08 PM, Dan Kenigsberg <danken@redhat.com> wrote:
On Wed, Dec 31, 2014 at 05:40:29AM -0500, Yedidyah Bar David wrote:
----- Original Message -----
From: "Andreas Mather" <andreas@allaboutapps.at> To: users@ovirt.org Sent: Wednesday, December 24, 2014 11:29:58 PM Subject: Re: [ovirt-users] hosted-engine --deploy fails
Hi All!
Just did more research on this and it seems as if the reason was related to my interface configuration. Disclaimer upfront: I've a public IP configured on this server (since it's a hosted root server), but changed the IP addr here to 192.168.0.99
I started with the output from ´vdsm-tool restore-nets': ipv4addr, prefix = addr['address'].split('/') ValueError: need more than 1 value to unpack
Thanks for this report.
I believe that this was fixed by http://gerrit.ovirt.org/35317 which would be in the long-delayed ovirt-3.5.1.
Unfortunately, the official build of vdsm-4.16.10 for el6 is being delayed http://lists.ovirt.org/pipermail/devel/2014-December/009566.html but for other platforms it is already testable.
I'd apreciate if you try it out!
So I dumped the addr dictionary: {'address': '192.168.0.99', 'family': 'inet', 'flags': frozenset(['permanent']), 'index': 2, 'label': 'eth0', 'prefixlen': 32, 'scope': 'universe'}
I've no clue why there's no "/32" at the end, but that's what my
actually is due to the special configuration I got from my hosting
netmask provider:
[root@vhost1 ~]# cat /etc/sysconfig/network-scripts/ifcfg-eth0 DEVICE=eth0 BOOTPROTO=none ONBOOT=yes HWADDR=00:52:9F:A8:AA:BB IPADDR=192.168.0.99 NETMASK=255.255.255.255 SCOPE="peer 192.168.0.1"
(again, public IPs changed to private one, if that matters. And I
skipped the
IPv6 config above...)
So what I did next, was to patch the netinfo.py: [root@vhost1 vdsm]# diff -u netinfo_orig.py netinfo.py --- netinfo_orig.py 2014-12-24 22:16:23.362198715 +0100 +++ netinfo.py 2014-12-24 22:16:02.567625247 +0100 @@ -368,7 +368,12 @@ if addr['family'] == 'inet': ipv4addrs.append(addr['address']) if 'secondary' not in addr['flags']: - ipv4addr, prefix = addr['address'].split('/') + """Assume /32 if no prefix was found""" + if addr['address'].find('/') == -1: + ipv4addr = addr['address'] + prefix = "32" + else: + ipv4addr, prefix = addr['address'].split('/') ipv4netmask = prefix2netmask(addr['prefixlen']) else: ipv6addrs.append(addr['address'])
and recompiled it: [root@vhost1 vdsm]# python -m py_compile netinfo.py
Et voilà: vdsm-tool ran fine: `hosted-engine --deploy' passed the previous failing stage!
Thanks for great analysis, report and patch! Would you like to push it to gerrit? See [1] and [2]
Adding Dan in case you do not want to, so that your patch isn't lost...
Hope this helps to find the root cause....
Not sure what you mean - did you have any other problem after applying your patch? Seems to me that the root cause is some code (the part you patched or something earlier) did not expect a prefix of /32, which is indeed quite rare. Not even certain how it works - did you also get a default gateway? How can you access it, if it's not in your subnet?
[1] http://www.ovirt.org/Develop [2] http://www.ovirt.org/Working_with_oVirt_Gerrit
Best regards, -- Didi
participants (4)
-
Andreas Mather -
Dan Kenigsberg -
Sven Kieske -
Yedidyah Bar David