oVirt + gluster with host and hosted VM on different subnets
Hi, I'm trying to install oVirt 3.4 + gluster looking at the following guides: http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/ It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway. But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones. Anyone installed oVirt in such a configuration and can give me any tip? Thanks Simone
Il 07/07/2014 15:38, Simone Marchioni ha scritto:
Hi,
I'm trying to install oVirt 3.4 + gluster looking at the following guides:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/
It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway.
But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones.
Anyone installed oVirt in such a configuration and can give me any tip?
Never tested such configuration. Andrew, something similar at your installation with an additional NIC?
Thanks Simone _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
On Wed, Jul 9, 2014 at 12:23 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 07/07/2014 15:38, Simone Marchioni ha scritto:
Hi,
I'm trying to install oVirt 3.4 + gluster looking at the following guides:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/
It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway.
But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones.
Anyone installed oVirt in such a configuration and can give me any tip?
Never tested such configuration. Andrew, something similar at your installation with an additional NIC?
If I understand correctly, you have your hosts on the 94.23.2.0/24 subnet but you need your hosted engine to be accessible as an address within 46.105.224.236? If that's true, then the easiest way to do it is simply run your hosted-engine install with the hosted-engine first on 94.23.2.0/24, you then add another nic to that hosted-engine VM which'll have the IP address for 46.105.224.237 (?)... alternatively, you could just use a nic alias? If you want to add an extra NIC to your hosted-engine to do that above scenario, here's a snippet from my notes: (storage_network is a bridge, replace that with ovirtmgmt or another bridge you may have created) hosted-engine --set-maintenance --mode=global # On all installed hosts nano /etc/ovirt-hosted-engine/vm.conf # insert under earlier nicModel # replace macaddress and uuid from above # increment slot devices={nicModel:pv,macAddr:00:16:3e:e1:7b:14,linkActive:true,network:storage_network,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:fdb11208-a888-e587-6053-32c9c0361f96,address:{bus:0x00,slot:0x04, domain:0x0000, type:pci,function:0x0},device:bridge,type:interface} hosted-engine --vm-shutdown hosted-engine --vm-start hosted-engine --set-maintenance --mode=none Although, re-reading your question, what do you mean by 'but I can't configure the VM network in a way it works.' ? Does the setup fail, or just when you create a VM you don't have any network connectivity..
Thanks Simone _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
Il 08/07/2014 16:47, Andrew Lau ha scritto:
On Wed, Jul 9, 2014 at 12:23 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 07/07/2014 15:38, Simone Marchioni ha scritto:
Hi,
I'm trying to install oVirt 3.4 + gluster looking at the following guides:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/
It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway.
But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones.
Anyone installed oVirt in such a configuration and can give me any tip? Never tested such configuration. Andrew, something similar at your installation with an additional NIC? If I understand correctly, you have your hosts on the 94.23.2.0/24 subnet but you need your hosted engine to be accessible as an address within 46.105.224.236?
Exactly
If that's true, then the easiest way to do it is simply run your hosted-engine install with the hosted-engine first on 94.23.2.0/24, you then add another nic to that hosted-engine VM which'll have the IP address for 46.105.224.237 (?)...
I'll try this
alternatively, you could just use a nic alias?
We made it work with the following changes (on the host machine in the subnet 94.23.2.0/24): - commented out and removed from running configuration ip rules in /etc/sysconfig/network-scripts/rule-ovirtmgmt - commented out and removed from running configuration ip routes in /etc/sysconfig/network-scripts/route-ovirtmgmt - added /etc/sysconfig/network-scripts/ovirtmgmt:0 with the following configuration: DEVICE=ovirtmgmt:238 ONBOOT=yes DELAY=0 IPADDR=46.105.224.238 NETMASK=255.255.255.252 BOOTPROTO=static NM_CONTROLLED=no - enabled ip forwarding in /proc/sys/net/ipv4/ip_forward After that installing the hosted-engine VM with the following IP stack: NETMASK=255.255.255.252 IPADDR=46.105.224.237 GATEWAY=46.105.224.238 seems to work ok.
If you want to add an extra NIC to your hosted-engine to do that above scenario, here's a snippet from my notes: (storage_network is a bridge, replace that with ovirtmgmt or another bridge you may have created)
hosted-engine --set-maintenance --mode=global
# On all installed hosts nano /etc/ovirt-hosted-engine/vm.conf # insert under earlier nicModel # replace macaddress and uuid from above # increment slot devices={nicModel:pv,macAddr:00:16:3e:e1:7b:14,linkActive:true,network:storage_network,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:fdb11208-a888-e587-6053-32c9c0361f96,address:{bus:0x00,slot:0x04, domain:0x0000, type:pci,function:0x0},device:bridge,type:interface}
hosted-engine --vm-shutdown hosted-engine --vm-start
hosted-engine --set-maintenance --mode=none
Ok: thanks for the advice!
Although, re-reading your question, what do you mean by 'but I can't configure the VM network in a way it works.' ? Does the setup fail, or just when you create a VM you don't have any network connectivity..
The setup works ok: it creates the VM and i can login to it with VNC on the host IP (94.23.2.X). I can install CentOS 6.5 as advised. After the reboot I login again by VNC and the host IP (94.23.2.X), and configure the IP stack with the other subnet (46.105.224.236/30) and after that the VM is isolated, unless I do the steps written above. Thanks for your support! Simone
Thanks Simone _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
Hi, I do not recommend running hosted engine on top of GlusterFS. Not even on top of NFS compatibility layer the GlusterFS provides. There have been a lot of issues with setups like that. GlusterFS does not ensure that the metadata writes are atomic and visible to all nodes at the same time and that causes serious trouble (the synchronization algorithm relies on the atomicity assumption). You can use GlusterFS storage domain for VMs, but the hosted engine storage domain needs something else - NFS or iSCSI (available in 3.5). -- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ ----- Original Message -----
Il 08/07/2014 16:47, Andrew Lau ha scritto:
On Wed, Jul 9, 2014 at 12:23 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 07/07/2014 15:38, Simone Marchioni ha scritto:
Hi,
I'm trying to install oVirt 3.4 + gluster looking at the following guides:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/
It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway.
But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones.
Anyone installed oVirt in such a configuration and can give me any tip? Never tested such configuration. Andrew, something similar at your installation with an additional NIC? If I understand correctly, you have your hosts on the 94.23.2.0/24 subnet but you need your hosted engine to be accessible as an address within 46.105.224.236?
Exactly
If that's true, then the easiest way to do it is simply run your hosted-engine install with the hosted-engine first on 94.23.2.0/24, you then add another nic to that hosted-engine VM which'll have the IP address for 46.105.224.237 (?)...
I'll try this
alternatively, you could just use a nic alias?
We made it work with the following changes (on the host machine in the subnet 94.23.2.0/24): - commented out and removed from running configuration ip rules in /etc/sysconfig/network-scripts/rule-ovirtmgmt - commented out and removed from running configuration ip routes in /etc/sysconfig/network-scripts/route-ovirtmgmt - added /etc/sysconfig/network-scripts/ovirtmgmt:0 with the following configuration: DEVICE=ovirtmgmt:238 ONBOOT=yes DELAY=0 IPADDR=46.105.224.238 NETMASK=255.255.255.252 BOOTPROTO=static NM_CONTROLLED=no - enabled ip forwarding in /proc/sys/net/ipv4/ip_forward
After that installing the hosted-engine VM with the following IP stack:
NETMASK=255.255.255.252 IPADDR=46.105.224.237 GATEWAY=46.105.224.238
seems to work ok.
If you want to add an extra NIC to your hosted-engine to do that above scenario, here's a snippet from my notes: (storage_network is a bridge, replace that with ovirtmgmt or another bridge you may have created)
hosted-engine --set-maintenance --mode=global
# On all installed hosts nano /etc/ovirt-hosted-engine/vm.conf # insert under earlier nicModel # replace macaddress and uuid from above # increment slot devices={nicModel:pv,macAddr:00:16:3e:e1:7b:14,linkActive:true,network:storage_network,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:fdb11208-a888-e587-6053-32c9c0361f96,address:{bus:0x00,slot:0x04, domain:0x0000, type:pci,function:0x0},device:bridge,type:interface}
hosted-engine --vm-shutdown hosted-engine --vm-start
hosted-engine --set-maintenance --mode=none
Ok: thanks for the advice!
Although, re-reading your question, what do you mean by 'but I can't configure the VM network in a way it works.' ? Does the setup fail, or just when you create a VM you don't have any network connectivity..
The setup works ok: it creates the VM and i can login to it with VNC on the host IP (94.23.2.X). I can install CentOS 6.5 as advised. After the reboot I login again by VNC and the host IP (94.23.2.X), and configure the IP stack with the other subnet (46.105.224.236/30) and after that the VM is isolated, unless I do the steps written above.
Thanks for your support! Simone
Thanks Simone _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Le 08/07/2014 21:15, Martin Sivak a écrit :
Hi,
I do not recommend running hosted engine on top of GlusterFS. Not even on top of NFS compatibility layer the GlusterFS provides.
Martin, It is very disturbing for us, final users, to read the comment above and the web page below, both written by Redhat employees. http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ We are about to try to setup a 3 nodes hosted ovirt + glusterfs cluster, but this kind of comment may delay our project. -- Nicolas Ecarnot
----- Original Message -----
From: "Nicolas Ecarnot" <nicolas@ecarnot.net> To: users@ovirt.org Sent: Tuesday, July 8, 2014 12:55:10 PM Subject: Re: [ovirt-users] oVirt + gluster with host and hosted VM on different subnets
Le 08/07/2014 21:15, Martin Sivak a écrit :
Hi,
I do not recommend running hosted engine on top of GlusterFS. Not even on top of NFS compatibility layer the GlusterFS provides.
Martin,
It is very disturbing for us, final users, to read the comment above and the web page below, both written by Redhat employees.
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/
I did note in that post that this is all new and somewhat in flux & that there are split-brain issues in play. FWIW, I'm actively working w/ a 3 node hosted ovirt + glusterfs cluster in my lab. I'll post about problems & solutions I encounter as I encounter them... Jason
We are about to try to setup a 3 nodes hosted ovirt + glusterfs cluster, but this kind of comment may delay our project.
-- Nicolas Ecarnot _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Hi Martin, Is that because of how the replication works? What if you had, the kernel-nfs server running ontop of the glusternfs share and a virtual IP to allow the hosted-engine to only access one of the shares. Thanks, Andrew On Wed, Jul 9, 2014 at 5:15 AM, Martin Sivak <msivak@redhat.com> wrote:
Hi,
I do not recommend running hosted engine on top of GlusterFS. Not even on top of NFS compatibility layer the GlusterFS provides.
There have been a lot of issues with setups like that. GlusterFS does not ensure that the metadata writes are atomic and visible to all nodes at the same time and that causes serious trouble (the synchronization algorithm relies on the atomicity assumption).
You can use GlusterFS storage domain for VMs, but the hosted engine storage domain needs something else - NFS or iSCSI (available in 3.5).
-- Martin Sivák msivak@redhat.com Red Hat Czech RHEV-M SLA / Brno, CZ
----- Original Message -----
Il 08/07/2014 16:47, Andrew Lau ha scritto:
On Wed, Jul 9, 2014 at 12:23 AM, Sandro Bonazzola <sbonazzo@redhat.com> wrote:
Il 07/07/2014 15:38, Simone Marchioni ha scritto:
Hi,
I'm trying to install oVirt 3.4 + gluster looking at the following guides:
http://community.redhat.com/blog/2014/05/ovirt-3-4-glusterized/ http://community.redhat.com/blog/2014/03/up-and-running-with-ovirt-3-4/
It went smooth until the hosted engine VM configuration: I can reach it by VNC and the host IP, but I can't configure the VM network in a way it works. Probably the problem is the assumption that the three hosts (2 hosts + the hosted engine) are on the same subnet sharing the same default gateway.
But my server is on OVH with the subnet 94.23.2.0/24 and my failover IPs are on the subnet 46.105.224.236/30, and my hosted engine need to use one IP of the last ones.
Anyone installed oVirt in such a configuration and can give me any tip? Never tested such configuration. Andrew, something similar at your installation with an additional NIC? If I understand correctly, you have your hosts on the 94.23.2.0/24 subnet but you need your hosted engine to be accessible as an address within 46.105.224.236?
Exactly
If that's true, then the easiest way to do it is simply run your hosted-engine install with the hosted-engine first on 94.23.2.0/24, you then add another nic to that hosted-engine VM which'll have the IP address for 46.105.224.237 (?)...
I'll try this
alternatively, you could just use a nic alias?
We made it work with the following changes (on the host machine in the subnet 94.23.2.0/24): - commented out and removed from running configuration ip rules in /etc/sysconfig/network-scripts/rule-ovirtmgmt - commented out and removed from running configuration ip routes in /etc/sysconfig/network-scripts/route-ovirtmgmt - added /etc/sysconfig/network-scripts/ovirtmgmt:0 with the following configuration: DEVICE=ovirtmgmt:238 ONBOOT=yes DELAY=0 IPADDR=46.105.224.238 NETMASK=255.255.255.252 BOOTPROTO=static NM_CONTROLLED=no - enabled ip forwarding in /proc/sys/net/ipv4/ip_forward
After that installing the hosted-engine VM with the following IP stack:
NETMASK=255.255.255.252 IPADDR=46.105.224.237 GATEWAY=46.105.224.238
seems to work ok.
If you want to add an extra NIC to your hosted-engine to do that above scenario, here's a snippet from my notes: (storage_network is a bridge, replace that with ovirtmgmt or another bridge you may have created)
hosted-engine --set-maintenance --mode=global
# On all installed hosts nano /etc/ovirt-hosted-engine/vm.conf # insert under earlier nicModel # replace macaddress and uuid from above # increment slot devices={nicModel:pv,macAddr:00:16:3e:e1:7b:14,linkActive:true,network:storage_network,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:fdb11208-a888-e587-6053-32c9c0361f96,address:{bus:0x00,slot:0x04, domain:0x0000, type:pci,function:0x0},device:bridge,type:interface}
hosted-engine --vm-shutdown hosted-engine --vm-start
hosted-engine --set-maintenance --mode=none
Ok: thanks for the advice!
Although, re-reading your question, what do you mean by 'but I can't configure the VM network in a way it works.' ? Does the setup fail, or just when you create a VM you don't have any network connectivity..
The setup works ok: it creates the VM and i can login to it with VNC on the host IP (94.23.2.X). I can install CentOS 6.5 as advised. After the reboot I login again by VNC and the host IP (94.23.2.X), and configure the IP stack with the other subnet (46.105.224.236/30) and after that the VM is isolated, unless I do the steps written above.
Thanks for your support! Simone
Thanks Simone _______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
-- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Am 08.07.2014 21:15, schrieb Martin Sivak:
Hi,
I do not recommend running hosted engine on top of GlusterFS. Not even on top of NFS compatibility layer the GlusterFS provides.
There have been a lot of issues with setups like that. GlusterFS does not ensure that the metadata writes are atomic and visible to all nodes at the same time and that causes serious trouble (the synchronization algorithm relies on the atomicity assumption).
You can use GlusterFS storage domain for VMs, but the hosted engine storage domain needs something else - NFS or iSCSI (available in 3.5).
You should maybe document this with a big fat warning sign on the wiki and in the release notes and on the feature pages. I think many people will just combine what ovirt offers if there is just a warning hidden somewhere on a mailing list. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen
participants (7)
-
Andrew Lau -
Jason Brooks -
Martin Sivak -
Nicolas Ecarnot -
Sandro Bonazzola -
Simone Marchioni -
Sven Kieske