[Users] oVirt Solaris support
René Koch (ovido)
r.koch at ovido.at
Mon Sep 16 16:27:31 UTC 2013
On Mon, 2013-09-16 at 16:29 +0100, Dan Kenigsberg wrote:
> On Mon, Sep 16, 2013 at 04:39:47PM +0200, René Koch (ovido) wrote:
> >
> > On Mon, 2013-09-16 at 15:14 +0100, Dan Kenigsberg wrote:
> >
> > > On Mon, Sep 16, 2013 at 12:33:34PM +0200, René Koch (ovido) wrote:
> > > >
> > > > On Fri, 2013-09-06 at 22:26 +0100, Dan Kenigsberg wrote:
> > > >
> > > > > On Fri, Sep 06, 2013 at 06:23:42PM +0200, René Koch (ovido) wrote:
> > > > > > Hi,
> > > > > >
> > > > > > I want to start the discussion about Solaris support on oVirt again, as
> > > > > > there was no solution for it yet.
> > > > > >
> > > > > > On my oVirt 3.2.2 environment I installed Solaris 11 U1 with the
> > > > > > following specs:
> > > > > > * Operating System: Other
> > > > > > * nic1: rtl8139
> > > > > > * Disk1: IDE (Thin Provision)
> > > > > > * Host: CentOS 6.4 with qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64
> > > > > >
> > > > > > These are the same settings as on my RHEL 6.4 KVM host (except I can
> > > > > > choose Solaris 10 as OS in virt-manager), which has KVM version:
> > > > > > qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2.x86_64 (I wanted to use this host
> > > > > > as a RHEV host, so the qemu-kvm-rhev package is installed in case you
> > > > > > wounder)...
> > > > > >
> > > > > > What's working:
> > > > > > * OS installation on IDE disk
> > > > > > * Bringing up network interface
> > > > > >
> > > > > > What's not working on oVirt:
> > > > > > * Network connections - on RHEL 6.4 with plain libvirt/kvm this is
> > > > > > working...
> > > > > >
> > > > > >
> > > > > > I can see the mac address on my CentOS host, but can't ping the Solaris
> > > > > > vm:
> > > > > >
> > > > > > # brctl showmacs ovirtmgmt | egrep '00:99:4a:00:64:83|port'
> > > > > > port no mac addr is local? ageing timer
> > > > > > 2 00:99:4a:00:64:83 no 10.72
> > > > > >
> > > > > > # arp -an | grep '00:99:4a:00:64:83'
> > > > > > ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
> > > > > >
> > > > > > When using tcpdump on the vnet interface which belongs to the Solaris vm
> > > > > > (ip 10.0.100.123) I can see ARP requests from the vm for ip address of
> > > > > > my CentOS host (10.0.100.42) but no response to it. Same when pinging
> > > > > > other ips in this network:
> > > > > >
> > > > > > # tcpdump -n -i vnet2
> > > > > > tcpdump: WARNING: vnet2: no IPv4 address assigned
> > > > > > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > > > > > decode
> > > > > > listening on vnet2, link-type EN10MB (Ethernet), capture size 65535
> > > > > > bytes
> > > > > > 18:15:35.987868 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > > > > 10.0.100.123, length 46
> > > > > > 18:15:36.487399 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > > > > 10.0.100.123, length 46
> > > > > > 18:15:36.987536 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > > > > 10.0.100.123, length 46
> > > > > >
> > > > > > I also compared the qemu-kvm process list on the KVM with the oVirt
> > > > > > machine and can't see much differences except that oVirt has more
> > > > > > information like smbios....
> > > > > > oVirt host:
> > > > > > /usr/libexec/qemu-kvm
> > > > > > <snip>
> > > > > > -netdev tap,fd=27,id=hostnet0
> > > > > > -device
> > > > > > rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3
> > > > > >
> > > > > > RHEL KVM host:
> > > > > > /usr/libexec/qemu-kvm
> > > > > > <snip>
> > > > > > -netdev tap,fd=32,id=hostnet0
> > > > > > -device
> > > > > > rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:c2:97,bus=pci.0,addr=0x3
> > > > > >
> > > > > > Any suggestions on how to troubleshoot / get Solaris networking running
> > > > > > is welcome.
> > > > > > Changing the interface to e1000 doesn't work either.
> > > >
> > > >
> > > >
> > > > Thanks a lot for your answer and sorry for my late response - I was sick
> > > > leaving last week.
> > > >
> > > >
> > > > >
> > > > > - Would you share the output of your `brctl show`?
> > > >
> > > >
> > > > # brctl show
> > > > bridge name bridge id STP enabled interfaces
> > > > ;vdsmdummy; 8000.000000000000 no
> > > > dmz 8000.0025901ad140 no eth0.200
> > > > ovirtmgmt 8000.0025901ad140 no eth0.100
> > > > vnet1
> > > > vnet2
> > > > setup 8000.0025901ad140 no eth0.500
> > > > test2 8000.0025901ad140 no eth0.602
> > > > vnet0
> > > >
> > > > vnet1 is a Debian 7 vm which has a perfectly working network.
> > > > vnet2 is the Solaris 11 vm which can't communicate over the network.
> > > >
> > > >
> > > > > - Particulalry, does the host device with 10.0.100.42 sit on the same
> > > > > bridge as your Solaris VM?
> > > >
> > > >
> > > > 10.0.100.42 is the ip of the oVirt host and it's ip is configured on
> > > > this bridge (and reachable from the Debian 7 vm):
> > > >
> > > > # ip a | grep ovirt
> > > > 5: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> > > > state UNKNOWN
> > > > inet 10.0.100.42/24 brd 10.0.100.255 scope global ovirtmgmt
> > > >
> > > >
> > > > > - Do you see the who-has packet when sniffing the bridge device?
> > > >
> > > >
> > > >
> > > > Yes, I can see them:
> > > > 12:24:16.311218 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > > 10.0.100.123, length 46
> > > > 12:24:16.311246 ARP, Reply 10.0.100.42 is-at 00:25:90:1a:d1:40, length
> > > > 28
> > > >
> > > > I can also see the MAC of the Solaris vm in arp cache of oVirt host:
> > > > # arp -an
> > > > ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
> > > >
> > > > But I can't see the MAC address of oVirt host (10.0.100.42) in arp table
> > > > of the Solaris 11 vm. To be more clear: I can't see any MAC address in
> > > > ARP table of Solaris vm except the MAC of the vm itself.
> > > >
> > > >
> > > > > - Have yout tried guest-to-guest communication (over the same bridge)?
> > > >
> > > >
> > > > I can communicate from the Debian 7 vm to all hosts in my network, but I
> > > > can't reach the Solaris 11 vm.
> > > > The network of the oVirt setup is working fine for Windows and various
> > > > Linux guests, but not for Solaris.
> > > >
> > > > Btw, excatly same behavior on RHEV 3.2 with Solaris 10/11 - Windows and
> > > > Linux is working as expected, but no networking for Solaris guests...
> > >
> > > I'm afraid I'm out of ideas. Is the kernel the same on both hosts?
> > > Would you reply with the complete qemu commandline? Maybe Michael could
> > > sport the difference between the working plain-KVM host and the
> > > non-working oVirt host.
> >
> >
> >
> > The kernel is not the same:
> > CentOS 6 (oVirt): 2.6.32-358.18.1.el6.x86_64
> > RHEL 6 (KVM): 2.6.32-358.11.1.el6.x86_64
> > But I think it's not kernel related as I successfully ran Solaris on the
> > KVM server 1 year ago (so kernel was an old RHEL 6.2 or 6.3 kernel) and
> > didn't worked this time on oVirt 3.1, too (hypervisor was CentOS 6.2 or
> > 6.3 using dreyou's-repository, now I'm running CentOS 6.4 with official
> > oVirt repository). So many kernel upgrades later Solaris still has a
> > working network on plain KVM but not on oVirt 3.2 or RHEV 3.2.
> >
> > Here's the complete qemu cmd:
> >
> > RHEL 6.4 (plain KVM - Solaris networking is fine):
> > /usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -enable-kvm -m
> > 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
> > e7d3b9a4-8605-dc4f-9092-41d68dd83735 -nodefconfig -nodefaults -chardev
> > socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/solaris11.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=40,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f0:27:d3,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:13 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
> > CentOS 6.4 (oVirt - Solaris can't reach other nodes):
> > /usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -cpu Nehalem
> > -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
> > cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5 -smbios
> > type=1,manufacturer=oVirt,product=oVirt
> > Node,version=6-4.el6.centos.10,serial=54443858-4E54-2500-901A-0025901AD140,uuid=cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5,sku=solaris11 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-09-06T15:08:22,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/484e62d7-7a01-4b5e-aec8-59d366100281/2055b80d-d59d-4fe6-80ec-5499674feed5/images/0786a273-780a-4f58-acc4-9b123953bde8/0c07f1f9-f9c0-4ac0-bfec-bee7a1c35a0b,if=none,id=drive-ide0-0-0,format=raw,serial=0786a273-780a-4f58-acc4-9b123953bde8,cache=none,werror=stop,rerror=stop,aio=threads -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/run/vdsm/payload/cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5.f3b4e68b93e9d35d3f17328fae2e178e.img,if=none,media=cdrom,id=drive-ide0-1-1,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=27,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/solaris11.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/solaris11.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
> >
>
> I don't think it's it, but maybe if you install and enabled the macspoof
> hook http://www.ovirt.org/Vdsm_Hooks#VM-level_hooks you could remove
> ebtables rules that might introduce some difference between the two
> hosts.
Thanks for the hint, but either I didn't use it right or it doesn't
solve the issue.
- Downloaded macspoof hook from github (as no package is available for
CentOS/RHEL in 3.2 repository)
- updated UserDefinedVMProperties and restarted engine
- set custom property macspoof = true
Btw, ebtables seems to be disabled/accept all anyway:
# ebtables -L
Bridge table: filter
Bridge chain: INPUT, entries: 0, policy: ACCEPT
Bridge chain: FORWARD, entries: 0, policy: ACCEPT
Bridge chain: OUTPUT, entries: 0, policy: ACCEPT
I also did something really nasty and created a Solaris 11 vm with
virt-manager on this hypervisor (it's not a production machine so I took
the risk to interfere with oVirt). And guess what - networking of this
Solaris 11 guest is fine on ovirtmgmt bridge.
So it seems to me that virt-manager is doing some magic for Solaris
guests oVirt doesn't...
Regards,
René
More information about the Users
mailing list