On Mon, 2013-09-16 at 15:14 +0100, Dan Kenigsberg wrote:
On Mon, Sep 16, 2013 at 12:33:34PM +0200, René Koch (ovido) wrote:
> 
> On Fri, 2013-09-06 at 22:26 +0100, Dan Kenigsberg wrote:
> 
> > On Fri, Sep 06, 2013 at 06:23:42PM +0200, René Koch (ovido) wrote:
> > > Hi,
> > > 
> > > I want to start the discussion about Solaris support on oVirt again, as
> > > there was no solution for it yet.
> > > 
> > > On my oVirt 3.2.2 environment I installed Solaris 11 U1 with the
> > > following specs:
> > > * Operating System: Other
> > > * nic1: rtl8139
> > > * Disk1: IDE (Thin Provision)
> > > * Host: CentOS 6.4 with qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64
> > > 
> > > These are the same settings as on my RHEL 6.4 KVM host (except I can
> > > choose Solaris 10 as OS in virt-manager), which has KVM version:
> > > qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2.x86_64 (I wanted to use this host
> > > as a RHEV host, so the qemu-kvm-rhev package is installed in case you
> > > wounder)...
> > > 
> > > What's working:
> > > * OS installation on IDE disk
> > > * Bringing up network interface
> > > 
> > > What's not working on oVirt:
> > > * Network connections  - on RHEL 6.4 with plain libvirt/kvm this is
> > > working...
> > > 
> > > 
> > > I can see the mac address on my CentOS host, but can't ping the Solaris
> > > vm:
> > > 
> > > # brctl showmacs ovirtmgmt | egrep '00:99:4a:00:64:83|port'
> > > port no	mac addr		is local?	ageing timer
> > >   2	00:99:4a:00:64:83	no		  10.72
> > > 
> > > # arp -an | grep '00:99:4a:00:64:83'
> > > ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
> > > 
> > > When using tcpdump on the vnet interface which belongs to the Solaris vm
> > > (ip 10.0.100.123) I can see ARP requests from the vm for ip address of
> > > my CentOS host (10.0.100.42) but no response to it. Same when pinging
> > > other ips in this network:
> > > 
> > > # tcpdump -n -i vnet2
> > > tcpdump: WARNING: vnet2: no IPv4 address assigned
> > > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > > decode
> > > listening on vnet2, link-type EN10MB (Ethernet), capture size 65535
> > > bytes
> > > 18:15:35.987868 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > > 18:15:36.487399 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > > 18:15:36.987536 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > > 
> > > I also compared the qemu-kvm process list on the KVM with the oVirt
> > > machine and can't see much differences except that oVirt has more
> > > information like smbios....
> > > oVirt host:
> > > /usr/libexec/qemu-kvm
> > > <snip>
> > >  -netdev tap,fd=27,id=hostnet0
> > >  -device
> > > rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3
> > > 
> > > RHEL KVM host:
> > > /usr/libexec/qemu-kvm
> > > <snip>
> > >  -netdev tap,fd=32,id=hostnet0
> > >  -device
> > > rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:c2:97,bus=pci.0,addr=0x3
> > > 
> > > Any suggestions on how to troubleshoot / get Solaris networking running
> > > is welcome.
> > > Changing the interface to e1000 doesn't work either.
> 
> 
> 
> Thanks a lot for your answer and sorry for my late response - I was sick
> leaving last week.
> 
> 
> > 
> > - Would you share the output of your `brctl show`?
> 
> 
> # brctl show
> bridge name	bridge id		STP enabled	interfaces
> ;vdsmdummy;		8000.000000000000	no		
> dmz		8000.0025901ad140	no		eth0.200
> ovirtmgmt		8000.0025901ad140	no		eth0.100
> 							vnet1
> 							vnet2
> setup		8000.0025901ad140	no		eth0.500
> test2		8000.0025901ad140	no		eth0.602
> 							vnet0
> 
> vnet1 is a Debian 7 vm which has a perfectly working network.
> vnet2 is the Solaris 11 vm which can't communicate over the network.
> 
> 
> > - Particulalry, does the host device with 10.0.100.42 sit on the same
> >   bridge as your Solaris VM?
> 
> 
> 10.0.100.42 is the ip of the oVirt host and it's ip is configured on
> this bridge (and reachable from the Debian 7 vm):
> 
> # ip a | grep ovirt
> 5: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UNKNOWN 
>     inet 10.0.100.42/24 brd 10.0.100.255 scope global ovirtmgmt
> 
> 
> > - Do you see the who-has packet when sniffing the bridge device?
> 
> 
> 
> Yes, I can see them:
> 12:24:16.311218 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> 10.0.100.123, length 46
> 12:24:16.311246 ARP, Reply 10.0.100.42 is-at 00:25:90:1a:d1:40, length
> 28
> 
> I can also see the MAC of the Solaris vm in arp cache of oVirt host:
> # arp -an
> ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
> 
> But I can't see the MAC address of oVirt host (10.0.100.42) in arp table
> of the Solaris 11 vm. To be more clear: I can't see any MAC address in
> ARP table of Solaris vm except the MAC of the vm itself.
> 
> 
> > - Have yout tried guest-to-guest communication (over the same bridge)?
> 
> 
> I can communicate from the Debian 7 vm to all hosts in my network, but I
> can't reach the Solaris 11 vm.
> The network of the oVirt setup is working fine for Windows and various
> Linux guests, but not for Solaris.
> 
> Btw, excatly same behavior on RHEV 3.2 with Solaris 10/11 - Windows and
> Linux is working as expected, but no networking for Solaris guests...

I'm afraid I'm out of ideas. Is the kernel the same on both hosts?
Would you reply with the complete qemu commandline? Maybe Michael could
sport the difference between the working plain-KVM host and the
non-working oVirt host.


The kernel is not the same:
CentOS 6 (oVirt): 2.6.32-358.18.1.el6.x86_64
RHEL 6 (KVM): 2.6.32-358.11.1.el6.x86_64
But I think it's not kernel related as I successfully ran Solaris on the KVM server 1 year ago (so kernel was an old RHEL 6.2 or 6.3 kernel) and didn't worked this time on oVirt 3.1, too (hypervisor was CentOS 6.2 or 6.3 using dreyou's-repository, now I'm running CentOS 6.4 with official oVirt repository). So many kernel upgrades later Solaris still has a working network on plain KVM but not on oVirt 3.2 or RHEV 3.2.

Here's the complete qemu cmd:

RHEL 6.4 (plain KVM - Solaris networking is fine):
/usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid e7d3b9a4-8605-dc4f-9092-41d68dd83735 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/solaris11.img,if=none,id=drive-ide0-0-0,format=raw -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=40,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f0:27:d3,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -vnc 127.0.0.1:13 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

CentOS 6.4 (oVirt - Solaris can't reach other nodes):
/usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -cpu Nehalem -enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5 -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-4.el6.centos.10,serial=54443858-4E54-2500-901A-0025901AD140,uuid=cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5,sku=solaris11 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2013-09-06T15:08:22,driftfix=slew -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive file=/rhev/data-center/484e62d7-7a01-4b5e-aec8-59d366100281/2055b80d-d59d-4fe6-80ec-5499674feed5/images/0786a273-780a-4f58-acc4-9b123953bde8/0c07f1f9-f9c0-4ac0-bfec-bee7a1c35a0b,if=none,id=drive-ide0-0-0,format=raw,serial=0786a273-780a-4f58-acc4-9b123953bde8,cache=none,werror=stop,rerror=stop,aio=threads -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/var/run/vdsm/payload/cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5.f3b4e68b93e9d35d3f17328fae2e178e.img,if=none,media=cdrom,id=drive-ide0-1-1,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=27,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/solaris11.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/solaris11.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev spicevmc,id=charchannel2,name=vdagent -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0 -spice port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on -k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5


Thanks,
René



Regrads,
Dan.