On Mon, 2013-09-16 at 15:14 +0100, Dan Kenigsberg wrote:
On Mon, Sep 16, 2013 at 12:33:34PM +0200, René Koch (ovido) wrote:
>
> On Fri, 2013-09-06 at 22:26 +0100, Dan Kenigsberg wrote:
>
> > On Fri, Sep 06, 2013 at 06:23:42PM +0200, René Koch (ovido) wrote:
> > > Hi,
> > >
> > > I want to start the discussion about Solaris support on oVirt again, as
> > > there was no solution for it yet.
> > >
> > > On my oVirt 3.2.2 environment I installed Solaris 11 U1 with the
> > > following specs:
> > > * Operating System: Other
> > > * nic1: rtl8139
> > > * Disk1: IDE (Thin Provision)
> > > * Host: CentOS 6.4 with qemu-kvm-0.12.1.2-2.355.0.1.el6.centos.7.x86_64
> > >
> > > These are the same settings as on my RHEL 6.4 KVM host (except I can
> > > choose Solaris 10 as OS in virt-manager), which has KVM version:
> > > qemu-kvm-rhev-0.12.1.2-2.295.el6_3.2.x86_64 (I wanted to use this host
> > > as a RHEV host, so the qemu-kvm-rhev package is installed in case you
> > > wounder)...
> > >
> > > What's working:
> > > * OS installation on IDE disk
> > > * Bringing up network interface
> > >
> > > What's not working on oVirt:
> > > * Network connections - on RHEL 6.4 with plain libvirt/kvm this is
> > > working...
> > >
> > >
> > > I can see the mac address on my CentOS host, but can't ping the
Solaris
> > > vm:
> > >
> > > # brctl showmacs ovirtmgmt | egrep '00:99:4a:00:64:83|port'
> > > port no mac addr is local? ageing timer
> > > 2 00:99:4a:00:64:83 no 10.72
> > >
> > > # arp -an | grep '00:99:4a:00:64:83'
> > > ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
> > >
> > > When using tcpdump on the vnet interface which belongs to the Solaris vm
> > > (ip 10.0.100.123) I can see ARP requests from the vm for ip address of
> > > my CentOS host (10.0.100.42) but no response to it. Same when pinging
> > > other ips in this network:
> > >
> > > # tcpdump -n -i vnet2
> > > tcpdump: WARNING: vnet2: no IPv4 address assigned
> > > tcpdump: verbose output suppressed, use -v or -vv for full protocol
> > > decode
> > > listening on vnet2, link-type EN10MB (Ethernet), capture size 65535
> > > bytes
> > > 18:15:35.987868 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > > 18:15:36.487399 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > > 18:15:36.987536 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> > > 10.0.100.123, length 46
> > >
> > > I also compared the qemu-kvm process list on the KVM with the oVirt
> > > machine and can't see much differences except that oVirt has more
> > > information like smbios....
> > > oVirt host:
> > > /usr/libexec/qemu-kvm
> > > <snip>
> > > -netdev tap,fd=27,id=hostnet0
> > > -device
> > > rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3
> > >
> > > RHEL KVM host:
> > > /usr/libexec/qemu-kvm
> > > <snip>
> > > -netdev tap,fd=32,id=hostnet0
> > > -device
> > > rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:c2:97,bus=pci.0,addr=0x3
> > >
> > > Any suggestions on how to troubleshoot / get Solaris networking running
> > > is welcome.
> > > Changing the interface to e1000 doesn't work either.
>
>
>
> Thanks a lot for your answer and sorry for my late response - I was sick
> leaving last week.
>
>
> >
> > - Would you share the output of your `brctl show`?
>
>
> # brctl show
> bridge name bridge id STP enabled interfaces
> ;vdsmdummy; 8000.000000000000 no
> dmz 8000.0025901ad140 no eth0.200
> ovirtmgmt 8000.0025901ad140 no eth0.100
> vnet1
> vnet2
> setup 8000.0025901ad140 no eth0.500
> test2 8000.0025901ad140 no eth0.602
> vnet0
>
> vnet1 is a Debian 7 vm which has a perfectly working network.
> vnet2 is the Solaris 11 vm which can't communicate over the network.
>
>
> > - Particulalry, does the host device with 10.0.100.42 sit on the same
> > bridge as your Solaris VM?
>
>
> 10.0.100.42 is the ip of the oVirt host and it's ip is configured on
> this bridge (and reachable from the Debian 7 vm):
>
> # ip a | grep ovirt
> 5: ovirtmgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UNKNOWN
> inet 10.0.100.42/24 brd 10.0.100.255 scope global ovirtmgmt
>
>
> > - Do you see the who-has packet when sniffing the bridge device?
>
>
>
> Yes, I can see them:
> 12:24:16.311218 ARP, Request who-has 10.0.100.42 (Broadcast) tell
> 10.0.100.123, length 46
> 12:24:16.311246 ARP, Reply 10.0.100.42 is-at 00:25:90:1a:d1:40, length
> 28
>
> I can also see the MAC of the Solaris vm in arp cache of oVirt host:
> # arp -an
> ? (10.0.100.123) at 00:99:4a:00:64:83 [ether] on ovirtmgmt
>
> But I can't see the MAC address of oVirt host (10.0.100.42) in arp table
> of the Solaris 11 vm. To be more clear: I can't see any MAC address in
> ARP table of Solaris vm except the MAC of the vm itself.
>
>
> > - Have yout tried guest-to-guest communication (over the same bridge)?
>
>
> I can communicate from the Debian 7 vm to all hosts in my network, but I
> can't reach the Solaris 11 vm.
> The network of the oVirt setup is working fine for Windows and various
> Linux guests, but not for Solaris.
>
> Btw, excatly same behavior on RHEV 3.2 with Solaris 10/11 - Windows and
> Linux is working as expected, but no networking for Solaris guests...
I'm afraid I'm out of ideas. Is the kernel the same on both hosts?
Would you reply with the complete qemu commandline? Maybe Michael could
sport the difference between the working plain-KVM host and the
non-working oVirt host.
The kernel is not the same:
CentOS 6 (oVirt): 2.6.32-358.18.1.el6.x86_64
RHEL 6 (KVM): 2.6.32-358.11.1.el6.x86_64
But I think it's not kernel related as I successfully ran Solaris on the
KVM server 1 year ago (so kernel was an old RHEL 6.2 or 6.3 kernel) and
didn't worked this time on oVirt 3.1, too (hypervisor was CentOS 6.2 or
6.3 using dreyou's-repository, now I'm running CentOS 6.4 with official
oVirt repository). So many kernel upgrades later Solaris still has a
working network on plain KVM but not on oVirt 3.2 or RHEV 3.2.
Here's the complete qemu cmd:
RHEL 6.4 (plain KVM - Solaris networking is fine):
/usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -enable-kvm -m
1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
e7d3b9a4-8605-dc4f-9092-41d68dd83735 -nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/var/lib/libvirt/images/solaris11.img,if=none,id=drive-ide0-0-0,format=raw -device
ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=40,id=hostnet0
-device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f0:27:d3,bus=pci.0,addr=0x3 -chardev
pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device
usb-tablet,id=input0 -vnc 127.0.0.1:13 -vga cirrus -device
intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
CentOS 6.4 (oVirt - Solaris can't reach other nodes):
/usr/libexec/qemu-kvm -name solaris11 -S -M rhel6.3.0 -cpu Nehalem
-enable-kvm -m 1024 -smp 1,sockets=1,cores=1,threads=1 -uuid
cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5 -smbios
type=1,manufacturer=oVirt,product=oVirt
Node,version=6-4.el6.centos.10,serial=54443858-4E54-2500-901A-0025901AD140,uuid=cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5,sku=solaris11
-nodefconfig -nodefaults -chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/solaris11.monitor,server,nowait -mon
chardev=charmonitor,id=monitor,mode=control -rtc base=2013-09-06T15:08:22,driftfix=slew
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device
virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 -drive
file=/rhev/data-center/484e62d7-7a01-4b5e-aec8-59d366100281/2055b80d-d59d-4fe6-80ec-5499674feed5/images/0786a273-780a-4f58-acc4-9b123953bde8/0c07f1f9-f9c0-4ac0-bfec-bee7a1c35a0b,if=none,id=drive-ide0-0-0,format=raw,serial=0786a273-780a-4f58-acc4-9b123953bde8,cache=none,werror=stop,rerror=stop,aio=threads
-device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -drive
if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device
ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive
file=/var/run/vdsm/payload/cbaa0cf5-d9e6-40e4-a269-9dc495f4c0d5.f3b4e68b93e9d35d3f17328fae2e178e.img,if=none,media=cdrom,id=drive-ide0-1-1,readonly=on,format=raw,serial=
-device ide-drive,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev
tap,fd=27,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=00:99:4a:00:64:83,bus=pci.0,addr=0x3 -chardev
socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/solaris11.com.redhat.rhevm.vdsm,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm
-chardev
socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/solaris11.org.qemu.guest_agent.0,server,nowait
-device
virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0
-chardev spicevmc,id=charchannel2,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=com.redhat.spice.0
-spice
port=5902,tls-port=5903,addr=0,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
-k en-us -vga qxl -global qxl-vga.ram_size=67108864 -global qxl-vga.vram_size=67108864
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
Thanks,
René
Regrads,
Dan.