On 11/28/2012 01:52 AM, Karli Sjöberg wrote: > > 27 nov 2012 kl. 16.01 skrev : > >> >> 27 nov 2012 kl. 15.59 skrev Itamar Heim: >> >>> On 11/27/2012 09:56 AM, Karli Sjöberg wrote: >>>> >>>> 27 nov 2012 kl. 15.42 skrev Itamar Heim: >>>> >>>>> On 11/27/2012 08:28 AM, Karli Sjöberg wrote: >>>>>> Hey all! >>>>>> >>>>>> Since recently patching our hosts, I´ve been having trouble running FreeBSD guests with more than one virtual core or socket. I have managed take a screenshot of how it looks like when it panics when booting kernel, right after ACPI: >>>>>> http://i47.tinypic.com/2u90qrr.png >>>>>> >>>>>> I´ve tried this with similar results using 8.2-RELEASE, 8.3-RELEASE, 9.0-RELEASE and 9-STABLE. >>>>>> >>>>>> If I edit the guest to have only one virtual core or socket, it boots up without issue. >>>>>> >>>>>> Since noticing this, I´ve tried updating the packages one more time, thinking maybe it had already been fixed but no, it remains. These are the package versions I´m using: >>>>>> # rpm -qa | egrep '(kernel|libvirt|qemu|vdsm|seabios)' | sort -d >>>>>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>>>>> kernel-3.6.2-4.fc17.x86_64 >>>>>> kernel-3.6.3-1.fc17.x86_64 >>>>>> kernel-3.6.7-4.fc17.x86_64 << This is the one that´s running >>>>>> libvirt-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-client-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-daemon-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 >>>>>> libvirt-python-0.9.11.7-1.fc17.x86_64 >>>>>> qemu-common-1.0.1-2.fc17.x86_64 >>>>>> qemu-img-1.0.1-2.fc17.x86_64 >>>>>> qemu-kvm-1.0.1-2.fc17.x86_64 >>>>>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>>>>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>>>>> seabios-1.7.1-1.fc17.x86_64 >>>>>> seabios-bin-1.7.1-1.fc17.noarch >>>>>> vdsm-4.10.0-10.fc17.x86_64 >>>>>> vdsm-cli-4.10.0-10.fc17.noarch >>>>>> vdsm-python-4.10.0-10.fc17.x86_64 >>>>>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>>>>> >>>>>> Do you have any insights as to what the problem might be? >>>>>> >>>>>> Best Regards >>>>>> Karli Sjöberg >>>>>> _______________________________________________ >>>>>> Users mailing list >>>>>> Users(a)ovirt.org >>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>> >>>>> >>>>> the first suspect would be qemu-kvm and maybe the bios, can you please downgrade it to the previously >>>> >>>> Oh the fun of downgrading...pass:) But it just so happens that we have another ovirt system running, apart from the production system, that may be less patched. I´ll check and see if there´s any difference there. Is there any data you wish me to share, like logs or something while testing? Or just yay/nay? >>> >>> if we identify the offending package, and versions, easier to report the regression and ask to fix it >> >> So yay/nay it is! Thanks. >> > > I´ve now tested to create a new FreeBSD server with dual cores in our experiment/test system and it worked, no problemo. > > oVirt test system - good: > ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch > kernel-3.3.4-5.fc17.x86_64 > libvirt-0.9.11.6-1.fc17.x86_64 > libvirt-client-0.9.11.6-1.fc17.x86_64 > libvirt-daemon-0.9.11.6-1.fc17.x86_64 > libvirt-daemon-config-network-0.9.11.6-1.fc17.x86_64 > libvirt-daemon-config-nwfilter-0.9.11.6-1.fc17.x86_64 > libvirt-lock-sanlock-0.9.11.6-1.fc17.x86_64 > libvirt-python-0.9.11.6-1.fc17.x86_64 > qemu-common-1.0.1-2.fc17.x86_64 > qemu-img-1.0.1-2.fc17.x86_64 > qemu-kvm-1.0.1-2.fc17.x86_64 > qemu-kvm-tools-1.0.1-2.fc17.x86_64 > qemu-system-x86-1.0.1-2.fc17.x86_64 > seabios-1.7.0-1.fc17.x86_64 > seabios-bin-1.7.0-1.fc17.noarch > vdsm-4.10.0-10.fc17.x86_64 > vdsm-cli-4.10.0-10.fc17.noarch > vdsm-python-4.10.0-10.fc17.x86_64 > vdsm-xmlrpc-4.10.0-10.fc17.noarch > > oVirt prod system - bad: > ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch > kernel-3.6.7-4.fc17.x86_64 > libvirt-0.9.11.7-1.fc17.x86_64 > libvirt-client-0.9.11.7-1.fc17.x86_64 > libvirt-daemon-0.9.11.7-1.fc17.x86_64 > libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 > libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 > libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 > libvirt-python-0.9.11.7-1.fc17.x86_64 > qemu-common-1.0.1-2.fc17.x86_64 > qemu-img-1.0.1-2.fc17.x86_64 > qemu-kvm-1.0.1-2.fc17.x86_64 > qemu-kvm-tools-1.0.1-2.fc17.x86_64 > qemu-system-x86-1.0.1-2.fc17.x86_64 > seabios-1.7.1-1.fc17.x86_64 > seabios-bin-1.7.1-1.fc17.noarch > vdsm-4.10.0-10.fc17.x86_64 > vdsm-cli-4.10.0-10.fc17.noarch > vdsm-python-4.10.0-10.fc17.x86_64 > vdsm-xmlrpc-4.10.0-10.fc17.noarch > so seems like you have different: kernel seabios libvirt can you please upgrade them one by one to find the culprit (I'd do in this order for simplicity of rollback: seabios, libvirt, kernel)

thanks, Itamar

On 11/28/2012 08:39 AM, Karli Sjöberg wrote: > > 28 nov 2012 kl. 09.19 skrev Itamar Heim: > >> On 11/28/2012 01:52 AM, Karli Sjöberg wrote: >>> >>> 27 nov 2012 kl. 16.01 skrev : >>> >>>> >>>> 27 nov 2012 kl. 15.59 skrev Itamar Heim: >>>> >>>>> On 11/27/2012 09:56 AM, Karli Sjöberg wrote: >>>>>> >>>>>> 27 nov 2012 kl. 15.42 skrev Itamar Heim: >>>>>> >>>>>>> On 11/27/2012 08:28 AM, Karli Sjöberg wrote: >>>>>>>> Hey all! >>>>>>>> >>>>>>>> Since recently patching our hosts, I´ve been having trouble running FreeBSD guests with more than one virtual core or socket. I have managed take a screenshot of how it looks like when it panics when booting kernel, right after ACPI: >>>>>>>> http://i47.tinypic.com/2u90qrr.png >>>>>>>> >>>>>>>> I´ve tried this with similar results using 8.2-RELEASE, 8.3-RELEASE, 9.0-RELEASE and 9-STABLE. >>>>>>>> >>>>>>>> If I edit the guest to have only one virtual core or socket, it boots up without issue. >>>>>>>> >>>>>>>> Since noticing this, I´ve tried updating the packages one more time, thinking maybe it had already been fixed but no, it remains. These are the package versions I´m using: >>>>>>>> # rpm -qa | egrep '(kernel|libvirt|qemu|vdsm|seabios)' | sort -d >>>>>>>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>>>>>>> kernel-3.6.2-4.fc17.x86_64 >>>>>>>> kernel-3.6.3-1.fc17.x86_64 >>>>>>>> kernel-3.6.7-4.fc17.x86_64 << This is the one that´s running >>>>>>>> libvirt-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-client-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-daemon-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 >>>>>>>> libvirt-python-0.9.11.7-1.fc17.x86_64 >>>>>>>> qemu-common-1.0.1-2.fc17.x86_64 >>>>>>>> qemu-img-1.0.1-2.fc17.x86_64 >>>>>>>> qemu-kvm-1.0.1-2.fc17.x86_64 >>>>>>>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>>>>>>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>>>>>>> seabios-1.7.1-1.fc17.x86_64 >>>>>>>> seabios-bin-1.7.1-1.fc17.noarch >>>>>>>> vdsm-4.10.0-10.fc17.x86_64 >>>>>>>> vdsm-cli-4.10.0-10.fc17.noarch >>>>>>>> vdsm-python-4.10.0-10.fc17.x86_64 >>>>>>>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>>>>>>> >>>>>>>> Do you have any insights as to what the problem might be? >>>>>>>> >>>>>>>> Best Regards >>>>>>>> Karli Sjöberg >>>>>>>> _______________________________________________ >>>>>>>> Users mailing list >>>>>>>> Users(a)ovirt.org >>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>> >>>>>>> >>>>>>> the first suspect would be qemu-kvm and maybe the bios, can you please downgrade it to the previously >>>>>> >>>>>> Oh the fun of downgrading...pass:) But it just so happens that we have another ovirt system running, apart from the production system, that may be less patched. I´ll check and see if there´s any difference there. Is there any data you wish me to share, like logs or something while testing? Or just yay/nay? >>>>> >>>>> if we identify the offending package, and versions, easier to report the regression and ask to fix it >>>> >>>> So yay/nay it is! Thanks. >>>> >>> >>> I´ve now tested to create a new FreeBSD server with dual cores in our experiment/test system and it worked, no problemo. >>> >>> oVirt test system - good: >>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>> kernel-3.3.4-5.fc17.x86_64 >>> libvirt-0.9.11.6-1.fc17.x86_64 >>> libvirt-client-0.9.11.6-1.fc17.x86_64 >>> libvirt-daemon-0.9.11.6-1.fc17.x86_64 >>> libvirt-daemon-config-network-0.9.11.6-1.fc17.x86_64 >>> libvirt-daemon-config-nwfilter-0.9.11.6-1.fc17.x86_64 >>> libvirt-lock-sanlock-0.9.11.6-1.fc17.x86_64 >>> libvirt-python-0.9.11.6-1.fc17.x86_64 >>> qemu-common-1.0.1-2.fc17.x86_64 >>> qemu-img-1.0.1-2.fc17.x86_64 >>> qemu-kvm-1.0.1-2.fc17.x86_64 >>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>> seabios-1.7.0-1.fc17.x86_64 >>> seabios-bin-1.7.0-1.fc17.noarch >>> vdsm-4.10.0-10.fc17.x86_64 >>> vdsm-cli-4.10.0-10.fc17.noarch >>> vdsm-python-4.10.0-10.fc17.x86_64 >>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>> >>> oVirt prod system - bad: >>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>> kernel-3.6.7-4.fc17.x86_64 >>> libvirt-0.9.11.7-1.fc17.x86_64 >>> libvirt-client-0.9.11.7-1.fc17.x86_64 >>> libvirt-daemon-0.9.11.7-1.fc17.x86_64 >>> libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 >>> libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 >>> libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 >>> libvirt-python-0.9.11.7-1.fc17.x86_64 >>> qemu-common-1.0.1-2.fc17.x86_64 >>> qemu-img-1.0.1-2.fc17.x86_64 >>> qemu-kvm-1.0.1-2.fc17.x86_64 >>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>> seabios-1.7.1-1.fc17.x86_64 >>> seabios-bin-1.7.1-1.fc17.noarch >>> vdsm-4.10.0-10.fc17.x86_64 >>> vdsm-cli-4.10.0-10.fc17.noarch >>> vdsm-python-4.10.0-10.fc17.x86_64 >>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>> >> >> so seems like you have different: >> kernel >> seabios >> libvirt >> >> can you please upgrade them one by one to find the culprit (I'd do in this order for simplicity of rollback: seabios, libvirt, kernel) > > Done. Kernel is the culprit! > > We have two hosts in the test system, so what I did was to first update seabios; test, then update libvirt; test, kernel; test on host1, where all was well until after booting new kernel. So I wanted to know for sure that it was only the kernel's blame, so on host2 only kernel was updated and afterwards the issue started appearing. So definitely, kernel. > > PS. As I expected, downgrading the packages again failed horribly. I friggin' hate trying to downgrade, even for just a few packages, I opt-out and rather just reinstall the whole thing to save me the headache:) > please open a bug on fedora kernel, and paste here. thanks well, i guess next question is does fedora 18 kernel solves the issue...

On 11/29/2012 01:52 AM, Karli Sjöberg wrote: > > 28 nov 2012 kl. 17.07 skrev Itamar Heim: > >> On 11/28/2012 08:39 AM, Karli Sjöberg wrote: >>> >>> 28 nov 2012 kl. 09.19 skrev Itamar Heim: >>> >>>> On 11/28/2012 01:52 AM, Karli Sjöberg wrote: >>>>> >>>>> 27 nov 2012 kl. 16.01 skrev : >>>>> >>>>>> >>>>>> 27 nov 2012 kl. 15.59 skrev Itamar Heim: >>>>>> >>>>>>> On 11/27/2012 09:56 AM, Karli Sjöberg wrote: >>>>>>>> >>>>>>>> 27 nov 2012 kl. 15.42 skrev Itamar Heim: >>>>>>>> >>>>>>>>> On 11/27/2012 08:28 AM, Karli Sjöberg wrote: >>>>>>>>>> Hey all! >>>>>>>>>> >>>>>>>>>> Since recently patching our hosts, I´ve been having trouble >>>>>>>>>> running FreeBSD guests with more than one virtual core or >>>>>>>>>> socket. I have managed take a screenshot of how it looks like >>>>>>>>>> when it panics when booting kernel, right after ACPI: >>>>>>>>>> http://i47.tinypic.com/2u90qrr.png >>>>>>>>>> >>>>>>>>>> I´ve tried this with similar results using 8.2-RELEASE, >>>>>>>>>> 8.3-RELEASE, 9.0-RELEASE and 9-STABLE. >>>>>>>>>> >>>>>>>>>> If I edit the guest to have only one virtual core or socket, >>>>>>>>>> it boots up without issue. >>>>>>>>>> >>>>>>>>>> Since noticing this, I´ve tried updating the packages one >>>>>>>>>> more time, thinking maybe it had already been fixed but no, >>>>>>>>>> it remains. These are the package versions I´m using: >>>>>>>>>> # rpm -qa | egrep '(kernel|libvirt|qemu|vdsm|seabios)' | sort -d >>>>>>>>>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>>>>>>>>> kernel-3.6.2-4.fc17.x86_64 >>>>>>>>>> kernel-3.6.3-1.fc17.x86_64 >>>>>>>>>> kernel-3.6.7-4.fc17.x86_64 << This is the one that´s running >>>>>>>>>> libvirt-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-client-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-daemon-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> libvirt-python-0.9.11.7-1.fc17.x86_64 >>>>>>>>>> qemu-common-1.0.1-2.fc17.x86_64 >>>>>>>>>> qemu-img-1.0.1-2.fc17.x86_64 >>>>>>>>>> qemu-kvm-1.0.1-2.fc17.x86_64 >>>>>>>>>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>>>>>>>>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>>>>>>>>> seabios-1.7.1-1.fc17.x86_64 >>>>>>>>>> seabios-bin-1.7.1-1.fc17.noarch >>>>>>>>>> vdsm-4.10.0-10.fc17.x86_64 >>>>>>>>>> vdsm-cli-4.10.0-10.fc17.noarch >>>>>>>>>> vdsm-python-4.10.0-10.fc17.x86_64 >>>>>>>>>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>>>>>>>>> >>>>>>>>>> Do you have any insights as to what the problem might be? >>>>>>>>>> >>>>>>>>>> Best Regards >>>>>>>>>> Karli Sjöberg >>>>>>>>>> _______________________________________________ >>>>>>>>>> Users mailing list >>>>>>>>>> Users(a)ovirt.org >>>>>>>>>> http://lists.ovirt.org/mailman/listinfo/users >>>>>>>>>> >>>>>>>>> >>>>>>>>> the first suspect would be qemu-kvm and maybe the bios, can >>>>>>>>> you please downgrade it to the previously >>>>>>>> >>>>>>>> Oh the fun of downgrading...pass:) But it just so happens that >>>>>>>> we have another ovirt system running, apart from the production >>>>>>>> system, that may be less patched. I´ll check and see if there´s >>>>>>>> any difference there. Is there any data you wish me to share, >>>>>>>> like logs or something while testing? Or just yay/nay? >>>>>>> >>>>>>> if we identify the offending package, and versions, easier to >>>>>>> report the regression and ask to fix it >>>>>> >>>>>> So yay/nay it is! Thanks. >>>>>> >>>>> >>>>> I´ve now tested to create a new FreeBSD server with dual cores in >>>>> our experiment/test system and it worked, no problemo. >>>>> >>>>> oVirt test system - good: >>>>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>>>> kernel-3.3.4-5.fc17.x86_64 >>>>> libvirt-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-client-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-daemon-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-daemon-config-network-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-daemon-config-nwfilter-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-lock-sanlock-0.9.11.6-1.fc17.x86_64 >>>>> libvirt-python-0.9.11.6-1.fc17.x86_64 >>>>> qemu-common-1.0.1-2.fc17.x86_64 >>>>> qemu-img-1.0.1-2.fc17.x86_64 >>>>> qemu-kvm-1.0.1-2.fc17.x86_64 >>>>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>>>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>>>> seabios-1.7.0-1.fc17.x86_64 >>>>> seabios-bin-1.7.0-1.fc17.noarch >>>>> vdsm-4.10.0-10.fc17.x86_64 >>>>> vdsm-cli-4.10.0-10.fc17.noarch >>>>> vdsm-python-4.10.0-10.fc17.x86_64 >>>>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>>>> >>>>> oVirt prod system - bad: >>>>> ipxe-roms-qemu-20120328-1.gitaac9718.fc17.noarch >>>>> kernel-3.6.7-4.fc17.x86_64 >>>>> libvirt-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-client-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-daemon-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-daemon-config-network-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-daemon-config-nwfilter-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-lock-sanlock-0.9.11.7-1.fc17.x86_64 >>>>> libvirt-python-0.9.11.7-1.fc17.x86_64 >>>>> qemu-common-1.0.1-2.fc17.x86_64 >>>>> qemu-img-1.0.1-2.fc17.x86_64 >>>>> qemu-kvm-1.0.1-2.fc17.x86_64 >>>>> qemu-kvm-tools-1.0.1-2.fc17.x86_64 >>>>> qemu-system-x86-1.0.1-2.fc17.x86_64 >>>>> seabios-1.7.1-1.fc17.x86_64 >>>>> seabios-bin-1.7.1-1.fc17.noarch >>>>> vdsm-4.10.0-10.fc17.x86_64 >>>>> vdsm-cli-4.10.0-10.fc17.noarch >>>>> vdsm-python-4.10.0-10.fc17.x86_64 >>>>> vdsm-xmlrpc-4.10.0-10.fc17.noarch >>>>> >>>> >>>> so seems like you have different: >>>> kernel >>>> seabios >>>> libvirt >>>> >>>> can you please upgrade them one by one to find the culprit (I'd do >>>> in this order for simplicity of rollback: seabios, libvirt, kernel) >>> >>> Done. Kernel is the culprit! >>> >>> We have two hosts in the test system, so what I did was to first >>> update seabios; test, then update libvirt; test, kernel; test on >>> host1, where all was well until after booting new kernel. So I >>> wanted to know for sure that it was only the kernel's blame, so on >>> host2 only kernel was updated and afterwards the issue started >>> appearing. So definitely, kernel. >>> >>> PS. As I expected, downgrading the packages again failed horribly. I >>> friggin' hate trying to downgrade, even for just a few packages, I >>> opt-out and rather just reinstall the whole thing to save me the >>> headache:) >>> >> >> please open a bug on fedora kernel, and paste here. thanks >> well, i guess next question is does fedora 18 kernel solves the issue... >> >> > > Done! https://bugzilla.redhat.com/show_bug.cgi?id=881579 > thanks. dor/karen - is there anyone who can take a look at this fedora regression causing SMP vm's to panic with a kernel upgrade?

thanks, Itamar