- Devel - oVirt List Archives

ovirt test day report - make ovirt easy configurable to allow redirection of all logs to syslog
by Alona Kaplan 06 Jul '14

06 Jul '14

During the test day I tested - Bug 1078738 - OVIRT35 - [RFE] make ovirt easy configurable to allow redirection of all logs to syslog There was no feature page so it was very hard to understand how this feature should be tested. The build contains only a patch that tailors the api to the engine. abarlev sent me an external rpm with the syslog extension- tested it on f19, seems to work fine. Alona.

2 1

Summary of my 3.5 oVirt Test Day - BZ1108866, BZ1108866
by Maor Lipchuk 06 Jul '14

06 Jul '14

Hi all, I was testing two features of 3.5 in the oVirt test day The first feature was: [RFE] Maintenance operations on a VM would ask for an optional reason (BZ1108866) The feature provides the ability to present a message to the user regarding the reason why this VM has been shutdown. It seems to work very well except when using shutdown while the VM is powering up or shutting down, then the message gets deleted. The other feature I was testing was: Allow to clone a (down) VM without snapshot/template (BZ1108866) This feature provides the ability to clone a VM from the VM itself, instead do it from a snapshot. I have tested many different scenarios such as cloning a VM which was in PREVIEW, VM containing shared disks and read only disks. The feature works very well, and the previewed VM which was cloned, was similar to the cloned one. Regards, Maor

1 0

compile vdsm and attach it to a engine
by aaron Beein 05 Jul '14

05 Jul '14

Hi, Thank you for your great job on ovirt and vdsm. Now I devote myself to compile vdsm on centos 6.3 host and attach it to a ovirt engine. But when I attach the host which contains a compiled vdsm to a ovirt engine , the status of the host is always ‘Non Responsive’(step 11 below). I reference the links below: http://www.ovirt.org/Vdsm_Developers http://www.ovirt.org/Installing_VDSM_from_rpm The steps( 1-9 ) are executed on centos6.3 host, and the steps(10--11) are executed on ovirt engine. So I would be very grateful if you can give me some clues that if I've missed anything or I done something wrong. The attachment is the same as the bellow which makes it easier for you to read. 1 Deployment platform Centos6.3 Linux bogon 2.6.32-431.20.3.el6.x86_64 #1 SMP Thu Jun 19 21:14:45 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Ip ： 10.1.8.252 CPU supports hardware virtualization extensions: # cat /proc/cpuinfo | egrep 'svm|vmx'| grep nx flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave lahf_lm arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 popcnt tsc_deadline_timer xsave lahf_lm arat epb xsaveopt pln pts dts tpr_shadow vnmi flexpriority ept vpid 2 Apply all updates # yum -y update 3 Installing required packages RHEL 6 users must add EPEL yum repository for installing python-ordereddict and pyton-pthreading. The rpm bellow will install the epel yum repo and required gpg keys. # yum install http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.r… RHEL 6 users must install a newer pep8 version than the one shipped in EPEL6. Older pep8 versions have a bug that's tickled by vdsm. You can use `pip`, or yum install http://danken.fedorapeople.org/python-pep8-1.4.5-2.el6.noarch.rpm oVirt repo: yum install http://resources.ovirt.org/releases/ovirt-release.noarch.rpm RHEL 6 users must add the glusterfs repository, providing newer glusterfs not available on RHEL 6. Optionally install 'wget' if not present rpm -q wget 2> /dev/null || yum install wget wget -O /etc/yum.repos.d/glusterfs-epel.repo *http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterfs-epel.repo* <http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/glusterf…> Fedora and RHEL 6 users must verify the following packages are installed before attempting to build: yum install make autoconf automake pyflakes logrotate gcc python-pep8 libvirt-python python-devel \ python-nose rpm-build sanlock-python genisoimage python-ordereddict python-pthreading libselinux-python\ python-ethtool m2crypto python-dmidecode python-netaddr python-inotify python-argparse git \ python-cpopen bridge-utils libguestfs-tools-c pyparted openssl libnl libtool gettext-devel python-ioprocess libvirt libvirt-client libvirt-lock-sanlock 4 Getting the source cd /root git clone *http://gerrit.ovirt.org/p/vdsm.git* <http://gerrit.ovirt.org/p/vdsm.git> cd vdsm 5 Building a Vdsm RPM ./autogen.sh –system ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --libdir=/usr/lib --enable-hooks make rpm NOSE_EXCLUDE=.* 6 Basic installation and start When building from source, you should enable the ovirt-beta repository, to satisfy dependencies that are not available yet in the release repository. # cd ~/rpmbuild/RPMS # yum install --skip-broken --enablerepo=ovirt-master-snapshot-static x86_64/* noarch/vdsm-xml* noarch/vdsm-cli* noarch/vdsm-python-zombiereaper* noarch/vdsm-*jsonrpc* Before starting vdsmd service for the first time vdsm requires some configuration procedures for external services that being used by vdsmd. To ease this process vdsm provides a utility (vdsm-tool). To perform full reconfiguration of external services perform: # vdsm-tool configure --force (for more information read "vdsm-tool --help") 7 Finally start the vdsmd service # service vdsmd start 8 Yum install -y bridge-utils Configuring the bridge Interface as below Disable the network manager service by executing as root: systemctl stop NetworkManager.service systemctl disable NetworkManager.service service network start chkconfig network on Add the following content into a new file named: */etc/sysconfig/network-scripts/ifcfg-ovirtmgmt*: DEVICE=ovirtmgmt TYPE=Bridge ONBOOT=yes DELAY=0 BOOTPROTO=static IPADDR=10.1.8.252 NETMASK=255.255.255.0 GATEWAY=10.1.8.254 Add the following line into the configuration file of your out going interface (usually em1/eth0) the file is located at: */etc/sysconfig/network-scripts/ifcfg-em1* (assuming the device is em1) BRIDGE=ovirtmgmt and remove the IPADDR, NETMASK and BOOTPROTO keys, since the interface should not have an IP address of its own. Full Example DEVICE=em1 ONBOOT=yes BRIDGE=ovirtmgmt Restart the network service by executing: service network restart *Note that if any other bridge (from ovirtmgmt) is present at the time of host installation, the bridge creation operation is skipped and you have to change the bridge settings to correspond to above shown configuration manually.* *9 Configuring VDSM* Add the following content into the file: */etc/vdsm/vdsm.conf* (you may need to create that file): [vars] ssl = false Restart the vdsmd service by executing: service vdsmd restart If Vdsm was started earlier with ssl=true, it would refuse to start and you may need to use the undocumented verb service vdsmd reconfigure service vdsmd start which edits */etc/libvirt/qemu.conf* and changes *spice_tls=1* to *spice_tls=0*. 10 Connect to overt-engine *ref: OVirt_-_connecting_development_vdsm_to_ovirt_engine <http://www.ovirt.org/OVirt_-_connecting_development_vdsm_to_ovirt_engine>.* *su - postgres -c "psql engine -c \"UPDATE vdc_options set option_value = 'true' where option_name = 'InstallVds'\""* *service overt-engine restart* 11 Attach the host to the engine I login the engine Administration Portal , and attach the centos host to a cluster. But it failed.

2 1

Re: [ovirt-devel] ovirt-engine-3.5 branching
by Oved Ourfali 03 Jul '14

03 Jul '14

On Jul 3, 2014 8:27 PM, Antoni Segura Puimedon <asegurap(a)redhat.com> wrote: > > > > ----- Original Message ----- > > From: "Allon Mureinik" <amureini(a)redhat.com> > > To: "Oved Ourfali" <ovedo(a)redhat.com> > > Cc: "Piotr Kliczewski" <pkliczew(a)redhat.com>, devel(a)ovirt.org > > Sent: Thursday, July 3, 2014 4:57:55 PM > > Subject: Re: [ovirt-devel] ovirt-engine-3.5 branching > > > > I concur. > > > > There are too many flows broken on /master/ to consider the 3.5 branch > > anything remotely near "stable". > > Wouldn't it be better to keep the current branch as "stabilization branch" > and test extensively every patch that goes into it instead of keeping adding > to the master branch and rebranch and then have the same or similar happen > in the next test day? > > If I remember correctly in the previous release cycle something similar happened > in the engine an teams tried to push non critical or stabilization patches > after feature freeze. At the time, it was argued that this release cycle it > would be branch and backport. > > I realize, of course, that it is painstaking to backport a great amount of > patches, but this is a direct result of letting features get merged too late > in the cycle and before being up to a certain standard of stability. > > I would say "let this backporting frenzy be a lesson to all to be more conservative > with the timelines in the next cycle" but I understand the other side of the > argument, so maybe instead we should just count with an extra week between > freeze and branching (note that this will delay review and merge of work > on master for the next feature reducing the chances of big features being > merged early-middle cycle. > I think that we all learn the lesson here, and we are just increasing the risk of having things not back ported due to human error, and by that reducing the stabilization... While we know that a lot was fixed. And also some bugs, and not just things that were added at the last minute. > > > > wrt to holding off 3.6 features, I can confirm that from the storage side > > nothing has been merged, and we can keep holding them back. > > > > > > -Allon > > > > ----- Original Message ----- > > > From: "Oved Ourfali" <ovedo(a)redhat.com> > > > To: devel(a)ovirt.org > > > Cc: "Piotr Kliczewski" <pkliczew(a)redhat.com> > > > Sent: Thursday, July 3, 2014 5:31:43 PM > > > Subject: [ovirt-devel] ovirt-engine-3.5 branching > > > > > > Hi all, > > > > > > The test day revealed a large amount of issues. These issues are being > > > addressed in the last few days. > > > To avoid the need to back-port each and every one of them to the > > > ovirt-engine-3.5 stable branch, I suggest to give a few days for that > > > effort, > > > and revisit it on mid next week, to asses it again and decide whether to do > > > the branching then. > > > > > > I ask the different maintainers not to push 3.6 relevant material into > > > master > > > in the next few days, until the branching is done. > > > To my knowledge no major (or any) patch related to 3.6 has been merge on > > > master, but please correct me if I'm wrong. > > > > > > Thanks all for your efforts in stabilizing the version. > > > > > > Regards, > > > Oved > > > _______________________________________________ > > > Devel mailing list > > > Devel(a)ovirt.org > > > http://lists.ovirt.org/mailman/listinfo/devel > > > > > _______________________________________________ > > Devel mailing list > > Devel(a)ovirt.org > > http://lists.ovirt.org/mailman/listinfo/devel > > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/devel

1 0

ovirt 3.5 Test day 1 - Import Storage Domain
by ybronhei 03 Jul '14

03 Jul '14

Hey, I assigned myself to the Import Storage Domain feature [1] in the first ovirt 3.5 test day Currently I checked that when setting OvfUpdateIntervalInMinutes (on vdc_options) to 2 minutes, it saves the ovf files as expected. After setting up new environment and importing the same nfs path that I used as storage domain, ovirt recovered my setup properly I'll play with it more on second test day. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1083307 -- Yaniv Bronhaim.

1 0

ovirt test day report - ovirt - foreman advanced integration feature
by Moti Asayag 03 Jul '14

03 Jul '14

Hi, I assigned myself to the ovirt-foreman advanced integration feature [1] in the first ovirt 3.5 test day. The setup required for testing that environment is a bit complex (foreman, isolated network, extending the foreman with extra plugins). I faced environmental issues most of the day, mainly regarding foreman installation and configuration. I provided specific feedback regarding the installed setup missing steps to Yaniv, the feature owner, which will incorporate them into the feature page. I haven't reached into a point where i can provision a host via foreman into ovirt, but I plan to get back to it in the second ovirt test day (or even before). [1] http://www.ovirt.org/Features/AdvancedForemanIntegration Regards, Moti

1 0

Fwd: [ovirt-users] ovirt 3.5 TestDay Results
by Vinzenz Feenstra 03 Jul '14

03 Jul '14

-------- Original Message -------- Subject: [ovirt-users] ovirt 3.5 TestDay Results Date: Thu, 03 Jul 2014 12:36:07 +0200 From: Vinzenz Feenstra <vfeenstr(a)redhat.com> Organization: Red Hat To: Users(a)ovirt.org List <Users(a)ovirt.org> Hi, I had to test *Bug 1080987* <https://bugzilla.redhat.com/show_bug.cgi?id=1080987> -OVIRT35 - [RFE] Support ethtool_opts functionality within oVirt Installation of the engine went very smooth without a single issue, even adding a previously used host through the host deploy went smoothly and upgraded VDSM without any issues. The feature worked as expected without any problems. All in all I have to say that this time I had the best beta experience since I am working with oVirt. Good Job guys! :-) -- Regards, Vinzenz Feenstra | Senior Software Engineer RedHat Engineering Virtualization R & D Phone: +420 532 294 625 IRC: vfeenstr or evilissimo Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

Test day 1 results
by Martin Betak 02 Jul '14

02 Jul '14

Hi all, During the first test day I tested BZ 1108602 "Implement REST API for oVirt scheduler". Most basic CRUD operations via rest worked well, but there was a problem with accessing the subcollections 'filters', 'weights' and 'balances' of the /api/schedulingpolicies resource by id. For this a bug was filed under https://bugzilla.redhat.com/show_bug.cgi?id=1115071 Martin

2 1

Re: [ovirt-devel] [libvirt] [RFC][scale] new API for querying domains stats
by Francesco Romani 02 Jul '14

02 Jul '14

----- Original Message ----- > From: "Daniel P. Berrange" <berrange(a)redhat.com> > To: "Francesco Romani" <fromani(a)redhat.com> > Cc: libvir-list(a)redhat.com > Sent: Tuesday, July 1, 2014 10:35:21 AM > Subject: Re: [libvirt] [RFC][scale] new API for querying domains stats > [...] > > We [in VDSM] currently use these APIs for our sempling: > > virDomainBlockInfo > > virDomainGetInfo > > virDomainGetCPUStats > > virDomainBlockStats > > virDomainBlockStatsFlags > > virDomainInterfaceStats > > virDomainGetVcpusFlags > > virDomainGetMetadata > > Why do you need to call virDomainGetMetadata so often ? That merely contains > a opaque data blob that can only have come from VDSM itself, so I'm surprised > you need to call that at all frequently. We store some QoS info in the domain metadata. Actually we can elide this API call from the list and fix our coude to make smarter use of it. > > please note that we are much more concerned about thread reduction then > > about performance numbers. We had report of thread number becoming a > > real harm, while performance so far is not yet a concern > > (https://bugzilla.redhat.com/show_bug.cgi?id=1102147#c54) > > > > * bulk APIs for querying domain stats > > (https://bugzilla.redhat.com/show_bug.cgi?id=1113116) > > would be really welcome as well. It is quite independent from the > > previous bullet point > > and would help us greatly with scale. > > If we did the first bullet point, we'd be adding another ~10 APIs for > async variants. If we then did the second bullet point we'd be adding > another ~10 APIs for bulk querying. So while you're right that they > are independent, it would be desirable to address them both at the > same time, so we only need to add 10 new APIs in total, not 20. I'm fine with this approach. > For the async API design, I could see two potential designs > > 1. A custom callback to run per API > > typedef (void)(*virDomainBlockInfoCallback)(virDomainPtr dom, > bool isError, > virDomainBlockInfoPtr info, > void *opaque); > > int virDomainGetBlockInfoAsync(virDomainPtr dom, > const char *disk, > virDomainBlockInfoCallback cb, > void *opaque, > unsigned int flags); > > > 2. A standard callback and a pair of APIs > > typedef void *virDomainAsyncResult; > typedef (void)(*virDomainAsyncCallback)(virDomainPtr dom, > virDomainAsyncResult res); > > void virDomainGetBlockInfoAsync(virDomainPtr dom, > const char *disk, > virDomainBlockInfoCallback cb, > void *opaque, > unsigned int flags); > int virDomainGetBlockInfoFinish(virDomainPtr dom, > virDomainAsyncResult res, > virDomainBlockInfoPtr info); > > This second approach is the way GIO works (see example in this page > https://developer.gnome.org/gio/stable/GAsyncResult.html ). The main > difference between them really is probably the way you get error > reporting from the APIs. In the first example, libvirt would raise > an error before it invoked the callback, with isError set to True. > In the second example, the Finish() func would raise the error and > return -1. I need to check in deeper detail and sync up with other VDSM developers, but I have a feel that the first approach is a bit easier for VDSM to consume. Bests, -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani

2 1

few notes from the scale setup
by Michal Skrivanek 02 Jul '14

02 Jul '14

Hi, I'd like to share couple of observations on the scale system, just a few chaotic notes mostly for the documentation purposes:-) 4 socket, 8 core, HT -> 64 CPU samples has been taken in +- stable conditions 1) running 100VMs; top sample with collapsed process usage (all threads sum up) top - 15:00:48 up 5 days, 6:31, 1 user, load average: 2.25, 1.90, 2.05 Tasks: 1989 total, 4 running, 1983 sleeping, 1 stopped, 1 zombie Cpu(s): 4.7%us, 1.9%sy, 0.0%ni, 93.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 396900992k total, 80656384k used, 316244608k free, 156632k buffers Swap: 41148408k total, 0k used, 41148408k free, 12225520k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9057 vdsm 0 -20 17.5g 482m 9m S 162.7 0.1 446:46.28 vdsm 8726 root 20 0 979m 17m 7996 R 135.6 0.0 68:56.42 libvirtd 36937 qemu 20 0 1615m 296m 6328 S 59.2 0.1 0:19.18 qemu-kvm 38174 root 20 0 16496 2724 880 R 32.1 0.0 0:00.54 top 2458 qemu 20 0 1735m 533m 6328 S 11.1 0.1 2:38.87 qemu-kvm 10203 qemu 20 0 1736m 511m 6328 S 11.1 0.1 2:32.53 qemu-kvm 27774 qemu 20 0 1730m 523m 6328 S 11.1 0.1 2:22.36 qemu-kvm 25208 qemu 20 0 1733m 514m 6328 S 9.9 0.1 2:22.47 qemu-kvm 51594 qemu 20 0 1733m 650m 6328 S 9.9 0.2 3:53.42 qemu-kvm … etc [ this one's not from stable conditions, unfortunately, as can be seen by PID 36937] ---------------------------- 2) running 185 VMs - all threads sum up. VDSM has 411 threads; load is much higher; also note high sys time top - 07:10:28 up 5 days, 22:41, 1 user, load average: 19.10, 14.28, 13.17 Tasks: 2318 total, 9 running, 2308 sleeping, 0 stopped, 1 zombie Cpu(s): 10.8%us, 21.0%sy, 0.0%ni, 67.8%id, 0.1%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 396900992k total, 157267616k used, 239633376k free, 175700k buffers Swap: 41148408k total, 0k used, 41148408k free, 12669856k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9057 vdsm 0 -20 30.1g 856m 10m S 883.4 0.2 9818:59 vdsm 8726 root 20 0 975m 19m 7996 R 142.4 0.0 1370:15 libvirtd 19542 root 20 0 16700 3108 1020 R 15.1 0.0 0:18.11 top 17614 qemu 20 0 1730m 692m 6328 S 6.5 0.2 49:05.53 qemu-kvm 55545 qemu 20 0 1732m 708m 6328 S 6.3 0.2 48:42.01 qemu-kvm 28542 qemu 20 0 1724m 696m 6328 S 6.2 0.2 44:44.50 qemu-kvm 12482 qemu 20 0 1738m 822m 6328 S 6.0 0.2 51:02.71 qemu-kvm … etc break-up per thread: top - 07:05:43 up 5 days, 22:36, 1 user, load average: 12.50, 11.15, 12.00 Tasks: 3357 total, 35 running, 3321 sleeping, 0 stopped, 1 zombie Cpu(s): 11.3%us, 16.0%sy, 0.0%ni, 72.5%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 396900992k total, 157238240k used, 239662752k free, 175700k buffers Swap: 41148408k total, 0k used, 41148408k free, 12669856k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8726 root 20 0 979m 19m 7996 R 81.6 0.0 860:14.21 libvirtd 9103 vdsm 0 -20 30.1g 855m 10m R 63.7 0.2 502:03.90 vdsm 9144 vdsm 0 -20 30.1g 855m 10m R 59.5 0.2 276:23.93 vdsm 11150 vdsm 0 -20 30.1g 855m 10m R 59.0 0.2 399:37.86 vdsm 9100 vdsm 0 -20 30.1g 855m 10m R 42.7 0.2 354:41.75 vdsm 18053 root 20 0 17708 3968 1020 R 17.2 0.0 0:17.46 top 11845 vdsm 0 -20 30.1g 855m 10m S 15.4 0.2 114:48.08 vdsm 8755 root 20 0 979m 19m 7996 S 13.0 0.0 81:25.64 libvirtd 8753 root 20 0 979m 19m 7996 S 12.7 0.0 81:21.16 libvirtd 64396 root 20 0 979m 19m 7996 S 12.4 0.0 80:03.68 libvirtd 8754 root 20 0 979m 19m 7996 R 10.0 0.0 81:26.52 libvirtd 8751 root 20 0 979m 19m 7996 S 9.9 0.0 81:18.83 libvirtd 8752 root 20 0 979m 19m 7996 R 9.7 0.0 81:28.07 libvirtd 52567 vdsm 0 -20 30.1g 855m 10m S 4.9 0.2 9:27.75 vdsm 30617 vdsm 0 -20 30.1g 855m 10m S 3.8 0.2 34:40.75 vdsm 40621 vdsm 0 -20 30.1g 855m 10m S 3.8 0.2 34:12.40 vdsm 8952 vdsm 0 -20 30.1g 855m 10m S 3.8 0.2 24:03.79 vdsm 29818 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 9:21.33 vdsm 31418 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 35:09.03 vdsm 6858 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 34:10.79 vdsm 18513 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 34:44.03 vdsm 46247 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 34:16.65 vdsm 50759 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 34:04.86 vdsm 58612 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 31:58.11 vdsm 25872 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 31:03.81 vdsm 31599 vdsm 0 -20 30.1g 855m 10m S 3.7 0.2 31:10.85 vdsm … etc overall network usage: 0.5-3Mbps varying, roughly corresponding to 15s ---------------------------- 3) special case when vdsm was down, 185 VMs top - 08:34:11 up 6 days, 4 min, 2 users, load average: 5.96, 5.20, 8.71 Tasks: 2314 total, 7 running, 2306 sleeping, 0 stopped, 1 zombie Cpu(s): 8.6%us, 4.9%sy, 0.0%ni, 86.1%id, 0.0%wa, 0.0%hi, 0.4%si, 0.0%st Mem: 396900992k total, 157512324k used, 239388668k free, 180100k buffers Swap: 41148408k total, 0k used, 41148408k free, 12726620k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 47776 root 20 0 17212 3592 1020 R 16.5 0.0 0:16.32 top 35549 qemu 20 0 1806m 809m 6328 S 6.7 0.2 53:46.20 qemu-kvm 43240 qemu 20 0 1735m 694m 6328 S 6.5 0.2 44:58.63 qemu-kvm 51594 qemu 20 0 1733m 822m 6328 S 6.3 0.2 54:19.51 qemu-kvm 24881 qemu 20 0 1726m 704m 6328 S 6.1 0.2 48:50.63 qemu-kvm 58563 qemu 20 0 1728m 699m 6328 S 6.1 0.2 50:14.10 qemu-kvm … etc -------------------------- 4) no disk space on /var/log (opened BZ 1115357): disk space - libvirtd 8726 root 4w REG 253,8 1638998016 34 /var/log/libvirtd.log (deleted) -------------------------- 5) startup of vdsm in 185 VMs environment: on vdsm service startup the "vdsm: Running nwfilter" took ~5 minutes to finish then VM recovery took ~20-30 minutes! overall we need to identify the specific threads and simulate specific issues in a debug-friendly environment to tell a bit more… Thanks, michal

1 0