Configuring Fedora 18 to run VDSM functional tests

Tue Mar 5 06:30:05 UTC 2013

Hi all,

Recently I get a oVirt Jenkins power user account. Firstly, on my own
machine, I have setup Jenkins in Fedora 18 to run VDSM functional tests,
then now I would like to configure the oVirt Jenkins to run those tests.

VDSM is for managing the hosts so the tests need some root privilege.
The system have to be properly configured to provide some dependencies.
This can not be done solely in Jenkins. I would like to list these
dependencies as follow and see if they can be setup with the help of the
server admin.

Dependency packages for building VDSM:

yum -y install git autoconf automake gcc python-devel python-nose libvirt libvirt-python python-pthreading m2crypto python-pep8 pyflakes rpm-build python-rtslib

Some configuration for the environment:

systemctl stop ksmtuned.service  # mom tests are conflicting with ksmtuned
systemctl disable ksmtuned.service
chmod a+r /boot/initramfs*
mkdir /rhev
restorecon -v -R /rhev
yum -y downgrade pyflakes

visudo:

jenkins ALL=    NOPASSWD: /usr/bin/make install, /usr/bin/yum, /usr/bin/systemctl, /usr/share/vdsm/tests/run_tests.sh
jenkins ALL=    NOPASSWD: /usr/bin/rm -rf /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/builder
jenkins ALL=    NOPASSWD: /usr/bin/rm -rf /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/rpmbuild
jenkins ALL=    NOPASSWD: /usr/bin/rm -f /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/nosetests.xml

On my machine, the Jenkins is run as user "jenkins". Its home is
/home/jenkins/.jenkins and the job name is vdsmFunctionalTest. We have
to change the sudo configuration to fit into the server.

To run glusterfs storage domain related test cases, selinux must be
turned to permissive mode, because there are some violations in the
latest glusterfs. If we can not give up selinux for security
considerations, I will skip those tests when configuring job in Jenkins.
The following gluster configuration is from Deepak and it works on my
machine.
Turn off selinux:

vim /etc/selinux/config: SELINUX=permissive
setenforce 0

Install glusterfs and setup the brick:

wget http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.0alpha/Fedora/glusterfs-alpha-fedora.repo
yum -y install glusterfs
systemctl start glusterd.service
mkdir /var/lib/vdsm/myglusterbrick
chmod 777 /var/lib/vdsm/myglusterbrick

Now start the gluster shell and issue the following commands:

gluster
gluster> volume create testvol 192.168.X.X:/var/lib/vdsm/myglusterbrick
gluster> volume start testvol
gluster> volume set testvol server.allow-insecure on

vim /etc/glusterfs/glusterd.vol and enable the below option:

option rpc-auth-allow-insecure on

So glusterd.vol should look somethign like this...

volume management
    type mgmt/glusterd
    option working-directory /var/lib/glusterd
    option transport-type socket,rdma
    option transport.socket.keepalive-time 10
    option transport.socket.keepalive-interval 2
    option rpc-auth-allow-insecure on
end-volume

If it's successful, we are able to see the glusterfsd process that owns
the brick:

ps -ef| grep testvol
root      2551     1  0 23:16 ?        00:00:00 /usr/sbin/glusterfsd -s localhost --volfile-id testvol.192.168.X.X.var-lib-vdsm-myglusterbrick -p /var/lib/glusterd/vols/testvol/run/192.168.X.X-var-lib-vdsm-myglusterbrick.pid -S /var/run/d5dd385ecebfdfc05ef54fa0b4d28960.socket --brick-name /var/lib/vdsm/myglusterbrick -l /var/log/glusterfs/bricks/var-lib-vdsm-myglusterbrick.log --xlator-option *-posix.glusterd-uuid=11bd6f47-a3ff-4969-a06b-e91e0f91a0e8 --brick-port 49152 --xlator-option testvol-server.listen-port=49152

I will run the following shell script as a build step in the Jenkins
job. Though the script works fine in my Jenkins on Fedora 18, please
give comments if you spot problems when it runs in oVirt Jenkins.

set -e
set -v

# BUILD #
# Make things clean.
sudo -n rm -f "$(pwd)/nosetests.xml"

BUILDERDIR="$(pwd)/builder"
RPMTOPDIR="$(pwd)/rpmbuild"

sudo -n rm -rf "$BUILDERDIR"
sudo -n rm -rf "$RPMTOPDIR"

test -f Makefile && make -k distclean || :
find . -name '*.pyc' | xargs rm -f
find . -name '*.pyo' | xargs rm -f

./autogen.sh --prefix="$BUILDERDIR"

# If the MAKEFLAGS envvar does not yet include a -j option,
# add -jN where N depends on the number of processors.
case $MAKEFLAGS in
  *-j*) ;;
  *) n=$(getconf _NPROCESSORS_ONLN 2> /dev/null)
    test "$n" -gt 0 || n=1
    n=$(expr $n + 1)
    MAKEFLAGS="$MAKEFLAGS -j$n"
    export MAKEFLAGS
    ;;
esac

make
sudo -n make install

rm -f *.tar.gz
make dist

if [ -n "$AUTOBUILD_COUNTER" ]; then
  EXTRA_RELEASE=".auto$AUTOBUILD_COUNTER"
else
  NOW=`date +"%s"`
  EXTRA_RELEASE=".$USER$NOW"
fi

NOSE_EXCLUDE=.* rpmbuild --nodeps \
   --define "extra_release $EXTRA_RELEASE" \
   --define "_sourcedir `pwd`" \
   --define "_topdir $RPMTOPDIR" \
   -ba --clean vdsm.spec

# INSTALL #
joinlines() {
    local lines="$1"
    local line=""
    local sep="$2"
    local joined=""
    for line in "$lines"; do
        joined="${joined}${sep}${line}"
    done
    printf "$joined"
}

( cd "$RPMTOPDIR"
packages=$(find . -name "*.rpm" | grep -v "\.src\.rpm")
sudo -n yum -y remove "vdsm*"
sudo -n yum -y localinstall $(joinlines "$packages" " ") )

# START SERVICE #
sudo -n systemctl start vdsmd.service
sleep 20
sudo -n systemctl status vdsmd.service

# RUN TESTS #
OLDDIR="$(pwd)"
( cd /usr/share/vdsm/tests
sudo -n ./run_tests.sh --with-xunit --xunit-file "$OLDDIR/nosetests.xml" functional/*.py )

# STOP SERVICE #
sudo -n systemctl stop vdsmd.service

The above setup may not be acceptable for security concerns. So I have
another setup plan.

I setup a virtual machine running in QEMU snapshot mode. Then add it to
Jenkins as a slave. There is a plugin for Jenkins to start and stop
slave using libvirt. So I configure Jenkins to start the VM slave to
build, install and run VDSM tests, then shutdown. Jenkins slave gets the
root privileges of the guest OS and do whatever it needs to. Since the
VM slave is in snapshot mode, it restores the original state after
shutdown. I also make a small script to switch snapshot mode on/off when
I needs to manage the configuration and packages for the guest OS.

Which plan do you prefer? Could someone help me setup this environment?
(I only get access to Jenkins.)

-- 
Thanks and best regards!

Zhou Zheng Sheng / 周征晟
E-mail: zhshzhou at linux.vnet.ibm.com
Telephone: 86-10-82454397