Configuring Fedora 18 to run VDSM functional tests

Tue Mar 5 14:25:20 UTC 2013

On Tue, Mar 05, 2013 at 02:30:05PM +0800, Zhou Zheng Sheng wrote:
> Hi all,
> 
> Recently I get a oVirt Jenkins power user account. Firstly, on my own
> machine, I have setup Jenkins in Fedora 18 to run VDSM functional tests,
> then now I would like to configure the oVirt Jenkins to run those tests.
> 
> VDSM is for managing the hosts so the tests need some root privilege.
> The system have to be properly configured to provide some dependencies.
> This can not be done solely in Jenkins. I would like to list these
> dependencies as follow and see if they can be setup with the help of the
> server admin.
> 
> Dependency packages for building VDSM:
> 
> yum -y install git autoconf automake gcc python-devel python-nose libvirt libvirt-python python-pthreading m2crypto python-pep8 pyflakes rpm-build python-rtslib
> 
> 
> Some configuration for the environment:
> 
> systemctl stop ksmtuned.service  # mom tests are conflicting with ksmtuned
> systemctl disable ksmtuned.service
> chmod a+r /boot/initramfs*
> mkdir /rhev
> restorecon -v -R /rhev
> yum -y downgrade pyflakes
> 
> 
> visudo:
> 
> jenkins ALL=    NOPASSWD: /usr/bin/make install, /usr/bin/yum, /usr/bin/systemctl, /usr/share/vdsm/tests/run_tests.sh
> jenkins ALL=    NOPASSWD: /usr/bin/rm -rf /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/builder
> jenkins ALL=    NOPASSWD: /usr/bin/rm -rf /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/rpmbuild
> jenkins ALL=    NOPASSWD: /usr/bin/rm -f /home/jenkins/.jenkins/jobs/vdsmFunctionalTest/workspace/nosetests.xml
> 
> On my machine, the Jenkins is run as user "jenkins". Its home is
> /home/jenkins/.jenkins and the job name is vdsmFunctionalTest. We have
> to change the sudo configuration to fit into the server.
> 
> 
> To run glusterfs storage domain related test cases, selinux must be
> turned to permissive mode, because there are some violations in the
> latest glusterfs. If we can not give up selinux for security
> considerations, I will skip those tests when configuring job in Jenkins.
> The following gluster configuration is from Deepak and it works on my
> machine.
> Turn off selinux:
> 
> vim /etc/selinux/config: SELINUX=permissive
> setenforce 0
> 
> Install glusterfs and setup the brick:
> 
> wget http://download.gluster.org/pub/gluster/glusterfs/qa-releases/3.4.0alpha/Fedora/glusterfs-alpha-fedora.repo
> yum -y install glusterfs
> systemctl start glusterd.service
> mkdir /var/lib/vdsm/myglusterbrick
> chmod 777 /var/lib/vdsm/myglusterbrick
> 
> Now start the gluster shell and issue the following commands:
> 
> gluster
> gluster> volume create testvol 192.168.X.X:/var/lib/vdsm/myglusterbrick
> gluster> volume start testvol
> gluster> volume set testvol server.allow-insecure on
> 
> vim /etc/glusterfs/glusterd.vol and enable the below option:
> 
> option rpc-auth-allow-insecure on
> 
> So glusterd.vol should look somethign like this...
> 
> volume management
>     type mgmt/glusterd
>     option working-directory /var/lib/glusterd
>     option transport-type socket,rdma
>     option transport.socket.keepalive-time 10
>     option transport.socket.keepalive-interval 2
>     option rpc-auth-allow-insecure on
> end-volume
> 
> If it's successful, we are able to see the glusterfsd process that owns
> the brick:
> 
> ps -ef| grep testvol
> root      2551     1  0 23:16 ?        00:00:00 /usr/sbin/glusterfsd -s localhost --volfile-id testvol.192.168.X.X.var-lib-vdsm-myglusterbrick -p /var/lib/glusterd/vols/testvol/run/192.168.X.X-var-lib-vdsm-myglusterbrick.pid -S /var/run/d5dd385ecebfdfc05ef54fa0b4d28960.socket --brick-name /var/lib/vdsm/myglusterbrick -l /var/log/glusterfs/bricks/var-lib-vdsm-myglusterbrick.log --xlator-option *-posix.glusterd-uuid=11bd6f47-a3ff-4969-a06b-e91e0f91a0e8 --brick-port 49152 --xlator-option testvol-server.listen-port=49152
> 
> 
> 
> I will run the following shell script as a build step in the Jenkins
> job. Though the script works fine in my Jenkins on Fedora 18, please
> give comments if you spot problems when it runs in oVirt Jenkins.
> 
> set -e
> set -v
> 
> # BUILD #
> # Make things clean.
> sudo -n rm -f "$(pwd)/nosetests.xml"
> 
> BUILDERDIR="$(pwd)/builder"
> RPMTOPDIR="$(pwd)/rpmbuild"
> 
> sudo -n rm -rf "$BUILDERDIR"
> sudo -n rm -rf "$RPMTOPDIR"
> 
> test -f Makefile && make -k distclean || :
> find . -name '*.pyc' | xargs rm -f
> find . -name '*.pyo' | xargs rm -f
> 
> ./autogen.sh --prefix="$BUILDERDIR"
> 
> # If the MAKEFLAGS envvar does not yet include a -j option,
> # add -jN where N depends on the number of processors.
> case $MAKEFLAGS in

You know what I feel about unquoted shell variables..

>   *-j*) ;;
>   *) n=$(getconf _NPROCESSORS_ONLN 2> /dev/null)
>     test "$n" -gt 0 || n=1
>     n=$(expr $n + 1)
>     MAKEFLAGS="$MAKEFLAGS -j$n"
>     export MAKEFLAGS
>     ;;
> esac
> 
> make
> sudo -n make install
> 
> rm -f *.tar.gz
> make dist
> 
> if [ -n "$AUTOBUILD_COUNTER" ]; then
>   EXTRA_RELEASE=".auto$AUTOBUILD_COUNTER"
> else
>   NOW=`date +"%s"`
>   EXTRA_RELEASE=".$USER$NOW"
> fi
> 
> NOSE_EXCLUDE=.* rpmbuild --nodeps \
>    --define "extra_release $EXTRA_RELEASE" \
>    --define "_sourcedir `pwd`" \
>    --define "_topdir $RPMTOPDIR" \
>    -ba --clean vdsm.spec
> 
> 
> # INSTALL #
> joinlines() {
>     local lines="$1"
>     local line=""
>     local sep="$2"
>     local joined=""
>     for line in "$lines"; do
>         joined="${joined}${sep}${line}"
>     done
>     printf "$joined"
> }
> 
> ( cd "$RPMTOPDIR"
> packages=$(find . -name "*.rpm" | grep -v "\.src\.rpm")
> sudo -n yum -y remove "vdsm*"
> sudo -n yum -y localinstall $(joinlines "$packages" " ") )

is "joinlines"" really needed? I think $() does this on its own.

> 
> 
> # START SERVICE #
> sudo -n systemctl start vdsmd.service
> sleep 20
> sudo -n systemctl status vdsmd.service
> 
> 
> # RUN TESTS #
> OLDDIR="$(pwd)"
> ( cd /usr/share/vdsm/tests
> sudo -n ./run_tests.sh --with-xunit --xunit-file "$OLDDIR/nosetests.xml" functional/*.py )
> 
> 
> # STOP SERVICE #
> sudo -n systemctl stop vdsmd.service
> 
> 
> 
> The above setup may not be acceptable for security concerns. So I have
> another setup plan.

yes, this is quite frightening.  `sudo make install` is just begging for
an exploit by a lazy attacker.

> 
> I setup a virtual machine running in QEMU snapshot mode. Then add it to
> Jenkins as a slave. There is a plugin for Jenkins to start and stop
> slave using libvirt. So I configure Jenkins to start the VM slave to
> build, install and run VDSM tests, then shutdown. Jenkins slave gets the
> root privileges of the guest OS and do whatever it needs to. Since the
> VM slave is in snapshot mode, it restores the original state after
> shutdown. I also make a small script to switch snapshot mode on/off when
> I needs to manage the configuration and packages for the guest OS.

This approach would make it easier to clean after a botched test (say, a
vdsm version with a broken /etc/sudoers.d/vdsm, which renders "sudo"
useless).

> 
> Which plan do you prefer? Could someone help me setup this environment?
> (I only get access to Jenkins.)
> 

I vote for the virtualized approach.