December 2021 - Devel - oVirt List Archives

Helper script
by Ales Musil 09 Dec '21

09 Dec '21

Hi, with recent switch to GH I made myself a little helper to sync fork and rebase your branch on top of that. Hopefully someone will find it useful. The usage is "./script.sh $NAME_OF_MASTER" (usually master for gerrit projects, newer GH have main). #!/bin/bash -e ORIGIN=$(git branch --show-current) # Go to master/main/base (gco) git checkout $1 # Fetch all remotes (gfa) git fetch --all --prune # Merge upstream remote into master/main/base (gm) git merge upstream/$1 # Push master/main/base into origin (gp) git push origin $(git branch --show-current) # Switch back to previous branch (gco) git checkout $ORIGIN # Rebase the branch (gl) git pull --rebase Best regards, Ales -- Ales Musil Software Engineer - RHV Network Red Hat EMEA <https://www.redhat.com> amusil(a)redhat.com IM: amusil <https://red.ht/sig>

1 0

Help installing oVirt on single machine, without cockpit
by Cameronsplaze222＠gmail.com 08 Dec '21

08 Dec '21

I'm trying to have a single machine, run both the engine (so I get the API), and hypervisor itself to run VM's on. My goal is to write some custom scripts that use the API, so I'm hoping cockpit isn't needed. I've tried two different routes, but both got blocked eventually. Maybe someone has an idea on how to continue? # Using CentOS 9 stream: Tried following the directions [here](https://www.ovirt.org/documentation/installing_ovirt_as_a_standalone_…, but in 3.2 where you add the modules, they couldn't be found. I tried skipping it, but in 3.3.2 where you run `dnf install ovirt-engine`, it says ovirt-engine doesn't exist. The closest thing I see is `ovirt-hosted-engine-setup-2.5.4-1.el9.noarch`, is that what I want maybe? # Using oVirt node iso, and installing an engine to that: I added the networking to the install, but it didn't save to the machine itself, and I wasn't sure on how to add it. I tried hooking a direct cable, but still unable to ping 8.8.8.8, or anything. My goal is for a minimal installation, only running the API and hypervisor, so I'm definitely open to other routes too. Is this something that's possible to do?

1 0

Switching to GH actions build: some not-yet-clear parts and some tips
by Martin Perina 06 Dec '21

06 Dec '21

Hi, I've tried to move ovirt-engine-extensions-api <https://github.com/ovirt/ovirt-engine-extensions-api> to github and setup GH action to build to have some sort of a basic template for all our Java based projects: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wo… During the work I discovered some parts which should be further discussed: 1. Most of projects requires packages from ovirt-master-release RPM repositories to build itself - In past we have been using https://resources.ovirt.org/pub/yum-repo/ovirt-release-master.rpm to fetch the latest version of this RPM - Do we plan to keep it and have this URL as the main source for oVirt master repositories? - ovirt-engine-extensions-api projects requires only Virt SIG repo so I've applied below hack to provide it: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automatio… - I don't like the solution, it would be much nicer to have a constant URL for the latest ovirt-release-master RPM 2. Providing RPM repositories for different OS inside single build artifact - I wanted to build the project for both CentOS Stream 8 and 9, but I figured out that upload-artifact action merges results for all different OS builds into a single directory, so created repositories are mixed together. - I've use GH strategy option and used centos-stream-8 for CS8 and centos-stream-9 for CS9 builds: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wo… - Do we already have that documented so an OST plugin to use those artifacts per OS can be created? If not, where to put it? 3. Using maven cache for Java project builds - When build maven projects in jenkins.ovirt.org we were using artifactory.ovirt.org to cache maven artifacts (and also save download bandwidth to PHX datacenter) - It doesn't make to use artifactory.ovirt.org inside GH actions, so I've tried to define a GH action specific cache: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wo… 4. Caching required dependencies - Above maven dependencies helps for maven invocations before RPM build but it doesn't help to fetch all required RPM dependencies for pro RPM maven based build - Right now it's almost 500 GB of packages to download and install on top of the default container and it for ovirt-engine-extensions-api it takes 50 % of the whole build time just to download and install those packages (I know it's just 1 minute, but anyway :-) - I've noticed that VDSM is using its own customized container <https://github.com/oVirt/vdsm/blob/master/.github/workflows/ci.yml#L8> - Wouldn't it be worthwhile to also create customized containers for Java based projects? 5. Differences between CS8 and CS9 - There are again some disturbing changes between CS8 and CS9: - PowerTools repo from CS8 is now called CRB in CS9 - pki-deps javapackages-tools modules from CS8 don't exist in CS9 - Core DNF plugins are not installed by default in CS9 container - To bypass those differences I've create 2 different shell scripts and plug them into a single workflow job: https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wo… https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automatio… https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automatio… 6. git commands cannot be easily used inside GH action - GH doesn't checkout the project completely during checkout action, so .git directory is not present - In the past I've been using 'git archive' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=…> to create .tar.gz source archive, but on GH it needed to be switched to a classic tar <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.automatio…> - In the past I've been using 'git rev-list' <https://gerrit.ovirt.org/gitweb?p=ovirt-engine-extensions-api.git;a=blob;f=…> to fetch short git hash for snapshot builds, but on GH it needed to be switched to below hack <https://github.com/oVirt/ovirt-engine-extensions-api/blob/master/.github/wo…> Regards, Martin -- Martin Perina Manager, Software Engineering Red Hat Czech s.r.o.

3 3

Proposal: moving to building oVirt RPMs via containers
by Yaniv Kaul 06 Dec '21

06 Dec '21

With the move to Github, I think we have an opportunity to move to building oVirt packages via containers. You may want to look at https://github.com/gluster/Gluster-Builds - building Gluster RPMs by running a container (the 'build container'), which has the required dependencies and is updated once a month, or when there's a new dependency. It fits nicely with Github actions or any other system that uses containers, from your laptop to K8S. What I personally really like about it is the fact that the Dockerfile becomes the self-describing build process. Thoughts? Y.

1 0

Updates on oVirt Node and oVirt appliance building outside Jenkins
by Sandro Bonazzola 04 Dec '21

04 Dec '21

Hi, just a quick update on current issues in trying to build oVirt Node and the engine appliance outside Jenkins. 1) Using GitHub Actions: an attempt to build it is in progress here: https://github.com/sandrobonazzola/ovirt-appliance/pull/1 it's currently failing due to lorax not being able to perform the build. It kind of make sense as we are trying to do a virt-install within a container without the needed virtualization hardware exposed. I'm currently investigating how to make use of software virtualization in order to drop the requirement on missing hardware / nested virtualization. Also investigating on how to use self hosted runners for providing a build system with usable virtualization hardware. 2) Using COPR: we have basically the same issue: lorax fails not having access to the virtualization hardware 3) Using CentOS Community Build System This is a fully fledged Koji instance and it allows to build using the image-build variant. It has a completely different configuration system and it is more similar to what we are doing within the downstream build of oVirt Node. An attempt of providing the configuration started here: https://gerrit.ovirt.org/c/ovirt-appliance/+/117801 The issue there is that all the packages needed to be included within Node and Appliance must be built within CentOS Community Build System build root. The system has no external access to the internet so everything we need it needs to come from CentOS infra. I haven't started digging into oVirt Node but the build flow is very similar to the appliance one, so once one is solved, the other should be simple. -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

2 2

oVirt Community - open discussion around repositories and workflows
by Sandro Bonazzola 03 Dec '21

03 Dec '21

Dear community members, We would like to take some next steps in improving usability and general decision making for our community and are interested in your thoughts and suggestions. The first step is a decision regarding our version control and workflows associated with it. Contributions to new features and general improvements for oVirt outside of Red Hat have been rare and we would like to hear from you if we could simplify this process for you. One suggestion we have is that we could simplify the contribution process and improve the contributor experience by moving our repositories to GitHub or GitLab. Currently, our Gerrit (gerrit.ovirt.org) instance is mirrored to GitHub and we already have several repositories that are exclusively developed there, including their CI running on GitHub Actions, for example: * https://github.com/oVirt/go-ovirt-client * https://github.com/oVirt/512-byte-vm There are also various related projects that run on GitLab, such as: * https://gitlab.com/qemu-project * https://gitlab.com/libvirt One recent project that moved from Gerrit to GitLab with a very similar discussion is mediawiki: https://www.mediawiki.org/wiki/GitLab_consultation/Discussion_summary Once we hear more of your thoughts around this and if the decision falls on moving to either GitHub or GitLab, our plan would be to start changing existing workflows and CI project by project as it makes sense. Both platforms offer similar features. Two key points that stand out for us are that: * GitLab is open source while GitHub isn't * GitHub is more visible which allows contributors to amplify their contributions and raise their profile by contributing to various big projects, including oVirt We are looking forward to hearing your thoughts, -- Sandro Bonazzola MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV Red Hat EMEA <https://www.redhat.com/> sbonazzo(a)redhat.com <https://www.redhat.com/> *Red Hat respects your work life balance. Therefore there is no need to answer this email out of your office hours.*

4 10

Integrating OST with artifacts built in github
by Nir Soffer 02 Dec '21

02 Dec '21

Looking this very helpful document https://ovirt.org/develop/developer-guide/migrating_to_github.html The suggested solution is to create artifacts.zip with all the rpms for a project. But to use the rpms in OST, we need to create a yum repository before uploading the artifacts, so we can pass a URL of a zip file with a yum repository. Here is what we have now in ovirt-imageio: 1. We build for multiple distros: https://github.com/nirs/ovirt-imageio/blob/790e6b79e756de24ef5134aa583bea46… 2. Every build creates a repo in exported-artifacts https://github.com/nirs/ovirt-imageio/blob/790e6b79e756de24ef5134aa583bea46… 3. Every build uploads the exported artifacts to rpm-{distro}.zip https://github.com/nirs/ovirt-imageio/blob/790e6b79e756de24ef5134aa583bea46… An example build: https://github.com/nirs/ovirt-imageio/actions/runs/1531658722 To start OST manually, developer can copy a link the right zip file (e.g for centos stream 8): https://github.com/nirs/ovirt-imageio/suites/4535392764/artifacts/121520882 And pass the link to OST build with parameters job. In this solution, OST side gets a repo that can be included in the build without any additional code or logic - just unzip and use the repo from the directory. I think this is the minimal solution to allow running OST with built artifacts from github. For triggering jobs automatically, we will need a way to find the right artifacts for a build, or use some convention for naming the artifacts in all projects. I started with the simple convention of jobname-containername since it is easy to integrate with the infrastructure we already have in the project. Nir

2 1

Another CI failure
by Milan Zamazal 02 Dec '21

02 Dec '21

Hi, as demonstrated in https://jenkins.ovirt.org/job/vdsm_standard-check-patch/31133/, OST builds can at least start now but still fail, apparently due to the following: + sudo -n usermod -a -G jenkins qemu usermod: user 'qemu' does not exist + log ERROR 'Failed to add user qemu to group jenkins' + local level=ERROR + shift + local 'message=Failed to add user qemu to group jenkins' + local prefix + [[ 4 -gt 1 ]] + prefix='global_setup[lago_setup]' + echo 'global_setup[lago_setup] ERROR: Failed to add user qemu to group jenkins' global_setup[lago_setup] ERROR: Failed to add user qemu to group jenkins + return 1 + failed=true What can be done about it? Thanks, Milan

3 4

IPV6 on github actions container
by Nir Soffer 02 Dec '21

02 Dec '21

I'm trying to add a github action workflow for ovirt-imageio. The yml file is simple: https://github.com/nirs/ovirt-imageio/commit/21da91274b7b4722167c2415d1c1b5… And all the jobs run, but tons of tests are failing: https://github.com/nirs/ovirt-imageio/actions/runs/1527728851 The typical failure is: 2021-12-01T20:50:07.6424921Z address = ('::', 43001), timeout = <object object at 0x7f2cb7d16270> 2021-12-01T20:50:07.6425749Z source_address = None 2021-12-01T20:50:07.6425985Z 2021-12-01T20:50:07.6426421Z def create_connection(address, timeout=_GLOBAL_DEFAULT_TIMEOUT, 2021-12-01T20:50:07.6426954Z source_address=None): ... 2021-12-01T20:50:07.6435579Z host, port = address 2021-12-01T20:50:07.6435942Z err = None 2021-12-01T20:50:07.6436563Z for res in getaddrinfo(host, port, 0, SOCK_STREAM): 2021-12-01T20:50:07.6437402Z af, socktype, proto, canonname, sa = res 2021-12-01T20:50:07.6437834Z sock = None 2021-12-01T20:50:07.6438146Z try: 2021-12-01T20:50:07.6438522Z sock = socket(af, socktype, proto) 2021-12-01T20:50:07.6439032Z if timeout is not _GLOBAL_DEFAULT_TIMEOUT: 2021-12-01T20:50:07.6439538Z sock.settimeout(timeout) 2021-12-01T20:50:07.6440109Z if source_address: 2021-12-01T20:50:07.6440522Z sock.bind(source_address) 2021-12-01T20:50:07.6440916Z > sock.connect(sa) 2021-12-01T20:50:07.6441573Z E OSError: [Errno 99] Cannot assign requested address I guess that ipv6 is disabled, and google confirms that github actions and ipv6 is an issue. In travis we enabled ipv6 in docker using this script: https://github.com/oVirt/ovirt-imageio/blob/master/travis/configure-docker.… Run before starting the job: https://github.com/oVirt/ovirt-imageio/blob/61f023bf1e45da4e98593278451030f… I did not try to run this script yet, since I'm not sure we have a way to run stuff *before* the container runs in github actions. I hope someone has some insight on a way to solve this. Nir

2 3

COPR's ovirt-master-snapshot replacing resources.ovirt.org's "tested" repo
by Michal Skrivanek 02 Dec '21

02 Dec '21

Hi all, COPR[1] is effectively replacing the "tested" repo - the one that gets all built artifacts after a patch is merged (and is actually not tested:) Sandro started to move projects to COPR for merged patches some time ago and we believe it's complete except appliance and node. So we can all start switching to it as it is currently more up to date anyway Please start modifying your CI setup, other automation, your own updates, wherever really, and just switch from [2] to [3] (adjust the platform/arch accordingly as well, so e.g. [4]) Thanks, michal [1] https://copr.fedorainfracloud.org/coprs/ovirt/ovirt-master-snapshot/ [2] https://resources.ovirt.org/repos/ovirt/tested/master/rpm/ [3] https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snaps… [4] https://download.copr.fedorainfracloud.org/results/ovirt/ovirt-master-snaps…

1 1