Hi, as a heads up: I'd like to contribute some basic support for the s390 architecture to oVirt. For that purpose I'll post some patches to gerrit, mostly for the following repositories: VDSM, ovirt-engine-api-model and ovirt-engine. There might be some collateral patches that need to go into ovirt-host[-deploy] since not all of the pre-requisite RPMs are available on s390, but this probably needs a separate discussion. I've started to hang out on the #ovirt OFTC channel, feel free to ping me if you have questions related to s390. Thanks!

Hi, as a heads up: I'd like to contribute some basic support for the s390 architecture to oVirt. For that purpose I'll post some patches to gerrit, mostly for the following repositories: VDSM, ovirt-engine-api-model and ovirt-engine. There might be some collateral patches that need to go into ovirt-host[-deploy] since not all of the pre-requisite RPMs are available on s390, but this probably needs a separate discussion. I've started to hang out on the #ovirt OFTC channel, feel free to ping me if you have questions related to s390. Thanks!

Koji is very opinionated about how RPMs and specfiles should look, AFAIK

More specifically, Koji usually assumes the starting point for the build process would be a specfile

Does fedora have an s390x server associated to it?

Does fedora have an s390x server associated to it?

On 16 November 2017 at 19:38, Martin Sivak <msivak(a)redhat.com&gt; wrote: >> Koji is very opinionated about how RPMs and specfiles should look, >> AFAIK > > Actually, koji does not care much. Fedora packaging rules are enforced > by package reviewers. > >> More specifically, Koji usually assumes the starting point for the >> build process would be a specfile > > This is correct though. > >> Does fedora have an s390x server associated to it? > > They used to have s390 emulators running for that purpose iirc. I wonder if someone could provide more information about this. Is this done via qemu? Or built-in to mock perhaps? It this exists, I can pave the way to enabling s390x build support on oVirt infra.

On 11/19/2017 09:53 AM, Barak Korren wrote: > On 18 November 2017 at 13:14, Barak Korren <bkorren(a)redhat.com&gt; > wrote: >> On 16 November 2017 at 19:38, Martin Sivak <msivak(a)redhat.com&gt; >> wrote: >>>> Koji is very opinionated about how RPMs and specfiles should >>>> look, AFAIK >>> >>> Actually, koji does not care much. Fedora packaging rules are >>> enforced by package reviewers. >>> >>>> More specifically, Koji usually assumes the starting point for >>>> the build process would be a specfile >>> >>> This is correct though. >>> >>>> Does fedora have an s390x server associated to it? >>> >>> They used to have s390 emulators running for that purpose iirc. >> >> I wonder if someone could provide more information about this. Is >> this done via qemu? Or built-in to mock perhaps? It this exists, I >> can pave the way to enabling s390x build support on oVirt infra. > > There seems to be a 'qemu-system-s390x.x86_64' package avalable for > CentOS7 in EPEL, so we might be able to use that to get some > emulated s390x Jenkins slaves up. As none of the oVirt infra team > members is currently experienced with using this, we would > appreciate so help there with: qemu s390x emulation does not work with code compiled for z12. Would a real virtual machine be what you need? The Fedora team DOES have access to a z13. Not sure how much resources are available, but can you contact Dan Horak (on cc) if there is enough spare capacity.

> * Figuring out the right libvirt commands to get an s390x VM running > on an x86_64 host. > * Figuring out the right way to get an OS image for that VM > (Probably using virt-install to run the Fedora installer) > * Making sure some JVM is available in those VMs > > Once we have that, the rest is quite straight forward. > > I've created a Jira ticket to map out, discuss and track the > required steps: https://ovirt-jira.atlassian.net/browse/OVIRT-1772 > > WRT the OS we would use, I can see there are s390x Fedora builds > available but no CentOS builds, is there ongoing work to remedy > this, we'd rather not have to upgrade our Jenkins slaves every 6 > months... > > As a side note - at this time it seems we would have to manually > spin up the s390x VMs with libvirt. It would be awesome if oVirt > could get support for foreign architecture emulation at some point. >

On 21.11.2017 11:26, Dan Horák wrote: [...] >> qemu s390x emulation does not work with code compiled for z12. >> Would a real virtual machine be what you need? >> The Fedora team DOES have access to a z13. Not sure how much >> resources are available, but can you contact Dan Horak (on cc) if >> there is enough spare capacity. > > Christian is right, we have a publicly accessible guest running > Fedora on the Marist College z13 mainframe. It's currently used by > ~5 projects (for example glibc and qemu) as their build and CI > host, so adding another project depends how intensive ovirt's usage > would be. As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters.

On 21.11.2017 11:26, Dan Horák wrote: [...] >> qemu s390x emulation does not work with code compiled for z12. >> Would a real virtual machine be what you need? >> The Fedora team DOES have access to a z13. Not sure how much >> resources are available, but can you contact Dan Horak (on cc) if >> there is enough spare capacity. > > Christian is right, we have a publicly accessible guest running Fedora > on the Marist College z13 mainframe. It's currently used by ~5 projects > (for example glibc and qemu) as their build and CI host, so adding > another project depends how intensive ovirt's usage would be. As a first step one could only build the packages needed for the KVM host. At this point in time that would be vdsm and ovirt-host, both are building rather quickly. It should be possible to ensure that only these are built on a s390 system using appropriate node filters. [...]

ok, I'm pretty sure we can make it work :-) Please send me your public SSH key and preferred username, then I'll set up you an account for you and we can work on the remaining details.

On 24 November 2017 at 14:36, Dan Horák <dan(a)danny.cz&gt; wrote: > On Fri, 24 Nov 2017 13:16:53 +0200 > Barak Korren <bkorren(a)redhat.com&gt; wrote: > >> On 24 November 2017 at 11:05, Viktor Mihajlovski >> <mihajlov(a)linux.vnet.ibm.com&gt; wrote: >> > On 21.11.2017 11:26, Dan Horák wrote: >> > [...] >> >>> qemu s390x emulation does not work with code compiled for z12. >> >>> Would a real virtual machine be what you need? >> >>> The Fedora team DOES have access to a z13. Not sure how much >> >>> resources are available, but can you contact Dan Horak (on cc) if >> >>> there is enough spare capacity. >> >> >> >> Christian is right, we have a publicly accessible guest running >> >> Fedora on the Marist College z13 mainframe. It's currently used by >> >> ~5 projects (for example glibc and qemu) as their build and CI >> >> host, so adding another project depends how intensive ovirt's >> >> usage would be. >> > As a first step one could only build the packages needed for the KVM >> > host. At this point in time that would be vdsm and ovirt-host, both >> > are building rather quickly. >> > It should be possible to ensure that only these are built on a s390 >> > system using appropriate node filters. >> > [...] >> >> We can get more accurate data by looking at the ppc64c build history >> (We support ppc64le only for hypervisor usage, similar to what is >> intended for s390). >> Here is the history for vdsm: >> http://jenkins.ovirt.org/job/vdsm_master_build-artifacts- el7-ppc64le/buildTimeTrend >> (~20 builds a day taking 1-2 minutes each) >> And here is the one for ovirt-host: >> http://jenkins.ovirt.org/job/ovirt-host_master_check-patch- el7-ppc64le/buildTimeTrend >> (only 1 build in history, taking 3-4 minutes) >> >> Looking at what else we have building on ppc64le: >> http://jenkins.ovirt.org/search/?q=master_build-artifacts-el7-ppc64le >> I can also see ioprocess with is a vdsm dependency, and the SDK which >> is probably not really needed. >> So for ioprocess: >> http://jenkins.ovirt.org/job/ioprocess_master_build- artifacts-el7-ppc64le/buildTimeTrend >> I'd say its very rarely built. >> >> So we end up with ~20 1-2 minute builds a day (Timed but the amount of >> Fedora versions we want to support, but what will probably be just >> one), with the rest being a statistical error... >> >> I wonder about sharing a VM with other project though. We do use mock >> for running the build script so the build itself should be fairly >> isolated, but we have some of our own wrapper scripts around mock that >> do things trying to keep build dependencies in the chroot cache over >> time. We're also incompatible with mock's new systemd-nspawn backend, >> so we force it to work with the older chroot-based backend. If other >> projects are using mock as well, I wonder if we may end up with race >> conditions arising from shared use of /var/lib/mock. > > it should work fine > >> Bottom line - we may end up being a little noisy neighbours if we >> share a VM, but we can try that and see what happens, how to we move >> foreward with trying that? > > ok, I'm pretty sure we can make it work :-) Please send me your > public SSH key and preferred username, then I'll set up you an account > for you and we can work on the remaining details. > An update for everyone woh may have been watching this thread - we made it work. With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs. For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired. The code changes I've made had already been tested and are now pending code review: https://gerrit.ovirt.org/c/85219 https://gerrit.ovirt.org/c/85221 Once those patches are merged it will become possible to add s390x jobs to any oVirt project by adding '390x' to the list of architectures targeted by the project in the JJB YAML, as well as setting the 'node_filter' to be 's390x' for that architecture.

-- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

On 10 December 2017 at 13:38, Greg Sheremeta <gshereme(a)redhat.com&gt; wrote: > > > Great job all! So I guess the question is, for those of us that maintain > oVirt projects, are there any that we know need s390x support added for? > > Well, Viktor Mihajlovski had been contributing patches to VDSM et el. to enable making an s390x hypervisor node. So those are the obvious candidates. Greg, I guess this is less relevant to your current line of work... -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

An update for everyone woh may have been watching this thread - we made it work.

With Dan's kind help we've attached an s390x VM to oVirt's CI infrastructure. I've then gone ahead and made some code changes to make our CI code play nice on it (So far we just assumed we own the execution slaves and can do what we want on them...). Following that I've gone ahead and added the basic configuration needed to make the oVirt CI system support s390x jobs. For now we only support using Fedora 26 on s390x. Please let me know if other distributions are desired.

The code changes I've made had already been tested and are now pending code review: https://gerrit.ovirt.org/c/85219 https://gerrit.ovirt.org/c/85221 Once those patches are merged it will become possible to add s390x jobs to any oVirt project by adding '390x' to the list of architectures targeted by the project in the JJB YAML, as well as setting the 'node_filter' to be 's390x' for that architecture.

On Thu, 14 Dec 2017 11:24:27 +0100

> Lately, I've been using Fedora 27 for the s390x porting, mostly > because the newer package levels of the key virtualization > components. If possible, Fedora 27 would be nice. The Marist College agrees to provide us more guests, so we shouldn't see capacity issues in the near (and mid) future. F-26 is good enough for your F-27 builds, because you use mock, right?

On 16 November 2017 at 19:38, Martin Sivak <msivak(a)redhat.com&gt; wrote: >> Koji is very opinionated about how RPMs and specfiles should look, >> AFAIK > > Actually, koji does not care much. Fedora packaging rules are enforced > by package reviewers. > >> More specifically, Koji usually assumes the starting point for the >> build process would be a specfile > > This is correct though. > >> Does fedora have an s390x server associated to it? > > They used to have s390 emulators running for that purpose iirc. I wonder if someone could provide more information about this. Is this done via qemu? Or built-in to mock perhaps? It this exists, I can pave the way to enabling s390x build support on oVirt infra. -- Barak Korren RHV DevOps team , RHCE, RHCi Red Hat EMEA redhat.com | TRIED. TESTED. TRUSTED. | redhat.com/trusted

On 16 November 2017 at 18:52, Viktor Mihajlovski <mihajlov(a)linux.vnet.ibm.com&gt; wrote: > > Short update, with yesterday's API model 4.2.25 release, there's basic > support for s390 available in ovirt-engine. At this point in time, there > are no ovirt yum repositories for the s390x architecture - not sure what > the process would be to add s390x repositories and how to build the > binary RPMs at least for the host packages (i.e. vdsm-*). Maybe it would > possible to use the s390-koji infrastructure used to build Fedora for s390x? Koji is very opinionated about how RPMs and specfiles should look, AFAIK A pretty massive amount of work would be needed to make everything that is needed for a node to build on it, and then we would end up with a process that is quite different with how we currently do builds for other platforms. (I might be wrong about this, since some oVirt packages get also built as part of the CentOS virt SIG, and that is done using Koji as well) More specifically, Koji usually assumes the starting point for the build process would be a specfile, while in oVirt we typically generated the specfile and then the RPM as part of a bigger build process. Does fedora have an s390x server associated to it?

We do use the same basic environment setup tool - mock - as the basis of our build infrastructure, so if Fedora is actually emulating s390x is some way while using mock, we might be able to do the same thing. Just for general knowledge, the process for building oVirt repos, is to have *-build-artifacts-* jobs for each project that build RPM's after patches get merged, and then have the change-queue to collect the built packages, rung them through ovirt-system-tests (a.k.a. OST) and finally deposit them into the 'tested' rpoe, from which they are copied nightly to the '*-snapshot' repos. OST only tests for CentOS 7/x86_64 ATM, but we bring along packages for other distros and architectures via the same process while assuming that if a package for a given commit works for CentOS 7/x86_64 it would probably work for other platforms as well...

We don't currently have s390x EPEL