If you have been using or monitoring any OST suits recently, you may have noticed we've been suffering from long delays in allocating CI hardware resources for running OST suits. I'd like to briefly discuss the reasons behind this, what are planning to do to resolve this and the implication of those actions for big suit owners.
As you might know, we have moved a while ago from running OST suits each on its own dedicated server to running them inside containers managed by OpenShift. That had allowed us to run multiple OST suits on the same bare-metal host which in turn increased our overall capacity by 50% while still allowing us to free up hardware for accommodating the kubevirt project on our CI hardware.
Our infrastructure is currently built in a way where we use the exact same POD specification (and therefore resource settings) for all suits. Making it more flexible at this point would require significant code changes we are not likely to make. What this means is that we need to make sure our PODs have enough resources to run the most demanding suits. It also means we waste some resources when running less demanding ones.
Given the set of OST suits we have ATM, we sized our PODs to allocate 32Gibs of RAM. Given the servers we have, this means we can run 15 suits at a time in parallel. This was sufficient for a while, but given increasing demand, and the expectation for it to increase further once we introduce the patch gating features we've been working on, we must find a way to significantly increase our suit running capacity.
We have measured the amount of RAM required by each suit and came to the conclusion that for the vast majority of suits, we could settle for PODs that allocate only 14Gibs of RAM. If we make that change, we would be able to run a total of 40 suits at a time, almost tripling our current capacity.
The downside of making this change is that our STDCI V2 infrastructure will no longer be able to run suits that require more then 14Gib of RAM. This effectively means it would no longer be possible to run these suits from OST's check-patch job or from the OST manual job.
The list of relevant suits that would be affected follows, the suit owners, as documented in the CI configuration, have be added as "to" recipients to the message:
- hc-basic-suite-4.3
- hc-basic-suite-master
- metrics-suite-4.3
Since we're aware people would still like to be able to work with the bigger suits, we will leverage the nightly suit invocation jobs to enable then to be run in the CI infra. We will support the following use cases:
- Periodically running the suit on the latest oVirt packages - this will be done by the nightly job like it is done today
- Running the suit to test changes to the suit`s code - while currently this is done automatically by check-patch, this would have to be done manually in the future by manually triggering the nightly job and setting the REFSPEC parameter to point to the examined patch
- Triggering the suit manually - This would be done by triggering the suit-specific nightly job (as opposed to the general OST manual job)
The patches listed below implement the changes outlined above:
- 102757 nightly-system-tests: big suits -> big containers
- 102771: stdci: Drop `big` suits from check-patch
We know that making the changes we presented will make things a little less convenient for users and maintainers of the big suits, but we believe the benefits of having vastly increased execution capacity for all other suits outweigh those shortcomings.
We would like to hear all relevant comment and questions from the quite owners and other interested parties, especially is you think we should not carry out the changes we propose.
Please take the time to respond on this thread, or on the linked patches.
Thanks,
--