But this is the reverse of what we need, the problem is how to make
the slave shut down in the first place, you can`t just do it from the
job that used it because it will make the job fail.

Hm. I think some hybrid option is needed. Once the job is finished we should unlabel the slave and then use some garbage collection to kill the used slaves. I believe this can be done using system groovy script. And I think instead of removing labels we should just add a new one, e..g. we add "to_be_removed" and just schedule based on slaves not having that label. Smth like how data are purged from the database with delete flag.

But maybe we can actually use the good old 'shutdown $TIME_DELAY' to
make the slave shut down a few seconds after the job is done...
I can't believe I forgot you can time delay a shut down... I was
initially thinking of 'at' and then I remebred this...

I do not like anything that accounts on any delays as it will rise race condition at some point. The probability is discussable, but first we should try to design the system without it if possible.

Anton Marchukov
Senior Software Engineer - RHEV CI - Red Hat