[snip]

Hi Derek,
today I played around with Ansible to accomplish, I think, what you currently do in oVirt shell.
It was the occasion to learn, as always, something new: as "blocks" in Ansible dont' support looping, a workaround to get that.
Furthermore I have a single host environment where it can turn usefull too...

The script is supposed to be run on engine, as you already do now.

There is a main script, startup.yml, that is the playbook expected to be run in a way such as:

/usr/bin/ansible-playbook --vault-password-file=pwfile startup.yml

Some explanations below.
On engine you need ansible and ovirt-engine-sdk-python packages, that I think are installed by default.
The script can be executed with a non root user.

In your scripts I see that you don't explicitly set a timeout.
In my ansible tasks where I have to wait (connection to engine, storage domains up and DC up), I set a delay of 10 seconds and a retries of 90 for each of them. So for each task where there is an until condition you have 15 minutes the job can wait before failing. You can adjust the values of delay and/or retries.

To accomplish the loop over VMs to start them up and set their custom timeout before starting next one, I have an include of a task file: vm_block.yml

Inside the directory where you put the two yml files, you will also have a vars directory, where you put your customized var files.
In my example I have encrypted two of them with the ansible-vault command and then put the password inside a file named pwfile (that has to simply contain the password).
Otherwise you can create in clear all the var files and call the playbook with:

/usr/bin/ansible-playbook startup.yml

The var files:

1) ovirt_credentials.yml
2) db_credentials.yml
3) vm_vars.yml

1) contains ovirt credentials. In clear it is something like:

---
ovirt_username: "admin@internal"
ovirt_password: "admin_password"
ovirt_ca: "my_ovirt.pem"
url_name: engine_hostname
...


2) contains engine db credentials. This is needed because I didn't find an oVirt ansible module or python API that gives you the status of storage domains. It would be nice for the ovirt_storage_domain_info ansible module to have it....
So I use connection to engine db and make a select against the storage_domains table, where the status up is expressed by the status column having a value of 3.
BTW: you find the engine db credentials for your engine inside file:
/etc/ovirt-engine/engine.conf.d/10-setup-database.conf

Contents of the var file (you have only to update the password one, randomly generated during the first engine-setup):

---
dbserver: localhost
db: engine
username: engine
password: "TLj63Wvw2yKyDPjlJ9fEku"
...


3) contains variables for your VM ordered names and timeouts, together with the default timeout
I have not encrypted this file, so its contents are in my test case

---
default_timeout: 10
vms:
  - name: f30
    timeout: 30
  - name: slax
...

At the end in the directory where you set up your files/directories you will have something like this:

-rwx------. 1 g.cecchi g.cecchi 1343 May 24 12:19 my_ovirt.pem
-rw-------. 1 g.cecchi g.cecchi 6 May 24 12:18 pwfile
-rwx------. 1 g.cecchi g.cecchi 2312 May 25 00:38 startup.yml
-rwx------. 1 g.cecchi g.cecchi  225 May 24 21:00 vm_block.yml

vars/:
total 12
-rw-------. 1 g.cecchi g.cecchi 679 May 25 00:14 db_credentials.yml
-rwx------. 1 g.cecchi g.cecchi 808 May 24 21:02 ovirt_credentials.yml
-rw-rw-r--. 1 g.cecchi g.cecchi  78 May 24 20:37 vm_vars.yml

You can find here:

startup.yml
https://drive.google.com/file/d/19F16TAAfneYbMnAuUE-BlTI-VNf8l4Yx/view?usp=sharing

vm_block.yml
https://drive.google.com/file/d/1tHCcP5pBlkjeixIF8GGdhGRNaNBGhwbR/view?usp=sharing

NOTE: in my case I have two storage domains unattached, the ovirt image one and an export one.  So inside startup.yml you find:

expected_sd_up: "{{ ( result_sd.ovirt_storage_domains | length ) - 2 }}"

Change "- 2" with "- 1" for your environment.

Executing the job in the rc.local you would get something like this where you redirect output:

[g.cecchi@ovirt ~]$ ansible-playbook --vault-password-file=pwfile startup.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit
localhost does not match 'all'

PLAY [Play to start oVirt VMs] *********************************************************************

TASK [Gathering Facts] *****************************************************************************
ok: [localhost]

TASK [Obtain SSO token using username/password credentials] ****************************************
FAILED - RETRYING: Obtain SSO token using username/password credentials (90 retries left).
....
ok: [localhost]

TASK [Sleep for 60 seconds] ************************************************************************
Pausing for 60 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [Get storage domains] *************************************************************************
ok: [localhost]

TASK [Set storage domains to be active] ************************************************************
ok: [localhost]

TASK [Print expected storage domains to be up] *****************************************************
ok: [localhost] => {
    "msg": "Expected storage domains to be up: 3"
}

TASK [Query storage_domains table] **************************************************************
[WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or
{% %}. Found: sd_up.query_result[0].up_sd == {{ expected_sd_up }}
FAILED - RETRYING: Query storage_domains table (90 retries left).
...
ok: [localhost]

TASK [Print storage domains that are up] ***********************************************************
ok: [localhost] => {
    "msg": "Number of storage domains up: 3"
}

TASK [Verify DC is up] *****************************************************************************
FAILED - RETRYING: Verify DC is up (90 retries left).
...
ok: [localhost]

TASK [External block to start VMs] *****************************************************************
included: /home/g.cecchi/vm_block.yml for localhost
included: /home/g.cecchi/vm_block.yml for localhost

TASK [Start VM f30] ************************************************************************************
changed: [localhost]

TASK [Pause before starting next VM] ***************************************************************
ok: [localhost]

TASK [Start VM slax] ************************************************************************************
changed: [localhost]

TASK [Pause before starting next VM] ***************************************************************
ok: [localhost]

TASK [Revoke SSO token] ****************************************************************************
ok: [localhost]

PLAY RECAP *****************************************************************************************
localhost                  : ok=15   changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0  

Cheers,