[ovirt-devel] Strange concurrency error on VM creation

[Juan Hernández]

On 03/07/2017 05:42 PM, Marc Young wrote:
> I've been fighting this for roughly two days and I'm starting to think
> that possibly it's not my code but an interaction with the server.
> 
> I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to spin up
> vagrant machines and run tests against them. I'm using Jenkins to run
> kitchen in containers in parallel.
> 
> Basically Jenkins runs a docker container with ruby + vagrant 1.9.2 and
> runs kitchen test all at the same time as another container with ruby +
> vagrant 1.9.1.
> 
> If I run these in parallel, on some occasions the server seems to
> respond with the wrong creation information. If you look at the logs
> here: http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console
> <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console>
> 
> 
> the container for vagrant 1.9.1 created a VM `vagrant-dynamic-1.9.1:
> 
>     [vagrant-1.9.1]        Bringing machine 'default' up with 'ovirt4' provider...
> 
>     [vagrant-1.9.1]        ==> default: Creating VM with the following settings...
> 
>     [vagrant-1.9.1]        ==> default:  -- Name:          dynamic-1.9.1
> 
> 
> And the container for vagrant 1.9.2 (nearly the same time) created a VM
> `vagrant-dynamic-1.9.2`:
> 
>     [vagrant-1.9.2]        ==> default: Creating VM with the following settings...
> 
>     [vagrant-1.9.2]        ==> default:  -- Name:          dynamic-1.9.2
> 
>     [vagrant-1.9.2]        ==> default:  -- Cluster:       Default
> 
> 
> If you look at the ss:
> 
> the container 1.9.1 will wait for dynamic-1.9.1 and try to contact it at
> 192.168.2.54
> 
> the container 1.9.2 will wait for dynamic-1.9.2 and try to contact it at
> 192.168.2.55
> 
> But if you look at the logs, the 1.9.1 container started trying to work
> with 192.168.2.55 by creating a new key then talking to it:
> 
>      [vagrant-1.9.1]            default: Key inserted! Disconnecting and reconnecting using new SSH key...
> 
>     [vagrant-1.9.1]        Waiting for SSH service on 192.168.2.55:22 <http://192.168.2.55:22>, retrying in 3 seconds
> 
> 
> Because 1.9.1 inserted a generated key into that box, the 1.9.2
> container which _should_ be talking to it cannot now:
> 
>     [vagrant-1.9.2]        ==> default: Rsyncing folder: /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache
>     [vagrant-1.9.2]        SSH authentication failed! This is typically caused by the public/private
>     [vagrant-1.9.2]        keypair for the SSH user not being properly set on the guest VM. Please
>     [vagrant-1.9.2]        verify that the guest VM is setup with the proper public key, and that
>     [vagrant-1.9.2]        the private key path for Vagrant is setup properly as well.
> 
> 
> 
> Via the ruby sdk I create the VM and store the ID it responded with.
> Then to get the IP:
> 
>     server = env[:vms_service].vm_service(env[:machine].id)
>     nics_service = server.nics_service
>     nics = nics_service.list
>     ip_addr = nics.collect { |nic_attachment|
>     env[:connection].follow_link(nic_attachment).reported_devices.collect {
>     |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' } }
>     }.flatten.reject {   |ip| ip.nil? }.first rescue nil
> 

Is this code running inside the same Ruby process for both virtual
machines? In multiple threads?

> Given this code I can't think of any way that I would get the wrong IP
> unless somehow the server responded incorrectly, since the NIC's i've
> scanned and compiled across are tied directly to the server I created.
> 
> Any thoughts? This only happpens randomly and it seems to happen if I
> bombard the server with a bunch of VM creations simultaneously
> 
> [1] https://github.com/test-kitchen/test-kitchen
> [2] https://github.com/test-kitchen/kitchen-vagrant
> 
> 
> _______________________________________________
> Devel mailing list
> Devel at ovirt.org
> http://lists.ovirt.org/mailman/listinfo/devel
>