I've been fighting this for roughly two days and I'm starting to think that possibly it's not my code but an interaction with the server. I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to spin up vagrant machines and run tests against them. I'm using Jenkins to run kitchen in containers in parallel. Basically Jenkins runs a docker container with ruby + vagrant 1.9.2 and runs kitchen test all at the same time as another container with ruby + vagrant 1.9.1. If I run these in parallel, on some occasions the server seems to respond with the wrong creation information. If you look at the logs here: http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... the container for vagrant 1.9.1 created a VM `vagrant-dynamic-1.9.1: [vagrant-1.9.1] Bringing machine 'default' up with 'ovirt4' provider... [vagrant-1.9.1] ==> default: Creating VM with the following settings... [vagrant-1.9.1] ==> default: -- Name: dynamic-1.9.1 And the container for vagrant 1.9.2 (nearly the same time) created a VM `vagrant-dynamic-1.9.2`: [vagrant-1.9.2] ==> default: Creating VM with the following settings... [vagrant-1.9.2] ==> default: -- Name: dynamic-1.9.2 [vagrant-1.9.2] ==> default: -- Cluster: Default If you look at the ss: the container 1.9.1 will wait for dynamic-1.9.1 and try to contact it at 192.168.2.54 the container 1.9.2 will wait for dynamic-1.9.2 and try to contact it at 192.168.2.55 But if you look at the logs, the 1.9.1 container started trying to work with 192.168.2.55 by creating a new key then talking to it: [vagrant-1.9.1] default: Key inserted! Disconnecting and reconnecting using new SSH key... [vagrant-1.9.1] Waiting for SSH service on 192.168.2.55:22 <http://192.168.2.55:22>;, retrying in 3 seconds Because 1.9.1 inserted a generated key into that box, the 1.9.2 container which _should_ be talking to it cannot now: [vagrant-1.9.2] ==> default: Rsyncing folder: /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache [vagrant-1.9.2] SSH authentication failed! This is typically caused by the public/private [vagrant-1.9.2] keypair for the SSH user not being properly set on the guest VM. Please [vagrant-1.9.2] verify that the guest VM is setup with the proper public key, and that [vagrant-1.9.2] the private key path for Vagrant is setup properly as well. Via the ruby sdk I create the VM and store the ID it responded with. Then to get the IP: server = env[:vms_service].vm_service(env[:machine].id) nics_service = server.nics_service nics = nics_service.list ip_addr = nics.collect { |nic_attachment| env[:connection].follow_link(nic_attachment).reported_devices.collect { |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' } } }.flatten.reject { |ip| ip.nil? }.first rescue nil

Given this code I can't think of any way that I would get the wrong IP unless somehow the server responded incorrectly, since the NIC's i've scanned and compiled across are tied directly to the server I created. Any thoughts? This only happpens randomly and it seems to happen if I bombard the server with a bunch of VM creations simultaneously [1] https://github.com/test-kitchen/test-kitchen [2] https://github.com/test-kitchen/kitchen-vagrant _______________________________________________ Devel mailing list Devel(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/devel

Completely isolated docker containers. Jenkins basically runs two separate calls to docker... [vagrant-1.9.1] $ docker run -t -d -u 997:994 -v /opt/gemcache:/opt/gemcache -w /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ -v /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:/var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:rw -v /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:/var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.1 [Pipeline] [vagrant-1.9.1] { [Pipeline] [vagrant-1.9.2] withDockerContainer [vagrant-1.9.2] $ docker run -t -d -u 997:994 -v /opt/gemcache:/opt/gemcache -w /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ -v /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:/var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:rw -v /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:/var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.2 Each of those containers in turn runs: +gem build *.gemspec +/usr/bin/vagrant plugin install *.gem +bundle install --path /opt/gemcache --without development plugins +bundle exec kitchen destroy all +rm -rf .kitchen +sleep \$(shuf -i 0-10 -n 1) #i did this to see if maybe i could stagger the creates +export VAGRANT_VERSION=\$(echo ${vagrantVersion} | sed 's/\\.//g') +bundle exec kitchen test ^[^singleton-] On Tue, Mar 7, 2017 at 11:01 AM, Juan Hernández <jhernand(a)redhat.com <mailto:jhernand@redhat.com>> wrote: On 03/07/2017 05:42 PM, Marc Young wrote: > I've been fighting this for roughly two days and I'm starting to think > that possibly it's not my code but an interaction with the server. > > I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to spin up > vagrant machines and run tests against them. I'm using Jenkins to run > kitchen in containers in parallel. > > Basically Jenkins runs a docker container with ruby + vagrant 1.9.2 and > runs kitchen test all at the same time as another container with ruby + > vagrant 1.9.1. > > If I run these in parallel, on some occasions the server seems to > respond with the wrong creation information. If you look at the logs > here: http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... > <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change... > > > the container for vagrant 1.9.1 created a VM `vagrant-dynamic-1.9.1: > > [vagrant-1.9.1] Bringing machine 'default' up with 'ovirt4' provider... > > [vagrant-1.9.1] ==> default: Creating VM with the following settings... > > [vagrant-1.9.1] ==> default: -- Name: dynamic-1.9.1 > > > And the container for vagrant 1.9.2 (nearly the same time) created a VM > `vagrant-dynamic-1.9.2`: > > [vagrant-1.9.2] ==> default: Creating VM with the following settings... > > [vagrant-1.9.2] ==> default: -- Name: dynamic-1.9.2 > > [vagrant-1.9.2] ==> default: -- Cluster: Default > > > If you look at the ss: > > the container 1.9.1 will wait for dynamic-1.9.1 and try to contact it at > 192.168.2.54 > > the container 1.9.2 will wait for dynamic-1.9.2 and try to contact it at > 192.168.2.55 > > But if you look at the logs, the 1.9.1 container started trying to work > with 192.168.2.55 by creating a new key then talking to it: > > [vagrant-1.9.1] default: Key inserted! Disconnecting and reconnecting using new SSH key... > > [vagrant-1.9.1] Waiting for SSH service on 192.168.2.55:22 <http://192.168.2.55:22> <http://192.168.2.55:22>;, retrying in 3 seconds > > > Because 1.9.1 inserted a generated key into that box, the 1.9.2 > container which _should_ be talking to it cannot now: > > [vagrant-1.9.2] ==> default: Rsyncing folder: /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache > [vagrant-1.9.2] SSH authentication failed! This is typically caused by the public/private > [vagrant-1.9.2] keypair for the SSH user not being properly set on the guest VM. Please > [vagrant-1.9.2] verify that the guest VM is setup with the proper public key, and that > [vagrant-1.9.2] the private key path for Vagrant is setup properly as well. > > > > Via the ruby sdk I create the VM and store the ID it responded with. > Then to get the IP: >

> server = env[:vms_service].vm_service(env[:machine].id) > nics_service = server.nics_service > nics = nics_service.list > ip_addr = nics.collect { |nic_attachment| > env[:connection].follow_link(nic_attachment).reported_devices.collect { > |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' } } > }.flatten.reject { |ip| ip.nil? }.first rescue nil > Is this code running inside the same Ruby process for both virtual machines? In multiple threads? > Given this code I can't think of any way that I would get the wrong IP > unless somehow the server responded incorrectly, since the NIC's i've > scanned and compiled across are tied directly to the server I created. > > Any thoughts? This only happpens randomly and it seems to happen if I > bombard the server with a bunch of VM creations simultaneously > > [1] https://github.com/test-kitchen/test-kitchen <https://github.com/test-kitchen/test-kitchen> > [2] https://github.com/test-kitchen/kitchen-vagrant <https://github.com/test-kitchen/kitchen-vagrant> > > > _______________________________________________ > Devel mailing list > Devel(a)ovirt.org <mailto:Devel@ovirt.org> > http://lists.ovirt.org/mailman/listinfo/devel <http://lists.ovirt.org/mailman/listinfo/devel> >

This is where the ID is retrieved and stored: https://github.com/myoung34/ vagrant-ovirt4/blob/master/lib/vagrant-ovirt4/action/create_vm.rb#L79 workflow is: create, wait til disks are OK, wait until vm status is "down", create network interfaces[1], start vm[2], wait until up[3] [1] https://github.com/myoung34/vagrant-ovirt4/blob/ master/lib/vagrant-ovirt4/action/create_network_interfaces.rb#L53 [2] https://github.com/myoung34/vagrant-ovirt4/blob/ master/lib/vagrant-ovirt4/action/start_vm.rb#L83 [3] https://github.com/myoung34/vagrant-ovirt4/blob/ master/lib/vagrant-ovirt4/action/wait_till_up.rb#L43 On Tue, Mar 7, 2017 at 11:34 AM, Juan Hernández <jhernand(a)redhat.com&gt; wrote: > On 03/07/2017 06:06 PM, Marc Young wrote: > > Completely isolated docker containers. Jenkins basically runs two > > separate calls to docker... > > > > [vagrant-1.9.1] $ docker run -t -d -u 997:994 -v > /opt/gemcache:/opt/gemcache -w /var/lib/jenkins/workspace/oun > g34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ > -v /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKV > M5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:/var/lib/ > jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPE > CFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:rw -v > /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKV > M5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:/var/ > lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ > 5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:rw -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.1 > > [Pipeline] [vagrant-1.9.1] { > > > > > > > > [Pipeline] [vagrant-1.9.2] withDockerContainer [vagrant-1.9.2] $ > docker run -t -d -u 997:994 -v /opt/gemcache:/opt/gemcache -w > /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKV > M5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ -v > /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKV > M5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:/var/lib/ > jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ5BGPE > CFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ:rw -v > /var/lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKV > M5TQ5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:/var/ > lib/jenkins/workspace/oung34_vagrant-ovirt4_PR-79-7BRKVM5TQ > 5BGPECFMXYIEOYZOICCET4GY37WXT4D65NSV4F5TADQ@tmp:rw -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e > ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.2 > > > > > > Each of those containers in turn runs: > > > > > > +gem build *.gemspec > > +/usr/bin/vagrant plugin install *.gem > > +bundle install --path /opt/gemcache --without > development plugins > > +bundle exec kitchen destroy all > > +rm -rf .kitchen > > +sleep \$(shuf -i 0-10 -n 1) #i did this to see if maybe > i could > > stagger the creates > > +export VAGRANT_VERSION=\$(echo ${vagrantVersion} | sed > 's/\\.//g') > > +bundle exec kitchen test ^[^singleton-] > > > > > > On Tue, Mar 7, 2017 at 11:01 AM, Juan Hernández <jhernand(a)redhat.com > > <mailto:jhernand@redhat.com>> wrote: > > > > On 03/07/2017 05:42 PM, Marc Young wrote: > > > I've been fighting this for roughly two days and I'm starting to > think > > > that possibly it's not my code but an interaction with the server. > > > > > > I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to > spin up > > > vagrant machines and run tests against them. I'm using Jenkins to > run > > > kitchen in containers in parallel. > > > > > > Basically Jenkins runs a docker container with ruby + vagrant > 1.9.2 and > > > runs kitchen test all at the same time as another container with > ruby + > > > vagrant 1.9.1. > > > > > > If I run these in parallel, on some occasions the server seems to > > > respond with the wrong creation information. If you look at the > logs > > > here: http://home.blindrage.us:8080/ > job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console > > <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovi > rt4/view/change-requests/job/PR-79/41/console> > > > <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovir > t4/view/change-requests/job/PR-79/41/console > > <http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovi > rt4/view/change-requests/job/PR-79/41/console>> > > > > > > > > > the container for vagrant 1.9.1 created a VM > `vagrant-dynamic-1.9.1: > > > > > > [vagrant-1.9.1] Bringing machine 'default' up with > 'ovirt4' provider... > > > > > > [vagrant-1.9.1] ==> default: Creating VM with the > following settings... > > > > > > [vagrant-1.9.1] ==> default: -- Name: > dynamic-1.9.1 > > > > > > > > > And the container for vagrant 1.9.2 (nearly the same time) > created a VM > > > `vagrant-dynamic-1.9.2`: > > > > > > [vagrant-1.9.2] ==> default: Creating VM with the > following settings... > > > > > > [vagrant-1.9.2] ==> default: -- Name: > dynamic-1.9.2 > > > > > > [vagrant-1.9.2] ==> default: -- Cluster: Default > > > > > > > > > If you look at the ss: > > > > > > the container 1.9.1 will wait for dynamic-1.9.1 and try to > contact it at > > > 192.168.2.54 > > > > > > the container 1.9.2 will wait for dynamic-1.9.2 and try to > contact it at > > > 192.168.2.55 > > > > > > But if you look at the logs, the 1.9.1 container started trying > to work > > > with 192.168.2.55 by creating a new key then talking to it: > > > > > > [vagrant-1.9.1] default: Key inserted! > Disconnecting and reconnecting using new SSH key... > > > > > > [vagrant-1.9.1] Waiting for SSH service on > > 192.168.2.55:22 <http://192.168.2.55:22> <http://192.168.2.55:22>;, > > retrying in 3 seconds > > > > > > > > > Because 1.9.1 inserted a generated key into that box, the 1.9.2 > > > container which _should_ be talking to it cannot now: > > > > > > [vagrant-1.9.2] ==> default: Rsyncing folder: > /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache > > > [vagrant-1.9.2] SSH authentication failed! This is > typically caused by the public/private > > > [vagrant-1.9.2] keypair for the SSH user not being > properly set on the guest VM. Please > > > [vagrant-1.9.2] verify that the guest VM is setup with > the proper public key, and that > > > [vagrant-1.9.2] the private key path for Vagrant is > setup properly as well. > > > > > > > > > > > > Via the ruby sdk I create the VM and store the ID it responded > with. > > > Then to get the IP: > > > > > Can you share this ^ code that creates and stores the ID of the virtual > machine? > > > > server = env[:vms_service].vm_service(env[:machine].id) > > > nics_service = server.nics_service > > > nics = nics_service.list > > > ip_addr = nics.collect { |nic_attachment| > > > env[:connection].follow_link(nic_attachment).reported_devices.collect > { > > > |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' > } } > > > }.flatten.reject { |ip| ip.nil? }.first rescue nil > > > > > > > Is this code running inside the same Ruby process for both virtual > > machines? In multiple threads? > > > > > Given this code I can't think of any way that I would get the > wrong IP > > > unless somehow the server responded incorrectly, since the NIC's > i've > > > scanned and compiled across are tied directly to the server I > created. > > > > > > Any thoughts? This only happpens randomly and it seems to happen > if I > > > bombard the server with a bunch of VM creations simultaneously > > > > > > [1] https://github.com/test-kitchen/test-kitchen > > <https://github.com/test-kitchen/test-kitchen> > > > [2] https://github.com/test-kitchen/kitchen-vagrant > > <https://github.com/test-kitchen/kitchen-vagrant> > > > > > > > > > _______________________________________________ > > > Devel mailing list > > > Devel(a)ovirt.org <mailto:Devel@ovirt.org> > > > http://lists.ovirt.org/mailman/listinfo/devel > > <http://lists.ovirt.org/mailman/listinfo/devel> > > > > > > > > >