<div dir="ltr">I just had the "aha" moment and i"m sorry. I think I'll take a nap.<div>In jenkins the local directory is mounted into the container in order to get the code there. This includes the `.kitchen` dir which has all the state information. So the 2nd run of `kitchen ...` will have the dir and try to use the vagrant state information.</div><div><br></div><div>Disregard, it's one of those stupid problems you have when you run things in parallel/isolation and dont question how isolated it actually is.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 7, 2017 at 11:47 AM, Marc Young <span dir="ltr"><<a href="mailto:3vilpenguin@gmail.com" target="_blank">3vilpenguin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This is where the ID is retrieved and stored: <a href="https://github.com/myoung34/vagrant-ovirt4/blob/master/lib/vagrant-ovirt4/action/create_vm.rb#L79" target="_blank">https://github.com/myoung34/<wbr>vagrant-ovirt4/blob/master/<wbr>lib/vagrant-ovirt4/action/<wbr>create_vm.rb#L79</a><br><div><br></div><div>workflow is: create, wait til disks are OK, wait until vm status is "down", create network interfaces[1], start vm[2], wait until up[3]</div><div><br></div><div>[1] <a href="https://github.com/myoung34/vagrant-ovirt4/blob/master/lib/vagrant-ovirt4/action/create_network_interfaces.rb#L53" target="_blank">https://github.com/<wbr>myoung34/vagrant-ovirt4/blob/<wbr>master/lib/vagrant-ovirt4/<wbr>action/create_network_<wbr>interfaces.rb#L53</a></div><div>[2] <a href="https://github.com/myoung34/vagrant-ovirt4/blob/master/lib/vagrant-ovirt4/action/start_vm.rb#L83" target="_blank">https://github.com/<wbr>myoung34/vagrant-ovirt4/blob/<wbr>master/lib/vagrant-ovirt4/<wbr>action/start_vm.rb#L83</a></div><div>[3] <a href="https://github.com/myoung34/vagrant-ovirt4/blob/master/lib/vagrant-ovirt4/action/wait_till_up.rb#L43" target="_blank">https://github.com/<wbr>myoung34/vagrant-ovirt4/blob/<wbr>master/lib/vagrant-ovirt4/<wbr>action/wait_till_up.rb#L43</a></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 7, 2017 at 11:34 AM, Juan Hernández <span dir="ltr"><<a href="mailto:jhernand@redhat.com" target="_blank">jhernand@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 03/07/2017 06:06 PM, Marc Young wrote:<br>
> Completely isolated docker containers. Jenkins basically runs two<br>
> separate calls to docker...<br>
><br>
> [vagrant-1.9.1] $ docker run -t -d -u 997:994 -v /opt/gemcache:/opt/gemcache -w /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ -v /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ:/var/lib/<wbr>jenkins/workspace/oung34_vagra<wbr>nt-ovirt4_PR-79-7BRKVM5TQ5BGPE<wbr>CFMXYIEOYZOICCET4GY37WXT4D65NS<wbr>V4F5TADQ:rw -v /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ@tmp:/var/<wbr>lib/jenkins/workspace/oung34_<wbr>vagrant-ovirt4_PR-79-7BRKVM5TQ<wbr>5BGPECFMXYIEOYZOICCET4GY37WXT4<wbr>D65NSV4F5TADQ@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.1<br>
> [Pipeline] [vagrant-1.9.1] {<br>
><br>
><br>
><br>
> [Pipeline] [vagrant-1.9.2] withDockerContainer [vagrant-1.9.2] $ docker run -t -d -u 997:994 -v /opt/gemcache:/opt/gemcache -w /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ -v /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ:/var/lib/<wbr>jenkins/workspace/oung34_vagra<wbr>nt-ovirt4_PR-79-7BRKVM5TQ5BGPE<wbr>CFMXYIEOYZOICCET4GY37WXT4D65NS<wbr>V4F5TADQ:rw -v /var/lib/jenkins/workspace/oun<wbr>g34_vagrant-ovirt4_PR-79-7BRKV<wbr>M5TQ5BGPECFMXYIEOYZOICCET4GY37<wbr>WXT4D65NSV4F5TADQ@tmp:/var/<wbr>lib/jenkins/workspace/oung34_<wbr>vagrant-ovirt4_PR-79-7BRKVM5TQ<wbr>5BGPECFMXYIEOYZOICCET4GY37WXT4<wbr>D65NSV4F5TADQ@tmp:rw -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** --entrypoint cat myoung34/vagrant:1.9.2<br>
><br>
><br>
> Each of those containers in turn runs:<br>
><br>
><br>
> +gem build *.gemspec<br>
> +/usr/bin/vagrant plugin install *.gem<br>
</span>> +bundle install --path /opt/gemcache --without development plugins<br>
<span>> +bundle exec kitchen destroy all<br>
> +rm -rf .kitchen<br>
</span>> +sleep \$(shuf -i 0-10 -n 1) #i did this to see if maybe i could<br>
> stagger the creates<br>
> +export VAGRANT_VERSION=\$(echo ${vagrantVersion} | sed 's/\\.//g')<br>
> +bundle exec kitchen test ^[^singleton-]<br>
<span>><br>
><br>
> On Tue, Mar 7, 2017 at 11:01 AM, Juan Hernández <<a href="mailto:jhernand@redhat.com" target="_blank">jhernand@redhat.com</a><br>
</span><div><div class="m_7328900949599692151h5">> <mailto:<a href="mailto:jhernand@redhat.com" target="_blank">jhernand@redhat.com</a>>> wrote:<br>
><br>
> On 03/07/2017 05:42 PM, Marc Young wrote:<br>
> > I've been fighting this for roughly two days and I'm starting to think<br>
> > that possibly it's not my code but an interaction with the server.<br>
> ><br>
> > I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to spin up<br>
> > vagrant machines and run tests against them. I'm using Jenkins to run<br>
> > kitchen in containers in parallel.<br>
> ><br>
> > Basically Jenkins runs a docker container with ruby + vagrant 1.9.2 and<br>
> > runs kitchen test all at the same time as another container with ruby +<br>
> > vagrant 1.9.1.<br>
> ><br>
> > If I run these in parallel, on some occasions the server seems to<br>
> > respond with the wrong creation information. If you look at the logs<br>
> > here: <a href="http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console" rel="noreferrer" target="_blank">http://home.blindrage.us:8080/<wbr>job/myoung34/job/vagrant-ovirt<wbr>4/view/change-requests/job/PR-<wbr>79/41/console</a><br>
> <<a href="http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console" rel="noreferrer" target="_blank">http://home.blindrage.us:808<wbr>0/job/myoung34/job/vagrant-ovi<wbr>rt4/view/change-requests/job/<wbr>PR-79/41/console</a>><br>
> > <<a href="http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console" rel="noreferrer" target="_blank">http://home.blindrage.us:8080<wbr>/job/myoung34/job/vagrant-ovir<wbr>t4/view/change-requests/job/<wbr>PR-79/41/console</a><br>
> <<a href="http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console" rel="noreferrer" target="_blank">http://home.blindrage.us:808<wbr>0/job/myoung34/job/vagrant-ovi<wbr>rt4/view/change-requests/job/<wbr>PR-79/41/console</a>>><br>
> ><br>
> ><br>
> > the container for vagrant 1.9.1 created a VM `vagrant-dynamic-1.9.1:<br>
> ><br>
> > [vagrant-1.9.1] Bringing machine 'default' up with 'ovirt4' provider...<br>
> ><br>
> > [vagrant-1.9.1] ==> default: Creating VM with the following settings...<br>
> ><br>
> > [vagrant-1.9.1] ==> default: -- Name: dynamic-1.9.1<br>
> ><br>
> ><br>
> > And the container for vagrant 1.9.2 (nearly the same time) created a VM<br>
> > `vagrant-dynamic-1.9.2`:<br>
> ><br>
> > [vagrant-1.9.2] ==> default: Creating VM with the following settings...<br>
> ><br>
> > [vagrant-1.9.2] ==> default: -- Name: dynamic-1.9.2<br>
> ><br>
> > [vagrant-1.9.2] ==> default: -- Cluster: Default<br>
> ><br>
> ><br>
> > If you look at the ss:<br>
> ><br>
> > the container 1.9.1 will wait for dynamic-1.9.1 and try to contact it at<br>
> > 192.168.2.54<br>
> ><br>
> > the container 1.9.2 will wait for dynamic-1.9.2 and try to contact it at<br>
> > 192.168.2.55<br>
> ><br>
> > But if you look at the logs, the 1.9.1 container started trying to work<br>
> > with 192.168.2.55 by creating a new key then talking to it:<br>
> ><br>
> > [vagrant-1.9.1] default: Key inserted! Disconnecting and reconnecting using new SSH key...<br>
> ><br>
> > [vagrant-1.9.1] Waiting for SSH service on<br>
</div></div>> <a href="http://192.168.2.55:22" rel="noreferrer" target="_blank">192.168.2.55:22</a> <<a href="http://192.168.2.55:22" rel="noreferrer" target="_blank">http://192.168.2.55:22</a>> <<a href="http://192.168.2.55:22" rel="noreferrer" target="_blank">http://192.168.2.55:22</a>>,<br>
<span>> retrying in 3 seconds<br>
> ><br>
> ><br>
> > Because 1.9.1 inserted a generated key into that box, the 1.9.2<br>
> > container which _should_ be talking to it cannot now:<br>
> ><br>
> > [vagrant-1.9.2] ==> default: Rsyncing folder: /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache<br>
> > [vagrant-1.9.2] SSH authentication failed! This is typically caused by the public/private<br>
> > [vagrant-1.9.2] keypair for the SSH user not being properly set on the guest VM. Please<br>
> > [vagrant-1.9.2] verify that the guest VM is setup with the proper public key, and that<br>
> > [vagrant-1.9.2] the private key path for Vagrant is setup properly as well.<br>
> ><br>
> ><br>
> ><br>
> > Via the ruby sdk I create the VM and store the ID it responded with.<br>
> > Then to get the IP:<br>
> ><br>
<br>
</span>Can you share this ^ code that creates and stores the ID of the virtual<br>
machine?<br>
<span><br>
> > server = env[:vms_service].vm_service(e<wbr>nv[:machine].id)<br>
> > nics_service = server.nics_service<br>
> > nics = nics_service.list<br>
> > ip_addr = nics.collect { |nic_attachment|<br>
> > env[:connection].follow_link(<wbr>nic_attachment).reported_devic<wbr>es.collect {<br>
> > |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' } }<br>
> > }.flatten.reject { |ip| ip.nil? }.first rescue nil<br>
> ><br>
><br>
> Is this code running inside the same Ruby process for both virtual<br>
> machines? In multiple threads?<br>
><br>
> > Given this code I can't think of any way that I would get the wrong IP<br>
> > unless somehow the server responded incorrectly, since the NIC's i've<br>
> > scanned and compiled across are tied directly to the server I created.<br>
> ><br>
> > Any thoughts? This only happpens randomly and it seems to happen if I<br>
> > bombard the server with a bunch of VM creations simultaneously<br>
> ><br>
> > [1] <a href="https://github.com/test-kitchen/test-kitchen" rel="noreferrer" target="_blank">https://github.com/test-kitche<wbr>n/test-kitchen</a><br>
> <<a href="https://github.com/test-kitchen/test-kitchen" rel="noreferrer" target="_blank">https://github.com/test-kitc<wbr>hen/test-kitchen</a>><br>
> > [2] <a href="https://github.com/test-kitchen/kitchen-vagrant" rel="noreferrer" target="_blank">https://github.com/test-kitche<wbr>n/kitchen-vagrant</a><br>
> <<a href="https://github.com/test-kitchen/kitchen-vagrant" rel="noreferrer" target="_blank">https://github.com/test-kitc<wbr>hen/kitchen-vagrant</a>><br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > Devel mailing list<br>
</span>> > <a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a> <mailto:<a href="mailto:Devel@ovirt.org" target="_blank">Devel@ovirt.org</a>><br>
> > <a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailman<wbr>/listinfo/devel</a><br>
> <<a href="http://lists.ovirt.org/mailman/listinfo/devel" rel="noreferrer" target="_blank">http://lists.ovirt.org/mailm<wbr>an/listinfo/devel</a>><br>
> ><br>
><br>
><br>
<br>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>