<div dir="ltr">I've been fighting this for roughly two days and I'm starting to think that possibly it's not my code but an interaction with the server.<div><br></div><div>I'm using test-kitchen[1] with the kitchen-vagrant[2] driver to spin up vagrant machines and run tests against them. I'm using Jenkins to run kitchen in containers in parallel.</div><div><br></div><div>Basically Jenkins runs a docker container with ruby + vagrant 1.9.2 and runs kitchen test all at the same time as another container with ruby + vagrant 1.9.1.</div><div><br></div><div>If I run these in parallel, on some occasions the server seems to respond with the wrong creation information. If you look at the logs here: <a href="http://home.blindrage.us:8080/job/myoung34/job/vagrant-ovirt4/view/change-requests/job/PR-79/41/console" target="_blank">http://home.blindrage.<wbr>us:8080/job/myoung34/job/<wbr>vagrant-ovirt4/view/change-<wbr>requests/job/PR-79/41/console</a></div><div><br></div><div><br></div><div>the container for vagrant 1.9.1 created a VM `vagrant-dynamic-1.9.1:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.1] Bringing machine 'default' up with 'ovirt4' provider...</pre></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.1] ==> default: Creating VM with the following settings...</pre></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.1] ==> default: -- Name: dynamic-1.9.1</pre><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><br></pre></div></blockquote><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">And the container for vagrant 1.9.2 (nearly the same time) created a VM `vagrant-dynamic-1.9.2`:</span></font><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.2] ==> default: Creating VM with the following settings...</pre></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.2] ==> default: -- Name: dynamic-1.9.2</pre></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.2] ==> default: -- Cluster: Default</pre><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"><br></pre></div></blockquote><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">If you look at the ss: <img src="http://i.imgur.com/NdqpGL6.png" width="544" height="34"><br></span></font><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">the container 1.9.1 will wait for dynamic-1.9.1 and try to contact it at 192.168.2.54</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">the container 1.9.2 will wait for dynamic-1.9.2 and try to contact it at 192.168.2.55</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">But if you look at the logs, the 1.9.1 container started trying to work with 192.168.2.55 by creating a new key then talking to it:</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px"> [vagrant-1.9.1] default: Key inserted! Disconnecting and reconnecting using new SSH key...</pre></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px;color:rgb(51,51,51);font-size:14px">[vagrant-1.9.1] Waiting for SSH service on <a href="http://192.168.2.55:22">192.168.2.55:22</a>, retrying in 3 seconds</pre></div></blockquote><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><div><br></div><div>Because 1.9.1 inserted a generated key into that box, the 1.9.2 container which _should_ be talking to it cannot now:</div><div><br></div></span></font><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><div><pre style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin-top:0px;margin-bottom:0px">[vagrant-1.9.2] ==> default: Rsyncing folder: /home/jenkins/.kitchen/cache/ => /tmp/omnibus/cache
[vagrant-1.9.2] SSH authentication failed! This is typically caused by the public/private
[vagrant-1.9.2] keypair for the SSH user not being properly set on the guest VM. Please
[vagrant-1.9.2] verify that the guest VM is setup with the proper public key, and that
[vagrant-1.9.2] the private key path for Vagrant is setup properly as well.</pre></div></span></font></blockquote><div><br></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">Via the ruby sdk I create the VM and store the ID it responded with.</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">Then to get the IP:</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">server = env[:vms_service].vm_service(env[:machine].id)</span><br></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><div>nics_service = server.nics_service</div><div>nics = nics_service.list</div><div>ip_addr = nics.collect { |nic_attachment| env[:connection].follow_link(nic_attachment).reported_devices.collect { |dev| dev.ips.collect { |ip| ip.address if ip.version == 'v4' } } }.flatten.reject { |ip| ip.nil? }.first rescue nil</div><div><br></div></span></font></div></blockquote><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">Given this code I can't think of any way that I would get the wrong IP unless somehow the server responded incorrectly, since the NIC's i've scanned and compiled across are tied directly to the server I created.</span></font><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">Any thoughts? This only happpens randomly and it seems to happen if I bombard the server with a bunch of VM creations simultaneously</span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap"><br></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">[1] <a href="https://github.com/test-kitchen/test-kitchen">https://github.com/test-kitchen/test-kitchen</a></span></font></div><div><font color="#333333" face="monospace"><span style="font-size:14px;white-space:pre-wrap">[2] <a href="https://github.com/test-kitchen/kitchen-vagrant">https://github.com/test-kitchen/kitchen-vagrant</a></span><br></font></div></div>