<div dir="ltr"><div>Progress: I managed to get a host setup and then a vm setup on the host, but the VM won't start. I'm looking for some help getting the VM to start. Details...</div><div><br></div>I was able to work around the VDSM host in a failed state issue (I think), by doing a host reinstall from the engine web GUI, after specifying the the root password for the host. That is, I changed to password authentication, instead of what was originally set to SSH PublicKey authentication. After the host reinstall completed, I was left with what appears to be an all-in-one setup with the engine and a host running on the same server. Maybe this would have worked automatically, without the re-install step, if I had ovirt SSH keys setup for root as expected? <div><br></div><div>Question: Any hints on to how to debug this in terms of host install SSH PublicKey auth not working? What user is attempting to SSH from the engine to the host with what private key to complete the setup? I see an ovirt-engine public key that was added to /root/.ssh/authorized_keys. Is this the public key that's expected to work with the automated all-in-one host setup? If so, where's the corresponding private key so I might attempt to debug further?<div><br></div><div>After adding a virtual local disk to the new host and setting up a new VM through the web interface, I tried to attempt to start the VM but got this permission denied error in engine.log:</div><div><br></div><div><div>2015-12-29 12:01:47,814 ERROR [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ForkJoinPool-1-worker-3) [] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: VM nycovirttest1 is down with error. Exit message: internal error: process exited while connecting to monitor: 2015-12-29T17:01:45.424728Z qemu-kvm: -drive file=/rhev/data-center/62bbbc20-00c6-46b3-8ac9-2677a4da69e4/954ebee1-521b-454d-b54c-db9516f36cff/images/e87e1633-7afc-4835-931e-c444e1cf72f5/f6dc9f53-b373-4c96-9015-25b316dcbf0a,if=none,id=drive-virtio-disk0,format=raw,serial=e87e1633-7afc-4835-931e-c444e1cf72f5,cache=none,werror=stop,rerror=stop,aio=threads: Could not open '/rhev/data-center/62bbbc20-00c6-46b3-8ac9-2677a4da69e4/954ebee1-521b-454d-b54c-db9516f36cff/images/e87e1633-7afc-4835-931e-c444e1cf72f5/f6dc9f53-b373-4c96-9015-25b316dcbf0a': Permission denied</div></div><div><br></div><div>Question: How can I debug this VM startup permission denied error further? Is there a command line I can issue that's similar to what ovirt engine is attempting that can give me more detail? Something I can strace?</div><div><br></div><div>The vdsm user can read the above referenced image just fine, so it's something more than just simple filesystem permissions I think?:</div><div><br></div><div><div>$ id</div><div>uid=36(vdsm) gid=36(kvm) groups=36(kvm),107(qemu),179(sanlock)</div><div><br></div></div><div><div>$ file /rhev/data-center/62bbbc20-00c6-46b3-8ac9-2677a4da69e4/954ebee1-521b-454d-b54c-db9516f36cff/images/e87e1633-7afc-4835-931e-c444e1cf72f5/f6dc9f53-b373-4c96-9015-25b316dcbf0a</div><div>/rhev/data-center/62bbbc20-00c6-46b3-8ac9-2677a4da69e4/954ebee1-521b-454d-b54c-db9516f36cff/images/e87e1633-7afc-4835-931e-c444e1cf72f5/f6dc9f53-b373-4c96-9015-25b316dcbf0a: data</div></div><div><br></div><div>Thanks for your help,</div><div><br></div><div>-Matthew</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 28, 2015 at 12:46 PM, Matthew Bohnsack <span dir="ltr"><<a href="mailto:bohnsack@gmail.com" target="_blank">bohnsack@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><font face="arial, helvetica, sans-serif">Hello,</font><div><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif">I am attempting to build a fairly basic proof of concept ovirt 3.6 machine with the engine, host and guests all on a single physical CentOS 7.2 box with local storage (to start), but am running into issues where engine-install produces the following error: </font></div><div><font face="arial, helvetica, sans-serif"><br></font></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="margin-top:0px"><font face="arial, helvetica, sans-serif">[ ERROR ] The VDSM host was found in a failed state. Please check engine and bootstrap installation logs.</font></div></div></div></blockquote><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div><div><font face="arial, helvetica, sans-serif"><font color="#000000"><span style="line-height:14.64px">The engine webconsole seems to be working fine after this - I can login and click around - but there's no host CPU/storage resources available, as expected. From the logs shown below, there seems to be an issue with the vdsm installation process being unable to contact the engine host during installation. Any ideas what's going wrong and/or what I can do to debug things further so I can move beyond this issue?</span></font></font></div><div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br>Thanks,</font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">-Matthew</font></div></div><div><br></div><div><br></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">Steps I took to install and diagnose the problem:</font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">1. Installed system with our configuration management system.</font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">2. Deleted users using UID=36 (needed by vdsm user) and UID=108 (needed by ovirt user).</font></div><div><font face="arial, helvetica, sans-serif" style="color:rgb(0,0,0);line-height:14.64px"><br></font></div><div><font face="arial, helvetica, sans-serif" style="color:rgb(0,0,0);line-height:14.64px">3. Ensured that /etc/sudoers contained the line "</font><font color="#000000" face="arial, helvetica, sans-serif"><span style="line-height:14.64px">#includedir /etc/sudoers.d" so that /etc/sudoers.d/50_vdsm will take effect.</span></font></div><div><font color="#000000" face="arial, helvetica, sans-serif"><span style="line-height:14.64px"><br></span></font></div><div><font color="#000000" face="arial, helvetica, sans-serif"><span style="line-height:14.64px">4. Made directories for isos and images:</span></font></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font color="#000000" face="arial, helvetica, sans-serif"><span style="line-height:14.64px"># mkdir /state/partition1/images/; chmod 777 /state/partition1/images/</span></font></div><div><span style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;line-height:14.64px"># mkdir /state/partition1/iso/; chmod 777 /state/partition1/iso/</span><font color="#000000" face="arial, helvetica, sans-serif"><span style="line-height:14.64px"><br></span></font></div></div></blockquote><div><div><div><br></div><div>5. Ensured selinux was disabled and no firewall rules were installed.</div><div><br></div><div>6. Installed RPMs:</div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><font face="arial, helvetica, sans-serif"># <span style="color:rgb(0,0,0);line-height:14.64px">yum -y install </span><a href="http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm" style="color:rgb(0,0,204);text-decoration:none;line-height:14.64px" target="_blank">http://resources.ovirt.org/pub/yum-repo/ovirt-release36.rpm</a></font></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0);line-height:14.64px"># yum -y install ovirt-engine ovirt-engine-setup-plugin-allinone</span></font></div><div><font face="arial, helvetica, sans-serif"><span style="color:rgb(0,0,0);line-height:14.64px"><div># rpm -qa | grep ovirt-release36</div><div>ovirt-release36-002-2.noarch</div></span></font></div></div></div></blockquote><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"></div></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">7. Installed engine with an all-in-one configuration (</font><span style="font-family:arial,helvetica,sans-serif;line-height:14.64px">Configure VDSM on this host? Yes</span><span style="font-family:arial,helvetica,sans-serif;line-height:14.64px">):</span></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><span style="line-height:14.64px"># cat /root/ovirt-engine.ans </span></div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"># action=setup</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">[environment:default]</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DIALOG/confirmSettings=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/applicationMode=str:virt</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/remoteEngineSetupStyle=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/sanWipeAfterDelete=bool:False</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/storageIsLocal=bool:False</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/firewallManager=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/remoteEngineHostRootPassword=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/firewallChangesReview=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/updateFirewall=bool:False</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/remoteEngineHostSshPort=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/fqdn=<...host.fqdn...></div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/storageType=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OSETUP_RPMDISTRO/requireRollback=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OSETUP_RPMDISTRO/enableUpgrade=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/secured=bool:False</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/host=str:localhost</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/user=str:engine</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/dumper=str:pg_custom</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/database=str:engine</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/fixDbViolations=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/port=int:5432</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/filter=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/restoreJobs=int:2</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_DB/securedHostValidation=bool:False</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_ENGINE_CORE/enable=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CORE/engineStop=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_SYSTEM/memCheckEnabled=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_SYSTEM/nfsConfigEnabled=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_PKI/organization=str:<...dn...></div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_PKI/renew=none:None</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/isoDomainName=str:ISO_DOMAIN</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/engineHeapMax=str:7975M</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/adminPassword=str:<...password...></div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/isoDomainACL=str:*(rw)</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/isoDomainMountPoint=str:/state/partition1/iso</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/engineDbBackupDir=str:/var/lib/ovirt-engine/backups</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/engineHeapMin=str:7975M</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_AIO/configure=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_AIO/storageDomainName=str:local_storage</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_AIO/storageDomainDir=str:/state/partition1/images/</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_PROVISIONING/postgresProvisioningEnabled=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_APACHE/configureRootRedirection=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_APACHE/configureSsl=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_VMCONSOLE_PROXY_CONFIG/vmconsoleProxyConfig=bool:True</div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_ENGINE_CONFIG/fqdn=str:<...fqdn...></div></div></div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">OVESETUP_CONFIG/websocketProxyConfig=bool:True</div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif"><br></font></div></div></div></blockquote><div><div><div><blockquote style="margin:0px 0px 0px 40px;color:rgb(0,0,0);line-height:14.64px;border:none;padding:0px"><div style="margin-top:0px"><div style="margin-top:0px"><div style="margin-top:0px"><font face="arial, helvetica, sans-serif"># engine-setup --config-append=/root/ovirt-engine.ans</font></div></div></div></blockquote><blockquote style="margin:0px 0px 0px 40px;color:rgb(0,0,0);line-height:14.64px;border:none;padding:0px"><div style="margin-top:0px"><div style="margin-top:0px"><div style="margin-top:0px"><font face="arial, helvetica, sans-serif">...</font></div></div></div><div><div style="margin-top:0px"><div style="margin-top:0px">[ INFO ] Starting engine service</div><div style="margin-top:0px">[ INFO ] Restarting httpd</div><div style="margin-top:0px">[ INFO ] Waiting for VDSM host to become operational. This may take several minutes...</div><div style="margin-top:0px">[ ERROR ] The VDSM host was found in a failed state. Please check engine and bootstrap installation logs.</div><div style="margin-top:0px">[WARNING] Local storage domain not added because the VDSM host was not up. Please add it manually.</div><div style="margin-top:0px">[ INFO ] Restarting ovirt-vmconsole proxy service</div><div style="margin-top:0px">[ INFO ] Stage: Clean up</div><div style="margin-top:0px"> Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20151228112813-iksxhe.log</div><div style="margin-top:0px">[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20151228113003-setup.conf'</div><div style="margin-top:0px">[ INFO ] Stage: Pre-termination</div><div style="margin-top:0px">[ INFO ] Stage: Termination</div><div style="margin-top:0px">[ INFO ] Execution of setup completed successfully</div><div><br></div></div></div></blockquote></div><div style="color:rgb(0,0,0);line-height:14.64px"><font face="arial, helvetica, sans-serif">8. Examined </font><span style="line-height:14.64px">/var/log/ovirt-engine/setup/ovirt-engine-setup-20151228112813-iksxhe.log and found this error message which seems to indicate that the vdsm installation process was unable to contact the engine:</span></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:29 DEBUG otopi.plugins.otopi.services.systemd plugin.execute:941 execute-output: ('/bin/systemctl', 'start', 'httpd.service') stderr:</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><br></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><br></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:29 DEBUG otopi.context context._executeMethod:142 Stage closeup METHOD otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi.Plugin._closeup</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:29 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:211 Connecting to the Engine</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:29 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitEngineUp:103 Waiting Engine API response</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:29 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitEngineUp:133 Cannot connect to engine</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">Traceback (most recent call last):</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/all-in-one/vdsmi.py", line 127, in _waitEngineUp</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> insecure=True,</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/api.py", line 191, in __init__</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> url=''</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 115, in request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> persistent_auth=self.__persistent_auth</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> persistent_auth)</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 155, in __do_request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> raise errors.RequestError(response_code, response_reason, response_body)</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">RequestError: ^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">status: 503^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">reason: Service Unavailable^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">detail:</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><html><head></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><title>503 Service Unavailable</title></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"></head><body></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><h1>Service Unavailable</h1></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><p>The server is temporarily unable to service your</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">request due to maintenance downtime or capacity</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">problems. Please try again later.</p></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"></body></html></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"><br></span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">2015-12-28 11:29:36 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitEngineUp:133 Cannot connect to engine</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">Traceback (most recent call last):</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/share/ovirt-engine/setup/bin/../plugins/ovirt-engine-setup/ovirt-engine/all-in-one/vdsmi.py", line 127, in _waitEngineUp</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> insecure=True,</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/api.py", line 191, in __init__</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> url=''</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/proxy.py", line 115, in request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> persistent_auth=self.__persistent_auth</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 79, in do_request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> persistent_auth)</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> File "/usr/lib/python2.7/site-packages/ovirtsdk/infrastructure/connectionspool.py", line 155, in __do_request</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px"> raise errors.RequestError(response_code, response_reason, response_body)</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">RequestError: ^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">status: 404^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">reason: Not Found^M</span></font></div></div></div></div><div><div><div><div><font color="#000000"><span style="line-height:14.64px">detail:</span></font></div><div><font color="#000000"><span style="line-height:14.64px">...</span></font></div><div><font color="#000000"><div><span style="line-height:14.64px"><!DOCTYPE html></span></div><div><span style="line-height:14.64px"><html></span></div><div><span style="line-height:14.64px"><head></span></div><div><span style="line-height:14.64px"> <meta http-equiv="Content-type" content="text/html; charset=utf-8" /></span></div><div><span style="line-height:14.64px"><br></span></div><div><span style="line-height:14.64px"><br></span></div><div><span style="line-height:14.64px"><link id="id-link-favicon" rel="shortcut icon" href="/ovirt-engine/theme-resource/favicon" type="image/x-icon" /></span></div><div><span style="line-height:14.64px"><br></span></div><div><span style="line-height:14.64px"> <title>404 - Page not found</title></span></div><div style="line-height:14.64px">...</div><div><div><span style="line-height:14.64px"></html></span></div><div><span style="line-height:14.64px"><br></span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:171 Database: 'None', Statement: '</span></div><div><span style="line-height:14.64px"> select version, option_value</span></div><div><span style="line-height:14.64px"> from vdc_options</span></div><div><span style="line-height:14.64px"> where option_name = %(name)s</span></div><div><span style="line-height:14.64px"> ', args: {'name': 'SupportedClusterLevels'}</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.ovirt_engine_setup.engine_common.database database.execute:221 Result: [{'version': 'general', 'option_value': '3.0,3.1,3.2,3.3,3.4,3.5,3.6'}]</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:225 engine SupportedClusterLevels [3.0,3.1,3.2,3.3,3.4,3.5,3.6], PACKAGE_VERSION [3.6.1.3],</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._getSupportedClusterLevels:181 Attempting to load the dsaversion vdsm module</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:236 VDSM SupportedClusterLevels [['3.4', '3.5', '3.6']], VDSM VERSION [4.17.13-0.el7.centos],</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:259 Creating the local data center</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:50 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:269 Creating the local cluster into the local data center</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:52 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:284 Adding the local host to the local cluster</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:55 INFO otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:58 Waiting for VDSM host to become operational. This may take several minutes...</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:55 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:56 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:58 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:29:59 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:30:00 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:30:01 DEBUG otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:87 VDSM host in installing state</span></div><div><span style="line-height:14.64px">2015-12-28 11:30:02 ERROR otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._waitVDSMHostUp:77 The VDSM host was found in a failed state. Please check engine and bootstrap installation logs.</span></div><div><span style="line-height:14.64px">2015-12-28 11:30:02 WARNING otopi.plugins.ovirt_engine_setup.ovirt_engine.all-in-one.vdsmi vdsmi._closeup:306 Local storage domain not added because the VDSM host was not up. Please add it manually.</span></div><div><span style="line-height:14.64px">2015-12-28 11:30:02 DEBUG otopi.context context._executeMethod:142 Stage closeup METHOD otopi.plugins.ovirt_engine_setup.vmconsole_proxy_helper.system.Plugin._closeup</span></div><div style="line-height:14.64px"><br></div></div></font></div></div></div></div></blockquote><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">9. Looked at the vdsm logs and examined service status:</div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"># ls -l /var/log/vdsm/</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">total 8</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">drwxr-xr-x 2 vdsm kvm 6 Dec 9 03:24 backup</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">-rw-r--r-- 1 vdsm kvm 0 Dec 28 11:13 connectivity.log</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">-rw-r--r-- 1 vdsm kvm 0 Dec 28 11:13 mom.log</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">-rw-r--r-- 1 root root 2958 Dec 28 11:30 supervdsm.log</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">-rw-r--r-- 1 root root 1811 Dec 28 11:30 upgrade.log</div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px">-rw-r--r-- 1 vdsm kvm 0 Dec 28 11:13 vdsm.log</div></div></div></div></div></blockquote><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px"><br></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px"># cat /var/log/vdsm/upgrade.log </div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,803::upgrade::90::upgrade::(apply_upgrade) Running upgrade upgrade-unified-persistence</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,806::libvirtconnection::160::root::(get) trying to connect libvirt</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,813::utils::669::root::(execCmd) /sbin/ip route show to <a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a> table main (cwd None)</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,826::utils::687::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,826::unified_persistence::46::root::(run) upgrade-unified-persistence upgrade persisting networks {} and bondings {}</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::INFO::2015-12-28 11:30:02,827::netconfpersistence::179::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,827::netconfpersistence::187::root::(_clearDisk) No existent config to clear.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::INFO::2015-12-28 11:30:02,827::netconfpersistence::179::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,827::netconfpersistence::187::root::(_clearDisk) No existent config to clear.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::INFO::2015-12-28 11:30:02,827::netconfpersistence::129::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,827::utils::669::root::(execCmd) /usr/share/vdsm/vdsm-store-net-config unified (cwd None)</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,836::utils::687::root::(execCmd) SUCCESS: <err> = 'cp: cannot stat \xe2\x80\x98/var/run/vdsm/netconf\xe2\x80\x99: No such file or directory\n'; <rc> = 0</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,837::upgrade::51::upgrade::(_upgrade_seal) Upgrade upgrade-unified-persistence successfully performed</div></div></div></div></div></div></blockquote><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px"><br></div></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px"># cat /var/log/vdsm/supervdsm.log </div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,415::supervdsmServer::539::SuperVdsm.Server::(main) Making sure I'm root - SuperVdsm</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,415::supervdsmServer::548::SuperVdsm.Server::(main) Parsing cmd args</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,415::supervdsmServer::551::SuperVdsm.Server::(main) Cleaning old socket /var/run/vdsm/svdsm.sock</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,415::supervdsmServer::555::SuperVdsm.Server::(main) Setting up keep alive thread</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,415::supervdsmServer::561::SuperVdsm.Server::(main) Creating remote object manager</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,416::fileUtils::192::Storage.fileUtils::(chown) Changing owner for /var/run/vdsm/svdsm.sock, to (36:36)</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">MainThread::DEBUG::2015-12-28 11:30:02,416::supervdsmServer::572::SuperVdsm.Server::(main) Started serving super vdsm object</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">sourceRoute::DEBUG::2015-12-28 11:30:02,416::sourceroutethread::79::root::(_subscribeToInotifyLoop) sourceRouteThread.subscribeToInotifyLoop started</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::DEBUG::2015-12-28 11:30:03,080::libvirtconnection::160::root::(get) trying to connect libvirt</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,188::vdsm-restore-net-config::86::root::(_restore_sriov_numvfs) SRIOV network device which is not persisted found at: 0000:01:00.1.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,189::vdsm-restore-net-config::86::root::(_restore_sriov_numvfs) SRIOV network device which is not persisted found at: 0000:01:00.0.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,189::vdsm-restore-net-config::86::root::(_restore_sriov_numvfs) SRIOV network device which is not persisted found at: 0000:01:00.3.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,189::vdsm-restore-net-config::86::root::(_restore_sriov_numvfs) SRIOV network device which is not persisted found at: 0000:01:00.2.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,189::vdsm-restore-net-config::385::root::(restore) starting network restoration.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::DEBUG::2015-12-28 11:30:03,189::vdsm-restore-net-config::183::root::(_remove_networks_in_running_config) Not cleaning running configuration since it is empty.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,205::netconfpersistence::179::root::(_clearDisk) Clearing /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::DEBUG::2015-12-28 11:30:03,206::netconfpersistence::187::root::(_clearDisk) No existent config to clear.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,206::netconfpersistence::129::root::(save) Saved new config RunningConfig({}, {}) to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::DEBUG::2015-12-28 11:30:03,207::vdsm-restore-net-config::329::root::(_wait_for_for_all_devices_up) All devices are up.</div></div></div></div></div></div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div style="line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,214::netconfpersistence::71::root::(setBonding) Adding bond0({'nics': [], 'options': ''})</div></div></div></div></div></div><div><div><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px">restore-net::INFO::2015-12-28 11:30:03,214::vdsm-restore-net-config::396::root::(restore) restoration completed successfully.</div><div style="color:rgb(0,0,0);line-height:14.64px"><br></div><div><div><font color="#000000"><span style="line-height:14.64px"># systemctl status supervdsmd</span></font></div><div><font color="#000000"><span style="line-height:14.64px">● supervdsmd.service - Auxiliary vdsm service for running helper functions as root</span></font></div><div><font color="#000000"><span style="line-height:14.64px"> Loaded: loaded (/usr/lib/systemd/system/supervdsmd.service; static; vendor preset: enabled)</span></font></div><div><font color="#000000"><span style="line-height:14.64px"> Active: active (running) since Mon 2015-12-28 11:30:02 EST; 1h 8min ago</span></font></div><div><font color="#000000"><span style="line-height:14.64px"> Main PID: 81535 (supervdsmServer)</span></font></div><div><font color="#000000"><span style="line-height:14.64px"> CGroup: /system.slice/supervdsmd.service</span></font></div><div><font color="#000000"><span style="line-height:14.64px"> └─81535 /usr/bin/python /usr/share/vdsm/supervdsmServer --sockfile /var/run/vdsm/svdsm.sock</span></font></div><div><font color="#000000"><span style="line-height:14.64px"><br></span></font></div><div><font color="#000000"><span style="line-height:14.64px">Dec 28 11:30:02 hostname systemd[1]: Started Auxiliary vdsm service for running helper functions as root.</span></font></div><div><font color="#000000"><span style="line-height:14.64px">Dec 28 11:30:02 hostname systemd[1]: Starting Auxiliary vdsm service for running helper functions as root.</span></font></div></div></div></div></div></div></div></blockquote><div><div><div><div style="color:rgb(0,0,0);line-height:14.64px"><div style="line-height:14.64px"><div><br></div></div></div></div></div><div style="color:rgb(0,0,0);font-family:'Courier New';font-size:12px;line-height:14.64px"></div></div></div>
</blockquote></div><br></div></div></div>