Unable to add secondary hosted engine hosts with ovirt3.5.5

</o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>I= have ovirt 3.5.5 with a hosted engine setup with the host node and engine = node running CentOS 7.1 with a shared NFS storage domain.<o:p></o:p></p><p = class=3DMsoNormal>I initially attempted to 2 additional hosted engine nodes= using CentOS 7.1 with the ‘hosted-engine –deploy’ comman= d.<o:p></o:p></p><p class=3DMsoNormal>Both nodes failed to add when the ans= wer file was unable to be retrieved from the first host.<o:p></o:p></p><p c= lass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span style=3D'f= ont-size:8.0pt;font-family:"Courier New"'>[root@bml-24 ~]# hosted-engine --= deploy<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8= .0pt;font-family:"Courier New"'>[ INFO ] Stage: Initializing<o:p></o:=
</p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Couri= er New"'> Continuing = will configure this host for serving as hypervisor and create a VM where yo= u have to install oVirt Engine afterwards.<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'> &= nbsp; Are you sure you want to co= ntinue? (Yes, No)[Yes]:<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-size:8.0pt;font-family:"Courier New"'> &n= bsp; Configuration files: []<o:p></o:p></span></p><=
<span style=3D'font-size:8.0pt;font-family:"Courier New"'><o:p> </o:p= </span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family= :"Courier New"'> --= =3D=3D STORAGE CONFIGURATION =3D=3D--<o:p></o:p></span></p><p class=3DMsoNo= rmal><span style=3D'font-size:8.0pt;font-family:"Courier New"'><o:p> <= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-fa= mily:"Courier New"'> = During customization use CTRL-D to abort.<o:p></o:p></span></p><p class=3DM= soNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'> &n= bsp; Please specify the storage y= ou would like to use (iscsi, nfs3, nfs4)[nfs3]:<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>&n= bsp; Please specify the ful= l shared storage connection path to use (example: host:/path): cantap01b.ci= ena.com:/vol/sta_fast/ovirt35/hosted_engine<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>&nbs=
[<span style=3D'background:yellow;mso-highlight:yellow'>WARNING] A configu= ration file must be supplied to deploy Hosted Engine on an additional host.= <o:p></o:p></span></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'>&nb= sp; The answer file may be = fetched from the first host using scp.<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-size:8.0pt;font-family:"Courier New";background:y= ellow;mso-highlight:yellow'>  = ; If you do not want to download it automatically you can abort the s= etup answering no to the following question.<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New";backg= round:yellow;mso-highlight:yellow'> &nbs=
Please provide the = FQDN or IP of the first host: ovirt.ca.stalab.ciena.com<o:p></o:p></span></=
</o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-= family:"Courier New"'>[ INFO ] Stage: Termination<o:p></o:p></span></=
<p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New"'>[root@bml-24 ~]# cat /etc/yu= m.repos.d/ovirt-3.5.repo<o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-size:8.0pt;font-family:"Courier New"'>[ovirt-3.5]<o:p></o:p></s=
</span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family= :"Courier New"'>#mirrorlist=3Dhttp://resources.ovirt.org/pub/yum-repo/mirro= rlist-ovirt-3.5-el$releasever<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-size:8.0pt;font-family:"Courier New"'>enabled=3D1<o:p></o:=
</o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-= family:"Courier New"'>gpgcheck=3D1<o:p></o:p></span></p><p class=3DMsoNorma= l><span style=3D'font-size:8.0pt;font-family:"Courier New"'>gpgkey=3Dhttps:= //dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7<o:p></o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'= <o:p> </o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New"'>[ovirt-3.5-glusterfs-epel]<o:p></o:p></spa= n></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Cour= ier New"'>name=3DGlusterFS is a clustered file-system capable of scaling to= several petabytes.<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New"'>baseurl=3Dhttp://download.gl= uster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$releasever/$basearch= /<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;= font-family:"Courier New"'>enabled=3D1<o:p></o:p></span></p><p class=3DMsoN= ormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>skip_if_una= vailable=3D1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New"'>gpgcheck=3D1<o:p></o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'= gpgkey=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.rep= o/pub.key<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-siz= e:8.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p class=3DM= soNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>[ovirt-3= .5-glusterfs-noarch-epel]<o:p></o:p></span></p><p class=3DMsoNormal><span s= tyle=3D'font-size:8.0pt;font-family:"Courier New"'>name=3DGlusterFS is a cl= ustered file-system capable of scaling to several petabytes.<o:p></o:p></sp= an></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Cou= rier New"'>baseurl=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATE= ST/EPEL.repo/epel-$releasever/noarch<o:p></o:p></span></p><p class=3DMsoNor= mal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>enabled=3D1<o= :p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;fon= t-family:"Courier New"'>skip_if_unavailable=3D1<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>gp= gcheck=3D1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-si= ze:8.0pt;font-family:"Courier New"'>gpgkey=3Dhttp://download.gluster.org/pu= b/gluster/glusterfs/LATEST/EPEL.repo/pub.key<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'><o:p= </o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt= ;font-family:"Courier New"'>[ovirt-3.5-patternfly1-noarch-epel]<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"= Courier New"'>name=3DCopr repo for patternfly1 owned by patternfly<o:p></o:=
gpgcheck=3D0<o:p></o:p></span></p><p class=3DMsoNormal><o:p> </o:p></=
</span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family= :"Courier New"'>2015-10-30 11:40:29 DEBUG otopi.context context._executeMet= hod:138 Stage customization METHOD otopi.plugins.ovirt_hosted_engine_setup.= core.remote_answerfile.Plugin._customization<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>2015= -10-30 11:40:29 WARNING otopi.plugins.ovirt_hosted_engine_setup.core.remote= _answerfile remote_answerfile._customization:244 A configuration file must = be supplied to deploy Hosted Engine on an additional h<o:p></o:p></span></p= <p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier N= ew"'>ost.<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-siz= e:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:29 DEBUG otopi.plugins.= otopi.dialog.human human.queryString:153 query OVEHOSTED_CORE_FETCH_ANSWER<= o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;fo= nt-family:"Courier New"'>2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dial= og.human dialog.__logString:215 DIALOG:SEND &n= bsp; The answer= file may be fetched from the first host using scp.<o:p></o:p></span></p><p= class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"= '>2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logSt= ring:215 DIALOG:SEND &= nbsp; If you do not want to download it= automatically you can abort the setup answering no to the follow<o:p></o:p= </span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family= :"Courier New"'>ing question.<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:29 = DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND&n= bsp;  = ; Do you want to scp the answer file from the first host?= (Yes, No)[Yes]:<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:43 DEBUG otopi.p= lugins.otopi.dialog.human human.queryString:153 query OVEHOSTED_NET_FIRST_H= OST_FQDN<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size= :8.0pt;font-family:"Courier New"'>2015-10-30 11:40:43 DEBUG otopi.plugins.o= topi.dialog.human dialog.__logString:215 DIALOG:SEND  = ; P= lease provide the FQDN or IP of the first host:<o:p></o:p></span></p><p cla= ss=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>20= 15-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString= :215 DIALOG:RECEIVE ovirt.ca.stalab.ciena.com<o:p></o:p><= /span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"= Courier New"'>2015-10-30 11:40:55 DEBUG otopi.plugins.ovirt_hosted_engine_s= etup.core.remote_answerfile remote_answerfile._fetch_answer_file:106 _fetch= _answer_file<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:55 DEBUG otopi.plugi= ns.otopi.dialog.human human.queryString:153 query HOST_FIRST_HOST_ROOT_PASS= WORD<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0=
self._tmp_ans<o:p></o:p></span></p><p class=3DMsoNormal= <span style=3D'font-size:8.0pt;font-family:"Courier New";background:yellow= ;mso-highlight:yellow'> File "/usr/lib/python2.7/site-packages/p= aramiko/sftp_client.py", line 718, in get<o:p></o:p></span></p><p clas= s=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New";back= ground:yellow;mso-highlight:yellow'> file_size =3D self.s= tat(remotepath).st_size<o:p></o:p></span></p><p class=3DMsoNormal><span sty= le=3D'font-size:8.0pt;font-family:"Courier New";background:yellow;mso-highl= ight:yellow'> File "/usr/lib/python2.7/site-packages/paramiko/sf= tp_client.py", line 413, in stat<o:p></o:p></span></p><p class=3DMsoNo= rmal><span style=3D'font-size:8.0pt;font-family:"Courier New";background:ye= llow;mso-highlight:yellow'> t, msg =3D self._request(CMD_= STAT, path)<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-s= ize:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'= File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py= ", line 729, in _request<o:p></o:p></span></p><p class=3DMsoNormal><sp= an style=3D'font-size:8.0pt;font-family:"Courier New";background:yellow;mso= -highlight:yellow'> return self._read_response(num)<o:p><= /o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-fa= mily:"Courier New";background:yellow;mso-highlight:yellow'> File &quo= t;/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 776,= in _read_response<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D= 'font-size:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:= yellow'> self._convert_status(msg)<o:p></o:p></span></p><=
</span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family= :"Courier New"'>2015-10-30 11:40:59 INFO otopi.context context.runSequence:= 417 Stage: Clean up<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG ot= opi.context context.runSequence:421 STAGE cleanup<o:p></o:p></span></p><p c= lass=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>= 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile= .Plugin._cleanup<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG otopi.c= ontext context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_= hosted_engine_setup.engine.add_host.Plugin._cleanup<o:p></o:p></span></p><p= class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"= '>2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage = cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._= cleanup<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG otopi.context co= ntext._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_en= gine_setup.storage.storage.Plugin._cleanup<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10= -30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage = storage._spmStop:692 spmStop<o:p></o:p></span></p><p class=3DMsoNormal><spa= n style=3D'font-size:8.0pt;font-family:"Courier New";background:yellow;mso-= highlight:yellow'>2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engi= ne_setup.storage.storage storage._cleanup:1055 Not SPM?<o:p></o:p></span></=
--_000_B6D287BF58D35D4B882E012AD3E175619A7F531FONWVEXCHMB04cie_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Ovirt users, I have ovirt 3.5.5 with a hosted engine setup with the host node and engine= node running CentOS 7.1 with a shared NFS storage domain. I initially attempted to 2 additional hosted engine nodes using CentOS 7.1 = with the 'hosted-engine -deploy' command. Both nodes failed to add when the answer file was unable to be retrieved fr= om the first host. [root@bml-24 ~]# hosted-engine --deploy [ INFO ] Stage: Initializing [ INFO ] Generating a temporary VNC password. [ INFO ] Stage: Environment setup Continuing will configure this host for serving as hypervisor and= create a VM where you have to install oVirt Engine afterwards. Are you sure you want to continue? (Yes, No)[Yes]: Configuration files: [] Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-= setup-20151030113943-c18yxd.log Version: otopi-1.3.2 (otopi-1.3.2-1.el7.centos) [ INFO ] Hardware supports virtualization [ INFO ] Bridge ovirtmgmt already created [ INFO ] Stage: Environment packages setup [ INFO ] Stage: Programs detection [ INFO ] Stage: Environment setup [ INFO ] Stage: Environment customization --=3D=3D STORAGE CONFIGURATION =3D=3D-- During customization use CTRL-D to abort. Please specify the storage you would like to use (iscsi, nfs3, nf= s4)[nfs3]: Please specify the full shared storage connection path to use (ex= ample: host:/path): cantap01b.ciena.com:/vol/sta_fast/ovirt35/hosted_engine The specified storage location already contains a data domain. Is= this an additional host setup (Yes, No)[Yes]? [ INFO ] Installing on additional host Please specify the Host ID [Must be integer, default: 2]: 4 Local storage datacenter name is an internal name and currently w= ill not be shown in engine's admin UI. Please enter local datacenter name [hosted_datacenter]: --=3D=3D SYSTEM CONFIGURATION =3D=3D-- [WARNING] A configuration file must be supplied to deploy Hosted Engine on = an additional host. The answer file may be fetched from the first host using scp. If you do not want to download it automatically you can abort the= setup answering no to the following question. Do you want to scp the answer file from the first host? (Yes, No)= [Yes]: Please provide the FQDN or IP of the first host: ovirt.ca.stalab.= ciena.com Enter 'root' user password for host ovirt.ca.stalab.ciena.com: [ ERROR ] Failed to execute stage 'Environment customization': [Errno 2] No= such file [ INFO ] Stage: Clean up [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answer= s/answers-20151030114103.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination I was able to add the additional hosts using the ovirt GUI and am able to u= se them for running VMs. I tried running 'hosted-engine -deploy' on each host again and get the same= error as before. Checking the logfile referenced above, I don't see that the download filena= me listed. I do see a couples of stack traces that might be useful. I also found this defect in the release notes for ovirt-3.5.5 which claims = to be resolved. Bug 1271272<https://bugzilla.redhat.com/show_bug.cgi?id=3D1271272> - [HE] F= ailed to deploy additional host using NFS I was careful when installing ovirt 3.5.5 to ensure that I used the baseurl= instead of the mirrors as it looked like the mirrors were behind. [root@bml-24 ~]# cat /etc/yum.repos.d/ovirt-3.5.repo [ovirt-3.5] name=3DLatest oVirt 3.5 Release baseurl=3Dhttp://resources.ovirt.org/pub/ovirt-3.5/rpm/el$releasever/ #mirrorlist=3Dhttp://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-= el$releasever enabled=3D1 skip_if_unavailable=3D1 gpgcheck=3D1 gpgkey=3Dfile:///etc/pki/rpm-gpg/RPM-GPG-ovirt-3.5 I see that there is another repo file for ovirt as well that I didn't modif= y. Could my issue be caused by not picking up the latest versions from these m= irrors? [root@bml-24 ~]# cat /etc/yum.repos.d/ovirt-3.5-dependencies.repo [ovirt-3.5-epel] name=3DExtra Packages for Enterprise Linux 7 - $basearch #baseurl=3Dhttp://download.fedoraproject.org/pub/epel/7/$basearch mirrorlist=3Dhttps://mirrors.fedoraproject.org/metalink?repo=3Depel-7&arch= =3D$basearch failovermethod=3Dpriority enabled=3D1 includepkgs=3Depel-release,python-uinput,puppet,python-lockfile,python-cpop= en,python-ordereddict,python-pthreading,python-inotify,python-argparse,novn= c,python-ply,python-kitchen,python-daemon,python-websockify,livecd-tools,sp= ice-html5,mom,python-IPy,python-ioprocess,ioprocess,python-paramiko,python-= crypto,python-cheetah,python-ecdsa,python-markdown,rubygem-rgen,ovirt-guest= -agent,userspace-rcu,protobuf-java gpgcheck=3D1 gpgkey=3Dhttps://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7 [ovirt-3.5-glusterfs-epel] name=3DGlusterFS is a clustered file-system capable of scaling to several p= etabytes. baseurl=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.rep= o/epel-$releasever/$basearch/ enabled=3D1 skip_if_unavailable=3D1 gpgcheck=3D1 gpgkey=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo= /pub.key [ovirt-3.5-glusterfs-noarch-epel] name=3DGlusterFS is a clustered file-system capable of scaling to several p= etabytes. baseurl=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.rep= o/epel-$releasever/noarch enabled=3D1 skip_if_unavailable=3D1 gpgcheck=3D1 gpgkey=3Dhttp://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo= /pub.key [ovirt-3.5-patternfly1-noarch-epel] name=3DCopr repo for patternfly1 owned by patternfly baseurl=3Dhttp://copr-be.cloud.fedoraproject.org/results/patternfly/pattern= fly1/epel-7-$basearch/ enabled=3D1 skip_if_unavailable=3D1 gpgcheck=3D0 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND --=3D=3D SYSTEM CONFIGURATION =3D=3D-- 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND 2015-10-30 11:40:29 DEBUG otopi.context context._executeMethod:138 Stage cu= stomization METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answ= erfile.Plugin._customization 2015-10-30 11:40:29 WARNING otopi.plugins.ovirt_hosted_engine_setup.core.re= mote_answerfile remote_answerfile._customization:244 A configuration file m= ust be supplied to deploy Hosted Engine on an additional h ost. 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human human.queryStrin= g:153 query OVEHOSTED_CORE_FETCH_ANSWER 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND The answer file may be fetched from the = first host using scp. 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND If you do not want to download it automa= tically you can abort the setup answering no to the follow ing question. 2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND Do you want to scp the answer file from = the first host? (Yes, No)[Yes]: 2015-10-30 11:40:43 DEBUG otopi.plugins.otopi.dialog.human human.queryStrin= g:153 query OVEHOSTED_NET_FIRST_HOST_FQDN 2015-10-30 11:40:43 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND Please provide the FQDN or IP of the fir= st host: 2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:RECEIVE ovirt.ca.stalab.ciena.com 2015-10-30 11:40:55 DEBUG otopi.plugins.ovirt_hosted_engine_setup.core.remo= te_answerfile remote_answerfile._fetch_answer_file:106 _fetch_answer_file 2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human human.queryStrin= g:153 query HOST_FIRST_HOST_ROOT_PASSWORD 2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logStri= ng:215 DIALOG:SEND Enter 'root' user password for host ovir= t.ca.stalab.ciena.com: 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:152 method e= xception Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _e= xecuteMethod method['method']() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hoste= d-engine-setup/core/remote_answerfile.py", line 304, in _customization self._fetch_answer_file() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hoste= d-engine-setup/core/remote_answerfile.py", line 143, in _fetch_answer_file self._tmp_ans File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 718= , in get file_size =3D self.stat(remotepath).st_size File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 413= , in stat t, msg =3D self._request(CMD_STAT, path) File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 729= , in _request return self._read_response(num) File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 776= , in _read_response self._convert_status(msg) File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 802= , in _convert_status raise IOError(errno.ENOENT, text) IOError: [Errno 2] No such file 2015-10-30 11:40:59 ERROR otopi.context context._executeMethod:161 Failed t= o execute stage 'Environment customization': [Errno 2] No such file 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:490 ENVIRON= MENT DUMP - BEGIN 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV BAS= E/error=3Dbool:'True' 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV BAS= E/exceptionInfo=3Dlist:'[(<type 'exceptions.IOError'>, IOError(2, u'No such= file'), <traceback object at 0x3daa320>)]' 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVE= HOSTED_FIRST_HOST/fetchAnswer=3Dbool:'True' 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVE= HOSTED_FIRST_HOST/fqdn=3Dstr:'ovirt.ca.stalab.ciena.com' 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVE= HOSTED_FIRST_HOST/rootPassword=3Dstr:'**FILTERED**' 2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:504 ENVIRON= MENT DUMP - END 2015-10-30 11:40:59 INFO otopi.context context.runSequence:417 Stage: Clean= up 2015-10-30 11:40:59 DEBUG otopi.context context.runSequence:421 STAGE clean= up 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile= .Plugin._cleanup 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.engine.add_host.Plugin= ._cleanup 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._cl= eanup 2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin= ._cleanup 2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.s= torage storage._spmStop:692 spmStop 2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.s= torage storage._cleanup:1055 Not SPM? Traceback (most recent call last): File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hoste= d-engine-setup/storage/storage.py", line 1053, in _cleanup self._spmStop() File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hoste= d-engine-setup/storage/storage.py", line 698, in _spmStop raise RuntimeError(status['status']['message']) RuntimeError: Not SPM 2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.s= torage storage._storagePoolConnection:657 disconnectStoragePool 2015-10-30 11:41:03 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._c= leanup 2015-10-30 11:41:03 DEBUG otopi.context context._executeMethod:138 Stage cl= eanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin= ._save_answers_at_cleanup 2015-10-30 11:41:03 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answe= rfile answerfile._save_answers:52 Generating answer file '/var/lib/ovirt-ho= sted-engine-setup/answers/answers-20151030114103.conf' 2015-10-30 11:41:03 INFO otopi.context context.runSequence:417 Stage: Pre-t= ermination Thanks for your help, Bruce. Bruce T. Jorgens | Technical Lead, System Test Automation bjorgens@ciena.com<mailto:bjorgens@ciena.com> | 115 North Sullivan Road | S= pokane Valley, WA 99037 USA Direct +1.509.242.9292 | Mobile +1.509.499.0215 | Fax +1.509.242.9001 --_000_B6D287BF58D35D4B882E012AD3E175619A7F531FONWVEXCHMB04cie_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV=3D"Content-Type" CONTENT= =3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros= oft Word 14 (filtered medium)"><style><!-- /* Font Definitions */ @font-face {font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4;} @font-face {font-family:Calibri; panose-1:2 15 5 2 2 2 4 3 2 4;} /* Style Definitions */ p.MsoNormal, li.MsoNormal, div.MsoNormal {margin:0in; margin-bottom:.0001pt; font-size:11.0pt; font-family:"Calibri","sans-serif";} a:link, span.MsoHyperlink {mso-style-priority:99; color:blue; text-decoration:underline;} a:visited, span.MsoHyperlinkFollowed {mso-style-priority:99; color:purple; text-decoration:underline;} span.EmailStyle17 {mso-style-type:personal-compose; font-family:"Calibri","sans-serif"; color:windowtext;} .MsoChpDefault {mso-style-type:export-only; font-family:"Calibri","sans-serif";} @page WordSection1 {size:8.5in 11.0in; margin:1.0in 1.0in 1.0in 1.0in;} div.WordSection1 {page:WordSection1;} --></style><!--[if gte mso 9]><xml> <o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" /> </xml><![endif]--><!--[if gte mso 9]><xml> <o:shapelayout v:ext=3D"edit"> <o:idmap v:ext=3D"edit" data=3D"1" /> </o:shapelayout></xml><![endif]--></head><body lang=3DEN-US link=3Dblue vli= nk=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal>Ovirt users,<o:p= p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-famil= y:"Courier New"'>[ INFO ] Generating a temporary VNC password.<o:p></= o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-fam= ily:"Courier New"'>[ INFO ] Stage: Environment setup<o:p></o:p></span= p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New= "'> Log file: /var/lo= g/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20151030113943-c18yxd= .log<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0= pt;font-family:"Courier New"'> &nb= sp; Version: otopi-1.3.2 (otopi-1.3.2-1.el7.centos)<o:p></o:p></span>= </p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courie= r New"'>[ INFO ] Hardware supports virtualization<o:p></o:p></span></= p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier = New"'>[ INFO ] Bridge ovirtmgmt already created<o:p></o:p></span></p>= <p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier Ne= w"'>[ INFO ] Stage: Environment packages setup<o:p></o:p></span></p><= p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New= "'>[ INFO ] Stage: Programs detection<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>[ IN= FO ] Stage: Environment setup<o:p></o:p></span></p><p class=3DMsoNorm= al><span style=3D'font-size:8.0pt;font-family:"Courier New"'>[ INFO ]= Stage: Environment customization<o:p></o:p></span></p><p class=3DMsoNormal= p; The specified storage lo= cation already contains a data domain. Is this an additional host setup (Ye= s, No)[Yes]?<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New"'>[ INFO ] Installing on addition= al host<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New"'> = Please specify the Host ID [Must be integer, default: 2]: 4<o:= p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font= -family:"Courier New"'> &nbs= p; Local storage datacenter name is an internal name and currently will not= be shown in engine's admin UI.<o:p></o:p></span></p><p class=3DMsoNormal><= span style=3D'font-size:8.0pt;font-family:"Courier New"'> = Please enter local datacenter name [ho= sted_datacenter]:<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'= font-size:8.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p c= lass=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>= --=3D=3D SYSTEM CONF= IGURATION =3D=3D--<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D= 'font-size:8.0pt;font-family:"Courier New"'><o:p> </o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'= p; Do you want to scp the answer file from the first host? (Yes= , No)[Yes]:<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-s= ize:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'= p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier = New";background:yellow;mso-highlight:yellow'> = Enter 'root' user password for host ovirt.ca.stala= b.ciena.com:<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow= '>[ ERROR ] Failed to execute stage 'Environment customization': [Errno 2] = No such file<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow= '>[ INFO ] Stage: Clean up</span><span style=3D'font-size:8.0pt;font-= family:"Courier New"'><o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'font-size:8.0pt;font-family:"Courier New"'>[ INFO ] Generating a= nswer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-201510301141= 03.conf'<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size= :8.0pt;font-family:"Courier New"'>[ INFO ] Stage: Pre-termination<o:p= p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>I was able= to add the additional hosts using the ovirt GUI and am able to use them fo= r running VMs.<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p c= lass=3DMsoNormal>I tried running ‘hosted-engine –deploy’ = on each host again and get the same error as before.<o:p></o:p></p><p class= =3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>Checking the logfile= referenced above, I don’t see that the download filename listed.&nbs= p; <o:p></o:p></p><p class=3DMsoNormal>I do see a couples of stack traces t= hat might be useful.<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></= p><p class=3DMsoNormal>I also found this defect in the release notes for ov= irt-3.5.5 which claims to be resolved.<o:p></o:p></p><p class=3DMsoNormal><= a href=3D"https://bugzilla.redhat.com/show_bug.cgi?id=3D1271272"><b>Bug&nbs= p;1271272</b></a> - [HE] Failed to deploy additional host using NFS<o:p></o= :p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>I was= careful when installing ovirt 3.5.5 to ensure that I used the baseurl inst= ead of the mirrors as it looked like the mirrors were behind.<o:p></o:p></p= pan></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Co= urier New"'>name=3DLatest oVirt 3.5 Release<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>base= url=3Dhttp://resources.ovirt.org/pub/ovirt-3.5/rpm/el$releasever/<o:p></o:p= p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-famil= y:"Courier New"'>skip_if_unavailable=3D1<o:p></o:p></span></p><p class=3DMs= oNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>gpgcheck= =3D1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0= pt;font-family:"Courier New"'>gpgkey=3Dfile:///etc/pki/rpm-gpg/RPM-GPG-ovir= t-3.5<o:p></o:p></span></p><p class=3DMsoNormal><o:p> </o:p></p><p cla= ss=3DMsoNormal>I see that there is another repo file for ovirt as well that= I didn’t modify.<o:p></o:p></p><p class=3DMsoNormal>Could my issue b= e caused by not picking up the latest versions from these mirrors?<o:p></o:= p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal>[<span= style=3D'font-size:8.0pt;font-family:"Courier New"'>root@bml-24 ~]# cat /e= tc/yum.repos.d/ovirt-3.5-dependencies.repo<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>[ovirt-= 3.5-epel]<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-siz= e:8.0pt;font-family:"Courier New"'>name=3DExtra Packages for Enterprise Lin= ux 7 - $basearch<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-size:8.0pt;font-family:"Courier New"'>#baseurl=3Dhttp://download.fedora= project.org/pub/epel/7/$basearch<o:p></o:p></span></p><p class=3DMsoNormal>= <span style=3D'font-size:8.0pt;font-family:"Courier New"'>mirrorlist=3Dhttp= s://mirrors.fedoraproject.org/metalink?repo=3Depel-7&arch=3D$basearch<o= :p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;fon= t-family:"Courier New"'>failovermethod=3Dpriority<o:p></o:p></span></p><p c= lass=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>= enabled=3D1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-s= ize:8.0pt;font-family:"Courier New"'>includepkgs=3Depel-release,python-uinp= ut,puppet,python-lockfile,python-cpopen,python-ordereddict,python-pthreadin= g,python-inotify,python-argparse,novnc,python-ply,python-kitchen,python-dae= mon,python-websockify,livecd-tools,spice-html5,mom,python-IPy,python-ioproc= ess,ioprocess,python-paramiko,python-crypto,python-cheetah,python-ecdsa,pyt= hon-markdown,rubygem-rgen,ovirt-guest-agent,userspace-rcu,protobuf-java<o:p= p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-famil= y:"Courier New"'>baseurl=3Dhttp://copr-be.cloud.fedoraproject.org/results/p= atternfly/patternfly1/epel-7-$basearch/<o:p></o:p></span></p><p class=3DMso= Normal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>enabled=3D= 1<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;= font-family:"Courier New"'>skip_if_unavailable=3D1<o:p></o:p></span></p><p = class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'= p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal><span styl= e=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:29 DEBUG o= topi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND<o:p></o:= p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-famil= y:"Courier New"'>2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human= dialog.__logString:215 DIALOG:SEND &nbs= p; --=3D=3D SYSTEM CO= NFIGURATION =3D=3D--<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:29 DEBUG ot= opi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND<o:p></o:p= pt;font-family:"Courier New"'>2015-10-30 11:40:55 DEBUG otopi.plugins.otopi= .dialog.human dialog.__logString:215 DIALOG:SEND &nb= sp; Enter= 'root' user password for host ovirt.ca.stalab.ciena.com:<o:p></o:p></span>= </p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courie= r New";background:yellow;mso-highlight:yellow'>2015-10-30 11:40:59 DEBUG ot= opi.context context._executeMethod:152 method exception<o:p></o:p></span></= p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier = New";background:yellow;mso-highlight:yellow'>Traceback (most recent call la= st):<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0= pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'> = File "/usr/lib/python2.7/site-packages/otopi/context.py", line 1= 42, in _executeMethod<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New";background:yellow;mso-highlig= ht:yellow'> method['method']()<o:p></o:p></span></p><p cl= ass=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New";ba= ckground:yellow;mso-highlight:yellow'> File "/usr/share/ovirt-ho= sted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/core/remote_= answerfile.py", line 304, in _customization<o:p></o:p></span></p><p cl= ass=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New";ba= ckground:yellow;mso-highlight:yellow'> self._fetch_answer= _file()<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'>&nb= sp; File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovir= t-hosted-engine-setup/core/remote_answerfile.py", line 143, in _fetch_= answer_file<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-s= ize:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'= p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New= ";background:yellow;mso-highlight:yellow'> File "/usr/lib/python= 2.7/site-packages/paramiko/sftp_client.py", line 802, in _convert_stat= us<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt= ;font-family:"Courier New";background:yellow;mso-highlight:yellow'> &n= bsp; raise IOError(errno.ENOENT, text)<o:p></o:p></span></p><p class= =3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New";backg= round:yellow;mso-highlight:yellow'>IOError: [Errno 2] No such file</span><s= pan style=3D'font-size:8.0pt;font-family:"Courier New"'><o:p></o:p></span><= /p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier= New"'>2015-10-30 11:40:59 ERROR otopi.context context._executeMethod:161 F= ailed to execute stage 'Environment customization': [Errno 2] No such file<= o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;fo= nt-family:"Courier New"'>2015-10-30 11:40:59 DEBUG otopi.context context.du= mpEnvironment:490 ENVIRONMENT DUMP - BEGIN<o:p></o:p></span></p><p class=3D= MsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10= -30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error= =3Dbool:'True'<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'fon= t-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG otopi.con= text context.dumpEnvironment:500 ENV BASE/exceptionInfo=3Dlist:'[(<type = 'exceptions.IOError'>, IOError(2, u'No such file'), <traceback object= at 0x3daa320>)]'<o:p></o:p></span></p><p class=3DMsoNormal><span style= =3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG ot= opi.context context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/fetchAnswe= r=3Dbool:'True'<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'fo= nt-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG otopi.co= ntext context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/fqdn=3Dstr:'ovir= t.ca.stalab.ciena.com'<o:p></o:p></span></p><p class=3DMsoNormal><span styl= e=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG o= topi.context context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/rootPassw= ord=3Dstr:'**FILTERED**'<o:p></o:p></span></p><p class=3DMsoNormal><span st= yle=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:40:59 DEBUG= otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END<o:p></o:p= p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Courier = New";background:yellow;mso-highlight:yellow'>Traceback (most recent call la= st):<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0= pt;font-family:"Courier New";background:yellow;mso-highlight:yellow'> = File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-h= osted-engine-setup/storage/storage.py", line 1053, in _cleanup<o:p></o= :p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-fami= ly:"Courier New";background:yellow;mso-highlight:yellow'> = self._spmStop()<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'f= ont-size:8.0pt;font-family:"Courier New";background:yellow;mso-highlight:ye= llow'> File "/usr/share/ovirt-hosted-engine-setup/scripts/../plu= gins/ovirt-hosted-engine-setup/storage/storage.py", line 698, in _spmS= top<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0p= t;font-family:"Courier New";background:yellow;mso-highlight:yellow'> &= nbsp; raise RuntimeError(status['status']['message'])<o:p></o:p></spa= n></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Cour= ier New";background:yellow;mso-highlight:yellow'>RuntimeError: Not SPM</spa= n><span style=3D'font-size:8.0pt;font-family:"Courier New"'><o:p></o:p></sp= an></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;font-family:"Cou= rier New"'>2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setu= p.storage.storage storage._storagePoolConnection:657 disconnectStoragePool<= o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:8.0pt;fo= nt-family:"Courier New"'>2015-10-30 11:41:03 DEBUG otopi.context context._e= xecuteMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_set= up.vm.boot_disk.Plugin._cleanup<o:p></o:p></span></p><p class=3DMsoNormal><= span style=3D'font-size:8.0pt;font-family:"Courier New"'>2015-10-30 11:41:0= 3 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi= .plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin._save_answers_at_= cleanup<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-size:= 8.0pt;font-family:"Courier New"'>2015-10-30 11:41:03 INFO otopi.plugins.ovi= rt_hosted_engine_setup.core.answerfile answerfile._save_answers:52 Generati= ng answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151030= 114103.conf'<o:p></o:p></span></p><p class=3DMsoNormal><span style=3D'font-= size:8.0pt;font-family:"Courier New"'>2015-10-30 11:41:03 INFO otopi.contex= t context.runSequence:417 Stage: Pre-termination<o:p></o:p></span></p><p cl= ass=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal> Thanks for your = help,<o:p></o:p></p><p class=3DMsoNormal>Bruce.<o:p></o:p></p><p class=3DMs= oNormal><o:p> </o:p></p><p class=3DMsoNormal><b><span style=3D'font-si= ze:7.5pt;font-family:"Arial","sans-serif"'>Bruce T. Jorgens</span></b><span= style=3D'font-size:7.5pt;font-family:"Arial","sans-serif"'> <span sty= le=3D'color:red'>|</span> <span style=3D'color:gray'>Technical Lead, System= Test Automation</span><br><u><span style=3D'color:blue'><a href=3D"mailto:= bjorgens@ciena.com"><span style=3D'color:blue'>bjorgens@ciena.com</span></a=
</span></u> <span style=3D'color:red'>|</span><span style=3D'color:purple'= </span><span style=3D'color:gray'>115 North Sullivan Road </span><span st= yle=3D'color:red'>|</span><span style=3D'color:purple'> </span><span s= tyle=3D'color:gray'>Spokane Valley, WA 99037 USA<br>Direct +1.509.242.9292 = </span><span style=3D'color:red'>|</span><span style=3D'color:gray'> Mobile= +1.509.499.0215</span> <span style=3D'color:red'>|</span><span style=3D'co= lor:purple'> </span><span style=3D'color:gray'>Fax +1.509.242.9001</span></= span><o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p></div></body>= </html>=
--_000_B6D287BF58D35D4B882E012AD3E175619A7F531FONWVEXCHMB04cie_--

On Fri, Oct 30, 2015 at 8:13 PM, Jorgens, Bruce <bjorgens@ciena.com> wrote:
Ovirt users,
I have ovirt 3.5.5 with a hosted engine setup with the host node and engine node running CentOS 7.1 with a shared NFS storage domain.
I initially attempted to 2 additional hosted engine nodes using CentOS 7.1 with the ‘hosted-engine –deploy’ command.
Both nodes failed to add when the answer file was unable to be retrieved from the first host.
[root@bml-24 ~]# hosted-engine --deploy
[ INFO ] Stage: Initializing
[ INFO ] Generating a temporary VNC password.
[ INFO ] Stage: Environment setup
Continuing will configure this host for serving as hypervisor and create a VM where you have to install oVirt Engine afterwards.
Are you sure you want to continue? (Yes, No)[Yes]:
Configuration files: []
Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20151030113943-c18yxd.log
Version: otopi-1.3.2 (otopi-1.3.2-1.el7.centos)
[ INFO ] Hardware supports virtualization
[ INFO ] Bridge ovirtmgmt already created
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup
[ INFO ] Stage: Environment customization
--== STORAGE CONFIGURATION ==--
During customization use CTRL-D to abort.
Please specify the storage you would like to use (iscsi, nfs3, nfs4)[nfs3]:
Please specify the full shared storage connection path to use (example: host:/path): cantap01b.ciena.com: /vol/sta_fast/ovirt35/hosted_engine
The specified storage location already contains a data domain. Is this an additional host setup (Yes, No)[Yes]?
[ INFO ] Installing on additional host
Please specify the Host ID [Must be integer, default: 2]: 4
Local storage datacenter name is an internal name and currently will not be shown in engine's admin UI.
Please enter local datacenter name [hosted_datacenter]:
--== SYSTEM CONFIGURATION ==--
[WARNING] A configuration file must be supplied to deploy Hosted Engine on an additional host.
The answer file may be fetched from the first host using scp.
If you do not want to download it automatically you can abort the setup answering no to the following question.
Do you want to scp the answer file from the first host? (Yes, No)[Yes]:
Please provide the FQDN or IP of the first host: ovirt.ca.stalab.ciena.com
Enter 'root' user password for host ovirt.ca.stalab.ciena.com:
[ ERROR ] Failed to execute stage 'Environment customization': [Errno 2] No such file
[ INFO ] Stage: Clean up
[ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151030114103.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
I was able to add the additional hosts using the ovirt GUI and am able to use them for running VMs.
I tried running ‘hosted-engine –deploy’ on each host again and get the same error as before.
Could you check if you see the file /etc/ovirt-hosted-engine/answers.conf on the host where you want to copy the answer file from? If not, most likely something went wrong when you set up the first node. You will find your ansers file then in this /var/lib/ovirt-hosted-engine-setup/answers/ folder. You might find some errors in /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine*.log on the first node. Could you provide the log from this node? After copying the ansers file from /var/lib/ovirt-hosted-engine-setup/answers/ over to /etc/ovirt-hosted-engine/answers.conf you should no longer see this error on the other hosts.
Checking the logfile referenced above, I don’t see that the download filename listed.
I do see a couples of stack traces that might be useful.
I also found this defect in the release notes for ovirt-3.5.5 which claims to be resolved.
*Bug 1271272* <https://bugzilla.redhat.com/show_bug.cgi?id=1271272> - [HE] Failed to deploy additional host using NFS
You should only be affected by that bug if you see something like No such file or directory: '/rhev/data-center/mnt/[...]/ha_agent/hosted-engine.metadata. This is related to accessing metadata of other nodes through the storage mounted by vdsm.
I was careful when installing ovirt 3.5.5 to ensure that I used the baseurl instead of the mirrors as it looked like the mirrors were behind.
[root@bml-24 ~]# cat /etc/yum.repos.d/ovirt-3.5.repo
[ovirt-3.5]
name=Latest oVirt 3.5 Release
baseurl=http://resources.ovirt.org/pub/ovirt-3.5/rpm/el$releasever/
#mirrorlist= http://resources.ovirt.org/pub/yum-repo/mirrorlist-ovirt-3.5-el$releasever
enabled=1
skip_if_unavailable=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-ovirt-3.5
I see that there is another repo file for ovirt as well that I didn’t modify.
Could my issue be caused by not picking up the latest versions from these mirrors?
[root@bml-24 ~]# cat /etc/yum.repos.d/ovirt-3.5-dependencies.repo
[ovirt-3.5-epel]
name=Extra Packages for Enterprise Linux 7 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/7/$basearch
mirrorlist= https://mirrors.fedoraproject.org/metalink?repo=epel-7&arch=$basearch
failovermethod=priority
enabled=1
includepkgs=epel-release,python-uinput,puppet,python-lockfile,python-cpopen,python-ordereddict,python-pthreading,python-inotify,python-argparse,novnc,python-ply,python-kitchen,python-daemon,python-websockify,livecd-tools,spice-html5,mom,python-IPy,python-ioprocess,ioprocess,python-paramiko,python-crypto,python-cheetah,python-ecdsa,python-markdown,rubygem-rgen,ovirt-guest-agent,userspace-rcu,protobuf-java
gpgcheck=1
gpgkey=https://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-7
[ovirt-3.5-glusterfs-epel]
name=GlusterFS is a clustered file-system capable of scaling to several petabytes.
baseurl= http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$rel...
enabled=1
skip_if_unavailable=1
gpgcheck=1
gpgkey= http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key
[ovirt-3.5-glusterfs-noarch-epel]
name=GlusterFS is a clustered file-system capable of scaling to several petabytes.
baseurl= http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/epel-$rel...
enabled=1
skip_if_unavailable=1
gpgcheck=1
gpgkey= http://download.gluster.org/pub/gluster/glusterfs/LATEST/EPEL.repo/pub.key
[ovirt-3.5-patternfly1-noarch-epel]
name=Copr repo for patternfly1 owned by patternfly
baseurl= http://copr-be.cloud.fedoraproject.org/results/patternfly/patternfly1/epel-7...
enabled=1
skip_if_unavailable=1
gpgcheck=0
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND --== SYSTEM CONFIGURATION ==--
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND
2015-10-30 11:40:29 DEBUG otopi.context context._executeMethod:138 Stage customization METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._customization
2015-10-30 11:40:29 WARNING otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile remote_answerfile._customization:244 A configuration file must be supplied to deploy Hosted Engine on an additional h
ost.
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human human.queryString:153 query OVEHOSTED_CORE_FETCH_ANSWER
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND The answer file may be fetched from the first host using scp.
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND If you do not want to download it automatically you can abort the setup answering no to the follow
ing question.
2015-10-30 11:40:29 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND Do you want to scp the answer file from the first host? (Yes, No)[Yes]:
2015-10-30 11:40:43 DEBUG otopi.plugins.otopi.dialog.human human.queryString:153 query OVEHOSTED_NET_FIRST_HOST_FQDN
2015-10-30 11:40:43 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND Please provide the FQDN or IP of the first host:
2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:RECEIVE ovirt.ca.stalab.ciena.com
2015-10-30 11:40:55 DEBUG otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile remote_answerfile._fetch_answer_file:106 _fetch_answer_file
2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human human.queryString:153 query HOST_FIRST_HOST_ROOT_PASSWORD
2015-10-30 11:40:55 DEBUG otopi.plugins.otopi.dialog.human dialog.__logString:215 DIALOG:SEND Enter 'root' user password for host ovirt.ca.stalab.ciena.com:
2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:152 method exception
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/otopi/context.py", line 142, in _executeMethod
method['method']()
File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/core/remote_answerfile.py", line 304, in _customization
self._fetch_answer_file()
File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/core/remote_answerfile.py", line 143, in _fetch_answer_file
self._tmp_ans
File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 718, in get
file_size = self.stat(remotepath).st_size
File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 413, in stat
t, msg = self._request(CMD_STAT, path)
File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 729, in _request
return self._read_response(num)
File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 776, in _read_response
self._convert_status(msg)
File "/usr/lib/python2.7/site-packages/paramiko/sftp_client.py", line 802, in _convert_status
raise IOError(errno.ENOENT, text)
IOError: [Errno 2] No such file
2015-10-30 11:40:59 ERROR otopi.context context._executeMethod:161 Failed to execute stage 'Environment customization': [Errno 2] No such file
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:490 ENVIRONMENT DUMP - BEGIN
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/error=bool:'True'
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV BASE/exceptionInfo=list:'[(<type 'exceptions.IOError'>, IOError(2, u'No such file'), <traceback object at 0x3daa320>)]'
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/fetchAnswer=bool:'True'
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/fqdn=str:'ovirt.ca.stalab.ciena.com'
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:500 ENV OVEHOSTED_FIRST_HOST/rootPassword=str:'**FILTERED**'
2015-10-30 11:40:59 DEBUG otopi.context context.dumpEnvironment:504 ENVIRONMENT DUMP - END
2015-10-30 11:40:59 INFO otopi.context context.runSequence:417 Stage: Clean up
2015-10-30 11:40:59 DEBUG otopi.context context.runSequence:421 STAGE cleanup
2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.remote_answerfile.Plugin._cleanup
2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.engine.add_host.Plugin._cleanup
2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.pki.vdsmpki.Plugin._cleanup
2015-10-30 11:40:59 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.storage.storage.Plugin._cleanup
2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._spmStop:692 spmStop
2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._cleanup:1055 Not SPM?
Traceback (most recent call last):
File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 1053, in _cleanup
self._spmStop()
File "/usr/share/ovirt-hosted-engine-setup/scripts/../plugins/ovirt-hosted-engine-setup/storage/storage.py", line 698, in _spmStop
raise RuntimeError(status['status']['message'])
RuntimeError: Not SPM
2015-10-30 11:40:59 DEBUG otopi.plugins.ovirt_hosted_engine_setup.storage.storage storage._storagePoolConnection:657 disconnectStoragePool
2015-10-30 11:41:03 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.vm.boot_disk.Plugin._cleanup
2015-10-30 11:41:03 DEBUG otopi.context context._executeMethod:138 Stage cleanup METHOD otopi.plugins.ovirt_hosted_engine_setup.core.answerfile.Plugin._save_answers_at_cleanup
2015-10-30 11:41:03 INFO otopi.plugins.ovirt_hosted_engine_setup.core.answerfile answerfile._save_answers:52 Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20151030114103.conf'
2015-10-30 11:41:03 INFO otopi.context context.runSequence:417 Stage: Pre-termination
Thanks for your help,
Bruce.
*Bruce T. Jorgens* | Technical Lead, System Test Automation *bjorgens@ciena.com <bjorgens@ciena.com>* | 115 North Sullivan Road | Spokane Valley, WA 99037 USA Direct +1.509.242.9292 | Mobile +1.509.499.0215 | Fax +1.509.242.9001
_______________________________________________ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Best Regards, Roman
participants (2)
-
Jorgens, Bruce
-
Roman Mohr