<!DOCTYPE html>
<html>
<head>
<title></title>
</head>
<body><div>On Wed, Nov 25, 2015, at 12:10, Simone Tiraboschi wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div> </div>
<div><div> </div>
<div><div>On Mon, Nov 23, 2015 at 10:17 PM, Giuseppe Ragusa <span dir="ltr"><<a href="mailto:giuseppe.ragusa@hotmail.com">giuseppe.ragusa@hotmail.com</a>></span> wrote:<br></div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex;"><div><div><div>On Tue, Oct 27, 2015, at 00:10, Giuseppe Ragusa wrote:<br></div>
<div>
> On Mon, Oct 26, 2015, at 09:48, Simone Tiraboschi wrote:<br></div>
<div>
> ><br></div>
<div>
> ><br></div>
<div>
> > On Mon, Oct 26, 2015 at 12:14 AM, Giuseppe Ragusa <<a href="mailto:giuseppe.ragusa@hotmail.com">giuseppe.ragusa@hotmail.com</a>> wrote:<br></div>
<div>
> >> Hi all,<br></div>
<div>
> >> I'm experiencing some difficulties using oVirt 3.6 latest snapshot.<br></div>
<div>
> >><br></div>
<div>
> >> I'm trying to trick the self-hosted-engine setup to create a custom engine vm with 3 nics (with fixed MACs/UUIDs).<br></div>
<div>
> >><br></div>
<div>
> >> The GlusterFS volume (3.7.5 hyperconverged, replica 3, for the engine vm) and the network bridges (ovirtmgmt and other two bridges, called nfs and lan, for the engine vm) have been preconfigured on the initial fully-patched CentOS 7.1 host (plus other two identical hosts which are awaiting to be added).<br></div>
<div>
> >><br></div>
<div>
> >> I'm stuck at a point with the engine vm successfully starting but with only one nic present (connected to the ovirtmgmt bridge).<br></div>
<div>
> >><br></div>
<div>
> >> I'm trying to obtain the modified engine vm by means of a trick which used to work in a previous (aborted because of lacking GlusterFS-by-libgfapi support) oVirt 3.5 test setup (about a year ago, maybe more): I'm substituting the standard /usr/share/ovirt-hosted-engine-setup/templates/<a href="http://vm.conf.in">vm.conf.in</a> with the following:<br></div>
<div>
> >><br></div>
<div>
> >> vmId=@VM_UUID@<br></div>
<div>
> >> memSize=@MEM_SIZE@<br></div>
<div>
> >> display=@CONSOLE_TYPE@<br></div>
<div>
> >> devices={index:2,iface:ide,address:{ controller:0, target:0,unit:0, bus:1, type:drive},specParams:{},readonly:true,deviceId:@CDROM_UUID@,path:@CDROM@,device:cdrom,shared:false,type:disk@BOOT_CDROM@}<br></div>
<div>
> >> devices={index:0,iface:virtio,format:raw,poolID:@SP_UUID@,volumeID:@VOL_UUID@,imageID:@IMG_UUID@,specParams:{},readonly:false,domainID:@SD_UUID@,optional:false,deviceId:@IMG_UUID@,address:{bus:0x00, slot:0x06, domain:0x0000, type:pci, function:0x0},device:disk,shared:exclusive,propagateErrors:off,type:disk@BOOT_DISK@}<br></div>
<div>
> >> devices={device:scsi,model:virtio-scsi,type:controller}<br></div>
<div>
> >> devices={index:4,nicModel:pv,macAddr:02:50:56:3f:c4:b0,linkActive:true,network:@BRIDGE@,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:@NIC_UUID@,address:{bus:0x00, slot:0x03, domain:0x0000, type:pci, function:0x0},device:bridge,type:interface@BOOT_PXE@}<br></div>
<div>
> >> devices={index:8,nicModel:pv,macAddr:02:50:56:3f:c4:a0,linkActive:true,network:lan,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:6c467650-1837-47ea-89bc-1113f4bfefee,address:{bus:0x00, slot:0x09, domain:0x0000, type:pci, function:0x0},device:bridge,type:interface@BOOT_PXE@}<br></div>
<div>
> >> devices={index:16,nicModel:pv,macAddr:02:50:56:3f:c4:c0,linkActive:true,network:nfs,filter:vdsm-no-mac-spoofing,specParams:{},deviceId:4d8e0705-8cb4-45b7-b960-7f98bb59858d,address:{bus:0x00, slot:0x0c, domain:0x0000, type:pci, function:0x0},device:bridge,type:interface@BOOT_PXE@}<br></div>
<div>
> >> devices={device:console,specParams:{},type:console,deviceId:@CONSOLE_UUID@,alias:console0}<br></div>
<div>
> >> vmName=@NAME@<br></div>
<div>
> >> spiceSecureChannels=smain,sdisplay,sinputs,scursor,splayback,srecord,ssmartcard,susbredir<br></div>
<div>
> >> smp=@VCPUS@<br></div>
<div>
> >> cpuType=@CPU_TYPE@<br></div>
<div>
> >> emulatedMachine=@EMULATED_MACHINE@<br></div>
<div>
> >><br></div>
<div>
> >> but unfortunately the vm gets created like this (output from "ps"; note that I'm attaching a CentOS7.1 Netinstall ISO with an embedded kickstart: the installation should proceed by HTTP on the lan network but obviously fails):<br></div>
<div>
> >><br></div>
<div>
> >> /usr/libexec/qemu-kvm -name HostedEngine -S -machine<br></div>
<div>
> >> pc-i440fx-rhel7.1.0,accel=kvm,usb=off -cpu Westmere -m 4096 -realtime mlock=off<br></div>
<div>
> >> -smp 2,sockets=2,cores=1,threads=1 -uuid f49da721-8aa6-4422-8b91-e91a0e38aa4a -s<br></div>
<div>
> >> mbios type=1,manufacturer=oVirt,product=oVirt Node,version=7-1.1503.el7.centos.2<br></div>
<div>
> >> .8,serial=2a1855a9-18fb-4d7a-b8b8-6fc898a8e827,uuid=f49da721-8aa6-4422-8b91-e91a<br></div>
<div>
> >> 0e38aa4a -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/li<br></div>
<div>
> >> b/libvirt/qemu/HostedEngine.monitor,server,nowait -mon chardev=charmonitor,id=mo<br></div>
<div>
> >> nitor,mode=control -rtc base=2015-10-25T11:22:22,driftfix=slew -global kvm-pit.l<br></div>
<div>
> >> ost_tick_policy=discard -no-hpet -no-reboot -boot strict=on -device piix3-usb-uh<br></div>
<div>
> >> ci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr<br></div>
<div>
> >> =0x4 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=<br></div>
<div>
> >> /var/tmp/engine.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 -drive file=/var/run/vdsm/storage/be4434bf-a5fd-44d7-8011-d5e4ac9cf523/b3abc1cb-8a78-4b56-a9b0-e5f41fea0fdc/8d075a8d-730a-4925-8779-e0ca2b3dbcf4,if=none,id=drive-virtio-disk0,format=raw,serial=b3abc1cb-8a78-4b56-a9b0-e5f41fea0fdc,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0 -netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:50:56:3f:c4:b0,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/f49da721-8aa6-4422-8b91-e91a0e38aa4a.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/f49da721-8aa6-4422-8b91-e91a0e38aa4a.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -chardev socket,id=charchannel2,path=/var/lib/libvirt/qemu/channels/f49da721-8aa6-4422-8b91-e91a0e38aa4a.org.ovirt.hosted-engine-setup.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.ovirt.hosted-engine-setup.0 -chardev socket,id=charconsole0,path=/var/run/ovirt-vmconsole-console/f49da721-8aa6-4422-8b91-e91a0e38aa4a.sock,server,nowait -device virtconsole,chardev=charconsole0,id=console0 -vnc 0:0,password -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -msg timestamp=on<br></div>
<div>
> >><br></div>
<div>
> >> There seem to be no errors in the logs.<br></div>
<div>
> >><br></div>
<div>
> >> I've tried reading some (limited) Python setup code but I've not found any obvious reason why the trick should not work anymore.<br></div>
<div>
> >><br></div>
<div>
> >> I know that 3.6 has different network configuration/management and this could be the hot point.<br></div>
<div>
> >><br></div>
<div>
> >> Does anyone have any further suggestion or clue (code/logs to read)?<br></div>
<div>
> ><br></div>
<div>
> > The VM creation path is now a bit different cause we use just vdscli library instead of vdsClient.<br></div>
<div>
> > Please take a a look at mixins.py<br></div>
<div>
><br></div>
<div>
> Many thanks for your very valuable hint:<br></div>
<div>
><br></div>
<div>
> I've restored the original /usr/share/ovirt-hosted-engine-setup/templates/<a href="http://vm.conf.in">vm.conf.in</a> and I've managed to obtain the 3-nics-customized vm by modifying /usr/lib/python2.7/site-packages/ovirt_hosted_engine_setup/mixins.py like this ("diff -Naur" output):<br></div>
<div> </div>
</div>
</div>
<div>Hi Simone,<br></div>
<div>
it seems that I spoke too soon :(<br></div>
<div> </div>
<div>
A separate network issue (already reported to the list) prevented me from successfully closing up the setup in its final phase (registering the host inside the Engine), so all seemed well while being stuck there :)<br></div>
<div> </div>
<div>
Now that I've solved that (for which I'm informing the list asap in a separate message) and the setup ended successfully, it seems that the last step of the HE setup (shutdown the Engine vm to place it under HA agent control) starts/creates a "different vm" and my virtual hardware customizations seem to be gone (only one NIC present, connected to ovirtmgmt).<br></div>
<div> </div>
<div>
My wild guess: maybe I need BOTH the mixins.py AND the <a href="http://vm.conf.in">vm.conf.in</a> customizations? ;)<br></div>
</blockquote><div> </div>
<div>Yes, you are right: the final configuration is still generated from the template so you need to fix both.<br></div>
<div> </div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex;"><div> </div>
<div>
It seems (from /etc/ovirt-hosted-engine/hosted-engine.conf) that the Engine vm definition is now in /var/run/ovirt-hosted-engine-ha/vm.conf<br></div>
<div> </div>
</blockquote><div> </div>
<div> </div>
<div>the vm configuration is now read from the shared domain converting back from the OVF_STORE, the idea is to let you edit it from the engine without the need to write it on each host.<br></div>
<div>/var/run/ovirt-hosted-engine-ha/vm.conf is just a temporary copy.<br></div>
<div> </div>
<div>As you are probably still not able to import the hosted-engine storage domain in the engine due to a well know bug, your system will fallback to the initial vm.conf configuration still on the shared storage and you can manually fix it:<br></div>
<div><div style="font-size:12.8px;">Please follow this procedure substituting '192.168.1.115:_Virtual_ext35u36' with the <span style="font-size:12.8px" class="size">mount point</span><span style="font-size:12.8px" class="size"> of </span><span style="font-size:12.8px" class="size">hosted-engine storage domain on your system:</span><br></div>
<div style="font-size:12.8px;"> </div>
<div><span style="font-family:monospace," monospace"" class="font"><span style="font-size:12.8px" class="size">mntpoint=/rhev/data-center/mnt/192.168.1.115:_Virtual_ext35u36<br>dir=`mktemp -d` && cd $dir<br>sdUUID_line=$(grep sdUUID /etc/ovirt-hosted-engine/hosted-engine.conf)<br>sdUUID=${sdUUID_line:7:36}<br>conf_volume_UUID_line=$(grep conf_volume_UUID /etc/ovirt-hosted-engine/hosted-engine.conf)<br>conf_volume_UUID=${conf_volume_UUID_line:17:36}<br>conf_image_UUID_line=$(grep conf_image_UUID /etc/ovirt-hosted-engine/hosted-engine.conf)<br>conf_image_UUID=${conf_image_UUID_line:16:36}<br>dd if=$mntpoint/$sdUUID/images/$conf_image_UUID/$conf_volume_UUID 2>/dev/null| tar -xvf -<br># directly edit vm.conf as you need <br>tar -cO * | dd of=$mntpoint/$sdUUID/images/$conf_image_UUID/$conf_volume_UUID</span></span></div>
</div>
<div> </div>
<div>When your engine will import the hosted-engine storage domain it will generate an OVF_STORE with the configuration of the engine VM, you will be able to edit some parameters from the engine and the agent will read the VM configuration from there.<br></div>
</div>
</div>
</div>
</blockquote><div> </div>
<div>Hi Simone,<br></div>
<div>I followed your advice and modified the vm.conf inside the conf_volume tar archive.<br></div>
<div> </div>
<div>Then I put the system (still with only one physical host) in global maintenance from the host:<br></div>
<div> </div>
<div>hosted-engine --set-maintenance --mode=global<br></div>
<div> </div>
<div>Then i regularly powered off the Engine vm and confirmed it from the host:<br></div>
<div> </div>
<div>hosted-engine --vm-status<br></div>
<div> </div>
<div>Then restarted it from the host:<br></div>
<div> </div>
<div>hosted-engine --vm-start<br></div>
<div> </div>
<div>Finally, I exited maintenance from the host:<br></div>
<div> </div>
<div>hosted-engine --set-maintenance --mode=none<br></div>
<div> </div>
<div>Waited a bit and verified it:<br></div>
<div> </div>
<div>hosted-engine --vm-status<br></div>
<div> </div>
<div>It all worked as expected!<br></div>
<div> </div>
<div>Many thanks again.<br></div>
<div> </div>
<div>Regards,<br></div>
<div>Giuseppe<br></div>
<div> </div>
<blockquote type="cite"><div dir="ltr"><div><div><div> </div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204, 204, 204);border-left-style:solid;padding-left:1ex;"><div> </div>
<div>
Many thanks for your assistance (and obviously I just sent a related wishlist item on the HE setup ;)<br></div>
<div> </div>
<div>
Regards,<br></div>
<div>
Giuseppe<br></div>
<div><div><div> </div>
<div> </div>
<div>
> ************************************************************************************<br></div>
<div>
><br></div>
<div>
> --- mixins.py.orig 2015-10-20 16:57:40.000000000 +0200<br></div>
<div>
> +++ mixins.py 2015-10-26 22:22:58.351223922 +0100<br></div>
<div>
> @@ -25,6 +25,7 @@<br></div>
<div>
> import random<br></div>
<div>
> import string<br></div>
<div>
> import time<br></div>
<div>
> +import uuid<br></div>
<div>
><br></div>
<div>
><br></div>
<div>
> from ovirt_hosted_engine_setup import constants as ohostedcons<br></div>
<div>
> @@ -247,6 +248,44 @@<br></div>
<div>
> ]['@BOOT_PXE@'] == ',bootOrder:1':<br></div>
<div>
> nic['bootOrder'] = '1'<br></div>
<div>
> conf['devices'].append(nic)<br></div>
<div>
> + nic2 = {<br></div>
<div>
> + 'nicModel': 'pv',<br></div>
<div>
> + 'macAddr': '02:50:56:3f:c4:a0',<br></div>
<div>
> + 'linkActive': 'true',<br></div>
<div>
> + 'network': 'lan',<br></div>
<div>
> + 'filter': 'vdsm-no-mac-spoofing',<br></div>
<div>
> + 'specParams': {},<br></div>
<div>
> + 'deviceId': str(uuid.uuid4()),<br></div>
<div>
> + 'address': {<br></div>
<div>
> + 'bus': '0x00',<br></div>
<div>
> + 'slot': '0x09',<br></div>
<div>
> + 'domain': '0x0000',<br></div>
<div>
> + 'type': 'pci',<br></div>
<div>
> + 'function': '0x0'<br></div>
<div>
> + },<br></div>
<div>
> + 'device': 'bridge',<br></div>
<div>
> + 'type': 'interface',<br></div>
<div>
> + }<br></div>
<div>
> + conf['devices'].append(nic2)<br></div>
<div>
> + nic3 = {<br></div>
<div>
> + 'nicModel': 'pv',<br></div>
<div>
> + 'macAddr': '02:50:56:3f:c4:c0',<br></div>
<div>
> + 'linkActive': 'true',<br></div>
<div>
> + 'network': 'nfs',<br></div>
<div>
> + 'filter': 'vdsm-no-mac-spoofing',<br></div>
<div>
> + 'specParams': {},<br></div>
<div>
> + 'deviceId': str(uuid.uuid4()),<br></div>
<div>
> + 'address': {<br></div>
<div>
> + 'bus': '0x00',<br></div>
<div>
> + 'slot': '0x0c',<br></div>
<div>
> + 'domain': '0x0000',<br></div>
<div>
> + 'type': 'pci',<br></div>
<div>
> + 'function': '0x0'<br></div>
<div>
> + },<br></div>
<div>
> + 'device': 'bridge',<br></div>
<div>
> + 'type': 'interface',<br></div>
<div>
> + }<br></div>
<div>
> + conf['devices'].append(nic3)<br></div>
<div>
><br></div>
<div>
> cli = self.environment[ohostedcons.VDSMEnv.VDS_CLI]<br></div>
<div>
> status = cli.create(conf)<br></div>
<div>
><br></div>
<div>
> ************************************************************************************<br></div>
<div>
><br></div>
<div>
> Obviously this is a horrible ad-hoc hack that I'm not able to generalize/clean-up now: doing so would involve (apart from a deeper understanding of the whole setup code/workflow) some well-thought-out design decisions and, given the effective deprecation of the aforementioned easy-to-modify <a href="http://vm.conf.in">vm.conf.in</a> template substituted by hardwired Python program logic, it seems that such a functionality is not very high on the development priority list atm ;)<br></div>
<div>
><br></div>
<div>
> Many thanks again!<br></div>
<div>
><br></div>
<div>
> Kind regards,<br></div>
<div>
> Giuseppe<br></div>
<div>
><br></div>
<div>
> >> Many thanks in advance.<br></div>
<div>
> >><br></div>
<div>
> >> Kind regards,<br></div>
<div>
> >> Giuseppe<br></div>
<div>
> >><br></div>
<div>
> >> PS: please keep also my address in replying because I'm experiencing some problems between Hotmail and oVirt-mailing-list<br></div>
<div>
> >><br></div>
<div>
> >> _______________________________________________<br></div>
<div>
> >><br></div>
<div>
> Users mailing list<br></div>
<div>
> >> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br></div>
<div>
> >> <a href="http://lists.ovirt.org/mailman/listinfo/users">http://lists.ovirt.org/mailman/listinfo/users</a><br></div>
<div>
> >><br></div>
</div>
</div>
</blockquote></div>
</div>
</div>
</blockquote><div> </div>
</body>
</html>