- Users - oVirt List Archives

Cannot change resolution on Ubuntu 16 LTS, using spice
by Филипп Панин 21 Sep '17

21 Sep '17

Hey, i have problem setting resolution higher than 1024x768 on guest using spice xserver-xorg-video - installed spice-vdagent - installed ovirt-guest-agent - installed /var/log/syslog: Sep 21 13:16:48 admin-oVirt-Node kernel: [ 1548.537892] qxl 0000:00:02.0: object_init failed for (8302592, 0x00000002) Sep 21 13:16:48 admin-oVirt-Node kernel: [ 1548.537901] [drm:qxl_gem_object_create [qxl]] *ERROR* Failed to allocate GEM object (8302080, 2, 4096, -12) Sep 21 13:16:48 admin-oVirt-Node kernel: [ 1548.537905] [drm:qxl_alloc_surf_ioctl [qxl]] *ERROR* qxl_alloc_surf_ioctl: failed to create gem ret=-12

1 0

[ANN] oVirt 4.1.7 First Release Candidate is now available
by Sandro Bonazzola 21 Sep '17

21 Sep '17

The oVirt Project is pleased to announce the availability of the First Release Candidate of oVirt 4.1.7, as of September 21st, 2017 This update is the seventh in a series of stabilization updates to the 4.1 series. Starting from 4.1.5 oVirt supports libgfapi [5]. Using libgfapi provides a real performance boost for ovirt when using GlusterFS . Due to a known issue [6], using this will break live storage migration. This is expected to be fixed soon. If you do not use live storage migration you can give it a try. Use [7] for more details on how to enable it. This release is available now for: * Red Hat Enterprise Linux 7.4 or later * CentOS Linux (or similar) 7.4 or later This release supports Hypervisor Hosts running: * Red Hat Enterprise Linux 7.4 or later * CentOS Linux (or similar) 7.4 or later * oVirt Node 4.1 See the release notes draft [3] for installation / upgrade instructions and a list of new features and bugs fixed. Notes: - oVirt Appliance is already available - oVirt Live is already available[4] - oVirt Node is already available[4] Additional Resources: * Read more about the oVirt 4.1.7 release highlights: http://www.ovirt.org/release/4.1.7/ * Get more oVirt Project updates on Twitter: https://twitter.com/ovirt * Check out the latest project news on the oVirt blog: http://www.ovirt.org/blog/ [1] https://www.ovirt.org/community/ [2] https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt [3] http://www.ovirt.org/release/4.1.7/ [4] http://resources.ovirt.org/pub/ovirt-4.1-pre/iso/ [5] http://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/lib… [6] https://bugzilla.redhat.com/show_bug.cgi?id=1306562 [7] http://www.ovirt.org/develop/release-management/features/storage/glusterfs-… -- SANDRO BONAZZOLA ASSOCIATE MANAGER, SOFTWARE ENGINEERING, EMEA ENG VIRTUALIZATION R&D Red Hat EMEA <https://www.redhat.com/> <https://red.ht/sig> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted> <http://www.teraplan.it/redhat-osd-2017/>

1 0

Still running in to problems trying to implement OVN
by Mitchell Smith 20 Sep '17

20 Sep '17

Hi List, I was very happy to see overt added support for OVN recently, and it was a fairly strong factor in choosing overt for our current project, however being a very new feature with very little in the way of publicly available documentation, we are having some difficulties with implementation. I have found a few very good articles on OVN itself and am comfortable with OVN cli for the most part, but translating that in to overt logical networks and NICs is proving to be quite a challenge. As you might remember from one of my earlier posts, we do have some limitations with the current hosting provider also that make our network topology for this project less than ideal, but hopefully with OVN I can get to the desired outcome somehow. So the first fairly big issue we run into relates to the ovirtmgmt network, obviously this is a hard dependency of overt and it can’t be removed. As we only have one physical NIC in each server though, this causes some problems since we can’t also attach OVS to the same NIC as the ovirtmgmt bridge. Ideally we could set up OVN and just plug ovirtmgmt in to that network, but I can’t find any way to do that. Secondly our public IPs each have a specific MAC attached to them, something in the 00:50:56 VMware prefix, I want to add these on OVN and then SNAT them to the VMs internal IP on the 10.x.x.x network, I can work out how to do this on the cli, but not in overt, should I just manually create these interfaces and then get overt to try and import them or similar? As an alternative, am I making this type of network structure to complex by using OVN, and should I think about just creating a private address space using Tinc / PeerVPN / FreeLAN / similar, and then just set up a single VM with multiple interfaces on it acting as a router? To try and explain the network structure a bit better, each server has one physical NIC with a public IP address, each server also has a /29, each individual IP has a MAC associated with it on the vendors firewall. I plan to have a private subnet across the whole cluster managed with OVN, which I will place VMs on. The public IPs will be transparently routed to the private IPs using OVN routers. It’s less than ideal, but those are the constraints we have to work in for this project. My final question is completely off topic for this mostly OVN themed email, but we have also been looking at ScaleIO from EMC, and it looks like a decent solution. We have built our proof of concept environment using Gluster but we are considering ScaleIO for the final implementation, and I was curious if anyone else had tried ScaleIO with overt, and did you find any advantages / disadvantages compared to Gluster? I will end this rather long email here, but if anyone has tried to implement a similar network structure on overt, I would very much appreciate any suggestions you might have for the best way to proceed. Thanks :) Mitch

1 0

Re: [ovirt-users] VM running on multiple hosts
by Yaniv Kaul 20 Sep '17

20 Sep '17

On Sep 20, 2017 9:50 PM, <support(a)jac-properties.com> wrote: This matches about with what we were thinking, thank you! To answer your questions We do not have power management configured due to it causing a cascading failure early in our deployment. The host was not fenced and "confirm host rebooted" was never used. The VMs were powered on via virsh (this shouldn't have happened) The way they were powered on is most likely why they were corrupted is our thought We'd be happy if you could share both engine and host logs, including vdsm.log, engine.log and /var/log/messages from both. Y. Logan On September 20, 2017 at 12:03 PM Michal Skrivanek < michal.skrivanek(a)redhat.com> wrote: On 20 Sep 2017, at 18:06, Logan Kuhn <support(a)jac-properties.com> wrote: We had an incident where a VM hosts' disk filled up, the VMs all went unknown in the web console, but were fully functional if you were to login or use the services of one. Hi, yes, that can happen since the VM’s storage is on NAS whereas the server itself is non-functional as the management and all other local processes are using local resources We couldn't migrate them so we powered them down on that host and powered them up and let ovirt choose the host for it, same as always. that’s a mistake. The host should be fenced in that case, you likely do not have a power management configured, do you? Even when you do not have a fencing device available it should have been resolved manually by rebooting it manually(after fixing the disk problem), or in case of permanent damage (e.g. server needs to be replaced, that takes a week, you need to run those VMs in the meantime elsewhere) it should have been powered off and VM states should be reset by “confirm host has been rebooted” manual action. Normally you should now be able to run those VMs while the status of the host is still Not Responding - was it not the case? How exactly you get to the situation that you were able to power up the VMs? However the disk image on a few of them were corrupted because once we fixed the host with the full disk, it still thought it should be running the VM. Which promptly corrupted the disk, the error seems to be this in the logs: this can only happen for VMs flagged as HA, is it a case? Thanks, michal 2017-09-19 21:59:11,058 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [36c806f6] VM '70cf75c7-0fc2-4bbe-958e-7d0095f70960'(testhub) is running in db and not running on VDS 'ef6dc2a3-af6e-4e00-aa4 0-493b31263417'(vm-int7) We upgraded to 4.1.6 from 4.0.6 earlier in the day, I don't really think it's anything more than coincidence, but it's worrying enough to send to the community. Regards, Logan _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users _______________________________________________ Users mailing list Users(a)ovirt.org http://lists.ovirt.org/mailman/listinfo/users

2 1

VM running on multiple hosts
by Logan Kuhn 20 Sep '17

20 Sep '17

We had an incident where a VM hosts' disk filled up, the VMs all went unknown in the web console, but were fully functional if you were to login or use the services of one. We couldn't migrate them so we powered them down on that host and powered them up and let ovirt choose the host for it, same as always. However the disk image on a few of them were corrupted because once we fixed the host with the full disk, it still thought it should be running the VM. Which promptly corrupted the disk, the error seems to be this in the logs: 2017-09-19 21:59:11,058 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [36c806f6] VM '70cf75c7-0fc2-4bbe-958e-7d0095f70960'(testhub) is running in db and not running on VDS 'ef6dc2a3-af6e-4e00-aa4 0-493b31263417'(vm-int7) We upgraded to 4.1.6 from 4.0.6 earlier in the day, I don't really think it's anything more than coincidence, but it's worrying enough to send to the community. Regards, Logan

3 2

VM running on multiple hosts
by Logan Kuhn 20 Sep '17

20 Sep '17

--=_0e272b5f-fcd6-4360-b961-925677073066 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit We had an incident where a VM hosts' disk filled up, the VMs all went unknown in the web console, but were fully functional if you were to login or use the services of one. We couldn't migrate them so we powered them down on that host and powered them up and let ovirt choose the host for it, same as always. However the disk image on a few of them were corrupted because once we fixed the host with the full disk, it still thought it should be running the VM. Which promptly corrupted the disk, the error seems to be this in the logs: 2017-09-19 21:59:11,058 INFO [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyzer] (DefaultQuartzScheduler3) [36c806f6] VM '70cf75c7-0fc2-4bbe-958e-7d0095f70960'(testhub) is running in db and not running on VDS 'ef6dc2a3-af6e-4e00-aa4 0-493b31263417'(vm-int7) We upgraded to 4.1.6 from 4.0.6 earlier in the day, I don't really think it's anything more than coincidence, but it's worrying enough to send to the community. Regards, Logan --=_0e272b5f-fcd6-4360-b961-925677073066 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: Arial; font-size: 12pt; color: #0000= 00"><div>We had an incident where a VM hosts' disk filled up, the VMs all w= ent unknown in the web console, but were fully functional if you were to lo= gin or use the services of one.  We couldn't migrate them so we powere= d them down on that host and powered them up and let ovirt choose the host = for it, same as always.  However the disk image on a few of them were = corrupted because once we fixed the host with the full disk, it still thoug= ht it should be running the VM.  Which promptly corrupted the disk, th= e error seems to be this in the logs:</div><div><br data-mce-bogus=3D"1"></= div><div><span style=3D"font-family: monospace;" data-mce-style=3D"font-fam= ily: monospace;"><span style=3D"color: #000000; background-color: #ffffff;"= data-mce-style=3D"color: #000000; background-color: #ffffff;">2017-09-19 2= 1:59:11,058 INFO  [org.ovirt.engine.core.vdsbroker.monitoring.VmAnalyz= er] (DefaultQuartzScheduler3) [36c806f6] VM '70cf75c7-0fc2-4bbe-958e-7d0095= f70960'(testhub) is </span><span style=3D"font-weight: bold; color: #ff5454= ; background-color: #ffffff;" data-mce-style=3D"font-weight: bold; color: #= ff5454; background-color: #ffffff;">running</span><span style=3D"color: #00= 0000; background-color: #ffffff;" data-mce-style=3D"color: #000000; backgro= und-color: #ffffff;"> in db and not </span><span style=3D"font-weight: bold= ; color: #ff5454; background-color: #ffffff;" data-mce-style=3D"font-weight= : bold; color: #ff5454; background-color: #ffffff;">running</span><span sty= le=3D"color: #000000; background-color: #ffffff;" data-mce-style=3D"color: = #000000; background-color: #ffffff;"> on VDS 'ef6dc2a3-af6e-4e00-aa4</span>= <br>0-493b31263417'(vm-int7)<br></span></div><div><br></div><div>We upgrade= d to 4.1.6 from 4.0.6 earlier in the day, I don't really think it's anythin= g more than coincidence, but it's worrying enough to send to the community.= </div><div><br data-mce-bogus=3D"1"></div><div data-marker=3D"__SIG_PRE__">= Regards,<br>Logan</div></div></body></html> --=_0e272b5f-fcd6-4360-b961-925677073066--

1 0

ovirt-engine service crash
by Andrew Dent 20 Sep '17

20 Sep '17

--------=_MBB3C20A1B-399A-4E02-90A9-15AD53725A32 Content-Type: text/plain; format=flowed; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi I have a hosted-engine 4.1.2.2-1.el7.centos setup The ovirt-engine service crashed and this is the result of querying the=20 status of the service. Restarting the service works, but then about 10 minutes later while=20 using the GUI it will crash again. I don't seem to need to restart apache or the webproxy services, though=20 if I do restart those two, the ovirt-engine service will still crash ~10=20 minutes later. How can I fix this, and where to I look for more hints as to why this is=20 happening? Redirecting to /bin/systemctl status ovirt-engine.service =E2=97=8F ovirt-engine.service - oVirt Engine Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enabled;= =20 vendor preset: disabled) Active: failed (Result: exit-code) since Sat 2017-08-12 21:01:52 AEST;= =20 2min 14s ago Process: 2028=20 ExecStart=3D/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py= =20 --redirect-output --systemd=3Dnotify $EXTRA_ARGS start (code=3Dexited,=20 status=3D1/FAILURE) Main PID: 2028 (code=3Dexited, status=3D1/FAILURE) Kind regards Andrew Dent --------=_MBB3C20A1B-399A-4E02-90A9-15AD53725A32 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <?xml version=3D"1.0" encoding=3D"utf-16"?><html><head> <style id=3D"signatureStyle"></style> <style></style> </head> <body><div>Hi</div><div><br /></div><div>I have a hosted-engine=C2=A0<span= style=3D"font-family: 'Arial Unicode MS', Arial, sans-serif; font-size: sma= ll; orphans: 2; text-align: -webkit-center; widows: 2; background-color: rg= b(255, 255, 255);">4.1.2.2-1.el7.centos</span><span style=3D"font-size: 12p= t;">=C2=A0setup=C2=A0</span></div><div>The ovirt-engine service crashed and = this is the result of querying the status of the service.=C2=A0</div><div>= Restarting the service works, but then about 10 minutes later while using t= he GUI it will crash again.=C2=A0</div><div>I don't seem to need to restart = apache or the webproxy services, though if I do restart those two, the ovi= rt-engine service will still crash ~10 minutes later.=C2=A0</div><div>How c= an I fix this, and where to I look for more hints as to why this is happeni= ng?</div><div><br /></div><div>Redirecting to /bin/systemctl status =C2=A0o= virt-engine.service<br />=E2=97=8F ovirt-engine.service - oVirt Engine<br /= > =C2=A0 Loaded: loaded (/usr/lib/systemd/system/ovirt-engine.service; enab= led; vendor preset: disabled)<br /> =C2=A0 Active: failed (Result: exit-cod= e) since Sat 2017-08-12 21:01:52 AEST; 2min 14s ago<br /> =C2=A0Process: 20= 28 ExecStart=3D/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.p= y --redirect-output --systemd=3Dnotify $EXTRA_ARGS start (code=3Dexited, st= atus=3D1/FAILURE)<br /> Main PID: 2028 (code=3Dexited, status=3D1/FAILURE)<= br /></div><div><br /></div><div id=3D"signature_old"><div id=3D"x545aa98cb= 8d44e8"> <div class=3D"WordSection1"> <div id=3D"xb71224f920234978acc74f4d23143069"><div class=3D"WordSection1"><= p class=3D"MsoNormal"><font face=3D"Tahoma"><font size=3D"3" style=3D"font-= size: 12pt;">Kind r</font><span style=3D"font-size: 12pt;">egards</span></f= ont></p><p class=3D"MsoNormal"><font size=3D"3" style=3D"font-size: 12pt;"= face=3D"Tahoma"><br /></font></p><p class=3D"MsoNormal"><font size=3D"3" st= yle=3D"font-size: 12pt;" face=3D"Tahoma">Andrew Dent</font></p> <p class=3D"MsoNormal"><br /></p></div></div> </div> </div></div><div><br /></div> </body></html> --------=_MBB3C20A1B-399A-4E02-90A9-15AD53725A32--

3 2

How to import OVA file into oVirt
by gabriel_skupien＠o2.pl 20 Sep '17

20 Sep '17

I am trying to import OVA file into my test oVirt env. I already have the export domain configured but when I go to the "VM Import" tab the "Import" button in inactive. oVirt 4.1.5/4.1.6 Any clue? G.

2 6

Impossible to add other host to hosted engine's cluster
by Demeter Tibor 20 Sep '17

20 Sep '17

--=_b3dd23ad-95be-449f-867c-dcd17f58f735 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi,=20 I've created a 3 host based hosted engine. I can't add other hosts to hoste= d engine's cluster.=20 Before the setup, I have created a 3 node glusterfs for hosted engine on th= em.=20 Because I would like use and separate vlans, I created two vlan bridge on a= ll host. First is for ovirtmgmt, and a second for glusterfs.=20 The hosted engine setup was successful but when I try to add other hosts, t= hat is does not work, because vdsmd could not started.=20 I found some network related errors in the logs:=20 restore-net::ERROR::2017-09-19 19:14:48,072::__init__::51::root::(__exit__)= Failed rollback transaction last known good network.=20 Traceback (most recent call last):=20 File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 150= , in _setup_legacy=20 in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 523, in bonds_setup=20 _bonds_remove(remove, configurator, _netinfo, in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 548, in _bonds_remove=20 if _is_bond_valid_for_removal(name, _netinfo, in_rollback):=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 573, in _is_bond_valid_for_removal=20 'interfaces %s' % (bond, bond_users))=20 ConfigNetworkError: (31, "Cannot remove bonding bond0: used by another inte= rfaces set(['bond0.105', 'bond0.104'])")=20 restore-net::ERROR::2017-09-19 19:14:48,073::vdsm-restore-net-config::478::= root::(restore) unified restoration failed.=20 Traceback (most recent call last):=20 File "/usr/share/vdsm/vdsm-restore-net-config", line 473, in restore=20 unified_restoration()=20 File "/usr/share/vdsm/vdsm-restore-net-config", line 123, in unified_restor= ation=20 _remove_networks_in_running_config()=20 File "/usr/share/vdsm/vdsm-restore-net-config", line 238, in _remove_networ= ks_in_running_config=20 '_inRollback': True})=20 File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 218, in s= etupNetworks=20 _setup_networks(networks, bondings, options)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 239, in _= setup_networks=20 netswitch.setup(networks, bondings, options, in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 132= , in setup=20 _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 150= , in _setup_legacy=20 in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 523, in bonds_setup=20 _bonds_remove(remove, configurator, _netinfo, in_rollback)=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 548, in _bonds_remove=20 if _is_bond_valid_for_removal(name, _netinfo, in_rollback):=20 File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line= 573, in _is_bond_valid_for_removal=20 'interfaces %s' % (bond, bond_users))=20 ConfigNetworkError: (31, "Cannot remove bonding bond0: used by another inte= rfaces set(['bond0.105', 'bond0.104'])")=20 The four host network config is same, the ovirtmgmt bond.104 is the default= route.=20 How can I add another host to cluster?=20 =C3=9Cdv=C3=B6zlettel:=20 Demeter Tibor=20 E-mail: tdemeter @itsmart.hu=20 Mobil: [ callto:+36 30 3777122 | +36 30 ] 4620500=20 Web: [ http://itsmart.hu/ | www.itsmart.hu ]=20 [ http://itsmart.hu/ ]=20 =09ItSmart Informatikai Kft.=20 2120 Dunakeszi, Wass Albert U. 2. 1/9.=20 Telefon: [ callto:+36 30 462 0500 | +36 30 462 0500 ]=20 [EN] This message and any attachments are confidential and privileged and i= ntended for the use of the addressee only. If you have received this commun= ication in error, please notify the sender by replay e-mail and delete this= message from your system. Please note that Internet e-mail guarantees neit= her the confidentiality nor the proper receipt of the message sent. The dat= a deriving from our correspondence with you are included in a file of ITSMA= RT Ltd which exclusive purpose is to manage the communications of the compa= ny; under the understanding that, in maintaining said correspondence, you a= uthorize the treatment of such data for the mentioned purpose. You are enti= tled to exercise your rights of access, rectification, cancellation and opp= osition by addressing such written application to address above.=20 [HUN] Ez az =C3=BCzenet =C3=A9s annak b=C3=A1rmely csatolt anyaga bizalmas,= a nyilv=C3=A1nos k=C3=B6zl=C3=A9stol v=C3=A9dett, kiz=C3=A1r=C3=B3lag a c= =C3=ADmzett haszn=C3=A1lhatja fel. Ha =C3=96n nem az =C3=BCzenet c=C3=ADmze= ttje, =C3=BAgy k=C3=A9rj=C3=BCk v=C3=A1lasz=C3=BCzenetben =C3=A9rtes=C3=ADt= se errol a felad=C3=B3t =C3=A9s t=C3=B6r=C3=B6lje az =C3=BCzenetet a rendsz= erbol. K=C3=A9rj=C3=BCk vegye figyelembe, hogy az email-en t=C3=B6rt=C3=A9n= o inform=C3=A1ci=C3=B3tov=C3=A1bb=C3=ADt=C3=A1s kock=C3=A1zattal j=C3=A1rha= t, nem garant=C3=A1lja sem a csatorna bizalmass=C3=A1g=C3=A1t, sem a k=C3= =A9zbes=C3=ADt=C3=A9st. A lev=C3=A9l az ITSMART Informatikai Kft. kommunik= =C3=A1ci=C3=B3j=C3=A1nak eszk=C3=B6ze, az adatokat kiz=C3=A1r=C3=B3lag erre= a c=C3=A9lra haszn=C3=A1ljuk. Jogosult t=C3=A1j=C3=A9koztat=C3=A1st k=C3= =A9rni szem=C3=A9lyes adatai kezel=C3=A9s=C3=A9rol, k=C3=A9rheti azok helye= sb=C3=ADt=C3=A9s=C3=A9t, illetve t=C3=B6rl=C3=A9s=C3=A9t =C3=ADr=C3=A1sos k= =C3=A9relemben a fenti e-mail c=C3=ADmen.=20 --=_b3dd23ad-95be-449f-867c-dcd17f58f735 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hi,</div><div>I've created a 3 host based h= osted engine. I can't add other hosts to hosted engine's cluster.</div><div= ><br></div><div>Before the setup, I have created a 3 node glusterfs for hos= ted engine on them.</div><div>Because I would like use and separate vlans, = I created two vlan bridge on all host. First is for ovirtmgmt, and a second= for glusterfs.</div><div>The hosted engine setup was successful but when I= try to add other hosts, that is does not work, because vdsmd could not sta= rted. </div><div>I found some network related errors in the logs:</div= ><div><br data-mce-bogus=3D"1"></div><div><div>restore-net::ERROR::2017-09-= 19 19:14:48,072::__init__::51::root::(__exit__) Failed rollback transaction= last known good network.</div><div>Traceback (most recent call last):</div= ><div>  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.= py", line 150, in _setup_legacy</div><div>    in_rollback)</div><= div>  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switc= h.py", line 523, in bonds_setup</div><div>    _bonds_remove(remov= e, configurator, _netinfo, in_rollback)</div><div>  File "/usr/lib/pyt= hon2.7/site-packages/vdsm/network/legacy_switch.py", line 548, in _bonds_re= move</div><div>    if _is_bond_valid_for_removal(name, _netinfo, = in_rollback):</div><div>  File "/usr/lib/python2.7/site-packages/vdsm/= network/legacy_switch.py", line 573, in _is_bond_valid_for_removal</div><di= v>    'interfaces %s' % (bond, bond_users))</div><div>ConfigNetwo= rkError: (31, "Cannot remove bonding bond0: used by another interfaces set(= ['bond0.105', 'bond0.104'])")</div><div>restore-net::ERROR::2017-09-19 19:1= 4:48,073::vdsm-restore-net-config::478::root::(restore) unified restoration= failed.</div><div>Traceback (most recent call last):</div><div>  File= "/usr/share/vdsm/vdsm-restore-net-config", line 473, in restore</div><div>=     unified_restoration()</div><div>  File "/usr/share/vdsm/= vdsm-restore-net-config", line 123, in unified_restoration</div><div> =   _remove_networks_in_running_config()</div><div>  File "/usr/sh= are/vdsm/vdsm-restore-net-config", line 238, in _remove_networks_in_running= _config</div><div>    '_inRollback': True})</div><div>  File= "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 218, in setup= Networks</div><div>    _setup_networks(networks, bondings, option= s)</div><div>  File "/usr/lib/python2.7/site-packages/vdsm/network/api= .py", line 239, in _setup_networks</div><div>    netswitch.setup(= networks, bondings, options, in_rollback)</div><div>  File "/usr/lib/p= ython2.7/site-packages/vdsm/network/netswitch.py", line 132, in setup</div>= <div>    _setup_legacy(legacy_nets, legacy_bonds, options, in_rol= lback)</div><div>  File "/usr/lib/python2.7/site-packages/vdsm/network= /netswitch.py", line 150, in _setup_legacy</div><div>    in_rollb= ack)</div><div>  File "/usr/lib/python2.7/site-packages/vdsm/network/l= egacy_switch.py", line 523, in bonds_setup</div><div>    _bonds_r= emove(remove, configurator, _netinfo, in_rollback)</div><div>  File "/= usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 548, i= n _bonds_remove</div><div>    if _is_bond_valid_for_removal(name,= _netinfo, in_rollback):</div><div>  File "/usr/lib/python2.7/site-pac= kages/vdsm/network/legacy_switch.py", line 573, in _is_bond_valid_for_remov= al</div><div>    'interfaces %s' % (bond, bond_users))</div><div>= ConfigNetworkError: (31, "Cannot remove bonding bond0: used by another inte= rfaces set(['bond0.105', 'bond0.104'])")</div><div><br></div><div>The four = host network config is same, the ovirtmgmt bond.104 is the default route.</= div><div><br data-mce-bogus=3D"1"></div><div>How can I add another host to = cluster?</div><div><br data-mce-bogus=3D"1"></div></div><div><br></div><div= data-marker=3D"__SIG_PRE__"><p style=3D"font-family: 'Times New Roman'; fo= nt-size: medium; margin: 0px;" data-mce-style=3D"font-family: 'Times New Ro= man'; font-size: medium; margin: 0px;"><strong><span style=3D"font-size: me= dium;" data-mce-style=3D"font-size: medium;"><span style=3D"color: #2d67b0;= " data-mce-style=3D"color: #2d67b0;"><br></span></span></strong></p><p styl= e=3D"font-family: 'Times New Roman'; font-size: medium; margin: 0px;" data-= mce-style=3D"font-family: 'Times New Roman'; font-size: medium; margin: 0px= ;"><span style=3D"font-family: georgia, serif; color: #000080;" data-mce-st= yle=3D"font-family: georgia, serif; color: #000080;"><strong><span style=3D= "font-size: medium;" data-mce-style=3D"font-size: medium;"><span>=C3=9Cdv= =C3=B6zlettel:</span></span></strong></span></p><p style=3D"font-family: 'T= imes New Roman'; font-size: medium; margin: 0px;" data-mce-style=3D"font-fa= mily: 'Times New Roman'; font-size: medium; margin: 0px;"><span style=3D"fo= nt-family: georgia, serif; color: #000080;" data-mce-style=3D"font-family: = georgia, serif; color: #000080;"><strong><span style=3D"font-size: medium;"= data-mce-style=3D"font-size: medium;"><span><br></span></span></strong></s= pan></p><p style=3D"font-family: 'Times New Roman'; font-size: medium; marg= in: 0px;" data-mce-style=3D"font-family: 'Times New Roman'; font-size: medi= um; margin: 0px;"><span style=3D"font-family: georgia, serif; color: #00008= 0;" data-mce-style=3D"font-family: georgia, serif; color: #000080;"><strong= ><span style=3D"font-size: medium;" data-mce-style=3D"font-size: medium;"><= span>Demeter Tibor</span></span></strong></span></p><p style=3D"font-family= : 'Times New Roman'; font-size: medium; margin: 0px;" data-mce-style=3D"fon= t-family: 'Times New Roman'; font-size: medium; margin: 0px;"><br><span sty= le=3D"color: #000000; font-family: arial, helvetica, sans-serif; font-style= : normal; font-variant-ligatures: normal; font-variant-caps: normal; font-w= eight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-= indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spa= cing: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;" data-mce-style= =3D"color: #000000; font-family: arial, helvetica, sans-serif; font-style: = normal; font-variant-ligatures: normal; font-variant-caps: normal; font-wei= ght: normal; letter-spacing: normal; orphans: 2; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: 2; word-spaci= ng: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;">E-mail: <span st= yle=3D"color: #0000ff;" data-mce-style=3D"color: #0000ff;">tdemeter<span cl= ass=3D"Object" id=3D"OBJ_PREFIX_DWT5299_ZmEmailObjectHandler" style=3D"curs= or: pointer;" data-mce-style=3D"cursor: pointer;"><span class=3D"Object" id= =3D"OBJ_PREFIX_DWT5313_ZmEmailObjectHandler" style=3D"cursor: pointer;" dat= a-mce-style=3D"cursor: pointer;">@itsmart.hu</span></span></span></span><br= style=3D"color: #000000; font-family: arial, helvetica, sans-serif; font-s= ize: 16px; font-style: normal; font-variant-ligatures: normal; font-variant= -caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; tex= t-align: start; text-indent: 0px; text-transform: none; white-space: normal= ; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-s= tyle=3D"color: #000000; font-family: arial, helvetica, sans-serif; font-siz= e: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-c= aps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-= align: start; text-indent: 0px; text-transform: none; white-space: normal; = widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style= =3D"color: #000000; font-family: arial, helvetica, sans-serif; font-style: = normal; font-variant-ligatures: normal; font-variant-caps: normal; font-wei= ght: normal; letter-spacing: normal; orphans: 2; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: 2; word-spaci= ng: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;" data-mce-style= =3D"color: #000000; font-family: arial, helvetica, sans-serif; font-style: = normal; font-variant-ligatures: normal; font-variant-caps: normal; font-wei= ght: normal; letter-spacing: normal; orphans: 2; text-align: start; text-in= dent: 0px; text-transform: none; white-space: normal; widows: 2; word-spaci= ng: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;">Mobil: <spa= n class=3D"Object" id=3D"OBJ_PREFIX_DWT5300_com_zimbra_phone" style=3D"colo= r: #336699; cursor: pointer;" data-mce-style=3D"color: #336699; cursor: poi= nter;"><a href=3D"callto:+36 30 3777122" target=3D"_blank" style=3D"color: = #336699; text-decoration: none; cursor: pointer;" data-mce-style=3D"color: = #336699; text-decoration: none; cursor: pointer;">+36 30 </a>4620500</span>= </span><br style=3D"color: #000000; font-family: arial, helvetica, sans-ser= if; font-size: 16px; font-style: normal; font-variant-ligatures: normal; fo= nt-variant-caps: normal; font-weight: normal; letter-spacing: normal; orpha= ns: 2; text-align: start; text-indent: 0px; text-transform: none; white-spa= ce: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;" = data-mce-style=3D"color: #000000; font-family: arial, helvetica, sans-serif= ; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font= -variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans= : 2; text-align: start; text-indent: 0px; text-transform: none; white-space= : normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><s= pan style=3D"color: #000000; font-family: arial, helvetica, sans-serif; fon= t-style: normal; font-variant-ligatures: normal; font-variant-caps: normal;= font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start= ; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; w= ord-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;" data-mc= e-style=3D"color: #000000; font-family: arial, helvetica, sans-serif; font-= style: normal; font-variant-ligatures: normal; font-variant-caps: normal; f= ont-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; widows: 2; wor= d-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 11pt;">Web:&nbsp= ;<span class=3D"Object" id=3D"OBJ_PREFIX_DWT5301_com_zimbra_url" style=3D"c= olor: #336699; cursor: pointer;" data-mce-style=3D"color: #336699; cursor: = pointer;"><span class=3D"Object" id=3D"OBJ_PREFIX_DWT5314_com_zimbra_url" s= tyle=3D"cursor: pointer;" data-mce-style=3D"cursor: pointer;"><a href=3D"ht= tp://itsmart.hu/" target=3D"_blank" style=3D"color: #336699; text-decoratio= n: none; cursor: pointer;" data-mce-style=3D"color: #336699; text-decoratio= n: none; cursor: pointer;">www.itsmart.hu</a></span></span></span><br></p><= br class=3D"Apple-interchange-newline"><table style=3D"color: #000000; font= -family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal;= font-variant-ligatures: normal; font-variant-caps: normal; font-weight: no= rmal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0= px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px= ; -webkit-text-stroke-width: 0px; table-layout: fixed;" cellpadding=3D"5px"= border=3D"0px" data-mce-style=3D"color: #000000; font-family: arial, helve= tica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatur= es: normal; font-variant-caps: normal; font-weight: normal; letter-spacing:= normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: n= one; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke= -width: 0px; table-layout: fixed;"><tbody><tr><td style=3D"font-family: Ver= dana, Arial, Helvetica, sans-serif; font-size: 11px;" data-mce-style=3D"fon= t-family: Verdana, Arial, Helvetica, sans-serif; font-size: 11px;"><span cl= ass=3D"Object" id=3D"OBJ_PREFIX_DWT5302_com_zimbra_url" style=3D"color: #33= 6699; cursor: pointer;" data-mce-style=3D"color: #336699; cursor: pointer;"= ><span class=3D"Object" id=3D"OBJ_PREFIX_DWT5315_com_zimbra_url" style=3D"c= ursor: pointer;" data-mce-style=3D"cursor: pointer;"><a href=3D"http://itsm= art.hu/" target=3D"_blank" style=3D"color: #336699; text-decoration: none; = cursor: pointer;" data-mce-style=3D"color: #336699; text-decoration: none; = cursor: pointer;"><img data-mce-src=3D"http://itsmart.hu/wp-content/uploads= /2016/09/xlogo_new.png.pagespeed.ic.2bojmSIfwB.png" src=3D"http://itsmart.h= u/wp-content/uploads/2016/09/xlogo_new.png.pagespeed.ic.2bojmSIfwB.png"></a= ></span></span><br></td><td style=3D"font-family: Verdana, Arial, Helvetica= , sans-serif; font-size: 11px; border-left: solid #b80004;" data-mce-style= =3D"font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 11px; bo= rder-left: solid #b80004;"><span style=3D"font-family: terminal, monaco; fo= nt-size: 11pt;" data-mce-style=3D"font-family: terminal, monaco; font-size:= 11pt;">ItSmart Informatikai Kft.</span><br><span style=3D"font-size: 11pt;= font-family: terminal, monaco;" data-mce-style=3D"font-size: 11pt; font-fa= mily: terminal, monaco;"><em>2120 Dunakeszi, Wass Albert U. 2. 1/9.</em></s= pan><br><span style=3D"font-size: 11pt;" data-mce-style=3D"font-size: 11pt;= "><em><span style=3D"font-family: terminal, monaco;" data-mce-style=3D"font= -family: terminal, monaco;">Telefon: <span class=3D"Object" id=3D"OBJ_= PREFIX_DWT5303_com_zimbra_phone" style=3D"color: #336699; cursor: pointer;"= data-mce-style=3D"color: #336699; cursor: pointer;"><a href=3D"callto:+36 = 30 462 0500" target=3D"_blank" style=3D"color: #336699; text-decoration: no= ne; cursor: pointer;" data-mce-style=3D"color: #336699; text-decoration: no= ne; cursor: pointer;">+36 30 462 0500</a></span></span></em></span><br></td= ></tr></tbody></table><div><br></div><div><span style=3D"color: #003366;" d= ata-mce-style=3D"color: #003366;"><span style=3D"font-size: xx-small;" data= -mce-style=3D"font-size: xx-small;"><span>[EN] This message and any attachm= ents are confidential and privileged and intended for the use of the addres= see only. If you have received this communication in error, please notify t= he sender by replay e-mail and delete this message from your system. Please= note that Internet e-mail guarantees neither the confidentiality nor the p= roper receipt of the message sent. The data deriving from our correspondenc= e with you are included in a file of  ITSMART Ltd which exclusive purp= ose is to manage the communications of the company; under the understanding= that, in maintaining said correspondence, you authorize the treatment of s= uch data for the mentioned purpose. You are entitled to exercise your right= s of access, rectification, cancellation and opposition by addressing such = written application to address above.</span></span><span style=3D"font-size= : 12pt;" data-mce-style=3D"font-size: 12pt;"> </span></span></div><div= ><span style=3D"color: #003366; font-size: xx-small;" data-mce-style=3D"col= or: #003366; font-size: xx-small;">[HUN] Ez az =C3=BCzenet =C3=A9s annak b= =C3=A1rmely csatolt anyaga bizalmas, a nyilv=C3=A1nos k=C3=B6zl=C3=A9stol v= =C3=A9dett, kiz=C3=A1r=C3=B3lag a c=C3=ADmzett haszn=C3=A1lhatja fel. Ha = =C3=96n nem az =C3=BCzenet c=C3=ADmzettje, =C3=BAgy k=C3=A9rj=C3=BCk v=C3= =A1lasz=C3=BCzenetben =C3=A9rtes=C3=ADtse errol a felad=C3=B3t =C3=A9s t=C3= =B6r=C3=B6lje az =C3=BCzenetet a rendszerbol. K=C3=A9rj=C3=BCk vegye figyel= embe, hogy az email-en t=C3=B6rt=C3=A9no inform=C3=A1ci=C3=B3tov=C3=A1bb=C3= =ADt=C3=A1s kock=C3=A1zattal j=C3=A1rhat, nem garant=C3=A1lja sem a csatorn= a bizalmass=C3=A1g=C3=A1t, sem a k=C3=A9zbes=C3=ADt=C3=A9st. A lev=C3=A9l a= z ITSMART Informatikai Kft. kommunik=C3=A1ci=C3=B3j=C3=A1nak eszk=C3=B6ze, = az adatokat kiz=C3=A1r=C3=B3lag erre a c=C3=A9lra haszn=C3=A1ljuk. Jogosult= t=C3=A1j=C3=A9koztat=C3=A1st k=C3=A9rni szem=C3=A9lyes adatai kezel=C3=A9s= =C3=A9rol, k=C3=A9rheti azok helyesb=C3=ADt=C3=A9s=C3=A9t, illetve t=C3=B6r= l=C3=A9s=C3=A9t =C3=ADr=C3=A1sos k=C3=A9relemben a fenti e-mail c=C3=ADmen.= </span></div><p></p></div></div></body></html> --=_b3dd23ad-95be-449f-867c-dcd17f58f735--

2 1

Hosted-engine --deploy 4.1.6 - Web GUI Down
by Mat Gomes 20 Sep '17

20 Sep '17

--_004_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_ Content-Type: multipart/alternative; boundary="_000_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_" --_000_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Guys, I'm attempting to rebuild my environment for production testing, We've mult= iple locations NY and CH with a 2+1arbiter setup, this will run Geo-Replica= tion. All goes smoothly, VM starts and I'm able to login onto it via SSH, I usual= ly stop Firewalld as it usually causes the Web-GUI to be unreachable This time around the Web GUI can't be reached, I've done more than 10 deplo= yments and tests but I'm not sure what's causing this, 4.1.5 was deployed s= ame exact way aswell. it all seems to be running fine, port 80 is listening= , I've seem most of the logs but nothing stands out. Please help. Steps: #Puppet Installation/deployment. class ovirt { package { 'centos-release-gluster310': ensure =3D> installed, } package { 'ovirt-release41-4.1.6-1.el7.centos.noarch': ensure =3D> installed, source =3D> "http://resources.ovirt.org/pub/yum-repo/ovirt-release4= 1.rpm", provider =3D> 'rpm', install_options =3D> ['--nosignature'], } $ovirt_packages =3D ['system-storage-manager','vdsm-gluster','ovirt-hos= ted-engine-setup','ovirt-engine-appliance','glusterfs-server',] package { $ovirt_packages: ensure =3D> installed, require =3D> [ Package['centos-release-gluster310','ovirt-release41= -4.1.6-1.el7.centos.noarch']], install_options =3D> ['--disablerepo=3Depel'], } service { 'glusterd': ensure =3D> running, enable =3D> true, require =3D> [ Package['vdsm-gluster'], File['/etc/glusterfs/gluste= rd.vol']], } file { '/etc/glusterfs/glusterd.vol': ensure =3D> file, source =3D> 'puppet:///modules/ovirt/glusterd.vol', owner =3D> root, group =3D> root, mode =3D> '0644', } <- All works fine. Once everything is up/running deployed, peer-probed, ssh-keys are created/s= hared, volumes settings are set and started: gluster volume create engine replica 3 arbiter 1 host1:/data/glusterfs/vol1= /engine host2:/data/glusterfs/vol1/engine host3:/data/glusterfs/vol1/engine gluster volume set engine cluster.quorum-type auto gluster volume set engine network.ping-timeout 10 gluster volume set engine auth.allow \* gluster volume set engine group virt gluster volume set engine storage.owner-uid 36 gluster volume set engine storage.owner-gid 36 gluster volume set engine server.allow-insecure on gluster volume start engine At last hosted-engine --deploy is ran, --> https://pastebin.com/QXEwmSwT = <-- answer file VM starts but no WEB GUI. Let me know if you need more info Best Regards, [cid:image001.png@01D3312F.03099560] Mat Gomes | Assistant Vice President, IT t. 212-531-8594 m. 954-254-1294 e. mgomes(a)clearpoolgroup.com <mailto:mgomes@clearpoolgroup.com>w. clearpoolgroup.com<www.clearpoolgroup.= com> --_000_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal">Hi Guys,<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">I’m attempting to rebuild my environment for p= roduction testing, We’ve multiple locations NY and CH with a 2+1a= rbiter setup, this will run Geo-Replication.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">All goes smoothly, VM starts and I’m able to l= ogin onto it via SSH, I usually stop Firewalld as it usually causes the Web= -GUI to be unreachable<o:p></o:p></p> <p class=3D"MsoNormal">This time around the Web GUI can’t be reached,= I’ve done more than 10 deployments and tests but I’m not sure = what’s causing this, 4.1.5 was deployed same exact way aswell. it all= seems to be running fine, port 80 is listening, I’ve seem most of the logs but nothing stands out.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Please help.<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Steps:<o:p></o:p></p> <p class=3D"MsoNormal">#Puppet Installation/deployment.<o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">class ovirt {<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    package { 'cento= s-release-gluster310':<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  ensure =3D> installed,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    }<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    package { 'ovirt= -release41-4.1.6-1.el7.centos.noarch':<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  ensure  =3D> installed,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  source =3D> "http://resources.ovirt.org/pub/yum-repo/ovirt-= release41.rpm",<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  provider =3D> 'rpm',<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  install_options =3D> ['--nosignature'],<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    }<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    $ovirt_packages = =3D ['system-storage-manager','vdsm-gluster','ovirt-hosted-engine-setup','o= virt-engine-appliance','glusterfs-server',]<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    package { $ovirt= _packages:<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  ensure =3D> installed,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  require =3D> [ Package['centos-release-gluster310','ovirt-releas= e41-4.1.6-1.el7.centos.noarch']],<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  install_options =3D> ['--disablerepo=3Depel'],<o:p></o:p></span>= </p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    }<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    service { 'glust= erd':<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  ensure =3D> running,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  enable =3D> true,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  require =3D> [ Package['vdsm-gluster'], File['/etc/glusterfs/glu= sterd.vol']],<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    }<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    file { '/etc/glu= sterfs/glusterd.vol':<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  ensure  =3D> file,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  source  =3D> 'puppet:///modules/ovirt/glusterd.vol',<o:p></= o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  owner   =3D> root,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  group   =3D> root,<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">     &nbsp= ;  mode    =3D> '0644',<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"font-size:8.0pt;font-family:Consolas;= background:silver;mso-highlight:silver">    }</span><span st= yle=3D"font-size:8.0pt;font-family:Consolas">  <- All works fine.<o= :p></o:p></span></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Once everything is up/running deployed, peer-probed,= ssh-keys are created/shared, volumes settings are set and started: <o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume create engine replica 3 arbiter 1 host1:/data/glusterfs/v= ol1/engine host2:/data/glusterfs/vol1/engine host3:/data/glusterfs/vol1/eng= ine<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine cluster.quorum-type auto<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine network.ping-timeout 10<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine auth.allow \*<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine group virt<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine storage.owner-uid 36<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine storage.owner-gid 36<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume set engine server.allow-insecure on</span><o:p></o:p></p> <p class=3D"MsoNormal"><span style=3D"background:silver;mso-highlight:silve= r">gluster volume start engine</span> <o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">At last <span style=3D"background:silver;mso-highlig= ht:silver"> hosted-engine --deploy</span> is ran,   --> <a href=3D"https:/= /pastebin.com/QXEwmSwT"> https://pastebin.com/QXEwmSwT</a> <--   answer file<o:p></o:p>= </p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">VM starts but no WEB GUI.<o:p></o:p></p> <p class=3D"MsoNormal">Let me know if you need more info<o:p></o:p></p> <p class=3D"MsoNormal"><o:p> </o:p></p> <p class=3D"MsoNormal">Best Regards,<o:p></o:p></p> <table class=3D"MsoNormalTable" border=3D"0" cellspacing=3D"0" cellpadding= =3D"0" style=3D"border-collapse:collapse"> <tbody> <tr> <td width=3D"400" valign=3D"top" style=3D"width:300.0pt;padding:0in 0in 0in= 0in"> <p class=3D"MsoNormal" style=3D"margin-bottom:8.0pt;line-height:107%"><span= style=3D"font-size:10.0pt;line-height:107%;font-family:"Helvetica&quo= t;,sans-serif;color:black"><img border=3D"0" width=3D"213" height=3D"50" id= =3D"Picture_x0020_1" src=3D"cid:image001.png@01D3312F.03099560"><o:p></o:p>= </span></p> </td> </tr> <tr> <td width=3D"400" valign=3D"top" style=3D"width:300.0pt;padding:0in 0in 0in= 0in"> <p class=3D"MsoNormal" style=3D"margin-bottom:8.0pt;line-height:107%"><b><s= pan style=3D"font-size:10.0pt;line-height:107%;font-family:"Helvetica&= quot;,sans-serif;color:black">Mat Gomes</span></b><span style=3D"font-size:= 10.0pt;line-height:107%;font-family:"Helvetica",sans-serif;color:= black"> | Assistant Vice President, IT<br> </span><b><span style=3D"font-size:10.0pt;line-height:107%;font-family:&quo= t;Helvetica",sans-serif;color:#00B0F0">t</span></b><span style=3D"font= -size:10.0pt;line-height:107%;font-family:"Helvetica",sans-serif;= color:black">. 212-531-8594</span><b><span style=3D"font-size:10.0pt;line-h= eight:107%;font-family:"Helvetica",sans-serif;color:#00B0F0">&nbs= p; m</span></b><span style=3D"font-size:10.0pt;line-height:107%;font-family:&= quot;Helvetica",sans-serif;color:black">. 954-254-1294<br> </span><b><span style=3D"font-size:10.0pt;line-height:107%;font-family:&quo= t;Helvetica",sans-serif;color:#00B0F0">e</span></b><span style=3D"font= -size:10.0pt;line-height:107%;font-family:"Helvetica",sans-serif;= color:black">. <a href=3D"mailto:mgomes@clearpoolgroup.com"><span style=3D"font-size:11.0p= t;line-height:107%;font-family:"Calibri",sans-serif">mgomes@clear= poolgroup.com<br> </span></a></span><b><span style=3D"font-size:10.0pt;line-height:107%;font-= family:"Helvetica",sans-serif;color:#00B0F0">w</span></b><span st= yle=3D"font-size:10.0pt;line-height:107%;font-family:"Helvetica",= sans-serif;color:black">. <a href=3D"www.clearpoolgroup.com"><span style=3D"font-size:11.0pt;line-hei= ght:107%;font-family:"Calibri",sans-serif">clearpoolgroup.com</sp= an></a><o:p></o:p></span></p> </td> </tr> </tbody> </table> <p class=3D"MsoNormal" style=3D"margin-bottom:8.0pt;line-height:107%"><span= style=3D"font-size:10.0pt;line-height:107%;font-family:"Helvetica&quo= t;,sans-serif;color:black"><o:p> </o:p></span></p> <p class=3D"MsoNormal"><o:p> </o:p></p> </div> </body> </html> --_000_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_-- --_004_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_ Content-Type: image/png; name="image001.png" Content-Description: image001.png Content-Disposition: inline; filename="image001.png"; size=4966; creation-date="Tue, 19 Sep 2017 14:17:47 GMT"; modification-date="Tue, 19 Sep 2017 14:17:47 GMT" Content-ID: <image001.png(a)01D3312F.03099560> Content-Transfer-Encoding: base64 iVBORw0KGgoAAAANSUhEUgAAANUAAAAyCAYAAADLAZ65AAAABGdBTUEAALGPC/xhBQAAAAlwSFlz AAAOwgAADsIBFShKgAAAABl0RVh0U29mdHdhcmUAcGFpbnQubmV0IDQuMC4xMkMEa+wAABLjSURB VHhe7VwNlBxVla4EZf0B4v7gAv4Ai6JmSWaqqieJAkbQ9eyuunsWMjMooGtED2dZRJa4uO7qAMe/ EFkSknS96pkQQhJxekLCguRvQhIkBFhXWREQkplJAiGCgSABkjBJ19vv3veqq3u6uru6ujuOO/Wd c091Vb1336tX93vv3vdetZEgQYIECRIkSJAgQQRks/KYvh2e2Ttw5Ot9A94SyAOQp7ID3o7e7blt vQPe1r4hb1l20Pu3O3d40zZJ+SadNUGCBIW4a5d3SnZ77gYQaFffoJSRZcB7HgSbc8c27wytKkGC sY3sPjmhb7s3NzuYOxRKmsjiDWM0W5R9XJ6kVSdIMPbQO3D4U9mB3HPhJIknvQO5l3q3HblEF5Eg wdhAV5ccj1GlC+KFEaNuYb3eLZs2/QHFW6Y707DdbxiW+JoxMXusvppgLOEDPcezDZCY4gJ9tTqY UIM5J5QMDRbEaHf+wRDLFr9EY0ocXzKmzDtBX00wljDNeZeyAbaDlfpqdcDdmxNGgKbJQG5Zl5Tj dfGjFwmpEsQhVe+QdylcvnDjb6KgzH/VVRi9SEiVoFZS9e70Tkes82qY0TdfvEPZAWnrqoxOJKRK UCup+rZ7s8MN/uhIdtD7kSHlOF2d0YeEVAlqJdWKIe9UGPb+MINvtqDcQ33bPFNXZXQiIVWCWDHV gPe530tMtd27RlehOZi++C2Glfm0Ybk/QGPcYVjiJ2iYrGE5Nxl2ZobxkZ7jdcryqJVUk+e8HWX+ A8r4oSoLZVriR5DvQM4zpnfVMevZNR46pkK6oHsp6nQPjnfifJ5hikuMSek/1gkVqL5UppU530jd +h59FYBnYHdP4eukj/QSKL2d+Tz0LkSb3WK0Z48x2kSK05niw/l0PqZkTkf+K5DWwfEupPsvHBdB rkG9PqhTxcPEBccZZvpC9a7QjvSsplhmmM51XBeqW1RMybwbeS6HLqrnKgjVsxv1vspIpf9CpyqP OKQiIK76fpjhh8mqnVLes1vKe/dIufo3EBzpfNUuKVcMhecpkYHcEtkst29a95+gAUEkd1/QGKHy KmSBYfaconOWIiqpyKC5TPFygf4w2WGYmctxfLPOWR1kQJbTEdSljFjO6zCWefj9Z5wv5bbh/Ii+ fzVfI0zf9CZc/5XKI36N49twvAxG9qJOS/KEcSp1SuK/1bnYzumIkFbmXKRdX6C7VCw3h+MGw3bO 1qVGQ9v8P4Xe2cj7uyJ9JSKe5I6kErksMQnpVqIuh8N1sOSQbrWRWtimc5UiLqnIwLMDuQV9g6Uj FhGFyLNxn5QPvi7lQwfKy1bc3/SylGuel/LOHcV6fIHb15uVMnpPUwtM8ddopN8UNJoSSwxDDqKn Gy65x+RzPssGMxJRSNUCw7HEMyN0+mUegJS+VEs8YExCD1oNRBB6kbbwivM7R6DjIOQNnBffs92d RlumNTKpaGQZaXiW+6sSUrUsfgfOZ0NGpiXDpLocxDmRqeAe2sB0b2SvoRpoVLXFs0X5WfCMSv+I dkSbkCfw4fQ7tQYF7oTcWUhzqCi9RW0oDildJfWEbnF9KEnjkopAxFox6P07XMEcGf9KkKJ/ryJK GIGqCeXb8KIa2ZhMA57XO5i7uWmEsp0vFRmH5b7ADdUK4yK3jDDtpreiAS0IuWNBz0zGZzmzSlyc aqRKpf8GeQ8U6HkRcoNhpW3D7kLPbowzpsOVaRMfBeHTuEeGp9La7lBF98MSZ+AZnirQTYbVA7J8 wmi5+R1IMc6Y2HWs0eq8H9e/Bnm6IO0eHt2qk+pVyH7126UOgEa6c5hANJoGpNqB+z34rQjM7UU9 fPoiHE9mY+xC203t/nOMZDSqrkU633DJ+FdVdLdpp4ItXtPpSfDu3Ou4c1CEHGdMWXoCP7vt3gYJ CGNhVPU7KHKvbTE3f0+V/XPIZfj9XuV+o55ti08yUhjpTPHT4rTO0hIXvR5S+Vi2zfv0yme94bhk Gimk55493pGlT3tf0EU0Hrbzt2g4ZUCqF1pgfGB25ZiJ3EQVmxQYCgyiEJVI1bLgL1HWK/o+vbwl JXHNSEzOnIn0W1UeLpOMe4K+G4DIRiOOn84WDxnmwg/pu+HgGBKEtl0avSjf7vyzlSOVL5bYCQNL 6RQKRaSiNhWaJEhrO3+lU5UBDDeV/ju0T+A1UNzFLuQIULm247t7OZwv43dTCa1OC/Q9WqD7EX4/ Jp4zeJ/USXy1ootI9WyFl5IPFZA3hdG4EPWS6qIt3imd/d6Wz2+WMo3RZctr4USJKuQu9mCkmnm/ lJ0bvMeg+0xdVOMwGcO/6tmoIT24eDCgqPEaGtV0bgxeBIyA/Hof5UhF+wAt52F9jwg1u/LLKwD1 2JbTr/JCUmJ+0QjJ5Cgi3noO3KNhHIwSQTncLj+/kvKkssRew+wuJWwxqXRadxuO79UpqqPVmQgd z6n83Mbf0HcUJt/+duj8Wf6+5agJkiigzshytwR5xR34TXEyncMjcP5ep6wOIraFd6ye8TBGxOn6 Tn2kmrH+jUmd/bndnRuIAEq+9FMp526Tcj3ipDDSlJP7XpFy4YCUX9kS6GLp917qWDf8UV1kY2CL 7+Uf2nJuLTLQKCAjs9270agU6L8OI/+OvlOeVK00MmpXk4J2csNqwVlz4Sa5Kg6zBI12gaFyXOjr FoNG25KA5FFARmmLjKp3XiqMVOI/9J1ijCQVuYlWd+2L9TSqWdpds8Tz7Hr5SLmXFuh/EMfSkawS WmDwVkkcRqNq+DNVAo2s+c5VPJzvmOOSqmOTd1ZHf25vEQFGyFcflvLGJ6W8dRfirRekXLcPMdfv cAThVuH8tmek/OGvpbz6ESkvCsmfl35vf+ea4XN10fWBen1/poh6molV3IaKoEYcMcKVI5Ul1uky KUaKN31sORcjr+9SaSNA+Za4V1/DqIuYJQ5SC98DPXuVHpZyMdVBHk3CUEoqoe/UBopRKKZSesi9 6+Tr75v3RyCbGmlUO57D12uF6fyz1q3ric4qynLJSFBnZHIsSDoOcyxOiEOqS9Z57+xcn9sVSoAm SUe/t7fzvgZ8FWx3fwwvRMdSzvLobl9EhJGKZ+R8Irv34xh9erwQFH8FbutmFWSzbnWNYpdqMVp5 gJxun9LDUo5UNFkT7s4VkoqMPtUTf6GepuH9UcB2snxt0rx3Q6+eJEF9qF5x0DrvROTXsS3pcr+t 79QOInxej/gmX6uVVLRTHLFOf5jhN1s61ud+0f64rO8bJZvXfNQDm+KL+mrjEEaqVNrENT37JP6X A9u4Yrsq3rDcQRwnGObCyfitZhNpxKqnk7AzFIBrYyhLqufLrtMVkcolF0utf8WBMvw9rIum58lF Z6Lp+pnuzTpl7aARz3YfYj0US9IMYVyQa+rXiRaJCbWSqmO9d1mYwR8tgct5va5KPKjZLv3AMd2H SggjlSk+jnLLL3zGERoxWhediN/nQLRLmJnL5cWF3X02yKB1VSLV/AikEk9FWmsqB5poobUvpWsP t2Vbhtxf/fyZL+uUtYPdS1e5zBQTl3Nno4BdUl77I12P8LVaSdXe780MM/ajJR3rj3TpqsRDIala C2ZsGoVqpOJpeF7krU9ssc9oW4BekheSfd3/yeXFBW3naRyptvEaX1wwqcRjWhdNs0/A8Qt8TlKP l0H1tF0dC4FUbZn4M8xqWcJf69zK12p2/7rY/VsbZvDNFsRVP2vP1uv+icD9s8RX9NXGIYxU1nwL 13z3L8M9Zb3iTyPT1hoyDPU8q/haXKTcC1UdWeojlS2e45E0LmgGMz9LJ4b4eS3xSXXO9QhmXGuF 2malCEu7L0wRf3aZCOnXyXZ/zNfiTFTMWO2d2LEhNxRm+M2Sjg3eC5/rl9U3M1aDlT4fD6pdMQqA j9JERRAY04p8vImKMFDsYYvfsm5LDPAugnigWcTluo4k9Y5UBzmWjAu1DqTek6kN0+w5lfWyfheu Vo1LIT5ID5FJPafHC8BxYYor83polwohDqkIMzZ4Ezv7vd+GESAvtOb0C8gTkO2QHZCd+kjnT0Ie hTwIuQ8SpgMCQr3SuXb4I7ro+qDcCH8G6eWihdtGIIxUBN4symUeMMzu9+mrtYH25tFCNUl+Zo2m 1J2Nqkxy3WpYwCwE7cy2XUVOJXWOVJx2vr5TG9rbaUPwUq0nZ6T0lLoaYf5HX38Nvy2+Xhuo8yiM q6meT8eK/9TzPqJ1DKMTOYuvxyUV4WIQCwb/TBEJNkN+CSHi7K5BiGyPQXgnRSA0lX7hWq8xhPJB sUf+od3aF395Jkp8Hw22mcVygqC5HKn4k5J87LMax9pGK97VjpFI6X7NMBcExKRPHvzJCoplal17 U4u/i5XuvDSAVO4rOK/d8K30NH5G1iN2F7mRphusMdHaX63/WNXinIZ8hetxJDm8w6t0iuhQ64Z+ XTbqq/WRinDxGu/k9v7c/Z0bNSmehYSRJqpQ/schIGdHf+7RGZu8eL16JahVdfXJBQfmmcuju4FI Z2euRV5txLxXLZg6LkcqNUvkxxu0veaGfFxUDRTw226vysv5M0UdAW3ApU8v8vedNdG3KfFIdxXq FmwsVlI/qTi9u40NOSpoDyPtClH5MUplrtB3FHjLlng8f586t6idIr0PSxRsU0IM5Me6HJdW259Y APIU/P1/NEq1ukHHXy+pCMcI7zMfWusNz9hVQI46pP0ZKVs2ekfenJH/qItoPGjRrnBGzna/W7XX U+4H7cr2RwXK167vKpQjFYHcA9/15JcqukvSjISKI4J9f5Z4Qu84LwZtBwo2mVK6h6tuqJ3W+1be x5gnFO258xdc6yUVf37hT6DsKtobFwqQm9aL/K1YnM+9mzujkVDLCGrPHnWK9KFj2CbjQqQWEVn1 3kvOt1Xvqfx2cA31Za+jEkmpU+2mzdh6tKP2ynxP31Soi1T00aCbu85wvZzhSvm2JVLaD0gZl1xE pikPSXn8UilJH/R60L/A6GrSf/5Z6Wvw0IogqlEH0KizjKnO+3mGjUCGQsbZmqY/yBwqSEtT418v eQGVSEWwxKcghZ9+7EE9voXfH8yXSbvlyQhpm49vmCrtEKT8rhIyNssJXBsO6tOL0Yt+IpjAQLxC BmZn/gV1pA8JdVr3OeSN8ulH1IkK1IPL0Gs4tDcRBmaKzxjnLNe7PmA//qcftHAdlE35N1eId2HY zmeRJmhH6hCIIK2LJuJcudbqs52puH4LrumNs5z2yfynH5S2aN8jd3bokMQX1SI36kigOlO8atLX 0/mOhzyO5fn35iM2qbrkeEN4aWX8xXLsIinPWC3luT+X8oIqsdWFiKU+9qiUZ66T8i2LS3WxiNwK HBs3Y1YI3mYCAuQbQQv13urboYB0wb39MPqZ+QYvRDVSEfhL2JCP69T6EwJwPRIWy/1Gy23v0hrK gyZB1EZT/8VrYWNBnJKf9QqER5K0yRMhjSQVjbK0z86fscsL14U2I+N63kCVqM6qJ8JePNpdTxuJ dxflZx0gML+7ke3I5fZDTtY6FIgUtJfS38QbCKWnjxTRsY2sJ8owxZwSQhFik0rk5pQYf4iMy0h5 3O1SnrxSytN+IuXp90JwPGWVlCdgRBqP+2H5SkTkljGRm4GzqLcktw4vorDhSoRdmuWQCqNFBFIR aOKBth3lv68qJ2InesMv43f0ToXSmumZ0B18sBgmZNT0Pwx+TNhoUvl7BG1nCs434zyss9BCnRdN +qTPR46I8S3AyxXOTdCv3epywqP8ZaEk8NGSaUW6e5AuGDFLBOTiDxUrfPYfi1SudylGqXDjb6aI I9fqGjQH/Och4gLDhI9siRVokPtgmHBL3FuMVvpKeEQPFwbaqcyfecMVizIRMTF7HF7QDOShz87v 4jLVn7PcwDFGLWQaCSqfd0mIb0LvMhypl15tmPQ/GxhpyfUqBHUCtkMbjs/jXet5YERuYWKch+PZ oXEOoRypGNDBC9XuLBxvh1CcuAZHuLiZK/hjzHpA7i3HyfQHOs7d0E1LGFlcu95oEx/H7+jtOJVm CJ1/4rqpOlJdb8PxGmOynjavBDUpdR6LP81eERnvdJDq1VCjb7p4h4zu4dH9Z5pjGRVJlaA8XO/G cIM/SiK8O0JjmQS/fySkiglHngbD3h9q8M0WgZEqPcr/THMsIyFVHXC8i2HgXqjhN00QwzneLF2D BKMRCanqhJP7QbjxN0u8JaP6P9QTJKSqG7xOlZsfToBGi/fjpi0AJ2gcElI1ADRypI98C+TiHRUN F5Hz4GbOM9qb9GeaCRqMLvpPPPpv9WshV1bdgpWgAsThT2I02R1KjLgivBeNjFf8R5UJEowpuHIC 77IQuQMlBKlFhPcGdAij2ytemEyQYMxigTwJpLgeMgSChBMnVDDSESkXy+ifCSRIMKagNtxOglwJ WQR5wHBzT0F24PfTINFWHG/H8WocU0Y2iZsSJEiQIEGCBAkSJEjw/wyG8X/EO188wIQwDwAAAABJ RU5ErkJggg== --_004_F305CD02A7055049A8ADC76AAD31F44A1D9C8288exchange2010dim_--

4 3