- Users - oVirt List Archives

Re: [ovirt-users] ovirt-ha-agent service errors
by Vadim 23 Aug '17

23 Aug '17

Hi, All after upgrade to 4.1.5 no any error messages ;-) Thanks to TEAM Пнд 21 Авг 2017 14:29:49 +0300, Vadim <ovirt(a)qip.ru> написал: > Hi, All > > ovirt 4.1.4 fresh install on two hosts with hosted-engine on both. > gluster volume is replica 3 with two vdsm hosts and one VM under esxi > > working only one vm for HE > > sometimes have such errors in ha-agent > > # service ovirt-ha-agent status > Redirecting to /bin/systemctl status ovirt-ha-agent.service > ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent > Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) > Active: active (running) since Thu 2017-08-17 11:43:44 MSK; 3 days ago > Main PID: 2534 (ovirt-ha-agent) > CGroup: /system.slice/ovirt-ha-agent.service > └─2534 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon > > Aug 21 00:29:11 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 00:48:32 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 01:12:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 02:12:09 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 03:55:08 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 08:14:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 08:25:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 08:46:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 09:20:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > Aug 21 09:21:40 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds > > > in agent log: > > MainThread::INFO::2017-08-21 09:21:40,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:c7b33006-e7c7-4e39-8d80-2301149ac8f9, volUUID:184f9e45-ab1b-44b8-8a68-238042dba1a7 > MainThread::INFO::2017-08-21 09:21:40,594::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e7a3c173-9f87-4f6c-a807-63118b9b7cb2, volUUID:92317b81-1bb0-43e6-b029-8931aa5d0af0 > MainThread::INFO::2017-08-21 09:21:40,716::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE > MainThread::INFO::2017-08-21 09:21:40,749::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/localhost:_ovha/7b6badfb-4986-4983-9f62-ae55da33d15e/images/e7a3c173-9f87-4f6c-a807-63118b9b7cb2/92317b81-1bb0-43e6-b029-8931aa5d0af0 > MainThread::INFO::2017-08-21 09:21:40,787::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert > MainThread::INFO::2017-08-21 09:21:40,792::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE > MainThread::ERROR::2017-08-21 09:21:40,792::states::606::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 300 seconds > MainThread::INFO::2017-08-21 09:21:40,792::states::430::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost > MainThread::INFO::2017-08-21 09:21:40,796::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUp'> > MainThread::INFO::2017-08-21 09:21:40,800::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1503296500.8 type=state_transition detail=EngineUpBadHealth-EngineUp hostname='kvm03' > MainThread::INFO::2017-08-21 09:21:40,853::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUpBadHealth-EngineUp) sent? sent > MainThread::INFO::2017-08-21 09:21:40,853::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM > > > There is no errors in web on dashboard > How is it possible to find the reason for an error? > > -- > Thanks, > Vadim > _______________________________________________ > Users mailing list > Users(a)ovirt.org > http://lists.ovirt.org/mailman/listinfo/users > -- Thanks, Vadim

1 0

Re: [ovirt-users] VM with attached host USB device failed to run
by Дровалев Роман 21 Aug '17

21 Aug '17

Sorry, did not notice that the letter directly from you. In the attachment vdsm.log 16.08.2017 14:36, Yaniv Kaul пишет: > No need to share it privately with me - share it with the users > mailing list. > I see that it's the latest VDSM. > It'd be interesting to get the VDSM log as well. > Y. > > > On Wed, Aug 16, 2017 at 2:07 PM, Дровалев Роман > <drovalev(a)miac.kaluga.ru <mailto:drovalev@miac.kaluga.ru>> wrote: > > Thanks for the answer. > > >Would you like to share with us your version of oVirt? > > ---- Manager ---- > oVirt Engine Version: 4.1.4.2-1.el7.centos > > #engine-config -g UserDefinedVMProperties > UserDefinedVMProperties: version: 3.6 > UserDefinedVMProperties: version: 4.0 > UserDefinedVMProperties: hostusb=[\w:&]+ version: 4.1 > > ---- host ----- > OS Version: > RHEL - 7 - 3.1611.el7.centos > OS Description: > CentOS Linux 7 (Core) > Kernel Version: > 3.10.0 - 514.26.2.el7.x86_64 > KVM Version: > 2.6.0 - 28.el7.10.1 > LIBVIRT Version: > libvirt-2.0.0-10.el7_3.9 > VDSM Version: > vdsm-4.19.24-1.el7.centos > SPICE Version: > 0.12.4 - 20.el7_3 > GlusterFS Version: > [N/A] > CEPH Version: > librbd1-0.94.5-1.el7 > > > > 16.08.2017 12:54, Yaniv Kaul пишет: >> >> >> On Wed, Aug 16, 2017 at 12:05 PM, Дровалев Роман >> <drovalev(a)miac.kaluga.ru <mailto:drovalev@miac.kaluga.ru>> wrote: >> >> Hello, >> >> I connected a USB to the VM, writes this error: VM Win2012 is >> down with >> error. Exit message: local variable 'device' referenced >> before assignment. >> >> >> Would you like to share with us your version of oVirt? >> >> >> >> I found this bug - >> https://bugzilla.redhat.com/show_bug.cgi?id=1261075 >> <https://bugzilla.redhat.com/show_bug.cgi?id=1261075> >> >> >> This is a duplicate of a bug that was fixed >1 year ago. >> Y. >> >> >> >> >> Does anyone have a working USB forwarding in a virtual >> machine? If >> "YES", how did you solve this problem? >> >> If this problem is not solved, unfortunately, the ovirt will >> have to >> completely abandon. (( >> >> >> >> >> >> >> >> _______________________________________________ >> Users mailing list >> Users(a)ovirt.org <mailto:Users@ovirt.org> >> http://lists.ovirt.org/mailman/listinfo/users >> <http://lists.ovirt.org/mailman/listinfo/users> >> >> > > -- > С уважением, Дровалев Роман > инженер ГБУЗ "МИАЦ", > Калужской области. > тел. 4842 705 004 > > -- С уважением, Дровалев Роман инженер ГБУЗ "МИАЦ", Калужской области. тел. 4842 705 004

2 1

ovirt-ha-agent service errors
by Vadim 21 Aug '17

21 Aug '17

Hi, All ovirt 4.1.4 fresh install on two hosts with hosted-engine on both. gluster volume is replica 3 with two vdsm hosts and one VM under esxi working only one vm for HE sometimes have such errors in ha-agent # service ovirt-ha-agent status Redirecting to /bin/systemctl status ovirt-ha-agent.service ● ovirt-ha-agent.service - oVirt Hosted Engine High Availability Monitoring Agent Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-agent.service; enabled; vendor preset: disabled) Active: active (running) since Thu 2017-08-17 11:43:44 MSK; 3 days ago Main PID: 2534 (ovirt-ha-agent) CGroup: /system.slice/ovirt-ha-agent.service └─2534 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-agent --no-daemon Aug 21 00:29:11 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 00:48:32 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 01:12:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 02:12:09 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 03:55:08 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 08:14:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 08:25:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 08:46:05 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 09:20:06 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds Aug 21 09:21:40 kvm03 ovirt-ha-agent[2534]: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine ERROR Engine VM has bad health status, timeout in 300 seconds in agent log: MainThread::INFO::2017-08-21 09:21:40,314::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:c7b33006-e7c7-4e39-8d80-2301149ac8f9, volUUID:184f9e45-ab1b-44b8-8a68-238042dba1a7 MainThread::INFO::2017-08-21 09:21:40,594::ovf_store::103::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(scan) Found OVF_STORE: imgUUID:e7a3c173-9f87-4f6c-a807-63118b9b7cb2, volUUID:92317b81-1bb0-43e6-b029-8931aa5d0af0 MainThread::INFO::2017-08-21 09:21:40,716::ovf_store::112::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) Extracting Engine VM OVF from the OVF_STORE MainThread::INFO::2017-08-21 09:21:40,749::ovf_store::119::ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore::(getEngineVMOVF) OVF_STORE volume path: /rhev/data-center/mnt/glusterSD/localhost:_ovha/7b6badfb-4986-4983-9f62-ae55da33d15e/images/e7a3c173-9f87-4f6c-a807-63118b9b7cb2/92317b81-1bb0-43e6-b029-8931aa5d0af0 MainThread::INFO::2017-08-21 09:21:40,787::config::431::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Found an OVF for HE VM, trying to convert MainThread::INFO::2017-08-21 09:21:40,792::config::436::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config::(_get_vm_conf_content_from_ovf_store) Got vm.conf from OVF_STORE MainThread::ERROR::2017-08-21 09:21:40,792::states::606::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine VM has bad health status, timeout in 300 seconds MainThread::INFO::2017-08-21 09:21:40,792::states::430::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(consume) Engine vm running on localhost MainThread::INFO::2017-08-21 09:21:40,796::state_decorators::88::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(check) Timeout cleared while transitioning <class 'ovirt_hosted_engine_ha.agent.states.EngineUpBadHealth'> -> <class 'ovirt_hosted_engine_ha.agent.states.EngineUp'> MainThread::INFO::2017-08-21 09:21:40,800::brokerlink::111::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Trying: notify time=1503296500.8 type=state_transition detail=EngineUpBadHealth-EngineUp hostname='kvm03' MainThread::INFO::2017-08-21 09:21:40,853::brokerlink::121::ovirt_hosted_engine_ha.lib.brokerlink.BrokerLink::(notify) Success, was notification of state_transition (EngineUpBadHealth-EngineUp) sent? sent MainThread::INFO::2017-08-21 09:21:40,853::hosted_engine::604::ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine::(_initialize_vdsm) Initializing VDSM There is no errors in web on dashboard How is it possible to find the reason for an error? -- Thanks, Vadim

1 0

broken authz
by Fabrice Bacchella 21 Aug '17

21 Aug '17

Since a few days, I can't connect any more to ovirt. I'm using an ldap authz. In the log I see: 2017-08-21 11:41:56,513+01 DEBUG [org.ovirt.engineextensions.aaa.ldap.Framework] (default task-14) [] Creating SearchRequest 2017-08-21 11:41:56,514+01 DEBUG [org.ovirt.engineextensions.aaa.ldap.Framework] (default task-14) [] SearchRequest: SearchRequest(baseDN='CN=Partitions,', scope=SUB, deref=NEVER, sizeLimit=0, timeLimit=0, filter='&(objectClass=crossRef)(nCName=)(nETBIOSName=*)', attrs={nCName, dnsRoot}) 2017-08-21 11:41:56,514+01 DEBUG [org.ovirt.engineextensions.aaa.ldap.Framework] (default task-14) [] getConnectionPoolEntry Entry name='authz', dn='CN=Partitions,' 2017-08-21 11:41:56,514+01 DEBUG [org.ovirt.engineextensions.aaa.ldap.Framework] (default task-14) [] Sequence ad-open-vars default domain failed due to exception: Unexpected comma or semicolon found at the end of the DN string. 2017-08-21 11:41:56,514+01 DEBUG [org.ovirt.engineextensions.aaa.ldap.Framework] (default task-14) [] Exception during sequence: LDAPException(resultCode=34 (invalid DN syntax), errorMessage='Unexpected comma or semicolon found at the end of the DN string.') at com.unboundid.ldap.sdk.DN.<init>(DN.java:629) [unboundid-ldapsdk.jar:3.2.0] at com.unboundid.ldap.sdk.DN.normalize(DN.java:1483) [unboundid-ldapsdk.jar:3.2.0] at com.unboundid.ldap.sdk.DN.normalize(DN.java:1460) [unboundid-ldapsdk.jar:3.2.0] at org.ovirt.engineextensions.aaa.ldap.Framework.getDNDomainComponent(Framework.java:235) [ovirt-engine-extension-aaa-ldap.jar:] at org.ovirt.engineextensions.aaa.ldap.Framework.getConnectionPoolEntry(Framework.java:827) [ovirt-engine-extension-aaa-ldap.jar:] Indeed it's looking for 'CN=Partitions,'. But that's not in my ovirt's configuration. So it's must be coming from AD. But I was unable to find it. I think I will need it to report it to my AD admins.

1 0

oVirt and FreeNAS
by Latchezar Filtchev 21 Aug '17

21 Aug '17

--_000_E1DA3226A946BC428B66CE34DAF3307FB1AFFD0C3Abl8esrv5aubgi_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Dear oVirt-ers, Just curious - did someone uses FreeNAS as storage for oVirt. My staging = environment is - two virtualization nodes, hosted engine, FreeNAS as storag= e (iSCSI hosted storage, iSCSI Data(Master) domain and NFS shares as ISO an= d export domains) Thank you! Best, Latcho --_000_E1DA3226A946BC428B66CE34DAF3307FB1AFFD0C3Abl8esrv5aubgi_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type content= =3D"text/html; charset=3Dus-ascii"><meta name=3DGenerator content=3D"Micros= oft Word 14 (filtered medium)"><style></style></head><body lang=3DEN-US link=3Dblue vli= nk=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal>Dear oVirt-ers,<= o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNorma= l>Just curious – did someone uses FreeNAS as storage  for oVirt.=  My staging environment is - two virtualization nodes, hosted engine,= FreeNAS as storage (iSCSI hosted storage, iSCSI Data(Master) domain and NF= S shares as ISO and export domains)<o:p></o:p></p><p class=3DMsoNormal><o:p= > </o:p></p><p class=3DMsoNormal>Thank you!<o:p></o:p></p><p class=3DM= soNormal><o:p> </o:p></p><p class=3DMsoNormal>Best,<o:p></o:p></p><p c= lass=3DMsoNormal>Latcho<o:p></o:p></p><p class=3DMsoNormal><o:p> </o:p= ></p></div></body></html>= --_000_E1DA3226A946BC428B66CE34DAF3307FB1AFFD0C3Abl8esrv5aubgi_--

4 5

Import task stuck in 'Tasks'
by Wesley Stewart 21 Aug '17

21 Aug '17

I had an issue importing VMs from an export domain I was trying out. The template would fail to copy over, and be stuck in a "paused" state where I could not delete or modify the VM. I was able to fix this by dropping into the SQL database and running: *psql engine -c "SELECT vm_guid, vm_name from vm_static WHERE vm_name = 'VM_Name_Here';"* *psql engine -c "update vm_dynamic SET status=0 where vm_guid='LongStringFromFirstCommand;"* This was able to set the status to a state where I could delete the VM, but now I still see the "import" task in the task log. I did some more database diving and found that if i ran: *SELECT * FROM audit_log WHERE vm_name = 'VM_Template_Name_Here';* These entries seem to correspond with the audit entries. Can I simply delete these entries from "audit_log"? Or would this cause problems? Or perhaps, to be safe, I should just see if I can set them to "Completed" As always, thanks!

1 0

High Availability Setup with oVirt
by Anantha Raghava 20 Aug '17

20 Aug '17

This is a multi-part message in MIME format. --------------2016BA6EE2CEFB11BE2C28C2 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, I am trying to setup ovirt clusters in two different locations and replicate the data between two sites so that in case of failure in primary site, I can bring the VM up in secondary site in quick time with minimum data loss. Can we do this oVirt? I was thinking of using DRBD, however read somewhere that oVirt plugins are still under development -- Thanks & Regards, Anantha Raghava Do not print this e-mail unless required. Save Paper & trees. --------------2016BA6EE2CEFB11BE2C28C2 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> <p>Hi,</p> <p>I am trying to setup ovirt clusters in two different locations and replicate the data between two sites so that in case of failure in primary site, I can bring the VM up in secondary site in quick time with minimum data loss. Can we do this oVirt?</p> <p>I was thinking of using DRBD, however read somewhere that oVirt plugins are still under development<br> </p> <div class="moz-signature"> <meta http-equiv="content-type" content="text/html; charset=utf-8"> <title></title> <meta name="generator" content="LibreOffice 5.0.3.2 (Linux)"> <meta name="created" content="00:00:00"> <meta name="changedby" content="Anantha Raghava"> <meta name="changed" content="2016-01-05T17:20:50.677541300"> <meta name="created" content="00:00:00"> <meta name="changedby" content="Anantha Raghava"> <meta name="changed" content="2015-12-20T09:03:26.251763811"> <meta name="created" content="2015-02-21T00:00:00"> <meta name="changedby" content="Anantha Raghava"> <meta name="changed" content="2015-12-20T09:02:11.666821134"> <style type="text/css"> @page { margin: 2cm } p { margin-bottom: 0.25cm; color: #000000; line-height: 120% } address { color: #000000 } a:link { so-language: zxx </style> <p>-- </p> <p style="margin-bottom: 0cm; line-height: 100%"><font face="Times New Roman, serif">Thanks & Regards,</font></p> <p style="margin-bottom: 0cm; line-height: 100%"><br> </p> <address style="line-height: 100%"><font face="Times New Roman, serif">Anantha Raghava</font></address> <br> <p style="margin-bottom: 0cm; line-height: 100%"><font color="#66cc00"><font face="Times New Roman, serif">Do not print this e-mail unless required. Save Paper & trees.</font></font></p> </div> </body> </html> --------------2016BA6EE2CEFB11BE2C28C2--

2 1

vdsm vds.dispatcher errors
by Gary Pedretty 19 Aug '17

19 Aug '17

--Apple-Mail=_3C58C5A6-FF18-4157-8715-87EEC5B8B53D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Why on even a new clean install of a glusterized self-hosted engine = setup do all the hosts fill the messages log with=20 journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof journal: vdsm vds.dispatcher ERROR SSL error during reading data: = unexpected eof An entry every few seconds even with no VMs setup and just the hosted = engine running. I know it is listed as a bug on redhat bugzilla page, = but is this going to be normal until version 4.1 comes out? Gary ------------------------------------------------------------------------ Gary Pedretty gary(a)ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Green, green as far as the eyes can see yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ --Apple-Mail=_3C58C5A6-FF18-4157-8715-87EEC5B8B53D Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><span style=3D"font-family: LucidaGrande;" class=3D"">Why on = even a new clean install of a glusterized self-hosted engine setup do = all the hosts fill the messages log with </span><div class=3D"" = style=3D"font-family: LucidaGrande;"><br class=3D""></div><div class=3D"" = style=3D"font-family: LucidaGrande;">journal: vdsm vds.dispatcher ERROR = SSL error during reading data: unexpected eof</div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div class=3D"">journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div></div><div class=3D"" style=3D"font-family: LucidaGrande;"><div = class=3D"">journal: vdsm vds.dispatcher ERROR SSL error during reading = data: unexpected eof</div></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><div class=3D"">journal: vdsm vds.dispatcher ERROR SSL = error during reading data: unexpected eof</div></div><div class=3D"" = style=3D"font-family: LucidaGrande;"><div class=3D"">journal: vdsm = vds.dispatcher ERROR SSL error during reading data: unexpected = eof</div></div><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;"><br class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">An entry every few seconds even with no VMs setup and = just the hosted engine running.  I know it is listed as a bug on = redhat bugzilla page, but is this going to be normal until version 4.1 = comes out?</div><div class=3D"" style=3D"font-family: LucidaGrande;"><br = class=3D""></div><div class=3D"" style=3D"font-family: = LucidaGrande;">Gary</div><div class=3D""><br class=3D""></div><div = class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty         =                     =            <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary(a)ravnalaska.net</a></div><div class=3D"">Systems Manager =                     =                     =  <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska   =                     =     /\                 =    907-450-7251</div><div class=3D"">5245 Airport Industrial = Road         /  \/\         =     907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska =  99709        /\  /    \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska          /  \/  /\  \ \/\ =   =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Green, = green as far as the eyes can see           =   yourself=E2=80=9D Matt 22:39</span></font><div style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: = Menlo;" class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""></body></html>= --Apple-Mail=_3C58C5A6-FF18-4157-8715-87EEC5B8B53D--

1 0

testing REST API
by wodel youchi 19 Aug '17

19 Aug '17

Hi; I am reading the REST API Guide, it's the first time that I am testing this. I have hard time to use OAuth authentication to make queries. I can get the token with curl and with a firefox REST module *curl --cacert ca.crt -X POST -H 'Content-Type: application/x-www-form-urlencoded' -H 'Accept: application/json' -d 'grant_type=password&scope=ovirt-appapi&username=admin%40internal&password=mypass' https://engine101.example.com/ovirt-engine/sso/oauth/token <https://engine101.example.com/ovirt-engine/sso/oauth/token> HTTP/1.1* engine log : [org.ovirt.engine.core.sso.utils.AuthenticationUtils] (default task-1) [] User admin@internal successfully logged in with scopes: ovirt-appapi But when I try to use it I get this error : *curl --cacert ca.crt -X GET -H 'application/xml' -H 'Authorization: Bearer fT0knxah-wEOyi-VdhmozKv-hz-wohVm268BBJts-MYxNZ548K0UZCSmv5nY18Z6gPiFdl-VAySjqr_N4gPGAA' https://engine101.example.com/ovirt-engine/api/vms <https://engine101.example.com/ovirt-engine/api/vms> HTTP/1.1* engine log: ERROR [org.ovirt.engine.core.aaa.filters.SsoRestApiAuthFilter] (default task-30) [] Cannot authenticate using authentication Headers: The required scope ovirt-app-api is not granted. With basic authentication, it's working, so I don't know where is the problem. Regards. <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> Garanti sans virus. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campai…> <#m_8952078234548713351_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2 1

Recovering from a multi-node failure
by Jim Kusznir 19 Aug '17

19 Aug '17

Hi all: Today has been rough. two of my three nodes went down today, and self heal has not been healing well. 4 hours later, VMs are running. but the engine is not happy. It claims the storage domain is down (even though it is up on all hosts and VMs are running). I'm getting a ton of these messages logging: VDSM engine3 command HSMGetAllTasksStatusesVDS failed: Not SPM Aug 4, 2017 7:23:00 PM VDSM engine3 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) Aug 4, 2017 7:22:49 PM VDSM engine3 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5868392a-0148-02cf-014d-000000000121, msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770' Aug 4, 2017 7:22:47 PM VDSM engine1 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5868392a-0148-02cf-014d-000000000121, msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770' Aug 4, 2017 7:22:46 PM VDSM engine2 command SpmStatusVDS failed: Error validating master storage domain: ('MD read error',) Aug 4, 2017 7:22:44 PM VDSM engine2 command ConnectStoragePoolVDS failed: Cannot find master domain: u'spUUID=5868392a-0148-02cf-014d-000000000121, msdUUID=cdaf180c-fde6-4cb3-b6e5-b6bd869c8770' Aug 4, 2017 7:22:42 PM VDSM engine1 command HSMGetAllTasksStatusesVDS failed: Not SPM: () ------------ I cannot set an SPM as it claims the storage domain is down; I cannot set the storage domain up. Also in the storage realm, one of my exports shows substantially less data than is actually there. Here's what happened, as best as I understood them: I went to do maintence on ovirt2 (needed to replace a faulty ram stick and rework the disk). I put it in maintence mode, then shut it down and did my work. In the process, much of the disk contents was lost (all the gluster data). I figure, no big deal, the gluster data is redundant on the network, it will heal when it comes back up. While I was doing maintence, all but one of the VMs were running on engine1. When I turned on engine2, all of the sudden, all vms including the main engine stop and go non-responsive. As far as I can tell, this should not have happened, as I turned ON one host, but none the less, I waited for recovery to occur (while customers started calling asking why everything stopped working....). As I waited, I was checking, and gluster volume status only showed ovirt1 and ovirt2....Apparently gluster had stopped/failed at some point on ovirt3. I assume that was the cause of the outage, still, if everything was working fine with ovirt1 gluster, and ovirt2 powers on with a very broke gluster (the volume status was showing NA for the port fileds for the gluster volumes), I would not expect to have a working gluster go stupid like that. After starting ovirt3 glusterd and checking the status, all three showed ovirt1 and ovirt3 as operational, and ovirt2 as NA. Unfortunately, recovery was still not happening, so I did some googling and found about the commands to inquire about the hosted-engine status. It appeared to be stuck "paused" and I couldn't find a way to unpause it, so I poweroff'ed it, then started it manually on engine 1, and the cluster came back up. It showed all VMs paused. I was able to unpause them and they worked again. So now I began to work the ovirt2 gluster healing problem. It didn't appear to be self-healing, but eventually I found this document: https://support.rackspace.com/how-to/recover-from-a-failed-server-in-a-glus… and from that found the magic xattr commands. After setting them, gluster volumes on ovirt2 came online. I told iso to heal, and it did but only came up about half as much data as it should have. I told it heal full, and it did finish off the remaining data, and came up to full. I then told engine to do a full heal (gluster volume heal engine full), and it transferred its data from the other gluster hosts too. However, it said it was done when it hit 9.7GB while there was 15GB on disk! It is still stuck that way; ovirt gui and gluster volume heal engine info both show the volume fully healed, but it is not: [root@ovirt1 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_ovirt-root 20G 4.2G 16G 21% / devtmpfs 16G 0 16G 0% /dev tmpfs 16G 16K 16G 1% /dev/shm tmpfs 16G 26M 16G 1% /run tmpfs 16G 0 16G 0% /sys/fs/cgroup /dev/mapper/gluster-engine 25G 12G 14G 47% /gluster/brick1 /dev/sda1 497M 315M 183M 64% /boot /dev/mapper/gluster-data 136G 124G 13G 92% /gluster/brick2 /dev/mapper/gluster-iso 25G 7.3G 18G 29% /gluster/brick4 tmpfs 3.2G 0 3.2G 0% /run/user/0 192.168.8.11:/engine 15G 9.7G 5.4G 65% /rhev/data-center/mnt/glusterSD/192.168.8.11:_engine 192.168.8.11:/data 136G 124G 13G 92% /rhev/data-center/mnt/glusterSD/192.168.8.11:_data 192.168.8.11:/iso 13G 7.3G 5.8G 56% /rhev/data-center/mnt/glusterSD/192.168.8.11:_iso This is from ovirt1, and before the work, both ovirt1 and ovirt2's brings had the same usage. ovirt2's bricks and the gluster mountpoints agree on iso and engine, but as you can see, not here. If I do a du -sh on /rhev/data-center/mnt/glusterSD/..../_engine, it comes back with the 12GB number (/brick1 is engine, brick2 is data and brick4 is iso). However, gluster still says its only 9.7G. I haven't figured out how to get it to finish "healing". data is in the process of healing currently. So, I think I have two main things to solve right now: 1) how do I get ovirt to see the data center/storage domain as online again? 2) How do I get engine to finish healing to ovirt2? Thanks all for reading this very long message! --Jim

2 3

Data domain on engine host
by Mitchell Smith 18 Aug '17

18 Aug '17

Hi List, I am in the process of deploying ovirt in a lab environment before implementing it in production. My lab consists of three compute hosts and one engine host. I am using the 4.2 nightly builds in this lab environment, mainly because I want to test the OVN functionality. I wanted to add a NFS data domain on the engine host to store VM templates and ISO images, however when I go in to Storage -> Domains -> New and try and add a NFS domain, it only gives me the three compute hosts to create storage on, I’m guessing because the engine host doesn’t have VDSM installed. Is there a way to do this, so I can make use of the considerable free space on the engine host, or do I have to add it as a compute host running VDSM, and if so how can I flag this host so no VMs get deployed on it, I only want to use the storage capacity. Any feedback or suggestions would be very much appreciated. Thanks in advance. Mitch

2 1

VM with attached host USB device failed to run
by Дровалев Роман 18 Aug '17

18 Aug '17

Hello, Please help! Does anyone have a working USB forwarding in a virtual machine? If "YES", how did you solve this problem? If this problem is not solved, unfortunately, the ovirt will have to completely abandon. (( oVirt Engine Version: 4.1.4.2-1.el7.centos Thanks.

3 2

glusterfs Error message constantly being reported
by Vadim 18 Aug '17

18 Aug '17

Hi, All ovirt 4.1.4 fresh install Constantly seeing this message in the logs, how to fix this: VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' VDSM kvm04 command GetGlusterVolumeAdvancedDetailsVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterVolumeStatvfs' VDSM kvm04 command GetGlusterVolumeAdvancedDetailsVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterVolumeStatvfs' VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' VDSM kvm04 command GlusterTasksListVDS failed: 'AutoProxy[instance]' object has no attribute 'glusterTasksList' -- Thanks, Vadim

3 6

install of engine
by david caughey 18 Aug '17

18 Aug '17

Hi folks, I have installed the ovirt OS on 3 dl380's. It has been deployed with gluster and all is fine up to a point. When I try to install the engine it is trying to get out to the internet but unfortunately I am behind a proxy with strict rules, is there a way to point to a local copy of the engine using the GUI install or can I scp the iso over to the first node for local install, is there a particular partition where the iso should be stored. Any help appreciated, BR/David

2 1

How to delay start ovirt service?
by Spickiy Nikita 17 Aug '17

17 Aug '17

Hi, i try install oVirt hosted-engine. For storage i use iscsi target (Infotrend DS1012RE). I make install, but have error to service ovirt-ha-broker.service and vdsmd.service: systemctl -l status ovirt-ha-broker.service ● ovirt-ha-broker.service - oVirt Hosted Engine High Availability Communications Broker ▽ Loaded: loaded (/usr/lib/systemd/system/ovirt-ha-broker.service; enabled; vendor preset: disabled) ▽ Active: active (running) since Thu 2017-08-17 15:28:57 +07; 48s ago ▽ Main PID: 1926 (ovirt-ha-broker) CGroup: /system.slice/ovirt-ha-broker.service └─1926 /usr/bin/python /usr/share/ovirt-hosted-engine-ha/ovirt-ha-broker --no-daemon Aug 17 15:28:57 vnode10.pi.local systemd[1]: Started oVirt Hosted Engine High Availability Communications Broker. Aug 17 15:28:57 vnode10.pi.local systemd[1]: Starting oVirt Hosted Engine High Availability Communications Broker... Aug 17 15:29:08 vnode10.pi.local ovirt-ha-broker[1926]: ovirt-ha-broker ovirt_hosted_engine_ha.broker.listener.ConnectionHandler ERROR Error handling request, data: 'set-storage-domain FilesystemBackend dom_type=iscsi sd_uuid=817a5b54-26c1-4d7d-a711-dafdb1ea477d' Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 166, in handle data) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/listener.py", line 299, in _dispatch .set_storage_domain(client, sd_type, **options) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/broker/storage_broker.py", line 66, in set_storage_domain self._backends[client].connect() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/storage_backends.py", line 470, in connect for lv in os.listdir(os.path.join("/dev", uuid)): OSError: [Errno 2] No such file or directory: '/dev/817a5b54-26c1-4d7d-a711-dafdb1ea477d’ systemctl -l status vdsmd.service ● vdsmd.service - Virtual Desktop Server Manager Loaded: loaded (/usr/lib/systemd/system/vdsmd.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2017-08-17 15:29:07 +07; 1min 30s ago Process: 3686 ExecStartPre=/usr/libexec/vdsm/vdsmd_init_common.sh --pre-start (code=exited, status=0/SUCCESS) Main PID: 3759 (vdsm) CGroup: /system.slice/vdsmd.service └─3759 /usr/bin/python2 /usr/share/vdsm/vdsm Aug 17 15:29:06 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running syslog_available Aug 17 15:29:06 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running nwfilter Aug 17 15:29:07 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running dummybr Aug 17 15:29:07 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running tune_system Aug 17 15:29:07 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running test_space Aug 17 15:29:07 vnode10.pi.local vdsmd_init_common.sh[3686]: vdsm: Running test_lo Aug 17 15:29:07 vnode10.pi.local systemd[1]: Started Virtual Desktop Server Manager. Aug 17 15:29:08 vnode10.pi.local vdsm[3759]: vdsm throttled WARN MOM not available. Aug 17 15:29:08 vnode10.pi.local vdsm[3759]: vdsm throttled WARN MOM not available, KSM stats will be missing. Aug 17 15:29:08 vnode10.pi.local vdsm[3759]: vdsm root ERROR failed to retrieve Hosted Engine HA info Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/host/api.py", line 231, in _getHaInfo stats = instance.get_all_stats() File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 103, in get_all_stats self._configure_broker_conn(broker) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/client/client.py", line 180, in _configure_broker_conn dom_type=dom_type) File "/usr/lib/python2.7/site-packages/ovirt_hosted_engine_ha/lib/brokerlink.py", line 177, in set_storage_domain .format(sd_type, options, e)) RequestError: Failed to set storage domain FilesystemBackend, options {'dom_type': 'iscsi', 'sd_uuid': '817a5b54-26c1-4d7d-a711-dafdb1ea477d'}: Request failed: <type 'exceptions.OSError’> But device exist: ls -l /dev/817a5b54-26c1-4d7d-a711-dafdb1ea477d total 0 lrwxrwxrwx. 1 root root 8 Aug 17 16:01 17bb9c12-c496-4fd3-b53e-ee4c8d319b43 -> ../dm-10 lrwxrwxrwx. 1 root root 8 Aug 17 16:01 1e9a3147-34ac-4020-8f95-71aa5d1a5258 -> ../dm-13 lrwxrwxrwx. 1 root root 8 Aug 17 16:01 436d8816-990e-4acd-b040-7cd948f7c0af -> ../dm-11 lrwxrwxrwx. 1 root root 8 Aug 17 16:01 eeb4b310-cc23-472c-9bce-36ce27881727 -> ../dm-12 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 ids -> ../dm-5 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 inbox -> ../dm-8 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 leases -> ../dm-6 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 master -> ../dm-9 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 metadata -> ../dm-4 lrwxrwxrwx. 1 root root 7 Aug 17 16:01 outbox -> ../dm-7 I suspect the service starting before mount. I think modify After in: /usr/lib/systemd/system/vdsmd.service /usr/lib/systemd/system/ovirt-ha-broker.service I add iscsid.service, but it not help. Maybe say how to fix it?

1 0

oVirt Node with bcache
by FERNANDO FREDIANI 16 Aug '17

16 Aug '17

This is a multi-part message in MIME format. --------------4B191FE3792DDE5D00F7F478 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello I just wanted to share a scenario with you and perhaps exchange more information with other people that may also have a similar scenario. For a couple of months I have been running a oVirt Node (CentOS 7.3 Minimal) with bcache (https://bcache.evilpiepirate.org/) for caching a SSD with HDD disks. The setup is simple and was made for a prof of concept and since them has been working better than expected. This is a standalone host with 4 disks being: 1 for Operating System, 2 x 2TB 7200 RPM in software RAID 1 and 1 x PCI-E NVMe 400GB SSD which plays the caching device for both reads and writes. The VM storage folder is mounted as a ext4 partition on the logical device created by bcache (/dev/bcache0). All this is transparent to oVirt as all it sees is a /folder to put the VMs. We monitor the IOPS on all block devices individually and see the behavior exactly as expected: random writes are all done on the SSD first and them streamed sequentially to the mechanical drives with pretty impressive performance. Also in the beginning while the total amount of data was less than 400GB ALL read used to come from the caching device and therefore didn't use IOPS from the mechanical drives leaving it free to do basically writes. Finally at sequential IOPS (as described by bcache) are intelligently passed directly to the mechanical drives (but they are not much). Although bcache is present on kernel 3.10 I had to use kernel-ml 4.12 (from Elrepo) and I had also to compile the bcache-tools as I could not find it available in any repository. Regards Fernando --------------4B191FE3792DDE5D00F7F478 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> <font face="arial, helvetica, sans-serif">Hello<br> <br> I just wanted to share a scenario with you and perhaps exchange more information with other people that may also have a similar scenario.<br> <br> For a couple of months I have been running a oVirt Node (CentOS 7.3 Minimal) with bcache (<a class="moz-txt-link-freetext" href="https://bcache.evilpiepirate.org/">https://bcache.evilpiepirate.org/</a>) for caching a SSD with HDD disks. The setup is simple and was made for a prof of concept and since them has been working better than expected.<br> This is a standalone host with 4 disks being: 1 for Operating System, 2 x 2TB 7200 RPM in software RAID 1 and 1 x PCI-E NVMe 400GB SSD which plays the caching device for both reads and writes. The VM storage folder is mounted as a ext4 partition on the logical device created by bcache (/dev/bcache0). All this is transparent to oVirt as all it sees is a /folder to put the VMs.<br> <br> We monitor the IOPS on all block devices individually and see the behavior exactly as expected: random writes are all done on the SSD first and them streamed sequentially to the mechanical drives with pretty impressive performance. Also in the beginning while the total amount of data was less than 400GB ALL read used to come from the caching device and therefore didn't use IOPS from the mechanical drives leaving it free to do basically writes. Finally at sequential IOPS (as described by bcache) are intelligently passed directly to the mechanical drives (but they are not much).<br> <br> Although bcache is present on kernel 3.10 I had to use kernel-ml 4.12 (from Elrepo) and I had also to compile the bcache-tools as I could not find it available in any repository.<br> <br> Regards<br> Fernando<br> </font> </body> </html> --------------4B191FE3792DDE5D00F7F478--

2 1

Communication Problems between Engine and Hosts
by FERNANDO FREDIANI 16 Aug '17

16 Aug '17

This is a multi-part message in MIME format. --------------7A7A94A488D7D26C7D1B2EB5 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello. Yesterday I had a pretty strange problem in one of our architectures. My oVirt which runs in one Datacenter and controls Nodes locally and also remotelly lost communication with the remote Nodes in another Datacenter. To this point nothing wrong as the Nodes can continue working as expected and running their Virtual Machines each without dependency of the oVirt Engine. What happened at some point is that when the communication between Engine and Hosts came back Hosts got confused and initiated a Live Migration of ALL VMs from one of the other. I had also to restart vdsmd agent on all Hosts in order to get sanity my environment. What adds up even more strangeness to this scenario is that one of the Hosts affected doesn't belong to the same Cluster as the others and had to have the vdsmd restarted. I understand the Hosts can survive without the Engine online with reduced possibilities but can communicated between them, but without affecting the VMs or even needing to do what happened in this scenario. Am I wrong on any of the assumptions ? Fernando --------------7A7A94A488D7D26C7D1B2EB5 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> <font face="arial, helvetica, sans-serif">Hello.<br> <br> Yesterday I had a pretty strange problem in one of our architectures. My oVirt which runs in one Datacenter and controls Nodes locally and also remotelly lost communication with the remote Nodes in another Datacenter.<br> To this point nothing wrong as the Nodes can continue working as expected and running their Virtual Machines each without dependency of the oVirt Engine.<br> <br> What happened at some point is that when the communication between Engine and Hosts came back Hosts got confused and initiated a Live Migration of ALL VMs from one of the other. I had also to restart vdsmd agent on all Hosts in order to get sanity my environment.<br> What adds up even more strangeness to this scenario is that one of the Hosts affected doesn't belong to the same Cluster as the others and had to have the vdsmd restarted.<br> <br> I understand the Hosts can survive without the Engine online with reduced possibilities but can communicated between them, but without affecting the VMs or even needing to do what happened in this scenario.<br> <br> Am I wrong on any of the assumptions ?<br> <br> Fernando<br> </font> </body> </html> --------------7A7A94A488D7D26C7D1B2EB5--

2 3

Ldap authentification filtering with custom attribute
by Jean-mathieu CHANTREIN 16 Aug '17

16 Aug '17

--=_43c7d111-b673-4361-8f5b-05dd6990fd98 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hello. Here is a way to filtering a group of ldap user by one of custom attribute and not by groups ? By example, I tryed (without success) to put this entry in /etc/ovirt-engine/extensions.d/my-ldap-authz.properties : search.simple-resolve-groups-memberOf.search-request.filter = &(myCustomAttribute=nameOfAttributeToFilter) And if it's possible, can I have filtering with more than one attribute (i.e.: each attribute will be discriminate like a group) ? Thanks for your help. Regards. Jean-Mathieu --=_43c7d111-b673-4361-8f5b-05dd6990fd98 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"font-family: arial, helvetica, sans-serif; font-s= ize: 12pt; color: #000000"><div>Hello.<br data-mce-bogus=3D"1"></div><div><= br data-mce-bogus=3D"1"></div><div>Here is  a way to filtering a group= of ldap user by one of custom attribute and not by groups ? By example, I = tryed (without success) to put this entry in /etc/ovirt-engine/extensions.d= /my-ldap-authz.properties :<br data-mce-bogus=3D"1"></div><div><br data-mce= -bogus=3D"1"></div><div>search.simple-resolve-groups-memberOf.search-reques= t.filter =3D &(myCustomAttribute=3DnameOfAttributeToFilter)<br data-mce= -bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>And if it's pos= sible, can I have filtering with more than one attribute (i.e.: each attrib= ute will be discriminate like a group) ?<br data-mce-bogus=3D"1"></div><div= ><br data-mce-bogus=3D"1"></div><div>Thanks for your help.<br data-mce-bogu= s=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>Regards.<br data-mce= -bogus=3D"1"></div><div><br data-mce-bogus=3D"1"></div><div>Jean-Mathieu<br= data-mce-bogus=3D"1"></div></div></body></html> --=_43c7d111-b673-4361-8f5b-05dd6990fd98--

1 0

Ovirt Network NAT
by Simms, Peter 16 Aug '17

16 Aug '17

--_000_B5BB66632CBAB644AAA49575398EB7160109F7EE003FCH1MPN20130_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, I am trying to setup a NAT network for my networks so my VMs can just use N= AT instead of getting their own IP. I seem to be struggling on how to do t= his, I found the below link, however can't seem to get this working. Is th= is still the best way to achieve this or what is the best way to achieve a = NAT network that my VM's can use? http://www.ovirt.org/develop/developer-guide/vdsm/hook/network-nat/ Any help would be appreciated, thank you ________________________________ This e-mail contains privileged and confidential information intended for t= he use of the addressees named above. If you are not the intended recipient= of this e-mail, you are hereby notified that you must not disseminate, cop= y or take any action in respect of any information contained in it. If you = have received this e-mail in error, please notify the sender immediately by= e-mail and immediately destroy this e-mail and its attachments. --_000_B5BB66632CBAB644AAA49575398EB7160109F7EE003FCH1MPN20130_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable <html xmlns:v=3D"urn:schemas-microsoft-com:vml" xmlns:o=3D"urn:schemas-micr= osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word" = xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml" xmlns=3D"http:= //www.w3.org/TR/REC-html40"> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii"= > <meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang=3D"EN-GB" link=3D"#0563C1" vlink=3D"#954F72"> <div class=3D"WordSection1"> <p class=3D"MsoNormal"><span style=3D"color:#2F5496">Hi all,<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496"><o:p> </o:p></spa= n></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496">I am trying to setup a= NAT network for my networks so my VMs can just use NAT instead of getting = their own IP.  I seem to be struggling on how to do this, I found the = below link, however can’t seem to get this working.  Is this still the best way to achieve this or what is the b= est way to achieve a NAT network that my VM’s can use?<o:p></o:p></sp= an></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496"><o:p> </o:p></spa= n></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496">http://www.ovirt.org/d= evelop/developer-guide/vdsm/hook/network-nat/<o:p></o:p></span></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496"><o:p> </o:p></spa= n></p> <p class=3D"MsoNormal"><span style=3D"color:#2F5496">Any help would be appr= eciated, thank you<o:p></o:p></span></p> </div> <br> <hr> <font face=3D"Arial" color=3D"Gray" size=3D"1"><br> This e-mail contains privileged and confidential information intended for t= he use of the addressees named above. If you are not the intended recipient= of this e-mail, you are hereby notified that you must not disseminate, cop= y or take any action in respect of any information contained in it. If you have received this e-mail in er= ror, please notify the sender immediately by e-mail and immediately destroy= this e-mail and its attachments.<br> </font> </body> </html> --_000_B5BB66632CBAB644AAA49575398EB7160109F7EE003FCH1MPN20130_--

1 0

NTP
by Moacir Ferreira 16 Aug '17

16 Aug '17

--_000_VI1P190MB0285EA69D255EBFB114A6167C8B50VI1P190MB0285EURP_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable I found that NTP does not get installed on oVirt node on the latest version= ovirt-node-ng-installer-ovirt-4.1-2017052309.iso. Also the installed repositories does not have it. So, is this a bug or NTP = is not considered appropriated anymore? Thanks. Moacir --_000_VI1P190MB0285EA69D255EBFB114A6167C8B50VI1P190MB0285EURP_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <html> <head> <meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Diso-8859-= 1"> <style type=3D"text/css" style=3D"display:none;"></style> </head> <body dir=3D"ltr"> <div id=3D"divtagdefaultwrapper" style=3D"font-size:12pt;color:#000000;font= -family:Calibri,Helvetica,sans-serif;" dir=3D"ltr"> <p>I found that NTP does not get installed on oVirt node <span>on the lates= t version</span> <span>ovirt-node-ng-installer-ovirt-</span>4.1-2017052309.iso.</p> <p><br> </p> <p>Also the installed repositories does not have it. So, is this a bug or N= TP is not considered appropriated anymore?</p> <p><br> </p> <p>Thanks.<br> </p> <p>Moacir<br> </p> </div> </body> </html> --_000_VI1P190MB0285EA69D255EBFB114A6167C8B50VI1P190MB0285EURP_--

4 6

VM with attached host USB device failed to run
by Дровалев Роман 16 Aug '17

16 Aug '17

Hello, I connected a USB to the VM, writes this error: VM Win2012 is down with error. Exit message: local variable 'device' referenced before assignment. I found this bug - https://bugzilla.redhat.com/show_bug.cgi?id=1261075 Does anyone have a working USB forwarding in a virtual machine? If "YES", how did you solve this problem? If this problem is not solved, unfortunately, the ovirt will have to completely abandon. ((

2 1

import qcow2 and sparse
by Marcin Kruk 16 Aug '17

16 Aug '17

After import machine from KVM which was based on qemu disk 2GB phisical and 50 GB virtual size I have got disk machine which occupy 50GB and even sparse option does not work. It still ocupy 50GB?

2 2

FORCE DELETE DATACENTER
by Erick Vogeler 16 Aug '17

16 Aug '17

Hello i had a host that now i dont have access to it, cant put it on maintenance, cant force remove, cant remove old vms. How do i force delete this datacenter?

2 1

Re: [ovirt-users] oVirt and FreeNAS
by Karli Sjöberg 15 Aug '17

15 Aug '17

1 0

hosted-engine restore
by Gary Pedretty 15 Aug '17

15 Aug '17

--Apple-Mail=_8210B568-81DD-4734-AAF7-CB9689618AB3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 When restoring a self hosted-engine from a backup, the documentation at = https://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Restor= ing_an_EL-Based_Self-Hosted_Environment/ = <https://www.ovirt.org/documentation/self-hosted/chap-Backing_up_and_Resto= ring_an_EL-Based_Self-Hosted_Environment/> Shows creating a new host server and then doing the hosted-engine = =E2=80=94deploy and manually installing the ovirt-engine instead of = using the ovirt-engine-appliance. If the hosted-engine deployment that = was backed up and is being restored, was created using the appliance, = should I still follow this documentation and manually install the = engine, run the restore, then do the engine-setup? Or should I use the = ovirt-engine-appliance and just run the restore before the engine-setup? Gary ------------------------------------------------------------------------ Gary Pedretty gary(a)ravnalaska.net = <mailto:gary@eraalaska.net> Systems Manager www.flyravn.com = <http://www.flyravn.com/> Ravn Alaska /\ 907-450-7251 5245 Airport Industrial Road / \/\ 907-450-7238 fax Fairbanks, Alaska 99709 /\ / \ \ Second greatest commandment Serving All of Alaska / \/ /\ \ \/\ =E2=80=9CLove your = neighbor as Green, green as far as the eyes can see yourself=E2=80=9D = Matt 22:39 ------------------------------------------------------------------------ --Apple-Mail=_8210B568-81DD-4734-AAF7-CB9689618AB3 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D"">When restoring a self hosted-engine from a backup, the = documentation at<div class=3D""><br class=3D""></div><div class=3D""><a = href=3D"https://www.ovirt.org/documentation/self-hosted/chap-Backing_up_an= d_Restoring_an_EL-Based_Self-Hosted_Environment/" = class=3D"">https://www.ovirt.org/documentation/self-hosted/chap-Backing_up= _and_Restoring_an_EL-Based_Self-Hosted_Environment/</a></div><div = class=3D""><br class=3D""></div><div class=3D"">Shows creating a new = host server and then doing the hosted-engine =E2=80=94deploy and = manually installing the ovirt-engine instead of using the = ovirt-engine-appliance.  If the hosted-engine deployment that was = backed up and is being restored, was created using the appliance, should = I still follow this documentation and manually install the engine, run = the restore, then do the engine-setup?  Or should I use the = ovirt-engine-appliance and just run the restore before the = engine-setup?</div><div class=3D""><br class=3D""></div><div = class=3D""><br class=3D""></div><div class=3D"">Gary</div><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""><div = class=3D""><br class=3D""></div><div class=3D""><br class=3D""><div = class=3D""> <div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"color: rgb(0, 0, 0); letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><div = style=3D"text-align: start; text-indent: 0px; word-wrap: break-word; = -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" = class=3D""><div style=3D"text-align: start; text-indent: 0px; word-wrap: = break-word; -webkit-nbsp-mode: space; -webkit-line-break: = after-white-space;" class=3D""><div style=3D"text-align: start; = text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; = -webkit-line-break: after-white-space;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); font-size: 12px; letter-spacing: normal; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px;" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div><div class=3D"">Gary Pedretty         =                     =            <a = href=3D"mailto:gary@eraalaska.net" = class=3D"">gary(a)ravnalaska.net</a></div><div class=3D"">Systems Manager =                     =                     =  <a href=3D"http://www.flyravn.com" = class=3D"">www.flyravn.com</a></div><div class=3D"">Ravn Alaska   =                     =     /\                 =    907-450-7251</div><div class=3D"">5245 Airport Industrial = Road         /  \/\         =     907-450-7238 fax</div><div class=3D"">Fairbanks, Alaska =  99709        /\  /    \ \ = Second greatest commandment</div></font><font face=3D"Monaco" = class=3D""><span style=3D"font-size: 12px;" class=3D"">Serving All of = Alaska          /  \/  /\  \ \/\ =   =E2=80=9CLove your neighbor as</span></font><br = style=3D"font-family: Monaco;" class=3D""><font face=3D"Menlo" = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px;" class=3D""><span style=3D"font-size: 12px;" class=3D"">Green, = green as far as the eyes can see           =   yourself=E2=80=9D Matt 22:39</span></font><div style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-family: = Menlo;" class=3D""></div><font face=3D"Menlo" style=3D"font-size: 12px;" = class=3D""></font><span style=3D"color: rgb(0, 0, 0); letter-spacing: = normal; text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; font-size: 12px;" class=3D""><font = face=3D"Menlo" class=3D""><div = class=3D"">---------------------------------------------------------------= ---------</div></font></span><div style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=3D""><font = face=3D"Menlo" style=3D"font-size: 12px;" class=3D""><br = class=3D""></font></div></div><span style=3D"color: rgb(0, 0, 0); = letter-spacing: normal; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: 12px;" = class=3D""><br class=3D"Apple-interchange-newline"></span></div><span = style=3D"color: rgb(0, 0, 0); letter-spacing: normal; text-transform: = none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: = 0px; font-size: 12px;" class=3D""><br = class=3D"Apple-interchange-newline"></span></div><span style=3D"color: = rgb(0, 0, 0); letter-spacing: normal; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; font-size: = 12px;" class=3D""><br class=3D"Apple-interchange-newline"></span></div><br= class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"></div><br = class=3D"Apple-interchange-newline"><br = class=3D"Apple-interchange-newline"> </div> <br class=3D""></div></div></body></html>= --Apple-Mail=_8210B568-81DD-4734-AAF7-CB9689618AB3--

2 2