- Users - oVirt List Archives

SELinux is preventing /usr/sbin/glusterfsd from write access on the sock_file
by Nathanaël Blanchet 19 Feb '15

19 Feb '15

On freshly installed el7 hosts, selinux prevents gluster from running. Setting selinux to permissive or building the relative .pp module resolves the issue. Does otopi configure selinux for gluster when installing?

2 1

Ovirt 3.5.1, Windows 2008 hangs
by wodel youchi 19 Feb '15

19 Feb '15

------=_Part_3325810_102070030.1423671665987 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Hi, I've a two nodes hosted-engine ovirt 3.5used storage is NFS4 for the: engine, vms, iso and export I have many Linux VMs and one Windows 2008 VM. The windows 2008 VMs is acting as a second AD, it's the second time in a month where the VM hangs and the cpu usage is 100%.I have this problem only with this VM only. I tried to look into vdsm logs, but no luck so far, all I found isGuestMonitor-Win2k8x64-AD2::DEBUG::2015-02-11 12:01:02,158::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=`ce0bcb92-90e8-45a7-9b9f-cfed10e920d6`::Domain Metadata is not set Is there a log file where I can follow the behaviour of the VM, because for now I have no clue about what is causing this problem. any advice? PS: is there a good solution to gather and analyze all the logs from ovirt's hypervisors and engine, storage...etc? thanks in advance. ------=_Part_3325810_102070030.1423671665987 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <html><body><div style=3D"color:#000; background-color:#fff; font-family:He= lveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;fo= nt-size:16px"><div id=3D"yui_3_16_0_1_1423652121743_87447">Hi,</div><div id= =3D"yui_3_16_0_1_1423652121743_87448"><br></div><div id=3D"yui_3_16_0_1_142= 3652121743_82672" dir=3D"ltr">I've a two nodes hosted-engine ovirt 3.5</div= ><div id=3D"yui_3_16_0_1_1423652121743_82673" dir=3D"ltr">used storage is N= FS4 for the: engine, vms, iso and export</div><div id=3D"yui_3_16_0_1_14236= 52121743_87449" dir=3D"ltr"><br></div><div id=3D"yui_3_16_0_1_1423652121743= _87446" dir=3D"ltr">I have many Linux VMs and one Windows 2008 VM.<br></div= ><div id=3D"yui_3_16_0_1_1423652121743_87445" dir=3D"ltr">The windows 2008 = VMs is acting as a second AD, it's the second time in a month where the VM = hangs and the cpu usage is 100%.</div><div dir=3D"ltr">I have this problem = only with this VM only.<br></div><div id=3D"yui_3_16_0_1_1423652121743_8744= 4" dir=3D"ltr"><br></div><div id=3D"yui_3_16_0_1_1423652121743_87443" dir= =3D"ltr">I tried to look into vdsm logs, but no luck so far, all I found is= </div><div dir=3D"ltr">GuestMonitor-Win2k8x64-AD2::DEBUG::2015-02-11 12:01:= 02,158::vm::486::vm.Vm::(_getUserCpuTuneInfo) vmId=3D`ce0bcb92-90e8-45a7-9b= 9f-cfed10e920d6`::Domain Metadata is not set<br></div><div dir=3D"ltr"><br>= </div><div dir=3D"ltr">Is there a log file where I can follow the behaviour= of the VM, because for now I have no clue about what is causing this probl= em.</div><div id=3D"yui_3_16_0_1_1423652121743_87455" dir=3D"ltr"><br></div= ><div id=3D"yui_3_16_0_1_1423652121743_87466" dir=3D"ltr">any advice?</div>= <div id=3D"yui_3_16_0_1_1423652121743_87454" dir=3D"ltr"><br></div><div id= =3D"yui_3_16_0_1_1423652121743_87467" dir=3D"ltr">PS: is there a good solut= ion to gather and analyze all the logs from ovirt's hypervisors and engine,= storage...etc?<br></div><div id=3D"yui_3_16_0_1_1423652121743_87451" dir= =3D"ltr"><br></div><div id=3D"yui_3_16_0_1_1423652121743_87450" dir=3D"ltr"= >thanks in advance.</div><div id=3D"yui_3_16_0_1_1423652121743_82653"><br><= /div></div></body></html> ------=_Part_3325810_102070030.1423671665987--

4 6

messages from journal in centos7 hosts
by Vadq 19 Feb '15

19 Feb '15

Hi, ovirt 3.5.1, engine on centos7 + storage type iscsi( using LIO™ on one of the vdsm host) + four hosts on centos7 After installation of vdsm at all cenos7 hosts in cluster in /var/log/messages every few seconds have such record Feb 19 16:47:08 kvm05 journal: metadata not found: Requested metadata element is not present Feb 19 16:47:19 kvm05 journal: metadata not found: Requested metadata element is not present Feb 19 16:47:23 kvm05 journal: metadata not found: Requested metadata element is not present Feb 19 16:47:34 kvm05 journal: metadata not found: Requested metadata element is not present Feb 19 16:47:38 kvm05 journal: metadata not found: Requested metadata element is not present is it possible to fix this without disabling systemd-journald? -- Thanks, Vadim

1 0

RHEV 3.5 problem with connecting rhev-m to nodes
by Jakub Bittner 19 Feb '15

19 Feb '15

Hello, after restart we had problem with manager to connect to nodes. All nodes are down and not reachable. When I try to remove and readd to engine it fails too. 10 days ago we changed SSL certs of rhev-m apache to IPA's generated. We have this in logs: 2015-02-19 11:41:33,491 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) Failure to refresh Vds runtime info: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.io.EOFException: SSL peer shut down incorrectly at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createNetworkException(VdsBrokerCommand.java:126) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:101) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) [vdsbroker.jar:] at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:31) [dal.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:571) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:648) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refresh(VdsUpdateRunTimeInfo.java:494) [vdsbroker.jar:] at org.ovirt.engine.core.vdsbroker.VdsManager.onTimer(VdsManager.java:236) [vdsbroker.jar:] at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) [:1.7.0_75] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_75] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: java.io.EOFException: SSL peer shut down incorrectly at sun.security.ssl.InputRecord.read(InputRecord.java:482) [jsse.jar:1.7.0_75] at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:934) [jsse.jar:1.7.0_75] at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1332) [jsse.jar:1.7.0_75] at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:709) [jsse.jar:1.7.0_75] at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122) [jsse.jar:1.7.0_75] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) [rt.jar:1.7.0_75] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) [rt.jar:1.7.0_75] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) [rt.jar:1.7.0_75] at java.io.FilterOutputStream.flush(FilterOutputStream.java:140) [rt.jar:1.7.0_75] at org.apache.xmlrpc.client.XmlRpcCommonsTransport$1$1.close(XmlRpcCommonsTransport.java:204) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcHttpTransport$ByteArrayReqWriter.write(XmlRpcHttpTransport.java:55) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcCommonsTransport$1.writeRequest(XmlRpcCommonsTransport.java:214) [xmlrpc-client.jar:3.1.3] at org.apache.commons.httpclient.methods.EntityEnclosingMethod.writeRequestBody(EntityEnclosingMethod.java:499) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2114) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) [commons-httpclient.jar:] at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) [commons-httpclient.jar:] at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) [xmlrpc-client.jar:3.1.3] at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) [xmlrpc-client.jar:3.1.3] at com.sun.proxy.$Proxy97.getVdsCapabilities(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_75] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_75] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_75] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_75] at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:255) [vdsbroker.jar:] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_75] at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_75] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_75] at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_75] 2015-02-19 11:41:33,506 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-30) Failed to refresh VDS , vds = 3c9f5126-8c0e-4037-ab61-29de6d02d487 : nodep01.example.com, VDS Network Error, continuing.

3 3

movirt -> ovirt 3.5.1 -> server error 500
by Markus Stockhausen 19 Feb '15

19 Feb '15

This is a multi-part message in MIME format. ------=_NextPartTM-000-01931aef-0e79-4067-9583-6cee81c33885 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi,=0A= =0A= just installed movirt on my mobile. Upon connection it breaks with the atta= ched error in the 3.5.1 engine server logs. =0A= =0A= Something I'm missing?=0A= =0A= Markus=0A= =0A= 2015-02-19 12:21:19,757 ERROR [org.apache.catalina.core.ContainerBase.[jbos= s.web].[default-host].[/ovirt-engine/api]] (ajp--127.0.0.1-8702-11) Standar= dWrapper.Throwable: java.lang.RuntimeException: java.lang.RuntimeException:= Unable to instantiate MessageBodyWriter=0A= at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(Re= gisterBuiltin.java:35) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.spi.ResteasyDeployment.start(ResteasyDeployme= nt.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispat= cher.init(ServletContainerDispatcher.java:67) [resteasy-jaxrs-2.3.2.Final.j= ar:]=0A= at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.= init(HttpServletDispatcher.java:36) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.apache.catalina.core.StandardWrapper.loadServlet(StandardWra= pper.java:1202) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.core.StandardWrapper.allocate(StandardWrappe= r.java:952) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWra= pperValve.java:188) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.core.StandardContextValve.invoke(StandardCon= textValve.java:161) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authe= nticatorBase.java:489) [jbossweb-7.0.13.Final.jar:]=0A= at org.jboss.as.web.security.SecurityContextAssociationValve.invoke= (SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.= 1.1.Final]=0A= at org.apache.catalina.core.StandardHostValve.invoke(StandardHostVa= lve.java:155) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportVa= lve.java:102) [jbossweb-7.0.13.Final.jar:]=0A= at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466)= [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngi= neValve.java:109) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapte= r.java:368) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:505= ) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(A= jpProtocol.java:445) [jbossweb-7.0.13.Final.jar:]=0A= at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.ja= va:930) [jbossweb-7.0.13.Final.jar:]=0A= at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_55]=0A= Caused by: java.lang.RuntimeException: Unable to instantiate MessageBodyWri= ter=0A= at org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(= ResteasyProviderFactory.java:772) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.plugins.providers.RegisterBuiltin.registerPro= viders(RegisterBuiltin.java:70) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.plugins.providers.RegisterBuiltin.register(Re= gisterBuiltin.java:31) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= ... 18 more=0A= Caused by: java.lang.RuntimeException: Failed to construct public org.ovirt= .engine.api.pdf.FOPMessageBodyWriter()=0A= at org.jboss.resteasy.core.ConstructorInjectorImpl.construct(Constr= uctorInjectorImpl.java:144) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.spi.ResteasyProviderFactory.getProviderInstan= ce(ResteasyProviderFactory.java:1039) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.spi.ResteasyProviderFactory.addMessageBodyWri= ter(ResteasyProviderFactory.java:519) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= at org.jboss.resteasy.spi.ResteasyProviderFactory.registerProvider(= ResteasyProviderFactory.java:768) [resteasy-jaxrs-2.3.2.Final.jar:]=0A= ... 20 more=0A= Caused by: java.lang.NoClassDefFoundError: org/apache/fop/apps/FopFactory= =0A= ------=_NextPartTM-000-01931aef-0e79-4067-9583-6cee81c33885 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-01931aef-0e79-4067-9583-6cee81c33885--

3 4

Poor iSCSI Performance
by John Florian 19 Feb '15

19 Feb '15

This is a multi-part message in MIME format. --------------070002020309040609070902 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi all, I've been trying to resolve a storage performance issue but have had no luck in identifying the exact cause. I have my storage domain on iSCSI and I can get the expected performance (limited by the Gbit Ethernet) when running bonnie++ on: * a regular physical machine configured with the iSCSI initiator connected to a dedicated iSCSI test target -- thus oVirt and VM technology are completely out of the picture * my oVirt host with the initiator connected to that same dedicated target -- thus I have an iSCSI connection on the oVirt host but I'm not using the iSCSI connection provided by oVirt's storage domain * a VM (hosted by oVirt) with the initiator (inside the VM) connected to that target -- thus bypassing oVirt's storage domain and the virtual disk it provides this VM However, if I just use a regular virtual disk via oVirt's storage domain the performance is much worse. I've tried both VirtIO and VirtIO-SCSI and have found no appreciable difference. Here's a typical example of the poor performance I get (as tested with bonnie++) with the normal virtual disk setup: # bonnie++ -d . -r 2048 -u root:root <snip> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP narvi-f21.double 4G 806 91 18507 1 15675 1 3174 56 33175 1 176.4 3 Latency 15533us 8142ms 2440ms 262ms 1289ms 780ms Version 1.96 ------Sequential Create------ --------Random Create-------- narvi-f21.doubledog -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 13641 24 +++++ +++ 22702 17 18919 31 +++++ +++ +++++ +++ Latency 27724us 247us 292us 71us 30us 172us For comparison, here's what I see if I run the same test, same VM, same host but this time the file system is mounted from a device obtained using iscsi-initiator-utils within the VM, i.e., the 3rd bullet config above: bonnie++ -d . -r 2048 -u root:root <snip> Version 1.96 ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP narvi-f21.double 4G 2051 89 103877 4 36286 3 4803 88 88166 4 163.6 3 Latency 7724us 191ms 396ms 48734us 73004us 1645ms Version 1.96 ------Sequential Create------ --------Random Create-------- narvi-f21.doubledog -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 6531 18 +++++ +++ 16388 20 5924 15 +++++ +++ 17906 23 Latency 15623us 64us 92us 1281us 14us 256us My host is Fedora 20 running oVirt 3.5 (hosted-engine). VM is running Fedora Server 21. Tonight I tried updating the host with the Fedora virt preview repo and I didn't see any significant change in the performance. Where should I look next? --------------070002020309040609070902 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> Hi all,<br> <br> I've been trying to resolve a storage performance issue but have had no luck in identifying the exact cause. I have my storage domain on iSCSI and I can get the expected performance (limited by the Gbit Ethernet) when running bonnie++ on:<br> <ul> <li>a regular physical machine configured with the iSCSI initiator connected to a dedicated iSCSI test target -- thus oVirt and VM technology are completely out of the picture<br> </li> <li>my oVirt host with the initiator connected to that same dedicated target -- thus I have an iSCSI connection on the oVirt host but I'm not using the iSCSI connection provided by oVirt's storage domain<br> </li> <li>a VM (hosted by oVirt) with the initiator (inside the VM) connected to that target -- thus bypassing oVirt's storage domain and the virtual disk it provides this VM</li> </ul> <p>However, if I just use a regular virtual disk via oVirt's storage domain the performance is much worse. I've tried both VirtIO and VirtIO-SCSI and have found no appreciable difference.<br> </p> <p>Here's a typical example of the poor performance I get (as tested with bonnie++) with the normal virtual disk setup:<br> </p> <p><tt># bonnie++ -d . -r 2048 -u root:root<br> <snip><br> Version 1.96 ------Sequential Output------ --Sequential Input- --Random-</tt><tt><br> </tt><tt>Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</tt><tt><br> </tt><tt>Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</tt><tt><br> </tt><tt>narvi-f21.double 4G 806 91 18507 1 15675 1 3174 56 33175 1 176.4 3</tt><tt><br> </tt><tt>Latency 15533us 8142ms 2440ms 262ms 1289ms 780ms</tt><tt><br> </tt><tt>Version 1.96 ------Sequential Create------ --------Random Create--------</tt><tt><br> </tt><tt>narvi-f21.doubledog -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--</tt><tt><br> </tt><tt> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP</tt><tt><br> </tt><tt> 16 13641 24 +++++ +++ 22702 17 18919 31 +++++ +++ +++++ +++</tt><tt><br> </tt><tt>Latency 27724us 247us 292us 71us 30us 172us</tt><br> <br> </p> <p>For comparison, here's what I see if I run the same test, same VM, same host but this time the file system is mounted from a device obtained using iscsi-initiator-utils within the VM, i.e., the 3rd bullet config above:<br> </p> <p><tt>bonnie++ -d . -r 2048 -u root:root</tt><tt><br> </tt><tt><snip></tt><tt><br> </tt><tt>Version 1.96 ------Sequential Output------ --Sequential Input- --Random-</tt><tt><br> </tt><tt>Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--</tt><tt><br> </tt><tt>Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP</tt><tt><br> </tt><tt>narvi-f21.double 4G 2051 89 103877 4 36286 3 4803 88 88166 4 163.6 3</tt><tt><br> </tt><tt>Latency 7724us 191ms 396ms 48734us 73004us 1645ms</tt><tt><br> </tt><tt>Version 1.96 ------Sequential Create------ --------Random Create--------</tt><tt><br> </tt><tt>narvi-f21.doubledog -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--</tt><tt><br> </tt><tt> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP</tt><tt><br> </tt><tt> 16 6531 18 +++++ +++ 16388 20 5924 15 +++++ +++ 17906 23</tt><tt><br> </tt><tt>Latency 15623us 64us 92us 1281us 14us 256us</tt><br> </p> <p>My host is Fedora 20 running oVirt 3.5 (hosted-engine). VM is running Fedora Server 21. Tonight I tried updating the host with the Fedora virt preview repo and I didn't see any significant change in the performance. Where should I look next?<br> </p> <p><br> </p> </body> </html> --------------070002020309040609070902--

2 1

How long do your migrations last?
by Nicolas Ecarnot 19 Feb '15

19 Feb '15

Hello list, Our storage domains are iSCSI on dedicated network, and when migrating VMs, the duration varies according to the size of the vDisks. The smallest VMs are migrated in about 20 seconds, while the biggest one may take more than 5 or 10 minutes. The average duration is 90 seconds. Questions : 1- Though I may have understood that the task of migration was made by the SPM, I don't know what it actually does? (which bytes goes where) 2- Do our times sound OK, or does it look like improvable? 3- What bottleneck should I investigate? I'm thinking about the dedicated hardware NICs setup of the hosts, the SAN, the MTU has already been setup at 9000... Any ideas welcomed. -- Nicolas Ecarnot

4 3

Where do you run oVirt?
by Sandro Bonazzola 18 Feb '15

18 Feb '15

Hi, As we continue developing oVirt 3.6 and future releases of oVirt, the development and integration team are seeking input from users on how oVirt is being run and on what platforms. We would appreciate you help by taking this quick survey! https://docs.google.com/forms/d/1u-YccuNRza4_WmS5VMdy0mEkXuKxt4lzrALKKcnEMB… Thanks, -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

2 2

Re: [ovirt-users] ovirt and glusterfs setup
by Donny D 18 Feb '15

18 Feb '15

----_com.android.email_82515713991120 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 CiAgICAKSSBkaWQgbm90IGhhdmUgYSBnb29kIGV4cGVyaWVuY2UgcHV0dGluZyBib3RoIGdsdXN0 ZXIgYW5kIHZpcnQgb24gdGhlIHNhbWUgbm9kZS4gSSB3YXMgZG9pbmcgaG9zdGVkIGVuZ2luZSB3 aXRoIHJlcGxpY2F0ZSBhY3Jvc3MgdHdvIG5vZGVzLCBhbmQgb25lIGRheSBpdCB3ZW50IGludG8g c3BsaXQgYnJhaW4gaGVsbC4uLiBpIHdhcyBuZXZlciBhYmxlIHRvIHRyYWNrIGRvd24gd2h5LiBI b3dldmVyIGkgZG8gaGF2ZSBhIGdsdXN0ZXIgd2l0aCBkaXN0cmlidXRlIGFuZCByZXBsaWNhIHNl dHVwIG9uIGl0cyBvd24gd2l0aCBhIGNvdXBsZSBub2RlcywgYW5kIGl0IGhhcyBnaXZlbiBtZSB6 ZXJvIHByb2JsZW1zIGluIHRoZSBsYXN0IDYwIGRheXMuIEl0IHNlZW1zIHRvIG1lIHRoYXQgZ2x1 c3RlciBhbmQgdmlydCBuZWVkIHRvIHN0YXkgc2VwZXJhdGUgZm9yIG5vdy4gQm90aCBhcmUgZ3Jl YXQgcHJvZHVjdHMgYW5kIGJvdGggd29yayBhcyBkZXNjcmliZWQsIGp1c3Qgbm90IG9uIHRoZSBz YW1lIG5vZGUgYXQgdGhlIHNhbWUgdGltZS4KCgpIYXBweSBDb25uZWN0aW5nLiBTZW50IGZyb20g bXkgU3ByaW50IFNhbXN1bmcgR2FsYXh5IFPCriA1CgotLS0tLS0tLSBPcmlnaW5hbCBtZXNzYWdl IC0tLS0tLS0tCkZyb206IEdlb3JnZSBTa29ydXAgPGdlb3JnZUBtd2NvbW0uY29tPiAKRGF0ZTog MDIvMTgvMjAxNSAgMTI6NTAgUE0gIChHTVQtMDc6MDApIApUbzogdXNlcnNAb3ZpcnQub3JnIApT dWJqZWN0OiBSZTogW292aXJ0LXVzZXJzXSBvdmlydCBhbmQgZ2x1c3RlcmZzIHNldHVwIAoKCiAg ICBCaWxsLAoKICAgIAoKICAgIEkgaGF2ZSBkb25lIGV4YWN0bHkgd2hhdCB5b3UncmUgbG9va2lu ZyB0byBkby4gSSB3YXMgdHJ5aW5nIHRvIG1pbWljCiAgICB2U0FOIGFzIHdlbGwuIFRoZXkgaGFk IFZTQSBmb3IgYSB3aGlsZSB3aGljaCB3YXMgYWNjZXB0YWJsZQogICAgbGljZW5zaW5nIGNvc3Rz LCBidXQgdGhhdCB3YXMgcmVwbGFjZWQgd2l0aCB2U0FOIHdoaWNoIGlzCiAgICByaWRpY3Vsb3Vz bHkgZXhwZW5zaXZlIGZvciBhbiBleHRyYS1zbWFsbCBidXNpbmVzcy4KCiAgICAKCiAgICBJIGhh dmUgYSBmb3VyIG5vZGUgY2x1c3RlciB3aXRoIDFUQiBvZiBzdG9yYWdlIGVhY2guIEdsdXN0ZXIg aXMKICAgIGNvbmZpZ3VyZWQgd2l0aCByZXBsaWNhIDQuIFNvIGJhc2ljYWxseSBJIGhhdmUgMVRC IG9mIHVzYWJsZQogICAgc3RvcmFnZSwgd2hpY2ggaXMgZmluZSBmb3IgbXkgbmVlZHMuIFZNIG1p Z3JhdGlvbiwgR2x1c3RlcgogICAgcmVwbGljYXRpb24sIGhvc3RlZCBlbmdpbmUgYW5kIGFsbCB0 aGF0IHdvcmtzIGZpbmUuIFBlcmZvcm1hbmNlIGlzCiAgICBnZW5lcmFsbHkgZmluZSwgZXZlbiB3 aXRoIG9ubHkgZHVhbCBMQUNQIGJvbmRlZCAxR2JFIE5JQ3MgaW4gZWFjaAogICAgbm9kZS4gSSBj YW4gZG8gd2hhdCBJIHdhbnQgd2l0aCBuZXR3b3JraW5nIHRvIGZpdCBvdXIgTk9DIGFuZCBvZmZp Y2UKICAgIG5ldHdvcmsgZW52aXJvbm1lbnQuCgogICAgCgogICAgSSBoYXZlIGJlZW4gcGxheWlu ZyB3aXRoIHRoaXMgZm9yIGFib3V0IHRocmVlIHdlZWtzLiBPdmVyIHRoZQogICAgd2Vla2VuZCwg SSBoYWQgYSBoYW5kZnVsIG9mIFZNcyBydW5uaW5nLCBpbmNsdWRpbmcgdGhlIGhvc3RlZAogICAg ZW5naW5lLiBUaGV5IHdlcmUgcHJldHR5IG11Y2ggc2l0dGluZyBpZGxlIGRvaW5nIG5vdGhpbmcu IEkgY2FtZSBpbgogICAgTW9uZGF5IGFuZCBmb3VuZCBldmVyeXRoaW5nIG9mZmxpbmUuIE5vIHBv d2VyIG91dGFnZXMsCiAgICBuZXR3b3JrL3N3aXRjaCBkaWRuJ3QgZmFpbCBvciByZWJvb3QsIG5v bmUgb2YgdGhlIGhhcmR3YXJlIHJlc2V0LgogICAgV2hhdCBJIGZvdW5kIGhhcHBlbmVkIHdhcyB0 aGF0IGdsdXN0ZXJkIHdlbnQgbnV0cyBvbiB0d28gb2YgdGhlCiAgICBub2RlcyAoSSBoYXZlIG5v IGlkZWEgd2h5KS4gR2x1c3RlciB3YXMgc3BpdHRpbmcgb3V0IGxvZ3MgbGlrZQogICAgY3Jhenks IC92YXIgZmlsbGVkIHVwLCB0aGVuIFJBTSBhbmQgc3dhcCB3YXMgZGVwbGV0ZWQuIFR3byBnbHVz dGVyCiAgICBwcm9jZXNzZXMvaG9zdHMgb2ZmbGluZSBvdXQgb2YgdGhlIGZvdXIgbWVhbnQgcXVv cnVtIGJyb2tlLAogICAgZXZlcnl0aGluZyBjYW1lIHRvIGEgaGFsdC4gSSB3YXMgdW5hYmxlIHRv IHJlY292ZXIgdGhlIGdsdXN0ZXIgbG9ncwogICAgc2luY2UgdGhleSBoYWQgdG8gYmUgZGVsZXRl ZCB0byBmcmVlIHVwIHNwYWNlIGluIC92YXIuCgogICAgCgogICAgSSB3YXMgYWJsZSB0byBnZXQg ZXZlcnl0aGluZyBmaXhlZCBhbmQgYmFjayBvbmxpbmUgaW4gYWJvdXQgMi0xLzIKICAgIGhvdXJz LiBTbyB0aGlzIGlzIGltcG9zc2libGUgdG8gcHV0IGludG8gcHJvZHVjdGlvbi4gU3RvcmFnZSBp cyB0aGUKICAgIHdlYWtlc3QgbGluayBhbmQgbW9zdGx5IGxpa2VseSB0byBmYWlsLCB3aGljaCBp biB0aGlzIGNhc2UgaXQgZGlkLgoKICAgIAoKICAgIEkgZG8gbm90IHJlY29tbWVuZCB0aGlzIGNv bmZpZ3VyYXRpb24gYXQgYWxsLiBBIGRlZGljYXRlZCBtYWNoaW5lCiAgICBmb3IgdGhlIGVuZ2lu ZSwgZGVkaWNhdGVkIGh5cGVydmlzb3JzIGFuZCBkZWRpY2F0ZWQgc3RvcmFnZSBub2RlcyBvcgog ICAgYSBTQU4gaXMgbmVlZGVkIGZvciBhbnl0aGluZyBiZXlvbmQgZXhwZXJpbWVudGF0aW9uLgoK ICAgIAoKICAgIEkgaGF2ZSB6ZXJvIGtub3dsZWRnZSBvZiBWTVdhcmUncyB2U0FOLCBvdGhlciB0 aGFuIGJhc2ljIGNvbmNlcHRzLAogICAgc28gSSBjYW5ub3Qgc2F5IHRoZSByZXN1bHRzIHdvdWxk IGJlIHNpbWlsYXIuCgogICAgCgogICAgCgogICAgT24gMi8xOC8yMDE1IDU6MzIgQU0sIEJpbGwg RG9zc2V0dAogICAgICB3cm90ZToKCiAgICAKICAgIAogICAgICAKICAgICAgCiAgICAgIDwhLS0K LyogRm9udCBEZWZpbml0aW9ucyAqLwpAZm9udC1mYWNlCgl7Zm9udC1mYW1pbHk6IkNhbWJyaWEg TWF0aCI7CglwYW5vc2UtMToyIDQgNSAzIDUgNCA2IDMgMiA0O30KQGZvbnQtZmFjZQoJe2ZvbnQt ZmFtaWx5OkNhbGlicmk7CglwYW5vc2UtMToyIDE1IDUgMiAyIDIgNCAzIDIgNDt9Ci8qIFN0eWxl IERlZmluaXRpb25zICovCnAuTXNvTm9ybWFsLCBsaS5Nc29Ob3JtYWwsIGRpdi5Nc29Ob3JtYWwK CXttYXJnaW46MGNtOwoJbWFyZ2luLWJvdHRvbTouMDAwMXB0OwoJZm9udC1zaXplOjExLjBwdDsK CWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7Cgltc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1VUzt9CmE6bGluaywgc3Bhbi5Nc29IeXBlcmxpbmsKCXttc28tc3R5bGUtcHJpb3JpdHk6 OTk7Cgljb2xvcjojMDU2M0MxOwoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9CmE6dmlzaXRl ZCwgc3Bhbi5Nc29IeXBlcmxpbmtGb2xsb3dlZAoJe21zby1zdHlsZS1wcmlvcml0eTo5OTsKCWNv bG9yOiM5NTRGNzI7Cgl0ZXh0LWRlY29yYXRpb246dW5kZXJsaW5lO30Kc3Bhbi5FbWFpbFN0eWxl MTcKCXttc28tc3R5bGUtdHlwZTpwZXJzb25hbC1jb21wb3NlOwoJZm9udC1mYW1pbHk6IkNhbGli cmkiLCJzYW5zLXNlcmlmIjsKCWNvbG9yOndpbmRvd3RleHQ7fQouTXNvQ2hwRGVmYXVsdAoJe21z by1zdHlsZS10eXBlOmV4cG9ydC1vbmx5OwoJZm9udC1mYW1pbHk6IkNhbGlicmkiLCJzYW5zLXNl cmlmIjsKCW1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLVVTO30KQHBhZ2UgV29yZFNlY3Rpb24xCgl7 c2l6ZTo2MTIuMHB0IDc5Mi4wcHQ7CgltYXJnaW46NzIuMHB0IDcyLjBwdCA3Mi4wcHQgNzIuMHB0 O30KZGl2LldvcmRTZWN0aW9uMQoJe3BhZ2U6V29yZFNlY3Rpb24xO30KLS0+CiAgICAgIAogICAg ICAgIEhpLAogICAgICAgIMKgCiAgICAgICAgSeKAmW0gaW4gYXQgdGhlIGRlZXAgZW5k4oCmCiAg ICAgICAgwqAKICAgICAgICBMb29raW5nIGZvciBzb21lIGFkdmljZSBvbiBpZiB0aGlzIGlzCiAg ICAgICAgICBwb3NzaWJsZSDigJMgYW5kIHdoYXQgdmVyc2lvbiBJIHNob3VsZCB0cnkgSSB3aXRo IGlmIHNvLgogICAgICAgIMKgCiAgICAgICAgV2UgYXJlIGEgaGVhdmlseSBWTXdhcmUgb3JpZW50 ZWQgY29tcGFueeKApgogICAgICAgICAgYnV0IEkgYW0gdHJ5aW5nIHRvIGdldCBSSEVWL29WaXJ0 IGluIHRoZSBkb29yLsKgIEhvbmVzdGx5IEkKICAgICAgICAgIHdvdWxkIHByZWZlciBvVmlydCwg YnV0IGlmIHRoZXkgaW5zaXN0IG9uIGVudGVycHJpc2Ugc3VwcG9ydCBJCiAgICAgICAgICBnbyB0 byBSZWRoYXQuCiAgICAgICAgwqAKICAgICAgICBTbyBJIGhhdmUgaGFkIGEgcGxheSB3aXRoIFZN d2FyZSBWU0FOIGFuZAogICAgICAgICAga25vdyBob3cgaXQgd29ya3MuwqAgSSBhbSB0cnlpbmcg dG8gbW9yZSBvciBsZXNzIHJlcGxpY2F0ZSB0aGlzCiAgICAgICAgICBzZXR1cCB1c2luZyBvcGVu IHNvdXJjZS4KICAgICAgICDCoAogICAgICAgIFNvIFZTQU4gdXNlcyAzIG9yIG1vcmUgRVNYaSBu b2Rlcywgd2l0aAogICAgICAgICAgbG9jYWwgaGFyZGRpc2tzIGFzIHRoZSBzdG9yYWdlIG1lZGl1 bSBhcyBhIHZpcnR1YWwgU0FOLsKgIFRoZQogICAgICAgICAgU0FOIHBlcmZvcm1hbmNlIGlzIHF1 aXRlIGhpZ2ggYXMgeW91IHB1dCBhdCBsZWFzdCAxIFNTRCBpbgogICAgICAgICAgZWFjaCBub2Rl LgogICAgICAgIMKgCiAgICAgICAgV2hhdCBpcyBuaWNlIGlzIHlvdSBkb27igJl0IGhhdmUgdGhl IE5BUwogICAgICAgICAgZWxlbWVudCBoZXJlLsKgIFZNcyBhcmUgb24gdGhlIGxvY2FsIHN0b3Jh Z2UsIHdoaWNoIGlzCiAgICAgICAgICBwYXJ0aWFsbHkgU1NELCBzbyBwZXJmb3JtYW5jZSBpcyBx dWl0ZSBnb29kLgogICAgICAgIMKgCiAgICAgICAgSSB3ZW50IHRvIGEgUmVkaGF0IHByZXNlbnRh dGlvbiBvbiBSZWRoYXQKICAgICAgICAgIHN0b3JhZ2UgYW5kIEdsdXN0ZXJGUyBhbmQgYmFzaWNh bGx5IHRoaXMgYWN0cyBhcyBhIGJpZwogICAgICAgICAgc29mdHdhcmUgZGVmaW5lZCBOQVMgd2hp Y2ggZG9lcyBzb21lIHByZXR0eSBjb29sIHRoaW5ncywgYnV0CiAgICAgICAgICB0aGF04oCZcyBu b3QgZXhhY3RseSB3aGF0IEkgbmVlZC4KICAgICAgICDCoAogICAgICAgIEkgd291bGQgbGlrZSB0 byBidWlsZCBvVmlydCBvbiB0b3Agb2YKICAgICAgICAgIENlbnRPUyBPc3PigKYgdGhhdCBoYXZl IGxvY2FsIHN0b3JhZ2UgaW4gdGhlbSB0aGF0IGlzCiAgICAgICAgICBkaXN0cmlidXRlZCBhbmQg cmVkdW5kYW50IGluIHRoZSBldmVudCBvZiBhIG5vZGUgZmFpbHVyZS4KICAgICAgICDCoAogICAg ICAgIEFuZCBJIHByb2JhYmx5IG5lZWQgdG8gdHJ5IGFuZCBidWlsZCB0aGlzCiAgICAgICAgICBp biBhIGxhYiB1bmRlciBGdXNpb24gb24gbXkgTWFjIEJvb2sgcHJvIHRvIGJlZ2luIHdpdGggYW55 d2F5CiAgICAgICAgICAodGhhdCBiaXQgSeKAmW0gZmFyaWx5IGNvbmZpZGVudCB3aXRoKSBhbmQg aWYgSSBnZXQgdGhhdCB3b3JraW5nCiAgICAgICAgICBjYW4gcHJvYmFibHkgZ2V0IHNvbWUgb2xk ZXIga2l0IHRvIHRyeSBpdCBvdXQgZm9yIHJlYWwgYXMgYQogICAgICAgICAgUG9DIHRvIGZldyBw ZW9wbGUuCiAgICAgICAgwqAKICAgICAgICBTbywgSeKAmXZlIHNldCB1cCBvVmlydCBiZWZvcmUg dGhhdCBzaG91bGQgYmUKICAgICAgICAgIG9rLCBJIGhhdmVu4oCZdCBzZXR1cCBHbHVzdGVyIGlz IHRoZXJlIGFueSBkb2N1bWVudHMgdGhhdCB3b3VsZAogICAgICAgICAgaGVscCBtZSBkb3duIHRo aXMgcm9hZCBhbmQgbWFrZSBzdXJlIEkgc3RhcnQgb3V0IHVzaW5nIHRoZQogICAgICAgICAgYmVz dCB2ZXJzaW9uLgogICAgICAgIMKgCiAgICAgICAgQW55IGFkdmljZSBvciBwb2ludGVycyB3b3Vs ZCBiZSBncmF0ZWZ1bGx5CiAgICAgICAgICByZWNlaXZlZC4KICAgICAgICDCoAogICAgICAgIFRo YW5rcwogICAgICAgIMKgCiAgICAgICAgCiAgICAgICAgICAKICAgICAgICAgICAgCiAgICAgICAg ICAgICAgCiAgICAgICAgICAgICAgICBCaWxsCiAgICAgICAgICAgICAgICAgICAgICBEb3NzZXR0 CiAgICAgICAgICAgICAgICBTeXN0ZW1zCiAgICAgICAgICAgICAgICAgICAgQXJjaGl0ZWN0CiAg ICAgICAgICAgICAgCiAgICAgICAgICAgIAogICAgICAgICAgICAKICAgICAgICAgICAgICAKICAg ICAgICAgICAgICAgIFRlY2gKICAgICAgICAgICAgICAgICAgICAgIENlbnRyYWwg4oCTIEdsb2Jh bCBFbmdpbmVlcmluZyBTZXJ2aWNlcwogICAgICAgICAgICAgICAgwqAKICAgICAgICAgICAgICAg IFTCoCs0NAogICAgICAgICAgICAgICAgICAgICgwKTE5MjMgMjc5MzUzCiAgICAgICAgICAgICAg ICBNwqArNDQKICAgICAgICAgICAgICAgICAgICAoMCk3NzcgNTkwIDg2MTIKICAgICAgICAgICAg ICAgIGJpbGwuZG9zc2V0dEBwYi5jb20KICAgICAgICAgICAgICAgIHBpdG5leWJvd2VzLmNvbQog ICAgICAgICAgICAgICAgCgogICAgICAgICAgICAgICAgICBQaXRuZXkKICAgICAgICAgICAgICAg ICAgICAgIEJvd2VzCiAgICAgICAgICAgICAgICA2CiAgICAgICAgICAgICAgICAgICAgSGVyY3Vs ZXMgV2F5IHwgTGVhdmVzZGVuIHwgV0QyNSA3R1MgfCBVSwogICAgICAgICAgICAgICAgwqAKICAg ICAgICAgICAgICAKICAgICAgICAgICAgCiAgICAgICAgICAgIAogICAgICAgICAgICAgIAoKICAg ICAgICAgICAgICAKICAgICAgICAgICAgCiAgICAgICAgICAgIAogICAgICAgICAgICAgIAogICAg ICAgICAgICAgICAgTGVhcm4gbW9yZSBhYm91dCBHbG9iYWwKICAgICAgICAgICAgICAgICAgICAg ICAgRW5naW5lZXJpbmcgU2VydmljZXMKICAgICAgICAgICAgICAgIMKgCiAgICAgICAgICAgICAg ICBJbgogICAgICAgICAgICAgICAgICAgIEVuZ2luZWVyaW5nPwogICAgICAgICAgICAgICAgUmFp c2UKICAgICAgICAgICAgICAgICAgICAgIGEgdGlja2V0IHZpYSBSZW1lZHkgQW55d2hlcmUgW0hF UkVdwqB0YWtlcyBsZXNzIHRoYW4gYQogICAgICAgICAgICAgICAgICAgICAgbWludXRlCiAgICAg ICAgICAgICAgICDCoAogICAgICAgICAgICAgICAgQ2xvdWRGb3JtcwogICAgICAgICAgICAgICAg ICAgICAgVXNlciBHdWlkZSBhdmFpbGFibGXCoFtIRVJFXQogICAgICAgICAgICAgIAogICAgICAg ICAgICAKICAgICAgICAgIAogICAgICAgIAogICAgICAgIMKgCiAgICAgICAgwqAKICAgICAgCiAg ICAgIAoKICAgICAgCiAgICAgIAoKICAgICAgCiAgICAgIAoKICAgICAgCiAgICAgIAoKICAgICAg X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVXNlcnMgbWFp bGluZyBsaXN0ClVzZXJzQG92aXJ0Lm9yZwpodHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4v bGlzdGluZm8vdXNlcnMKCiAgICAKICAgIAoKICA= ----_com.android.email_82515713991120 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGh0bWw+PGhlYWQ+PG1ldGEgaHR0cC1lcXVpdj0iQ29udGVudC1UeXBlIiBjb250ZW50PSJ0ZXh0 L2h0bWw7IGNoYXJzZXQ9VVRGLTgiPjwvaGVhZD48Ym9keT4KICAgIAo8ZGl2PkkgZGlkIG5vdCBo YXZlIGEgZ29vZCBleHBlcmllbmNlIHB1dHRpbmcgYm90aCBnbHVzdGVyIGFuZCB2aXJ0IG9uIHRo ZSBzYW1lIG5vZGUuIEkgd2FzIGRvaW5nIGhvc3RlZCBlbmdpbmUgd2l0aCByZXBsaWNhdGUgYWNy b3NzIHR3byBub2RlcywgYW5kIG9uZSBkYXkgaXQgd2VudCBpbnRvIHNwbGl0IGJyYWluIGhlbGwu Li4gaSB3YXMgbmV2ZXIgYWJsZSB0byB0cmFjayBkb3duIHdoeS4gSG93ZXZlciBpIGRvIGhhdmUg YSBnbHVzdGVyIHdpdGggZGlzdHJpYnV0ZSBhbmQgcmVwbGljYSBzZXR1cCBvbiBpdHMgb3duIHdp dGggYSBjb3VwbGUgbm9kZXMsIGFuZCBpdCBoYXMgZ2l2ZW4gbWUgemVybyBwcm9ibGVtcyBpbiB0 aGUgbGFzdCA2MCBkYXlzLiBJdCBzZWVtcyB0byBtZSB0aGF0IGdsdXN0ZXIgYW5kIHZpcnQgbmVl ZCB0byBzdGF5IHNlcGVyYXRlIGZvciBub3cuIEJvdGggYXJlIGdyZWF0IHByb2R1Y3RzIGFuZCBi b3RoIHdvcmsgYXMgZGVzY3JpYmVkLCBqdXN0IG5vdCBvbiB0aGUgc2FtZSBub2RlIGF0IHRoZSBz YW1lIHRpbWUuPC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PC9kaXY+PGRpdj48YnI+PC9k aXY+PGRpdiBpZD0iY29tcG9zZXJfc2lnbmF0dXJlIj48ZGl2IHN0eWxlPSJmb250LXNpemU6ODUl O2NvbG9yOiM1NzU3NTciPkhhcHB5IENvbm5lY3RpbmcuIFNlbnQgZnJvbSBteSBTcHJpbnQgU2Ft c3VuZyBHYWxheHkgU8KuIDU8L2Rpdj48L2Rpdj48YnI+PGJyPi0tLS0tLS0tIE9yaWdpbmFsIG1l c3NhZ2UgLS0tLS0tLS08YnI+RnJvbTogR2VvcmdlIFNrb3J1cCAmbHQ7Z2VvcmdlQG13Y29tbS5j b20mZ3Q7IDxicj5EYXRlOiAwMi8xOC8yMDE1ICAxMjo1MCBQTSAgKEdNVC0wNzowMCkgPGJyPlRv OiB1c2Vyc0BvdmlydC5vcmcgPGJyPlN1YmplY3Q6IFJlOiBbb3ZpcnQtdXNlcnNdIG92aXJ0IGFu ZCBnbHVzdGVyZnMgc2V0dXAgPGJyPjxicj4KICAgIEJpbGwsPGJyPgogICAgPGJyPgogICAgSSBo YXZlIGRvbmUgZXhhY3RseSB3aGF0IHlvdSdyZSBsb29raW5nIHRvIGRvLiBJIHdhcyB0cnlpbmcg dG8gbWltaWMKICAgIHZTQU4gYXMgd2VsbC4gVGhleSBoYWQgVlNBIGZvciBhIHdoaWxlIHdoaWNo IHdhcyBhY2NlcHRhYmxlCiAgICBsaWNlbnNpbmcgY29zdHMsIGJ1dCB0aGF0IHdhcyByZXBsYWNl ZCB3aXRoIHZTQU4gd2hpY2ggaXMKICAgIHJpZGljdWxvdXNseSBleHBlbnNpdmUgZm9yIGFuIGV4 dHJhLXNtYWxsIGJ1c2luZXNzLjxicj4KICAgIDxicj4KICAgIEkgaGF2ZSBhIGZvdXIgbm9kZSBj bHVzdGVyIHdpdGggMVRCIG9mIHN0b3JhZ2UgZWFjaC4gR2x1c3RlciBpcwogICAgY29uZmlndXJl ZCB3aXRoIHJlcGxpY2EgNC4gU28gYmFzaWNhbGx5IEkgaGF2ZSAxVEIgb2YgdXNhYmxlCiAgICBz dG9yYWdlLCB3aGljaCBpcyBmaW5lIGZvciBteSBuZWVkcy4gVk0gbWlncmF0aW9uLCBHbHVzdGVy CiAgICByZXBsaWNhdGlvbiwgaG9zdGVkIGVuZ2luZSBhbmQgYWxsIHRoYXQgd29ya3MgZmluZS4g UGVyZm9ybWFuY2UgaXMKICAgIGdlbmVyYWxseSBmaW5lLCBldmVuIHdpdGggb25seSBkdWFsIExB Q1AgYm9uZGVkIDFHYkUgTklDcyBpbiBlYWNoCiAgICBub2RlLiBJIGNhbiBkbyB3aGF0IEkgd2Fu dCB3aXRoIG5ldHdvcmtpbmcgdG8gZml0IG91ciBOT0MgYW5kIG9mZmljZQogICAgbmV0d29yayBl bnZpcm9ubWVudC48YnI+CiAgICA8YnI+CiAgICBJIGhhdmUgYmVlbiBwbGF5aW5nIHdpdGggdGhp cyBmb3IgYWJvdXQgdGhyZWUgd2Vla3MuIE92ZXIgdGhlCiAgICB3ZWVrZW5kLCBJIGhhZCBhIGhh bmRmdWwgb2YgVk1zIHJ1bm5pbmcsIGluY2x1ZGluZyB0aGUgaG9zdGVkCiAgICBlbmdpbmUuIFRo ZXkgd2VyZSBwcmV0dHkgbXVjaCBzaXR0aW5nIGlkbGUgZG9pbmcgbm90aGluZy4gSSBjYW1lIGlu CiAgICBNb25kYXkgYW5kIGZvdW5kIGV2ZXJ5dGhpbmcgb2ZmbGluZS4gTm8gcG93ZXIgb3V0YWdl cywKICAgIG5ldHdvcmsvc3dpdGNoIGRpZG4ndCBmYWlsIG9yIHJlYm9vdCwgbm9uZSBvZiB0aGUg aGFyZHdhcmUgcmVzZXQuCiAgICBXaGF0IEkgZm91bmQgaGFwcGVuZWQgd2FzIHRoYXQgZ2x1c3Rl cmQgd2VudCBudXRzIG9uIHR3byBvZiB0aGUKICAgIG5vZGVzIChJIGhhdmUgbm8gaWRlYSB3aHkp LiBHbHVzdGVyIHdhcyBzcGl0dGluZyBvdXQgbG9ncyBsaWtlCiAgICBjcmF6eSwgL3ZhciBmaWxs ZWQgdXAsIHRoZW4gUkFNIGFuZCBzd2FwIHdhcyBkZXBsZXRlZC4gVHdvIGdsdXN0ZXIKICAgIHBy b2Nlc3Nlcy9ob3N0cyBvZmZsaW5lIG91dCBvZiB0aGUgZm91ciBtZWFudCBxdW9ydW0gYnJva2Us CiAgICBldmVyeXRoaW5nIGNhbWUgdG8gYSBoYWx0LiBJIHdhcyB1bmFibGUgdG8gcmVjb3ZlciB0 aGUgZ2x1c3RlciBsb2dzCiAgICBzaW5jZSB0aGV5IGhhZCB0byBiZSBkZWxldGVkIHRvIGZyZWUg dXAgc3BhY2UgaW4gL3Zhci48YnI+CiAgICA8YnI+CiAgICBJIHdhcyBhYmxlIHRvIGdldCBldmVy eXRoaW5nIGZpeGVkIGFuZCBiYWNrIG9ubGluZSBpbiBhYm91dCAyLTEvMgogICAgaG91cnMuIFNv IHRoaXMgaXMgaW1wb3NzaWJsZSB0byBwdXQgaW50byBwcm9kdWN0aW9uLiBTdG9yYWdlIGlzIHRo ZQogICAgd2Vha2VzdCBsaW5rIGFuZCBtb3N0bHkgbGlrZWx5IHRvIGZhaWwsIHdoaWNoIGluIHRo aXMgY2FzZSBpdCBkaWQuPGJyPgogICAgPGJyPgogICAgSSBkbyBub3QgcmVjb21tZW5kIHRoaXMg Y29uZmlndXJhdGlvbiBhdCBhbGwuIEEgZGVkaWNhdGVkIG1hY2hpbmUKICAgIGZvciB0aGUgZW5n aW5lLCBkZWRpY2F0ZWQgaHlwZXJ2aXNvcnMgYW5kIGRlZGljYXRlZCBzdG9yYWdlIG5vZGVzIG9y CiAgICBhIFNBTiBpcyBuZWVkZWQgZm9yIGFueXRoaW5nIGJleW9uZCBleHBlcmltZW50YXRpb24u PGJyPgogICAgPGJyPgogICAgSSBoYXZlIHplcm8ga25vd2xlZGdlIG9mIFZNV2FyZSdzIHZTQU4s IG90aGVyIHRoYW4gYmFzaWMgY29uY2VwdHMsCiAgICBzbyBJIGNhbm5vdCBzYXkgdGhlIHJlc3Vs dHMgd291bGQgYmUgc2ltaWxhci48YnI+CiAgICA8YnI+CiAgICA8YnI+CiAgICA8ZGl2IGNsYXNz PSJtb3otY2l0ZS1wcmVmaXgiPk9uIDIvMTgvMjAxNSA1OjMyIEFNLCBCaWxsIERvc3NldHQKICAg ICAgd3JvdGU6PGJyPgogICAgPC9kaXY+CiAgICA8YmxvY2txdW90ZSBjaXRlPSJtaWQ6QjQ3RjAw ODI1RDY4NDQ0QUJDMDM3MzMwNzgyNzUzNjAzNzcxQTM2NEAwMTMtQlIxTVBOMS0wMTIuTUdEUEJJ Lmdsb2JhbC5wdnQiIHR5cGU9ImNpdGUiPgogICAgICA8bWV0YSBodHRwLWVxdWl2PSJDb250ZW50 LVR5cGUiIGNvbnRlbnQ9InRleHQvaHRtbDsKICAgICAgICBjaGFyc2V0PXdpbmRvd3MtMTI1MiI+ CiAgICAgIDxtZXRhIG5hbWU9IkdlbmVyYXRvciIgY29udGVudD0iTWljcm9zb2Z0IFdvcmQgMTUg KGZpbHRlcmVkCiAgICAgICAgbWVkaXVtKSI+CiAgICAgIDxzdHlsZT48IS0tCi8qIEZvbnQgRGVm aW5pdGlvbnMgKi8KQGZvbnQtZmFjZQoJe2ZvbnQtZmFtaWx5OiJDYW1icmlhIE1hdGgiOwoJcGFu b3NlLTE6MiA0IDUgMyA1IDQgNiAzIDIgNDt9CkBmb250LWZhY2UKCXtmb250LWZhbWlseTpDYWxp YnJpOwoJcGFub3NlLTE6MiAxNSA1IDIgMiAyIDQgMyAyIDQ7fQovKiBTdHlsZSBEZWZpbml0aW9u cyAqLwpwLk1zb05vcm1hbCwgbGkuTXNvTm9ybWFsLCBkaXYuTXNvTm9ybWFsCgl7bWFyZ2luOjBj bTsKCW1hcmdpbi1ib3R0b206LjAwMDFwdDsKCWZvbnQtc2l6ZToxMS4wcHQ7Cglmb250LWZhbWls eToiQ2FsaWJyaSIsInNhbnMtc2VyaWYiOwoJbXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tVVM7fQph OmxpbmssIHNwYW4uTXNvSHlwZXJsaW5rCgl7bXNvLXN0eWxlLXByaW9yaXR5Ojk5OwoJY29sb3I6 IzA1NjNDMTsKCXRleHQtZGVjb3JhdGlvbjp1bmRlcmxpbmU7fQphOnZpc2l0ZWQsIHNwYW4uTXNv SHlwZXJsaW5rRm9sbG93ZWQKCXttc28tc3R5bGUtcHJpb3JpdHk6OTk7Cgljb2xvcjojOTU0Rjcy OwoJdGV4dC1kZWNvcmF0aW9uOnVuZGVybGluZTt9CnNwYW4uRW1haWxTdHlsZTE3Cgl7bXNvLXN0 eWxlLXR5cGU6cGVyc29uYWwtY29tcG9zZTsKCWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1z ZXJpZiI7Cgljb2xvcjp3aW5kb3d0ZXh0O30KLk1zb0NocERlZmF1bHQKCXttc28tc3R5bGUtdHlw ZTpleHBvcnQtb25seTsKCWZvbnQtZmFtaWx5OiJDYWxpYnJpIiwic2Fucy1zZXJpZiI7Cgltc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1VUzt9CkBwYWdlIFdvcmRTZWN0aW9uMQoJe3NpemU6NjEyLjBw dCA3OTIuMHB0OwoJbWFyZ2luOjcyLjBwdCA3Mi4wcHQgNzIuMHB0IDcyLjBwdDt9CmRpdi5Xb3Jk U2VjdGlvbjEKCXtwYWdlOldvcmRTZWN0aW9uMTt9Ci0tPjwvc3R5bGU+PCEtLVtpZiBndGUgbXNv IDldPjx4bWw+CjxvOnNoYXBlZGVmYXVsdHMgdjpleHQ9ImVkaXQiIHNwaWRtYXg9IjEwMjYiIC8+ CjwveG1sPjwhW2VuZGlmXS0tPjwhLS1baWYgZ3RlIG1zbyA5XT48eG1sPgo8bzpzaGFwZWxheW91 dCB2OmV4dD0iZWRpdCI+CjxvOmlkbWFwIHY6ZXh0PSJlZGl0IiBkYXRhPSIxIiAvPgo8L286c2hh cGVsYXlvdXQ+PC94bWw+PCFbZW5kaWZdLS0+CiAgICAgIDxkaXYgY2xhc3M9IldvcmRTZWN0aW9u MSI+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+SGksPG86cD48L286cD48L3A+CiAgICAg ICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CiAgICAgICAgPHAg Y2xhc3M9Ik1zb05vcm1hbCI+SeKAmW0gaW4gYXQgdGhlIGRlZXAgZW5k4oCmPG86cD48L286cD48 L3A+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CiAg ICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+TG9va2luZyBmb3Igc29tZSBhZHZpY2Ugb24gaWYg dGhpcyBpcwogICAgICAgICAgcG9zc2libGUg4oCTIGFuZCB3aGF0IHZlcnNpb24gSSBzaG91bGQg dHJ5IEkgd2l0aCBpZiBzby48bzpwPjwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0iTXNvTm9y bWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0iTXNvTm9ybWFsIj5X ZSBhcmUgYSBoZWF2aWx5IFZNd2FyZSBvcmllbnRlZCBjb21wYW554oCmCiAgICAgICAgICBidXQg SSBhbSB0cnlpbmcgdG8gZ2V0IFJIRVYvb1ZpcnQgaW4gdGhlIGRvb3IuJm5ic3A7IEhvbmVzdGx5 IEkKICAgICAgICAgIHdvdWxkIHByZWZlciBvVmlydCwgYnV0IGlmIHRoZXkgaW5zaXN0IG9uIGVu dGVycHJpc2Ugc3VwcG9ydCBJCiAgICAgICAgICBnbyB0byBSZWRoYXQuPG86cD48L286cD48L3A+ CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CiAgICAg ICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+U28gSSBoYXZlIGhhZCBhIHBsYXkgd2l0aCBWTXdhcmUg VlNBTiBhbmQKICAgICAgICAgIGtub3cgaG93IGl0IHdvcmtzLiZuYnNwOyBJIGFtIHRyeWluZyB0 byBtb3JlIG9yIGxlc3MgcmVwbGljYXRlIHRoaXMKICAgICAgICAgIHNldHVwIHVzaW5nIG9wZW4g c291cmNlLjxvOnA+PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+ Jm5ic3A7PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPlNvIFZTQU4gdXNl cyAzIG9yIG1vcmUgRVNYaSBub2Rlcywgd2l0aAogICAgICAgICAgbG9jYWwgaGFyZGRpc2tzIGFz IHRoZSBzdG9yYWdlIG1lZGl1bSBhcyBhIHZpcnR1YWwgU0FOLiZuYnNwOyBUaGUKICAgICAgICAg IFNBTiBwZXJmb3JtYW5jZSBpcyBxdWl0ZSBoaWdoIGFzIHlvdSBwdXQgYXQgbGVhc3QgMSBTU0Qg aW4KICAgICAgICAgIGVhY2ggbm9kZS48bzpwPjwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0i TXNvTm9ybWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0iTXNvTm9y bWFsIj5XaGF0IGlzIG5pY2UgaXMgeW91IGRvbuKAmXQgaGF2ZSB0aGUgTkFTCiAgICAgICAgICBl bGVtZW50IGhlcmUuJm5ic3A7IFZNcyBhcmUgb24gdGhlIGxvY2FsIHN0b3JhZ2UsIHdoaWNoIGlz CiAgICAgICAgICBwYXJ0aWFsbHkgU1NELCBzbyBwZXJmb3JtYW5jZSBpcyBxdWl0ZSBnb29kLjxv OnA+PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9v OnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPkkgd2VudCB0byBhIFJlZGhhdCBw cmVzZW50YXRpb24gb24gUmVkaGF0CiAgICAgICAgICBzdG9yYWdlIGFuZCBHbHVzdGVyRlMgYW5k IGJhc2ljYWxseSB0aGlzIGFjdHMgYXMgYSBiaWcKICAgICAgICAgIHNvZnR3YXJlIGRlZmluZWQg TkFTIHdoaWNoIGRvZXMgc29tZSBwcmV0dHkgY29vbCB0aGluZ3MsIGJ1dAogICAgICAgICAgdGhh dOKAmXMgbm90IGV4YWN0bHkgd2hhdCBJIG5lZWQuPG86cD48L286cD48L3A+CiAgICAgICAgPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CiAgICAgICAgPHAgY2xhc3M9 Ik1zb05vcm1hbCI+SSB3b3VsZCBsaWtlIHRvIGJ1aWxkIG9WaXJ0IG9uIHRvcCBvZgogICAgICAg ICAgQ2VudE9TIE9zc+KApiB0aGF0IGhhdmUgbG9jYWwgc3RvcmFnZSBpbiB0aGVtIHRoYXQgaXMK ICAgICAgICAgIGRpc3RyaWJ1dGVkIGFuZCByZWR1bmRhbnQgaW4gdGhlIGV2ZW50IG9mIGEgbm9k ZSBmYWlsdXJlLjxvOnA+PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxv OnA+Jm5ic3A7PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPkFuZCBJIHBy b2JhYmx5IG5lZWQgdG8gdHJ5IGFuZCBidWlsZCB0aGlzCiAgICAgICAgICBpbiBhIGxhYiB1bmRl ciBGdXNpb24gb24gbXkgTWFjIEJvb2sgcHJvIHRvIGJlZ2luIHdpdGggYW55d2F5CiAgICAgICAg ICAodGhhdCBiaXQgSeKAmW0gZmFyaWx5IGNvbmZpZGVudCB3aXRoKSBhbmQgaWYgSSBnZXQgdGhh dCB3b3JraW5nCiAgICAgICAgICBjYW4gcHJvYmFibHkgZ2V0IHNvbWUgb2xkZXIga2l0IHRvIHRy eSBpdCBvdXQgZm9yIHJlYWwgYXMgYQogICAgICAgICAgUG9DIHRvIGZldyBwZW9wbGUuPG86cD48 L286cD48L3A+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PG86cD4mbmJzcDs8L286cD48 L3A+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+U28sIEnigJl2ZSBzZXQgdXAgb1ZpcnQg YmVmb3JlIHRoYXQgc2hvdWxkIGJlCiAgICAgICAgICBvaywgSSBoYXZlbuKAmXQgc2V0dXAgR2x1 c3RlciBpcyB0aGVyZSBhbnkgZG9jdW1lbnRzIHRoYXQgd291bGQKICAgICAgICAgIGhlbHAgbWUg ZG93biB0aGlzIHJvYWQgYW5kIG1ha2Ugc3VyZSBJIHN0YXJ0IG91dCB1c2luZyB0aGUKICAgICAg ICAgIGJlc3QgdmVyc2lvbi48bzpwPjwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0iTXNvTm9y bWFsIj48bzpwPiZuYnNwOzwvbzpwPjwvcD4KICAgICAgICA8cCBjbGFzcz0iTXNvTm9ybWFsIj5B bnkgYWR2aWNlIG9yIHBvaW50ZXJzIHdvdWxkIGJlIGdyYXRlZnVsbHkKICAgICAgICAgIHJlY2Vp dmVkLjxvOnA+PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5i c3A7PC9vOnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPlRoYW5rczxvOnA+PC9v OnA+PC9wPgogICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxvOnA+Jm5ic3A7PC9vOnA+PC9w PgogICAgICAgIDx0YWJsZSBjbGFzcz0iTXNvTm9ybWFsVGFibGUiIHN0eWxlPSJ3aWR0aDo0ODAu MHB0IiBib3JkZXI9IjAiIGNlbGxwYWRkaW5nPSIwIiBjZWxsc3BhY2luZz0iMCIgd2lkdGg9IjY0 MCI+CiAgICAgICAgICA8dGJvZHk+CiAgICAgICAgICAgIDx0cj4KICAgICAgICAgICAgICA8dGQg c3R5bGU9IndpZHRoOjQ4MC4wcHQ7cGFkZGluZzowY20gMGNtIDBjbSAwY20iIHdpZHRoPSI2NDAi PgogICAgICAgICAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PGI+PHNwYW4gc3R5bGU9ImZv bnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1z ZXJpZiZxdW90Oztjb2xvcjojMDA4MkQ1O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj5CaWxs CiAgICAgICAgICAgICAgICAgICAgICBEb3NzZXR0PC9zcGFuPjwvYj48c3BhbiBzdHlsZT0ibXNv LWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KICAgICAgICAg ICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0 O2ZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNv LWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPlN5c3RlbXMKICAgICAgICAgICAgICAgICAgICBBcmNo aXRlY3Q8L3NwYW4+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj48bzpw PjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgPC90ZD4KICAgICAgICAgICAgPC90cj4K ICAgICAgICAgICAgPHRyPgogICAgICAgICAgICAgIDx0ZCBzdHlsZT0id2lkdGg6NDgwLjBwdDtw YWRkaW5nOjBjbSAwY20gMGNtIDBjbSIgd2lkdGg9IjY0MCI+CiAgICAgICAgICAgICAgICA8cCBj bGFzcz0iTXNvTm9ybWFsIj48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZh bWlseTomcXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0 LWxhbmd1YWdlOkVOLUdCIj5UZWNoCiAgICAgICAgICAgICAgICAgICAgICBDZW50cmFsIOKAkyBH bG9iYWwgRW5naW5lZXJpbmcgU2VydmljZXM8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJtc28tZmFy ZWFzdC1sYW5ndWFnZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAg ICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7bXNv LWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPiZuYnNwOzwvc3Bhbj48c3BhbiBzdHlsZT0ibXNvLWZh cmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KICAgICAgICAgICAg ICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0 O2ZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNv LWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPlQ8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJmb250LXNp emU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3NhbnMtc2VyaWYm cXVvdDs7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPiZuYnNwOys0NAogICAgICAgICAgICAg ICAgICAgICgwKTE5MjMgMjc5MzUzPC9zcGFuPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5n dWFnZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgPHAgY2xh c3M9Ik1zb05vcm1hbCI+PGI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1p bHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1s YW5ndWFnZTpFTi1HQiI+TTwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7 Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28t ZmFyZWFzdC1sYW5ndWFnZTpFTi1HQiI+Jm5ic3A7KzQ0CiAgICAgICAgICAgICAgICAgICAgKDAp Nzc3IDU5MCA4NjEyPC9zcGFuPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1H QiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgPHAgY2xhc3M9Ik1zb05v cm1hbCI+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj48YSBtb3otZG8t bm90LXNlbmQ9InRydWUiIGhyZWY9Im1haWx0bzpqZWZmLmdvbnphbGV6QHBiLmNvbSI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1 b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjpwdXJwbGUiPmI8L3NwYW4+PC9hPjwvc3Bhbj48c3Bh biBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtBcmlhbCZxdW90Oywm cXVvdDtzYW5zLXNlcmlmJnF1b3Q7O2NvbG9yOiMwMDgyRDU7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6 RU4tR0IiPjxhIG1vei1kby1ub3Qtc2VuZD0idHJ1ZSIgaHJlZj0ibWFpbHRvOmlsbC5kb3NzZXR0 QHBiLmNvbSI+PHNwYW4gc3R5bGU9ImNvbG9yOmJsdWUiPmlsbC5kb3NzZXR0QHBiLmNvbTwvc3Bh bj48L2E+PC9zcGFuPjxzcGFuIHN0eWxlPSJmb250LXNpemU6MTIuMHB0O2ZvbnQtZmFtaWx5OiZx dW90O1RpbWVzIE5ldwpSb21hbiZxdW90OywmcXVvdDtzZXJpZiZxdW90Ozttc28tZmFyZWFzdC1s YW5ndWFnZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1p bHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMDA4MkQ1 O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj48YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhy ZWY9InBpdG5leWJvd2VzLmNvbSI+PHNwYW4gc3R5bGU9ImNvbG9yOiMwMDgyRDUiPnBpdG5leWJv d2VzLmNvbTwvc3Bhbj48L2E+PC9zcGFuPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFn ZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgPHAgY2xhc3M9 Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7bXNvLWZhcmVhc3QtbGFu Z3VhZ2U6RU4tR0IiPjxicj4KICAgICAgICAgICAgICAgICAgPC9zcGFuPjxiPjxzcGFuIHN0eWxl PSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7LCZxdW90O3Nh bnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPlBpdG5leQogICAgICAg ICAgICAgICAgICAgICAgQm93ZXM8L3NwYW4+PC9iPjxzcGFuIHN0eWxlPSJtc28tZmFyZWFzdC1s YW5ndWFnZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9wPgogICAgICAgICAgICAgICAgPHAg Y2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1p bHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1s YW5ndWFnZTpFTi1HQiI+NgogICAgICAgICAgICAgICAgICAgIEhlcmN1bGVzIFdheSB8IExlYXZl c2RlbiB8IFdEMjUgN0dTIHwgVUs8L3NwYW4+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1 YWdlOkVOLUdCIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8cCBjbGFz cz0iTXNvTm9ybWFsIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTom cXVvdDtBcmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWxhbmd1 YWdlOkVOLUdCIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdl OkVOLUdCIj48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgPC90ZD4KICAgICAg ICAgICAgPC90cj4KICAgICAgICAgICAgPHRyPgogICAgICAgICAgICAgIDx0ZCBzdHlsZT0id2lk dGg6NDgwLjBwdDtwYWRkaW5nOjBjbSAwY20gMGNtIDBjbSIgd2lkdGg9IjY0MCI+PGJyPgogICAg ICAgICAgICAgIDwvdGQ+CiAgICAgICAgICAgIDwvdHI+CiAgICAgICAgICAgIDx0cj4KICAgICAg ICAgICAgICA8dGQgc3R5bGU9IndpZHRoOjQ4MC4wcHQ7cGFkZGluZzowY20gMGNtIDBjbSAwY20i IHdpZHRoPSI2NDAiPgogICAgICAgICAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4g c3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1 b3Q7c2Fucy1zZXJpZiZxdW90Oztjb2xvcjojMDA4MkQ1O21zby1mYXJlYXN0LWxhbmd1YWdlOkVO LUdCIj48YSBtb3otZG8tbm90LXNlbmQ9InRydWUiIGhyZWY9Imh0dHA6Ly90ZWFtc2l0ZS5pbnNp ZGUucGIuY29tL3NpdGVzL1RDUFBNL0dFUy9TaXRlUGFnZXMvSG9tZS5hc3B4Ij48c3BhbiBzdHls ZT0iY29sb3I6IzAwODJENSI+TGVhcm4gbW9yZSBhYm91dCBHbG9iYWwKICAgICAgICAgICAgICAg ICAgICAgICAgRW5naW5lZXJpbmcgU2VydmljZXM8L3NwYW4+PC9hPjwvc3Bhbj48c3BhbiBzdHls ZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPjxvOnA+PC9vOnA+PC9zcGFuPjwvcD4KICAg ICAgICAgICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiIHN0eWxlPSJwYWdlLWJyZWFrLWFmdGVy OmF2b2lkIj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtB cmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWxhbmd1YWdlOkVO LUdCIj4mbmJzcDs8L3NwYW4+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdC Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8cCBjbGFzcz0iTXNvTm9y bWFsIiBzdHlsZT0icGFnZS1icmVhay1hZnRlcjphdm9pZCI+PHNwYW4gc3R5bGU9ImZvbnQtc2l6 ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJpYWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZx dW90Ozttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1HQiI+SW4KICAgICAgICAgICAgICAgICAgICBF bmdpbmVlcmluZz88L3NwYW4+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdC Ij48bzpwPjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8cCBjbGFzcz0iTXNvTm9y bWFsIj48Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtB cmlhbCZxdW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWxhbmd1YWdlOkVO LUdCIj5SYWlzZQogICAgICAgICAgICAgICAgICAgICAgYSB0aWNrZXQgdmlhIFJlbWVkeSBBbnl3 aGVyZSBbPHNwYW4gc3R5bGU9ImNvbG9yOiMwMDgyRDUiPjxhIG1vei1kby1ub3Qtc2VuZD0idHJ1 ZSIgaHJlZj0idGVhbXNpdGUuaW5zaWRlLnBiLmNvbS9zaXRlcy9UQ1BQTS9HRVMvTGlzdHMvQ3Jl YXRlJTIwSW5jaWRlbnQvbmV3SW5jaWRlbnQuYXNweCI+PHNwYW4gc3R5bGU9ImNvbG9yOiMwMDgy RDUiPkhFUkU8L3NwYW4+PC9hPjwvc3Bhbj5dPHNwYW4gc3R5bGU9ImNvbG9yOmJsdWUiPiZuYnNw Ozwvc3Bhbj50YWtlcyBsZXNzIHRoYW4gYQogICAgICAgICAgICAgICAgICAgICAgbWludXRlPC9z cGFuPjwvYj48c3BhbiBzdHlsZT0ibXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPjxvOnA+PC9v OnA+PC9zcGFuPjwvcD4KICAgICAgICAgICAgICAgIDxwIGNsYXNzPSJNc29Ob3JtYWwiPjxiPjxz cGFuIHN0eWxlPSJmb250LXNpemU6MTAuMHB0O2ZvbnQtZmFtaWx5OiZxdW90O0FyaWFsJnF1b3Q7 LCZxdW90O3NhbnMtc2VyaWYmcXVvdDs7bXNvLWZhcmVhc3QtbGFuZ3VhZ2U6RU4tR0IiPiZuYnNw Ozwvc3Bhbj48L2I+PHNwYW4gc3R5bGU9Im1zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj48bzpw PjwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgICAgICAgICA8cCBjbGFzcz0iTXNvTm9ybWFsIj48 Yj48c3BhbiBzdHlsZT0iZm9udC1zaXplOjEwLjBwdDtmb250LWZhbWlseTomcXVvdDtBcmlhbCZx dW90OywmcXVvdDtzYW5zLXNlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWxhbmd1YWdlOkVOLUdCIj5D bG91ZEZvcm1zCiAgICAgICAgICAgICAgICAgICAgICBVc2VyIEd1aWRlIGF2YWlsYWJsZTwvc3Bh bj48L2I+PHNwYW4gc3R5bGU9ImZvbnQtc2l6ZToxMC4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7QXJp YWwmcXVvdDssJnF1b3Q7c2Fucy1zZXJpZiZxdW90Ozttc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1H QiI+Jm5ic3A7WzxiPjxzcGFuIHN0eWxlPSJjb2xvcjojMDA4MkQ1Ij48YSBtb3otZG8tbm90LXNl bmQ9InRydWUiIGhyZWY9Imh0dHBzOi8vZGwuZHJvcGJveHVzZXJjb250ZW50LmNvbS91LzYyMTk0 NDEvTWFuYWdlSVElMjAtJTIwVXNlciUyME1hbnVhbCUyMFBCJTIwdjUucGRmIj48c3BhbiBzdHls ZT0iY29sb3I6IzAwODJENSI+SEVSRTwvc3Bhbj48L2E+PC9zcGFuPjwvYj5dPC9zcGFuPjxzcGFu IHN0eWxlPSJtc28tZmFyZWFzdC1sYW5ndWFnZTpFTi1HQiI+PG86cD48L286cD48L3NwYW4+PC9w PgogICAgICAgICAgICAgIDwvdGQ+CiAgICAgICAgICAgIDwvdHI+CiAgICAgICAgICA8L3Rib2R5 PgogICAgICAgIDwvdGFibGU+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05vcm1hbCI+PHNwYW4gc3R5 bGU9ImZvbnQtc2l6ZToxMi4wcHQ7Zm9udC1mYW1pbHk6JnF1b3Q7VGltZXMgTmV3CiAgICAgICAg ICAgIFJvbWFuJnF1b3Q7LCZxdW90O3NlcmlmJnF1b3Q7O21zby1mYXJlYXN0LWxhbmd1YWdlOkVO LUdCIj48bzpwPiZuYnNwOzwvbzpwPjwvc3Bhbj48L3A+CiAgICAgICAgPHAgY2xhc3M9Ik1zb05v cm1hbCI+PG86cD4mbmJzcDs8L286cD48L3A+CiAgICAgIDwvZGl2PgogICAgICA8YnI+CiAgICAg IDxocj4KICAgICAgPGZvbnQgY29sb3I9IkJsYWNrIiBmYWNlPSJBcmlhbCIgc2l6ZT0iMiI+PGJy PgogICAgICA8L2ZvbnQ+CiAgICAgIDxicj4KICAgICAgPGZpZWxkc2V0IGNsYXNzPSJtaW1lQXR0 YWNobWVudEhlYWRlciI+PC9maWVsZHNldD4KICAgICAgPGJyPgogICAgICA8cHJlIHdyYXA9IiI+ X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVXNlcnMgbWFp bGluZyBsaXN0CjxhIGNsYXNzPSJtb3otdHh0LWxpbmstYWJicmV2aWF0ZWQiIGhyZWY9Im1haWx0 bzpVc2Vyc0BvdmlydC5vcmciPlVzZXJzQG92aXJ0Lm9yZzwvYT4KPGEgY2xhc3M9Im1vei10eHQt bGluay1mcmVldGV4dCIgaHJlZj0iaHR0cDovL2xpc3RzLm92aXJ0Lm9yZy9tYWlsbWFuL2xpc3Rp bmZvL3VzZXJzIj5odHRwOi8vbGlzdHMub3ZpcnQub3JnL21haWxtYW4vbGlzdGluZm8vdXNlcnM8 L2E+CjwvcHJlPgogICAgPC9ibG9ja3F1b3RlPgogICAgPGJyPgogIDwvYm9keT48L2h0bWw+ ----_com.android.email_82515713991120--

1 0

how to prepare for an engine re-install
by Robert Story 18 Feb '15

18 Feb '15

--Sig_/9FfbJVEsL_QD6_U8ceRqc4+ Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable I'm running ovirt (engine 3.4.0-1.el6) on a dedicated engine machine with a degraded raid array. This has caused the root filesystem to go read-only, and the engine crashed. I can restart the engine, but after doing so, the web page just loads a blank page. All nodes and VMs seem to be fine, I just can't manage them via oVirt. Because I'm offsite, my recovery option is likely limited to a reinstall of the OS and ovirt. I'm hoping for some pointers on what I can do in advance to minimize the pain of this transition.=20 - Am I going to be able to recover without having to shut down VMs, exporting and importing? - what directories should I back up? - when I reinstall, should I upgrade and install 3.5.1, or first install 3.4, get it working, and then think about upgrading? - Is it possible/recommended/crazy to migrate to a hosted engine instead of a dedicated machine? So I want to prepare for the worst before I try a reboot. If I'm lucky, the reboot will at least come up with a writable filesystem, and I won't need to re-install. If it doesn't come up, I want to be as ready as possible for the reinstall. Any hints and/or tips appreciated.. (I'll also be hanging out in the irc channel if anyone wants to respond in that venue..) Robert --=20 Senior Software Engineer @ Parsons --Sig_/9FfbJVEsL_QD6_U8ceRqc4+ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAEBAgAGBQJU5KvuAAoJEMHFVuy5l8Y48rwQAKbV/omCGFyztV/QDBF+T4qX ZtI7+j263nisA38eN7XHbfdfvEux00rqYMy9n7e+IPmsvbMIdPqps9BO9iH7fFjx 7ZtkMqyhKEju6vCiIwdWi3/KWPAhfufQvPXGOqMdgro2tim6yC1cwWQS/bAVexKh RnvntPo3WAdDdE986OdQ4JofJQqUwKAde7oPEWMjhbzx4Mxb/NSq7ExhyfWj0+wh 2iWqEEQLs2KaSOSzaqbXjIoJhxVmQPxToKdUxs1hmAhB4X/lFdIRkSY/x7SMhrKV eDB+4vd55saNz5dPfNSokQ1tTFR7z4iRkOEh5knKlRJwNdrabm+oIwrtrUbR4BX+ L0sHdB12x08aM2aio1OzY5xMLhpJSD/YiAfXs8oCPtHRbYrO2HyvOjilktY4azeO bZkCard2zxUO+UVWOEllcrXVpEgYRMlaCwhs1vsEI0mArpe0Ip9zEGYVN3IgHR6w sKrWi9u2JueqTiJOmuvGwE4EEsYKW4KRId6r1sS16neWAcbuBLjLMVzZkZ7pET5j hoERFPonAURGz1wXUuW8xK/hpZJ4xW71QpqKQfs86rwP1rIMnw3KaFqtf0AFUEPs LTKT44ibC6DNMNj3g3Kdj0iUW6/IM5xRL9QZ4exKipEyJxPf2Er6SZoYC2HffKfo gqTOxAdaa3bCVIn19Zro =ife/ -----END PGP SIGNATURE----- --Sig_/9FfbJVEsL_QD6_U8ceRqc4+--

2 2