<div dir="ltr"><div><br></div><div>I found vdsClient and ran a few commands.</div><div><br></div><div>
<p class="p1"><span class="s1">[root@geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo 88b69eba-ef4f-4dbe-ba53-20dadd424d0e</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>version = 3</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>role = Master</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>remotePath = hostedstorage.pazion.nl:/opt/hosted-engine</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>type = NFS</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>class = Data</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']</span></p>
<p class="p1"><span class="s1"><span class="Apple-tab-span"> </span>name = hostedengine_nfs</span></p><p class="p1"><span class="s1"><br></span></p><p class="p1"><span class="s1">[root@geisha-1 ~]# vdsClient -s 0 connectStoragePool 499b208c-9de9-4a2a-97de-30f410b4e6d4</span></p><p class="p1"><span class="s1">Error using command: list index out of range </span></p><p class="p2"><span class="s1"></span><br></p><p class="p1"><span class="s1"></span></p><p class="p1">...</p><div><br></div>
<p class="p2"><span class="s1"></span><br></p>
<p class="p1"><span class="s1">[root@geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo 499b208c-9de9-4a2a-97de-30f410b4e6d4</span></p>
<p class="p1"><span class="s1">Unknown pool id, pool not connected: ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)</span></p><p class="p1"><span class="s1"><br></span></p>
<p class="p1"><span class="s1">[root@geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList</span></p>
<p class="p1"><span class="s1">00000002-0002-0002-0002-000000000385</span></p><p class="p1"><br></p></div>Can I manually create the missing storage pool which is missing with:<div><div><br><div>vdsClient -s 0<span class="Apple-tab-span" style="white-space:pre"> </span>createStoragePool 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs 88b69eba-ef4f-4dbe-ba53-20dadd424d0e<br></div><div><br></div><div>If this would be the solution, I still wonder why there is no existing storagePool for the storageDomain.</div><div><br></div></div></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <<a href="mailto:paul@pazion.nl">paul@pazion.nl</a>>:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>I still keep getting <span style="white-space:pre-wrap"> </span></div></div></div><div dir="ltr"><div><div><br></div><div><span style="white-space:pre-wrap"> </span>The Hosted Engine Storage Domain doesn't exist. It should be imported into the setup.</div><div><br></div></div></div><div dir="ltr"><div><div>I went digging a little further and found this in the engine.log</div><div><br></div><div><span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span>2016-02-08 22:30:25,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (org.ovirt.thread.pool-8-thread-41) [] FINISH, HSMGetStorageDomainInfoVDSCommand, return: <StorageDomainStatic:{name='hostedengine_nfs', id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'}, 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427</div><div><span style="white-space:pre-wrap"> </span>2016-02-08 22:30:25,083 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (org.ovirt.thread.pool-8-thread-41) [] FINISH, GetExistingStorageDomainListQuery, log id: 6ca437b1</div><div><span style="white-space:pre-wrap"> </span>2016-02-08 22:30:25,084 INFO [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'</div><div><span style="white-space:pre-wrap"> </span>2016-02-08 22:30:25,102 WARN [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons: VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST</div><div><br></div><div>So it looks like it can't find the storage domain. When I check vdsm.log I found an error:</div><div><span style="white-space:pre-wrap"> </span></div><div><span style="white-space:pre-wrap"> </span>Thread-711902::ERROR::2016-02-08 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': {'message': "Unknown pool id, pool not connected: ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}</div><div><br></div><div>It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is also trying to connect to in the engine.log.</div><div><br></div><div>I did found <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1269768" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1269768</a> and <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1294457" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1294457</a></div><div><br></div><div><div>Can it be I have setup another DC with a master iscsi storage which is now offline? ( 1 DC with Cluster, VMs with hosted engine and master storage FC all up and running. )</div></div><div><br></div><div>Not sure where to go from here. Can I somehow restart this storage domain / do a new hosted-engine deploy with different settings?<br></div><div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr">Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <<a href="mailto:paul@pazion.nl" target="_blank">paul@pazion.nl</a>>:<br></div></div></div></div><div dir="ltr"><div dir="ltr"><div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>>:<br></div></div></div><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <span dir="ltr"><<a href="mailto:paul@pazion.nl" target="_blank">paul@pazion.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks for your response.<div><br></div><div>I have reverted the commented entries and restarted ovirt-ha-agent.</div><div>Now status on the daemon gives the error again:</div><div><a href="http://screencast.com/t/DCZo6QJM1" target="_blank">http://screencast.com/t/DCZo6QJM1</a><br></div><div><br></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Here I'm a bit confused: is this a fresh hosted-engine deployment with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?</div></div></div></div></blockquote><div><br></div></div></div><div dir="ltr"><div class="gmail_quote"><div>I have reinstalled host ( complete OS reinstall) and ran hosted-engine --deploy for redeployment on an existing oVirt ( hosted-engine ).</div><div><br></div><div>As described in previous mail, during the deployment I got errors about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/... </div><div>I created these by hand and then I ran deployment again and it worked.</div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div>The hosted engine domain is not shown in the web storage overview...<br></div><div>When I start import it shows a warning: <a href="http://screencast.com/t/7HfJ9qWFwb2L" target="_blank">http://screencast.com/t/7HfJ9qWFwb2L</a><br></div><div><br></div><div><div>Can I manually import the hostedengine storage? </div><div>( Hosted engine is visible in web interface and I have running VMs on another storage domain ).</div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It should auto-import it.</div><div>Adding Roy here.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div></div></div></div></blockquote></div></div><div dir="ltr"><div class="gmail_quote"><div>I believe this auto-import is failing... </div><div>Where can I find logs about this auto-import?</div></div></div><div dir="ltr"><div class="gmail_quote"><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div><br><div class="gmail_quote"><div dir="ltr">Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <<a href="mailto:stirabos@redhat.com" target="_blank">stirabos@redhat.com</a>>:<br></div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <span dir="ltr"><<a href="mailto:paul@pazion.nl" target="_blank">paul@pazion.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Update:</div></div></blockquote><div><br></div><div>The OVF_STORE volume will contain the engine VM description in a format that the engine can easily handle.</div><div>It will be created by the engine when:</div><div>- it imported the hosted-engine storage domain</div><div>- the engineVM is visible</div><div>- the datacenter is up (you need at least one additional regular storage domain)</div><div><br></div><div>Please make sure that all these conditions are met.</div><div><br></div><div>If the OVF_STORE is not there or it's not accessible, ovirt-ha-agent will fallback to the initial vm.conf from deployment time. You see a warning since we are failing back to an initial status and this can temporary override your latest customization to the engine VM configuration.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>1. I was able to remove the warnirngs about the not found ovf / vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were gone. ( I believe this is not a permanent solution ).</div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Please revert it.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><br></div><div>2. This morning I noticed the warning in the event log.<br><div>"The Hosted Engine Storage Domain doesn't exist. It should be imported into the setup."<br></div></div></div></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It's just a side effect of your point 1, please revert it.</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div></div><div><br></div><div>Any help would be appreciated.<br><br><div class="gmail_quote"><div><div dir="ltr">Best Regards,<br><br><div>Paul Groeneweg<div><br></div></div></div></div><div dir="ltr">Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <<a href="mailto:paul@pazion.nl" target="_blank">paul@pazion.nl</a>>:<br></div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><br></div><div>I have been upgrading our ovirt hosts to el7 to be able to run version 3.6.</div><div>When deploying a new host have problems with the shared OVF storage config part.</div><div><br></div><div>1. I get an error during deploy:</div><div><br></div><div>[ INFO ] Stage: Setup validation</div><div>[ ERROR ] Failed to execute stage 'Setup validation': failed to read metadata: [Errno 2] No such file or directory: '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'</div><div>[ INFO ] Stage: Clean up</div><div><br></div><div>It seems the symlink of hosted-engine.metadata is missing.</div><div>So I added manually the /var/run/vdsm/storage folder with subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.</div><div>Then I reran the hosted-engine deploy and this time all went well.</div><div><br></div><div>2. Now when the host is installed I notice the following in the logs</div><div><br></div><div><span style="white-space:pre-wrap"> </span>....</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing images</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading vm.conf from the shared storage domain</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying to get a fresher copy of vm configuration from the OVF_STORE</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find OVF_STORE</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable to get vm.conf from OVF_STORE, falling back to initial vm.conf</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - trying to restart agent</div><div><span style="white-space:pre-wrap"> </span>Feb 5 22:33:08 geisha-2 ovirt-ha-agent: ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - trying to restart agent</div><div><br></div><div>So at this point I was a bit stuck.</div><div>I removed complete ovirt36 release and installed ovirt35 for the hosted engine deploy.</div><div><br></div><div>This time all went without any problem directly.</div><div>From there I retried upgrade and same problems again.</div><div><br></div><div>I run ovirt hosted-engine on nfs3. I did found <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1159314" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1159314</a> </div><div>I added no_subtree_check and tried deploy again, but again without luck.</div><div><br></div><div>I believe the hosted engine configs are not being migrated to the shared storage. And my problem has todo with <a href="http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage" target="_blank">http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage</a></div><div><br></div><div><div>And found /etc/ovirt-hosted-engine/hosted-engine.conf, </div><div>conf_volume_UUID, conf_image_UUID</div><div> </div></div><div><br></div><div>Hosts run el7</div><div>Hosted engine runs el6</div><div>Hosted engine storage is through NFS3</div><div><br></div><div>Can I import the hosted engine storage in to engine ( in web interface )?</div><div>Or how can I create or trigger manually the OVF store the logs complain about?</div></div><div><br><div><div dir="ltr">Best Regards,</div><div dir="ltr">Paul Groeneweg<div><br></div></div></div></div></div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T <a href="tel:%2B31%2026%203020038" value="+31263020038" target="_blank">+31 26 3020038</a></div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div></blockquote></div></div></div></div></div></div><div><div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T <a href="tel:%2B31%2026%203020038" value="+31263020038" target="_blank">+31 26 3020038</a></div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div>
</div></div></blockquote></div></div></div></blockquote></div></div></div></div></div><div><div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T <a href="tel:%2B31%2026%203020038" value="+31263020038" target="_blank">+31 26 3020038</a></div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div>
</div></div></blockquote></div></div></div></blockquote></div></div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T +31 26 3020038</div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div></blockquote></div></div></div></div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T +31 26 3020038</div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div></blockquote></div><div dir="ltr">-- <br></div><div dir="ltr"><span>Met vriendelijke groeten,</span><br><br><div>Paul Groeneweg<div>Pazion</div><div>Webdevelopment - Hosting - Apps</div><div><br></div><div>T +31 26 3020038</div><div>M +31 614 277 577</div><div>E <a>paul@pazion.nl</a></div><div><br></div><div> ***disclaimer*** </div><div>"This e-mail and any attachments thereto may contain information which is confidential and/or protected by intellectual property rights and are intended for the sole use of the recipient(s) named above. Any use of the information contained herein (including, but not limited to, total or partial reproduction, communication or distribution in any form) by persons other than the designated recipient(s) is prohibited. If you have received this e-mail in error, please notify the sender either by telephone or by e-mail and delete the material from any computer. Thank you for your cooperation."</div></div></div>