Re: [ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE

On Tue, Feb 9, 2016 at 10:53 AM, Paul Groeneweg | Pazion <paul(a)pazion.nl&gt; wrote: > The original (first host) was removed ( about a year a go I removed this > one ) and it is also removed from hosted-engine --vm-status overview. > > I ran 'hosted-engine --deploy' again on all the hosts because I upgraded > all hosts to EL7. > Upgrading hosted-engine from 3.5 on el6 to 3.6 on el7 requires some additional steps: https://bugzilla.redhat.com/show_bug.cgi?id=1188679 Adding Didi and Rafael who are working on that flow. > > ( I have als ran engine-setup for the upgrade ) > > How can I fix this missing spUUID issue? > Can I manually add this somehow or run somsetup again to fix this? > > > Op di 9 feb. 2016 om 10:43 schreef Simone Tiraboschi <stirabos(a)redhat.com > >: > >> On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <paul(a)pazion.nl&gt; >> wrote: >> >>> I have added 2 logs from 2 hosts >>> >>> >> Both the hosts are just additional host: >> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human >> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST >> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human >> dialog.__logString:219 DIALOG:SEND The specified storage >> location already contains a data domain. Is this an additional host setup >> (Yes, No)[Yes]? >> 2016-02-07 20:32:37 INFO >> otopi.plugins.ovirt_hosted_engine_setup.storage.storage >> storage._handleHostId:249 Installing on additional host >> >> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human >> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST >> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human >> dialog.__logString:219 DIALOG:SEND The specified storage >> location already contains a data domain. Is this an additional host setup >> (Yes, No)[Yes]? >> 2016-02-05 23:47:07 INFO >> otopi.plugins.ovirt_hosted_engine_setup.storage.storage >> storage._handleHostId:123 Installing on additional host >> >> Did you redeployed also the engine VM on a fresh storage domain or di >> you just run 'hosted-engine --deploy' again on all the hosts? >> If it's just an upgrade you don't have to run 'hosted-engine --deploy' >> again. >> >> Please follow this guide: >> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine >> >> >> >>> >>> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi < >>> stirabos(a)redhat.com&gt;: >>> >>>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion < >>>> paul(a)pazion.nl&gt; wrote: >>>> >>>>> >>>>> I found vdsClient and ran a few commands. >>>>> >>>>> [root@geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo >>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>> >>>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>> >>>>> version = 3 >>>>> >>>>> role = Master >>>>> >>>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine >>>>> >>>>> type = NFS >>>>> >>>>> class = Data >>>>> >>>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4'] >>>>> >>>>> name = hostedengine_nfs >>>>> >>>>> >>>>> >>>> Can you please attach hosted-engine-setup logs? at the end of the >>>> setup procedure the hosted-engine setup shouldn't b attached to any >>>> storagePool. >>>> >>>> >>>>> [root@geisha-1 ~]# vdsClient -s 0 connectStoragePool >>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 >>>>> >>>>> Error using command: list index out of range >>>>> >>>>> >>>>> ... >>>>> >>>>> >>>>> [root@geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo >>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 >>>>> >>>>> Unknown pool id, pool not connected: >>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',) >>>>> >>>>> >>>>> [root@geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList >>>>> >>>>> 00000002-0002-0002-0002-000000000385 >>>>> >>>>> >>>>> Can I manually create the missing storage pool which is missing with: >>>>> >>>>> vdsClient -s 0 createStoragePool >>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs >>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e >>>>> >>>>> If this would be the solution, I still wonder why there is no >>>>> existing storagePool for the storageDomain. >>>>> >>>>> >>>>> >>>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion < >>>>> paul(a)pazion.nl&gt;: >>>>> >>>>>> I still keep getting >>>>>> >>>>>> The Hosted Engine Storage Domain doesn't exist. It should be >>>>>> imported into the setup. >>>>>> >>>>>> I went digging a little further and found this in the engine.log >>>>>> >>>>>> 2016-02-08 22:30:25,083 INFO >>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] >>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >>>>>> HSMGetStorageDomainInfoVDSCommand, return: >>>>>> <StorageDomainStatic:{name='hostedengine_nfs', >>>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'}, >>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427 >>>>>> 2016-02-08 22:30:25,083 INFO >>>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] >>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH, >>>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1 >>>>>> 2016-02-08 22:30:25,084 INFO >>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object >>>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}' >>>>>> 2016-02-08 22:30:25,102 WARN >>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand] >>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action >>>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons: >>>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST >>>>>> >>>>>> So it looks like it can't find the storage domain. When I check >>>>>> vdsm.log I found an error: >>>>>> Thread-711902::ERROR::2016-02-08 >>>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status': >>>>>> {'message': "Unknown pool id, pool not connected: >>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}} >>>>>> >>>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is >>>>>> also trying to connect to in the engine.log. >>>>>> >>>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768 >>>>>> and https://bugzilla.redhat.com/show_bug.cgi?id=1294457 >>>>>> >>>>>> Can it be I have setup another DC with a master iscsi storage which >>>>>> is now offline? ( 1 DC with Cluster, VMs with hosted engine and master >>>>>> storage FC all up and running. ) >>>>>> >>>>>> Not sure where to go from here. Can I somehow restart this storage >>>>>> domain / do a new hosted-engine deploy with different settings? >>>>>> >>>>>> >>>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion < >>>>>> paul(a)pazion.nl&gt;: >>>>>> >>>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi < >>>>>>> stirabos(a)redhat.com&gt;: >>>>>>> >>>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion < >>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>> >>>>>>>>> Thanks for your response. >>>>>>>>> >>>>>>>>> I have reverted the commented entries and restarted >>>>>>>>> ovirt-ha-agent. >>>>>>>>> Now status on the daemon gives the error again: >>>>>>>>> http://screencast.com/t/DCZo6QJM1 >>>>>>>>> >>>>>>>>> >>>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment >>>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment? >>>>>>>> >>>>>>> >>>>>>> I have reinstalled host ( complete OS reinstall) and ran >>>>>>> hosted-engine --deploy for redeployment on an existing oVirt ( >>>>>>> hosted-engine ). >>>>>>> >>>>>>> As described in previous mail, during the deployment I got errors >>>>>>> about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/... >>>>>>> I created these by hand and then I ran deployment again and it >>>>>>> worked. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> The hosted engine domain is not shown in the web storage >>>>>>>>> overview... >>>>>>>>> When I start import it shows a warning: >>>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L >>>>>>>>> >>>>>>>>> Can I manually import the hostedengine storage? >>>>>>>>> ( Hosted engine is visible in web interface and I have running >>>>>>>>> VMs on another storage domain ). >>>>>>>>> >>>>>>>> >>>>>>>> It should auto-import it. >>>>>>>> Adding Roy here. >>>>>>>> >>>>>>>> >>>>>>> I believe this auto-import is failing... >>>>>>> Where can I find logs about this auto-import? >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi < >>>>>>>>> stirabos(a)redhat.com&gt;: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion < >>>>>>>>>> paul(a)pazion.nl&gt; wrote: >>>>>>>>>> >>>>>>>>>>> Update: >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The OVF_STORE volume will contain the engine VM description in a >>>>>>>>>> format that the engine can easily handle. >>>>>>>>>> It will be created by the engine when: >>>>>>>>>> - it imported the hosted-engine storage domain >>>>>>>>>> - the engineVM is visible >>>>>>>>>> - the datacenter is up (you need at least one additional regular >>>>>>>>>> storage domain) >>>>>>>>>> >>>>>>>>>> Please make sure that all these conditions are met. >>>>>>>>>> >>>>>>>>>> If the OVF_STORE is not there or it's not accessible, >>>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from deployment time. >>>>>>>>>> You see a warning since we are failing back to an initial status and this >>>>>>>>>> can temporary override your latest customization to the engine VM >>>>>>>>>> configuration. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf / >>>>>>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the >>>>>>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were >>>>>>>>>>> gone. ( I believe this is not a permanent solution ). >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Please revert it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> 2. This morning I noticed the warning in the event log. >>>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be >>>>>>>>>>> imported into the setup." >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> It's just a side effect of your point 1, please revert it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Any help would be appreciated. >>>>>>>>>>> >>>>>>>>>>> Best Regards, >>>>>>>>>>> >>>>>>>>>>> Paul Groeneweg >>>>>>>>>>> >>>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion < >>>>>>>>>>> paul(a)pazion.nl&gt;: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run >>>>>>>>>>>> version 3.6. >>>>>>>>>>>> When deploying a new host have problems with the shared OVF >>>>>>>>>>>> storage config part. >>>>>>>>>>>> >>>>>>>>>>>> 1. I get an error during deploy: >>>>>>>>>>>> >>>>>>>>>>>> [ INFO ] Stage: Setup validation >>>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed >>>>>>>>>>>> to read metadata: [Errno 2] No such file or directory: >>>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata' >>>>>>>>>>>> [ INFO ] Stage: Clean up >>>>>>>>>>>> >>>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing. >>>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with >>>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks. >>>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went >>>>>>>>>>>> well. >>>>>>>>>>>> >>>>>>>>>>>> 2. Now when the host is installed I notice the following in >>>>>>>>>>>> the logs >>>>>>>>>>>> >>>>>>>>>>>> .... >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing >>>>>>>>>>>> images >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading >>>>>>>>>>>> vm.conf from the shared storage domain >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying >>>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find >>>>>>>>>>>> OVF_STORE >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable >>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable >>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent >>>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume >>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - >>>>>>>>>>>> trying to restart agent >>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent: >>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume >>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' - >>>>>>>>>>>> trying to restart agent >>>>>>>>>>>> >>>>>>>>>>>> So at this point I was a bit stuck. >>>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for >>>>>>>>>>>> the hosted engine deploy. >>>>>>>>>>>> >>>>>>>>>>>> This time all went without any problem directly. >>>>>>>>>>>> From there I retried upgrade and same problems again. >>>>>>>>>>>> >>>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found >>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314 >>>>>>>>>>>> I added no_subtree_check and tried deploy again, but again >>>>>>>>>>>> without luck. >>>>>>>>>>>> >>>>>>>>>>>> I believe the hosted engine configs are not being migrated to >>>>>>>>>>>> the shared storage. And my problem has todo with >>>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage >>>>>>>>>>>> >>>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf, >>>>>>>>>>>> conf_volume_UUID, conf_image_UUID >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hosts run el7 >>>>>>>>>>>> Hosted engine runs el6 >>>>>>>>>>>> Hosted engine storage is through NFS3 >>>>>>>>>>>> >>>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web >>>>>>>>>>>> interface )? >>>>>>>>>>>> Or how can I create or trigger manually the OVF store the logs >>>>>>>>>>>> complain about? >>>>>>>>>>>> >>>>>>>>>>>> Best Regards, >>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>>> >>>>>>>>>>>> Paul Groeneweg >>>>>>>>>>>> Pazion >>>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>>> >>>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>>> E paul(a)pazion.nl >>>>>>>>>>>> >>>>>>>>>>>> ***disclaimer*** >>>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>>> information which is confidential and/or protected by intellectual property >>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above. >>>>>>>>>>>> Any use of the information contained herein (including, but not limited to, >>>>>>>>>>>> total or partial reproduction, communication or distribution in any form) >>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you >>>>>>>>>>>> have received this e-mail in error, please notify the sender either by >>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you >>>>>>>>>>>> for your cooperation." >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Met vriendelijke groeten, >>>>>>>>>>> >>>>>>>>>>> Paul Groeneweg >>>>>>>>>>> Pazion >>>>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>>>> >>>>>>>>>>> T +31 26 3020038 >>>>>>>>>>> M +31 614 277 577 >>>>>>>>>>> E paul(a)pazion.nl >>>>>>>>>>> >>>>>>>>>>> ***disclaimer*** >>>>>>>>>>> "This e-mail and any attachments thereto may contain >>>>>>>>>>> information which is confidential and/or protected by intellectual property >>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above. >>>>>>>>>>> Any use of the information contained herein (including, but not limited to, >>>>>>>>>>> total or partial reproduction, communication or distribution in any form) >>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you >>>>>>>>>>> have received this e-mail in error, please notify the sender either by >>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you >>>>>>>>>>> for your cooperation." >>>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Met vriendelijke groeten, >>>>>>>>> >>>>>>>>> Paul Groeneweg >>>>>>>>> Pazion >>>>>>>>> Webdevelopment - Hosting - Apps >>>>>>>>> >>>>>>>>> T +31 26 3020038 >>>>>>>>> M +31 614 277 577 >>>>>>>>> E paul(a)pazion.nl >>>>>>>>> >>>>>>>>> ***disclaimer*** >>>>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>>>> which is confidential and/or protected by intellectual property rights and >>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>>>>>> the information contained herein (including, but not limited to, total or >>>>>>>>> partial reproduction, communication or distribution in any form) by persons >>>>>>>>> other than the designated recipient(s) is prohibited. If you have received >>>>>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>>>> cooperation." >>>>>>>>> >>>>>>>> -- >>>>>>> Met vriendelijke groeten, >>>>>>> >>>>>>> Paul Groeneweg >>>>>>> Pazion >>>>>>> Webdevelopment - Hosting - Apps >>>>>>> >>>>>>> T +31 26 3020038 >>>>>>> M +31 614 277 577 >>>>>>> E paul(a)pazion.nl >>>>>>> >>>>>>> ***disclaimer*** >>>>>>> "This e-mail and any attachments thereto may contain information >>>>>>> which is confidential and/or protected by intellectual property rights and >>>>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>>>> the information contained herein (including, but not limited to, total or >>>>>>> partial reproduction, communication or distribution in any form) by persons >>>>>>> other than the designated recipient(s) is prohibited. If you have received >>>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>>> cooperation." >>>>>>> >>>>>> -- >>>>>> Met vriendelijke groeten, >>>>>> >>>>>> Paul Groeneweg >>>>>> Pazion >>>>>> Webdevelopment - Hosting - Apps >>>>>> >>>>>> T +31 26 3020038 >>>>>> M +31 614 277 577 >>>>>> E paul(a)pazion.nl >>>>>> >>>>>> ***disclaimer*** >>>>>> "This e-mail and any attachments thereto may contain information >>>>>> which is confidential and/or protected by intellectual property rights and >>>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>>> the information contained herein (including, but not limited to, total or >>>>>> partial reproduction, communication or distribution in any form) by persons >>>>>> other than the designated recipient(s) is prohibited. If you have received >>>>>> this e-mail in error, please notify the sender either by telephone or by >>>>>> e-mail and delete the material from any computer. Thank you for your >>>>>> cooperation." >>>>>> >>>>> -- >>>>> Met vriendelijke groeten, >>>>> >>>>> Paul Groeneweg >>>>> Pazion >>>>> Webdevelopment - Hosting - Apps >>>>> >>>>> T +31 26 3020038 >>>>> M +31 614 277 577 >>>>> E paul(a)pazion.nl >>>>> >>>>> ***disclaimer*** >>>>> "This e-mail and any attachments thereto may contain information >>>>> which is confidential and/or protected by intellectual property rights and >>>>> are intended for the sole use of the recipient(s) named above. Any use of >>>>> the information contained herein (including, but not limited to, total or >>>>> partial reproduction, communication or distribution in any form) by persons >>>>> other than the designated recipient(s) is prohibited. If you have received >>>>> this e-mail in error, please notify the sender either by telephone or by >>>>> e-mail and delete the material from any computer. Thank you for your >>>>> cooperation." >>>>> >>>> -- >>> Met vriendelijke groeten, >>> >>> Paul Groeneweg >>> Pazion >>> Webdevelopment - Hosting - Apps >>> >>> T +31 26 3020038 >>> M +31 614 277 577 >>> E paul(a)pazion.nl >>> >>> ***disclaimer*** >>> "This e-mail and any attachments thereto may contain information which >>> is confidential and/or protected by intellectual property rights and are >>> intended for the sole use of the recipient(s) named above. Any use of the >>> information contained herein (including, but not limited to, total or >>> partial reproduction, communication or distribution in any form) by persons >>> other than the designated recipient(s) is prohibited. If you have received >>> this e-mail in error, please notify the sender either by telephone or by >>> e-mail and delete the material from any computer. Thank you for your >>> cooperation." >>> >> -- > Met vriendelijke groeten, > > Paul Groeneweg > Pazion > Webdevelopment - Hosting - Apps > > T +31 26 3020038 > M +31 614 277 577 > E paul(a)pazion.nl > > ***disclaimer*** > "This e-mail and any attachments thereto may contain information which is > confidential and/or protected by intellectual property rights and are > intended for the sole use of the recipient(s) named above. Any use of the > information contained herein (including, but not limited to, total or > partial reproduction, communication or distribution in any form) by persons > other than the designated recipient(s) is prohibited. If you have received > this e-mail in error, please notify the sender either by telephone or by > e-mail and delete the material from any computer. Thank you for your > cooperation." > --