[ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE
Simone Tiraboschi
stirabos at redhat.com
Tue Feb 9 10:45:13 UTC 2016
On Tue, Feb 9, 2016 at 10:53 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
wrote:
> The original (first host) was removed ( about a year a go I removed this
> one ) and it is also removed from hosted-engine --vm-status overview.
>
> I ran 'hosted-engine --deploy' again on all the hosts because I upgraded
> all hosts to EL7.
>
Upgrading hosted-engine from 3.5 on el6 to 3.6 on el7 requires some
additional steps:
https://bugzilla.redhat.com/show_bug.cgi?id=1188679
Adding Didi and Rafael who are working on that flow.
>
> ( I have als ran engine-setup for the upgrade )
>
> How can I fix this missing spUUID issue?
> Can I manually add this somehow or run somsetup again to fix this?
>
>
> Op di 9 feb. 2016 om 10:43 schreef Simone Tiraboschi <stirabos at redhat.com
> >:
>
>> On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
>> wrote:
>>
>>> I have added 2 logs from 2 hosts
>>>
>>>
>> Both the hosts are just additional host:
>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>> dialog.__logString:219 DIALOG:SEND The specified storage
>> location already contains a data domain. Is this an additional host setup
>> (Yes, No)[Yes]?
>> 2016-02-07 20:32:37 INFO
>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>> storage._handleHostId:249 Installing on additional host
>>
>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>> dialog.__logString:219 DIALOG:SEND The specified storage
>> location already contains a data domain. Is this an additional host setup
>> (Yes, No)[Yes]?
>> 2016-02-05 23:47:07 INFO
>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>> storage._handleHostId:123 Installing on additional host
>>
>> Did you redeployed also the engine VM on a fresh storage domain or di you
>> just run 'hosted-engine --deploy' again on all the hosts?
>> If it's just an upgrade you don't have to run 'hosted-engine --deploy'
>> again.
>>
>> Please follow this guide:
>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
>>
>>
>>
>>>
>>> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi <
>>> stirabos at redhat.com>:
>>>
>>>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <
>>>> paul at pazion.nl> wrote:
>>>>
>>>>>
>>>>> I found vdsClient and ran a few commands.
>>>>>
>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo
>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>
>>>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>
>>>>> version = 3
>>>>>
>>>>> role = Master
>>>>>
>>>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine
>>>>>
>>>>> type = NFS
>>>>>
>>>>> class = Data
>>>>>
>>>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']
>>>>>
>>>>> name = hostedengine_nfs
>>>>>
>>>>>
>>>>>
>>>> Can you please attach hosted-engine-setup logs? at the end of the setup
>>>> procedure the hosted-engine setup shouldn't b attached to any storagePool.
>>>>
>>>>
>>>>> [root at geisha-1 ~]# vdsClient -s 0 connectStoragePool
>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>
>>>>> Error using command: list index out of range
>>>>>
>>>>>
>>>>> ...
>>>>>
>>>>>
>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo
>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>
>>>>> Unknown pool id, pool not connected:
>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)
>>>>>
>>>>>
>>>>> [root at geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList
>>>>>
>>>>> 00000002-0002-0002-0002-000000000385
>>>>>
>>>>>
>>>>> Can I manually create the missing storage pool which is missing with:
>>>>>
>>>>> vdsClient -s 0 createStoragePool
>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs
>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>
>>>>> If this would be the solution, I still wonder why there is no existing
>>>>> storagePool for the storageDomain.
>>>>>
>>>>>
>>>>>
>>>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <
>>>>> paul at pazion.nl>:
>>>>>
>>>>>> I still keep getting
>>>>>>
>>>>>> The Hosted Engine Storage Domain doesn't exist. It should be imported
>>>>>> into the setup.
>>>>>>
>>>>>> I went digging a little further and found this in the engine.log
>>>>>>
>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand]
>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>> HSMGetStorageDomainInfoVDSCommand, return:
>>>>>> <StorageDomainStatic:{name='hostedengine_nfs',
>>>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'},
>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427
>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery]
>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1
>>>>>> 2016-02-08 22:30:25,084 INFO
>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object
>>>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
>>>>>> 2016-02-08 22:30:25,102 WARN
>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action
>>>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
>>>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
>>>>>>
>>>>>> So it looks like it can't find the storage domain. When I check
>>>>>> vdsm.log I found an error:
>>>>>> Thread-711902::ERROR::2016-02-08
>>>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
>>>>>> {'message': "Unknown pool id, pool not connected:
>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}
>>>>>>
>>>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is
>>>>>> also trying to connect to in the engine.log.
>>>>>>
>>>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768 and
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1294457
>>>>>>
>>>>>> Can it be I have setup another DC with a master iscsi storage which
>>>>>> is now offline? ( 1 DC with Cluster, VMs with hosted engine and master
>>>>>> storage FC all up and running. )
>>>>>>
>>>>>> Not sure where to go from here. Can I somehow restart this storage
>>>>>> domain / do a new hosted-engine deploy with different settings?
>>>>>>
>>>>>>
>>>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <
>>>>>> paul at pazion.nl>:
>>>>>>
>>>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <
>>>>>>> stirabos at redhat.com>:
>>>>>>>
>>>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <
>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>
>>>>>>>>> Thanks for your response.
>>>>>>>>>
>>>>>>>>> I have reverted the commented entries and restarted ovirt-ha-agent.
>>>>>>>>> Now status on the daemon gives the error again:
>>>>>>>>> http://screencast.com/t/DCZo6QJM1
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment
>>>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?
>>>>>>>>
>>>>>>>
>>>>>>> I have reinstalled host ( complete OS reinstall) and ran
>>>>>>> hosted-engine --deploy for redeployment on an existing oVirt (
>>>>>>> hosted-engine ).
>>>>>>>
>>>>>>> As described in previous mail, during the deployment I got errors
>>>>>>> about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/...
>>>>>>> I created these by hand and then I ran deployment again and it
>>>>>>> worked.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>> The hosted engine domain is not shown in the web storage
>>>>>>>>> overview...
>>>>>>>>> When I start import it shows a warning:
>>>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L
>>>>>>>>>
>>>>>>>>> Can I manually import the hostedengine storage?
>>>>>>>>> ( Hosted engine is visible in web interface and I have running VMs
>>>>>>>>> on another storage domain ).
>>>>>>>>>
>>>>>>>>
>>>>>>>> It should auto-import it.
>>>>>>>> Adding Roy here.
>>>>>>>>
>>>>>>>>
>>>>>>> I believe this auto-import is failing...
>>>>>>> Where can I find logs about this auto-import?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <
>>>>>>>>> stirabos at redhat.com>:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <
>>>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Update:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> The OVF_STORE volume will contain the engine VM description in a
>>>>>>>>>> format that the engine can easily handle.
>>>>>>>>>> It will be created by the engine when:
>>>>>>>>>> - it imported the hosted-engine storage domain
>>>>>>>>>> - the engineVM is visible
>>>>>>>>>> - the datacenter is up (you need at least one additional regular
>>>>>>>>>> storage domain)
>>>>>>>>>>
>>>>>>>>>> Please make sure that all these conditions are met.
>>>>>>>>>>
>>>>>>>>>> If the OVF_STORE is not there or it's not accessible,
>>>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from deployment time.
>>>>>>>>>> You see a warning since we are failing back to an initial status and this
>>>>>>>>>> can temporary override your latest customization to the engine VM
>>>>>>>>>> configuration.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf /
>>>>>>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the
>>>>>>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were
>>>>>>>>>>> gone. ( I believe this is not a permanent solution ).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please revert it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2. This morning I noticed the warning in the event log.
>>>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>>>>>> imported into the setup."
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It's just a side effect of your point 1, please revert it.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>>
>>>>>>>>>>> Best Regards,
>>>>>>>>>>>
>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>
>>>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <
>>>>>>>>>>> paul at pazion.nl>:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run
>>>>>>>>>>>> version 3.6.
>>>>>>>>>>>> When deploying a new host have problems with the shared OVF
>>>>>>>>>>>> storage config part.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. I get an error during deploy:
>>>>>>>>>>>>
>>>>>>>>>>>> [ INFO ] Stage: Setup validation
>>>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed to
>>>>>>>>>>>> read metadata: [Errno 2] No such file or directory:
>>>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'
>>>>>>>>>>>> [ INFO ] Stage: Clean up
>>>>>>>>>>>>
>>>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing.
>>>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with
>>>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.
>>>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went
>>>>>>>>>>>> well.
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Now when the host is installed I notice the following in the
>>>>>>>>>>>> logs
>>>>>>>>>>>>
>>>>>>>>>>>> ....
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing
>>>>>>>>>>>> images
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading
>>>>>>>>>>>> vm.conf from the shared storage domain
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying
>>>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find
>>>>>>>>>>>> OVF_STORE
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable
>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume
>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume
>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>>
>>>>>>>>>>>> So at this point I was a bit stuck.
>>>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for
>>>>>>>>>>>> the hosted engine deploy.
>>>>>>>>>>>>
>>>>>>>>>>>> This time all went without any problem directly.
>>>>>>>>>>>> From there I retried upgrade and same problems again.
>>>>>>>>>>>>
>>>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found
>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314
>>>>>>>>>>>> I added no_subtree_check and tried deploy again, but again
>>>>>>>>>>>> without luck.
>>>>>>>>>>>>
>>>>>>>>>>>> I believe the hosted engine configs are not being migrated to
>>>>>>>>>>>> the shared storage. And my problem has todo with
>>>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage
>>>>>>>>>>>>
>>>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf,
>>>>>>>>>>>> conf_volume_UUID, conf_image_UUID
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Hosts run el7
>>>>>>>>>>>> Hosted engine runs el6
>>>>>>>>>>>> Hosted engine storage is through NFS3
>>>>>>>>>>>>
>>>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web
>>>>>>>>>>>> interface )?
>>>>>>>>>>>> Or how can I create or trigger manually the OVF store the logs
>>>>>>>>>>>> complain about?
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>
>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>> Pazion
>>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>>
>>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>>
>>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>>> "This e-mail and any attachments thereto may contain
>>>>>>>>>>>> information which is confidential and/or protected by intellectual property
>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above.
>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to,
>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form)
>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you
>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by
>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you
>>>>>>>>>>>> for your cooperation."
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>
>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>> Pazion
>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>
>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>
>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>>>> cooperation."
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>
>>>>>>>>> Paul Groeneweg
>>>>>>>>> Pazion
>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>
>>>>>>>>> T +31 26 3020038
>>>>>>>>> M +31 614 277 577
>>>>>>>>> E paul at pazion.nl
>>>>>>>>>
>>>>>>>>> ***disclaimer***
>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>> cooperation."
>>>>>>>>>
>>>>>>>> --
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>> Pazion
>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>
>>>>>>> T +31 26 3020038
>>>>>>> M +31 614 277 577
>>>>>>> E paul at pazion.nl
>>>>>>>
>>>>>>> ***disclaimer***
>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>> cooperation."
>>>>>>>
>>>>>> --
>>>>>> Met vriendelijke groeten,
>>>>>>
>>>>>> Paul Groeneweg
>>>>>> Pazion
>>>>>> Webdevelopment - Hosting - Apps
>>>>>>
>>>>>> T +31 26 3020038
>>>>>> M +31 614 277 577
>>>>>> E paul at pazion.nl
>>>>>>
>>>>>> ***disclaimer***
>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>> the information contained herein (including, but not limited to, total or
>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>> cooperation."
>>>>>>
>>>>> --
>>>>> Met vriendelijke groeten,
>>>>>
>>>>> Paul Groeneweg
>>>>> Pazion
>>>>> Webdevelopment - Hosting - Apps
>>>>>
>>>>> T +31 26 3020038
>>>>> M +31 614 277 577
>>>>> E paul at pazion.nl
>>>>>
>>>>> ***disclaimer***
>>>>> "This e-mail and any attachments thereto may contain information which
>>>>> is confidential and/or protected by intellectual property rights and are
>>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>>> information contained herein (including, but not limited to, total or
>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>> cooperation."
>>>>>
>>>> --
>>> Met vriendelijke groeten,
>>>
>>> Paul Groeneweg
>>> Pazion
>>> Webdevelopment - Hosting - Apps
>>>
>>> T +31 26 3020038
>>> M +31 614 277 577
>>> E paul at pazion.nl
>>>
>>> ***disclaimer***
>>> "This e-mail and any attachments thereto may contain information which
>>> is confidential and/or protected by intellectual property rights and are
>>> intended for the sole use of the recipient(s) named above. Any use of the
>>> information contained herein (including, but not limited to, total or
>>> partial reproduction, communication or distribution in any form) by persons
>>> other than the designated recipient(s) is prohibited. If you have received
>>> this e-mail in error, please notify the sender either by telephone or by
>>> e-mail and delete the material from any computer. Thank you for your
>>> cooperation."
>>>
>> --
> Met vriendelijke groeten,
>
> Paul Groeneweg
> Pazion
> Webdevelopment - Hosting - Apps
>
> T +31 26 3020038
> M +31 614 277 577
> E paul at pazion.nl
>
> ***disclaimer***
> "This e-mail and any attachments thereto may contain information which is
> confidential and/or protected by intellectual property rights and are
> intended for the sole use of the recipient(s) named above. Any use of the
> information contained herein (including, but not limited to, total or
> partial reproduction, communication or distribution in any form) by persons
> other than the designated recipient(s) is prohibited. If you have received
> this e-mail in error, please notify the sender either by telephone or by
> e-mail and delete the material from any computer. Thank you for your
> cooperation."
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160209/1bf93ed5/attachment-0001.html>
More information about the Users
mailing list