[ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE
Paul Groeneweg | Pazion
paul at pazion.nl
Tue Feb 9 10:51:38 UTC 2016
For the record, my hosted-engine still runs el6.
The hosts now run el7.
This means I need to save backup and restore to a new instance?
I do need to upgrade hosted-engine to el7?
I don't get the overall picture, but what I find strange is, the spUUID in
my hosted-engine.conf is not found. When I check the NFS hosted-engine
storageDomain it reports this same pool ID. Which in my opinion is the
reason for the failed ovf / vm.conf auto-import on shared storage is.
So to get this correct I need to backup and reinstall hosted-engine to get
the hosted engine working on my reinstalled hosts now runnign el7?
Op di 9 feb. 2016 om 11:45 schreef Simone Tiraboschi <stirabos at redhat.com>:
> On Tue, Feb 9, 2016 at 10:53 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
> wrote:
>
>> The original (first host) was removed ( about a year a go I removed this
>> one ) and it is also removed from hosted-engine --vm-status overview.
>>
>> I ran 'hosted-engine --deploy' again on all the hosts because I upgraded
>> all hosts to EL7.
>>
>
> Upgrading hosted-engine from 3.5 on el6 to 3.6 on el7 requires some
> additional steps:
> https://bugzilla.redhat.com/show_bug.cgi?id=1188679
>
> Adding Didi and Rafael who are working on that flow.
>
>
>>
>> ( I have als ran engine-setup for the upgrade )
>>
>> How can I fix this missing spUUID issue?
>> Can I manually add this somehow or run somsetup again to fix this?
>>
>>
>> Op di 9 feb. 2016 om 10:43 schreef Simone Tiraboschi <stirabos at redhat.com
>> >:
>>
>>> On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
>>> wrote:
>>>
>>>> I have added 2 logs from 2 hosts
>>>>
>>>>
>>> Both the hosts are just additional host:
>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>>> dialog.__logString:219 DIALOG:SEND The specified storage
>>> location already contains a data domain. Is this an additional host setup
>>> (Yes, No)[Yes]?
>>> 2016-02-07 20:32:37 INFO
>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>>> storage._handleHostId:249 Installing on additional host
>>>
>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>>> dialog.__logString:219 DIALOG:SEND The specified storage
>>> location already contains a data domain. Is this an additional host setup
>>> (Yes, No)[Yes]?
>>> 2016-02-05 23:47:07 INFO
>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>>> storage._handleHostId:123 Installing on additional host
>>>
>>> Did you redeployed also the engine VM on a fresh storage domain or di
>>> you just run 'hosted-engine --deploy' again on all the hosts?
>>> If it's just an upgrade you don't have to run 'hosted-engine --deploy'
>>> again.
>>>
>>> Please follow this guide:
>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
>>>
>>>
>>>
>>>>
>>>> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi <
>>>> stirabos at redhat.com>:
>>>>
>>>>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <
>>>>> paul at pazion.nl> wrote:
>>>>>
>>>>>>
>>>>>> I found vdsClient and ran a few commands.
>>>>>>
>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo
>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>
>>>>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>
>>>>>> version = 3
>>>>>>
>>>>>> role = Master
>>>>>>
>>>>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine
>>>>>>
>>>>>> type = NFS
>>>>>>
>>>>>> class = Data
>>>>>>
>>>>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']
>>>>>>
>>>>>> name = hostedengine_nfs
>>>>>>
>>>>>>
>>>>>>
>>>>> Can you please attach hosted-engine-setup logs? at the end of the
>>>>> setup procedure the hosted-engine setup shouldn't b attached to any
>>>>> storagePool.
>>>>>
>>>>>
>>>>>> [root at geisha-1 ~]# vdsClient -s 0 connectStoragePool
>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>>
>>>>>> Error using command: list index out of range
>>>>>>
>>>>>>
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo
>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>>
>>>>>> Unknown pool id, pool not connected:
>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)
>>>>>>
>>>>>>
>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList
>>>>>>
>>>>>> 00000002-0002-0002-0002-000000000385
>>>>>>
>>>>>>
>>>>>> Can I manually create the missing storage pool which is missing with:
>>>>>>
>>>>>> vdsClient -s 0 createStoragePool
>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs
>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>
>>>>>> If this would be the solution, I still wonder why there is no
>>>>>> existing storagePool for the storageDomain.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <
>>>>>> paul at pazion.nl>:
>>>>>>
>>>>>>> I still keep getting
>>>>>>>
>>>>>>> The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>> imported into the setup.
>>>>>>>
>>>>>>> I went digging a little further and found this in the engine.log
>>>>>>>
>>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand]
>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>>> HSMGetStorageDomainInfoVDSCommand, return:
>>>>>>> <StorageDomainStatic:{name='hostedengine_nfs',
>>>>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'},
>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427
>>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery]
>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1
>>>>>>> 2016-02-08 22:30:25,084 INFO
>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object
>>>>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
>>>>>>> 2016-02-08 22:30:25,102 WARN
>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action
>>>>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
>>>>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
>>>>>>>
>>>>>>> So it looks like it can't find the storage domain. When I check
>>>>>>> vdsm.log I found an error:
>>>>>>> Thread-711902::ERROR::2016-02-08
>>>>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
>>>>>>> {'message': "Unknown pool id, pool not connected:
>>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}
>>>>>>>
>>>>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is
>>>>>>> also trying to connect to in the engine.log.
>>>>>>>
>>>>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768
>>>>>>> and https://bugzilla.redhat.com/show_bug.cgi?id=1294457
>>>>>>>
>>>>>>> Can it be I have setup another DC with a master iscsi storage which
>>>>>>> is now offline? ( 1 DC with Cluster, VMs with hosted engine and master
>>>>>>> storage FC all up and running. )
>>>>>>>
>>>>>>> Not sure where to go from here. Can I somehow restart this storage
>>>>>>> domain / do a new hosted-engine deploy with different settings?
>>>>>>>
>>>>>>>
>>>>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <
>>>>>>> paul at pazion.nl>:
>>>>>>>
>>>>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <
>>>>>>>> stirabos at redhat.com>:
>>>>>>>>
>>>>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <
>>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for your response.
>>>>>>>>>>
>>>>>>>>>> I have reverted the commented entries and restarted
>>>>>>>>>> ovirt-ha-agent.
>>>>>>>>>> Now status on the daemon gives the error again:
>>>>>>>>>> http://screencast.com/t/DCZo6QJM1
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment
>>>>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I have reinstalled host ( complete OS reinstall) and ran
>>>>>>>> hosted-engine --deploy for redeployment on an existing oVirt (
>>>>>>>> hosted-engine ).
>>>>>>>>
>>>>>>>> As described in previous mail, during the deployment I got errors
>>>>>>>> about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/...
>>>>>>>> I created these by hand and then I ran deployment again and it
>>>>>>>> worked.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> The hosted engine domain is not shown in the web storage
>>>>>>>>>> overview...
>>>>>>>>>> When I start import it shows a warning:
>>>>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L
>>>>>>>>>>
>>>>>>>>>> Can I manually import the hostedengine storage?
>>>>>>>>>> ( Hosted engine is visible in web interface and I have running
>>>>>>>>>> VMs on another storage domain ).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> It should auto-import it.
>>>>>>>>> Adding Roy here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>> I believe this auto-import is failing...
>>>>>>>> Where can I find logs about this auto-import?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <
>>>>>>>>>> stirabos at redhat.com>:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <
>>>>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Update:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> The OVF_STORE volume will contain the engine VM description in a
>>>>>>>>>>> format that the engine can easily handle.
>>>>>>>>>>> It will be created by the engine when:
>>>>>>>>>>> - it imported the hosted-engine storage domain
>>>>>>>>>>> - the engineVM is visible
>>>>>>>>>>> - the datacenter is up (you need at least one additional regular
>>>>>>>>>>> storage domain)
>>>>>>>>>>>
>>>>>>>>>>> Please make sure that all these conditions are met.
>>>>>>>>>>>
>>>>>>>>>>> If the OVF_STORE is not there or it's not accessible,
>>>>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from deployment time.
>>>>>>>>>>> You see a warning since we are failing back to an initial status and this
>>>>>>>>>>> can temporary override your latest customization to the engine VM
>>>>>>>>>>> configuration.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf /
>>>>>>>>>>>> vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in the
>>>>>>>>>>>> hosted-engine.conf. After a restart of the ovirt-ha-agent the errors were
>>>>>>>>>>>> gone. ( I believe this is not a permanent solution ).
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please revert it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2. This morning I noticed the warning in the event log.
>>>>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>>>>>>> imported into the setup."
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It's just a side effect of your point 1, please revert it.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>
>>>>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <
>>>>>>>>>>>> paul at pazion.nl>:
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to run
>>>>>>>>>>>>> version 3.6.
>>>>>>>>>>>>> When deploying a new host have problems with the shared OVF
>>>>>>>>>>>>> storage config part.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. I get an error during deploy:
>>>>>>>>>>>>>
>>>>>>>>>>>>> [ INFO ] Stage: Setup validation
>>>>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed
>>>>>>>>>>>>> to read metadata: [Errno 2] No such file or directory:
>>>>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'
>>>>>>>>>>>>> [ INFO ] Stage: Clean up
>>>>>>>>>>>>>
>>>>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing.
>>>>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with
>>>>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.
>>>>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went
>>>>>>>>>>>>> well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. Now when the host is installed I notice the following in
>>>>>>>>>>>>> the logs
>>>>>>>>>>>>>
>>>>>>>>>>>>> ....
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing
>>>>>>>>>>>>> images
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading
>>>>>>>>>>>>> vm.conf from the shared storage domain
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying
>>>>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find
>>>>>>>>>>>>> OVF_STORE
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable
>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume
>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume
>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>>>
>>>>>>>>>>>>> So at this point I was a bit stuck.
>>>>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for
>>>>>>>>>>>>> the hosted engine deploy.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This time all went without any problem directly.
>>>>>>>>>>>>> From there I retried upgrade and same problems again.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found
>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314
>>>>>>>>>>>>> I added no_subtree_check and tried deploy again, but again
>>>>>>>>>>>>> without luck.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I believe the hosted engine configs are not being migrated to
>>>>>>>>>>>>> the shared storage. And my problem has todo with
>>>>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage
>>>>>>>>>>>>>
>>>>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf,
>>>>>>>>>>>>> conf_volume_UUID, conf_image_UUID
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hosts run el7
>>>>>>>>>>>>> Hosted engine runs el6
>>>>>>>>>>>>> Hosted engine storage is through NFS3
>>>>>>>>>>>>>
>>>>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web
>>>>>>>>>>>>> interface )?
>>>>>>>>>>>>> Or how can I create or trigger manually the OVF store the logs
>>>>>>>>>>>>> complain about?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>> Pazion
>>>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>>>
>>>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>>>
>>>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain
>>>>>>>>>>>>> information which is confidential and/or protected by intellectual property
>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above.
>>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to,
>>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form)
>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you
>>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by
>>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you
>>>>>>>>>>>>> for your cooperation."
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>
>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>> Pazion
>>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>>
>>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>>
>>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>>> "This e-mail and any attachments thereto may contain
>>>>>>>>>>>> information which is confidential and/or protected by intellectual property
>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above.
>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to,
>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form)
>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you
>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by
>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you
>>>>>>>>>>>> for your cooperation."
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>
>>>>>>>>>> Paul Groeneweg
>>>>>>>>>> Pazion
>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>
>>>>>>>>>> T +31 26 3020038
>>>>>>>>>> M +31 614 277 577
>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>
>>>>>>>>>> ***disclaimer***
>>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>>> cooperation."
>>>>>>>>>>
>>>>>>>>> --
>>>>>>>> Met vriendelijke groeten,
>>>>>>>>
>>>>>>>> Paul Groeneweg
>>>>>>>> Pazion
>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>
>>>>>>>> T +31 26 3020038
>>>>>>>> M +31 614 277 577
>>>>>>>> E paul at pazion.nl
>>>>>>>>
>>>>>>>> ***disclaimer***
>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>> cooperation."
>>>>>>>>
>>>>>>> --
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>> Pazion
>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>
>>>>>>> T +31 26 3020038
>>>>>>> M +31 614 277 577
>>>>>>> E paul at pazion.nl
>>>>>>>
>>>>>>> ***disclaimer***
>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>> cooperation."
>>>>>>>
>>>>>> --
>>>>>> Met vriendelijke groeten,
>>>>>>
>>>>>> Paul Groeneweg
>>>>>> Pazion
>>>>>> Webdevelopment - Hosting - Apps
>>>>>>
>>>>>> T +31 26 3020038
>>>>>> M +31 614 277 577
>>>>>> E paul at pazion.nl
>>>>>>
>>>>>> ***disclaimer***
>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>> the information contained herein (including, but not limited to, total or
>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>> cooperation."
>>>>>>
>>>>> --
>>>> Met vriendelijke groeten,
>>>>
>>>> Paul Groeneweg
>>>> Pazion
>>>> Webdevelopment - Hosting - Apps
>>>>
>>>> T +31 26 3020038
>>>> M +31 614 277 577
>>>> E paul at pazion.nl
>>>>
>>>> ***disclaimer***
>>>> "This e-mail and any attachments thereto may contain information which
>>>> is confidential and/or protected by intellectual property rights and are
>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>> information contained herein (including, but not limited to, total or
>>>> partial reproduction, communication or distribution in any form) by persons
>>>> other than the designated recipient(s) is prohibited. If you have received
>>>> this e-mail in error, please notify the sender either by telephone or by
>>>> e-mail and delete the material from any computer. Thank you for your
>>>> cooperation."
>>>>
>>> --
>> Met vriendelijke groeten,
>>
>> Paul Groeneweg
>> Pazion
>> Webdevelopment - Hosting - Apps
>>
>> T +31 26 3020038
>> M +31 614 277 577
>> E paul at pazion.nl
>>
>> ***disclaimer***
>> "This e-mail and any attachments thereto may contain information which is
>> confidential and/or protected by intellectual property rights and are
>> intended for the sole use of the recipient(s) named above. Any use of the
>> information contained herein (including, but not limited to, total or
>> partial reproduction, communication or distribution in any form) by persons
>> other than the designated recipient(s) is prohibited. If you have received
>> this e-mail in error, please notify the sender either by telephone or by
>> e-mail and delete the material from any computer. Thank you for your
>> cooperation."
>>
> --
Met vriendelijke groeten,
Paul Groeneweg
Pazion
Webdevelopment - Hosting - Apps
T +31 26 3020038
M +31 614 277 577
E paul at pazion.nl
***disclaimer***
"This e-mail and any attachments thereto may contain information which is
confidential and/or protected by intellectual property rights and are
intended for the sole use of the recipient(s) named above. Any use of the
information contained herein (including, but not limited to, total or
partial reproduction, communication or distribution in any form) by persons
other than the designated recipient(s) is prohibited. If you have received
this e-mail in error, please notify the sender either by telephone or by
e-mail and delete the material from any computer. Thank you for your
cooperation."
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160209/9aeff20f/attachment-0001.html>
More information about the Users
mailing list