[ovirt-users] ovirt 3.6 hostedengine.metadata, missing OVF_STORE
Simone Tiraboschi
stirabos at redhat.com
Tue Feb 9 11:05:04 UTC 2016
On Tue, Feb 9, 2016 at 11:51 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
wrote:
> For the record, my hosted-engine still runs el6.
>
You can run the 3.6 engine also on el6 so no issue there.
> The hosts now run el7.
>
To reach 3.6 cluster compatibility level you need to upgrade all the hosts
to el7.
Now I don't understand where the engine VM is running.
In order to be able to (re)deploy the host you upgraded to el7 you need a
running engine.
Is your engine VM running?
If so, on which host?
> This means I need to save backup and restore to a new instance?
> I do need to upgrade hosted-engine to el7?
>
> I don't get the overall picture, but what I find strange is, the spUUID in
> my hosted-engine.conf is not found. When I check the NFS hosted-engine
> storageDomain it reports this same pool ID. Which in my opinion is the
> reason for the failed ovf / vm.conf auto-import on shared storage is.
>
This is a bit tricky:
when you setup hosted-engine on 3.4 or 3.5, the hosted-engine storage
domain was on a dedicated (boostrap) storage pool that nobody was really
using.
In order to be able to manage and edit the engine VM from the engine, on
3.6 the hosted-engine storage domain should be attached to the storagePool
of the datacenter as seen by the engine but to do that the hosted-engine
storage domain should be detached from its bootstrap storage pool.
The flow is this:
1. put everything into maintenance
2. upgrade ovirt-ha-agent to 3.6
3. restart ovirt-ha-agent: the new agent will
- create a new volume with all the hosted-engine configuration on the
shared volume
- detach hosted-engine storage domain from its boostrap storage pool
4. when all the hosts are at 3.6, you can set 3.6 cluster compatibility
level into the engine; this will trigger the autoimport of the
hosted-engine storage domain from the engine and the hosted-engine storage
domain will be attached to the datacenter storage pool.
In your case is a bit different since your hosts was with el6 and so you
cannot upgrade ovirt-ha-agent to 3.6.
>
> So to get this correct I need to backup and reinstall hosted-engine to get
> the hosted engine working on my reinstalled hosts now runnign el7?
>
> Op di 9 feb. 2016 om 11:45 schreef Simone Tiraboschi <stirabos at redhat.com
> >:
>
>> On Tue, Feb 9, 2016 at 10:53 AM, Paul Groeneweg | Pazion <paul at pazion.nl>
>> wrote:
>>
>>> The original (first host) was removed ( about a year a go I removed this
>>> one ) and it is also removed from hosted-engine --vm-status overview.
>>>
>>> I ran 'hosted-engine --deploy' again on all the hosts because I
>>> upgraded all hosts to EL7.
>>>
>>
>> Upgrading hosted-engine from 3.5 on el6 to 3.6 on el7 requires some
>> additional steps:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1188679
>>
>> Adding Didi and Rafael who are working on that flow.
>>
>>
>>>
>>> ( I have als ran engine-setup for the upgrade )
>>>
>>> How can I fix this missing spUUID issue?
>>> Can I manually add this somehow or run somsetup again to fix this?
>>>
>>>
>>> Op di 9 feb. 2016 om 10:43 schreef Simone Tiraboschi <
>>> stirabos at redhat.com>:
>>>
>>>> On Tue, Feb 9, 2016 at 9:45 AM, Paul Groeneweg | Pazion <paul at pazion.nl
>>>> > wrote:
>>>>
>>>>> I have added 2 logs from 2 hosts
>>>>>
>>>>>
>>>> Both the hosts are just additional host:
>>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>>>> 2016-02-07 20:32:30 DEBUG otopi.plugins.otopi.dialog.human
>>>> dialog.__logString:219 DIALOG:SEND The specified storage
>>>> location already contains a data domain. Is this an additional host setup
>>>> (Yes, No)[Yes]?
>>>> 2016-02-07 20:32:37 INFO
>>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>>>> storage._handleHostId:249 Installing on additional host
>>>>
>>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>>>> human.queryString:156 query OVEHOSTED_ADDITIONAL_HOST
>>>> 2016-02-05 23:47:05 DEBUG otopi.plugins.otopi.dialog.human
>>>> dialog.__logString:219 DIALOG:SEND The specified storage
>>>> location already contains a data domain. Is this an additional host setup
>>>> (Yes, No)[Yes]?
>>>> 2016-02-05 23:47:07 INFO
>>>> otopi.plugins.ovirt_hosted_engine_setup.storage.storage
>>>> storage._handleHostId:123 Installing on additional host
>>>>
>>>> Did you redeployed also the engine VM on a fresh storage domain or di
>>>> you just run 'hosted-engine --deploy' again on all the hosts?
>>>> If it's just an upgrade you don't have to run 'hosted-engine --deploy'
>>>> again.
>>>>
>>>> Please follow this guide:
>>>> http://www.ovirt.org/Hosted_Engine_Howto#Upgrade_Hosted_Engine
>>>>
>>>>
>>>>
>>>>>
>>>>> Op di 9 feb. 2016 om 09:07 schreef Simone Tiraboschi <
>>>>> stirabos at redhat.com>:
>>>>>
>>>>>> On Mon, Feb 8, 2016 at 11:25 PM, Paul Groeneweg | Pazion <
>>>>>> paul at pazion.nl> wrote:
>>>>>>
>>>>>>>
>>>>>>> I found vdsClient and ran a few commands.
>>>>>>>
>>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStorageDomainInfo
>>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>>
>>>>>>> uuid = 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>>
>>>>>>> version = 3
>>>>>>>
>>>>>>> role = Master
>>>>>>>
>>>>>>> remotePath = hostedstorage.pazion.nl:/opt/hosted-engine
>>>>>>>
>>>>>>> type = NFS
>>>>>>>
>>>>>>> class = Data
>>>>>>>
>>>>>>> pool = ['499b208c-9de9-4a2a-97de-30f410b4e6d4']
>>>>>>>
>>>>>>> name = hostedengine_nfs
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Can you please attach hosted-engine-setup logs? at the end of the
>>>>>> setup procedure the hosted-engine setup shouldn't b attached to any
>>>>>> storagePool.
>>>>>>
>>>>>>
>>>>>>> [root at geisha-1 ~]# vdsClient -s 0 connectStoragePool
>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>>>
>>>>>>> Error using command: list index out of range
>>>>>>>
>>>>>>>
>>>>>>> ...
>>>>>>>
>>>>>>>
>>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getStoragePoolInfo
>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4
>>>>>>>
>>>>>>> Unknown pool id, pool not connected:
>>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)
>>>>>>>
>>>>>>>
>>>>>>> [root at geisha-1 ~]# vdsClient -s 0 getConnectedStoragePoolsList
>>>>>>>
>>>>>>> 00000002-0002-0002-0002-000000000385
>>>>>>>
>>>>>>>
>>>>>>> Can I manually create the missing storage pool which is missing with:
>>>>>>>
>>>>>>> vdsClient -s 0 createStoragePool
>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4 hostedengine_nfs
>>>>>>> 88b69eba-ef4f-4dbe-ba53-20dadd424d0e
>>>>>>>
>>>>>>> If this would be the solution, I still wonder why there is no
>>>>>>> existing storagePool for the storageDomain.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Op ma 8 feb. 2016 om 22:54 schreef Paul Groeneweg | Pazion <
>>>>>>> paul at pazion.nl>:
>>>>>>>
>>>>>>>> I still keep getting
>>>>>>>>
>>>>>>>> The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>>> imported into the setup.
>>>>>>>>
>>>>>>>> I went digging a little further and found this in the engine.log
>>>>>>>>
>>>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>>>> [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand]
>>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>>>> HSMGetStorageDomainInfoVDSCommand, return:
>>>>>>>> <StorageDomainStatic:{name='hostedengine_nfs',
>>>>>>>> id='88b69eba-ef4f-4dbe-ba53-20dadd424d0e'},
>>>>>>>> 499b208c-9de9-4a2a-97de-30f410b4e6d4>, log id: 56723427
>>>>>>>> 2016-02-08 22:30:25,083 INFO
>>>>>>>> [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery]
>>>>>>>> (org.ovirt.thread.pool-8-thread-41) [] FINISH,
>>>>>>>> GetExistingStorageDomainListQuery, log id: 6ca437b1
>>>>>>>> 2016-02-08 22:30:25,084 INFO
>>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] Lock Acquired to object
>>>>>>>> 'EngineLock:{exclusiveLocks='[]', sharedLocks='null'}'
>>>>>>>> 2016-02-08 22:30:25,102 WARN
>>>>>>>> [org.ovirt.engine.core.bll.ImportHostedEngineStorageDomainCommand]
>>>>>>>> (org.ovirt.thread.pool-8-thread-41) [364719ce] CanDoAction of action
>>>>>>>> 'ImportHostedEngineStorageDomain' failed for user SYSTEM. Reasons:
>>>>>>>> VAR__ACTION__ADD,VAR__TYPE__STORAGE__DOMAIN,ACTION_TYPE_FAILED_STORAGE_DOMAIN_NOT_EXIST
>>>>>>>>
>>>>>>>> So it looks like it can't find the storage domain. When I check
>>>>>>>> vdsm.log I found an error:
>>>>>>>> Thread-711902::ERROR::2016-02-08
>>>>>>>> 21:17:01,240::dispatcher::76::Storage.Dispatcher::(wrapper) {'status':
>>>>>>>> {'message': "Unknown pool id, pool not connected:
>>>>>>>> ('499b208c-9de9-4a2a-97de-30f410b4e6d4',)", 'code': 309}}
>>>>>>>>
>>>>>>>> It looks like 499b208c-9de9-4a2a-97de-30f410b4e6d4 is the id it is
>>>>>>>> also trying to connect to in the engine.log.
>>>>>>>>
>>>>>>>> I did found https://bugzilla.redhat.com/show_bug.cgi?id=1269768
>>>>>>>> and https://bugzilla.redhat.com/show_bug.cgi?id=1294457
>>>>>>>>
>>>>>>>> Can it be I have setup another DC with a master iscsi storage
>>>>>>>> which is now offline? ( 1 DC with Cluster, VMs with hosted engine and
>>>>>>>> master storage FC all up and running. )
>>>>>>>>
>>>>>>>> Not sure where to go from here. Can I somehow restart this storage
>>>>>>>> domain / do a new hosted-engine deploy with different settings?
>>>>>>>>
>>>>>>>>
>>>>>>>> Op ma 8 feb. 2016 om 12:05 schreef Paul Groeneweg | Pazion <
>>>>>>>> paul at pazion.nl>:
>>>>>>>>
>>>>>>>>> Op ma 8 feb. 2016 om 11:28 schreef Simone Tiraboschi <
>>>>>>>>> stirabos at redhat.com>:
>>>>>>>>>
>>>>>>>>>> On Mon, Feb 8, 2016 at 10:55 AM, Paul Groeneweg | Pazion <
>>>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks for your response.
>>>>>>>>>>>
>>>>>>>>>>> I have reverted the commented entries and restarted
>>>>>>>>>>> ovirt-ha-agent.
>>>>>>>>>>> Now status on the daemon gives the error again:
>>>>>>>>>>> http://screencast.com/t/DCZo6QJM1
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Here I'm a bit confused: is this a fresh hosted-engine deployment
>>>>>>>>>> with 3.6 or an upgrade from an existing 3.5 hosted-engine deployment?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have reinstalled host ( complete OS reinstall) and ran
>>>>>>>>> hosted-engine --deploy for redeployment on an existing oVirt (
>>>>>>>>> hosted-engine ).
>>>>>>>>>
>>>>>>>>> As described in previous mail, during the deployment I got errors
>>>>>>>>> about a missing hostedengine.metadata symlink to /var/run/vdsd/storage/...
>>>>>>>>> I created these by hand and then I ran deployment again and it
>>>>>>>>> worked.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> The hosted engine domain is not shown in the web storage
>>>>>>>>>>> overview...
>>>>>>>>>>> When I start import it shows a warning:
>>>>>>>>>>> http://screencast.com/t/7HfJ9qWFwb2L
>>>>>>>>>>>
>>>>>>>>>>> Can I manually import the hostedengine storage?
>>>>>>>>>>> ( Hosted engine is visible in web interface and I have running
>>>>>>>>>>> VMs on another storage domain ).
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> It should auto-import it.
>>>>>>>>>> Adding Roy here.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> I believe this auto-import is failing...
>>>>>>>>> Where can I find logs about this auto-import?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Op ma 8 feb. 2016 om 10:36 schreef Simone Tiraboschi <
>>>>>>>>>>> stirabos at redhat.com>:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Feb 8, 2016 at 9:47 AM, Paul Groeneweg | Pazion <
>>>>>>>>>>>> paul at pazion.nl> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Update:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> The OVF_STORE volume will contain the engine VM description in
>>>>>>>>>>>> a format that the engine can easily handle.
>>>>>>>>>>>> It will be created by the engine when:
>>>>>>>>>>>> - it imported the hosted-engine storage domain
>>>>>>>>>>>> - the engineVM is visible
>>>>>>>>>>>> - the datacenter is up (you need at least one additional
>>>>>>>>>>>> regular storage domain)
>>>>>>>>>>>>
>>>>>>>>>>>> Please make sure that all these conditions are met.
>>>>>>>>>>>>
>>>>>>>>>>>> If the OVF_STORE is not there or it's not accessible,
>>>>>>>>>>>> ovirt-ha-agent will fallback to the initial vm.conf from deployment time.
>>>>>>>>>>>> You see a warning since we are failing back to an initial status and this
>>>>>>>>>>>> can temporary override your latest customization to the engine VM
>>>>>>>>>>>> configuration.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 1. I was able to remove the warnirngs about the not found ovf
>>>>>>>>>>>>> / vm.conf by commenting the conf_volume_UUID, conf_image_UUID entries in
>>>>>>>>>>>>> the hosted-engine.conf. After a restart of the ovirt-ha-agent the errors
>>>>>>>>>>>>> were gone. ( I believe this is not a permanent solution ).
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Please revert it.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. This morning I noticed the warning in the event log.
>>>>>>>>>>>>> "The Hosted Engine Storage Domain doesn't exist. It should be
>>>>>>>>>>>>> imported into the setup."
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> It's just a side effect of your point 1, please revert it.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Any help would be appreciated.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>>
>>>>>>>>>>>>> Op zo 7 feb. 2016 om 22:14 schreef Paul Groeneweg | Pazion <
>>>>>>>>>>>>> paul at pazion.nl>:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have been upgrading our ovirt hosts to el7 to be able to
>>>>>>>>>>>>>> run version 3.6.
>>>>>>>>>>>>>> When deploying a new host have problems with the shared OVF
>>>>>>>>>>>>>> storage config part.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. I get an error during deploy:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [ INFO ] Stage: Setup validation
>>>>>>>>>>>>>> [ ERROR ] Failed to execute stage 'Setup validation': failed
>>>>>>>>>>>>>> to read metadata: [Errno 2] No such file or directory:
>>>>>>>>>>>>>> '/rhev/data-center/mnt/[NFS3_SHARE]/88b69eba-ef4f-4dbe-ba53-20dadd424d0e/ha_agent/hosted-engine.metadata'
>>>>>>>>>>>>>> [ INFO ] Stage: Clean up
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It seems the symlink of hosted-engine.metadata is missing.
>>>>>>>>>>>>>> So I added manually the /var/run/vdsm/storage folder with
>>>>>>>>>>>>>> subfolder 88b69eba-ef4f-4dbe-ba53-20dadd424d0e and symlinks.
>>>>>>>>>>>>>> Then I reran the hosted-engine deploy and this time all went
>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. Now when the host is installed I notice the following in
>>>>>>>>>>>>>> the logs
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ....
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Preparing
>>>>>>>>>>>>>> images
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.lib.image.Image:Preparing images
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine:Reloading
>>>>>>>>>>>>>> vm.conf from the shared storage domain
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> INFO:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Trying
>>>>>>>>>>>>>> to get a fresher copy of vm configuration from the OVF_STORE
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> WARNING:ovirt_hosted_engine_ha.lib.ovf.ovf_store.OVFStore:Unable to find
>>>>>>>>>>>>>> OVF_STORE
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config ERROR Unable
>>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.hosted_engine.HostedEngine.config:Unable
>>>>>>>>>>>>>> to get vm.conf from OVF_STORE, falling back to initial vm.conf
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 journal: ovirt-ha-agent
>>>>>>>>>>>>>> ovirt_hosted_engine_ha.agent.agent.Agent ERROR Error: 'Path to volume
>>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>>>> Feb 5 22:33:08 geisha-2 ovirt-ha-agent:
>>>>>>>>>>>>>> ERROR:ovirt_hosted_engine_ha.agent.agent.Agent:Error: 'Path to volume
>>>>>>>>>>>>>> 6a5b55c3-3f03-47cc-b322-d6d7562371be not found in /rhev/data-center/mnt' -
>>>>>>>>>>>>>> trying to restart agent
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> So at this point I was a bit stuck.
>>>>>>>>>>>>>> I removed complete ovirt36 release and installed ovirt35 for
>>>>>>>>>>>>>> the hosted engine deploy.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This time all went without any problem directly.
>>>>>>>>>>>>>> From there I retried upgrade and same problems again.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I run ovirt hosted-engine on nfs3. I did found
>>>>>>>>>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1159314
>>>>>>>>>>>>>> I added no_subtree_check and tried deploy again, but again
>>>>>>>>>>>>>> without luck.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I believe the hosted engine configs are not being migrated to
>>>>>>>>>>>>>> the shared storage. And my problem has todo with
>>>>>>>>>>>>>> http://www.ovirt.org/Features/Hosted_Engine_configuration_on_shared_storage
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And found /etc/ovirt-hosted-engine/hosted-engine.conf,
>>>>>>>>>>>>>> conf_volume_UUID, conf_image_UUID
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hosts run el7
>>>>>>>>>>>>>> Hosted engine runs el6
>>>>>>>>>>>>>> Hosted engine storage is through NFS3
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can I import the hosted engine storage in to engine ( in web
>>>>>>>>>>>>>> interface )?
>>>>>>>>>>>>>> Or how can I create or trigger manually the OVF store the
>>>>>>>>>>>>>> logs complain about?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best Regards,
>>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>>> Pazion
>>>>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain
>>>>>>>>>>>>>> information which is confidential and/or protected by intellectual property
>>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above.
>>>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to,
>>>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form)
>>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you
>>>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by
>>>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you
>>>>>>>>>>>>>> for your cooperation."
>>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>>>> Pazion
>>>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>>>
>>>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>>>
>>>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>>>> "This e-mail and any attachments thereto may contain
>>>>>>>>>>>>> information which is confidential and/or protected by intellectual property
>>>>>>>>>>>>> rights and are intended for the sole use of the recipient(s) named above.
>>>>>>>>>>>>> Any use of the information contained herein (including, but not limited to,
>>>>>>>>>>>>> total or partial reproduction, communication or distribution in any form)
>>>>>>>>>>>>> by persons other than the designated recipient(s) is prohibited. If you
>>>>>>>>>>>>> have received this e-mail in error, please notify the sender either by
>>>>>>>>>>>>> telephone or by e-mail and delete the material from any computer. Thank you
>>>>>>>>>>>>> for your cooperation."
>>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>>>
>>>>>>>>>>> Paul Groeneweg
>>>>>>>>>>> Pazion
>>>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>>>
>>>>>>>>>>> T +31 26 3020038
>>>>>>>>>>> M +31 614 277 577
>>>>>>>>>>> E paul at pazion.nl
>>>>>>>>>>>
>>>>>>>>>>> ***disclaimer***
>>>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>>>> cooperation."
>>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>> Met vriendelijke groeten,
>>>>>>>>>
>>>>>>>>> Paul Groeneweg
>>>>>>>>> Pazion
>>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>>
>>>>>>>>> T +31 26 3020038
>>>>>>>>> M +31 614 277 577
>>>>>>>>> E paul at pazion.nl
>>>>>>>>>
>>>>>>>>> ***disclaimer***
>>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>>> cooperation."
>>>>>>>>>
>>>>>>>> --
>>>>>>>> Met vriendelijke groeten,
>>>>>>>>
>>>>>>>> Paul Groeneweg
>>>>>>>> Pazion
>>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>>
>>>>>>>> T +31 26 3020038
>>>>>>>> M +31 614 277 577
>>>>>>>> E paul at pazion.nl
>>>>>>>>
>>>>>>>> ***disclaimer***
>>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>>> cooperation."
>>>>>>>>
>>>>>>> --
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> Paul Groeneweg
>>>>>>> Pazion
>>>>>>> Webdevelopment - Hosting - Apps
>>>>>>>
>>>>>>> T +31 26 3020038
>>>>>>> M +31 614 277 577
>>>>>>> E paul at pazion.nl
>>>>>>>
>>>>>>> ***disclaimer***
>>>>>>> "This e-mail and any attachments thereto may contain information
>>>>>>> which is confidential and/or protected by intellectual property rights and
>>>>>>> are intended for the sole use of the recipient(s) named above. Any use of
>>>>>>> the information contained herein (including, but not limited to, total or
>>>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>>>> cooperation."
>>>>>>>
>>>>>> --
>>>>> Met vriendelijke groeten,
>>>>>
>>>>> Paul Groeneweg
>>>>> Pazion
>>>>> Webdevelopment - Hosting - Apps
>>>>>
>>>>> T +31 26 3020038
>>>>> M +31 614 277 577
>>>>> E paul at pazion.nl
>>>>>
>>>>> ***disclaimer***
>>>>> "This e-mail and any attachments thereto may contain information which
>>>>> is confidential and/or protected by intellectual property rights and are
>>>>> intended for the sole use of the recipient(s) named above. Any use of the
>>>>> information contained herein (including, but not limited to, total or
>>>>> partial reproduction, communication or distribution in any form) by persons
>>>>> other than the designated recipient(s) is prohibited. If you have received
>>>>> this e-mail in error, please notify the sender either by telephone or by
>>>>> e-mail and delete the material from any computer. Thank you for your
>>>>> cooperation."
>>>>>
>>>> --
>>> Met vriendelijke groeten,
>>>
>>> Paul Groeneweg
>>> Pazion
>>> Webdevelopment - Hosting - Apps
>>>
>>> T +31 26 3020038
>>> M +31 614 277 577
>>> E paul at pazion.nl
>>>
>>> ***disclaimer***
>>> "This e-mail and any attachments thereto may contain information which
>>> is confidential and/or protected by intellectual property rights and are
>>> intended for the sole use of the recipient(s) named above. Any use of the
>>> information contained herein (including, but not limited to, total or
>>> partial reproduction, communication or distribution in any form) by persons
>>> other than the designated recipient(s) is prohibited. If you have received
>>> this e-mail in error, please notify the sender either by telephone or by
>>> e-mail and delete the material from any computer. Thank you for your
>>> cooperation."
>>>
>> --
> Met vriendelijke groeten,
>
> Paul Groeneweg
> Pazion
> Webdevelopment - Hosting - Apps
>
> T +31 26 3020038
> M +31 614 277 577
> E paul at pazion.nl
>
> ***disclaimer***
> "This e-mail and any attachments thereto may contain information which is
> confidential and/or protected by intellectual property rights and are
> intended for the sole use of the recipient(s) named above. Any use of the
> information contained herein (including, but not limited to, total or
> partial reproduction, communication or distribution in any form) by persons
> other than the designated recipient(s) is prohibited. If you have received
> this e-mail in error, please notify the sender either by telephone or by
> e-mail and delete the material from any computer. Thank you for your
> cooperation."
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20160209/8811ad72/attachment-0001.html>
More information about the Users
mailing list