[Users] Abysmal network performance inside hypervisor node
This is a multi-part message in MIME format. ------=_NextPartTM-000-c6aeb801-f168-4270-8537-5789abc26f22 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello,=0A= =0A= we tried to migrate our first VM into the OVirt cluster. Lacking a separate= =0A= host for that task we simply jumped onto on of our FC19 OVirt nodes and=0A= installed the virt-v2v packages. Converting speed ESX->NFS is less than =0A= 10 MB/sec. OUCH!=0A= =0A= To make sure we have no problems in our environment we reinstalled FC19 =0A= on the node and made a direct NFS test: 450MB/sec read speed =0A= =0A= Inserting node into OVirt cluster once again we are down to 15MB/sec=0A= =0A= Could someone clarify which kind of throtteling is activated for the Host= =0A= and if that might be an issue for the VMS.=0A= =0A= Attached a short log:=0A= =0A= Before Ovirt Install:=0A= =0A= [root@colovn01 _ISOs]# date=0A= Mi 11. Dez 08:33:53 CET 2013=0A= [root@colovn01 _ISOs]# pwd=0A= /mnt/_von_nas2/_ISOs=0A= [root@colovn01 _ISOs]# mount | grep "mnt "=0A= 10.10.30.251:/var/nas1 on /mnt type nfs4 (rw,relatime,vers=3D4.0,rsize=3D10= 48576,wsize=3D1048576,namlen=3D255,hard,proto=3Dtcp,port=3D0,timeo=3D600,re= trans=3D2,sec=3Dsys,clientaddr=3D10.10.30.1,local_lock=3Dnone,addr=3D10.10.= 30.251)=0A= [root@colovn01 _ISOs]# echo 3 > /proc/sys/vm/drop_caches=0A= [root@colovn01 _ISOs]# dd if=3DSLES-10-SP2-DVD-x86_64-GM-DVD1.iso of=3D/dev= /null=0A= 5480236+0 Datens=E4tze ein=0A= 5480236+0 Datens=E4tze aus=0A= 2805880832 Bytes (2,8 GB) kopiert, 6,13422 s, 457 MB/s=0A= =0A= After OVirt Install (test done twice to ensure NFS server cache is hot):=0A= =0A= [root@colovn01 _ISOs]# date=0A= Mi 11. Dez 11:33:53 CET 2013=0A= [root@colovn01 _ISOs]# pwd=0A= /mnt/_von_nas2/_ISOs=0A= [root@colovn01 _ISOs]# mount | grep "mnt "=0A= 10.10.30.251:/var/nas1=0A= on /mnt type nfs4 =0A= (rw,relatime,vers=3D4.0,rsize=3D1048576,wsize=3D1048576,namlen=3D255,hard,p= roto=3Dtcp,port=3D0,timeo=3D600,retrans=3D2,sec=3Dsys,clientaddr=3D10.10.30= .1,local_lock=3Dnone,addr=3D10.10.30.251)=0A= [root@colovn01 _ISOs]# echo 3 > /proc/sys/vm/drop_caches=0A= [root@colovn01 _ISOs]# dd if=3DSLES-10-SP2-DVD-x86_64-GM-DVD1.iso of=3D/dev= /null=0A= 5480236+0 Datens=E4tze ein=0A= 5480236+0 Datens=E4tze aus=0A= 2805880832 Bytes (2,8 GB) kopiert, 192,587 s, 14,6 MB/s=0A= [root@colovn01 _ISOs]# echo 3 > /proc/sys/vm/drop_caches=0A= [root@colovn01 _ISOs]# dd if=3DSLES-10-SP2-DVD-x86_64-GM-DVD1.iso of=3D/dev= /null=0A= 5480236+0 Datens=E4tze ein=0A= 5480236+0 Datens=E4tze aus=0A= 2805880832 Bytes (2,8 GB) kopiert, 193,131 s, 14,5 MB/s=0A= =0A= Markus=0A= ------=_NextPartTM-000-c6aeb801-f168-4270-8537-5789abc26f22 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-c6aeb801-f168-4270-8537-5789abc26f22--
Could this have something to do with the bridging config used by vdsm? @Markus: what does your network config look like? And what is the interface you use to access the NFS store?
Von: sander.grendelman@gmail.com [sander.grendelman@gmail.com]" =0A= Gesendet: Mittwoch, 11. Dezember 2013 13:27=0A= An: Markus Stockhausen=0A= Cc: ovirt-users=0A= Betreff: Re: [Users] Abysmal network performance inside hypervisor node= =0A= =0A= Could this have something to do with the bridging config used by vdsm?=0A= =0A= @Markus: what does your network config look like?=0A= And what is the interface you use to access the NFS store?=0A= =0A= We have an IB IPoIB network for storage. Thats why you see a throughput=0A= at 400 MB/sec. For a cross check I mounted the NFS share through the=0A= ovirtmgmt interface (our normal trusted internal network) on a 1GBit=0A= Intel network card.=0A= =0A= The throughput is exatly the same - throttled at round about 16MB/sec:=0A= =0A= Maybe that is no problem at all for the VMs. But in my case it would be hel=
This is a multi-part message in MIME format. ------=_NextPartTM-000-ffd0bae2-b260-475b-bf63-62d301d76f7c Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable pful=0A= to know the switch to temporarily flood the channels for fast migration.=0A= =0A= [root@colovn01 _ISOs]# df .=0A= Dateisystem 1K-blocks Benutzt Verf=FCgbar Verw% Eingeh=E4= ngt auf=0A= 192.168.10.30:/var/nas1 11611801600 9618833408 1992968192 83% /mnt=0A= [root@colovn01 _ISOs]# dd if=3DSLES-11-DVD-x86_64-GM-DVD1.iso of=3D/dev/nul= l=0A= 5148409+0 Datens=E4tze ein=0A= 5148408+0 Datens=E4tze aus=0A= 2635984896 Bytes (2,6 GB) kopiert, 157,247 s, 16,8 MB/s=0A= =0A= [root@colovn01 _ISOs]# ifconfig ovirtmgmt=0A= ovirtmgmt: flags=3D4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500=0A= inet 192.168.10.51 netmask 255.255.255.0 broadcast 192.168.10.255= =0A= inet6 fe80::230:48ff:fed7:9ec4 prefixlen 64 scopeid 0x20<link>=0A= ether 00:30:48:d7:9e:c4 txqueuelen 0 (Ethernet)=0A= RX packets 203382 bytes 2673889310 (2.4 GiB)=0A= RX errors 0 dropped 2092 overruns 0 frame 0=0A= TX packets 160826 bytes 19946957 (19.0 MiB)=0A= TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0=0A= =0A= Markus=0A= ------=_NextPartTM-000-ffd0bae2-b260-475b-bf63-62d301d76f7c Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt" **************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-ffd0bae2-b260-475b-bf63-62d301d76f7c--
I'm just wondering if the tested interfaces are bridged because I've seen some issues with network througput and bridged interfaces on my local system (F19). Basically, if an IP is configured on the bridge itself ( in oVirt this is the case if the network is configured as a VM network ) latency goes up and throughput goes down. Can you rule this one out by using an unbridged interface? On Wed, Dec 11, 2013 at 2:30 PM, Markus Stockhausen <stockhausen@collogia.de> wrote:
Von: sander.grendelman@gmail.com [sander.grendelman@gmail.com]" Gesendet: Mittwoch, 11. Dezember 2013 13:27 An: Markus Stockhausen Cc: ovirt-users Betreff: Re: [Users] Abysmal network performance inside hypervisor node
Could this have something to do with the bridging config used by vdsm?
@Markus: what does your network config look like? And what is the interface you use to access the NFS store?
We have an IB IPoIB network for storage. Thats why you see a throughput at 400 MB/sec. For a cross check I mounted the NFS share through the ovirtmgmt interface (our normal trusted internal network) on a 1GBit Intel network card.
The throughput is exatly the same - throttled at round about 16MB/sec:
Maybe that is no problem at all for the VMs. But in my case it would be helpful to know the switch to temporarily flood the channels for fast migration.
[root@colovn01 _ISOs]# df . Dateisystem 1K-blocks Benutzt Verfügbar Verw% Eingehängt auf 192.168.10.30:/var/nas1 11611801600 9618833408 1992968192 83% /mnt [root@colovn01 _ISOs]# dd if=SLES-11-DVD-x86_64-GM-DVD1.iso of=/dev/null 5148409+0 Datensätze ein 5148408+0 Datensätze aus 2635984896 Bytes (2,6 GB) kopiert, 157,247 s, 16,8 MB/s
[root@colovn01 _ISOs]# ifconfig ovirtmgmt ovirtmgmt: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.10.51 netmask 255.255.255.0 broadcast 192.168.10.255 inet6 fe80::230:48ff:fed7:9ec4 prefixlen 64 scopeid 0x20<link> ether 00:30:48:d7:9e:c4 txqueuelen 0 (Ethernet) RX packets 203382 bytes 2673889310 (2.4 GiB) RX errors 0 dropped 2092 overruns 0 frame 0 TX packets 160826 bytes 19946957 (19.0 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Markus
This is a multi-part message in MIME format. ------=_NextPartTM-000-7ddbe9c9-062b-4b5d-8d87-a12dd295ea78 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Von: sander.grendelman@gmail.com [sander.grendelman@gmail.com]" im A= uftrag von "=0A= Gesendet: Mittwoch, 11. Dezember 2013 14:43=0A= An: Markus Stockhausen=0A= Cc: ovirt-users=0A= Betreff: Re: [Users] Abysmal network performance inside hypervisor node= =0A= =0A= I'm just wondering if the tested interfaces are bridged because I've=0A= seen some issues with=0A= network througput and bridged interfaces on my local system (F19).=0A= =0A= Basically, if an IP is configured on the bridge itself ( in oVirt this=0A= is the case if the network=0A= is configured as a VM network ) latency goes up and throughput goes down.= =0A= =0A= Can you rule this one out by using an unbridged interface?=0A= =0A= I see. But my storage infiniband network is working without a bridge.=0A= That was the mounting network device of my initial mail. So with=0A= (ovirtmgmt) and without bridge (infiniband) I have the same problem.=0A= =0A= [root@colovn01 ~]# ifconfig ib1=0A= ib1: flags=3D4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044=0A= inet 10.10.30.1 netmask 255.255.255.0 broadcast 10.10.30.255=0A= inet6 fe80::223:7dff:ff94:d3fe prefixlen 64 scopeid 0x20<link>=0A= Infiniband hardware address can be incorrect! Please read BUGS section in i= fconfig(8).=0A= infiniband 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00= :00 txqueuelen 256 (InfiniBand)=0A= RX packets 3575120 bytes 7222538528 (6.7 GiB)=0A= RX errors 0 dropped 10 overruns 0 frame 0=0A= TX packets 416942 bytes 29156648 (27.8 MiB)=0A= TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0=0A= =0A= Markus=0A= ------=_NextPartTM-000-7ddbe9c9-062b-4b5d-8d87-a12dd295ea78 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt"
**************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-7ddbe9c9-062b-4b5d-8d87-a12dd295ea78--
Then I'm also out of ideas. You could take al look at the options in /etc/vdsm/vdsm.conf ( information at /usr/share/doc/vdsm-*/vdsm.conf.sample ) On Wed, Dec 11, 2013 at 2:48 PM, Markus Stockhausen <stockhausen@collogia.de> wrote:
Von: sander.grendelman@gmail.com [sander.grendelman@gmail.com]" im Auftrag von " Gesendet: Mittwoch, 11. Dezember 2013 14:43 An: Markus Stockhausen Cc: ovirt-users Betreff: Re: [Users] Abysmal network performance inside hypervisor node
I'm just wondering if the tested interfaces are bridged because I've seen some issues with network througput and bridged interfaces on my local system (F19).
Basically, if an IP is configured on the bridge itself ( in oVirt this is the case if the network is configured as a VM network ) latency goes up and throughput goes down.
Can you rule this one out by using an unbridged interface?
I see. But my storage infiniband network is working without a bridge. That was the mounting network device of my initial mail. So with (ovirtmgmt) and without bridge (infiniband) I have the same problem.
[root@colovn01 ~]# ifconfig ib1 ib1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 2044 inet 10.10.30.1 netmask 255.255.255.0 broadcast 10.10.30.255 inet6 fe80::223:7dff:ff94:d3fe prefixlen 64 scopeid 0x20<link> Infiniband hardware address can be incorrect! Please read BUGS section in ifconfig(8). infiniband 80:00:00:49:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 txqueuelen 256 (InfiniBand) RX packets 3575120 bytes 7222538528 (6.7 GiB) RX errors 0 dropped 10 overruns 0 frame 0 TX packets 416942 bytes 29156648 (27.8 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Markus
This is a multi-part message in MIME format.
------=_NextPartTM-000-f1ba4135-ebf4-40cf-9db9-3a185596e813
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
> Von: sander.grendelman@gmail.com [sander.grendelman@gmail.com]" im A=
uftrag von "=0A=
> Gesendet: Mittwoch, 11. Dezember 2013 14:51=0A=
> An: Markus Stockhausen=0A=
> Cc: ovirt-users=0A=
> Betreff: Re: [Users] Abysmal network performance inside hypervisor node=
=0A=
> =0A=
> Then I'm also out of ideas.=0A=
> =0A=
> You could take al look at the options in /etc/vdsm/vdsm.conf=0A=
> ( information at /usr/share/doc/vdsm-*/vdsm.conf.sample )=0A=
=0A=
Ok, now it is getting crazy. Something is going on inside debugfs. =0A=
I do not know what activated it, but I have two candidates:=0A=
=0A=
1) DMA API debugging has been enabled. So I deactivated it with=0A=
echo Y > ./sys/kernel/debug/dma-api/disabled and instantly=0A=
throughput raises from 17MB/sec to 33MB/sec=0A=
=0A=
2) Another location I cannot identify up to know. A perf trace shows=0A=
the following consumption path.=0A=
=0A=
- 12,51% kworker/1:2 [kernel.kallsyms] [k] lock_is_held =
=0A=
- lock_is_held =0A=
- 82,25% __module_address =0A=
- 99,97% __module_text_address =0A=
is_module_text_address =0A=
__kernel_text_address =0A=
print_context_stack =0A=
dump_trace =0A=
save_stack_trace =0A=
- set_track =0A=
- 52,61% free_debug_processing =0A=
- __slab_free =0A=
- 93,67% kmem_cache_free =0A=
- 72,64% kfree_skbmem =0A=
- 91,34% __kfree_skb =0A=
- 92,58% napi_gro_receive =0A=
ipoib_ib_handle_rx_wc =0A=
ipoib_poll =0A=
net_rx_action =0A=
__do_softirq =0A=
call_softirq =0A=
- do_softirq =0A=
- 75,12% irq_exit =0A=
- 95,67% smp_apic_timer_interrupt =0A=
- apic_timer_interrupt =0A=
- 79,47% __slab_free =0A=
- 99,51% kmem_cache_free =0A=
- 99,19% nfs_free_request =0A=
nfs_release_request =0A=
nfs_readpage_release =0A=
nfs_read_completion =0A=
nfs_readdata_release =0A=
nfs_readpage_release_common =0A=
rpc_free_task =0A=
rpc_async_release =0A=
process_one_work =0A=
worker_thread =0A=
kthread =0A=
ret_from_fork =0A=
+ 0,81% nfs_readhdr_free =0A=
=0A=
Any idea what activates tracing for nfs kmem allocations?=0A=
=0A=
Markus=0A=
------=_NextPartTM-000-f1ba4135-ebf4-40cf-9db9-3a185596e813
Content-Type: text/plain;
name="InterScan_Disclaimer.txt"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="InterScan_Disclaimer.txt"
****************************************************************************
Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte
Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail
irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und
vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte
Weitergabe dieser Mail ist nicht gestattet.
Über das Internet versandte E-Mails können unter fremden Namen erstellt oder
manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine
rechtsverbindliche Willenserklärung.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
Vorstand:
Kadir Akin
Dr. Michael Höhnerbach
Vorsitzender des Aufsichtsrates:
Hans Kristian Langva
Registergericht: Amtsgericht Köln
Registernummer: HRB 52 497
This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.
e-mails sent over the internet may have been written under a wrong name or
been manipulated. That is why this message sent as an e-mail is not a
legally binding declaration of intention.
Collogia
Unternehmensberatung AG
Ubierring 11
D-50678 Köln
executive board:
Kadir Akin
Dr. Michael Höhnerbach
President of the supervisory board:
Hans Kristian Langva
Registry office: district court Cologne
Register number: HRB 52 497
****************************************************************************
------=_NextPartTM-000-f1ba4135-ebf4-40cf-9db9-3a185596e813--
This is a multi-part message in MIME format. ------=_NextPartTM-000-b641f2c8-6895-425e-af42-61d356e0b268 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Ok, now it is getting crazy. Something is going on inside debugfs.=0A= I do not know what activated it, but I have two candidates:=0A= =0A= incredible simple solution: after attaching a FC 19 host to a OVirt =0A= cluster the machine will boot a debug kernel=0A= =0A= [root@colovn03 ~]# dmesg | grep debug | head -1=0A= [ 0.000000] Linux version 3.11.10-200.fc19.x86_64.debug=0A= =0A= Fixed that with:=0A= =0A= [root@colovn03 ~]# vi /etc/default/grub=0A= GRUB_DEFAULT=3D1=0A= [root@colovn03 ~]# grub2-mkconfig -o /boot/grub2/grub.cfg=0A= =0A= Everything is back to normal. Should I file a BZ for that?=0A= =0A= Markus= ------=_NextPartTM-000-b641f2c8-6895-425e-af42-61d356e0b268 Content-Type: text/plain; name="InterScan_Disclaimer.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="InterScan_Disclaimer.txt"
**************************************************************************** Diese E-Mail enthält vertrauliche und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese Mail. Das unerlaubte Kopieren sowie die unbefugte Weitergabe dieser Mail ist nicht gestattet. Über das Internet versandte E-Mails können unter fremden Namen erstellt oder manipuliert werden. Deshalb ist diese als E-Mail verschickte Nachricht keine rechtsverbindliche Willenserklärung. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln Vorstand: Kadir Akin Dr. Michael Höhnerbach Vorsitzender des Aufsichtsrates: Hans Kristian Langva Registergericht: Amtsgericht Köln Registernummer: HRB 52 497 This e-mail may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden. e-mails sent over the internet may have been written under a wrong name or been manipulated. That is why this message sent as an e-mail is not a legally binding declaration of intention. Collogia Unternehmensberatung AG Ubierring 11 D-50678 Köln executive board: Kadir Akin Dr. Michael Höhnerbach President of the supervisory board: Hans Kristian Langva Registry office: district court Cologne Register number: HRB 52 497 **************************************************************************** ------=_NextPartTM-000-b641f2c8-6895-425e-af42-61d356e0b268--
participants (2)
-
Markus Stockhausen -
Sander Grendelman