Re: [ovirt-users] Datacenter unresponsive: recovery procedure?

16 Apr 2015


      This is a multi-part message in MIME format.
--------------090805020300030003020405
Content-Type: text/plain;
	charset="windows-1252"
Content-Transfer-Encoding: quoted-printable


On 04/16/2015 11=3A12 AM=2C Andrea Ghelardi wrote=3A
=3E
=3E =28sorry=3A resending as I wasn=92t part of the list=2C yet=29
=3E
=3E =20
=3E
=3E hi=2C
=3E
=3E this is my first post so hallo all and thank you for reading=2E
=3E
=3E I have an issue with my production Ovirt environment =283=2E5=2E1=2E1-1=
=2Eel6=29=2E
=3E
=3E =20
=3E
=3E My system consists of several datancers=2E
=3E
=3E 2 of them are connected to an iSCSI SAN and they were working fine=2E=

=3E
=3E Until the moment I had the bad idea of deleting a SAN volume from the=

=3E SAN manager before deleting the associated storage on Ovirt=2E From tha=
t
=3E moment=2C the DC where this storage was mounted became not responsive=
=3A
=3E it cannot attach the master storage =28or any other=29=2E
=3E
=3E I tried to
=3E
=3E 1=29 manually destroy the offending storage =28select -=3E destroy=29 b=
ut
=3E still cannot recover the situation=2E
=3E
=3E 2=29 right click on master storage and activate it
=3E
=3E 3=29 re-initialize the datacenter using a NFS storage from the working=

=3E sister DC=2E
=3E
=3E =20
=3E
=3E All Hosts are still running even though their status is =22unknown=22=
=2E
=3E
=3E All VM are still running even though their status is =22not responding=
=22=2E
=3E
=3E =20
=3E
=3E I half resolved the issue by manually restarting the host where the
=3E datastore was originally mounted=2E This cleared the orphaned multipath=
=2E
=3E
=3E However=2C the SPM does not come up still=2E
=3E
=3E This is an extract of the log
=3E
=3E /2015-04-16 03=3A51=3A48=2C069 WARN=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmStopVDSComm=
and=5D
=3E =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D could not stop=
 spm of
=3E pool *00000002-0002-0002-0002-00000000009c*on vds
=3E *89254f23-8748-402a-afc9-08438dca0975*- reason=3A
=3E org=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2EVDSNetworkExcepti=
on=3A
=3E VDSGenericException=3A VDSNetworkException=3A Message timeout which can=
 be
=3E caused by communication issues/
=3E
=3E /2015-04-16 03=3A51=3A48=2C072 INFO=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmStopVDSComm=
and=5D
=3E =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D FINISH=2C
=3E SpmStopVDSCommand=2C log id=3A 4354cf46/
=3E
=3E /2015-04-16 03=3A51=3A48=2C072 WARN=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsProxyData=
=5D
=3E =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D spm stop on spm=
 failed=2C
=3E stopping spm selection!/
=3E
=3E /2015-04-16 03=3A51=3A58=2C223 INFO=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsProxyData=
=5D
=3E =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D hostFromVds=3A=
=3AselectedVds
=3E - Brachetto=2C spmStatus Free=2C storage pool IRDC-INTEL/
=3E
=3E /2015-04-16 03=3A51=3A58=2C225 ERROR
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsProxyData=
=5D
=3E =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D SPM Init=3A coul=
d not find
=3E reported vds or not up - pool=3AIRDC-INTEL vds=5Fspm=5Fid=3A 3/
=3E
=3E /2015-04-16 03=3A51=3A58=2C239 INFO=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsProxyData=
=5D
=3E =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D SPM selection -=
 vds seems
=3E as spm sovana/
=3E
=3E /2015-04-16 03=3A51=3A58=2C252 INFO=20
=3E =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmStopVDSComm=
and=5D
=3E =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D START=2C
=3E SpmStopVDSCommand=28HostName =3D sovana=2C HostId =3D
=3E 89254f23-8748-402a-afc9-08438dca0975=2C storagePoolId =3D
=3E 00000002-0002-0002-0002-00000000009c=29=2C log id=3A 63a17687/
=3E
=3E =20
=3E
=3E storagePoolId =3D 00000002-0002-0002-0002-00000000009c is =28was=29
=3E hertz-dstore2 which does not exists anymore on SAN adn ovirt
=3E
=3E hostid  89254f23-8748-402a-afc9-08438dca0975 is sovana server =28curren=
t
=3E SPM=29
=3E
=3E =20
=3E
=3E =20
=3E
=3E =20
=3E
=3E =20
=3E
=3E I=92m thinking about
=3E
=3E /Put the hosted engine host into Maintenance///
=3E
=3E /Shutdown Ovirt Manager///
=3E
=3E /Rebooted SPM server///
=3E
=3E /Restarted Ovirt Manager///
=3E
=3E /Took hosted engine host out of Maintenance///
=3E
=3E =20
=3E
=3E =20
=3E
=3E any help or clue is highly welcomed with cheers and beers
=3E
=3E thank you!
=3E
=3E


I had comparable issues nearly a year ago after a failed iSCSI failover
that ended in a split brain=2E Wasn=27t able to recover from it=2E

https=3A//bugzilla=2Eredhat=2Ecom/show=5Fbug=2Ecgi=3Fid=3D1108576


Met vriendelijke groet=2C With kind regards=2C

Jorick Astrego

Netbulae Virtualization Experts=20

----------------

=09Tel=3A 053 20 30 270 =09info=40netbulae=2Eeu =09Staalsteden 4-3A =09KvK=
 08198180
 =09Fax=3A 053 20 30 271 =09www=2Enetbulae=2Eeu =097547 TA Enschede =09BTW=
 NL821234584B01

----------------


--------------090805020300030003020405
Content-Type: text/html;
	charset="windows-1252"
Content-Transfer-Encoding: quoted-printable

=3Chtml=3E
  =3Chead=3E
    =3Cmeta content=3D=22text/html=3B charset=3Dwindows-1252=22
      http-equiv=3D=22Content-Type=22=3E
  =3C/head=3E
  =3Cbody bgcolor=3D=22=23FFFFFF=22 text=3D=22=23000000=22=3E
    =3Cbr=3E
    =3Cbr=3E
    =3Cdiv class=3D=22moz-cite-prefix=22=3EOn 04/16/2015 11=3A12 AM=2C Andr=
ea Ghelardi
      wrote=3A=3Cbr=3E
    =3C/div=3E
    =3Cblockquote
      cite=3D=22mid=3A4d3f0edc7c9e811c8c2b43e3880a8af6=40mail=2Egmail=2Ecom=
=22
      type=3D=22cite=22=3E
      =3Cmeta http-equiv=3D=22Content-Type=22 content=3D=22text/html=3B
        charset=3Dwindows-1252=22=3E
      =3Cmeta name=3D=22Generator=22 content=3D=22Microsoft Word 14 =28filt=
ered
        medium=29=22=3E
      =3Cstyle=3E=3C!--
/* Font Definitions */
=40font-face
=09=7Bfont-family=3ACalibri=3B
=09panose-1=3A2 15 5 2 2 2 4 3 2 4=3B=7D
=40font-face
=09=7Bfont-family=3ATahoma=3B
=09panose-1=3A2 11 6 4 3 5 4 4 2 4=3B=7D
/* Style Definitions */
p=2EMsoNormal=2C li=2EMsoNormal=2C div=2EMsoNormal
=09=7Bmargin=3A0in=3B
=09margin-bottom=3A=2E0001pt=3B
=09font-size=3A11=2E0pt=3B
=09font-family=3A=22Calibri=22=2C=22sans-serif=22=3B=7D
a=3Alink=2C span=2EMsoHyperlink
=09=7Bmso-style-priority=3A99=3B
=09color=3A=230563C1=3B
=09text-decoration=3Aunderline=3B=7D
a=3Avisited=2C span=2EMsoHyperlinkFollowed
=09=7Bmso-style-priority=3A99=3B
=09color=3A=23954F72=3B
=09text-decoration=3Aunderline=3B=7D
span=2EEmailStyle17
=09=7Bmso-style-type=3Apersonal=3B
=09font-family=3A=22Calibri=22=2C=22sans-serif=22=3B
=09color=3Awindowtext=3B=7D
span=2EEmailStyle18
=09=7Bmso-style-type=3Apersonal-reply=3B
=09font-family=3A=22Calibri=22=2C=22sans-serif=22=3B
=09color=3A=2344546A=3B=7D
=2EMsoChpDefault
=09=7Bmso-style-type=3Aexport-only=3B
=09font-size=3A10=2E0pt=3B=7D
=40page WordSection1
=09=7Bsize=3A8=2E5in 11=2E0in=3B
=09margin=3A1=2E0in 1=2E0in 1=2E0in 1=2E0in=3B=7D
div=2EWordSection1
=09=7Bpage=3AWordSection1=3B=7D
--=3E=3C/style=3E
      =3Cdiv class=3D=22WordSection1=22=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan
style=3D=22font-size=3A10=2E0pt=3Bfont-family=3A=26quot=3BTahoma=26quot=3B=
=2C=26quot=3Bsans-serif=26quot=3B=3Bcolor=3A=2344546a=22=3E=28sorry=3A
            resending as I wasn=92t part of the list=2C yet=29=3C/span=3E=
=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3Ehi=2C=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3Ethis is my first post so hallo all a=
nd
          thank you for reading=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EI have an issue with my production O=
virt
          environment =283=2E5=2E1=2E1-1=2Eel6=29=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=A0=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EMy system consists of several datanc=
ers=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E2 of them are connected to an iSCSI=
 SAN and
          they were working fine=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EUntil the moment I had the bad idea=
 of
          deleting a SAN volume from the SAN manager before deleting the
          associated storage on Ovirt=2E From that moment=2C the DC where=

          this storage was mounted became not responsive=3A it cannot
          attach the master storage =28or any other=29=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EI tried to=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E1=29 manually destroy the offending=
 storage
          =28select -=26gt=3B destroy=29 but still cannot recover the situa=
tion=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E2=29 right click on master storage a=
nd
          activate it=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E3=29 re-initialize the datacenter us=
ing a NFS
          storage from the working sister DC=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=A0=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EAll Hosts are still running even tho=
ugh
          their status is =22unknown=22=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3EAll VM are still running even though=
 their
          status is =22not responding=22=2E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=A0=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3EI half resolved
            the issue by manually restarting the host where the
            datastore was originally mounted=2E This cleared the orphaned=

            multipath=2E=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3EHowever=2C the
            SPM does not come up still=2E=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3EThis is an
            extract of the log=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A48=2C069 WARN=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmS=
topVDSCommand=5D
              =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D could=
 not
              stop spm of pool =3C/span=3E=3Cb=3E=3Cspan style=3D=22color=
=3Ared=22=3E00000002-0002-0002-0002-00000000009c=3C/span=3E=3C/b=3E=3Cspan=

              style=3D=22color=3A=2344546a=22=3E on vds =3C/span=3E=3Cb=3E=
=3Cspan
                style=3D=22color=3Ared=22=3E89254f23-8748-402a-afc9-08438dc=
a0975=3C/span=3E=3C/b=3E=3Cspan
              style=3D=22color=3A=2344546a=22=3E - reason=3A
              org=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2EVDSNetw=
orkException=3A
              VDSGenericException=3A VDSNetworkException=3A Message timeout=

              which can be caused by communication issues=3C/span=3E=3C/i=
=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A48=2C072 INFO=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmS=
topVDSCommand=5D
              =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D FINIS=
H=2C
              SpmStopVDSCommand=2C log id=3A 4354cf46=3C/span=3E=3C/i=3E=3C=
/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A48=2C072 WARN=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsP=
roxyData=5D
              =28DefaultQuartzScheduler=5FWorker-14=29 =5B61a44b19=5D spm s=
top on
              spm failed=2C stopping spm selection!=3C/span=3E=3C/i=3E=3C/p=
=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A58=2C223 INFO=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsP=
roxyData=5D
              =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D
              hostFromVds=3A=3AselectedVds - Brachetto=2C spmStatus Free=2C=

              storage pool IRDC-INTEL=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A58=2C225 ERROR
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsP=
roxyData=5D
              =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D SPM In=
it=3A
              could not find reported vds or not up - pool=3AIRDC-INTEL
              vds=5Fspm=5Fid=3A 3=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A58=2C239 INFO=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Eirsbroker=2EIrsP=
roxyData=5D
              =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D SPM se=
lection
              - vds seems as spm sovana=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Ci=3E=3Cspan style=3D=22color=3A=
=2344546a=22=3E2015-04-16
              03=3A51=3A58=2C252 INFO=A0
              =5Borg=2Eovirt=2Eengine=2Ecore=2Evdsbroker=2Evdsbroker=2ESpmS=
topVDSCommand=5D
              =28DefaultQuartzScheduler=5FWorker-4=29 =5B4ca2d938=5D START=
=2C
              SpmStopVDSCommand=28HostName =3D sovana=2C HostId =3D
              89254f23-8748-402a-afc9-08438dca0975=2C storagePoolId =3D
              00000002-0002-0002-0002-00000000009c=29=2C log id=3A 63a17687=
=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3EstoragePoolId =3D
            00000002-0002-0002-0002-00000000009c is =28was=29 hertz-dstore2=

            which does not exists anymore on SAN adn ovirt=3C/span=3E=3C/p=
=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3Ehostid
            =A089254f23-8748-402a-afc9-08438dca0975 is sovana server
            =28current SPM=29=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3EI=92m thinking
            about =3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22 style=3D=22background=3Awhite=22=3E=3C=
i=3E=3Cspan
              style=3D=22color=3A=231f497d=22=3EPut the hosted engine host=
 into
              Maintenance=3C/span=3E=3C/i=3E=3Ci=3E=3Cspan
              style=3D=22font-size=3A12=2E0pt=3Bfont-family=3A=26quot=3BTim=
es New
              Roman=26quot=3B=2C=26quot=3Bserif=26quot=3B=3Bcolor=3A=232222=
22=22=3E=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22 style=3D=22background=3Awhite=22=3E=3C=
i=3E=3Cspan
              style=3D=22color=3A=231f497d=22=3EShutdown=A0Ovirt=A0Manager=
=3C/span=3E=3C/i=3E=3Ci=3E=3Cspan
              style=3D=22font-size=3A12=2E0pt=3Bfont-family=3A=26quot=3BTim=
es New
              Roman=26quot=3B=2C=26quot=3Bserif=26quot=3B=3Bcolor=3A=232222=
22=22=3E=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22 style=3D=22background=3Awhite=22=3E=3C=
i=3E=3Cspan
              style=3D=22color=3A=231f497d=22=3ERebooted SPM server=3C/span=
=3E=3C/i=3E=3Ci=3E=3Cspan
              style=3D=22font-size=3A12=2E0pt=3Bfont-family=3A=26quot=3BTim=
es New
              Roman=26quot=3B=2C=26quot=3Bserif=26quot=3B=3Bcolor=3A=232222=
22=22=3E=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22 style=3D=22background=3Awhite=22=3E=3C=
i=3E=3Cspan
              style=3D=22color=3A=231f497d=22=3ERestarted=A0Ovirt=A0Manager=
=3C/span=3E=3C/i=3E=3Ci=3E=3Cspan
              style=3D=22font-size=3A12=2E0pt=3Bfont-family=3A=26quot=3BTim=
es New
              Roman=26quot=3B=2C=26quot=3Bserif=26quot=3B=3Bcolor=3A=232222=
22=22=3E=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22 style=3D=22background=3Awhite=22=3E=3C=
i=3E=3Cspan
              style=3D=22color=3A=231f497d=22=3ETook hosted engine host out=
 of
              Maintenance=3C/span=3E=3C/i=3E=3Ci=3E=3Cspan
              style=3D=22font-size=3A12=2E0pt=3Bfont-family=3A=26quot=3BTim=
es New
              Roman=26quot=3B=2C=26quot=3Bserif=26quot=3B=3Bcolor=3A=232222=
22=22=3E=3C/span=3E=3C/i=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=3Cspan style=3D=22color=3A=2344546a=
=22=3E=A0=3C/span=3E=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3E=A0=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3Eany help or clue is highly welcomed=
 with
          cheers and beers=3C/p=3E
        =3Cp class=3D=22MsoNormal=22=3Ethank you!=3C/p=3E
      =3C/div=3E
      =3Cbr=3E
    =3C/blockquote=3E
    =3Cbr=3E
    =3Cbr=3E
    I had comparable issues nearly a year ago after a failed iSCSI
    failover that ended in a split brain=2E Wasn=27t able to recover from=

    it=2E=3Cbr=3E
    =3Cbr=3E
    =3Ca class=3D=22moz-txt-link-freetext=22 href=3D=22https=3A//bugzilla=
=2Eredhat=2Ecom/show=5Fbug=2Ecgi=3Fid=3D1108576=22=3Ehttps=3A//bugzilla=2Er=
edhat=2Ecom/show=5Fbug=2Ecgi=3Fid=3D1108576=3C/a=3E=3Cbr=3E
    =3Cbr=3E
    =3Cbr=3E
 =20=
=3CBR /=3E
=3CBR /=3E
=3Cb style=3D=22color=3A=23604c78=22=3E=3C/b=3E=3Cbr=3E=3Cspan style=3D=22c=
olor=3A=23604c78=3B=22=3E=3Cfont color=3D=22000000=22=3E=3Cspan style=3D=22=
mso-fareast-language=3Aen-gb=3B=22 lang=3D=22NL=22=3EMet vriendelijke groet=
=2C With kind regards=2C=3Cbr=3E=3Cbr=3E=3C/span=3EJorick Astrego=3C/font=
=3E=3C/span=3E=3Cb style=3D=22color=3A=23604c78=22=3E=3Cbr=3E=3Cbr=3ENetbul=
ae Virtualization Experts =3C/b=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3B=
border-top=3A1px solid =23ccc=3B=22=3E=3Ctable style=3D=22width=3A 522px=22=
=3E=3Ctbody=3E=3Ctr=3E=3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=
=22=3ETel=3A  053 20 30 270=3C/td=3E    =3Ctd style=3D=22width=3A 130px=3Bf=
ont-size=3A 10px=22=3Einfo=40netbulae=2Eeu=3C/td=3E    =3Ctd style=3D=22wid=
th=3A 130px=3Bfont-size=3A 10px=22=3EStaalsteden 4-3A=3C/td=3E    =3Ctd sty=
le=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EKvK 08198180=3C/td=3E=3C/tr=
=3E=3Ctr=3E    =3Ctd style=3D=22width=3A 130px=3Bfont-size=3A 10px=22=3EFax=
=3A 053 20 30 271=3C/td=3E    =3Ctd style=3D=22width=3A 130px=3Bfont-size=
=3A 10px=22=3Ewww=2Enetbulae=2Eeu=3C/td=3E    =3Ctd style=3D=22width=3A 130=
px=3Bfont-size=3A 10px=22=3E7547 TA Enschede=3C/td=3E    =3Ctd style=3D=22w=
idth=3A 130px=3Bfont-size=3A 10px=22=3EBTW NL821234584B01=3C/td=3E=3C/tr=3E=
=3C/tbody=3E=3C/table=3E=3Cbr=3E=3Chr style=3D=22border=3Anone=3Bborder-top=
=3A1px solid =23ccc=3B=22=3E=3CBR /=3E
=3C/body=3E
=3C/html=3E


--------------090805020300030003020405--