4:35 p.m.
--_000_DC9ABD90CB55EC448B1F682D5731AF7407A78DB15FIAD2MBX17mex0_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Hi,
there was no issue here, it seems I was a totally benighted...
...when trying to add a new ISO domain but using "import domain" function f=
or NFS...
Sorry if somebody already wasted time here!
Best,
Sven.
Sven Knohsalla | System Administration | Netbiscuits
Office +49 631 68036 433 | Fax +49 631 68036 111 |E-Mail s.knohsalla@netbi=
scuits.com | Skype: netbiscuits.admin
Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY
Von: Sven Knohsalla
Gesendet: Mittwoch, 31. Juli 2013 14:54
An: users(a)ovirt.org
Cc: Sven Knohsalla (s.knohsalla(a)netbiscuits.com)
Betreff: oVirt 3.2.1 adding NFS ISO/Export fails
Hi,
did anybody run into the issue following issue under oVirt 3.2.1 + F18 base=
d nodes ?
-->
NFS ISO/Export mount via webgui gives error message "There is no storage do=
main under the specified path. Check event log for more details."
(no advanced mount parameters given)
Mounting NFS volume under console is working fine, no special mount options=
given, NFS server is a QNAP NAS which did work under oVirt 3.0 without any=
problems (disks cleaned!).
Created a NFS mount point "de-ovirt-iso" for testing NFS mount /ISO&Export
=
functionality under oVirt 3.2.1.
Watching the mount process on console initiated from the WebGui, shows a co=
rrect mounted ISO/Export domain for ~1 sec.
under /rhev/..
Then, oVirt engine triggers a disconnection (Task: Disconnecting from Stora=
ge server on <hypervisor>)
engine log snippet when performing NFS mount:
2013-07-31 14:16:49,204 INFO [org.ovirt.engine.core.bll.storage.AddStorage=
ServerConnectionCommand] (ajp--127.0.0.1-8702-2) [74556c31] Running command=
: AddStorageServerConnectionCommand internal: false. Entities affected : I=
D: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-07-31 14:16:49,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co=
nnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [74556c31] START, Con=
nectStorageServerVDSCommand(HostName =3D deovn-a04, HostId =3D c6618b0e-e56=
b-4e38-ae2b-cf4944e6648c, storagePoolId =3D 00000000-0000-0000-0000-0000000=
00000, storageType =3D NFS, connectionList =3D [{ id: e9911b63-a50c-441f-af=
02-0f7eb5be0312, connection: 172.16.101.95:/de-ovirt-iso, iqn: null, vfsTyp=
e: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: =
null };]), log id: 44b45cc7
2013-07-31 14:16:49,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co=
nnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [74556c31] FINISH, Co=
nnectStorageServerVDSCommand, return: {e9911b63-a50c-441f-af02-0f7eb5be0312=
=3D0}, log id: 44b45cc7
2013-07-31 14:16:49,345 INFO [org.ovirt.engine.core.bll.storage.GetExistin=
gStorageDomainListQuery] (ajp--127.0.0.1-8702-2) START, GetExistingStorageD=
omainListQuery(org.ovirt.engine.core.common.queries.GetExistingStorageDomai=
nListParameters@24b89633<mailto:org.ovirt.engine.core.common.queries.GetExi=
stingStorageDomainListParameters@24b89633>), log id: 2fe40ba5
2013-07-31 14:16:49,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-2) START, HSMGetStor=
ageDomainsListVDSCommand(HostName =3D deovn-a04, HostId =3D c6618b0e-e56b-4=
e38-ae2b-cf4944e6648c, storagePoolId=3D00000000-0000-0000-0000-000000000000=
, storageType=3DNFS, storageDomainType=3DISO, path=3D172.16.101.95:/de-ovir=
t-iso), log id: 7e399cee
2013-07-31 14:16:51,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HS=
MGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-2) FINISH, HSMGetSto=
rageDomainsListVDSCommand, return: [], log id: 7e399cee
2013-07-31 14:16:51,632 INFO [org.ovirt.engine.core.bll.storage.GetExistin=
gStorageDomainListQuery] (ajp--127.0.0.1-8702-2) FINISH, GetExistingStorage=
DomainListQuery, log id: 2fe40ba5
2013-07-31 14:16:51,687 INFO [org.ovirt.engine.core.bll.storage.RemoveStor=
ageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [51bf485f] Running comm=
and: RemoveStorageServerConnectionCommand internal: false. Entities affecte=
d : ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-07-31 14:16:51,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Di=
sconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [51bf485f] START, =
DisconnectStorageServerVDSCommand(HostName =3D deovn-a04, HostId =3D c6618b=
0e-e56b-4e38-ae2b-cf4944e6648c, storagePoolId =3D 00000000-0000-0000-0000-0=
00000000000, storageType =3D NFS, connectionList =3D [{ id: null, connectio=
n: 172.16.101.95:/de-ovirt-iso, iqn: null, vfsType: null, mountOptions: nul=
l, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63ebbd8=
4
2013-07-31 14:16:52,436 INFO [org.ovirt.engine.core.bll.LoginUserCommand] =
(ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
2013-07-31 14:16:52,443 WARN [org.ovirt.engine.core.bll.GetConfigurationVa=
lueQuery] (ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (Appli=
cationMode) with null version, using default general for version
2013-07-31 14:16:52,444 WARN [org.ovirt.engine.core.bll.GetConfigurationVa=
lueQuery] (ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (VdcVe=
rsion) with null version, using default general for version
2013-07-31 14:16:53,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Di=
sconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [51bf485f] FINISH,=
DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000=
000000=3D0}, log id: 63ebbd84
snippet from vdsm log affected hypervisor:
[root@deovn-a04 data-center]# tail /var/log/vdsm/vdsm.log -n 1000 | grep 1=
72.16.101.95
Thread-1757::DEBUG::2013-07-31 14:38:49,531::misc::84::Storage.Misc.excCmd:=
:(<lambda>) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,ti=
meo=3D600,retrans=3D6,nfsvers=3D3 172.16.101.95:/de-ovirt-iso /rhev/data-ce=
nter/mnt/172.16.101.95:_de-ovirt-iso' (cwd None)
Thread-1767::INFO::2013-07-31 14:38:52,021::logUtils::41::dispatcher::(wrap=
per) Run and protect: disconnectStorageServer(domType=3D1, spUUID=3D'000000=
00-0000-0000-0000-000000000000', conList=3D[{'connection':
'172.16.101.95:/=
de-ovirt-iso', 'iqn': '', 'portal': '',
'user': '', 'password': '******', '=
id': '00000000-0000-0000-0000-000000000000', 'port': ''}],
options=3DNone)
As the mount is working under console and I can't see an obvious error in t=
he engine/vdsm log I'm a bit confused where to look at here...
I'm running a POSIXFS storage domain, I tried to mount NFS iso/export as "d=
ata" storage, both were working fine! (Detached NAS mounts from oVirt and c=
leaned created content from engine after testing)
So, it seems there's a difference between mounting NFS mountpoints via Pos=
ixFS and ISO/Export?
Thanks for any suggests!
Best regards,
Sven.
Sven Knohsalla | System Administration | Netbiscuits
Office +49 631 68036 433 | Fax +49 631 68036 111 |E-Mail s.knohsalla@netbi=
scuits.com<mailto:s.knohsalla@netbiscuits.com> | Skype: netbiscuits.admin
Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY
Von: Sven Knohsalla
Gesendet: Mittwoch, 31. Juli 2013 10:45
An: users@ovirt.org<mailto:users@ovirt.org>
Cc: Sven Knohsalla (s.knohsalla@netbiscuits.com<mailto:s.knohsalla@netbiscu=
its.com>)
Betreff: oVirt 3.2.1 /F18 node vdsm issue
Hi,
I just set up a vanilla oVirt 3.2.1 environment, containing
oVirt engine
3.2.1 F18 based oVirt Engine Version: 3.2.1-1.fc18
F18 minimal node
vdsm-4.10.3-10.fc18 libvirt-0.10.2.6-1.fc18 1.2.2 - 13.fc18 K ernel 3.9.11 =
- 200.fc18.x86_64
gluster-storage, replicated, EL6
glusterfs 3.4.0
F18 minimal node was installed via oVirt engine, worked fine, expect vdsmd =
wasn't started
(node non-responsive for engine)
Manually starting vdsm works (HV up in engine), autostart of vdsm (during b=
oot) fails:
Jul 31 10:04:47 deovn-a04 systemd-vdsmd[711]: Note: Forwarding request to '=
systemctl disable libvirt-guests.service'.
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: vdsm: libvirt already configu=
red for vdsm [ OK ]
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting ntpd...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl=
start ntpd.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting wdmd...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl=
start wdmd.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting sanlock...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl=
start sanlock.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting iscsid:
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl=
start libvirtd.service
Jul 31 10:06:17 deovn-a04 systemd[1]: vdsmd.service operation timed out. Te=
rminating.
Jul 31 10:06:17 deovn-a04 systemd[1]: Unit vdsmd.service entered failed sta=
te.
Did find similar issues:
https://bugzilla.redhat.com/show_bug.cgi?id=3D832199
https://bugzilla.redhat.com/show_bug.cgi?id=3D720359
http://lists.ovirt.org/pipermail/users/2012-September/009624.html
Unfortunately, there's no hint.. libvirtd starts without any latency when m=
anually triggered.
The strange thing here: F18 node deploy worked with engine 3.2.0 3-4 weeks =
ago.
Did update engine / F18.
Can anybody give me an advice here?
Thanks in advance!
Best,
Sven
Sven Knohsalla | System Administration
Office +49 631 68036 433 | Fax +49 631 68036 111 |E-Mail s.knohsalla@netbi=
scuits.com<mailto:s.knohsalla@netbiscuits.com> | Skype: netbiscuits.admin
Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY
[https://my.netbiscuits.com/image/image_gallery?uuid=3D3a1a9d19-c305-4032-8=
cef-00b03c3d4c79&groupId=3D10211&t=3D1361534926402]<http://www.netbiscuits.=
com/>
[https://my.netbiscuits.com/image/image_gallery?uuid=3D9e553e7b-3e7d-4784-=
b274-15aa1dfb48e2&groupId=3D10211&t=3D1361533377340] <https://www.netbiscui=
ts.com/news> [https://my.netbiscuits.com/image/image_gallery?uuid=3D1d1a5e=
29-ceda-4ab1-9353-67a1e838364d&groupId=3D10211&t=3D1347281040591] <https://=
twitter.com/netbiscuits> [https://my.netbiscuits.com/image/image_gallery?u=
uid=3Dc99bf866-be25-4236-a0ee-dca68ec828a5&groupId=3D10211&t=3D134728098384=
8] <http://www.linkedin.com/company/netbiscuits> [https://my.netbiscuits.c=
om/image/image_gallery?uuid=3Dd62ba951-14dc-450d-b5f1-be33884225e3&groupId=
=3D10211&t=3D1347280983872] <http://www.xing.com/companies/netbiscuitsgmbh>=
[https://my.netbiscuits.com/image/image_gallery?uuid=3D7b28f500-f415-40bb=
-851f-0cd55beeaf45&groupId=3D10211&t=3D1347280983791] <https://www.facebook=
.com/Netbiscuits> [https://my.netbiscuits.com/image/image_gallery?uuid=3Dc=
c8764d0-a5ac-4623-bb63-da3ca7c97f94&groupId=3D10211&t=3D1347280983836] <htt=
ps://plus.google.com/u/0/112410769451962733032> [https://my.netbiscuits.co=
m/image/image_gallery?uuid=3Da15e871c-a11b-419c-acca-da5a0ebd5856&groupId=
=3D10211&t=3D1347281040599] <http://www.youtube.com/user/netbiscuits>
Register Court: Local Court Kaiserslautern | Commercial Register ID: HR B 3=
604
Management Board: Guido Moggert, Michael Neidh=F6fer, Christian Reitz, Mart=
in S=FC=DF
This message and any files transmitted with it are confidential and intende=
d solely for the use of the individual or entity to whom they are addressed=
. It may also be privileged or otherwise protected by work product immunity=
or other legal rules. Please notify the sender immediately by e-mail if yo=
u have received this e-mail by mistake and delete this e-mail from your sys=
tem. If you are not the intended recipient you are notified that disclosing=
, copying, distributing or taking any action in reliance on the contents of=
this information is strictly prohibited.
Warning: Although Netbiscuits has taken reasonable precautions to ensure no=
viruses are present in this email, the company cannot accept responsibilit=
y for any loss or damage arising from the use of this email or attachments.
[http://www.netbiscuits.com/image/image_gallery?uuid=3D0ba7711a-a277-4ea0-a=
cb0-17fe13c3089d&groupId=3D10211&t=3D1348560850164]Please consider the envi=
ronment before printing
--_000_DC9ABD90CB55EC448B1F682D5731AF7407A78DB15FIAD2MBX17mex0_
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
<html xmlns:v=3D"urn:schemas-microsoft-com:vml"
xmlns:o=3D"urn:schemas-micr=
osoft-com:office:office" xmlns:w=3D"urn:schemas-microsoft-com:office:word"
=
xmlns:m=3D"http://schemas.microsoft.com/office/2004/12/omml"
xmlns=3D"http:=
//www.w3.org/TR/REC-html40"><head><meta http-equiv=3DContent-Type
content=
=3D"text/html; charset=3Diso-8859-1"><meta name=3DGenerator
content=3D"Micr=
osoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:*
{behavior:url(#d=
efault#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Sprechblasentext Zchn";
margin:0cm;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.SprechblasentextZchn
{mso-style-name:"Sprechblasentext Zchn";
mso-style-priority:99;
mso-style-link:Sprechblasentext;
font-family:"Tahoma","sans-serif";}
span.E-MailFormatvorlage19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.gwt-inlinelabel
{mso-style-name:gwt-inlinelabel;}
span.E-MailFormatvorlage21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.E-MailFormatvorlage22
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1" />
</o:shapelayout></xml><![endif]--></head><body lang=3DEN-US
link=3Dblue vli=
nk=3Dpurple><div class=3DWordSection1><p class=3DMsoNormal><span
style=3D'c=
olor:#1F497D'>Hi,<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'=
color:#1F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal><span style=
=3D'color:#1F497D'>there was no issue here, it seems I was a totally benigh=
ted…<br>…when trying to add a new ISO domain but using
“i=
mport domain” function for NFS…<br><br>Sorry if somebody
alread=
y wasted time here!<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'color:#1F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal><span s=
tyle=3D'color:#1F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal><sp=
an
style=3D'color:#1F497D'>Best,<o:p></o:p></span></p><p
class=3DMsoNormal>=
<span
style=3D'color:#1F497D'>Sven.<o:p></o:p></span></p><p
class=3DMsoNorm=
al><span
style=3D'color:#1F497D'><o:p> </o:p></span></p><div><p
class=
=3DMsoNormal><span style=3D'color:#5A5A5A'>Sven Knohsalla
</span><span styl=
e=3D'color:#C00000'>|</span><span style=3D'color:#5A5A5A'>
System Administr=
ation | Netbiscuits<o:p></o:p></span></p><p
class=3DMsoNormal><span style=
=3D'color:gray'><o:p> </o:p></span></p><p
class=3DMsoNormal><span styl=
e=3D'font-size:9.0pt;color:#5A5A5A'>Office +49 631 68036 433 | Fax +49 631 =
68036 111=A0 |E-Mail</span><span
style=3D'font-size:9.0pt;color:#505050'> <=
/span><span
style=3D'font-size:9.0pt;color:#0070C0'>s.knohsalla@netbiscuits=
.com</span><span style=3D'font-size:9.0pt;color:#5A5A5A'> | Skype:
</span><=
span
style=3D'font-size:9.0pt;color:#0070C0'>netbiscuits.admin</span><span
=
style=3D'font-size:9.0pt;color:#5A5A5A'>
<o:p></o:p></span></p><p class=3DM=
soNormal><span lang=3DDE
style=3D'font-size:9.0pt;color:#5A5A5A'>Netbiscuit=
s GmbH | Europaallee 10 | 67657 |
GERMANY<o:p></o:p></span></p></div><p cla=
ss=3DMsoNormal><span
style=3D'color:#1F497D'><o:p> </o:p></span></p><d=
iv><div style=3D'border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0=
cm 0cm 0cm'><p class=3DMsoNormal><b><span lang=3DDE
style=3D'font-size:10.0=
pt;font-family:"Tahoma","sans-serif"'>Von:</span></b><span
lang=3DDE style=
=3D'font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
Sven Knohsalla <br=
<b>Gesendet:</b> Mittwoch, 31. Juli 2013
14:54<br><b>An:</b> users(a)ovirt.o=
rg<br><b>Cc:</b> Sven Knohsalla
(s.knohsalla(a)netbiscuits.com)<br><b>Betreff=
:</b> oVirt 3.2.1 adding NFS ISO/Export fails
<o:p></o:p></span></p></div><=
/div><p class=3DMsoNormal><o:p> </o:p></p><p
class=3DMsoNormal><span s=
tyle=3D'color:#1F497D'>Hi,<br><br>did anybody run into the issue
following =
issue under oVirt 3.2.1 + F18 based nodes ?<br>--><br><br>NFS
ISO/Export=
mount via webgui </span><span style=3D'color:#376092'>gives error
message =
“There is no storage domain under the specified path. Check event log=
for more details.”<o:p></o:p></span></p><p
class=3DMsoNormal><span s=
tyle=3D'color:#376092'>(no advanced mount parameters
given)<br><br>Mounting=
NFS volume under console is working fine, no special mount options given, =
NFS server is a QNAP NAS which did work under oVirt 3.0 without any problem=
s (disks cleaned!).<br>Created a NFS mount point
“de-ovirt-iso”=
for testing NFS mount /ISO&Export functionality under oVirt 3.2.1.<br>=
<br>Watching the mount process on console initiated from the WebGui, shows =
a correct mounted ISO/Export domain for ~1 sec.<br>under /rhev/..
<o:p></o:=
p></span></p><p class=3DMsoNormal><span
style=3D'color:#002060'><br></span>=
<span style=3D'color:#1F497D'>Then, oVirt engine triggers a disconnection
(=
Task: Disconnecting from Storage server on
<hypervisor>)<br><br><u>en=
gine log snippet when performing NFS mount:</u><br><br>2013-07-31
14:16:49,=
204 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectio=
nCommand] (ajp--127.0.0.1-8702-2) [74556c31] Running command: AddStorageSer=
verConnectionCommand internal: false. Entities affected : ID: aaa0000=
0-0000-0000-0000-123456789aaa Type:
System<o:p></o:p></span></p><p class=3D=
MsoNormal><span style=3D'color:#1F497D'>2013-07-31 14:16:49,209
INFO =
[org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] =
(ajp--127.0.0.1-8702-2) [74556c31] START, ConnectStorageServerVDSCommand(Ho=
stName =3D deovn-a04, HostId =3D c6618b0e-e56b-4e38-ae2b-cf4944e6648c, stor=
agePoolId =3D 00000000-0000-0000-0000-000000000000, storageType =3D NFS, co=
nnectionList =3D [{ id: e9911b63-a50c-441f-af02-0f7eb5be0312, connection: 1=
72.16.101.95:/de-ovirt-iso, iqn: null, vfsType: null, mountOptions: null, n=
fsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 44b45cc7<o:=
p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-=
07-31 14:16:49,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Co=
nnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [74556c31] FINISH, Co=
nnectStorageServerVDSCommand, return: {e9911b63-a50c-441f-af02-0f7eb5be0312=
=3D0}, log id: 44b45cc7<o:p></o:p></span></p><p
class=3DMsoNormal><span sty=
le=3D'color:#1F497D'>2013-07-31 14:16:49,345 INFO
[org.ovirt.engine.c=
ore.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-2) =
START, GetExistingStorageDomainListQuery(<a href=3D"mailto:org.ovirt.engine=
.core.common.queries.GetExistingStorageDomainListParameters(a)24b89633">org.o=
virt.engine.core.common.queries.GetExistingStorageDomainListParameters@24b8=
9633</a>), log id: 2fe40ba5<o:p></o:p></span></p><p
class=3DMsoNormal><span=
style=3D'color:#1F497D'>2013-07-31 14:16:49,348 INFO
[org.ovirt.engi=
ne.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0=
.0.1-8702-2) START, HSMGetStorageDomainsListVDSCommand(HostName =3D deovn-a=
04, HostId =3D c6618b0e-e56b-4e38-ae2b-cf4944e6648c, storagePoolId=3D000000=
00-0000-0000-0000-000000000000, storageType=3DNFS, storageDomainType=3DISO,=
path=3D172.16.101.95:/de-ovirt-iso), log id:
7e399cee<o:p></o:p></span></p=
<p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-07-31 14:16:51,632=
INFO
[org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomains=
ListVDSCommand] (ajp--127.0.0.1-8702-2) FINISH, HSMGetStorageDomainsListVDS=
Command, return: [], log id: 7e399cee<o:p></o:p></span></p><p
class=3DMsoNo=
rmal><span style=3D'color:#1F497D'>2013-07-31 14:16:51,632 INFO
[org.=
ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.=
0.0.1-8702-2) FINISH, GetExistingStorageDomainListQuery, log id: 2fe40ba5<o=
:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013=
-07-31 14:16:51,687 INFO [org.ovirt.engine.core.bll.storage.RemoveSto=
rageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [51bf485f] Running com=
mand: RemoveStorageServerConnectionCommand internal: false. Entities affect=
ed : ID: aaa00000-0000-0000-0000-123456789aaa Type:
System<o:p></o:p>=
</span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-07-31 14=
:16:51,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.Disconnect=
StorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [51bf485f] START, Disconne=
ctStorageServerVDSCommand(HostName =3D deovn-a04, HostId =3D c6618b0e-e56b-=
4e38-ae2b-cf4944e6648c, storagePoolId =3D 00000000-0000-0000-0000-000000000=
000, storageType =3D NFS, connectionList =3D [{ id: null, connection: 172.1=
6.101.95:/de-ovirt-iso, iqn: null, vfsType: null, mountOptions: null, nfsVe=
rsion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63ebbd84<o:p></=
o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-07-3=
1 14:16:52,436 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp=
--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.<o:p>=
</o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-07=
-31 14:16:52,443 WARN [org.ovirt.engine.core.bll.GetConfigurationValu=
eQuery] (ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (Applica=
tionMode) with null version, using default general for
version<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>2013-07-31 14:1=
6:52,444 WARN [org.ovirt.engine.core.bll.GetConfigurationValueQuery] =
(ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (VdcVersion) wit=
h null version, using default general for
version<o:p></o:p></span></p><p c=
lass=3DMsoNormal><span style=3D'color:#1F497D'>2013-07-31 14:16:53,955
INFO=
[org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVD=
SCommand] (ajp--127.0.0.1-8702-2) [51bf485f] FINISH, DisconnectStorageServe=
rVDSCommand, return: {00000000-0000-0000-0000-000000000000=3D0}, log id: 63=
ebbd84<o:p></o:p></span></p><p class=3DMsoNormal
style=3D'margin-bottom:12.=
0pt'><span style=3D'color:#1F497D'><br><u>snippet from
vdsm log affected hy=
pervisor:<o:p></o:p></u></span></p><p
class=3DMsoNormal><span style=3D'colo=
r:#1F497D'>[root@deovn-a04 data-center]# tail /var/log/vdsm/vdsm.log =
-n 1000 | grep 172.16.101.95<o:p></o:p></span></p><p
class=3DMsoNormal><spa=
n style=3D'color:#1F497D'>Thread-1757::DEBUG::2013-07-31 14:38:49,531::misc=
::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /usr/bin/moun=
t -t nfs -o soft,nosharecache,timeo=3D600,retrans=3D6,nfsvers=3D3 172.16.10=
1.95:/de-ovirt-iso /rhev/data-center/mnt/172.16.101.95:_de-ovirt-iso' (cwd =
None)<br>Thread-1767::INFO::2013-07-31 14:38:52,021::logUtils::41::dispatch=
er::(wrapper) Run and protect: disconnectStorageServer(domType=3D1, spUUID=
=3D'00000000-0000-0000-0000-000000000000', conList=3D[{'connection':
'172.1=
6.101.95:/de-ovirt-iso', 'iqn': '', 'portal': '',
'user': '', 'password': '=
******', 'id': '00000000-0000-0000-0000-000000000000', 'port':
''}], option=
s=3DNone)<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1=
F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal style=3D'margin-bot=
tom:12.0pt'><span
style=3D'color:#1F497D'><o:p> </o:p></span></p><p
cl=
ass=3DMsoNormal style=3D'margin-bottom:12.0pt'><span
style=3D'color:#1F497D=
'>As the mount is working under console and <u>I can’t see an
obvious=
error in the engine/vdsm log</u> I’m a bit confused where to look at=
here…<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'color=
:#1F497D'>I’m running a POSIXFS storage domain, I tried to mount NFS =
iso/export as “data” storage, both were working fine! (Detached=
NAS mounts from oVirt and cleaned created content from engine after testin=
g)<o:p></o:p></span></p><p class=3DMsoNormal
style=3D'margin-bottom:12.0pt'=
<span style=3D'color:#1F497D'>So, it seems
there’s a difference betw=
een mounting NFS mountpoints via
PosixFS and ISO/Export?<o:p></o:p></=
span></p><p class=3DMsoNormal><span
style=3D'color:#1F497D'>Thanks for any =
suggests!<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:#1=
F497D'><o:p> </o:p></span></p><p
class=3DMsoNormal><span style=3D'colo=
r:#1F497D'>Best
regards,<br>Sven.<o:p></o:p></span></p><p
class=3DMsoNormal=
<span
style=3D'color:#1F497D'><o:p> </o:p></span></p><div><p
class=3D=
MsoNormal><span style=3D'color:#5A5A5A'>Sven Knohsalla
</span><span style=
=3D'color:#C00000'>|</span><span style=3D'color:#5A5A5A'>
System Administra=
tion | Netbiscuits<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D=
'color:gray'><o:p> </o:p></span></p><p
class=3DMsoNormal><span style=
=3D'font-size:9.0pt;color:#5A5A5A'>Office +49 631 68036 433 | Fax +49 631 6=
8036 111 |E-Mail</span><span
style=3D'font-size:9.0pt;color:#505050'>=
</span><span style=3D'font-size:9.0pt;color:#0070C0'><a
href=3D"mailto:s.k=
nohsalla@netbiscuits.com">s.knohsalla(a)netbiscuits.com</a></span><span
style=
=3D'font-size:9.0pt;color:#5A5A5A'> | Skype: </span><span
style=3D'font-siz=
e:9.0pt;color:#0070C0'>netbiscuits.admin</span><span
style=3D'font-size:9.0=
pt;color:#5A5A5A'> <o:p></o:p></span></p><p
class=3DMsoNormal><span lang=3D=
DE style=3D'font-size:9.0pt;color:#5A5A5A'>Netbiscuits GmbH | Europaallee 1=
0 | 67657 | GERMANY<o:p></o:p></span></p></div><p
class=3DMsoNormal><span s=
tyle=3D'color:#1F497D'><o:p> </o:p></span></p><div><div
style=3D'borde=
r:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=
=3DMsoNormal><b><span lang=3DDE
style=3D'font-size:10.0pt;font-family:"Taho=
ma","sans-serif"'>Von:</span></b><span lang=3DDE
style=3D'font-size:10.0pt;=
font-family:"Tahoma","sans-serif"'> Sven Knohsalla
<br><b>Gesendet:</b> Mit=
twoch, 31. Juli 2013 10:45<br><b>An:</b> <a
href=3D"mailto:users@ovirt.org"=
users@ovirt.org</a><br><b>Cc:</b> Sven
Knohsalla (<a href=3D"mailto:s.knoh=
salla@netbiscuits.com">s.knohsalla@netbiscuits.com</a>)<br><b>Betreff:</b>
=
oVirt 3.2.1 /F18 node vdsm
issue<o:p></o:p></span></p></div></div><p class=
=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal
style=3D'margin-bott=
om:12.0pt'>Hi,<br><br>I just set up a vanilla oVirt 3.2.1 environment,
cont=
aining <o:p></o:p></p><p class=3DMsoNormal>oVirt engine
<br>3.2.1 F18 based=
<span class=3Dgwt-inlinelabel>oVirt Engine Version:</span> <span
class=3Dg=
wt-inlinelabel>3.2.1-1.fc18</span><br><br>F18 minimal node
<br>vdsm-4.10.3-=
10.fc18 libvirt-0.10.2.6-1.fc18 1.2.2 - 13.fc18 K ernel 3.9.11 - 200.fc18.x=
86_64<br><br>gluster-storage, replicated, EL6<br>glusterfs
3.4.0<br><br>F18=
minimal node was installed via oVirt engine, worked fine, expect vdsmd was=
n’t started<br>(node non-responsive for
engine)<br><br>Manually start=
ing vdsm works (HV up in engine), autostart of vdsm (during boot) fails:<br=
<br>Jul 31 10:04:47 deovn-a04 systemd-vdsmd[711]: Note:
Forwarding request=
to 'systemctl disable
libvirt-guests.service'.<o:p></o:p></p><p class=3DMs=
oNormal>Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: vdsm: libvirt already=
configured for vdsm [ OK ]<o:p></o:p></p><p
class=3DMsoNormal>=
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting
ntpd...<o:p></o:p></=
p><p class=3DMsoNormal>Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redire=
cting to /bin/systemctl start ntpd.service<o:p></o:p></p><p
class=3DM=
soNormal>Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting wdmd...<o:p=
</o:p></p><p class=3DMsoNormal>Jul 31 10:04:48
deovn-a04 systemd-vdsmd[711=
]: Redirecting to /bin/systemctl start
wdmd.service<o:p></o:p></p><p =
class=3DMsoNormal>Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting sa=
nlock...<o:p></o:p></p><p class=3DMsoNormal>Jul 31 10:04:48
deovn-a04 syste=
md-vdsmd[711]: Redirecting to /bin/systemctl start sanlock.service<o:=
p></o:p></p><p class=3DMsoNormal>Jul 31 10:04:48 deovn-a04
systemd-vdsmd[71=
1]: Starting iscsid:<o:p></o:p></p><p class=3DMsoNormal>Jul 31
10:04:48 deo=
vn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl start libvir=
td.service<o:p></o:p></p><p class=3DMsoNormal>Jul 31 10:06:17
deovn-a04 sys=
temd[1]: vdsmd.service operation timed out.
Terminating.<o:p></o:p></p><p c=
lass=3DMsoNormal>Jul 31 10:06:17 deovn-a04 systemd[1]: Unit vdsmd.service e=
ntered failed state.<br><br>Did find similar issues:<br><a
href=3D"https://=
bugzilla.redhat.com/show_bug.cgi?id=3D832199">https://bugzilla.re...
how_bug.cgi?id=3D832199</a><br><a
href=3D"https://bugzilla.redhat.com/show_=
bug.cgi?id=3D720359">https://bugzilla.redhat.com/show_bug.cgi?id=...
a><o:p></o:p></p><p class=3DMsoNormal><a
href=3D"http://lists.ovirt.org/pip=
ermail/users/2012-September/009624.html">http://lists.ovirt.org/p...
sers/2012-September/009624.html</a><o:p></o:p></p><p
class=3DMsoNormal><o:p=
</o:p></p><p
class=3DMsoNormal>Unfortunately, there’s no hint.=
. libvirtd starts
without any latency when manually triggered.<br><br>The s=
trange thing here: F18 node deploy worked with engine 3.2.0 3-4 weeks ago.<=
o:p></o:p></p><p class=3DMsoNormal>Did update engine /
F18.<o:p></o:p></p><=
p class=3DMsoNormal><br>Can anybody give me an advice
here?<o:p></o:p></p><=
p class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal
style=3D'marg=
in-bottom:12.0pt'>Thanks in advance!<o:p></o:p></p><p
class=3DMsoNormal>Bes=
t,<br>Sven<o:p></o:p></p><p
class=3DMsoNormal><o:p> </o:p></p><p class=
=3DMsoNormal><span style=3D'color:#5A5A5A'>Sven Knohsalla
</span><span styl=
e=3D'color:#C00000'>|</span><span style=3D'color:#5A5A5A'>
System Administr=
ation<o:p></o:p></span></p><p class=3DMsoNormal><span
style=3D'color:gray'>=
<o:p> </o:p></span></p><p
class=3DMsoNormal><span style=3D'font-size:9=
.0pt;color:#5A5A5A'>Office +49 631 68036 433 | Fax +49 631 68036 111 =
|E-Mail</span><span style=3D'font-size:9.0pt;color:#505050'>
</span><span s=
tyle=3D'font-size:9.0pt;color:#0070C0'><a
href=3D"mailto:s.knohsalla@netbis=
cuits.com">s.knohsalla(a)netbiscuits.com</a></span><span
style=3D'font-size:9=
.0pt;color:#5A5A5A'> | Skype: </span><span
style=3D'font-size:9.0pt;color:#=
0070C0'>netbiscuits.admin</span><span
style=3D'font-size:9.0pt;color:#5A5A5=
A'> <o:p></o:p></span></p><p
class=3DMsoNormal><span lang=3DDE style=3D'fon=
t-size:9.0pt;color:#5A5A5A'>Netbiscuits GmbH | Europaallee 10 | 67657 | GER=
MANY<o:p></o:p></span></p><p class=3DMsoNormal><span
lang=3DDE style=3D'fon=
t-size:9.0pt;color:#5A5A5A'><o:p> </o:p></span></p><p
class=3DMsoNorma=
l><a href=3D"http://www.netbiscuits.com/"><span
style=3D'text-decoration:no=
ne'><img border=3D0 width=3D231 height=3D38 id=3D"_x0000_i1025"
src=3D"http=
s://my.netbiscuits.com/image/image_gallery?uuid=3D3a1a9d19-c305-4032-8cef-0=
0b03c3d4c79&groupId=3D10211&t=3D1361534926402"
alt=3D"https://my.ne=
tbiscuits.com/image/image_gallery?uuid=3D3a1a9d19-c305-4032-8cef-00b03c3d4c=
79&groupId=3D10211&t=3D1361534926402"></span></a><o:p></o:p></p><p
=
class=3DMsoNormal><o:p> </o:p></p><p class=3DMsoNormal
style=3D'margin=
-bottom:12.0pt'><span
style=3D'color:black'> </span><a href=3D"https:/=
/www.netbiscuits.com/news"><span
style=3D'color:windowtext;text-decoration:=
none'><img border=3D0 width=3D24 height=3D24 id=3D"_x0000_i1026"
src=3D"htt=
ps://my.netbiscuits.com/image/image_gallery?uuid=3D9e553e7b-3e7d-4784-b274-=
15aa1dfb48e2&groupId=3D10211&t=3D1361533377340"
alt=3D"https://my.n=
etbiscuits.com/image/image_gallery?uuid=3D3031deca-7e56-4417-9822-3d6d72f71=
ef0&groupId=3D10211&t=3D1347280983812"></span></a><span
style=3D'co=
lor:black'> </span><a
href=3D"https://twitter.com/netbiscuits"><span s=
tyle=3D'color:windowtext;text-decoration:none'><img border=3D0 width=3D24
h=
eight=3D24 id=3D"_x0000_i1027"
src=3D"https://my.netbiscuits.com/image/imag=
e_gallery?uuid=3D1d1a5e29-ceda-4ab1-9353-67a1e838364d&groupId=3D10211&a=
mp;t=3D1347281040591" alt=3D"https://my.netbiscuits.com/image/image_gallery=
?uuid=3D1d1a5e29-ceda-4ab1-9353-67a1e838364d&groupId=3D10211&t=3D13=
47281040591"></span></a><span
style=3D'color:black'> </span><a href=3D=
"http://www.linkedin.com/company/netbiscuits"><span
style=3D'color:windowte=
xt;text-decoration:none'><img border=3D0 width=3D24 height=3D24
id=3D"_x000=
0_i1028" src=3D"https://my.netbiscuits.com/image/image_gallery?uuid=3Dc99bf=
866-be25-4236-a0ee-dca68ec828a5&groupId=3D10211&t=3D1347280983848" =
alt=3D"https://my.netbiscuits.com/image/image_gallery?uuid=3Dc99bf86...
4236-a0ee-dca68ec828a5&groupId=3D10211&t=3D1347280983848"></span></=
a><span style=3D'color:black'> </span><a
href=3D"http://www.xing.com/c=
ompanies/netbiscuitsgmbh"><span
style=3D'color:windowtext;text-decoration:n=
one'><img border=3D0 width=3D24 height=3D24 id=3D"_x0000_i1029"
src=3D"http=
s://my.netbiscuits.com/image/image_gallery?uuid=3Dd62ba951-14dc-450d-b5f1-b=
e33884225e3&groupId=3D10211&t=3D1347280983872"
alt=3D"https://my.ne=
tbiscuits.com/image/image_gallery?uuid=3Dd62ba951-14dc-450d-b5f1-be33884225=
e3&groupId=3D10211&t=3D1347280983872"></span></a><span
style=3D'col=
or:black'> </span><a
href=3D"https://www.facebook.com/Netbiscuits"><sp=
an style=3D'color:windowtext;text-decoration:none'><img border=3D0
width=3D=
24 height=3D24 id=3D"_x0000_i1030"
src=3D"https://my.netbiscuits.com/image/=
image_gallery?uuid=3D7b28f500-f415-40bb-851f-0cd55beeaf45&groupId=3D102=
11&t=3D1347280983791"
alt=3D"https://my.netbiscuits.com/image/image_gal=
lery?uuid=3D7b28f500-f415-40bb-851f-0cd55beeaf45&groupId=3D10211&t=
=3D1347280983791"></span></a><span
style=3D'color:black'> </span><a hr=
ef=3D"https://plus.google.com/u/0/112410769451962733032"><...
style=3D'col=
or:windowtext;text-decoration:none'><img border=3D0 width=3D24 height=3D24 =
id=3D"_x0000_i1031"
src=3D"https://my.netbiscuits.com/image/image_gallery?u=
uid=3Dcc8764d0-a5ac-4623-bb63-da3ca7c97f94&groupId=3D10211&t=3D1347=
280983836" alt=3D"https://my.netbiscuits.com/image/image_gallery?uuid=3Dcc8=
764d0-a5ac-4623-bb63-da3ca7c97f94&groupId=3D10211&t=3D1347280983836=
"></span></a><span
style=3D'color:black'> </span><a href=3D"http://www=
.youtube.com/user/netbiscuits"><span
style=3D'color:windowtext;text-decorat=
ion:none'><img border=3D0 width=3D24 height=3D24 id=3D"_x0000_i1032"
src=3D=
"https://my.netbiscuits.com/image/image_gallery?uuid=3Da15e871c-a11b-419c-a=
cca-da5a0ebd5856&groupId=3D10211&t=3D1347281040599"
alt=3D"https://=
my.netbiscuits.com/image/image_gallery?uuid=3Da15e871c-a11b-419c-acca-da5a0=
ebd5856&groupId=3D10211&t=3D1347281040599"></span></a><o:p></o:p></=
p><p class=3DMsoNormal><b><span
style=3D'font-size:9.0pt;color:#5A5A5A'>Reg=
ister Court:</span></b><span
style=3D'font-size:9.0pt;color:#5A5A5A'> Local=
Court Kaiserslautern | <b>Commercial Register ID:</b> HR B
3604<br><b>Mana=
gement Board</b>: Guido Moggert, Michael Neidh=F6fer, Christian Reitz, Mart=
in S=FC=DF<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-si=
ze:9.0pt;color:#5A5A5A'><o:p> </o:p></span></p><p
class=3DMsoNormal><s=
pan style=3D'font-size:9.0pt;color:#5A5A5A'>This message and any files tran=
smitted with it are confidential and intended solely for the use of the ind=
ividual or entity to whom they are addressed. It may also be privileged or =
otherwise protected by work product immunity or other legal rules. Please n=
otify the sender immediately by e-mail if you have received this e-mail by =
mistake and delete this e-mail from your system. If you are not the intende=
d recipient you are notified that disclosing, copying, distributing or taki=
ng any action in reliance on the contents of this information is strictly p=
rohibited.<o:p></o:p></span></p><p
class=3DMsoNormal><span style=3D'font-si=
ze:9.0pt;color:#5A5A5A'>Warning: Although Netbiscuits has taken reasonable =
precautions to ensure no viruses are present in this email, the company can=
not accept responsibility for any loss or damage arising from the use of th=
is email or attachments.<o:p></o:p></span></p><p
class=3DMsoNormal><span st=
yle=3D'font-size:9.0pt;color:#5A5A5A'><o:p> </o:p></span></p><p
class=
=3DMsoNormal><span style=3D'font-size:9.0pt;color:#5A5A5A'><img
border=3D0 =
width=3D33 height=3D19 id=3D"_x0000_i1033"
src=3D"http://www.netbiscuits.co=
m/image/image_gallery?uuid=3D0ba7711a-a277-4ea0-acb0-17fe13c3089d&group=
Id=3D10211&t=3D1348560850164" alt=3D"Beschreibung: Beschreibung:
http:/=
/www.netbiscuits.com/image/image_gallery?uuid=3D0ba7711a-a277-4ea0-acb0-17f=
e13c3089d&groupId=3D10211&t=3D1348560850164"></span><span
style=3D'=
font-size:9.0pt;color:#6CA62C'>Please consider the environment before print=
ing<o:p></o:p></span></p><p
class=3DMsoNormal><o:p> </o:p></p></div></=
body></html>=
--_000_DC9ABD90CB55EC448B1F682D5731AF7407A78DB15FIAD2MBX17mex0_--