[Users] oVirt 3.2.1 adding NFS ISO/Export fails

Wed Jul 31 12:54:06 UTC 2013

Hi,

did anybody run into the issue following issue under oVirt 3.2.1 + F18 based nodes ?
-->

NFS ISO/Export mount via webgui gives error message "There is no storage domain under the specified path. Check event log for more details."
(no advanced mount parameters given)

Mounting NFS volume under console is working fine, no special mount options given, NFS server is a QNAP NAS which did work under oVirt 3.0 without any problems (disks cleaned!).
Created a NFS mount point "de-ovirt-iso" for testing NFS mount /ISO&Export functionality under oVirt 3.2.1.

Watching the mount process on console initiated from the WebGui, shows a correct mounted ISO/Export domain for ~1 sec.
under /rhev/..

Then, oVirt engine triggers a disconnection (Task: Disconnecting from Storage server on <hypervisor>)

engine log snippet when performing NFS mount:

2013-07-31 14:16:49,204 INFO  [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [74556c31] Running command: AddStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-07-31 14:16:49,209 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [74556c31] START, ConnectStorageServerVDSCommand(HostName = deovn-a04, HostId = c6618b0e-e56b-4e38-ae2b-cf4944e6648c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: e9911b63-a50c-441f-af02-0f7eb5be0312, connection: 172.16.101.95:/de-ovirt-iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 44b45cc7
2013-07-31 14:16:49,309 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [74556c31] FINISH, ConnectStorageServerVDSCommand, return: {e9911b63-a50c-441f-af02-0f7eb5be0312=0}, log id: 44b45cc7
2013-07-31 14:16:49,345 INFO  [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-2) START, GetExistingStorageDomainListQuery(org.ovirt.engine.core.common.queries.GetExistingStorageDomainListParameters at 24b89633), log id: 2fe40ba5
2013-07-31 14:16:49,348 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-2) START, HSMGetStorageDomainsListVDSCommand(HostName = deovn-a04, HostId = c6618b0e-e56b-4e38-ae2b-cf4944e6648c, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ISO, path=172.16.101.95:/de-ovirt-iso), log id: 7e399cee
2013-07-31 14:16:51,632 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp--127.0.0.1-8702-2) FINISH, HSMGetStorageDomainsListVDSCommand, return: [], log id: 7e399cee
2013-07-31 14:16:51,632 INFO  [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp--127.0.0.1-8702-2) FINISH, GetExistingStorageDomainListQuery, log id: 2fe40ba5
2013-07-31 14:16:51,687 INFO  [org.ovirt.engine.core.bll.storage.RemoveStorageServerConnectionCommand] (ajp--127.0.0.1-8702-2) [51bf485f] Running command: RemoveStorageServerConnectionCommand internal: false. Entities affected :  ID: aaa00000-0000-0000-0000-123456789aaa Type: System
2013-07-31 14:16:51,690 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [51bf485f] START, DisconnectStorageServerVDSCommand(HostName = deovn-a04, HostId = c6618b0e-e56b-4e38-ae2b-cf4944e6648c, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 172.16.101.95:/de-ovirt-iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63ebbd84
2013-07-31 14:16:52,436 INFO  [org.ovirt.engine.core.bll.LoginUserCommand] (ajp--127.0.0.1-8702-7) Running command: LoginUserCommand internal: false.
2013-07-31 14:16:52,443 WARN  [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (ApplicationMode) with null version, using default general for version
2013-07-31 14:16:52,444 WARN  [org.ovirt.engine.core.bll.GetConfigurationValueQuery] (ajp--127.0.0.1-8702-7) calling GetConfigurationValueQuery (VdcVersion) with null version, using default general for version
2013-07-31 14:16:53,955 INFO  [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp--127.0.0.1-8702-2) [51bf485f] FINISH, DisconnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 63ebbd84

snippet from vdsm log affected hypervisor:

[root at deovn-a04 data-center]# tail  /var/log/vdsm/vdsm.log -n 1000 | grep 172.16.101.95
Thread-1757::DEBUG::2013-07-31 14:38:49,531::misc::84::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /usr/bin/mount -t nfs -o soft,nosharecache,timeo=600,retrans=6,nfsvers=3 172.16.101.95:/de-ovirt-iso /rhev/data-center/mnt/172.16.101.95:_de-ovirt-iso' (cwd None)
Thread-1767::INFO::2013-07-31 14:38:52,021::logUtils::41::dispatcher::(wrapper) Run and protect: disconnectStorageServer(domType=1, spUUID='00000000-0000-0000-0000-000000000000', conList=[{'connection': '172.16.101.95:/de-ovirt-iso', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '00000000-0000-0000-0000-000000000000', 'port': ''}], options=None)

As the mount is working under console and I can't see an obvious error in the engine/vdsm log I'm a bit confused where to look at here...

I'm running a POSIXFS storage domain, I tried to mount NFS iso/export as "data" storage, both were working fine! (Detached NAS mounts from oVirt and cleaned created content from engine after testing)
So, it seems there's a difference between mounting NFS  mountpoints via PosixFS and ISO/Export?

Thanks for any suggests!

Best regards,
Sven.

Sven Knohsalla | System Administration | Netbiscuits

Office +49 631 68036 433 | Fax +49 631 68036 111  |E-Mail s.knohsalla at netbiscuits.com | Skype: netbiscuits.admin
Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY

Von: Sven Knohsalla
Gesendet: Mittwoch, 31. Juli 2013 10:45
An: users at ovirt.org
Cc: Sven Knohsalla (s.knohsalla at netbiscuits.com)
Betreff: oVirt 3.2.1 /F18 node vdsm issue

Hi,

I just set up a vanilla oVirt 3.2.1 environment, containing
oVirt engine
3.2.1 F18 based oVirt Engine Version: 3.2.1-1.fc18

F18 minimal node
vdsm-4.10.3-10.fc18 libvirt-0.10.2.6-1.fc18 1.2.2 - 13.fc18 K ernel 3.9.11 - 200.fc18.x86_64

gluster-storage, replicated, EL6
glusterfs 3.4.0

F18 minimal node was installed via oVirt engine, worked fine, expect vdsmd wasn't started
(node non-responsive for engine)

Manually starting vdsm works (HV up in engine), autostart of vdsm (during boot) fails:

Jul 31 10:04:47 deovn-a04 systemd-vdsmd[711]: Note: Forwarding request to 'systemctl disable libvirt-guests.service'.
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: vdsm: libvirt already configured for vdsm [  OK  ]
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting ntpd...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl start  ntpd.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting wdmd...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl start  wdmd.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting sanlock...
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl start  sanlock.service
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Starting iscsid:
Jul 31 10:04:48 deovn-a04 systemd-vdsmd[711]: Redirecting to /bin/systemctl start  libvirtd.service
Jul 31 10:06:17 deovn-a04 systemd[1]: vdsmd.service operation timed out. Terminating.
Jul 31 10:06:17 deovn-a04 systemd[1]: Unit vdsmd.service entered failed state.

Did find similar issues:
https://bugzilla.redhat.com/show_bug.cgi?id=832199
https://bugzilla.redhat.com/show_bug.cgi?id=720359
http://lists.ovirt.org/pipermail/users/2012-September/009624.html

Unfortunately, there's no hint.. libvirtd starts without any latency when manually triggered.

The strange thing here: F18 node deploy worked with engine 3.2.0 3-4 weeks ago.
Did update engine / F18.

Can anybody give me an advice here?

Thanks in advance!
Best,
Sven

Sven Knohsalla | System Administration

Office +49 631 68036 433 | Fax +49 631 68036 111  |E-Mail s.knohsalla at netbiscuits.com<mailto:s.knohsalla at netbiscuits.com> | Skype: netbiscuits.admin
Netbiscuits GmbH | Europaallee 10 | 67657 | GERMANY

[https://my.netbiscuits.com/image/image_gallery?uuid=3a1a9d19-c305-4032-8cef-00b03c3d4c79&groupId=10211&t=1361534926402]<http://www.netbiscuits.com/>

 [https://my.netbiscuits.com/image/image_gallery?uuid=9e553e7b-3e7d-4784-b274-15aa1dfb48e2&groupId=10211&t=1361533377340] <https://www.netbiscuits.com/news>  [https://my.netbiscuits.com/image/image_gallery?uuid=1d1a5e29-ceda-4ab1-9353-67a1e838364d&groupId=10211&t=1347281040591] <https://twitter.com/netbiscuits>  [https://my.netbiscuits.com/image/image_gallery?uuid=c99bf866-be25-4236-a0ee-dca68ec828a5&groupId=10211&t=1347280983848] <http://www.linkedin.com/company/netbiscuits>  [https://my.netbiscuits.com/image/image_gallery?uuid=d62ba951-14dc-450d-b5f1-be33884225e3&groupId=10211&t=1347280983872] <http://www.xing.com/companies/netbiscuitsgmbh>  [https://my.netbiscuits.com/image/image_gallery?uuid=7b28f500-f415-40bb-851f-0cd55beeaf45&groupId=10211&t=1347280983791] <https://www.facebook.com/Netbiscuits>  [https://my.netbiscuits.com/image/image_gallery?uuid=cc8764d0-a5ac-4623-bb63-da3ca7c97f94&groupId=10211&t=1347280983836] <https://plus.google.com/u/0/112410769451962733032>  [https://my.netbiscuits.com/image/image_gallery?uuid=a15e871c-a11b-419c-acca-da5a0ebd5856&groupId=10211&t=1347281040599] <http://www.youtube.com/user/netbiscuits>
Register Court: Local Court Kaiserslautern | Commercial Register ID: HR B 3604
Management Board: Guido Moggert, Michael Neidhöfer, Christian Reitz, Martin Süß

This message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. It may also be privileged or otherwise protected by work product immunity or other legal rules. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.
Warning: Although Netbiscuits has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.

[http://www.netbiscuits.com/image/image_gallery?uuid=0ba7711a-a277-4ea0-acb0-17fe13c3089d&groupId=10211&t=1348560850164]Please consider the environment before printing

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ovirt.org/pipermail/users/attachments/20130731/42a45fbf/attachment-0001.html>