- Users - oVirt List Archives

[Users] Hosts going offline due to storage issues.
by Ian Mordey 17 Dec '13

17 Dec '13

Hi there I've got a 3 node oVirt 3.3.1 setup connecting to an Equallogic iSCSI SAN array. My nodes keep going offline with these errors: Host reports about one of the Active Storage Domains as Problematic. Host cannot access one of the Storage Domains attached to the Data Center <DC> Setting Host state to Non-Operational The host is then fenced, reboots and comes back up normally. I see some of these errors prior to the event: 2013-12-10 13:36:49,442 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-14) Command ListVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused 2013-12-10 13:36:49,448 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Failed to refresh VDS , vds = 7f74b28a-149a-4f87-9689-b23db6d435f7 : <hostname> , VDS Network Error, continuing. 2013-12-10 13:36:52,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused Any ideas where to look? I can't see any errors on the switches or SAN.. Thanks Ian -- Listed on the London Stock Exchange: WAND<http://www.bloomberg.com/quote/WAND:LN> THIS MESSAGE AND ANY ATTACHMENTS ARE CONFIDENTIAL, PROPRIETARY, AND MAY BE PRIVILEGED. If this message was misdirected, WANdisco, Inc. and its subsidiaries, ("WANdisco") does not waive any confidentiality or privilege. If you are not the intended recipient, please notify us immediately and destroy the message without disclosing its contents to anyone. Any distribution, use or copying of this e-mail or the information it contains by other than an intended recipient is unauthorized. The views and opinions expressed in this e-mail message are the author's own and may not reflect the views and opinions of WANdisco, unless the author is authorized by WANdisco to express such views or opinions on its behalf. All email sent to or from this address is subject to electronic storage and review by WANdisco. Although WANdisco operates anti-virus programs, it does not accept responsibility for any damage whatsoever caused by viruses being passed.

2 1

[Users] AcquireHostId problem
by Pascal Jakobi 17 Dec '13

17 Dec '13

Context : vdsm-4.13.0-11.fc19.x86_64 sanlock-2.8-1.fc19.x86_64 Creating storage domain (NFS) fails with error in engine.log such as Error code AcquireHostIdFailure and error message VDSGenericException: VDSErrorException: Failed to CreateStorage And Cannot acquire host id: ('b4f05e3e-d714-45c4-adf9-8d269418338f', SanlockException(-203, 'Sanlock lockspace add failure', 'Sanlock exception')) Reading the mail archive, I can see that this is a know issue but I do not see the workaround. Can someone enlight me ? Also, is there somewhere a coherent instruction set in order to make NFSv3 work (including iptables settings). I had to stop iptables to unblock myself...

2 1

Re: [Users] Failing migration, inconsistent state
by Martijn Grendelman 17 Dec '13

17 Dec '13

Martijn Grendelman schreef op 5-12-2013 11:00: > Hi, > > I tried to migrate several VMs from one host to another. Two VMs > migrated without issues, but for one VM, the migration didn't happen. It > seems to be hanging, but the UI is now in an inconsistent state: > > - The 'Tasks' tab reports 0 active tasks, but the last task (the > migration in question) is still reported as 'Executing'. > - The VM status is 'Up' (not migrating) > - 'Migrate' action is choosable from menu, while 'Cancel Migration' is > greyed out, but when I choose 'Migrate' and pick a host, I am told > 'Cannot migrate VM. VM <name> is being migrated.'. > > What is the best way to fix this? And perhaps this information is useful to oVirt developers: > Thread-600648::DEBUG::2013-12-03 12:37:47,926::vm::180::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Destination server is: onode0.isaac.local:54321 > Thread-600648::DEBUG::2013-12-03 12:37:47,927::vm::182::vm.Vm::(_setupVdsConnection) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Initiating connection with destination > Thread-600648::DEBUG::2013-12-03 12:37:47,990::vm::232::vm.Vm::(_prepareGuest) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Migration started > Thread-600648::DEBUG::2013-12-03 12:37:48,006::vm::299::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::migration semaphore acquired > Thread-600648::DEBUG::2013-12-03 12:37:48,115::vm::357::vm.Vm::(_startUnderlyingMigration) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::starting migration to qemu+tls://onode0.isaac.local/system with miguri tcp://onode0.isaac.local > Thread-600648::DEBUG::2013-12-03 12:43:10,819::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 9 edom: 10 level: 2 message: operation failed: migration job: unexpectedly failed > Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::742::vm.Vm::(cancel) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::canceling migration downtime thread > Thread-600648::DEBUG::2013-12-03 12:43:10,819::vm::812::vm.Vm::(stop) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::stopping migration monitor thread > Thread-600648::ERROR::2013-12-03 12:43:10,820::vm::238::vm.Vm::(_recover) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::operation failed: migration job: unexpectedly failed > Thread-600648::ERROR::2013-12-03 12:43:11,276::vm::321::vm.Vm::(run) vmId=`0669e3c2-9cfd-4d4e-a0a3-56070902a8c8`::Failed to migrate > Traceback (most recent call last): > File "/usr/share/vdsm/vm.py", line 308, in run > self._startUnderlyingMigration() > File "/usr/share/vdsm/vm.py", line 385, in _startUnderlyingMigration > None, maxBandwidth) > File "/usr/share/vdsm/vm.py", line 835, in f > ret = attr(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper > ret = f(*args, **kwargs) > File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1178, in migrateToURI2 > if ret == -1: raise libvirtError ('virDomainMigrateToURI2() failed', dom=self) > libvirtError: operation failed: migration job: unexpectedly failed Cheers, Martijn.

2 1

[Users] Problem introducing VDSM hosts
by Garry Tiedemann 17 Dec '13

17 Dec '13

Hi folks, We are setting up ovirt on Proliant BL460c blades (mix of G1 & G6), and found an interesting problem. Can anyone here can offer guidance on how to approach this please? Also, I would be interested to know others' experiences of using ovirt on BladeSystem gear. The problem: When trying to introduce a new VDSM host, the install fails, immediately. It's reproducible; we tried more than a dozen times, with different combinations of NICs, to eliminate hardware faults. These messages appear on the VDSM host: Dec 13 10:55:24 bl05 sshd[2627]: pam_unix(sshd:session): session opened for user root by (uid=0) Dec 13 10:55:24 bl05 sshd[2627]: Corrupted MAC on input. Dec 13 10:55:24 bl05 sshd[2627]: Disconnecting: Packet corrupt Dec 13 10:55:24 bl05 sshd[2627]: pam_unix(sshd:session): session closed for user root It seems to be a checksum problem. Google has plenty of similar reports (none ovirt-related), and very little in the way of troubleshooting tips. We have eliminated physical hardware; memory has tested OK, and alternative network ports have been tried, with the same result. I expected to be able to reproduce the issue outside ovirt, by scping large files back and forth, however, this did not cause errors. I believe it's a software matter, possibly TOE, but I don't know how to take it further. We are using FC19 for both the ovirt-engine and vdsm hosts. Any guidance on how to solve this would be much appreciated. Thanks in advance, Garry

2 1

[Users] Keepalived on oVirt Hosts has engine networking issues
by Andrew Lau 17 Dec '13

17 Dec '13

Hi, I have the scenario where I have gluster and ovirt hosts on the same box, to keep the gluster volumes highly available incase a box drops I'm using keepalived across the boxes and using that IP as the means for the storage domain. I documented my setup here in case anyone needs a little more info http://www.andrewklau.com/returning-to-glusterized-ovirt-3-3/ However, the engine seems to be picking up the floating IP assigned to keepalived as the interface and messing with the ovirtmgmt migration network, so migrations are failing as my floating IP gets assigned to the ovirtmgmt bridge in the engine however it's not actually there on most hosts (except one) so vdsm seems to report destination same as source. I've since created a new vlan interface just for storage to avoid the ovirtmgmt conflict, but the engine will still pick up the wrong IP on the storage vlan because of keepalived. This means I can't use the save network feature within the engine as it'll save the floating ip rather than the one already there. Is this a bug or just the way it's designed. eth0.2 -> ovirtmgmt (172.16.0.11) -> management and migration network -> engine sees, sets and saves 172.16.0.11 eth0.3 -> storagenetwork (172.16.1.11) -> gluster network -> engine sees, sets and saves 172.16.1.5 (my floating IP) I hope this makes sense. p.s. can anyone also confirm, does gluster support multi pathing by default? If I'm using this keepalived method, am I bottle necking myself to one host? Thanks, Andrew

4 5

[Users] errors in engine log and tones of sync messages
by Juan Pablo Lorier 16 Dec '13

16 Dec '13

Hi, I'm trying to debug why I can't import a VM to my brand new glusterfs domain and I see that engine.log is full of gluster volumelist calls and less but often, errors of not been able to call async, I'm I missing any package to install in the engine other than vdsm-gluster? Is there a way not to log all the VolumeList commands (without lowering log level) to avoid having so many entries that difficult searching in the log? Regards, : 5c8baf53 2013-12-13 13:27:58,504 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-33) Failed to invoke scheduled method gluster_async_task_poll_event: java.lang.reflect.InvocationTargetException at sun.reflect.GeneratedMethodAccessor108.invoke(Unknown Source) [:1.7.0_45] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_45] at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_45] at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] Caused by: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: org.apache.xmlrpc.XmlRpcException: <type 'exceptions.Exception'>:method "glusterTasksList" is not supported (Failed with error VDS_NETWORK_ERROR and code 5022) at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:122) [bll.jar:] at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.runVdsCommand(GlusterTasksService.java:60) [bll.jar:] at org.ovirt.engine.core.bll.gluster.tasks.GlusterTasksService.getTaskListForCluster(GlusterTasksService.java:28) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateTasksInCluster(GlusterTasksSyncJob.java:67) [bll.jar:] at org.ovirt.engine.core.bll.gluster.GlusterTasksSyncJob.updateGlusterAsyncTasks(GlusterTasksSyncJob.java:56) [bll.jar:] ... 6 more 2013-12-13 13:28:00,655 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) START, GlusterVolumesListVDSCommand(HostName = Dell4, HostId = 001f72ef-fac6-42bc-be8e-b17e249e80bd), log id: 75cbf869 2013-12-13 13:28:00,692 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, GlusterVolumesListVDSCommand, return: {30759593-86cf-47d3-8527-d36623674ea6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ac95f510}, log id: 75cbf869 2013-12-13 13:28:05,743 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) START, GlusterVolumesListVDSCommand(HostName = Dell3, HostId = 342b037f-e4ab-4c3b-b804-806d3ec30ba9), log id: 76386ce 2013-12-13 13:28:07,241 INFO [org.ovirt.engine.core.vdsbroker.gluster.GlusterVolumesListVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, GlusterVolumesListVDSCommand, return: {30759593-86cf-47d3-8527-d36623674ea6=org.ovirt.engine.core.common.businessentities.gluster.GlusterVolumeEntity@ee3bee7d}, log id: 76386ce

4 6

[Users] oVirt 3.3.2 RC
by Sandro Bonazzola 16 Dec '13

16 Dec '13

The oVirt team is pleased to announce that the 3.3.2 Release Candidate is now available in ovirt-updates-testing [1]. We planned to release it on Wed Dec 18th 2013 but it will probably be postponed to next week for allowing more testing on the RC. The release date will be decided in next oVrit sync meeting on Wed Dec 18th. Feel free to join us testing it[2] and verifying the bugzilla entries actually under verification [3]. Release notes for this update are available on the wiki [4]. A new oVirt Node build will be available soon as well. [1] http://resources.ovirt.org/releases/updates-testing [2] http://www.ovirt.org/Testing/Ovirt_3.3.2_testing [3] http://red.ht/19NoJKf [4] http://www.ovirt.org/OVirt_3.3.2_release_notes -- Sandro Bonazzola Better technology. Faster innovation. Powered by community collaboration. See how it works at redhat.com

1 0

[Users] simple networking?
by Ted Miller 16 Dec '13

16 Dec '13

I am trying to set up a testing network using o-virt, but the networking is refusing to cooperate. I am testing for possible use in two different production setups. My previous experience has been with VMWare. I have always set up a single bridged network on each host. All my hosts, VMs, and non-VM computers were peers on the LAN. They could all talk to each other, and things worked very well. There was a firewall/gateway that provided access to the Internet, and hosts, VMs, and could all communicate with the Internet as needed. o-virt seems to be compartmentalizing things beyond all reason. Is there any way to set up simple networking, so ALL computers can see each other? Is there anywhere that describes the philosophy behind the networking setup? What reason is there that networks are so divided? After banging my head against the wall trying to configure just one host, I am very frustrated. I have spent several HOURS Googling for a coherent explanation of how/why networking is supposed to work, but only fine obscure references like "letting non-VMs see VM traffic would be a huge security violation". I have no concept of what king of an installation the o-virt designers have in mind, but it is obviously worlds different from what I am trying to do. The best I can tell, o-virt networking works like this (at least when you have only one NIC): there must be an ovirtmgt network, which cannot be combined with any other network. the ovirtmgt network cannot talk to VMs (unless that VM is running the engine) the ovirtmgt network can only talk to hosts, not to other non-VM computers a VM network can talk only to VMs cannot talk to hosts cannot talk to non-VMs hosts cannot talk to my LAN hosts cannot talk to VMs VMs cannot talk to my LAN All of the above are enforced by a boatload of firewall rules that o-virt puts into every host and VM under its jurisdiction. All of the above is inferred from things I Googled, because I can't find anywhere that explains what or how things are supposed to work--only things telling people WHAT THEY CANT DO. All I see on the mailing lists is people getting their hands slapped because they are trying to do SIMPLE SETUPS that should work, but don't (due to either design restrictions or software bugs). My use case A: * My (2 or 3) hosts have only one physical NIC. * My VMs exist to provide services to non-VM computers. * The VMs do not run X-windows, but they provide GUI programs to non-VMs via "ssh -X" connections. * MY VMs need access to storage that is shared with hosts and non-VMs on the LAN. Is there some way to TURN OFF network control in o-virt? My systems are small and static. I can hand-configure the networking a whole lot easier than I can deal with o-virt (as I have used it so far). Mostly I would need to be able to turn off the firewall rules on both hosts and VMs. banging head against wall, Ted

7 13

[Users] Upgrade from 3.3.1-2 el6 to 3.3.2
by Sven Kieske 16 Dec '13

16 Dec '13

Hi, got a question regarding engine upgrades: My current setup: I got 3.3.1-2.el6 installed from ovirt-beta repo and want to upgrade to 3.3.2 RC. According to: http://www.ovirt.org/OVirt_3.3.2_release_notes "oVirt 3.3.2 release candidate is available for testing. In order to install it you've to enable ovirt-updates-testing repository." I need ovirt-updates-testing enabled? Shall I disable ovirt-beta repo? Additional info: the base os is still CentOS 6.4 Did anyone perform an inplace upgrade to 6.5? Is this recommended? -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen

2 2

[Users] Documentation incoming
by Sven Kieske 16 Dec '13

16 Dec '13

Hi, just for all who are curious: There is a large package of documentation for oVirt at redhat.com: https://access.redhat.com/site/documentation/Red_Hat_Enterprise_Virtualizat… There is also beta documentation for 3.3. This is more structured than the ovirt.org wiki. Maybe this helps someone. PS: In case you do not know: Red Hat Enterprise Virtualization (RHEV for short) is a downstream product build from oVirt, so almost everything in the docs should be transferable to oVirt. -- Mit freundlichen Grüßen / Regards Sven Kieske Systemadministrator Mittwald CM Service GmbH & Co. KG Königsberger Straße 6 32339 Espelkamp T: +49-5772-293-100 F: +49-5772-293-333 https://www.mittwald.de Geschäftsführer: Robert Meyer St.Nr.: 331/5721/1033, USt-IdNr.: DE814773217, HRA 6640, AG Bad Oeynhausen Komplementärin: Robert Meyer Verwaltungs GmbH, HRB 13260, AG Bad Oeynhausen

1 0